Ignition - Alarm is both active and shelfed at the same time

I got a issue, where I've got an active alarm, which i then shelve.
I have not activated "Shelved" as a dropout condition on the alarm pipeline.
The alarm will have to reside inside the alarm pipeline, as I want it to resume sending out notifications after the shelfing time is done.

After a while, I seem to get a new alarm, with the same name, same tag and it seems to confuse the alarmpipeline, since the notifications keeps going out.

firefox_Apq0yFU7DK

Any idea what might be causing this to happen?
I'm not receiving any clear messages or ack messages.(both clear and ack have pipelines attached)