OK, I figured this one out. Might be a possible bug in the email parser.
There was 3 large emails that were over the email size limit and thus being ignored. However they were not being removed from the inbox so they were parsed every time. While these were in the inbox the alerts were never sent.
After removed these the alerts started up again. It seems the alerts are not sending even if 3 emails go unprocessed while other do. The alerts should at least be sent for emails that were processed which is strange...
This actually brings up an interesting feature request now
