Large file attachments causing parser to hang

[eger]

I read a lot of threads on this issue. Yet I found no real solution or what even causes the problem.

The issue is that any emails with attachments over a couple megabytes (2+ mB) grind the parser to a halt. New emails don't come in and the parser the hangs forever until something eventually crashes.

php.ini settings:

max_execution_time = 60
max_input_time = 120
memory_limit = 64M
upload_max_filesize = 32M
post_max_size = 32M

mysql my.cnf:

max_allowed_packet = 64M

I am running the parser every 2 minutes from cron so I can't see the output. But when I run the parser URL in my local browser it just eventually times out.

Can anyone shed any light on this? Seems to be a long history issue with no resolution.

I am grabbing emails using POP3. PHP version 4.3.9, Zend Engine v1.3.0, MySQL 4.1.20, Apache 2.0.52.

[craigbrass]

Maybe try increasing he execution time and input time to see if that makes any difference. Doubt it will but worth a try.

[eger]

Yea I've raised them up to 600 seconds. Then what happens is the cron keeps running the parser and eventually... well I don't think I need to explain what happens after a couple hours of that being unnoticed

[BigDawgRob]

These issues can be a right barrel to sort out.
Firstly I'd suggest increasing the parser time to 10 minutes, it can cause problems if one parser kicks off whilst the other is executing.
Secondly try ramping up the memory limit, 64M seems ok though the mail processor is very memory inefficient. Try looking in your php error log file for clues....

[craigbrass]

Hmm, I suggest you submit a ticket about it then. http://members.kayako.net -> "Get Support". If you do get a solution from them, please post it here for other users.

[supportskins]

Originally Posted by eger

I read a lot of threads on this issue. Yet I found no real solution or what even causes the problem.

The issue is that any emails with attachments over a couple megabytes (2+ mB) grind the parser to a halt. New emails don't come in and the parser the hangs forever until something eventually crashes.

php.ini settings:

max_execution_time = 60
max_input_time = 120
memory_limit = 64M
upload_max_filesize = 32M
post_max_size = 32M

mysql my.cnf:

max_allowed_packet = 64M

I am running the parser every 2 minutes from cron so I can't see the output. But when I run the parser URL in my local browser it just eventually times out.

Can anyone shed any light on this? Seems to be a long history issue with no resolution.

I am grabbing emails using POP3. PHP version 4.3.9, Zend Engine v1.3.0, MySQL 4.1.20, Apache 2.0.52.

Which version of Kayako are you using?

[craigbrass]

He is running 3.11.01 according to his previous posts. I looked it up earlier when he first posted.

[eger]

I raised all my limits another 32mb. Now, the parser seems to timeout. However there is no error message in any logs and no emails downloaded or parsed. It simply stops and no emails are parsed. Really troubling... We have had to setup a dummy email client to download emails and leave them on server. But move email messages larger than 1mb to a separate folder so the parser doesn't lock up.

Nobody else experiences this?

[supportskins]

max_execution_time, change it to 600 and increase your cron time from 2 minutes to every 5-8 minutes.

[eger]

I already said I tried this in post #3. I've let them run for up to 20 minutes. The wget never ends and just continuously fills up the process list. If the parser doesn't finish within 2 minutes for one message it doesn't seem like waiting 5 minutes will work...

It shouldn't take more than a couple seconds to parse the actual message of a 5mb file right?

[craigbrass]

I would say it should be a max of 15-20 seconds for that size of file. I would expect a lot lower. I think submitting a ticket is your best option.

[supportskins]

My best guess is the issue is with the PHP configuration or Mail Parser setup. Btw, how many POP3 accounts have you added in the software?

[eger]

I use just one POP3 account to get all the mail then use different queues set to pipe method which take the To: address into the appropriate queue. I am now wondering if it's possible the parser is overloaded because I'm trying to use piping queues to handle the different To: addresses rather than Pre parser rules.. Not sure how this might affect downloading and parsing files though.

[supportskins]

I think you need someone to look into your server as well as PHP configuration. I suggest you create a support ticket from the Kayako Members Area.

[jreed]

FYI, I figured out that the parsing code is really inneficient. When it searches the body of an email for the ticket ID it searches through the raw mime data, which includes all the binary attachments. I changed my code so it only searches the body and does not search attachments. Parsing 6MB emails went from 80 seconds to 4 seconds on my system.

File: /modules/parser/functions_parsercore.php
Original code:

  if (empty($ticketiddata["id"]))
  {
   // We try to look it up in the entire mail
   $ticketiddata = parseTicketID($_email["mimedata"]); // try to look it up in the mail body
   parserDebug("LOOKUP: Found Ticket ID In Body!");
  }

Change to:

  if (empty($ticketiddata["id"]))
  {
   // We try to look it up in the text body
   $ticketiddata = parseTicketID($_email["text"]); // try to look it up in the mail body
   parserDebug("LOOKUP: Searching Ticket # In text Body!");
  }
  if (empty($ticketiddata["id"]))
  {
   // We try to look it up in the html body
   $ticketiddata = parseTicketID($_email["html"]); // try to look it up in the mail body
   parserDebug("LOOKUP: Searching Ticket # In HTML Body!");
  }