Attachement-Bug exeeds CPU-Limit

[turbo]

We mailed, assigned tickets and posted in the forum for 10 days now. And nothing ...

The Attachement-Bug is known since october 2005 (look at http://bugs.kayako.net/?do=details&id=919) and still no solution.

If you get only one mail with a big attachement, no more mails will be parsed...
Hey, isn't it a big bug?

[AKL-MFCU]

i can't replicate it- is your php.ini settings allowing the large file? also make sure the sql database doesn't have limitations. The only other thing i can think of is that the e-mail account becomes "full" based on size so you actually don't receive any more. However, since im not equipped with all the details, i'll wait until you respond.

[turbo]

We did some steps to find the problem. So we figured out, that it was not the php.ini or any other adjustment beside the cpu-limit.

1. Step: Hiren looked into the administration, server and the php-settings
2. Step: We did some cpu measuring while the mail-parser runs
3. Step: Looking for bugs in the kayako source and logging the process

So we found that the mail-parser-cpu-time-waste is nearly linear to the attachementsize.

[AKL-MFCU]

have you tried the difference between the files directory and the database for uploads in e-mail size? or is it the ticket size itself (i was assuming attachments were the cause of the large file)

[turbo]

Originally Posted by AKL-MFCU

have you tried the difference between the files directory and the database for uploads in e-mail size? or is it the ticket size itself (i was assuming attachments were the cause of the large file)

Yes, we tried it ourself an Hiren made some changes too, until he decided to forward our problem to some developer. Afterwards we heard nothing.

The problem is the attachement of abaout 8MB, since the whole attachements are parsed for a ticket-ID.

[eiden]

Originally Posted by Mark [ger]

... well this issue is still open and Kayako did not reply for weeks. Actually they are obviously not interested in giving support.

Mark

You are not giving any useful feedback to this thread by posting this ****! We are all aware of the current support-situation. If you have the need to flamepost, please create your own "I like to repeat what everyone knows"-thread somewhere.

[turbo]

Originally Posted by eiden

You are not giving any useful feedback to this thread by posting this ****! .......

Every post in this thread is useful on showing the importance to the admins.

I can't laugh any more about this, after we heard nothing for some Weeks. No reply on the ticket or the threads. Then the following advises and now nothing for another week.

As Varun wrote, the problem should be the php-memory-limit (which we increased to 500MB without any effort) or we should try piping (which is impossible on our server). We did a logging(1) of the parser-process and found out, that it is definitively the cpu-usage-time which exceeds (not to mix up with the process-time-limit ! ). This means, that the process exceeds 60 real-cpu-seconds.

So, these hints perhaps may be ok on handling the problem, but not solving it. The Bug is still the handling of attachements, as I wrote before.
All attachements are parsed for a ticket id by using regular expressions. Each parsing-process will use 6 RegEx, if the ticketId isn't found.

So I figured out that IMHO this has to be optimized.
1. Only parse html / text (and never images, pdf, etc)
2. Max. 3 RegEx seem to be needed to be executed


(1): Try using command 'top' while logged in via ssh and then execute the mail-parser and look the process-information.

Regards,
Torben

[AKL-MFCU]

heres your hacked solution in the code, go to the cron_parser.php code in the folder (forget exactly where) go down to the point that you see:
PHP Code:

echo "Fetching Msg UID: $_msguid<BR />";
$structd = imap_fetchstructure($mbox, $_msguid, $_uidtype);
$db = $structd->bytes;
$kb = $db/1024;
$mb = $kb/1024;
if ($kb > $_SWIFT["settings"]["pr_sizelimit"]) {
// Reject the Message
$rejected++;

At this point, its doing nothing but sitting there while the other ones are being deleted after they are parsed. While Varun recommends that you increase the size of the mailbox limit (forgot exactly where), you can simply copy the code:
PHP Code:

if ($_account["fetchtype"] == "pop3" || $_account["fetchtype"] == "pop3ssl")
{
imap_delete($mbox, $msgid);
} else {
imap_delete($mbox, $_msguid, $_uidtype);
}

which you see right underneath it on when its successful. Just a solution, although i don't recommend it as the best. It will delete the e-mail if it can't parse it due to size.

[turbo]

Originally Posted by AKL-MFCU

heres your hacked solution in the code, go to the cron_parser.php code in the folder (forget exactly where) go down to the point that you see:
.......
......
which you see right underneath it on when its successful. Just a solution, although i don't recommend it as the best. It will delete the e-mail if it can't parse it due to size.

Hi AKL-MFCU,
thanks for the code. As you say it's a a kind of workaround, but no real solution.
I know that some mails may have some big attachements and thats ok. But IMHO there is a bug in the search-function and it can't be the right solution just to delete these mails.
If I have some time I will do some changes in the source of the ID-parsing-process. Afterwards in worst case there will be 3 regex executed instead of 6. Maybe this will solve the problem AND save the mail.


Regards,
Torben

[AKL-MFCU]

definitely interested in that when you find it. i feel like when i go through their code that it always references other files, and i'm just trying to figure out which to change sometimes. I went in there mainly to try to figure out if i could edit the automated tasks to change that imap retrieval to 1 minute intervals isntead of 10, because i honestly hate using wget on our system to keep bouncing it against the server. Its annoying to see that black window pop up every minute

[turbo]

I did what I said before and seperated the TicketID-Parsing-Process. In my case it works for a mail with about 10MB. Before this mail wasn't processed, now it takes about 57s (realtime) .

I think that this is a way to solve the problem, BUT depends on the machines cpu power and size of the mail.

This solutions just processes max. 3 regular expressions (ereg) instead of 6 like before. So the chance to successfully reach the end of the script grows.

File: helpdesk/modules/parser/functions_parsercore.php
Function: function parseTicketID($subject)

Change the code like :

if ($_SWIFT["settings"]["t_eticketid"] == "random"){
.....if-statements
}
elseif ( $_SWIFT["settings"]["t_eticketid"] == "seq"){
.....if-statements
}

Regards,
Torben

[Mark [ger]]

Originally Posted by eiden

You are not giving any useful feedback to this thread by posting this ****! We are all aware of the current support-situation. If you have the need to flamepost, please create your own "I like to repeat what everyone knows"-thread somewhere.

Hey, you're a cool guy, eh? We bought a software, asked for support and got nothing but nonsense by the support team. So we started to fix it on our own and found a workaround that still doesn't solve the bug but makes it a bit easier to deal with. Kayako is unable to handle this bug - gimme your Helpdesk-E-Mail and I'll send you some large attachments.

So at the moment the software runs... and we'll probably change to another server to have some backup power in case a customer sends a large mail. I do not want to check the inbox every day to make sure that there's no large mail stopping Kayako.

BTW: Having made the fix requires to do it any time a new version of Kayako is released. That's acceptable for OpenSource - but not for commercial software. And even if it's the 100.000th time that someone posts that Kayako's support isn't worth a dime... the way kayako (and Varun) behaved on this bug makes me wondering if they don't want to provide support or if they are unable to do so. We really had the impression they bought the software somewhere and had no idea what could be wrong.

So, may I say that your posting was no contribution to solving this issue, either?

BTW: We didn't even get a feedback from Varun after he got the code hack (he asked for it). So we're not wure whether these guys will fix it in the next release or if we'll have to apply the hack to the next version, too.

Regards

Mark

[AKL-MFCU]

hey mark, was the hacks we provided helpful at all? I just want to know what else we are running into with it, if i can dig anymore into it to fix it. Problem is, i don't think that the 1% of the code the encode is only licensing, because it seems like some of these reference the swift code yet have no other places to edit. Let me know if there is anything in particular you want it to check for and i'll see if i can write my own code into it.

[turbo]

Hi AKL,

as Varun wrote, my hack is useable for handling mails, while always generating TicketIDs the same way. This means, as long as you don't change the method from sequential to random generating of TicketIDs or vice versa, my hack will work. If you change the method and a ticket with another generated TicketID is received (as reply, ...), this ticket won't be handled.
This is the reason why all of the regex have to be done.

So, if you haven't changed the method and do not plan to change the method, my hack will solve the bug.