Free HTML/Outlook Tickets and Queue Signature Problems - Solved!

[Matthew]

Well, hopefully solved, anyway. Enough to be usable.

Does your helpdesk workflow require you to receive and view emailed posts with HTML formatting? Has it been a real pain getting Kayako to handle HTML tickets well? Have you been to these threads recently...?

• remove double line breaks from signature in palin text message
• Line breaks, <BR> tags, et al.
• Display HTML ticket posts properly (without breaking text-only posts)
• HTML ticket posts from Outlook butchered
• Fix formatting when replying to client ticket from Outlook?
• Incoming HTML mail formatting problem

<Rant>
These problems have been around long enough. Kayako thinks they are merely 'cosmetic' and seems to push them to the backburner. The fact is, issues like this affect how our clients see us--literally. Signature lines that run together, clients that need to submit tables and other HTML formats that end up looking like garbage in the Staff CP and in the client ticket list... all these things make SupportSuite managers look bad in a big way. It's a pain having to spend so many hours pouring over forums and code for things like this. Argghh!
</Rant>

I do not claim that all of these fixes are pretty, or the most efficient way to do things. They work for me; they could work for you too, but don't blame me if you break anything. These fixes are for SupportSuite v3.11.02; I haven't tried them in any version later than that.

The following settings and fixes will allow you to:

• View HTML email posts (including from MS Outlook) in the Staff CP and on the client side, in almost all of their original glory, and without imbedded styles affecting the rest of the page.
• Mouseover ticket previews work too, but with a reduced number of HTML tags to maintain a more consistent look. (Not perfect, but it usually works.)
• Send HTML emails with correctly-formatted queue signatures (no missing line breaks, or having to insert <BR> tags at the end of every line of the signature. Ack!)
• Allow staff replies to ticket alerts via email clients (including Outlook) without the reply losing all it's formatting
• Receive plain text ticket posts without breaking their formatting
• (optional) Filter out imbedded <IMG> tags in HTML posts that might be dangerous, or that just muck of the display with ugly placeholders for inline images
• Turn off HTML emails and just use plain text with one setting change

Backup Your Files
You'll be modifying the following files, so back them up:

1. includes/functions_html.php
2. modules/tickets/functions_ticketcore.php
3. modules/tickets/functions_ticketmain.php
4. modules/tickets/functions_ticketsui.php

Configure Admin CP Settings
Having the correct settings is critical.

Settings->CPU Optimization & Server

• HTML Email: 'Yes'. This turns on outgoing HTML email templates.
• Enable SMTP: 'Yes' (and configure the SMTP options). This is not actually required for proper HTML formatting, but is a 'fix' needed to prevent double line breaks in the signatures of outgoing plain text emails, if you should ever decide to turn that back on. See remove double line breaks from signature in palin text message

Settings->Tickets

• HTML Conversion: 'No Processing'. This is done so that (nearly) all HTML formatting shows up in both the client and staff side (although we are still going to restrict this a bit to avoid some nasty surprises). You might want to be careful about this if your setup allows new tickets to be submitted by email.

Settings->Mail Parser

• E-mail Content Priority: 'HTML'. If there's HTML, we want to process and display it.
• Automatically Convert Encoding Of Incoming E-mails: 'Yes'. Allow character sets to be automatically converted.
• Strip <SCRIPT> Tags: 'Yes'. Prevent nasty code from running in your pretty HTML posts.
• Allow HTML Tags: 'Yes'. This is worth noting: Since we've already set HTML Conversion to 'No Processing' (and not 'Strip Tags') this setting won't affect what we see in ticket view. However, it will affect what the mouseover ticket preview displays, if you have that enabled. If you use the preview, you'll want to have HTML tags enabled to make sure the previews give a reasonable approximation of the full HTML content. The specific tags to be retained are specified in the next setting...
• Valid HTML Tags: '<b><i><u><p><div><hr><strong><em><ol><ul><li><tabl e><tr><td><th><blockquote>'. These are my suggestions. This setting determines which HTML tags are actually allowed to show up in ticket posts (if Strip Tags is enabled) and in the mouseover ticket previews.

Fix for issue #595
This is an issue where plaintext queue signatures get their end-of-line characters stripped when included as part of an HTML mail. See Kayako Bug Tracker - Viewing Issue #595 - Newline stripping in plaintext signatures. What we need to do is make sure that line feeds get replaced by <br> tags for the HTML templates.

In functions_ticketmain.php, locate the following code:

PHP Code:

        $template->assign("queuesignature", $_SWIFT["queuecache"]["list"][$emailqueueid]["contents"]);
        $template->assign("ishtml", true);
        $_templateresulthtml = $template->displayTemplate("email_autoresponder", TEMPLATE_DB);
        $template->assign("ishtml", false);
        $_templateresulttext = $template->displayTemplate("email_autoresponder", TEMPLATE_DB); 


Replace with:

PHP Code:

        // Fix for issue #595 - Newline stripping in plaintext signatures (matthew@xentana.com)
        $queuesignature = $_SWIFT["queuecache"]["list"][$emailqueueid]["contents"]; // get a copy
        $template->assign("ishtml", true);
        $template->assign("queuesignature",preg_replace('#(\n)#s','<br>', $queuesignature)); // add <br> tags for HTML
        $_templateresulthtml = $template->displayTemplate("email_autoresponder", TEMPLATE_DB);
        $template->assign("ishtml", false);
        $template->assign("queuesignature", $queuesignature);
        $_templateresulttext = $template->displayTemplate("email_autoresponder", TEMPLATE_DB); 


Fix for 'Style Creep' Issue
This issue and my fix is mentioned in Display HTML ticket posts properly (without breaking text-only posts). When SupportSuite displays an HTML post originating in Outlook, the surrounding menu styles will be overridden by the styles embedded in the HTML post, causing colored underlines to appear everywhere.

In functions_ticketmain.php, locate the function getTicketPosts, and the following code around line 537:

PHP Code:

if ($nobr == true)
{
    $_posts[$dbCore->Record["ticketpostid"]]["contents"] = $dbCore->Record["contents"];
} else {
    $_posts[$dbCore->Record["ticketpostid"]]["contents"] = $this->returnProcessedContent($dbCore->Record["contents"]);
} 


Replace it all with:

PHP Code:

if ($nobr == true)
{
    $_posts[$dbCore->Record["ticketpostid"]]["contents"] = $dbCore->Record["contents"];
} else {
    // 'Style Creep' fix to display HTML ticket posts from Outlook without breaking 
    // the display of hyperlinks elsewhere on the page(matthew@xentana.com)
    $search = array(
        '@a:link,(?:.*?)}@si',        // hyperlink
        '@a:visited,(?:.*?)}@si',    // visited hyperlink
        '@<img(?:.*?)>@si',            // IMG tags
        );
    $text = preg_replace($search, '', $dbCore->Record["contents"]);
    $_posts[$dbCore->Record["ticketpostid"]]["contents"] = $this->returnProcessedContent($text);
    // End 'Style Creep' fix
} 


Ticket Previews
To get a good and reasonably consistent display for ticket mouseover previews, we're going to split off the HTML tag-stripping function and customize it a bit. We'll also cut out some 'fat' in the ticket details displayed by the mouseover.

1. In functions_html.php, locate the stripHTMLTags function near the end of the file, and place this modified version right after it:

PHP Code:

function stripHTMLTagsMO($str) 
{
    global $_SWIFT;
    
    // Some HTML tags are allowed, if the setting is enabled.
    $allowed = array();
    
    if (1 == $_SWIFT["settings"]["pr_allowhtml"])
    {
        if (preg_match_all('@<([^>]+)>@i', $_SWIFT["settings"]["pr_allowabletags"], $matches))
        {
            $allowed = array_merge($allowed, $matches[1]);
        }
    }
    
    if (count($allowed))
    {
        $strRegex = '@<(?![\\/]?(' . implode("|", $allowed) . '))[\\/\\!]*?[^<>]*?>@si';
    }
    else
    {
        $strRegex = '@<[\\/\\!]*?[^<>]*?>@si';    
    }
    
        $search = array(
                    '@class[^<]*?/>@si',                        // class attributes
                    '@<style[^<]*?/>@si',                        // "Empty" style tags
                    '@<style(?:.*?)>.*?</style(?:[\s]*?)>@si',    // "Full" style tags
                    '@style=\'(?:.*?)\'@si',                    // style attributes
                    '@<img(?:.*?)>@si',                            // IMG tags
                    '@\bMso(?:.*?)\b@si',                        // Pesky Mso's
                    '@<o:(?:.*?)>@si',                            // Smart tags
                    $strRegex                                    // All other tags and HTML comments 
                    );
    $str = preg_replace('@&nbsp;@', ' ', $str);
    $str = preg_replace($search, '', $str);
    $str = preg_replace('@<html(?:.*?)>@si','<html>',$str);
    return preg_replace('@\n@si', '<p>', $str); 
} 


2. Around line 860 of functions_ticketsui.php, locate a line that begins:

PHP Code:

$tooltip = $_SWIFT["language"]["f_ticketid"].": ".$record["t.ticketid"]."<BR />". $_SWIFT["language"]["f_subject"].": ".htmlspecialchars($record["subject"])."<BR />". $_SWIFT["language"]["f_department"].": " 


Replace it with:

PHP Code:

$tooltip = $_SWIFT["language"]["f_ticketid"].": ".$record["t.ticketid"]."<BR />". $_SWIFT["language"]["f_subject"].": ".htmlspecialchars($record["subject"])."<BR />". $_SWIFT["language"]["f_department"].": ".$record["t.departmentid"]."<BR />". $_SWIFT["language"]["f_ticketstatus"].": ".$record["t.ticketstatusid"]."<BR />". $_SWIFT["language"]["f_date"].": ".$record["t.dateline"]."<BR />". $_SWIFT["language"]["f_owner"].": ".$record["t.ownerstaffid"]."<BR />". $_SWIFT["language"]["f_priority"].": ".$record["t.priorityid"]."<BR />". $_SWIFT["language"]["f_lastreplier"].": ".htmlspecialchars($record["lastreplier"])."<BR />". $_SWIFT["language"]["f_fullname"].": ".htmlspecialchars($record["fullname"])."<BR />". $_SWIFT["language"]["f_email"].": ".htmlspecialchars($record["email"])."<BR />". $_SWIFT["language"]["f_totalreplies"].": ".$record["t.totalreplies"]."<BR />". $_SWIFT["language"]["f_lastactivity"].": ".$record["t.lastactivity"]."<BR />". $_SWIFT["language"]["f_duedate"].": ".$record["t.duetime"]."<BR />".iif(!empty($record["preview"]), "<HR />".htmlspecialchars(preg_replace("#(\r\n|\r|\n)#s", "<BR>", stripHTMLTagsMO($record["preview"])))); 


Fix Tag Stripping
If you don't want/need to use 'No Processing' to show all the HTML in ticket posts, you can enable 'Strip Tags' to get a more uniform appearance in your tickets. I hacked the stripHTMLTags function to make this work. (I didn't have much luck with Sheep's fix.)

In functions_html.php, locate the stripHTMLTags function (yes, again), and the following lines:

PHP Code:

        $search = array(
                    '@<style[^<]*?/>@si',                        // "Empty" style tags
                    '@<style(?:.*?)>.*?</style(?:[\s]*?)>@si',    // "Full" style tags
                    $strRegex                                    // All other tags and HTML comments
                    ); 


Replace them with:

PHP Code:

        $search = array(
                    '@class[^<]*?/>@si',                        // class attributes
                    '@<style[^<]*?/>@si',                        // "Empty" style tags
                    '@<style(?:.*?)>.*?</style(?:[\s]*?)>@si',    // "Full" style tags
                    '@style=\'(?:.*?)\'@si',                    // style attributes
                    '@&nbsp;@',                                    // get rid of extra line breaks, actually
                    '@<img(?:.*?)>@si',                            // IMG tags
                    '@\bMso(?:.*?)\b@si',                        // Pesky Mso's
                    '@<o:(?:.*?)>@si',                            // Smart tags
                    $strRegex                                    // All other tags and HTML comments
                    ); 


We're just expanding the number of tags that are stripped out of those tricky Outlook HTML emails, so that they display quite nicely in the Staff CP and client-side ticket lists.

Clean <IMG> Tags From Alerts
This is optional. New ticket and client reply alerts sent to staff normally contain the contents of the ticket post. By default any <IMG> tags included in the post (a corporate logo, for example, included as an in-line attachment) will be there as well. Since SupportSuite does not have the ability to forward these in-line images, the <IMG> tag remains in the post, but width and height attributes may cause it to take up screen space, or an alt attribute causes it to display default text, such as the filename of the image. I would just as soon not see these things. To filter them out, you need to add a regular expression in two different files to strip these <IMG> tags completely out of the forwarded message text.

1. In functions_ticketmain.php, around line 1438, locate this code:

PHP Code:

        if ($isnewticket != true)
        {
            if ($isemailed && $ishtml)
            {
                $alertcontents = strip_tags(str_replace("<br />", "\n", str_replace("<BR />", "\n", $contents)));
            } else {
                $alertcontents = $contents;
            } 


Replace it with the following, which actually only inserts one new line:

PHP Code:

        if ($isnewticket != true)
        {
            $contents = preg_replace('@<img[^>]*>@si', '', $contents); // Clean IMG tags (matthew@xentana.com)
            if ($isemailed && $ishtml)
            {
                $alertcontents = strip_tags(str_replace("<br />", "\n", str_replace("<BR />", "\n", $contents)));
            } else {
                $alertcontents = $contents;
            } 


2. in functions_ticketcore.php, around line 480, locate this code:

PHP Code:

    // Load the ticket object.
    $_SWIFT["ticketobj"] =& new TicketMain($ticketid);
    $_ticket = &$_SWIFT["ticketobj"];
    if (_is_array($attachments))
    {
        $_ticket->alerts->addAttachments($attachments);
    } 


Insert this line immediately after it, so that it looks like:

PHP Code:

    // Load the ticket object.
    $_SWIFT["ticketobj"] =& new TicketMain($ticketid);
    $_ticket = &$_SWIFT["ticketobj"];
    if (_is_array($attachments))
    {
        $_ticket->alerts->addAttachments($attachments);
    }
    $contents = preg_replace('@<img[^>]*>@si', '', $contents); // Clean IMG tags (matthew@xentana.com) 


Customizing Templates
Now that your HTML emails are working great, you might want to think about customizing your templates to show off that pretty formatting. I recommend having a look at this article: http://forums.kayako.com/f56/howto-s...oks-good-7617/. Using this as a starting point, I customized all my HTML templates to display the company logo, and much more. Here's a quick hint:

All these templates have the same basic structure, which as a beginner you should look for:

<{if $ishtml == true}>
<!-- HTML content goes here -->
<{else}>
Plain text content goes here.
<{/if}>

Breaklines
If you have lots of clients using Outlook, and if you allow replies to tickets or ticket alerts via email, then you've probably seen plenty of reply posts that have fragments of the original message attached as a result of the person hitting 'Reply' and typing their message without first deleting the old text. This is where breaklines come in.

If the user has HTML enabled in their Outlook as the default format for new messages, then there should be no need for the breakline to kick in, as the <div> tags that contain the message separator will be stripped out by the email parser as a result of our settings.

However, if the client uses plain text by default and replies to a SupportSuite staff post, then we need to think about the breakline. Other threads have dealt extensively with breaklines, so I won't go into this in detail. It can get complicated. I'll only note that most of our clients are English speakers who use MS Outlook 2007, and so the most common breakline we see in plain text messages is:

> -----Original Message-----

or simply

-----Original Message-----

If you decide to go with plain text message templates instead of HTML, then I recommend manually inserting a 'standard' breakline at the beginning of the plain text content for the autoresponder, staffreply, and various alert templates. Something along the lines of:

====== Please reply above this line ======

Referring to the structure mentioned above in Customizing Templates, this will go in the following location:

<{if $ishtml == true}>
  <!-- HTML content goes here -->
<{else}>
====== Please reply above this line ======
The rest of the plain test message goes here.
<{/if}>

You then must make sure that you've inserted this breakline into your list of breaklines at AdminCP->Mail Parser->Manage Breaklines. This has been discussed in other threads.

All Done!
That's a lot to do, but hopefully you've now got a SupportSuite that displays HTML emails beautifully, doesn't mess up HTML email replies to ticket alerts, and doesn't break any plain text formatting either. Just like I have!

[Matthew]

I wanted to further mention one 'bug' that I haven't been able to resolve.

Bug prevents body of New Ticket Alerts from appearing in plain text format when alert is triggered by new ticket in HTML format received via email.

This bug occurs when HTML Email is set to 'No', which should cause all emails to be in plain text. Incoming HTML email tickets are processed correctly as HTML, but any New Ticket Alerts are issued with the 'contentstext' variable full of HTML tags, producing a garbled message. This is issue is not seen when a new ticket is created via the web interface.

New ticket alerts (and the 'contentstext' variable) are generated by /modules/functions_ticketcore.php at about line 512:

PHP Code:

$_ticket->alerts->raiseAlert(ALERT_NEWTICKET, array("contentshtml" => getProcessedHTML($contents), "contentstext" => getProcessedText($contents), "contentssms" => substr($_ticket->returnProcessedContent($contents),0,40), "fullname" => $fullname)); 


I haven't been able to discover an effective fix for this problem. Maybe someone else can.

[jnet]

Emails coming from clients have always some useless signes and I do not want to do the above modifications because then updating becomes painful

[Matthew]

Originally Posted by jnet

I do not want to do the above modifications because then updating becomes painful

It's the price we pay, and you basically get what you pay for. Updating is never really fun, but a guaranteed adventure.

[supportskins]

Alot of clients are having issues with this. I am sure this is more than useful for them!

[mopa]

In functions_html.php, locate the stripHTMLTags function near the end of the file, and place this modified version right after it:

Could you be a little more specific - I can't seem to find this place?

[Matthew]

Originally Posted by mopa

Could you be a little more specific - I can't seem to find this place?

It's probably the 2nd to the last function. Anyway, just use your editor's search tool.

[mopa]

I must be blind.
We are talking about the file includes/fucntions_html.php right?

These are the last lines from my file:

PHP Code:

function printSelectRow($name, $title, $description = "", $selectoptionsarray, $onchange = "", $style = "") {
 global $_SWIFT;
 if (empty($name)) {
  return false;
 }
 if ($_POST[$name] == 1) {
  $isyes = true;
 }
 $data = '<select name="'. $name .'" class="swiftselect"'. iif($onchange!="", ' onChange="'. $onchange .'"') .'>';
 $selecteddone = false;
 for ($ii=0; $ii<count($selectoptionsarray); $ii++)
 {
  $data .= '<option value="'. $selectoptionsarray[$ii]["value"] .'"';
  if ((($_POST[$name] == $selectoptionsarray[$ii]["value"]) || ($_POST[$name] == "" && $selectoptionsarray[$ii]["selected"] == true)) && $selecteddone == false)
  {
   $data .= ' selected';
   $selecteddone = true;
  }
  $data .= '>'. str_replace(" ", "&nbsp;", htmlspecialchars($selectoptionsarray[$ii]["title"])) .'</option>'.SWIFT_CRLF;
 }
 $data .= '</select>'.SWIFT_CRLF;
 
 $info = '<span class="tabletitle">'. $title .'</span>'. iif($description, '<BR /><span class="tabledescription">'. $description .'</span>');
 
 return printDefaultRow($info, $data, $name, $style);
}
 
/**
* Prints a row with select multiple box allowing selection of multiple options
*/
function printSelectMultipleRow($name, $title, $description = "", $selectoptionsarray, $size = 5, $width = "") {
 global $_SWIFT;
 if (empty($name)) {
  return false;
 }
 if ($_POST[$name] == 1) {
  $isyes = true;
 }
 $data = '<select name="'. $name .'[]" class="swiftselect" size="'. $size .'"'. iif(!empty($width), ' style="WIDTH: '. $width .';"') .' multiple>';
 for ($ii=0; $ii<count($selectoptionsarray); $ii++)
 {
  $data .= '<option value="'. $selectoptionsarray[$ii]["value"] .'"';
  if ((@in_array($selectoptionsarray[$ii]["value"], $_POST[$name])) || (!is_array($_POST[$name]) && $selectoptionsarray[$ii]["selected"] == true))
  {
   $data .= ' selected';
  }
  $data .= '>'. str_replace(" ", "&nbsp;", htmlspecialchars($selectoptionsarray[$ii]["title"])) .'</option>'.SWIFT_CRLF;
 }
 $data .= '</select>'.SWIFT_CRLF;
 
 $info = '<span class="tabletitle">'. $title .'</span>'. iif($description, '<BR /><span class="tabledescription">'. $description .'</span>');
 
 return printDefaultRow($info, $data);
}
/*
* Prints a redirect, can be unix or windows style depending upon setting
*/
function printRedirect($message, $url, $printredir = false)
{
 global $_SWIFT, $settings, $template;
 if ($settings->store["general_redirecttype"] == "windows" || $printredir == true) {
  $template->assign("redirecturl", $url);
  if (SWIFT_AREA == SWIFT_CLIENT || SWIFT_AREA == SWIFT_VISITOR)
  {
   echo $template->displayTemplate("redirect");
  } else {
   echo $template->display("redirect.tpl");
  }
  printTableHeader(array(), "100%", "2", "1");
  printInfoRow('<table border="0" cellpading="0" cellspacing="3"><tr><td align="center"><span class="mediumtext">'.$message.'</span></td></tr><tr><td><span class="smalltext"><a href="'. $url .'">'.$_SWIFT["language"]["no_wait"].'</a></span></td></tr></table>');
  printTableFooter();
 } else if ($settings->store["general_redirecttype"] == "unix") {
  header("location: $url");
 }
}
/**
* Prints a select row with all current user groups
*/
function printUserGroupSelectRow($name, $title, $description, $onlyregistered = false)
{
 global $dbCore, $_UGROUPCACHE;
 if (_is_array($_UGROUPCACHE))
 {
  $_options = $_UGROUPCACHE;
 } else {
  $_options = array();
  $index = $firstone = 0;
  $dbCore->query("SELECT `usergroupid`, `title` FROM `". TABLE_PREFIX ."usergroups`". iif($onlyregistered, " WHERE `grouptype` = '". USERGROUP_REGISTERED ."'")." ORDER BY `title` ASC;", 3);
  while ($dbCore->nextRecord(3))
  {
   $_options[$index]["title"] = $dbCore->Record3["title"];
   $_options[$index]["value"] = $dbCore->Record3["usergroupid"];
   if ($firstone == 0)
   {
    $_options[$index]["selected"] = 1;
    $firstone = 1;
   }
   $index++;
  }
  $_UGROUPCACHE = $_options;  
 }
 printSelectRow($name, $title, $description, $_options);
}
?> 


Could you point it out because as you write it I think I should look for something called "strip..."

[Matthew]

Originally Posted by mopa

I must be blind.
We are talking about the file includes/fucntions_html.php right?

These are the last lines from my file:
[/php]Could you point it out because as you write it I think I should look for something called "strip..."

Hmmm... not sure what to say Mopa. I don't see it there either. Are we using the same version?

(You might be thinking of the built-in php function, strip_tags.)

Incidentally, I've actually gone a bit more extreme than when I posted this mod. I'm in the final stages of integrating the TinyMCE editor, which is really fantastic. I plan on sharing when the dust settles a little more.

[mopa]

Ok - that sounds very cool - and no it's not the same version as I still use 3.11.01 due to a lot of mods... will wait for your updated TinyMCE version and then hope it works with my version as well.

FYI: I really appreciate that you look into getting this to work - would have loved that it worked out of the box though but hey - at least we got some people like your self doing the hard work to get it all done

[webawere]

Mannn... I am a bit shocked that this all this is needed (and not supported by Kayako) to accept normal html messages...

My(our) customers have NO idea if they are sending text or html mail and it looks very unprofessional if there emails are messed-up.