UTF-8 and UTF-16

A client of ours using our BlackBerry Ticket Client contacted us to say that sync suddenly stopped working. Another client then contacted us to say the same thing.

It took a while but we finally nailed down the issue. It was down to the character Ascii Table being in one of the ticket replies.

The developer of this application spent a while trying to fix the issue on the BlackBerry but didn't get very far. Both of the parsers we tried to use just point blank refused to do anything once it hit the bad character. One thing we tried was scanning the XML file and removing bad characters first but this took 30 minutes to do so was not an option.

I passed this over to one of my PHP developers and he found the ultimate problem is the following lines of Kayako code in /includes/functions_xml.php :-

Code:

	/**
	* Function to be called before XML building starts
	*/
	function buildXML($charset)
	{
		global $_SWIFT;

		if (SWIFT_AREA == SWIFT_MOBILE)
		{
			$this->charset = "UTF-16";
		} else {
			$this->charset = $charset;
		}

		$this->xmldata = array();
		$this->xmldata[] = '<?xml version="1.0" encoding="'. $this->charset .'"?>'.$this->crlf;

		return true;
	}

For some reason (my developer has no idea why it has been coded this when UTF-8 works much better), it has been set to use UTF-16 for all mobile devices. The problem, however, is it is putting characters in UTF-8 format (such as the one I listed above) into the UTF-16 output.

This really needs to get resolved and I hope it can be patched in the next minor build.

Right, it also looks like characters 0-31 from the table at ASCII Table - ASCII and Unicode Characters need to be ignored and not included in the XML too. They appear to be the ones that cause problems.

Well we have done some more testing and it is just my second post that I need resolving to be able to get our application working. We need characters 0-31 from ASCII Table - ASCII and Unicode Characters stripped before data gets sent out. There is NO way around this locally on the BlackBerry due to the parser.

I would appreciate getting this resolved before the next build.

Bump. I want to make sure this makes the next release...

Bump. Had another user who had problems with this today again having to provide them with a patched file to remove those characters. Please can we get this resolved?

Great stuff. I take it this is planned for 3.70.03 and in which case, when is this due out?