Character set issues after upgrading to 3.20.02

[Ryan Lederman]

Originally Posted by iczeh

I've converted the tables to utf-8 by John's instructions, but the messages still go back to the customers with wrong characters. What else can I do?

Check the content-type header that your server is sending when it serves pages (both by viewing the source and looking at the meta tags and by viewing the actual HTTP headers); you can use this FF plugin to see the headers:

https://addons.mozilla.org/en-US/firefox/addon/3829

If the content type is being served wrong there are a couple places I would look:

1. Search all active language packs for "charset" and make sure all those entries are also set to UTF-8.
2. If you're using Apache 2, there is a new setting called AddDefaultCharset that causes the headers to include a default character set if none is provided (which is the case with Kayako products). Look at httpd.conf and make sure this setting is disabled.

If the content type is right and you're still getting garbled characters, it is possible that the database connection settings are causing problems.

[BoxAdmin]

a week ago, I sent email to Ryan to send me the patch but unfortunately he did not reply.

sveide, could you please forward that character set conversion patch to me please.

I will pm you my email.


thanks.

[BoxAdmin]

never mind, I managed to change the coding from using mbstring to iconv

and now it works perfectly.


PLEASE Kayako, I already written a request in the bug portal about this. please replace mbstring with iconv and no one will have any problem with encoding.
Kayako Bug Tracker - Viewing Issue #689 - Parser Encoding & Charactersets Conversion Problems

[John Haugeland]

Originally Posted by BoxAdmin

please replace mbstring with iconv and no one will have any problem with encoding.

Yeah, it's not actually that simple. For example, iconv doesn't exist by default on Windows systems, and is difficult to get running with IIS, which represents a fairly large subset of our userbase. There are charactersets that iconv doesn't cover. iconv doesn't mesh with regular expressions the way mbstring does. Until recently, SupportSuite officially supported versions of PHP that predated iconv support. Et cetera.

There's also the question of what to do about PHP6, which has fully native unicode and characterset conversion support.

Believe me, sir, this conversion issue is actually one of my personal bugbears; I want to see this changed very badly. Like you, I think that mbstring is a problem, especially because it's basically stopping certain language groups cold - particularly cp1256/arabic. However, as an application vendor, we have to take a very broad view of people's differing servers.

We do, actually, have a plan about this. I'm not supposed to be talking about it yet, but we're not ignoring this issue. This is a red flag for us right now, and we're working on a solution that should work for a whole lot of people. We're just not going to start talking about it when we're only halfway done planning it ourselves.

Seriously, though. This Is Going To Get Fixed (tm), and iconv will be one of the options.

[asiams]

What kind of settings should I make in order for Japanese to work both on showing them on the web, and to send email in ISO-2022 (JIS) through pipe setup?

[Ryan Lederman]

Originally Posted by asiams

What kind of settings should I make in order for Japanese to work both on showing them on the web, and to send email in ISO-2022 (JIS) through pipe setup?

You should use UTF-8 for your database, tables, database connection and main charset. Then, set text_charset and html_charset to ISO-2022-JP in the language pack. That will cause outgoing e-mail to always be sent in ISO-2022. Incoming e-mail will be converted to UTF-8.

Important: The steps mentioned above will only work in 3.20.02 and higher.

[asiams]

Originally Posted by Ryan Lederman

You should use UTF-8 for your database, tables, database connection and main charset. Then, set text_charset and html_charset to ISO-2022-JP in the language pack. That will cause outgoing e-mail to always be sent in ISO-2022. Incoming e-mail will be converted to UTF-8.

Important: The steps mentioned above will only work in 3.20.02 and higher.

Now where can I get the Japanese language package for the 3.20.02 version?

Thanks.

[supportskins]

You might want to check the "Language Packs & Translation" sub-forum if someone has shared it. If not, then you can create one for yourself by editing the language pack XML file.

[asiams]

Hello,

Currently I use DB with different character set. In this case, how can I convert the DB into UTF-8?

For DB connection to be UTF-8, should I set this at the config file? --> Self Answer: Yet it is located in config.php under config directory.

To set the main character set to be UTF-8, which file should I modify? --> This is also in the config.php file as default charset.

In order to set text_charset and html_charset to ISO-2022-JP, which files should I modify? --> This is in the language pack or CP->Language-> Manage Phrases -> search for charset, it will show the current set up.

At the admin, I can find the html_charset section to modify. But could not find text_charset section to modify. Where can I find this file? --> This was answered above.

These are my findings on this issue. Hope this will help.

Thanks.

[asiams]

You need to change the character set and the collation for any table being converted. You can do those both at the same time.

Please remember to back up your database before working in SQL; especially in the case of character sets, errors can cause loss. Backups are crucial.

For any table that you would like to convert to UTF-8 (these instructions written for a table named "foo"), issue the following command:

alter table foo convert to character set 'utf8' collate utf8_unicode_ci;

Of course, you'll want to go through the table afterwards, and make sure your data is intact. (It should be, but with character set issues and MySQL, one just checks.)

Please be aware that though this is safe for Kayako products, there is a bug in MySQL which causes the conversion of mediumtext and largetext columns to text columns during this conversion. As such, this command would not be safe for applications which stored very large texts (in excess of 2000 printed pages).

This seems to be the general instruction on how to convert DB to UTF-8.

Is there any patch that can just convert this process for us?

Or does the update process include this process to 3.30.02?

Any better solution to this?

Thanks.

[asiams]

deleted since it was no longer valid