Author: Tatsuo Ishii (<ishii@postgresql.org>), last updated 2000-03-22. Check Tatsuo's web site for more information.
Multibyte (MB) support is intended to allow PostgreSQL to handle multiple-byte character sets such as EUC (Extended Unix Code), Unicode and Mule internal code. With MB enabled you can use multibyte character sets in regular expressions (regexp), LIKE, and some other functions. The default encoding system is selected while initializing your PostgreSQL installation using initdb. Note that this can be overridden when you create a database using createdb or by using the SQL command CREATE DATABASE. So you can have multiple databases each with a different encoding system.
MB also fixes some problems concerning 8-bit single byte character sets including ISO8859. (I would not say all problems have been fixed. I just confirmed that the regression test ran fine and a few French characters could be used with the patch. Please let me know if you find any problem while using 8-bit characters.)
Run configure with the multibyte option:
% ./configure --enable-multibyte[=encoding_system]
where encoding_system can be one of the values in the following table:
Table 5-1. Character Set Encodings
Encoding | Description |
---|---|
SQL_ASCII | US ASCII |
EUC_JP | Japanese EUC |
EUC_CN | Chinese EUC |
EUC_KR | Korean EUC |
EUC_TW | Taiwan EUC |
UNICODE | Unicode (UTF-8) |
MULE_INTERNAL | Mule internal code |
LATIN1 | ISO 8859-1 ECMA-94 Latin Alphabet No.1 |
LATIN2 | ISO 8859-2 ECMA-94 Latin Alphabet No.2 |
LATIN3 | ISO 8859-3 ECMA-94 Latin Alphabet No.3 |
LATIN4 | ISO 8859-4 ECMA-94 Latin Alphabet No.4 |
LATIN5 | ISO 8859-9 ECMA-128 Latin Alphabet No.5 |
LATIN6 | ISO 8859-10 ECMA-144 Latin Alphabet No.6 |
LATIN7 | ISO 8859-13 Latin Alphabet No.7 |
LATIN8 | ISO 8859-14 Latin Alphabet No.8 |
LATIN9 | ISO 8859-15 Latin Alphabet No.9 |
LATIN10 | ISO 8859-16 ASRO SR 14111 Latin Alphabet No.10 |
ISO-8859-5 | ECMA-113 Latin/Cyrillic |
ISO-8859-6 | ECMA-114 Latin/Arabic |
ISO-8859-7 | ECMA-118 Latin/Greek |
ISO-8859-8 | ECMA-121 Latin/Hebrew |
KOI8 | KOI8-R(U) |
WIN | Windows CP1251 |
ALT | Windows CP866 |
CAUTION1: Note that before 7.2 LATIN5 meant ISO 8859-5 mistakely. In 7.2 LATIN5 measn ISO 8859-9. If you have LATIN5 database created on 7.1 or before and want to migrate to 7.2, you should be very carefull about this change.
CAUTION2: Not all API supports encodings listed above. For example, PostgreSQL JDBC driver does not support MULE_INTERNAL, LATIN6, LATIN8 and LATIN10.
Here is an example of configuring PostgreSQL to use a Japanese encoding by default:
% ./configure --enable-multibyte=EUC_JP
If the encoding system is omitted (./configure --enable-multibyte), SQL_ASCII is assumed.
initdb defines the default encoding for a PostgreSQL installation. For example:
% initdb -E EUC_JP
sets the default encoding to EUC_JP (Extended Unix Code for Japanese). Note that you can use --encoding instead of -E if you prefer to type longer option strings. If no -E or --encoding option is given, the encoding specified at configure time is used.
You can create a database with a different encoding:
% createdb -E EUC_KR korean
will create a database named korean with EUC_KR encoding. Another way to accomplish this is to use a SQL command:
CREATE DATABASE korean WITH ENCODING = 'EUC_KR';
The encoding for a database is represented as an encoding column in the pg_database system catalog. You can see that by using -l or \l of psql command.
$ psql -l List of databases Database | Owner | Encoding ---------------+---------+--------------- euc_cn | t-ishii | EUC_CN euc_jp | t-ishii | EUC_JP euc_kr | t-ishii | EUC_KR euc_tw | t-ishii | EUC_TW mule_internal | t-ishii | MULE_INTERNAL regression | t-ishii | SQL_ASCII template1 | t-ishii | EUC_JP test | t-ishii | EUC_JP unicode | t-ishii | UNICODE (9 rows)
PostgreSQL supports an automatic encoding translation between backend and frontend for some encodings.
Table 5-2. Client/Server Character Set Encodings
Server Encoding | Available Client Encodings |
---|---|
SQL_ASCII | SQL_ASCII, UNICODE, MULE_INTERNAL |
EUC_JP | EUC_JP, SJIS, UNICODE, MULE_INTERNAL |
EUC_TW | EUC_TW, BIG5, UNICODE, MULE_INTERNAL |
LATIN1 | LATIN1, UNICODE MULE_INTERNAL |
LATIN2 | LATIN2, WIN1250, UNICODE, MULE_INTERNAL |
LATIN3 | LATIN3, UNICODE MULE_INTERNAL |
LATIN4 | LATIN4, UNICODE MULE_INTERNAL |
LATIN5 | LATIN5, UNICODE MULE_INTERNAL |
LATIN6 | LATIN6, UNICODE MULE_INTERNAL |
LATIN7 | LATIN7, UNICODE MULE_INTERNAL |
LATIN8 | LATIN8, UNICODE MULE_INTERNAL |
LATIN9 | LATIN9, UNICODE MULE_INTERNAL |
LATIN10 | LATIN10, UNICODE MULE_INTERNAL |
ISO_8859_5 | ISO_8859_5, UNICODE |
ISO_8859_6 | ISO_8859_6, UNICODE |
ISO_8859_7 | ISO_8859_7, UNICODE |
ISO_8859_8 | ISO_8859_8, UNICODE |
ISO_8859_9 | ISO_8859_9, WIN, ALT, KOI8R, UNICODE, MULE_INTERNAL |
UNICODE | EUC_JP, SJIS, EUC_KR, EUC_CN, EUC_TW, BIG5, LATIN1 to LATIN10, ISO_8859_5, ISO_8859_6, ISO_8859_7, ISO_8859_8, WIN, ALT, KOI8 |
MULE_INTERNAL | EUC_JP, SJIS, EUC_KR, EUC_CN, EUC_TW, BIG5, LATIN1 to LATIN5, WIN, ALT, WIN1250 |
KOI8 | ISO_8859_9, WIN, ALT, KOI8, UNICODE, MULE_INTERNAL |
WIN | ISO_8859_9, WIN, ALT, KOI8, UNICODE, MULE_INTERNAL |
ALT | ISO_8859_9, WIN, ALT, KOI8, UNICODE, MULE_INTERNAL |
To enable the automatic encoding translation, you have to tell PostgreSQL the encoding you would like to use in frontend. There are several ways to accomplish this.
Using the \encoding command in psql. \encoding allows you to change frontend encoding on the fly. For example, to change the encoding to SJIS, type:
\encoding SJIS
Using libpq functions. \encoding actually calls PQsetClientEncoding() for its purpose.
int PQsetClientEncoding(PGconn *conn, const char *encoding)
where conn is a connection to the backend, and encoding is an encoding you want to use. If it successfully sets the encoding, it returns 0, otherwise -1. The current encoding for this connection can be shown by using:
int PQclientEncoding(const PGconn *conn)
Note that it returns the encoding id, not the encoding symbol string such as EUC_JP. To convert an encoding id to an encoding symbol, you can use:
char *pg_encoding_to_char(int encoding_id)
Using SET CLIENT_ENCODING TO. Setting the frontend side encoding can be done by this SQL command:
SET CLIENT_ENCODING TO 'encoding';
Also you can use SQL92 syntax SET NAMES for this purpose:
SET NAMES 'encoding';
To query the current frontend encoding:
SHOW CLIENT_ENCODING;
To return to the default encoding:
RESET CLIENT_ENCODING;
Using PGCLIENTENCODING. If environment variable PGCLIENTENCODING is defined in the client's environment, that client encoding is automatically selected when a backend connection is made. (This can subsequently be overridden using any of the other methods mentioned above.)
An automatic encoding translation between Unicode and other encodings has been supported since PostgreSQL 7.1. For 7.1 it was not enabled by default. To enable this feature, run configure with the --enable-unicode-conversion option. Note that this requires the --enable-multibyte option also.
For 7.2, --enable-unicode-conversion is not necessary. The unicode conversion functionality is automatically enabled if --enable-multibyte is specified.
Suppose you choose EUC_JP for the backend, LATIN1 for the frontend, then some Japanese characters could not be translated into LATIN1. In this case, a letter that cannot be represented in the LATIN1 character set would be transformed as:
(HEXA DECIMAL)
These are good sources to start learning about various kinds of encoding systems.
Detailed explanations of EUC_JP, EUC_CN, EUC_KR, EUC_TW appear in section 3.2.
The web site of the Unicode Consortium
UTF-8 is defined here.
Dec 7, 2000 * An automatic encoding translation between Unicode and other encodings are implemented * Changes above will appear in 7.1 May 20, 2000 * SJIS UDC (NEC selection IBM kanji) support contributed by Eiji Tokuya * Changes above will appear in 7.0.1 Mar 22, 2000 * Add new libpq functions PQsetClientEncoding, PQclientEncoding * ./configure --with-mb=EUC_JP now deprecated. use ./configure --enable-multibyte=EUC_JP instead * Add SQL_ASCII regression test case * Add SJIS User Defined Character (UDC) support * All of above will appear in 7.0 July 11, 1999 * Add support for WIN1250 (Windows Czech) as a client encoding (contributed by Pavel Behal) * fix some compiler warnings (contributed by Tomoaki Nishiyama) Mar 23, 1999 * Add support for KOI8(KOI8-R), WIN(CP1251), ALT(CP866) (thanks Oleg Broytmann for testing) * Fix problem with MB and locale Jan 26, 1999 * Add support for Big5 for fronend encoding (you need to create a database with EUC_TW to use Big5) * Add regression test case for EUC_TW (contributed by Jonah Kuo <jonahkuo@mail.ttn.com.tw>) Dec 15, 1998 * Bugs related to SQL_ASCII support fixed Nov 5, 1998 * 6.4 release. In this version, pg_database has "encoding" column that represents the database encoding Jul 22, 1998 * determine encoding at initdb/createdb rather than compile time * support for PGCLIENTENCODING when issuing COPY command * support for SQL92 syntax "SET NAMES" * support for LATIN2-5 * add UNICODE regression test case * new test suite for MB * clean up source files Jun 5, 1998 * add support for the encoding translation between the backend and the frontend * new command SET CLIENT_ENCODING etc. added * add support for LATIN1 character set * enhance 8 bit cleaness April 21, 1998 some enhancements/fixes * character_length(), position(), substring() are now aware of multi-byte characters * add octet_length() * add --with-mb option to configure * new regression tests for EUC_KR (contributed by Soonmyung Hong <hong@lunaris.hanmesoft.co.kr>) * add some test cases to the EUC_JP regression test * fix problem in regress/regress.sh in case of System V * fix toupper(), tolower() to handle 8bit chars Mar 25, 1998 MB PL2 is incorporated into PostgreSQL 6.3.1 Mar 10, 1998 PL2 released * add regression test for EUC_JP, EUC_CN and MULE_INTERNAL * add an English document (this file) * fix problems concerning 8-bit single byte characters Mar 1, 1998 PL1 released
The WIN1250 character set on Windows client platforms can be used with PostgreSQL with locale support enabled.
The following should be kept in mind:
Success depends on proper system locales. This has been tested with Red Hat 6.0 and Slackware 3.6, with cs_CZ.iso8859-2 locale.
Never try to set the server multibyte database encoding to WIN1250. Always use LATIN2 instead since there is not a WIN1250 locale in Unix.
WIN1250 encoding is usable only for Windows ODBC clients. The characters are recoded on the fly, to be displayed and stored back properly.
When running, it is important to remember the following:
This configuration reorders your sort order depending on your LC_x settings. Don't be confused with the regression test results since they don't use locale.
A locale such as ch is correctly sorted only if your system supports that locale; older systems may not do so but new ones (e.g. RH6.0) do.
You have to insert money as '162,50' (note comma within the single-quotes).
At the time of writing (early 1999), this configuration has not received extensive testing. Please let us know of any changes you had to make!
WIN1250 on Windows/ODBC
Compile PostgreSQL with locale enabled and the multibyte encoding set to LATIN2.
Set up your installation. Do not forget to create locale variables in your profile (environment). For example (this may not be correct for your environment):
LC_ALL=cs_CZ.ISO8859-2 LC_COLLATE=cs_CZ.ISO8859-2 LC_CTYPE=cs_CZ.ISO8859-2 LC_MONETARY=cs_CZ.ISO8859-2 LC_NUMERIC=cs_CZ.ISO8859-2 LC_TIME=cs_CZ.ISO8859-2
You have to start the postmaster with locales set!
Try it with Czech language, and have it sort on a query.
Install ODBC driver for PostgreSQL on your Windows machine.
Setup properly your data source. Include this line in your ODBC configuration dialog in the field Connect Settings:
SET CLIENT_ENCODING = 'WIN1250';
Now try it again, but in Windows with ODBC.