Fix Japanese character issue in unix box - database

Question

Our company build process uses gradle and copies over the deliverables including sql file to a shared drive. Unix boxes can refer to these paths and then uses a shell script. The shell script invokes the db2 command line utility to run the sql files in the corresponding environment's database.
This works pretty well. However I have hit a wall where we are updating japanese characters in the sql file. When I download the file from the browser and inspect the file, I can see the characters properly. However, in the unix box, I can see that the content in the sql file looks pretty different. I used cat, more and vi to view the contents of the file. Value of the LANG environment variable is C. I see the environment variable set is TERM=XTERM. So I think xterm is the terminal evaluator.
LANG=C
When I run the command in the linux terminal window-
locale charmap
I see the encoding as ANSI_X3.4-1968. UTF-8 is one of the possiblities but that is not set in the box I think.
When I run the command in the linux terminal -
file -bi 1.sql
I see the charset encoding of the file as UTF-8.
When we run the shellscript which invokes the db2 command line utility, it inserts whatever I can see in the unix file and not the actual japanese characters.
I tries searching for options as to what could go wrong. Please find below my analysis -
When I run the sql from the Aqua Data Studio, data is inserted properly. So there is no issue with the actual database setup or the table configuration.
There are some other xml files in the shared path which have japanese characters. In unix box, I again see the distorted characters as I see in the sql file. When these xml files get copied to the reporting server, they work fine and the data is shown properly in the pdf.
So my line of thinking is that there can be some way I can specify in the db2 command line utility to use the correct character set and that should work fine. But I am unable to find anything useful on the internet.
Now there are chances that there is something small which I am not able to figure out. Hence I will request you to help me in this regard. This is not very critical as we can always leave the automated process of deployment and sql insertion , and instead go with the manual insertion by giving the scripts to the DBA. However this is highly discouraged.
Do let me know if you could help me with this regard. Just FYI, I unzipped the war produced after the gradle build (from the jenkins server). I can see the file properly in the notepad++. Do let me know if you need any other info.

Answer 1

If the Japanese characters do not render correctly in cat or more in your terminal window then it means that the LANG variable has the wrong setting. In your case LANG=C is not suitable for Unicode.
Some terminal emulators such as Putty, need to have the translation set to support UTF-8 and that may not be default - otherwise Unicode characters can render incorrectly on screen.
The correct LANG setting depends on your country, on which locales are installed in Linux, and also on the target database territory and codepage and Db2-server operating-system (i.e. Db2 for Z/OS , or i-series, or Linux/Unix/Windows). Sometimes you have to install additional locales if they were not installed by default, it depends on your Linux distro , platform, and version and installation policies.
The solution is normally to ensure you set the LANG variable correctly before you connect to any Db2 database. It's wise to assert such things in configuration checks that get automatically run by scripts. You get the correct value for LANG from the Db2 documentation, and that depends on your Db2-server operating system and version and your knowledge of the target database encoding and territory (or CCSID of the target database/tables/columns/tablespaces etc). For example, if your Db2-database lives on Linux/Unix/Windows, look at this page. Other pages exist for Db2 for Z/OS or i-series.
Once you change the LANG variable ( export LANG=...), again verify that more or cat renders the Japanese characters correctly. If you are already connected to the database, changing the LANG has no effect, so remember to disconnect first. Test the LANG values until the Japanese characters look correct, and only then try the Db2 connection and examine the new behavior.