I have a problem executing insert commands that are loaded from a text file. I use codeIgniter "file" helper to load an sql line then I perform a simple db->query(content of my file). The problem is when the sql is loaded from a file the special character trims the rest of the string.
Here is an example that works
INSERT INTO test(test) VALUES("<p>There is <strong>no special character</strong> in this string</p>");
Example that will not work
INSERT INTO test(test) VALUES("<p>this character <em>é</em> is a <strong>special character</strong></p>");
In the second example, only "<p> this character <em>" will be saved. This is weird because if I execute the same line in phpMyAdmin it works fine.
Anyone knows why this happens? or what I do wrong?
Thanks
Here is a simple "step to reproduce".
Simple table :
CREATE TABLE `test` (
`test` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
) ENGINE = InnoDB;
A file "application/view/text.txt" that contains :
INSERT INTO test(test) VALUES("<p>this character <em>é</em> is a <strong>special character</strong></p>");
The code I use to perform the insert
$this->load->helper('file');
$loaded_sql = read_file(BASEPATH . "../application/views/test.txt");
$this->db->query($loaded_sql);
My database config
$db['default']['char_set'] = 'utf8';
$db['default']['dbcollat'] = 'utf8_general_ci';
CI Config
$config['charset'] = 'UTF-8';
I finally got it. Needed to use "utf8_encode()" when reading the file to unsure that special character gets encoded properly. Text file must be encoded in ANSI (default notepad encoding). If file is UTF-8 or UNICODE it won't work!
Code that resolved the problem :
$loaded_sql = utf8_encode( read_file(BASEPATH . "../application/views/test.txt") );
Try to use before inserting record in table.
$this->db->db_set_charset('latin1', 'latin1_swedish_ci');
Make sure you have same setting in table & table column.
Related
I am trying to "COPY INTO" command to load data from s3 to the snowflake
Below are the steps I followed to create the stage and loading file from stage to Snowflake
JSON file
{
"Name":"Umesh",
"Desigantion":"Product Manager",
"Location":"United Kingdom"
}
create or replace stage emp_json_stage
url='s3://mybucket/emp.json'
credentials=(aws_key_id='my id' aws_secret_key='my key');
# create the table with variant
CREATE TABLE emp_json_raw (
json_data_raw VARIANT
);
#load data from stage to snowflake
COPY INTO emp_json_raw from #emp_json_stage;
I am getting below error
Field delimiter ',' found while expecting record delimiter '\n' File
'emp.json', line 2, character 18 Row 2, column
"emp_json_raw"["JSON_DATA_RAW":1]
I am using a simple JSON file, and I don't understand this error.
What causes it and how can I solve it?
File format is not specified and is defaulting to CSV format hence the error.
Try this:
COPY INTO emp_json_raw
from #emp_json_stage
file_format=(TYPE=JSON);
There are other options too that can be specified with file_format other than TYPE. Refer the documentation here: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#type-json
try:
file_format = (type = csv field_optionally_enclosed_by='"')
The default settings do not expect the " wrapping around your data.
So you could strip all the " or ... just set the field_optionally_enclosed_by to a ". This does mean if your data has " in it things get messy.
https://docs.snowflake.com/en/user-guide/getting-started-tutorial-copy-into.html
https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html#type-csv
Also have a standard practice to mention type of file either it could be CSV, JSON ,AVRO , Parquet etc.
https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html
In a file, few of the rows have \ in a column value for example, i have rows in below format.
101,Path1,Z:\VMC\PSPS,abc
102,Path5,C:\wintm\PSPS,abc
I was wondering how to load \ character
COPY INTO TEST_TABLE from #database.schema.stage_name FILE_FORMAT = ( TYPE = CSV FIELD_OPTIONALLY_ENCLOSED_BY = '\"' SKIP_HEADER = 1 );
is there any thing that i can mention the file_format line?
Are you still getting this error? I just tried to recreate it by creating a CSV based off your sample data and a test table. I loaded the CSV into an internal stage and then ran your COPY command. It worked for me. Please see the screenshot below.
Could you provide more details on the error you are facing? Perhaps there was something off with your table definition.
I am trying to load data from azure blob storage.
The data has already been staged.
But, the issue is when I try to run
copy into random_table_name
from #stage_name_i_created
file_format = (type='csv')
pattern ='*.csv'
Below is the error I encounter:
raise error_class(
snowflake.connector.errors.ProgrammingError: 001757 (42601): SQL compilation error:
Table 'random_table_name' does not exist
Basically, it says table does not exist, which it does not, but the syntax on website is the same as mine.
COPY INTO query on Snowflake returns TABLE does not exist error
In my case the table name is case-sensitive. Snowflake seems to convert everything to upper case. I changed the database/schema/table names to all upper-case and it started working.
First run the below query to fetch the column headers
select $1 FROM #stage_name_i_created/filename.csv limit 1
Assuming below are the header lines from your csv file
id;first_name;last_name;email;age;location
Create a file_format csv
create or replace file format semicolon
type = 'CSV'
field_delimiter = ';'
skip_header=1;
Then you should define the datatype and field name as below
create or replace table <yourtable> as
select $1::varchar as id
,$2::varchar as first_name
,$3::varchar as last_name
,$4::varchar as email
,$5::int as age
,$6::varchar as location
FROM #stage_name_i_created/yourfile.csv
(file_format => semicolon );
The table must exist prior to running a COPY INTO command. In your post, you say that the table does not exist...so that is your issue.
If your table exist, try by forcing the table path like this:
copy into <database>.<schema>.<random_table_name>
from #stage_name_i_created
file_format = (type='csv')
pattern ='*.csv'
or by steps like this:
use database <database_name>;
use schema <schema_name>;
copy into database.schema.random_table_name
from #stage_name_i_created
file_format = (type='csv')
pattern ='*.csv';
rbachkaniwala, what do you mean by 'How do I create a table?( according to snowflake syntax it is not possible to create empty tables)'.
You can just do below to create a table
CREATE TABLE random_table_name (FIELD1 VARCHAR, FIELD2 VARCHAR)
The table does need to exist. You should check the documentation for COPY INTO.
Other areas to consider are
do you have the right context set for the database & schema
does the user / role have access to the table or object.
It basically seems like you don't have the table defined yet. You should
ensure the table is created
ensure all columns in the CSV exist as columns in the table
ensure the order of the columns are the same as in the CSV
I'd check data types too.
"COPY INTO" is not a query command, it is the actual data transfer execution from source to destination, which both must exist as others commented here but If you want just to query without loading the files then run the following SQL:
//Display list of files in the stage to verify stage
LIST #stage_name_i_created;
//Create a file format
CREATE OR REPLACE FILE FORMAT RANDOM_FILE_CSV
type = csv
COMPRESSION = 'GZIP' FIELD_DELIMITER = ',' RECORD_DELIMITER = '\n' SKIP_HEADER = 0 FIELD_OPTIONALLY_ENCLOSED_BY = '\042'
TRIM_SPACE = FALSE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE ESCAPE = 'NONE' ESCAPE_UNENCLOSED_FIELD = 'NONE' DATE_FORMAT = 'AUTO' TIMESTAMP_FORMAT = 'AUTO'
NULL_IF = ('\\N');
//Now select the data in the files
Select $1 as first_col,$2 as second_col //can add as necessary number of columns ...etc
from #stage_name_i_created
(FILE_FORMAT => RANDOM_FILE_CSV)
More information can be found in the documentation link here
https://docs.snowflake.com/en/user-guide/querying-stage.html
I have a text file that contains Persian words and is saved using ANSI encoding. When I try to read the Persian words from the text file, I get some characters like '?'. To solve the problem, I changed the file encoding to UTF8 and re-wrote the text file. Here's the method for changing file encoding:
public void Convert2UTF8(string filePath)
{
//first, read the text file with "ANSI" endocing
StreamReader fileStream = new StreamReader(filePath, Encoding.Default);
string fileContent = fileStream.ReadToEnd();
fileStream.Close();
//Now change the file encoding and replace it with the UTF8
StreamWriter utf8Writer = new StreamWriter(filePath.Replace(".txt", ".txt"), false, Encoding.UTF8);
utf8Writer.Write(fileContent);
utf8Writer.Close();
}
Now the first problem is solved; However, there is another issue here: every time that I want to search a Persian word from the SQL server database table, the result is null while the record does exist in the database table.
What's the solution to find my Persian words that exist in the table? The code that I currently use is simply like the following:
SELECT * FROM [dbo].[WordDirectory]
WHERE Word = N'کلمه'
Word is the field that Persian words are saved in. The type of the field is NVARCHAR. My SQL server version is 2012.
Should I change the collation?
DECLARE #Table TABLE(Field NVARCHAR(4000) COLLATE Frisian_100_CI_AI)
INSERT INTO #Table (Field) VALUES
(N'همهٔ افراد بش'),
(N'میآیند و حیثیت '),
(N'ميشه آهسته تر صحبت کنيد؟'),
(N'روح'),
(N' رفتار')
SELECT * FROM #Table
WHERE Field LIKE N'%آهسته%'
The both Queries return the same result
RESULT Set: ميشه آهسته تر صحبت کنيد؟
You have to make sure that when you are inserting the values you prefix then witn N thats to tell sql server there can be unicode character in the passed string. Same is true when you are searching for them strings in Select statement.
Probably you have problem with Persian and Arabic versions of the 'ي' and 'ك' during search. These characters even look the same, have different Unicode numbers:
select NCHAR(1740), -- Persian ى
NCHAR(1610), -- Arabic ي
NCHAR(1705), -- Persian ك
NCHAR(1603) -- Arabic ك
more info: http://www.dotnettips.info/post/90
I was trying to export a table in H2 DB into CSV using CSVWRITE function and found out if double quotes are included in a varchar column they will be duplicated.
Eg. - 'hello"howareyou' will be 'hello""howareyou' in the written csv.
Tried saving this varchar column with escape characters and few other combinations but result is the same.
Following is my table column I created to test this issue and the resulted CSV value I got.
My column CSV written value
------------------------------
hello"how hello""how
hello\"how hello\""how
hello""how hello""""how
hello\""how hello\""""how
hello\\"how hello\\""how
hello\\\\"how hello\\\\""how
hello["]how hello[""]how
hello"e;how hello"e;how
Following is my CSVWrite command:
CALL CSVWRITE(
'#DELTA_CSV_DIR#/DELTA.csv',
'SELECT ccc from temptemp',
null, '|', '');
Am I doing this wrong? or is there any option or workaround I can use to avoid this situation?
Thanks in advanced.
You are currently using the built-in CSVWRITE function with the following options:
fileName = '#DELTA_CSV_DIR#/DELTA.csv'
query = 'SELECT ccc from temptemp'
characterSet = default (UTF-8)
fieldSeparator = '|'
fieldDelimiter = '' (empty string)
As documented, the default escape character is a double quote, so that double quotes are escaped using a double quote (in the same way as you need to escape a backslash within a Java string with a backslash). The escape character is needed to escape the field separator.
You can disable the escape character as follows:
CALL CSVWRITE(
'#DELTA_CSV_DIR#/DELTA.csv',
'SELECT ccc from temptemp',
'fieldSeparator=| fieldDelimiter= escape=');
This is also using the more readable new format for options.