I am copying the data into snowflake table which has three columns: ID, DATA and ETL_LOAD_TIMESTAMP.
I have a column ETL_LOAD_TIMESTAMP in snowflake of type TIMESTAMP_TZ(9) and I have set its default value as CURRENT_TIMESTAMP().
I get my data from a CSV file, which is of type:
ID, DATA
1, Dummy
I download the csv file at tmpdir location on local. I load the data of this csv into snowflake as:
create_cmd = "CREATE TEMPORARY STAGE teamp123 COMMENT = 'TEMPORARY STAGE FOR TEST_TABLE1 DATA LOAD'"
self.connection.execute("ALTER SESSION SET TIMEZONE = 'UTC';")
self.connection.execute(create_cmd)
self.connection.execute(f"put file://tmpdir/* #temp123 PARALLEL=8")
self.connection.execute("COPY INTO TEST_TABLE1 FROM #temp123 PURGE = TRUE FILE_FORMAT = (TYPE = 'CSV' field_delimiter = ',' FIELD_OPTIONALLY_ENCLOSED_BY = '\"' ESCAPE_UNENCLOSED_FIELD = None error_on_column_count_mismatch=false SKIP_HEADER = 1)")
I get the values of ID and Data but the ETL_LOAD_TIMESTAMP is null.
How do I modify this copy command so that I get the default value of ETL_LOAD_TIMESTAMP which is current timestamp instead of null?
you can use default current_timestamp() while defining datatypes or explicit to_timestamp
https://docs.snowflake.com/en/user-guide/data-load-transform.html#current-time-current-timestamp-default-column-values
Related
I am trying to load a CSV file from S3. which has a null value in the integer type data field in the snowflake table.
So I try to use IFFNULL function but gets the error.
Numeric value 'null' is not recognized.
For example when I try
select IFNULL(null,0)
I get the answer as 0.
but the same thing when I try while reading the CSV file won't work
select $1,$2,ifnull($2,0)
from
#stage/path
(file_format => csv)
I get the null not recognized Error.
and it fails when $2 is null.
My csv format is as below.
create FILE FORMAT CSV
COMPRESSION = 'AUTO' FIELD_DELIMITER = ','
RECORD_DELIMITER = '\n' SKIP_HEADER = 0
FIELD_OPTIONALLY_ENCLOSED_BY = '\042'
TRIM_SPACE = FALSE
ERROR_ON_COLUMN_COUNT_MISMATCH = TRUE ESCAPE = '\134'
ESCAPE_UNENCLOSED_FIELD = '\134' DATE_FORMAT = 'AUTO'
TIMESTAMP_FORMAT = 'AUTO' NULL_IF = ('\\N');
Basically, I am just trying to convert null to 0, when reading from the stage.
The null string literal could be handled by setting NULL_IF:
CREATE FILE FORMAT CSV
...
NULL_IF = ('null', '\\N');
I used the second option listed in the Snowflake documentation specifying FIELD_OPTIONALLY_ENCLOSED_BY=NONE and EMPTY_FIELD_AS_NULL = FALSE in which case I'd need to provide a value to be used for NULLs (NULL_IF=('NULL')
https://docs.snowflake.com/en/user-guide/data-unload-considerations.html
"Leave string fields unenclosed by setting the FIELD_OPTIONALLY_ENCLOSED_BY option to NONE (default), and set the EMPTY_FIELD_AS_NULL value to FALSE to unload empty strings as empty fields.
If you choose this option, make sure to specify a replacement string for NULL data using the NULL_IF option, to distinguish NULL values from empty strings in the output file. If you later choose to load data from the output files, you will specify the same NULL_IF value to identify the NULL values in the data files."
So my query looked something like the following:
COPY INTO #~/unload/table FROM (
SELECT * FROM table
)
FILE_FORMAT = (TYPE = 'CSV' COMPRESSION = 'GZIP'
FIELD_DELIMITER = '\u0001'
EMPTY_FIELD_AS_NULL = FALSE
FIELD_OPTIONALLY_ENCLOSED_BY = NONE
NULL_IF=('NULL'))
OVERWRITE = TRUE;
When I try to load the data below from a stage file it fails with invalid date, is there a way to resolve this issue? Without changing the source file.
I am trying to setup a Snowpipe
Orig_Int_Date
04-21-2020
create or replace file format Ally_format
type = csv
field_delimiter = '|'
skip_header = 1
empty_field_as_null = true
REPLACE_INVALID_CHARACTERS = TRUE
DATE_FORMAT = '<MM-DD-YYYY>'
EMPTY_FIELD_AS_NULL = TRUE;**
Copy into NAM_FIN_DB.FIN_PUBLIC.ALLY
from #NAM_FIN_DB.PUBLIC.FP_FINANCE
file_format = Ally_format
pattern='ALLY.*';**
I think your date format line should be:
DATE_FORMAT = 'MM-DD-YYYY'
not
DATE_FORMAT = '<MM-DD-YYYY>'
I am trying to load a CSV file into Snowflake. The sample format of the input csv table in s3 location is as follows (with 2 columns: ID, Location_count):
Input csv table
I need to transform it in the below format:(with 3 columns:ID, Location, Count)
Output csv table
However when I am trying to load the input file using the following query after creating database, external stage and file format, it returns LOAD_FAILED
create or replace table table_name
(
id integer,
Location_count variant
);
select parse_json(Location_count) as c;
list #stage_name;
copy into table_name from #stage_name file_format = 'fileformatname' on_error = 'continue';
you will probably need to parse_json that 2nd column as part of a copy-transformation. For example:
create file format myformat
type = csv field_delimiter = ','
FIELD_OPTIONALLY_ENCLOSED_BY = '"';
create or replace stage csv_stage file_format = (format_name = myformat);
copy into #csv_stage from
( select '1',
'{"SHS-TRN":654738,"PRN-UTN":78956,"NCT-JHN":96767}') ;
create or replace table blah (id integer, something variant);
copy into blah from (select $1, parse_json($2) from #csv_stage);
I am trying to load .csv file data to Snowflake table and using following command
COPY INTO MYTABLE
FROM #S3PATH PATTERN='.*TEST.csv'
FILE_FORMAT = (type = csv skip_header = 1) ON_ERROR = CONTINUE PURGE=TRUE FORCE=TRUE;
Following scenario I am seeing
1) if even one column of the table is numeric it will throw error
Numeric value '""' is not recognized
2) if i change all the columns data type to varchar, then it will load the data but it will populate
all the columns data with "" double quotes ( instead of 15 , "15")
Thanks in advance for your response!
You're likely missing FIELD_OPTIONALLY_ENCLOSED_BY = '\042' in your file_format. Add that in and try.
https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html#type-csv
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html
Thanks CodeMonkey!
One issue is solved
current scenario:
One column is defines as " NUMBER" in SF table and if the csv file has a value populated for that columns then those were the only rows loaded in the table. basically if the numeric column in csv file is null (or blank) those record as not loaded.
also tried using
EMPTY_FIELD_AS_NULL = TRUE
still the same result as above.
"first_error" message: Numeric value '' is not recognized
here is what i did and it is working
FILE_FORMAT = (type = csv field_delimiter = ',' skip_header = 1 FIELD_OPTIONALLY_ENCLOSED_BY = '\042' EMPTY_FIELD_AS_NULL = TRUE NULL_IF = ('NULL','null','')) ON_ERROR = CONTINUE PURGE=TRUE FORCE=TRUE;
I'm trying to load the entirety of each row in a csv file into a variant column.
my copy into statement fails with the below
Error parsing JSON:
Which is really odd as my data isn't JSON and I've never told it to try and validate it as json.
create or replace file format NeilTest
RECORD_DELIMITER = '0x0A'
field_delimiter = NONE
TYPE = CSV
VALIDATE_UTF8 = FALSE;
with
create table Stage_Neil_Test
(
Data VARIANT,
File_Name string
);
copy into Stage_Neil_Test(Data, File_Name
)
from (select
s.$1, METADATA$FILENAME
from #Neil_Test_stage s)
How do I stop snowflake from thinking it is JSON?
You need to explicitly cast the text into a VARIANT type, since it cannot auto-interpret it as it would if the data were JSON.
Simply:
copy into Stage_Neil_Test(Data, File_Name
)
from (select
s.$1::VARIANT, METADATA$FILENAME
from #Neil_Test_stage s)