I have to load csv file to snowflake using copy command.
These is a timestamp column in csv file, which I need to convert to date and load to snowflake
any idea on the command?
First of all, Snowflake can convert timestamp to date implicitly. If you get any errors (because of your format), please check the following docs:
Convert Data Types During a Load:
https://docs.snowflake.com/en/user-guide/data-load-transform.html#convert-data-types-during-a-load
TO_DATE function:
https://docs.snowflake.com/en/sql-reference/functions/to_date.html
Related
I am trying to get the last refresh date of table using a system function SYSTEM$LAST_CHANGE_COMMIT_TIME
select SYSTEM$LAST_CHANGE_COMMIT_TIME( 'table_name') -- returning a token
when I try
select to_timestamp_LTZ( SYSTEM$LAST_CHANGE_COMMIT_TIME( 'table_name')/1000) -- Invalid date
can anyone please help ?
TIA
You should note that the Snowflake documentation strongly discourages the use of the output of SYSTEM$LAST_CHANGE_COMMIT_TIME as a timestamp. Given that, …
The output of this function is epoch seconds so you just need to convert it to a timestamp e.g.
to_timestamp_LTZ( SYSTEM$LAST_CHANGE_COMMIT_TIME( 'table_name'))
The following instruction aims at using a specific format to import DATEs
alter session set DATE_INPUT_FORMAT = 'YYYY-MM-DD HH24:MI:SS.FF';
However, it seems to have no effect on the following:
copy into schema.table
from s3://bucket/file.parquet
credentials=(aws_key_id='...' aws_secret_key='...')
match_by_column_name=case_insensitive
file_format=(type=parquet);
Which results in errors like below:
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 100071 (22000):
Failed to cast variant value "2020-06-16 00:00:00.000" to DATE
When a column in the imported Parquet file has a format as specified above for a date field.
This really sounds like a bug, as the above COPY INTO scenario should in theory be a typical use case for altering the DATE_INPUT_FORMAT.
Is there a way to address this?
The DATE_INPUT_FORMAT should affect the copy command. The documentation talks about not supporting a timestamp from a variant column on a date conversion.
Although TO_DATE accepts a TIMESTAMP value, it does not accept a TIMESTAMP inside a VARIANT.
I am calling a stored procedure from a data flow task in SSIS in which I am selecting the HOUR datepart of a datetime field. (code below from the stored procedure)
SELECT
DATEPART (HOUR, R.received_date) AS [Hour] ,
CONVERT (VARCHAR(10), R.received_date, 101) AS [Date] ,
COUNT (R.id) AS [NumberofFilings]
And in my data flow task, I have a OLE DB Source task in which I call the stored procedure:
And when I preview the data with the OLE DB source task, the data looks like I would expect - with the hour column displaying an integer between 0 & 24:
The issue occurs after I export the results to a CSV file and the hour becomes a datetime field where the values become '1/11/1900 0:00' which is not what I'm expecting.
In my flat file destination connection manager, I set the Hour properties to be four-byte signed integer but the hour will not display as an integer but as a datetime.
I've tried other datatypes for the Hour column but nothing will convert this to a single integer / character. Any suggetions?
If you are opening the .csv file in Excel, I suspect that Excel is looking at a column named "Hour" and thinking, "Must be a datetime field. I'll just help my user out and make it so."
Try opening the .csv file in notepad and see what the actual contents look like.
EDIT:
I am unable to reproduce your results. When I follow your steps I get a CSV file that looks like this in notepad:
"Col1","Col2"
"0","04/05/2016"
"0","04/02/2016"
"0","04/01/2016"
...
You must be doing something that you are not including in your description of the issue.
Or maybe your package has gotten corrupted. You could try re-building it from scratch to eliminate that possibility.
But I have tested and proved that what you are trying to do should work.
I have a data file that contains a datetime field in (yyyy mm dd) format.
I have created a bcp format file to import the data but when run the statement, I get an error
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 16 (ReleaseDate).
How can I tell the bcp utility to treat the field in (yyyy mm dd) format or convert the format to that sql server expect ?
I have two comments on the problem.
First, make sure you are using a ASCII code page, not UNICODE which is two bytes.
Second, if BCP is having issues you can play around with the format file.
If that does not work, change from ETL to Extract Load Translate (ELT).
Bulk load from file to a varchar() column in a table. Translate with a stored procedure to the right data type.
Is there any way to check Date(like isDate function in TSQL) column in SSIS package derived column expression after extraction from Sourcefile before loading to target dtabase?
Thanks
there is no built in function but you can run a script task and use vb.net code to check if the column is a date and operate on it as you wish...
I had a similar issue. I had a date/time in a text file, but the field had a dash between the date and the time. I created a derived column to do a replace on the dash:
REPLACE([TimeField], "- ", "")
I then added a convert column to convert the field to a date. I chose to ignore errors. I then added another Derived Column to check if the converted field was NULL, indicating that it could not convert the value. If it was, I had it set to the current date.
There is a a data conversion task you can drop in. Then redirect the rows as needed, either failing the import entirely or redircting the rows that don't work.
Or you could try a conditional split wher eyou cast the field to a date data type and then send the failures along another path (either deleting the records or nulling out the field would be the common action.)
See also http://www.sqlis.com/sqlis/post/Expression-Date-Functions.aspx > "IsDate workaround" for a technique that can be adapted
You can check whether your variable has a date or not using a conditional statement like this:
testDateVariable?true:false
For example, if date > 2 then it is true (and put the date, or format the date as you wish). If it is false, put null (you replace true with the date format and false with null).
All this is in a drived column in SSIS.