Hello Techies,
Thanks for your time in reading this and trying to help.
I have a peculiar situation where I have a CSV file with data (separated by commas) in an external stage. I have snowflake stored procedure, that accesses, reads this file, loads the data into a snowflake table.
Everything is good except, Some of the data from the file gets automatically converted into Uppercase.
My T-SQL in the procedure does not do any Transformation or Conversion of data to upper case.
The Optional parameters in the Snowflake File Format for CSV file type does not have any parameters to preserve the proper case or current case of data in the file.
Please help
Thanks in advance.
Regards,
Sathya
Related
We have staged the log files in external stage s3.The staged log files are in CEF file format.How to parse CEF files from stage to move the data to snowflake?
If the files have a fixed format (i.e. there are record and field delimiters and each record has the same number of columns) then you can just treat it as a text file and create an appropriate file format.
If the file has a semi-structured format then you should be able to load it into a variant column - whether you can create multiple rows per file or only one depends in the file structure. If you can only create one record per file then you may run into issues with file size as a variant column has a maximum file size.
Once the data is in a variant column you should be able to process it to extract usable data from it. If there is a structure Snowflake can process (e.g. xml or json) then you can use the native capabilities. If there is no recognisable structure then you'd have to write your own parsing logic in a stored procedure.
Alternatively, you could try and find another tool that will convert your files to an xml/json format and then Snowflake can easily process those files.
I have a Azure Logic App in which I am executing the SQL Select queries and converting the output of SQL Queries in CSV files and Mail them. The SQL view contains column names with special characters such as +,%,() or whitespaces. But after creating the CSV file from the Query result the column are replaced with unknown values in place of special characters. Example -
a column name "total(value + value)" is replace in CSV with
"total_x005c_value_x0027_value_x005c_". Please help me with this I want the exact column names in CSV file also.
Thanks.
You get this it is because the source data in sql server is UCS-2. And in logic app it doesn't support to decode it.
The simplest way is set the custom header when you create the csv table.
However if you have multiple columns it will be hard to solve it. So you could use Azure Function to convert the json data. You pass the data to HTTP trigger Function, convert it and return data to the logic app and create the CSV.
I'm into a task of importing a CSV file to SQL server table. I'm using bcp tool as my data can be large. The issue im facing with bcp is that the table where I'm gonna import CSV into can have a mix of data types like date, int, etc and if I use bcp using native mode (-n), I will need bcp file as the input but I have CSV file.
Is there any way to convert CSV file into bcp file? or
How can I import a CSV file into SQL server table given that my table columns can have any data type and not just character types?
Had it been that all columns are of character type, i would have used bcp tool with -c option.
Actually... the safest thing to do when importing data, especially when it ins bulk like this, is to import it into a staging table first. In this case where all of the fields are string/varchars. That then allows you to scrub/validate the data and make sure it's safe for consumption. Then once you've verified it, move/copy it to your production tables converting it to the proper type as you go. That's typically what I do when dealing with import data.
a CSV file is just a text file that is delimited by commas. With regard to importing text files, there is no such thing as a 'BCP' file. BCP has an option to work with native SQL data (unreadable to the human eye with a text editor), but the default is to just work with text the same as what you have in your CSV file. There is no conversion needed, with using textual data, there is no such thing as a "BCP file". It's just a ascii text file.
Whoever created the text file has already completed a conversion from their natural datatypes into text. As others have suggested, you will save yourself some pain later if you just load the textual CSV data file you have into a "load" table of all "VARCHAR" fields. Then from that load table you can manipulate the data into whatever datatypes you require in your final destination table. Better to do this than to make SQL do implied conversions by having BCP insert data directly into the final destination table.
I have a task where I need to generate a CSV file from data coming out of two views including a Header having hard coded values and a Trailer at the bottom of the CSV file having These fields- Record_Type = E99, Row_count, and Blank field with 190 length.
I'm able to get the desired output file but I am not able to figure out how to retrieve the NO. of rows coming out of the two vies and write it in between the record type and the blank field at the bottom of the CSV as the whole line is trailer with | delimited.
Please help me figure this out.
Thanks.
My suggestion:
I assume you are using the SSIS Package to solve this problem.
Create a SQL Staging table to store the content which you want to export in CSV file. You may use stored procedure to truncate and refill this staging table by executing it. Execute this store procedure through Execute SQL Task in SSIS Package
Use Data Flow Task to export the data from staging table to CSV file. Input will be SQL Staging table and output will be flat file with Comma(,) delimiter.
I hope it will help you
I am using Slowly Changing Dimension task from SSIS 2008 for delta load. Flat file is the input to slowly changing dimension task. I have observed that '--' character from file is converted into ' â€' after delta load.
Input is the flat file and destination is the database table. Flat file contains few strings having '--' character but somehow after inserting this data to table this character is getting converted to 'â€'.
What can be the issue?
Kindly help me to resolve this issue.
Regards,
Sameer K.
In essence you need to scrub these characters from the data. This can be done in several places, but it's a well accepted design pattern to populate from the source file to a staging table where you can scrub the offending characters before bringing it into your slowly changing dimension. It's also possible to scrub the file prior to import, but it's typically easier to work with the data once it's in a database rather than in a flat file. You could also include a derived column task within SSIS to extract these characters one in the SSIS Pipeline, but you would need to manage this column by column which can become difficult to maintain.