I have a CSV file I am attempting to create, and the recipient requires a header row. In this header row (and in the data) there is a field that used to be present that was removed. However, they did not remove the column that that held that data, so now, there is an empty column name surrounded by delimiters ("|"). How can I recreate this?
The expected results for the following columns should be:
RxType1|RxType2|RxType3|RxType4|RxType5||DelivID
(There is an empty column between RxType5 and DelivID) and the results would be:
|Rx|OTC|Legend|Generic|Other||Express
I am using SSRS, and have attempted adding an extra pipe the the column header for RxType5 with an empty column behind it, but the CSV seems to generate a header row based on the column names from the stored procedure and not from the RDL data. I have also attempted in the Stored Proc to create the column by using:
Select
'' AS ""
OR
'' AS "|"
but when I refresh the fields in SSRS, it puts that the column is called "ID_" (because a space, no character, or pipe is non-CLS compliant.
Any suggestions on how I can achieve this? Thanks so much :)
Try creating the column with a known name, like SELECT '' AS [RemoveMe], and then just remove that name from the row header text box.
Related
I'm facing the issue in ADF copy activity while loading the CSV data to the snowflake table,
The issue is while loading the CSV file to the snowflake table using the ADF COPY ACTIVITY, it is treating data of a single column as a multiple columns data,
for example: "My brother often watches different cricket shows on different ""screens"", but on the same different platform"
This is the value of single column_A but ADF copy activity is reading as a value for two-column instead of one
i.e col_A=My brother often watches different cricket shows on different ""screens"
col_B= but in the same different platform
But I want this value to be in single-column i,e column_A
column_A="My brother often watches different cricket shows on different ""screens"", but on the same different platform"
Any alternatives I could do for this?
In your source data, the column value contains comma , and double quotes " which are the same as your dataset properties column delimiter and Quote character.
Column delimiter is to separate the column based on the given delimiter value.
If the column value also contains the same delimiter character, the quote character is used to identify the complete value as a single column
Example:
sample data : "1,abc",def
Preview of data in Azure Data Factory dataset:
In your case you have both column delimiter and quotes character within your column value, so it is not identified as a single column but instead separated into 2 columns based on dataset property values (comma , and double-quotes ".)
Your sample data :
"My brother often watches different cricket shows on different ""screens"", but on the same different platform"
To fix this you can change the column delimiter in your source file or replace double quotes within column value with something else.
Example:
At the moment, when I create a stage from a CSV, I need to manually specify the columns by $column_position and then aliasing each.
e.g. CSV headers: employer_code, employer_name...etc.
Select Statement: SELECT $1 employer_code, $2, employer_name... etc.
I've been looking through the options in the documentation and searching for this answer, but I can't seem to find options that will basically take;
every column in the CSV, and
the first row of data as the header value
This seems like it should be something that's available OoTB, but can't see how to work it.
Any clues?
Thx in advance.
Find here a Snowflake that basically does what you are requesting:
https://github.com/Snowflake-Labs/Excelerator/blob/master/SnowflakeExcelAddin_Stored_Procedures.sql
The name of the UDF is create_table_from_file_and_load_work() - it does it's best to guess headers and data types from the staged file.
It allows you to use the first row of the file for the data types, or the name of the columns:
firstColumnFromFirstowFromFile = arrayFirstRow[0].toUpperCase();
// Check to see if the first col is a datatype. If not then its' the column headers
If the type of each column is not provided, the UDF get_datatype() is called - which makes its best guess.
Created tables as below :
source:([id:`symbol$()] ric:();source:();Date:`datetime$())
property:([id:`symbol$()] Value:())
Then i have two .csv files which include two tables datas.
property.csv showing as below :
id,Value
TEST1,1
TEST2,2
source.csv showing as below :
id,ric,source,Date
1,TRST,QO,2017-07-07 11:42:30.603
2,TRST2,QOT,2018-07-07 11:42:30.603
Now , how to load csv file data into each tables one time
You can use the 0: to load delimited records. https://code.kx.com/wiki/Reference/ZeroColon
The most simple form of the function is (types; delimiter) 0: filehandle
The types should be given as their uppercase letter representations, one for each column or a blank space to ignore a column. e.g using "SJ" for source.csv would mean I wanted to read in the id column as a symbol and the value column as a long.
The delimiter specifies how each columns is separated, in your case Comma Separated Values (CSV). You can pass in the delimiter as a string "," which will treat every row as part of the data and return a nested list of the columns which you can either insert into a table with matching schema or you can append on headers and flip the dictionary manually and then flip to get a table like so: flip `id`value!("IS";",") 0: `:test.txt.
If you have column headers as the first row in the csv you can pass an enlisted delimeter enlist "," which will then use the column headers and return a table in kdb with these as the headers, which you can then rename if you see fit.
As the files you want to read in have different types for the columns and are to bed into you could create a function to read them in for examples
{x insert (y;enlist ",") 0:z}'[(`source;`property);("SSSP";"SJ");(`:source.csv;`:property.csv)]
Which would allow you to specify the name of the table that should be created, the column types and the file handle of the file.
I would suggest a timestamp instead of the (depreciated) datetime as it is stored as a long instead of a float so there will be no issues with comparison.
you can use key to list the contents of the dir ;
files: key `:.; /get the contents of the dir
files:files where files like "*.csv"; /filter the csv files
m:`property.csv`source.csv!("SJ";"JSSZ"); /create the mappings for each csv file
{[f] .[first ` vs f;();:; (m#f;enlist csv) 0: hsym f]}each files
and finally, load each csv file; please note here the directory is 'pwd', you might need to add the dir path to each file before using 0:
I want to import *.csv file. There is head row with the column names, and data rows below. The problem is in column description.This column has multi-line text and each line is recognised as the record.
Document has {CR}{LF} for the end of row, and {LF} for the end of line in multi-line text. Like this:
(0)"Name","Description" {CR}{LF}
(1)"John","adsaddsadas" {CR}{LF}
(2)"Mike","dasdsadsdsda
dsadadsdasdsa {LF}
dsadadsadsad {LF}
dasdsadsadsd"{CR}{LF}
(3)"Dave","dsada"{CR}{LF}
It returns an error saying the row (2) is truncated and is missing data
I have selected {CR}{LF} as delimiter, but it still recognises this as 6 records instead of 3, i suppose it, for some reason, recognises {LF} as row delimiter.
Is there anyone who had similar issue here, or knows how to get over this.
Also i want to mention i don't have a lot of experience with this, so i don't know if there is data missing.
In your flat file connection manager make sure that the Header Row Delimiter is set to {CR}{LF}
And That the Row Delimiter is also set to {CR}{LF}
And Check that Description Column has a length of 4000 (to prevent text from getting truncated) and that the last column delimiter is {CR}{LF}
I have data in the csv file similar to this:
Name,Age,Location,Score
"Bob, B",34,Boston,0
"Mike, M",76,Miami,678
"Rachel, R",17,Richmond,"1,234"
While trying to BULK INSERT this data into a SQL Server table, I encountered two problems.
If I use FIELDTERMINATOR=',' then it splits the first (and sometimes the last) column
The last column is an integer column but it has quotes and comma thousand separator whenever the number is greater than 1000
Is there a way to import this data (using XML Format File or whatever) without manually parsing the csv file first?
I appreciate any help. Thanks.
You can parse the file with http://filehelpers.sourceforge.net/
And with that result, use the approach here: SQL Bulkcopy YYYYMMDD problem or straight into SqlBulkCopy
Use MySQL load data:
LOAD DATA LOCAL INFILE 'path-to-/filename.csv' INTO TABLE `sql_tablename`
CHARACTER SET 'utf8'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
IGNORE 1 LINES;
The part optionally enclosed by '\"', or escape character and quote, will keep the data in the first column together for the first field.
IGNORE 1 LINES will leave the field name row out.
UTF8 line is optional but good to use if names have diacritics, like in José.