I’m trying to export a table into excel/csv , but I’m having trouble because of one column, which is long and has been concatenated with delimiter of “char(10) + char(13)” for a new lines . When I copy all the data from sql server management studio and use “save as” csv file, the output gets broken . Every place that there is a use of a new line , the output get stretched to more than 1 row and breaks the columns position.
I also tried using the export wizard ( don’t know if it will make a difference ) but with no success as the export keeps failing on the last step (getting a warning of “potential lost conversion from nvarchar to longtext) with error of “data conversion failed ..”
To allow multiline fields in csv, those fields have to be enclosed in quotes:
123,"multiline
field",456
789,second record,147
If this is not the case in your generated csv you might have to tell the generator to quote the fields.
If the quotes are already there the csv is valid and any decent reader should take care of those multiline fields. Of course, if you open the file in Notepad you'll still see multiple lines per record, which is normal.
To avoid such issues, you need to clean the data by replacing the carriage return (char(13)) and line feed (char(10)) in your SELECT statement using the following query:
SELECT replace(replace([ColumnName], char(10), ''), char(13), '')
FROM [dbo].[yourTableName]
Related
Whenever I try to import a CSV file into sql server with more than one column I get an error (well, nothing is imported). I know the file is terminated fine because it works with 1 column ok if I modify the file and table. I am limiting the rows so it never gets to the end, the line terminator is the correct and valid one (also shown by working when having 1 column only).
All I get is this and no errors
0 rows affected
I've also check all the other various questions like this and they all point to a bad end of file or line terminator, but all is well here...
I have tried quotes and no quotes. For example, I have a table with 2 columns of varchar(max).
I run:
bulk insert mytable from 'file.csv' WITH (FIRSTROW=2,lastrow=4,rowterminator='\n')
My sample file is:
name,status
TEST00040697,OK
TEST00042142,OK
TEST00042782,OK
TEST00043431,BT
If I drop a column then delete the second column in the csv ensuring it has the same line terminator \n, it works just fine.
I have also tried specifying the 'errorfile' parameter but it never seems to write anything or even create the file.
Well, that was embarrassing.
SQL Server in it's wisdom is using \t as the default field terminator for a CSV file, but I guess when the documentation says 'FORMAT = 'CSV'' it's an example and not the default.
If only it produced actual proper and useful error messages...
I am exporting a file that is going to be picked up by another system. To avoid rework in the other system I am trying to match an existing excel csv output exactly. I have a date column in the DB which I want to export as dd/mm/yyy. In the data flow task I have the following SQL as the source where I do the appropriate conversion. If I run this query in ssms I get the right output.
SELECT [Code]
,[Agency_Name]
,[Region_Group]
,CONVERT( varchar(20), [GrossAmtYrly] , 1) GrossAmtYrly
,CONVERT ( varchar(20), [SaleDate] , 103) SaleDate
,[MemberNo]
,[Surname]
,[Scale]
FROM [Land].[Sales]
I then link this to a flat file destination, the column that this is mapped to is set to DT_SR width 20 not text qualified.
But the output file is spitting out a date in format yyyy-mm-dd.
Similarly for the grossamtyrly the old excel generated csv had the amount with commas after each 3 digits, wrapped in ". The output column it is mapped to is DT_SR width 20 with text qualified to true.
The output file for that column is missing the commas for grossamtyrly.
So it seems like my conversions in the SQL are being ignored completely, but I can't work out why.
Thanks in advance for any suggestions!
Using SSIS 2012 - Visual Basic 2010, DB is SQL Server 2012
I'd use a derived column in the data flow to convert it to the format you want. If it's coming in as a text field in format yyyy-mm-dd, you can convert it to dd/mm/yyyy with the following expression:
SUBSTRING(dt,9,2) + "//" + SUBSTRING(dt,6,2) + "//" + SUBSTRING(dt,1,4)
Thanks Custodian, I figure out how to get it to work.
I double clicked on the flow arrow between the tasks and the metadata tab shows the data type of each column. When I first set this up I did access mode as table or view and so date and grossamt were set to DT_DATE and DT_CY, so I think SSIS was implictly converting the column back again to its original type.
Now I couldn't work out how to change them, So I deleted the DB Source and recreated it starting with the SQL Command option, and everything works as expected.
I have data in the csv file similar to this:
Name,Age,Location,Score
"Bob, B",34,Boston,0
"Mike, M",76,Miami,678
"Rachel, R",17,Richmond,"1,234"
While trying to BULK INSERT this data into a SQL Server table, I encountered two problems.
If I use FIELDTERMINATOR=',' then it splits the first (and sometimes the last) column
The last column is an integer column but it has quotes and comma thousand separator whenever the number is greater than 1000
Is there a way to import this data (using XML Format File or whatever) without manually parsing the csv file first?
I appreciate any help. Thanks.
You can parse the file with http://filehelpers.sourceforge.net/
And with that result, use the approach here: SQL Bulkcopy YYYYMMDD problem or straight into SqlBulkCopy
Use MySQL load data:
LOAD DATA LOCAL INFILE 'path-to-/filename.csv' INTO TABLE `sql_tablename`
CHARACTER SET 'utf8'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
IGNORE 1 LINES;
The part optionally enclosed by '\"', or escape character and quote, will keep the data in the first column together for the first field.
IGNORE 1 LINES will leave the field name row out.
UTF8 line is optional but good to use if names have diacritics, like in José.
I extracted some 10 tables in CSV with " as the text qualifier. Problem is my extract does not look right in Excel because of special characters in a few columns. Some columns are breaking into a new row when it should stay in the column.
I've been doing it manually using the management studio export feature, but what's the best extract the 10 tables to CSV with the double quote qualifier using a script?
Will I have to escape commas and double quotes? Best way to do this?
How should I handle newline codes in my columns, we need them for migration to a new system, but the PM wants to open the files and make modifications using Excel. Can they have it both ways?
I understand that much of the problem is that Excel is interpreting the file where a load utility into another database might not do anything special with new line, but what about double quotes and commas in the data, if I don't care about excel, must I escape that?
Many Thanks.
If you are using SQL Server 2005 or later, the export wizard will export the excel file out for you.
Right click the database, select Tasks-> Export Data...
Set the source to be the database.
Set the destination to excel.
At the end of the wizard, select the option to create an SSIS package. You can then create a job to execute the package on a schedule or on demand.
I'd suggest never using commas for your delimiter - they show up too frequently in other places. Use a tab, since a tab isn't too easy to include in Excel tables.
Make sure you never start a field with a space unless you want that space in the field.
Try changing your text lf's into the literal text \n. That is:
You might have:
0,1,"Line 1
Line 2", 3
I suggest you want:
0 1 "Line 1\nLine 2" 3
(assuming the spacing between lines are tabs)
Good luck
As far as I know, you cannot have new line in csv columns. If you know a column could have comma, double quotes or new line, then you can use this SQL statement to extract the value as valid csv
SELECT '"' + REPLACE(REPLACE(REPLACE(CAST([yourColumnName] AS VARCHAR(MAX)), '"', '""'), char(13), ''), char(10), '') + '"' FROM yourTable.
I have a text file (txt) containing formatted text (just line breaks, carriage returns and tabs)
It also contains German language characters.
I want to use the Bulk Insert comment in T-SQL to read in the text file into one field within a database table.
I ran this command:
CREATE TABLE #MyTestTable (
MyData NVARCHAR(MAX)
)
BULK INSERT [#MyTestTable]
FROM 'D:\MyTextFile.txt'
SELECT * FROM #MyTestTable
The problem is that it reads each line of the text file into a new row in the Temp table. I want it to read the whole file (formatting and all) into one row.
Also the German language characters appear to be lost - replaced by a non-printable character default in the Results View.
Anyone any ideas how I can achieve this?
Thanks.
You can use ROWTERMINATOR and CODEPAGE parameters. Default row terminator is '\r\n'. For the CODEPAGE, you need to know encoding of your raw file and default collation of your DB.
BULK INSERT [#MyTestTable]
FROM 'D:\MyTextFile.txt'
WITH (ROWTERMINATOR = '\0',
CODEPAGE = 'ACP')
Also see http://msdn.microsoft.com/en-us/library/ms188365.aspx
Use this:
FIELDTERMINATOR = '|',
ROWTERMINATOR = '\n'
Where | is your column delimiter.
don't use bulk insert. it is made to take one record per line. You need to write code.
Properly handle the transition from you text file to the unicode (nvarchar) in code. bulk insert probably appplied the standard codepage, loosing your characters.
This really cries for some minor programming job - an hour work or so, plus naother testing and as long for running as it takes.