I am getting extra double quotes (") when exporting to CSV using "Insert into OpenRowSet" command from a SQL Server stored procedure (this is part of a C# Visual Studio Core2 automation program, so I need this to run without any intervention.)
When I run:
SELECT data1 as DeleteThisLine
FROM tmpExportData
I get the following results:
DeleteThisLine
-----------------------------------------------------
"#HDR","#BATCH",20190611,Date Range: 06/11-06/19/2019
"#HDR","JOURNAL",,
If I right click and save as .csv, the resulting file looks exactly like the above results.
However when I run the "INSERT INTO OPENROWSET" command to export the results into a csv file:
INSERT INTO OPENROWSET('Microsoft.ACE.OLEDB.12.0','Text;Database=\\FileSever229\file\;HDR=YES;H=-1;FMT=Delimited','SELECT * FROM [ExportFile_06_24_120522.csv]')
SELECT data1 AS DeleteThisLine
FROM tmpExportData
It exports the file, but with Extra double quotes at the beginning and end of every line and double quotes around the first 2 fields:
DeleteThisLine
-----------------------------------------------------------
"""#HDR"",""#BATCH"",20190611,Date Range: 06/11-06/19/2019"
"""#HDR"",""JOURNAL"",,"
How do I eliminate these extra quotes?
Background:
This is an unusual situation where I am creating a Batch file that is to be uploaded into an accounting system. It holds all that information in a single cell (data1) as each row in the upload file has a different number of columns and each row has a set of variables of different types. So all the data is in a single column. It was working great, and then these extra quotes appeared. I must have changed something in my "insert into OPENROWSET" command.
Related
I have an issue with commas in the column while exporting SQL Server query results to a .csv file. The data is splitting up to next columns with double quotes, using SSIS.
This is my sample data:
EmpId Location Dept
---------------------------
101 Nyc,AUS It,HR
102 Nyc,AUS It,HR
When exporting this data into a .csv using SSIS, the data is splitting next column though I used text-qualifier double quote (").
Any suggestions will be appreciated
You might want to use another column delimiter, such as ";" instead of "," when exporting it to .csv
When using text qualifier you have to make sure they are applied on the export also.
Open your .csv by notepad++ or any text editor and see if each Location-entry is surrounded by double quotes (")
Every time that I try to import an Excel file into SQL Server I'm getting a particular error. When I try to edit the mappings the default value for all numerical fields is float. None of the fields in my table have decimals in them and they aren't a money data type. They're only 8 digit numbers. However, since I don't want my primary key stored as a float when it's an int, how can I fix this? It gives me a truncation error of some sort, I'll post a screen cap if needed. Is this a common problem?
It should be noted that I cannot import Excel 2007 files (I think I've found the remedy to this), but even when I try to import .xls files every value that contains numerals is automatically imported as a float and when I try to change it I get an error.
http://imgur.com/4204g
SSIS doesn't implicitly convert data types, so you need to do it explicitly. The Excel connection manager can only handle a few data types and it tries to make a best guess based on the first few rows of the file. This is fully documented in the SSIS documentation.
You have several options:
Change your destination data type to float
Load to a 'staging' table with data type float using the Import Wizard and then INSERT into the real destination table using CAST or CONVERT to convert the data
Create an SSIS package and use the Data Conversion transformation to convert the data
You might also want to note the comments in the Import Wizard documentation about data type mappings.
Going off of what Derloopkat said, which still can fail on conversion (no offense Derloopkat) because Excel is terrible at this:
Paste from excel into Notepad and save as normal (.txt file).
From within excel, open said .txt file.
Select next as it is obviously tab delimited.
Select "none" for text qualifier, then next again.
Select the first row, hold shift, select the last row, and select the text radial button. Click Finish
It will open, check it to make sure it's accurate and then save as an excel file.
There is a workaround.
Import excel sheet with numbers as float (default).
After importing, Goto Table-Design
Change DataType of the column from Float to Int or Bigint
Save Changes
Change DataType of the column from Bigint to any Text Type (Varchar, nvarchar, text, ntext etc)
Save Changes.
That's it.
When Excel finds mixed data types in same column it guesses what is the right format for the column (the majority of the values determines the type of the column) and dismisses all other values by inserting NULLs. But Excel does it far badly (e.g. if a column is considered text and Excel finds a number then decides that the number is a mistake and insert a NULL instead, or if some cells containing numbers are "text" formatted, one may get NULL values into an integer column of the database).
Solution:
Create a new excel sheet with the name of the columns in the first row
Format the columns as text
Paste the rows without format (use CVS format or copy/paste in Notepad to get only text)
Note that formatting the columns on an existing Excel sheet is not enough.
There seems to be a really easy solution when dealing with data type issues.
Basically, at the end of Excel connection string, add ;IMEX=1;"
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\\YOURSERVER\shared\Client Projects\FOLDER\Data\FILE.xls;Extended Properties="EXCEL 8.0;HDR=YES;IMEX=1";
This will resolve data type issues such as columns where values are mixed with text and numbers.
To get to connection property, right click on Excel connection manager below control flow and hit properties. It'll be to the right under solution explorer. Hope that helps.
To avoid float type field in a simple way:
Open your excel sheet..
Insert blank row after header row and type (any text) in all cells.
Mouse Right-Click on the head of the columns that cause a float issue and select (Format Cells), then choose the category (Text) and press OK.
And then export the excel sheet to your SQL server.
This simple way worked with me.
A workaround to consider in a pinch:
save a copy of the excel file, modify the column to format type 'text'
copy the column values and paste to a text editor, save the file (call it tmp.txt).
modify the data in the text file to start and end with a character so that the SQL Server import mechanism will recognize as text. If you have a fancy editor, use included tools. I use awk in cygwin on my windows laptop. For example, I start end end the column value with a single quote, like "$ awk '{print "\x27"$1"\x27"}' ./tmp.txt > ./tmp2.txt"
copy and paste the data from tmp2.txt over top of the necessary column in the excel file, and save the excel file
run the sql server import for your modified excel file... be sure to double check the data type chosen by the importer is not numeric... if it is, repeat the above steps with a different set of characters
The data in the database will have the quotes once the import is done... you can update the data later on to remove the quotes, or use the "replace" function in your read query, such as "replace([dbo].[MyTable].[MyColumn], '''', '')"
I am trying to upload a bunch of csv's to SybaseIQ using dbisql's Load command.
My CSV's look like this
"ps,abc","jgh","kyj"
"gh",""jkl","qr,t"
and my Load command and other options are these:
set temporary option ISQL_LOG = '/path/for/log/file';
set temporary option DATE_ORDER = 'MDY';
Load table test (a, b, c)
Format bcp
Strip RTRIM
Escapes OFF
quotes ON
Delimited by ','
I create a '.ctl' file like the one above and then execute it with the following command:
dbisql -c "uid = xxx; pwd = yyyy" -host aaaa - port 1089 -nogui test.ctl
On execution I get the following error:
Non-space text found after ending quote character for an enclosed field.
--(db_RecScanner.cxx 2473)
SQLCODE=-1005014, ODBC 3 state="HY000"
HOWEVER
It works fine with the following csv format
"ps,abc","jgh",kyj
"gh",""jkl",qrt
i.e if the last column doesn't have the double quotes it works. but all my files have double quotes around each element.
Also the following .ctl file
set temporary option ISQL_LOG = '/path/for/log/file';
set temporary option DATE_ORDER = 'MDY';
Load table test (a ASCII(8), b ASCII(7), c ASCII(7))
Strip RTRIM
Escapes OFF
quotes ON
Delimited by ','
is able to insert data as in the first csv sample into the database but it also inserts the quotes and the commas and also messes up the data:
ex: first row in the db would look like
"ps,abc" ,"jgh", "kyj""g
as opposed to
ps,abc jgh kyj
I am going nuts over trying to figure out what the issue is I have read the manual for sybase and dbisql and according to that the first control file should be able to load the data properly, but it doesn't do that. Any help on this will be appreciated. Thanks in advance
Also my table structure is like this:
a(varchar(8)) b(varchar(7)) c(varchar(7))
I have data in the csv file similar to this:
Name,Age,Location,Score
"Bob, B",34,Boston,0
"Mike, M",76,Miami,678
"Rachel, R",17,Richmond,"1,234"
While trying to BULK INSERT this data into a SQL Server table, I encountered two problems.
If I use FIELDTERMINATOR=',' then it splits the first (and sometimes the last) column
The last column is an integer column but it has quotes and comma thousand separator whenever the number is greater than 1000
Is there a way to import this data (using XML Format File or whatever) without manually parsing the csv file first?
I appreciate any help. Thanks.
You can parse the file with http://filehelpers.sourceforge.net/
And with that result, use the approach here: SQL Bulkcopy YYYYMMDD problem or straight into SqlBulkCopy
Use MySQL load data:
LOAD DATA LOCAL INFILE 'path-to-/filename.csv' INTO TABLE `sql_tablename`
CHARACTER SET 'utf8'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
IGNORE 1 LINES;
The part optionally enclosed by '\"', or escape character and quote, will keep the data in the first column together for the first field.
IGNORE 1 LINES will leave the field name row out.
UTF8 line is optional but good to use if names have diacritics, like in José.
I have a CSV I'm trying to import into SQL using SSIS packages through code.
A line might look something like this
321,1234,"SOME MACHINE, MACHINE ACCESSORIES 1 1/2"" - 4"""
In this example they're using a double quote to symbolize inches. They are trying to escape the inches double quote with a double quote. SSIS, however, does not honour this escapism and fails.
Is there anyway I can still use the double quote symbol for inches and escape it within the quoted text?
Many suggestions are to replace the double quote with two single quotes. Is this the only work around or can I use some other escape technique?
I've seen people talk about using the Derived Column transformation but in my case SSIS fails at the Flat File Source step and I therefore cannot get to a derived column transform step.
I'm currently running a script task in the control flow, just before the data flow, to manipulate the Csv with some regex's to cleanup the data.
I need the string to be text qualified with the 2 outer double quotes because of potential commas in the description column.
What can I do about the double quotes within the text qualified string?
Wow, I expected to be able to answer with "Just set the text qualifier", but figured you would have already tried that so I tried it before I answered. Surprise, SSIS doesn't support standard CSV files!
Looks like this is a common complaint. There is one comment in there from Microsoft about some samples that may help; Here is the codeplex project, they mentioned that the Regular Expression Flat File Source sample and the Delimited File Reader Source sample in particular may help -- I'm guessing the Delimited File Reader would be more worthwhile.
I ran into a similar problem yesterday.
We got the csv file that using comma , as delimiter and double quote " as text qualifier, but there is a field that contain double quote within double quote(non-escaped double quote within a string).
After spending half day searching, came up with the solution below:
// load the file into a one dimensional string array.
// fullFilePath is the full path + file name.
var fileContent = File.ReadAllLines(fullFilePath);
// Find double quotes within double quotes and replace with a single quote
var fileContentUpdated = fileContent.Select(
x => new Regex(#"(?<!^)(?<!\,)""(?!\,)(?!$)"
).Replace(x, "'")).ToArray();
// write the string array into the csv file.
File.WriteAllLines(fullFilePath, fileContentUpdated);
I don't see any other way than replace the double quote with something else to avoid the issue.
This answer is not applicable to 2005 as referenced here, but in case someone comes across this while searching and is using 2008, this other question appears to have a working answer: SSIS 2008 and Undouble
There is a workaround if in the File connection you remove the " as text qualifier you can remove all the double quotes later with a derived column expression REPLACE(Item_Name,"\"",""). The downside is that you will need to do it for every field
I didn't find a direct way to achieve this so I wrote a script:
Add a Script component to the workflow (make sure to connect the input arrow or it won't recognize the columns)
Right click on the Script component -> Input Columns, change the column Usage Type to READWRITE
Click Ok
Edit the Script, replace double quotes with two single quotes
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
var descr = Row.Description;
Row.Description = Row.Description.Replace("\"", "''");
}
Probably old news now, but this issue was fixed in SQL Server 2012. I was able to import the same file on a 2012 server that failed on my 2008 server.