In general I know how to import tab delimited file or comma delimited file. One of my client sent me delimited file. Example you can see below. And I don't how to import it without quotes.
"Make","Model","ModelYear","Trim","BodyStyle","Mileage","EngineDescription","Cylinders","FuelType","Transmission"
"Dodge","Stealth","1993","ES","Hatchback","107000","V-6","6-Cylinder","Gasoline",""
"GMC","Envoy XL","2003","SLE","Sport Utility","116000","6cyl","6-Cylinder","Gasoline","Automatic"
Could you guide me how to import and what settings do I need to change in order to import it with import wizard?
Thank you
You need to set the "Text qualifier" to ".
People might say it's a bit krufty, but the easiest way might be to open it as a CSV in excel and then copy/paste it right into your table using Management Studio.
If you are trying to do anything fancier than that, this method won't work. If not, you would be hard pressed to find a faster way to do it.
If the data is very simplistic you can use the SSMS import wizard. Right click the database, select tasks, select Import Data, and has been pointed out identify quotes as the text qualifier.
For a csv, tab delimited file you would change Data Source option to: Flat File Source
From there just answer the questions and follow the prompts.
I will warn you however. Unless the data is very simplistic, this method tends to produce a lot of errors. I almost always end up using SSIS which is a little more hassle and complex but a whole lot more powerful and dependable.
Related
I'm trying to import a CSV file, which includes commas and quotes in the fields, into a SQL Server database. There's about a million questions and topics about it online, but none really works. I've come to understand that when it comes to CSV there are slightly different standards, but SSMS doesn't seem to be able to import either and I feel like there really should be a convenient way.
The files contain free text strings where they use both double-quotations and commas within the fields.
Here's the test CSV file I'm using:
"Value 1","Notes"
""8-pooln" grupp 7:6 To11:13","As extracted"
"""8-pooln"" grupp 7:6 To11:13","With escaped quotes"
"""""""""""8-pooln"""""""""""""""" grupp 7:6 To11:13","With loads of quotes"
I used a 3rd-party program extract the data to CSV. So the first record is how I got it from that program. According to some site you need to escape double-quotes within a field by adding another double-quote, that's what you see at record 2. The last one just contains a lot of them for testing. I also used another application to validate the file as CSV, where 2nd and 3rd records pass.
By using the SSMS Import Wizard I get:
_Value_1_,_Notes_
8-pooln" grupp 7:6 To11:13,As extracted
8-pooln"" grupp 7:6 To11:13,With escaped quotes
8-pooln"""""""""""""""" grupp 7:6 To11:13,With loads of quotes
So double-quotations in the start are all always ignored regardless how many they are. I haven't found any settings that could change this at all.
I've also tried to manually write an SQL command such as:
BULK INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n')
Which gives us:
Value_1,Notes
"Value 1","Notes"
""8-pooln" grupp 7:6 To11:13","As extracted"
"""8-pooln"" grupp 7:6 To11:13","With escaped quotes"
"""""""""""8-pooln"""""""""""""""" grupp 7:6 To11:13","With loads of quotes"
It only recognizes commas and newlines as any type of control character and there doesn't seem to be any additional lines you can add to fix it.
Lastly I found some solution where you can write a "format file", where you can basically define the column delimiter for each column manually. Which would probably work, but I have way over 50 columns for one file and about 20 files.
I also found a possible solution in settings for SSMS Import Wizard but it's for an old version and looks like it no longer exists.
To clarify:
The fields have both commas and double-quotes in them so the double-quotes opening and closing the fields are necessary. I rather not change anything at all (like from double- to single-quotes) as I don't know exactly what the values mean.
There are about 20 files, one with 95000 records and 50+ columns. Creating format-files seems unreasonable.
It's really not that bad formatted files. SSMS intuitively really should be able to import this without any fix. I can maybe live with manually editing the CSV-file to match the standards (as I did with the 2nd record in my test file).
At this point I am just happy with insight of why it just doesn't work or why my problem seem to be unique.
I'm not sure if using SSIS is an option for you, but if so importing data with quotes within the text fields would be fairly easy to do. An overview of this process is below.
Create an OLE DB connection to the SQL Server instance where the destination table is.
This can be done by right-clicking in the Connections Managers window, selecting New Connection... then choosing the OLE DB option. Configure the login credentials and initial catalog for where the data will be loaded to.
Next create a Flat File Connection Manager. For File Name field, navigate to an existing folder and select an example data file. If you don't see the file, change to file extension to all files in file explorer. Choose Delimited for the Format field and check the "Columns names in the first data row" option if this applies to your file. Set the header row delimiter appropriately. Judging by your example, I'm guessing you would use the carriage return/line field combination, which is the {CR}{LF} value.
On the Columns pane, set the row delimiter accordingly, which also appears to be {CR}{LF} from your sample. For column delimiter, use the ,. This will apply to all columns in the file thus you won't need to set this for each field. I couldn't quite tell by your question, but if a , separates all fields then use this option, otherwise type Mixed for the column delimiter. This option may not appear in the drop-down, but tying it will allow you to use different delimiters for each column. More on this follows in the item belows
In the Advanced pane, add names, data types, and lengths for the columns. If you're unsure about what SSIS data types correspond to the SQL Server ones see the mapping table in this link which shows which data types relate to each other. If you used the Mixed option above, here you can set the delimiter in the ColumnDelimiter field for each column. You can type in values here as well. For example, if fields will always be separated by a certain combination of characters this can be used as well.
After the connection manager has been created, create a Data Flow Task and within this add a Flat File Source component. Use the connection manager that you just created for the connection manager of this component.
Next add either an OLE DB or SQL Server destination. I've found that SQL Server destination tend to perform better, but of course this can vary between different environments. Use the OLE DB connection manager that was created for the destination SQL Server instance and map the columns on the Mappings pane. Connect to Flat File Source to the SQL Server Destination and you can now load the data into your table from source file.
If this is something you will be doing on a regular basis, look into setting this up as a SQL Agent job. You can find more details on this process here.
I'm struggling to find a built-in way to redirect empty rows as flat file source read errors in SSIS (without resorting to a custom script task).
as an example, you could have a source file with an empty row in the middle of it:
DATE,CURRENCY_NAME
2017-13-04,"US Dollar"
2017-11-04,"Pound Sterling"
2017-11-04,"Aus Dollar"
and your column types defined as:
DATE: database time [DT_DBTIME]
CURRENCY_NAME: string [DT_STR]
with all that, package still runs and takes the empty row all the way to destination where it, naturally fails. I was to be able to catch it early and identify as a source read failure. Is it possible w/o a script task? A simple derived column perhaps but I would prefer if this could be configured at the Connection Manager / Flat File Source level.
The only way to not rely on a script task is to define your source flat file with only one varchar(max) column, chose a delimiter that is never used within and write all the content into a SQL Server staging table. You can then clean those empty lines and parse the rest to a relational output using SQL.
This approach is not very clean and a takes lot more effort than using a script task to dump empty lines or ones not matching a pattern. It isn't that hard to create a transformation with the script component
This being said, my advise is to document a clear interface description and distribute it to all clients using your interface. Handle all files that throw an error while reading the flat file and send a mail with the file to the responsible client with information that it doesn't follow the interface rules and needs to be fixed.
Just imagine the flat file is manually generated, even worse using something like excel, you will struggle with wrong file encoding, missing columns, non ascii characters, wrong date format etc.
You will be working on handling all exceptions caused by quality issues.
Just add a Conditional Split component, and use the following expression to split rows
[DATE] == ""
And connect the default output connector to the destination
References
Conditional Split Transformation
I have a file without any extension and I need to download data into a SQL Server table.
Here's an example of one string opened in Notepad:
7600 20160701 20160701 20160630 20160630 20160630 ZSO ### 5501 850170371
In Excel it looks like one string without any spaces.
How to break string over columns when I have no comma or something like that?
I recommend you to check SQL Server Import wizard which is smart enough to figure out delimiter for you. You may need to validate column width and type, though.
If that is a regular activity you can create a SSIS package (you can actually create SSIS package at the end of import-export wizard) and can implement advance error handling features.
If you are looking for TSQL solution only please post more data/specification so we can work on that. it can be done in any of the following way:
Bulk Insert
OPENQUERY
OPENDATASOURCE
OPENROWSET
For one-off tasks like this with small datasets and variable data quality, my personal preference is to use Excel to create a script which can be run in SSMS.
To accomplish this I would do the following steps:
Split the data using Excels "Text to columns"
There are similar options here as in SQL Import Wizard as to how columns are calculated (fixed width/delimiters/...) with a pretty intuitive and immediate interface
Copy and paste special Transpose to shift the data into rows
Possibly do some manual data cleanup
Add formula to "scriptify" the values
Create and store script
You can use Excel (there is a function for this in german it is called (text in spalten, should be something like "text in rows") or LibreOffice Calc (it should provide a wizard) for importing the data.
If it looks good in one of these programmes, save it as csv ore another format that you can import to MSSQL.
Hi I need some help in understanding what is the best way to go about this problem I have. I have these text files that are placed in a folder throughout the day so there can me multiple files in the folder, and each file has thousands of rows. I need to import these comma delimited text files to a table in my sql database.
There are supposed to be 9 columns separated by a comma, sometimes the client sends bad data like files such as: "1245,5656,HG,," when its supposed to be "1244,9000,ET,TI,09,V3,,,6.9". I need to do this via VB.net since the other parts of the overall process are in vb.net so I need to modify the code to incorporate the import part.
So my idea is to create a datable with premade columns then import the data to the datatable and somehow catch the bad files and send them to another folder. Is there a way to loop through the folder, import the delimited files to a datatable to confirm the right amount of columns are used and then import them to a DB datatabase table and move the bad data files elsewhere?
I am searching on this site ways to do this as I have never imported text files to sqll via vb.net before but I found another question and I was wondering if that first answer is in the right direction? The question I am referring to:Link
I'm trying to output the results of a SELECT query to a tab delimited text file in Advantage Data Architect. I know I can use the 'Export to' feature to do this, but there are a lot of tables and that is going to take forever. I would rather use the SQL editor, but I found out it does not accept the OUTPUT TO argument, even though that command is part of Sybase SQL.
I would like to do this:
SELECT * FROM tablename;
OUTPUT TO 'C:/ExportDirectory' DELIMITED BY '\t' FORMAT TEXT;
Is there another way?
Maybe someone else knows a clever way to do this, but I don't think this functionality exists natively in the Advantage SQL engine. If you are Delphi developer, one possibility comes to mind. The Advantage Data Architect ships with the source. The file texttable.pas has the bulk of the logic that writes the text file for the export functionality you mentioned.