I am using Microsoft SQL Server Management studio and I am currently importing some CSV files in a database. I am importing the CSV files using the BULK INSERT command into already existing tables, using the following query.
BULK INSERT myTable
FROM >>'D:\myfolder\file.csv'
WITH
(FIRSTROW = 2,
FIELDTERMINATOR = ';', --CSV Field Delimiter
ROWTERMINATOR = '\n', -- Used to shift to the next row
ERRORFILE = 'D:\myfolder\Error Files\myErrrorFile.csv',
TABLOCK
)
This works fine for me thus far, but I would like to automate the process of naming columns in tables. More specifically I would like to create a table and use as column names, the contents of the first row of the CSV file. Is that possible?
The easiest way I can think of is:
right-click on the database, select: Tasks -> Import Data...
After that, SQL Server Import and Export Wizard will display. There you have everything to specify and custom settings on importing data from any sources (such as getting column names from first row in a file).
In your case, your data source will be Flat file source.
Related
I receive update information for items on a daily basis via a CSV file that includes date/time information in the format YYY-MM-DDThh:mm:ss
I used the Management Studio task "Import Flat File..." to create a table dbo.fullItemList and import the contents of the initial file. It identified the date/time columns as type datetime2(7) and imported the data correctly. I then copied this table to create a blank table dbo.dailyItemUpdate.
I want to create a script that imports the CSV file to dbo.dailyItemUpdate, uses a MERGE function to update dbo.fullItemList, then wipes dbo.dailyItemUpdate ready for the next day.
The bit I can't get to work is the import. As the table already exists I'm using the following
BULK INSERT dbo.dailyItemUpdate
FROM 'pathToFile\ReceivedFile.csv'
WITH
(
DATAFILETYPE = 'char',
FIELDQUOTE = '"',
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n',
TABLOCK
)
But I get a "type mismatch..." error on the date/time columns. How come the BULK INSERT fails, even though the data type was picked up by the "Import Flat File" function?
I am migrating my data from SQL Server to Hive using following steps but there is data issue with the resulting table. I tried various options including checking datatype, Using csvSerde but not able to get data aligned properly in respective columns. I followed following steps:
Export SQL Server data to flat file with fields separated by comma.
Create external table in Hive as given below and load data.
CREATE EXTERNAL TABLE IF NOT EXISTS myschema.mytable (
r_date timestamp
, v_nbr varchar(12)
, d_account int
, d_amount decimal(19,4)
, a_account varchar(14)
)
row format delimited
fields terminated by ','
stored as textfile;
LOAD DATA INPATH 'gs://mybucket/myschema.db/mytable/mytable.txt' OVERWRITE INTO TABLE myschema.mytable;
There is issue with data with all combination I could try.
I also tried OpenCSVSerde but the result was worse than simple text file. I also tried by changing delimiter to semicolon but no luck.
row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
with serdeproperties ( "separatorChar" = ",") stored as textfile
location 'gs://mybucket/myschema.db/mytable/';
Can you please suggest some robust approach so that I don't have to deal with data issue.
Note: Currently I don't have option of connecting my SQL Server table with Sqoop.
I have CSV file on Azure blob storage. It has 4 columns in it without headers and one blank row at starting. I am inserting CSV file into SQL managed instance by bulkinsert and I have 5 columns in the database table. I don't have 5th column in CSV file.
Therefore it is throwing this error:
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 5 (uId2)
As I want to insert that 4 columns from CSV file to table in database and I want that 5th column in table as NULL.
I am using this code:
BULK INSERT testing
FROM 'test.csv'
WITH (DATA_SOURCE = 'BULKTEST',
FIELDTERMINATOR = ',',
FIRSTROW = 0,
CODEPAGE = '65001',
ROWTERMINATOR = '0x0a'
);
Want that 5th row as NULL in database table, if there are 4 columns in CSV file.
Sorry, we achieve that in bulk insert. None of other ways according my experience.
Azure SQL managed instance is also not supported as dataset in Data Factory Data flow. Otherwise we can using Data Flow derived column to create a new column to mapping to the Azure SQL database.
The best way is that you editor your csv file: just add new column as header in you csv files.
Hope this helps.
I want to import flat file using SQL Server query. Flat file is PIPE delimited with one HEADER row and one FOOTER row.
Flat file sample is as follows,
H|201501204|01
1|abc|123
2|efg|456
4|hij|789
T|03
in above file H and T are fixed defining HEADER and FOOTER respectively, '201501204' will be the date, '01'will be the flat file number, '03'will be count of data rows in flat file.
I have tried using Bulk Insert but I am losing one Data Row while importing Flat file.
I am using FIRSTROW=2 and LASTROW=2651, number of data rows in my Flat file are 2650 but after importing i am getting only 2649 rows in table.
Use the BULK INSERT statement (see https://msdn.microsoft.com/en-us/library/ms188365.aspx), the option you want is FIRSTROW and set that to 2 (the 2nd row in the file). You can also use LASTROW.
Alternatively load the data into a staging table first - single column and then split it in SQL Server having excluded the H and T by WHERE LEFT( input_line, 1 ) not in ( 'H', 'T' )
How can I import CSV file data into SQL Server 2000 table? I need to insert data from CSV file to table twice a day. Table has more then 20 fields but I only need to insert value into 6 fields.
i face same problem before i can suggest start reading here. The author covers:"This is very common request recently – How to import CSV file into SQL Server? How to load CSV file into SQL Server Database Table? How to load comma delimited file into SQL Server? Let us see the solution in quick steps."
I need to insert data from CSV file to table twice a day.
Use DTS to perform the import, then schedule it.
For SQL 2000, I would use DTS. You can then shedule this as a job when your happy with it.
Below is a good Microsoft link explaining how to use it.
Data Transformation Services (DTS)
You describe two distinct problems:
the CSV import, and
the extraction of data into only those 6 fields.
So break your solution down into two steps:
import the CSV into a raw staging table, and
then insert into your six 'live' fields from that staging table.
There is a function for the first part, called BULK INSERT, the syntax looks like this:
BULK INSERT target_staging_table_in_database
FROM 'C:\Path_to\CSV_file.csv'
WITH
(
DATAFILETYPE = 'CHAR'
,FIRSTROW = 2
,FIELDTERMINATOR = ','
,ROWTERMINATOR = '\n'
);
Adjust to taste, and consult the docs for more options. You might also want to TRUNCATE or DELETE FROM your staging table before doing the bulk insert so you don't have any old data in there.
Once you get the information into the database, doing an UPDATE or INSERT into those six fields should be straightforward.
You can make of use SQL Server Integration services(SSIS). It's jusy one time task to create the Package. Next time onwards just run that package.
You can also try Bulk Insert as daniel explained.
You can also try Import export wizard in SQL Server 2000.