I am trying to import csv (1 million records) file to sql server table using bulk insert
Example of the csv data is as below:
"Phone Numbers","Opstype"
"9827259163","D"
"9827961481","D"
"9827202228","A"
"9827529897","A"
"9827700249","A"
Screenshot from Notepad++:
My BULK INSERT code is as below
BULK INSERT dbo.tbl_dnd FROM "C:\DND_1.csv" WITH (DATAFILETYPE='char',FIRSTROW=2,FIELDTERMINATOR='","',ROWTERMINATOR='"\n"');
the result is as below from sql server (example):
Phone Numbers Opstype
1120029090 A" "9868321244
1120220004 A" "9868863100
1120291549 A" "9868843285
1120299308 A" "9868908545
1120314169 A" "9868929393
1120502244 A" "1122230590
1120907054 A" "1126148672
1122029480 A" "9868545672
1122070793 A" "9868541277
1122078765 A" "9968140674
Screenshot from sqlserver table:
Please also note that total records in sql server table after bulk insert is 500,000 records only. (Half the expected)
What am I doing wrong?
I am expecting to see only letters in Opstype without numbers and quotes
Your data contains CR LF as the row terminator.
Your BULK INSERT statement only has LF (\n).
You need '\r\n' as the row terminator.
(And to remove the " from your field and row terminators.)
Related
My actual Excel has about 1,000 rows/records and up to 5 comma separated tags per record in a "TAGS" column. Very open to any and all suggested solutions besides manually populating the TEXTs&TAGs tbl.
The Excel & SQL Server resemble this:
Excel:
TEXT TAGS
--------------------------------
1.derivatives math, calculus
2.triangles math, geometry
Database:
TEXTs tbl
1.derivatives
2.triangles
TAGs tbl
1.math
2.calculus
3.geometry
4.science
TEXTs&TAGs tbl (many to many)
1.1,1
2.1,2
3.2,1
4.2,3
You can use ADODB (tutorial, another one) to connect to SQL Server from your VBA macro in Excel. Then just insert records one by one like this. It won't by very fast, but should be OK for your needs.
Sub update()
For I = 1 To 100
txt = Sheet1.Cells(I, 1)
txt_id = insertAndGetIdForText(txt)
tags = Split(Sheet1.Cells(I, 2), ",")
For Each Tag In tags
tag_id = getIdForTag(Tag) #insert if not in DB, otherwise just return id
Call insertTextTag(txt_id, tag_id)
Next Tag
Next I
End Sub
I'm using tablediff utility to transfert data from serval databases sources to a destination database and I get a result having all the differences between the source and destination databases with something like this
Dest. Only N'1027' N'799' N'91443' N'1'
Mismatch N'103A' N'799' N'13010' N'1' DATE_CURRENT DATE_OPERATION MATRICULE_UTILISATEUR QTE QTE_FINAL QTE_INIT QTE_OPERATION REFERENCE_DOCUMENT TYPE_DOCUMENT
Src. Only N'103A' N'310' N'30129' N'1'
so the generated sql file contain delete the Dest. Only rows, update the Mismatch rows and insert the Src. Only rows
My question is: Is there any way using tablediff to get the result of only Mismatch and Src. Only rows??
In the end of your tablediff tool command add the following
-dt -et DiffResults
It will drop an existing table with name DiffResults and create a new one in the destination server and database.
Then you can query the DiffResults table to get the desired rows.
In my test I run the following
SELECT * FROM DiffResults
WHERE MSdifftool_ErrorDescription in ('Mismatch','Src. Only')
or
SELECT * FROM DiffResults
WHERE MSdifftool_ErrorCode in (0,2) -- 0 is for 'Mismatch'; 1 is for 'Dest. Only' and 2 is for 'Src. Only'
Some more details can be found here - https://technet.microsoft.com/en-us/library/ms162843.aspx
If you want to use results from command line, you can pipe output with findstr:
tablediff <your parameters> | findstr /i "^Mismatch ^Src"
I'm using the following BULK INSERT command
BULK INSERT dbo.A
FROM 'd:\AData.csv'
WITH (FIELDTERMINATOR = ',',ROWTERMINATOR = ',\n',FIRSTROW = 2)
to process the data shown. My import skips the first row but also skips the second row. In this case i believe this is because my header and data rows have different delimiters,the data rows have a training comma.
DATASET 1
Trial,Timestep,Column1 - line 1
1,0,0,- line 2
1,1,0.00687237750794734, - line 3
1,2,-0.00190074803257245,- line 4
The import works with this data (note the comma on line 1)
DATASET 2
Trial,Timestep,Column1, - line 1
1,0,0,- line 2
1,1,0.00687237750794734, - line 3
1,2,-0.00190074803257245,- line 4
Is there a way to tweak the parameters of the BULK INSERT command to handle DATASET1 without using a custom formatting file?
Delete header row from your file and you should be good to go.
Your data rows have a comma at the end, but your header row doesn't. Get rid of the last commas in the data rows and try again.
my issue is that when i try to access at the database after a change (insert, update...) the query return the old data.
I explain further:
i have two datatables:
- the first one is filled with the current month datas as per below:
tbastorico.Connection.ConnectionString = "Data Source=(LocalDB)\v11.0;AttachDbFilename=C:\Users\user\Documents\Visual Studio 2012\Projects\MC Capital\MC Capital\App_Data\MC Capital Ltd.mdf;Integrated Security=True"
tbastorico.InsertQuerydDaTabellone(riga.IDcliente, riga.Cognome, riga.SubAm, ......)
after loading the data, when i call the select query on the first table returns 0 rows as the table was empty. But the data are in the mdf file.
Can you help me?
Am having 6 fields in the record set
I want to insert into the table? How can I insert.
Used Query
INSERT INTO table values (recordset (0), recordset (1) ….. recordset (6))
But It showing Undefined function “recordset”
Please how to insert record set value in to the table?
Dim cmdCommand As New ADODB.Command
If recordSet.EOF = False Then
recordSet.MoveFirst
cmdCommand.CommandText = "insert into table ( field1) values ( " + recordSet.Fields(0) + ")"
cmdCommand.Execute()
recordSet.MoveNext
Loop Until recordSet.EOF = True
Keep in mind that you will need quotes in varchar fields
Konstantinos gives a good answer, but you just need to be careful of the potential for SQL Injection issues.
The simplest fix would be to just replace any single apostrophes with two.
.CommandText = "insert into table (field1) values ( '" & Replace(recordSet.Fields(0), "'", "''") & "')"
It sounds like you're attempting to INSERT multiple records using a single statement (one INSERT for 6 records instead of 6 INSERT for 1 record each). If that's the case, the INSERT syntax fully supports it.
Here's an example:
INSERT INTO MyTable
(row1, row2)
SELECT
value1, value2
FROM
OtherTable
When you look at it like this, you can basically take any SELECT statement and put the INSERT clause on top of it. Of course, the column names need to line up correctly for it to work.
You may insert data into database using the codeigniter Framework.
Database
Table Name : user
Fields Name : name , email
Insert a record using codeigniter:
//storing data **record** into an **Array**
$data(
'name' => 'Rudra',
'email' => 'rud.test#gmail.com',
)
//Inserting Array into Database Table Name : **user**
$this->db->insert('user',$data);
Try with the recordsetname.fields("field_name") and check that the recorsetname variable was defined. You can also use the syntax recordsetname!field_name.