I want to restore my table with BCP by code in the below.
BCP framework.att.attendance in "D:\test\mhd.txt" -T -c
But the column (id) is identity in this table.
When data is restored with BCP I want id columns to be unchanged.
In other words, if the id of the first row is '7' before BCP, I want to import data and the id of the first row will be still be '7'.
What should I do?
BCP IMPORT
-E
-E Specifies that identity value or values in the imported data file are to be used for the identity column.
If -E is not given, the identity values for this column in the data file being imported are ignored.
Related
I have a data set and want to import it into my database with the condition:
In case there is a record that cannot be imported, it can be extracted into a log
Although existing records can not be imported but still allow import of records that can be imported (other records) and continue to process
Currently I use the BCP utility to import data into the table from the csv file with:
bcp table_name IN C:\Users\09204086121\Desktop\data.csv -T -c -o C:\Users\09204086121\Desktop\logOut.log -e C:\Users\09204086121\Desktop\errOut.log
It just satisfies my condition 1 above.
I need that when the record has error (duplicate primary key,...), write to log (1) and continue to insert into the table the other normal records (2).
I came up with the idea that combining trigger with bcp, after creating a trigger and adding the parameter -h "FIRE_TRIGGERS" to the bcp statement, the insert will ignore records that have the same key but it won't write to the log.
This is my trigger.
ALTER TRIGGER [PKGORDERCOMMON].[T_ImportData] ON [PKGORDERCOMMON].[IF_R_BUNRUI1]
INSTEAD OF INSERT
AS
BEGIN
--Insert non duplicate records
INSERT INTO [IF_R_BUNRUI1]
(
SYSTEM_KB,
BUNRUI1_CD,
BUNRUI1_KANJI_NA,
BUNRUI1_KANA_NA,
CREATE_TS
)
SELECT SYSTEM_KB,
BUNRUI1_CD,
BUNRUI1_KANJI_NA,
BUNRUI1_KANA_NA,
CREATE_TS
FROM inserted i
WHERE NOT EXISTS
(
SELECT *
FROM [IF_R_BUNRUI1] c
WHERE c.BUNRUI1_CD = i.BUNRUI1_CD
AND c.SYSTEM_KB = i.SYSTEM_KB
);
END;
Is there anyone who can help me.
BCP is not meant for what you are asking it to do (separate good and bad records). For instance, bcp -e option has a limit to how many records it will show. Im not sure if this limit is tied to the "max errors" option, but regardless there is a limit.
Your best option is to load all the records and address bad data in t-sql.
Load all records in such a way to ignore conversion errors. Either:
load entire line from file into a single, large varchar column. Parse out columns and qc data as needed.
or
load all columns from source file into generic varchar columns with large enough size to accomodate your source data.
Either way, when done, use t-sql to inspect your data and split among good/bad records.
I'm using BCP because I want to export some columns from a table:
bcp "SELECT csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn FROM [DbJamaica].[dbo].[CS_Usage]" queryout "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
Here I exclude my primary key, now I want to import this txt and I want that my primary key will be auto generate:
bcp DbJamaica.dbo.CS_Usage out "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
I have always format not valid why? I also used format file but I have the same error.
This is my question: how to exclude identify column or some columns?
You can't, as far as I know. Bcp basically bulk inserts blindly into a table, if the columns don't match, you get an error. What you can do, though, is create a staging table like:
SELECT TOP 0 csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn
INTO [DbJamaica].[dbo].[CS_Usage_TEMP]
FROM [DbJamaica].[dbo].[CS_Usage]
Then you can use bcp on your staging table:
bcp DbJamaica.dbo.CS_Usage_TEMP out "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
Next you can insert the data from the staging to the actual table, where you set the PK column as IDENTITY:
INSERT INTO DbJamaica.dbo.CS_Usage (csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn)
SELECT csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn
FROM DbJamaica.dbo.CS_Usage_TEMP
And finally cleanup:
DROP TABLE DbJamaica.dbo.CS_Usage_TEMP
Your source file should match the target table structure. Which means, the number of columns in the table should match the number of column in your csv/txt (source file).
In your case, even if your PK column is the identity column you must have the column in the source file. SQL server will take care of identifying the column as identity and it will ignore the values you put there. So you can either have any value or not, your bcp will work.
It's a different use case if you want to retain the identity values. Refer to -E argument in the documentation (bcp utility)
We are working on to import data from MS SQL Server to hive through Sqoop. If we use the incremental & append mode which is the requirement then we need to specify the --last-value of the row id which we inserted last time.
I have to update about 100 tables into Hive.
What is the practice to save the value of row id for all tables and specify in the sqoop --last-value command ?
Why does not Sqoop itself check the row id of the source & destination table, finally update the rows onwards the last row id value of the destination table?
If i save the last value of row id for all tables in a hive table and want to use those values in Sqoop job then how it's possible?
All and above, i want to automate the data importing job so that i do not have to provide the value manually for each table data import per day
Any pointers ?
Thanks
I want to import my Excel workbook with single worksheet into SQL server, but after trying Import and Export Data I found a problem with my source document. My excel document is multilined, so when I trying to import that wizard wants to imports each row in each column but I want to import my data with IDs and insert every rows with single ID in once columns.
How can I do this?
please look at my sample picture, hope helpful for understanding what I want to do.
excel to sql problem
I would manipulate the excel data to make it database friendly. To do this, I would add an extra worksheet and copy the data from the first sheet to the new sheet.
Then, since you want IDs in every row in col A, I would change the value in col A (of the new sheet) to a formula that copies the value from the same cell in the first sheet, unless it is blank, in which case, copies the value from the cell above the cell with the formula.
Something like: =IF(ISBLANK(Sheet1!A2), A1, Sheet1!A2)
This will give you a column of IDs with every row having a value. The Import and Export data process should work happily now.
with bcp we can insert data intotables use the following queries
exec xp_cmdshell 'bcp MyTasks.dbo.emp out d:\f\yes.xls -T -c -U nagababu -P Test123 '
exec xp_cmdshell 'bcp "SELECT * FROM MyTasks.dbo.emp" queryout d:\f\PersonData_n.xls -c -S (local) -T -L 1'
exec xp_cmdshell 'bcp MyTasks.dbo.emp format nul -c -t, -f d:\f\EmployeeData_n.xls -S (local) -T'
we have to use in instead of out while importing data from sheet to table
use it to import daata to table
exec xp_cmdshell 'bcp MyTasks.dbo.emp in d:\f\yes.xls -T -c -U nagababu -P Test123 '
I have an extremely large database I need to send to the developer, the table has over 120million rows. the developer says he only needs about 10,000 or so rows so I was going to use the sqlcmd -S -d -Q "select top 10000 * from table" -s "," -o "C:\temp\filename.csv"
I decided rather than truncate immediately I would script out the table, rename and test bulk inserting, I tried using
bulk insert tablename from 'c:\temp\filename.csv'
with (
fieldterminator = ',',
rowterminator = '\n'
)
this ends in "Bulk load data conversion error (truncation) for row 1..." error. I also tried in import/export wizard and it fails for the same problem (truncation). increasing the size of the field lengths, solves the problem but I really am having problems understanding why I need to do this. Its the same data from the same table, it should bulk insert right back in?!?
also the problem is happening on every column in the table and by varying lengths, so there is no column with the same number of chars I have to add. all the columns are varchar data type. could the sqlcmd be inserting some kind of corruption in the file? I have tried to look for a problem, I also tried rtrim(ltrim(columname)) to make sure there is no whitespace but I'm not sure this is how it works. I'm using sql server 2012 if this helps.
thanks
You should look into BCP Queryout and BULK INSERT options. Use NATIVE format if you're going from SQL to SQL.
(BCP is command-line):
bcp "select top(10000) * from table" queryout "OUTPUTFILENAME.DAT" -S serverInstanceName -d databaseName -T -n
The Bulk Insert command is SQL (not command line):
bulk insert table from 'path\and\OUTPUTFILENAME.DAT' with (keepidentity,datafiletype = 'native');
(If the table doesn't have an identity, you can eliminate keepidentity,