I'm using BCP because I want to export some columns from a table:
bcp "SELECT csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn FROM [DbJamaica].[dbo].[CS_Usage]" queryout "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
Here I exclude my primary key, now I want to import this txt and I want that my primary key will be auto generate:
bcp DbJamaica.dbo.CS_Usage out "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
I have always format not valid why? I also used format file but I have the same error.
This is my question: how to exclude identify column or some columns?
You can't, as far as I know. Bcp basically bulk inserts blindly into a table, if the columns don't match, you get an error. What you can do, though, is create a staging table like:
SELECT TOP 0 csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn
INTO [DbJamaica].[dbo].[CS_Usage_TEMP]
FROM [DbJamaica].[dbo].[CS_Usage]
Then you can use bcp on your staging table:
bcp DbJamaica.dbo.CS_Usage_TEMP out "C:\temp\CS_Usage.txt" /U.. /P.. -c -T
Next you can insert the data from the staging to the actual table, where you set the PK column as IDENTITY:
INSERT INTO DbJamaica.dbo.CS_Usage (csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn)
SELECT csusUsageDate, csusType, csusTrack1, csusTrack2, csusTrack3, csusDateReaded, csusLoggedIn
FROM DbJamaica.dbo.CS_Usage_TEMP
And finally cleanup:
DROP TABLE DbJamaica.dbo.CS_Usage_TEMP
Your source file should match the target table structure. Which means, the number of columns in the table should match the number of column in your csv/txt (source file).
In your case, even if your PK column is the identity column you must have the column in the source file. SQL server will take care of identifying the column as identity and it will ignore the values you put there. So you can either have any value or not, your bcp will work.
It's a different use case if you want to retain the identity values. Refer to -E argument in the documentation (bcp utility)
Related
I have a data set and want to import it into my database with the condition:
In case there is a record that cannot be imported, it can be extracted into a log
Although existing records can not be imported but still allow import of records that can be imported (other records) and continue to process
Currently I use the BCP utility to import data into the table from the csv file with:
bcp table_name IN C:\Users\09204086121\Desktop\data.csv -T -c -o C:\Users\09204086121\Desktop\logOut.log -e C:\Users\09204086121\Desktop\errOut.log
It just satisfies my condition 1 above.
I need that when the record has error (duplicate primary key,...), write to log (1) and continue to insert into the table the other normal records (2).
I came up with the idea that combining trigger with bcp, after creating a trigger and adding the parameter -h "FIRE_TRIGGERS" to the bcp statement, the insert will ignore records that have the same key but it won't write to the log.
This is my trigger.
ALTER TRIGGER [PKGORDERCOMMON].[T_ImportData] ON [PKGORDERCOMMON].[IF_R_BUNRUI1]
INSTEAD OF INSERT
AS
BEGIN
--Insert non duplicate records
INSERT INTO [IF_R_BUNRUI1]
(
SYSTEM_KB,
BUNRUI1_CD,
BUNRUI1_KANJI_NA,
BUNRUI1_KANA_NA,
CREATE_TS
)
SELECT SYSTEM_KB,
BUNRUI1_CD,
BUNRUI1_KANJI_NA,
BUNRUI1_KANA_NA,
CREATE_TS
FROM inserted i
WHERE NOT EXISTS
(
SELECT *
FROM [IF_R_BUNRUI1] c
WHERE c.BUNRUI1_CD = i.BUNRUI1_CD
AND c.SYSTEM_KB = i.SYSTEM_KB
);
END;
Is there anyone who can help me.
BCP is not meant for what you are asking it to do (separate good and bad records). For instance, bcp -e option has a limit to how many records it will show. Im not sure if this limit is tied to the "max errors" option, but regardless there is a limit.
Your best option is to load all the records and address bad data in t-sql.
Load all records in such a way to ignore conversion errors. Either:
load entire line from file into a single, large varchar column. Parse out columns and qc data as needed.
or
load all columns from source file into generic varchar columns with large enough size to accomodate your source data.
Either way, when done, use t-sql to inspect your data and split among good/bad records.
I'm trying to import a TXT file using bcp.
My TXT file is like this:
abc|cba
xyz|zyx
My Table is like this:
Field_1 -> Identity field
Field_2 -> Varchar(3)
Field_3 -> Varchar(3)
Filed_4 -> Varchar(1) In this case I must set with default value 'P'
Filed_5 -> Varchar(1) In this case I must set with default value 'C'
My table with values must be:
1,abc,cba,P,C
2,xyz,zyx,P,C
Note-> My TXT file is huge (around 200GB), I can't import into another table to then pass all value to this table (just saying).
##Version-> SQL Server 2014 (SP2)
You cannot generate data via BCP, you must depend on SQL Server to do that as Jeroen commented. To add to his comment, the identity value is not a default, you should continue to use the identity property of the column.
For both (identity and default), you must use the -f option to BCP. This the option to include a format file to direct the BCP utility to see and handle the data as stated in the format file.
Using a format file, you can specify which columns in the file are mapped to which columns are in the destination table. To exclude a column, just set its destination value to "0".
The format files and the bcp utility are much larger topics in and of themselves, but to answer your question; yes it is possible and using a format file with modified destination values (set to "0") is the way to do it.
Doing this, you can process the data once. Using powershell to append data is possible, but unnecessary and less efficient. To do this in one action with bcp, you need to use a format file.
I want to restore my table with BCP by code in the below.
BCP framework.att.attendance in "D:\test\mhd.txt" -T -c
But the column (id) is identity in this table.
When data is restored with BCP I want id columns to be unchanged.
In other words, if the id of the first row is '7' before BCP, I want to import data and the id of the first row will be still be '7'.
What should I do?
BCP IMPORT
-E
-E Specifies that identity value or values in the imported data file are to be used for the identity column.
If -E is not given, the identity values for this column in the data file being imported are ignored.
I have a possibly peculiar data migration problem:
I have an existing and populated SQLite3 database.
I do receive a new schema for a (hopefully compatible) database.
Result should be a new database, built according to new schema, containing as much as possible of the old database content.
Given limitations in both SQLite3 ALTER statement and our workflow it is safe to assume:
normal case will be new columns are added to end of table.
added columns (fields) will either have a default or can be left NULL.
rarely some table will be added.
very rarely some table or column may be dropped.
no table/column renaming will happen.
no column reshuffling will happen.
NOTE: if the new schema is not compatible with the old one (i.e.: any of the above assumptions does not hold true) it's accepted to fail badly.
I tried this script (old database is data.sql3 and new schema is data.schema):
mkdir tmp
cd tmp
#compute old DB schema
sqlite3 ../data.sql3 .schema >old_s
#purge new schema for any initialization...
grep -v ^INSERT ../data.schema >data.schema
#... create a dew, empty DB...
sqlite3 new.sql3 <data.schema
#... and compute a standard schema
#(this is done to avoid typing differences)
sqlite3 new.sql3 .schema >new_s
#iff the schemas are different
if ! diff -q old_s new_s
then
#save old DB
mv ../data.sql3 .
#dump contents
sqlite3 data.sql3 .dump >old_d
#expunge all statements needed to recreate DB/Tables
#new_d contains only INSERT statements
grep -v -f old_s old_d >new_d
#add old DB content to new DB
sqlite3 new.sql3 <new_d
#move new DB in place
mv new.sql3 ../data.sql3
fi
cd ..
This works to detect changes, but fails to repopulate the new database because .dump does not contain column names and thus insertion fails (missing value).
What I'm looking for is some way to force sqlite3 DB .dump to output INSERT statements containing all field names (normally it relies on position) or, it that's not possible, some way to tell sqlite3 DB <new_d to consider any undefined field as null or default (without failing).
Any other way to achieve the same result (without requiring knowledge of what, exactly, has been modified) would be equally welcome.
To be able to insert/import dumps with less columns into a table you can provide default values for the new, appended columns, or simply enable them to be set to NULL. The constraint clause is the same for CREATE TABLE and ALTER TABLE:
http://www.sqlite.org/syntax/column-constraint.html
-- newColumn is set to a default value if not provided with INSERT
alter table myTable
add column newColumn INTEGER NOT NULL default 0;
-- newColumn may be NULL, which is the default if not provided with INSERT
alter table myTable
add column newColumn INTEGER;
-- It is also valid to combine NULL and DEFAULT constraints
alter table myTable
add column newColumn INTEGER default 0;
Note that in order for the INSERT statement to work with the new columns it must provide the column names.
I have an extremely large database I need to send to the developer, the table has over 120million rows. the developer says he only needs about 10,000 or so rows so I was going to use the sqlcmd -S -d -Q "select top 10000 * from table" -s "," -o "C:\temp\filename.csv"
I decided rather than truncate immediately I would script out the table, rename and test bulk inserting, I tried using
bulk insert tablename from 'c:\temp\filename.csv'
with (
fieldterminator = ',',
rowterminator = '\n'
)
this ends in "Bulk load data conversion error (truncation) for row 1..." error. I also tried in import/export wizard and it fails for the same problem (truncation). increasing the size of the field lengths, solves the problem but I really am having problems understanding why I need to do this. Its the same data from the same table, it should bulk insert right back in?!?
also the problem is happening on every column in the table and by varying lengths, so there is no column with the same number of chars I have to add. all the columns are varchar data type. could the sqlcmd be inserting some kind of corruption in the file? I have tried to look for a problem, I also tried rtrim(ltrim(columname)) to make sure there is no whitespace but I'm not sure this is how it works. I'm using sql server 2012 if this helps.
thanks
You should look into BCP Queryout and BULK INSERT options. Use NATIVE format if you're going from SQL to SQL.
(BCP is command-line):
bcp "select top(10000) * from table" queryout "OUTPUTFILENAME.DAT" -S serverInstanceName -d databaseName -T -n
The Bulk Insert command is SQL (not command line):
bulk insert table from 'path\and\OUTPUTFILENAME.DAT' with (keepidentity,datafiletype = 'native');
(If the table doesn't have an identity, you can eliminate keepidentity,