I'm attempting to create a bcp format file for importing a number of small files into SQL Server. When I run the bcp command without a format file, and specify the column and row delimiters, it works fine. When I run it with a format file, I get the following error: "Unexpected EOF encountered in BCP data-file"
The file is pipe delimited, has one header row, one data row and uses LF, not CR-LF as the row delimiter.
-- This command works:
BCP.exe Validation.CONTROL_TEST IN "C:\Users\rliben\Documents\MW_Data_Extracts\Sample_LF.ctl" -F2 -T -S "MyServer" -d MSO_MT -c -t"|" -r"0x0A"
-- This command fails:
BCP.exe Validation.CONTROL_TEST IN "C:\Users\rliben\Documents\MW_Data_Extracts\Sample_LF.ctl" -F2 -T -S "MyServer" -d MSO_MT -f "C:\Users\rliben\Documents\MW_Data_Extracts\Control_LF.fmt"
The files are very simple. Each one looks like this. They each literally have only one line of data:
FileName|FeedVersion|LineCount
File_Name_20220308.ctl|200|3339
If you copy this to try it yourself, remember that it uses LF, not CR-LF as the row delimiter.
Here's the format file:
13.0
3
1 SQLCHAR 2 100 "|" 1 CONTROL_FILE_NAME SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 2 10 "|" 2 FILE_VERSION SQL_Latin1_General_CP1_CI_AS
3 SQLINT 0 4 "0x0A" 3 CONTROL_RECORD_COUNT ""
Here's the table it's going in to:
CREATE TABLE Validation.CONTROL_TEST (
CONTROL_FILE_NAME varchar(100) NOT NULL,
FILE_VERSION varchar(10) NOT NULL,
CONTROL_RECORD_COUNT int NOT NULL
)
I'm connecting to SQL Server 2016, hence the 13.0 at the start of the format file. I've tried different versions. I've tried changing the row delimiters in the file to CR-LF, as a test, and using "\n" and "\r\n" as the row delimiter. Nothing seems to work!
I think it's something pretty simple. I'm hoping your eyes will see it immediately.
Thank You,
Robbie
**** Update – 3/14/2022 ****
One part of the problem is resolved: The “Host file data type” must by SQLCHAR for all columns, even integer columns. Originally I had SQLINT for the last column.
I am now able to import the file when it has CR-LF row terminators. Here is the format file for CR-LF files:
14.0
3
1 SQLCHAR 0 100 "|" 1 CONTROL_FILE_NAME SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 10 "|" 2 FILE_VERSION SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 0 4 "\r\n" 3 CONTROL_RECORD_COUNT ""
I’m not yet able to import files with only LF row terminator. Here’s the format file I’m using that doesn’t work:
14.0
3
1 SQLCHAR 0 100 "|" 1 CONTROL_FILE_NAME SQL_Latin1_General_CP1_CI_AS
2 SQLCHAR 0 10 "|" 2 FILE_VERSION SQL_Latin1_General_CP1_CI_AS
3 SQLCHAR 1 12 "0x0A" 3 CONTROL_RECORD_COUNT ""
Almost identical except for the row terminator. Anyone have any ideas about why it should treat CRLF and LF files differently? That is, aside from the obvious EOL character difference?
It’s not clear to me what the “prefix length” is supposed to do. I’ve tried different ones. When I use 0 for the first two columns, it doesn’t throw an error. It just doesn’t import any rows! When I use any of the other acceptable options, 1, 2, 4 or 8, I get "Unexpected EOF encountered in BCP data-file", as usual. When I use a -e switch to generate an error file, there is no further meaningful information in it.
Here is a Microsoft graphic with the terminology of what’s in a BCP format file.
Here is a link to MS documentation on creating format files.
https://learn.microsoft.com/en-us/sql/relational-databases/import-export/non-xml-format-files-sql-server?view=sql-server-ver15
BTW, so far as I can tell, using format nul creates a completely erroneous format file because it uses SQLINT instead of SQLCHAR as the host file data type of the integer column. Using bcp without -f or -c switches, it allows me to use “char” as the data type for the integer column. But, it fails anyway.
Thanks again, Robbie
Related
I am attempting to load a .csv file from ubuntu 16.04 using msobdcsql17 driver loaded via bcp. The Linux server is sitting in an active directory environment, and I am using a username/password combo to access the data.
When I run the command, it loads no files, and it gives no output.
I decided to make it as simple as possible, so here is my example.
(code via this site)
The Table (SomeTable)
CREATE TABLE dbo.SomeTable
(
SomeTableID INT IDENTITY(1,1) NOT NULL --This is column "1"
ColA INT --This is column "2"
,ColB INT --This is column "3"
,ColC INT --This is column "4"
)
;
The Text File (test.csv)
Column_C,Extra,Column_A
1,2,3
4,5,6
7,8,9
The Format File (format.fmt)
14.0
3
1 SQLCHAR 0 8000 "," 4 ColC ""
2 SQLCHAR 0 8000 "," 0 NotUsed ""
3 SQLCHAR 0 8000 "\r\n" 2 ColA ""
The Command
bcp SomeTable in "/full/path/to/test.csv" -d DatabaseName -U"username" -P"password" -S"SERVERNAME,1432" -f"/full/path/to/format.fmt" -e error.txt
The Result
[Tue 12 9:16AM]:~/
ॐ bcp SomeTable in "/full/path/to/test.csv" -d DatabaseName -U"username" -P"password" -S"SERVERNAME,1234" -f"/full/path/to/format.fmt" -e error.txt
[Tue 12 9:16AM]:~/
♻
While the result shows nothing, that change in the little symbol says that it returned with an error.
NOTE: I am trying to access an instance (reason for the port number 1234). Using a named instance does not work with the linux driver, so a fixed port must be assigned and used.
What I have tried:
Adjust credentials: It shows an error indicating it can't login with the wrong credentials
Exporting a format file: Works fine
Dumping from the same credentials/config: Works fine
Uninstalling all ODBC drivers, the reinstalling msodbcsql17: Fails, no output
Using Sql Server ODBC driver v13: Fails, no output
Using an XML format file: Fails, no output
Tried removing the format file and using '-c' option: Fails, no output
Possible ideas of what could be wrong:
it is unable to access the file on the server: would I need to map a drive to the linux server on the sql box?
???????????
If anyone knows how to get some verbose output so I can troubleshoot, or make this happen, would love the help...
I am exporting data out of Teradata and using BCP to import into Sql Server (SS) to an empty staging table via a Linux script. After I wrote it I ran a small volume through and all was well. Now I am doing the full volume load of 146 million rows. Around 27+ million rows into it I start getting data thrown out due to invalid characters. If I just extract a segment of the file that includes the keys to the bad records and run that smaller file through there are no errors and data loads fine. If I feed the big file in using a sorted input that is ascending and then do the same again with descending sort the load fails at exactly the same point in the file even though the records at that point in the file are not the same. So I have concluded there is nothing wrong with the input file. It seems to be that when we get to a certain record number in the input (repeatable each time) the errors start occurring. I calculated that this point in the file is just about 2gb. Is it possible there is a limit in Sql Server I am hitting or DBMS setting that needs to be tuned upwards? I appreciate any suggestions. Thank you.
**Update #2 2016-5-17 21:55 EST - the following is the format file for the BCP plus the BCP command I used.
/john/bcp # more *fmt
11.0
7
1 SQLCHAR 0 41 "|" 1 lgx_cus_ext_id ""
2 SQLCHAR 0 12 "|" 2 dgtl_cpn_id ""
3 SQLCHAR 0 7 "|" 3 dgtl_cpn_src_id ""
4 SQLCHAR 0 41 "|" 4 shopr_cpn_rnk_finl_pc ""
5 SQLCHAR 0 24 "|" 5 rw_udt_ts ""
6 SQLCHAR 0 30 "|" 6 rw_udt_prc_id SQL_Latin1_General_CP1_CI_AS
7 SQLCHAR 0 1 "\n" 7 cpn_not_sug_flg ""
bcp DIGITALCOUPONSIMPLE.DBO.SHOPR_CPN_RNK_STGG \
in ${one_split_file} -c -D -S MSSQLTestServer \
-U 103866 -P KafRaS4e -f benchmark.fmt \
-e /root/john/bcp/benchmark_errors.out \
-h -h "TABLOCK,ORDER(LGX_CUS_EXT_ID ASC, DGTL_CPN_ID ASC, DGTL_CPN_SRC_ID ASC)"
Sample errors:
Row 27183442, Column 1: Invalid character value for cast specification
RK 0 46800846218 00016323936 000012 0.1272 2016-05-06
18:18:40.570000 BCP-JQUIRK|0
Row 27183623, Column 4: Numeric value out of range ## 46800846225 00016380967 000011 00011 0.3610 2016-05-06
18:18:40.570000 BCP-JQUIRK|0
Row 27183804, Column 5: Invalid date format ## 46800846231 00016380968 000011 0.5020 .7362|2016-05-06
18:18:40.570000 BCP-JQUIRK 0
Row 27183986, Column 3: Numeric value out of range
As previous commenter said you do need to provide us with error messages you recieve.
In the meantime check the Connection / Query timeouts, I assume your file is taking a long time to load.
Also check that you are not running out of disk space in the Database.
I'm trying to use a non-XML bcp format file to import data into LocalDB on Win7 64 bit. Simplest possible use case.
OS Name: Microsoft Windows 7 Home Premium
OS Version: 6.1.7601 Service Pack 1 Build 7601
LocalDB version: Microsoft SQL Server 2014 (12.0.2000.8)
BCP version: 12.0.2000.8
Basically, latest version of everything downloaded from Microsoft SQL Server 2014 site a few days ago.
I'm able to connect to the LocalDB instance via bcp to create a format file, but the generated format file doesn't work when trying to re-import the simplest possible data using it. No matter what I try, bcp loads zero rows, fails silently and prints no error information to the specified error file.
/* create the table */
use try_db;
create table try(num integer);
/* create the format file based on the table. */
bcp try_db.dbo.TRY format nul -n -T -f TRY.fmt -S (localdb)\default_db
/* above command creates a file TRY.fmt with the following contents */
12.0
1
1 SQLINT 1 4 "" 1 num ""
/* then I create a file data.txt, with just the number 99 in it, followed by a Windows line terminator (\r\n) */
/* then try importing the file into the table */
bcp try_db.dbo.TRY in data.txt -f TRY.fmt -T -S (localdb)\default_db -e errors.txt
Result:
Starting copy...
0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 1
Nothing is written to errors.txt. I am just not able to get bcp to import anything at all using a format file!
I haven't tried it with SQL Server itself (only with LocalDB) but it shouldn't matter for stuff as simple as this.
I tried editing the TRY.fmt file line as follows:
1 SQLINT 1 4 "\r\n" 1 num ""
But that didn't help either.
I am able to get it to successfully import using -c instead of -f:
bcp try_db.dbo.TRY in data.txt -c -T -S (localdb)\default_db -e errors.txt
Any thoughts on (a) why bcp won't import using the format file, and (b) why it prints no errors to the specified error file? There must be something really simple I'm getting wrong here.
Please, no recommendations to use BULK INSERT or SSIS (etc) instead. bcp should just work as documented!
The format file is describing the source data not the destination. When you use -c or datafiletype='char' your input datatypes must be SQLCHAR. Native datatypes are only valid when using -n or datafiletype='native'. A source file in native format is always binary so bcp needs to know the data type of each field in order to read the correct amount of bytes and interpret them correctly.
I think I found the answer. The bcp format spec doesn't work properly! It seems that even for numeric or datetime import fields, you have to specify "SQLCHAR" as the datatype in the .fmt file. Any attempt to use the actual .fmt file generated by "bcp format" is hopeless -- if it gives you SQLINT or SQLDATE lines back, you have to replace those with SQLCHAR for the thing to work, even if the db columns are in fact numeric or date/datetime types.
What a crock!
I've got a couple of issues with bcp.
I'm trying to load data from a text file to Sybase ASE 16.0 . The DB is on my local machine (Win 7 64-bit).
I'm using this command:
bcp dbname.owner.TO_INTS in "SomeInts.txt" -P password -S server -U sa -f "bcp_tblTO_INTS.fmt" -e "bcp.errlog.txt"
Here is the format file ("bcp_tblTO_INTS.fmt"):
10.0
1
1 SYBINT4 0 5 "" 1 SEEMS_THIS_IS_IGNORED
The target table has a single column, defined as an INT:
CREATE TABLE owner.TO_INTS
(
FROM_INT INT NOT NULL
)
LOCK ALLPAGES
ON 'default'
GO
I have two problems.
bcp will only populate the target table if the data has just four characters in it (ie, values from 1000 to 9999). If I use values outside this range I get:
Unexpected EOF encountered in BCP data-file.
bcp copy in failed
The values are not being populated correctly:
Input value (in file) : 1000 Populated as (in Sybase):808464433
Input value (in file) : 9999 Populated as (in Sybase):960051513
The file encoding is ANSI. The EOL character is LF.
What am I doing wrong?
Thanks
Dave.
Since your file is only one column, remove the format file and use the -c option instead. This will BCP the file in character (plaintext) mode, and should resolve the issue.
As noted, you may need to change the EOL character to CRLF
I am using bcp utility to import data into Sybase database on Solaris system
Since the bcp file is generated by another xml parsing script, I have no control on that to make it generate the fields in the order I want. And now the the order of fields in the bcp file is a little different with the order in the database table.
I want to use a format file for bcp tool to control the order of fields loaded into the database, so I have the sample bcp file as below and constructed a format file accordingly:
the bcp file:
603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8
603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06
the format file:
10.0
4
1 SYBCHAR 0 12 "|" 3 ver
2 SYBCHAR 0 26 "|" 2 first_dt
3 SYBCHAR 0 60 "|" 4 name1
4 SYBCHAR 0 10 "|" 1 name2
Although I am stuck on the following error:
$bcp my_db..my_tbl in test.bcp -e error -f format.fmt -r\\n -S Sever -U user -P pw
Starting copy...
CSLIB Message: - L0/O0/S0/N24/1/0:
cs_convert: cslib user api layer: common library error: The conversion/operation was stopped due to a syntax error in the source field.
Unexpected EOF encountered in BCP data-file.
bcp copy in partially failed
1 rows copied.
I suspect the cause for the error is the bcp utility can't recognize the row delimiter which is '\n' character (I have used od -c test.bcp to check this), although the -r\\n option in the command seems doesn't work, which should specify the row delimiter as '\n' character.
Does anyone have a clue?
Edit:
I did a change on the format file and it works fine now, I changed the field delimiter for the last field from '|' to '\n' as below:
the new format file:
10.0
4
1 SYBCHAR 0 12 "|" 3 ver
2 SYBCHAR 0 26 "|" 2 first_dt
3 SYBCHAR 0 60 "|" 4 name1
4 SYBCHAR 0 10 "\n" 1 name2
As Michael mentioned in comments, it's tricky to use format files. When using the original format file I have tried to add '|' at end of each records, although the file were not processed correctly.
Now the file can be processed correctly, although I am not sure what is used as row delimiter by bcp utility
As most of the loader utilities, BCP does not read rows, it just gets a bunch of bytes from the source file, therefore, you have to specify the end line character on the .fmt file mostly when the end of line character can be different depending on the encoding of the file.
For example, in the example you provided, you have this on the source file:
603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8
603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06
But depending on the Encoding you will have this:
603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8\n603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06
or this:
603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8\n\r603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06
That's why the change of the terminator you did on the last column is so important:
10.0
4
1 SYBCHAR 0 12 "|" 3 ver
2 SYBCHAR 0 26 "|" 2 first_dt
3 SYBCHAR 0 60 "|" 4 name1
4 SYBCHAR 0 10 "\n" 1 name2 * The last terminator was not "|"
The following image shows what means each component on the format file (I got it from here):
By the way, please notice the last terminator!
Sybase format files can be tricky to use, and I have yet to get them to work to redorder columns. Without a format file, and since you can not control the input file you're given, your options are:
Rebuild the table with the columns in the correct order
Use a temp table, or work table for the bcp in, and move the data into your production table from there.
If for some reason there is logic in place the requires the table to keep it's existing column order (which should be unlikely), you can rebuild the table with the correct column order, then create a view of the table that has the columns in the old order.
Given your data format, the following bcp command line should work.
bcp DBNAME..TABLENAME in MY_FILE.bcp -U username -P password -S SERVERNAME -c -t\| -r\\n