Unicode flat file import to SQL - sql-server

Question

I am trying to bulk import data into ms-sql 2016, but, because of 2-bytes length characters (like Ü, Ä, etc), I am facing problem:
wrapping fields
Source is fixed-length, unicode (utf-8) text file with special (wide) characters:
this a sample part of file:
ABS525 0128211024200
ABS526 0128211024200
ABS527 0128211024200
ABS528 0128211024200
ABS529 0128211024200
Ölrücklaufleitung 0128211037390
Ölzu- und Ölrücklaufle0128211037390
Ölzulaufleitung 0128211037390
field lengths are: 22 - 4 - 3 - 5 - 1
I tried every way:
- import wizard in Management Studio,
- SSDT import,
- bulk import,
- openrowset,
- bcp command line
nothing worked, actually, they work unless there is a special character in the row.
This is my bulk insert code:
BULK INSERT [tecdoc2].[dbo].[211]
FROM 'C:\Users\Administrator\Desktop\D_TAF24\211yeni.0128'
WITH (MAXERRORS=50, CODEPAGE = '65001', DATAFILETYPE = 'widechar', FORMATFILE = 'C:\Users\Administrator\Desktop\BCP_Formats\a211.xml')
This is my format file (here, I tried a lot of combinations):
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharFixed" LENGTH="22" />
<FIELD ID="2" xsi:type="CharFixed" LENGTH="4" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharFixed" LENGTH="3" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharFixed" LENGTH="5" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharFixed" LENGTH="1" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="6" xsi:type="CharTerm" TERMINATOR="\r\n" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="ArtNr" xsi:type="SQLNVARCHAR" LENGTH="22" />
<COLUMN SOURCE="2" NAME="DLNr" xsi:type="SQLNCHAR" />
<COLUMN SOURCE="3" NAME="SA" xsi:type="SQLNCHAR" />
<COLUMN SOURCE="4" NAME="GenArtNr" xsi:type="SQLNCHAR" />
<COLUMN SOURCE="5" NAME="Losch-Flag" xsi:type="SQLNCHAR" />
</ROW>
</BCPFORMAT>
all fields in sql are nvarchar (with the specified lengths, actually I made a lot of trials here: double the specified lengths, or 'max', etc)
would you have any advice? I would appreciate.
With Kind Regards,
Murat

Answer 1

This is exactly the problem I am having but with OPENROWSET. If the file is delimited it work fine.
The only way around this issue I have found is to import the whole row into a single nvarchar(Big Enough) column and parse it out with the database. Works fine then, but a royal pain in the bottom.
If you change your format file to be:
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharFixed" LENGTH="35" />
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR="\r\n"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="RowData" xsi:type="SQLNVARCHAR" LENGTH="35"/>
</ROW>
</BCPFORMAT>
Then you import query can be:
INSERT INTO [tecdoc2].[dbo].[211]
(
ArtNr
,DLNr
,SA
,GenArtNr
,[Losch-Flag]
)
SELECT SUBSTRING(src.RowData, 0, 22) AS ArtNr
,SUBSTRING(src.RowData, 23, 4) AS DLNr
,SUBSTRING(src.RowData, 27, 3) AS SA
,SUBSTRING(src.RowData, 30, 5) AS GenArtNr
,SUBSTRING(src.RowData, 35, 1) AS 'Losch-Flag'
FROM OPENROWSET ( BULK 'C:\Users\Administrator\Desktop\D_TAF24\211yeni.0128'
,FORMATFILE = 'C:\Users\Administrator\Desktop\BCP_Formats\a211.xml'
,CODEPAGE = '65001' -- Unicode
,FIRSTROW = 1
) AS src