How to create a virtual SQL Server table from a csv file - sql-server

Question

I have multiple use cases whereby I need to effectively create a virtual sql server table from a csv source and I need to do it preferably using pure sql on the fly- ie not using a procedure. To provide some context, I need to wire up this code in a third party reporting engine; which is talking to sql server in addition to other multiple data sources such as oracle etc.
This can be done in POSTGRE using a simple function split_part because that function includes the ability to search the csv concate string for the position of the field delimiter; example ',' comma separator. Unfortunately, sql server has a similar function STRING_SPLIT ( string , separator ) but notice it does NOT have the position aspect SPLIT_PART(string, delimiter, position) offered in POSTGre.
Example source csv:
Row 1 a,b,c,d
Row 2 e,f,g,h
etc
Output - using POSTGre db but require same for sql server
select split_part(p.Lines, ',',1) As Col1,
split_part(p.Lines, ',',2) As Col2,
split_part(p.Lines, ',',3) As Col3,
split_part(p.Lines, ',',4) As Col4
from ( select unnest(string_to_array(:csvdata, chr(10))) as Lines)p
Any solution?

Answer 1

SQL Server allows to query *.csv file as a virtual DB table on a file system.
Here is a minimal reproducible example.
SQL
SELECT *
FROM OPENROWSET(BULK 'e:\Temp\Quark.csv'
, FORMATFILE = 'e:\Temp\Quark.xml'
, ERRORFILE = 'e:\Temp\Quark.err'
, FIRSTROW = 2 -- real data starts on the 2nd row
, MAXERRORS = 100
) AS tbl;
Quark.csv
"ID"|"Name"|"Color"|"LogDate"|"Unknown"
41|Orange|Orange|2018-09-09 16:41:02.000|
42|Cherry, Banana|Red,Yellow||
43|Apple|Yellow|2017-09-09 16:41:02.000|
Quark.xml
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
<FIELD ID="1" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="70"/>
<FIELD ID="2" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="70" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="3" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="70" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="4" xsi:type="CharTerm" TERMINATOR='|' MAX_LENGTH="70" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
<FIELD ID="5" xsi:type="CharTerm" TERMINATOR='\r\n' MAX_LENGTH="70" COLLATION="SQL_Latin1_General_CP1_CI_AS"/>
</RECORD>
<ROW>
<COLUMN SOURCE="1" NAME="ID" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="2" NAME="Name" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="3" NAME="Color" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="4" NAME="LogDate" xsi:type="SQLVARYCHAR"/>
<COLUMN SOURCE="5" NAME="Unknown" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>
Output
+----+----------------+------------+-------------------------+---------+
| ID | Name | Color | LogDate | Unknown |
+----+----------------+------------+-------------------------+---------+
| 41 | Orange | Orange | 2018-09-09 16:41:02.000 | NULL |
| 42 | Cherry, Banana | Red,Yellow | NULL | NULL |
| 43 | Apple | Yellow | 2017-09-09 16:41:02.000 | NULL |
+----+----------------+------------+-------------------------+---------+

Answer 2

Unless I am mistaken - split_part parses a delimited string into component elements and returns the desired element from that string.
To accomplish this in SQL Server - we can use charindex and substring, or you can use a combination of splitting the string to a table, and rolling the values back up in a GROUP BY.
Here is a function that can be used to return up to 12 elements from a delimited string:
ALTER Function [dbo].[fnSplitString_12Columns] (
#pString varchar(8000)
, #pDelimiter char(1)
)
Returns Table
With schemabinding
As
Return
Select InputString = #pString
, p01_pos = p01.pos
, p02_pos = p02.pos
, p03_pos = p03.pos
, p04_pos = p04.pos
, p05_pos = p05.pos
, p06_pos = p06.pos
, p07_pos = p07.pos
, p08_pos = p08.pos
, p09_pos = p09.pos
, p10_pos = p10.pos
, p11_pos = p11.pos
, p12_pos = p12.pos
, col_01 = ltrim(substring(v.inputString, 1, p01.pos - 2))
, col_02 = ltrim(substring(v.inputString, p01.pos, p02.pos - p01.pos - 1))
, col_03 = ltrim(substring(v.inputString, p02.pos, p03.pos - p02.pos - 1))
, col_04 = ltrim(substring(v.inputString, p03.pos, p04.pos - p03.pos - 1))
, col_05 = ltrim(substring(v.inputString, p04.pos, p05.pos - p04.pos - 1))
, col_06 = ltrim(substring(v.inputString, p05.pos, p06.pos - p05.pos - 1))
, col_07 = ltrim(substring(v.inputString, p06.pos, p07.pos - p06.pos - 1))
, col_08 = ltrim(substring(v.inputString, p07.pos, p08.pos - p07.pos - 1))
, col_09 = ltrim(substring(v.inputString, p08.pos, p09.pos - p08.pos - 1))
, col_10 = ltrim(substring(v.inputString, p09.pos, p10.pos - p09.pos - 1))
, col_11 = ltrim(substring(v.inputString, p10.pos, p11.pos - p10.pos - 1))
, col_12 = ltrim(substring(v.inputString, p11.pos, p12.pos - p11.pos - 1))
From (Values (concat(#pString, replicate(#pDelimiter, 12)))) As v(inputString)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, 1) + 1)) As p01(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p01.pos) + 1)) As p02(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p02.pos) + 1)) As p03(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p03.pos) + 1)) As p04(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p04.pos) + 1)) As p05(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p05.pos) + 1)) As p06(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p06.pos) + 1)) As p07(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p07.pos) + 1)) As p08(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p08.pos) + 1)) As p09(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p09.pos) + 1)) As p10(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p10.pos) + 1)) As p11(pos)
Cross Apply (Values (charindex(#pDelimiter, v.inputString, p11.pos) + 1)) As p12(pos);
The function also returns the starting position in the string for each element. If you use the function to return only the first 4 elements - the rest of the code will be factored out of the final query.
If you have more than 12 columns you can extend to as many as needed. Performance may be affected by the size of the string (8000 vs MAX) and how many elements are parsed - but the only way to be sure will be to test different methods to see which performs best for your data.