Output XML directly to file from SQL server - sql-server

Question

Output the results (XML) of a stored procedure to a file.
I have a stored procedure in SQL server that creates an XML file. It currently displays the resulting XML and I have to manually save as a file.
I have tried to call the procedure from Powershell, as in this question, this works for small files but not for large (>1gb files) as Powershell tries to store the entire thing as a variable and it quickly runs out of memory.
I'm opening this as a new question as I think there should be a way of doing this within SQL server (or a better way of doing it with Powershell).

Answer 1

You shouldn't use a stored procedure here. Just use better PowerShell. You can stream large types to and from SQL Server with SqlClient. So you just need to drop down and use ADO.NET instead of using the invoke-sqlcmd convenience method.
EG:
$conString = "server=localhost;database=tempdb;integrated security=true"
$sql = #"
select top (1000*1000) *
from sys.messages m
cross join sys.objects o
for xml auto
"#
$fn = "c:\temp\out.xml"
$con = new-object System.Data.SqlClient.SqlConnection
$con.connectionstring = $conString
$con.Open()
$cmd = $con.createcommand()
$cmd.CommandText = $sql
$cmd.CommandTimeout = 0
$rdr = $cmd.ExecuteXmlReader()
$w = new-object System.Xml.XmlTextWriter($fn,[System.Text.Encoding]::UTF8)
$w.WriteNode($rdr,$true)
$w.Close()
$rdr.Close()
write-host "Process Memory: $( [System.GC]::GetTotalMemory($false) )"
write-host "File Size: $( (ls $fn)[0].Length )"
outputs
Process Memory: 34738200
File Size: 468194885

Answer 2

Other solution if you can you have to build your XML file in Temporary Table line by line and then output and read the result line by line from Powershell or other code :
SQL Example :
/*
**
** Stored procedure
**
*/
/*** Effacement: ********************************************************
IF EXISTS ( SELECT name FROM sysobjects
WHERE type = 'P' AND name = 'procTEST' )
DROP PROCEDURE procTEST
*** Effacement: ********************************************************/
CREATE PROCEDURE procTEST
AS
CREATE TABLE #TEMP (vInfo VARCHAR(MAX), nLine int)
INSERT INTO #TEMP
SELECT 'Line 1',1
UNION ALL
SELECT 'Line 2',2
UNION ALL
SELECT 'Line 3',3
SELECT vInfo FROM #TEMP ORDER BY nLine ASC
SET NOCOUNT OFF
/*** TESTS ****************************************************************************************************************************************
sp_helptext procTEST
-- DROP PROCEDURE procTEST
EXEC procTEST
*** TESTS ****************************************************************************************************************************************/
Powershell Script :
$readconn = New-Object System.Data.OleDb.OleDbConnection
$writeconn = New-Object System.Data.OleDb.OleDbConnection
[string]$connstr="Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=TEST;Data Source=.\XXXXX;Workstation ID=OMEGA2"
$readconn.connectionstring = $connstr
$readconn.open()
$readcmd = New-Object system.Data.OleDb.OleDbCommand
$readcmd.connection=$readconn
$readcmd.commandtext='EXEC procTEST'
$reader = $readcmd.executereader()
# generate header
$hash=#{}
for ($i=0;$i -lt $reader.FieldCount;$i++){
$hash+=#{$reader.getname($i)=''}
}
$dbrecords=while($reader.read()) {
for ($i=0;$i -lt $reader.FieldCount;$i++){
$hash[$reader.getname($i)] = $reader.GetValue($i)
}
New-Object PSObject -Property $hash
}
$reader.close()
$readconn.close()
$dbrecords