BCP BLOB exporting corrupted file - sql-server

I have looked thoroughly for an answer regarding BCP and extracting a BLOB from my SQL server database. I have followed the various steps found in other threads and I keep coming up with the same file corrupted error.
My database has a column with data type IMAGE in BLOB format. On the users end they can enter photos, pdfs, anything to this field and the server converts them to extremely long BLOBs. My task is to extract these so we can replace them with file name extensions, putting less stress on the database.
My current command is:
bcp "select notes_activex from myDatabase where NCR_NO = '1361'" queryout "C:\BCPtest\testOutput" -S -U -P -f C:\BCPtest\testOutput.fmt
My format file is correct for an image file as per some of the other files posted. I have tried casting the image file first to a varbinary(max) and that still doesnt solve my solution. No matter what I try I can get the BLOB to export but it is a corrupted file.
Added my format file:
11.0
1
1 SQLIMAGE 0 0 "" 1 notes_activex ""

As far as binary data, BCP is intended for extracting data so that it can be later inserted into another SQL Server. It's not saved in a format that's compatible with a binary data file. You'll need to extract the data with a program or script, basically, that's capable of converting the data to bytes.
I've done this in the past with a PowerShell script. I'd use something like the script below. I strongly recommend determining the file from the SQL query if at all possible if you're fetching more than one record at a time.
# Define the connection parameters
$SqlServer = 'MyServerName';
$SqlDatabase = 'MyDatabase';
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase;
# Define the query. Note that one field is the file name and the other is the data.
# Modify the WHERE clause to pull the records you need. I am assuming that NCR_NO is not a unique identifier.
$SqlQuery = "select SomeUniqueID AS FileName, notes_activex AS FileData from myDatabase where NCR_NO = '1361'";
# Or use a GUID for the filename
# $SqlQuery = "select NEWID() AS FileName, notes_activex AS FileData from myDatabase";
# Define the path pattern for the output files. {0} will get filled in with the filename.
$OutputFileFullNamePattern = 'C:\Path\To\Output\{0}';
# Create the Connection object and the Command object
$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString;
$SqlCommand = $SqlConnection.CreateCommand();
$SqlCommand.CommandText = $SqlQuery;
# Open the connection
$SqlConnection.Open();
# Create the Sql Data Reader
$SqlDataReader = $SqlCommand.ExecuteReader();
while ($SqlDataReader.Read()) {
# Set in the file name
$OutputFileFullName = $OutputFileFullNamePattern -f $SqlDataReader['FileName'];
# Save the data to the file
[System.IO.File]::WriteAllBytes($OutputFileFullName,$SqlDataReader['FileData']);
}
# Close and dispose of the SQL connection
$SqlConnection.Close();
$SqlConnection.Dispose();
This uses an SqlDataReader, which loads records one at a time. This means your system won't need to load the entire table into memory, but it does mean that you'll have a shared lock on the table until it's done if you're dumping the whole table. If possible, run this query during downtime.

Related

How to stream web service data via PowerShell into a SQL Server VarBinary column

We have a working solution in PowerShell to stream web service (WS) data into a SQL Server 2019 table with a column of datatype NVARCHAR(max). This WS data is just business data in Json format. I'd like to enhance this solution to also be able to stream the WS data into a VARBINARY(max) column.
The currently working code looks like this (abbreviated):
$SQL_ImpInsert = #"
Insert Into [Database].dbo.ImportTable (SeqNo, BulkDataNVarChar)
Values (#SeqNo, #WSData)
"#
# Create the SQL objects.
$SQLConn = New-Object System.Data.SqlClient.SqlConnection $ConnStr
$SQLConn.Open()
$SQLCommand = $SQLConn.CreateCommand()
$SQLCommand.CommandText = $SQL_ImpInsert
# Prep the stream
$WS_URL = 'https://a_valid_ws_request'
$WebRequest = [System.Net.WebRequest]::Create($WS_URL)
$WebRequest.ClientCertificates.Add($WS_Cert) | Out-Null
$WebRequest.Method = "Get"
$WebResponse = $WebRequest.GetResponse()
$WebStream = $WebResponse.GetResponseStream()
$WebStreamRdr = New-Object System.IO.StreamReader $WebStream
# Prep the SQL command to use the stream
$SeqNo = 1
$SQLCommand.Parameters.Clear()
$SQLCommand.Parameters.Add("#SeqNo", [Data.SQLDBType]::INT, -1).Value = $SeqNo
$SQLCommand.Parameters.Add("#WSData", [Data.SQLDBType]::NVARCHAR, -1).Value = $WebStreamRdr
# Perform the streamed insert from the WS into the SQL table
$ExecOutput = $SQLCommand.ExecuteNonQuery();
To attempt to enhance the solution I added a VARBINARY column to the table and changed the command parameter datatype to VARBINARY as follows (only changes to above code shown):
$SQL_ImpInsert = #"
Insert Into [Database].dbo.ImportTable (SeqNo, BulkDataBlob)
Values (#SeqNo, #WSData)
"#
$SQLCommand.Parameters.Add("#WSData", [Data.SQLDBType]::VARBINARY, -1).Value = $WebStreamRdr
When I run the new version of the PowerShell script I get the following error raised during the ExecuteNonQuery statement:
Exception calling "ExecuteNonQuery" with "0" argument(s): "Failed to convert parameter value from a StreamReader to a Byte[]."
I've looked into the command Parameters.Add arguments to see if there was anything I needed to adjust there. There are some variations to the Add method but I don't know\think they are relevant.
I don't know enough about the streaming calls to know if there is anything that should be tweaked there. Since I'm only changing the destination SQL column datatype I'm expecting the setup of the stream to remain the same.
I checked the WS data and there shouldn't be any data bigger than the SQL datatype VARBINARY can handle. I don't think the error has anything to do about the data size coming in though.
Any ideas on how to adjust this code to stream the WS data into the VARBINARY parameter and therefore the SQL column?
StreamReader implements a TextReader, which gives you a string that will fail to convert to your varbinary byte array. I am guessing that your first example using nvarchar succeeds because you get an implicit conversion of the string.
You can likely refactor this to not use the StreamReader at all, but to answer the question you will need to convert that string to a byte array:
$output = $WebStreamRdr.ReadToEnd() # todo: flush & close
$bytes = [system.Text.Encoding]::UTF8.GetBytes($output)
Then use $bytes:
$SQLCommand.Parameters.Add("#WSData", [Data.SQLDBType]::VARBINARY, -1).Value = $bytes
You could also let the string pass into the command as before:
$SQLCommand.Parameters.Add("#WSData", [Data.SQLDBType]::NVARCHAR, -1).Value = $WebStreamRdr
Then let the command convert it to varbinary:
$SQL_ImpInsert = #"
Insert Into [Database].dbo.ImportTable (SeqNo, BulkDataBlob)
Values (#SeqNo, cast(#WSData as varbinary(max)))
"#

Import 2 Excel Files via SSIS with different sheet names

So as the title suggest, I need to do an import of 2 Excel (.xlsx) files from my local machine (c:\temp) into one SQL Server table. Each of the files only contains one sheet, but the sheet names will differ. The columnnames and no of columns on each file is identical.
If I select one specific excel file through SSIS via Excel Connection Manager, it extracts the data perfectly and inserts it into my destination SQL table.
The problem comes in when I add a ForEach Loop Container and want to loop through the c:\temp directory to read the 2 files. Somewhere I am missing a setting and keep getting various "connect to Excel" errors.
Please assist with the following:
I am unsure how to specify the Excel file path. Is the below correct? I used to select the exact file here when loading only 1 file:
Then it seems I need to create variables, so I did below:
Then I am not sure if I should add an expression to my ForEach loop and which mappings would be correct?
And lastly, I am not sure whether to put the filename or sheetname as variable below. I tried the filepath, but get the following error:
Please help as I am totally lost with this.
UPDATE
OK, I have now done the following:
Added a SheetName variable (which I think the Value is maybe incorrect). I am trying to tell it to only read the first sheet.
Then my Excel connection string looks like this:
Provider=Microsoft.ACE.OLEDB.12.0;Data Source=;Extended Properties="EXCEL 12.0 XML;HDR=NO";
My ForEach loop:
And my Excel source:
I get the following error:
[Book 2] Error: Opening a rowset for "Sheet1$" failed. Check that the object exists in the database.
It seems like your biggest issue is in regards to getting the sheetname which can vary, and the only way I know how to do this is with a script task.
So inside your foreach loop (store filepath to the Excel file) as variable, add a script task before you enter the data flow.
First of all start with knowing you connection string (I use this site for help: https://www.connectionstrings.com/excel/)
Set your read/write variable to [SheetName] and read to FilePath
Code:
var cstr = string.Format(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source={0};Extended Properties=""Excel 12.0 Xml; HDR = YES"";"
, Dts.Variables["FilePath"].ToString()); //Enter your connection string for OleDB conn
using (var conn = new System.Data.OleDb.OleDbConnection(cstr))
{
conn.Open();
var sheets = conn.GetOleDbSchemaTable(System.Data.OleDb.OleDbSchemaGuid.Tables, null);
//Since there is only 1 for sure.
Dts.Variables["SheetName"] = sheets.Rows[0]["TABLE_NAME"].ToString();
}
Now you have the SheetName in a variable (this will have the $ in the sheetname that you need as well), set up another variable called SQL and define it as "Select * from [" + SheetName + "]".
Now use the variable SQL in your DataFlow Source.

Invoke-Sqlcmd ran SQL script capture verbose output

I need to do health check on various servers on daily basis. I have SQL script which includes missing indexes, duplicate indexes, CPU and memory, recovery, failed job, last backup etc in one script.
I am running this script manually on sever and changing result to text (I have made script which use print statement so I can directly copy and paste result).
Now I want to run this script with PowerShell but I am not getting result to text format with row-column format.
Code:
$RESULT = (Invoke-Sqlcmd -InputFile $sqlscript -ServerInstance $server -Database $databse -Verbose 4>&1) |
Out-File $outfile
I am getting all the print statement but not the result in text file.
A more appropriate way to do this is as follows.
[string]$SqlsSript = '..\path\'
[string]$ConnectionString = '...'
$DataSet = Invoke-SqlCmd -ConnectionString $Connectionstring -InputFile $SqlScript -As DataSet
You can then explore the $DataSet collection, i.e
$DataSet.Tables[0].Rows
Which you can then transform / output to a file if you like.
Do ensure that your SqlScript returns a table, e.g. a table variable.

The given value of type String from the data source cannot be converted to type uniqueidentifier of the specified target column

What I'm trying here is, to load the data from .xml file to the SQL server database using PowerShell Script.
My Script goes like:
$dataSource = 'dtsrc';
$database = 'tempdb'; #DB
$connection = New-Object System.Data.SqlClient.SqlConnection; #setting connection
$connection.ConnectionString = "Server=$dataSource;Database=$database;Integrated Security=True;";
$connection.Open();
$command = New-Object System.Data.SqlClient.SqlCommand;
$command.Connection = $connection;
$as = New-Object System.Data.SqlClient.SqlDataAdapter;
$as.SelectCommand = $command;
$filename = 'C:\DCT\XML\apc.xml'; #file to be loaded into sql server database table
$ds = New-Object System.Data.DataSet;
$ds.ReadXml($filename); #reading from the file -- line gives output InferSchema
$dt = New-Object System.Data.DataTable;
$dt.Columns.Add("StartTime",[datetime]);
$dt.Columns.Add("EndTime",[datetime]);
$dt.Columns.Add("Status",[int]);
$dt.Columns.Add("ProcessedTime",[datetime]);
$dt.Columns.Add("ScheduleId",[guid]);
$dt.Columns.Add("Model",[string]);
$dt.Columns.Add("ModelVersion",[string]);
$dt.Columns.Add("ModelVersionState",[string]);
$dt.Columns.Add("ModifiedTime",[datetime]);
$dt.Columns.Add("WriteBackLastRunTime",[datetime]);
$dt.Columns.Add("ModifiedBy",[string]);
$dt.Columns.Add("ModelType",[int]);
$dt.Columns.Add("IsTimeRange",[int]);#b
$dt.Columns.Add("WriteBackStatus",[int]);
$dt.Columns.Add("RetryWriteBack",[int]);#b
$dt.Columns.Add("NumOfRetry",[int]);
$dt.Columns.Add("FrequencyUnits",[int]);
$dt.Columns.Add("ScheduleType",[int]);
$dt.Columns.Add("CustomType",[int]);
$dt.Columns.Add("ShiftStartTime",[datetime]);
$dt.Columns.Add("StartWeekDay",[int]);
$dt.Columns.Add("EndWeekDay",[int]);
$dt.Columns.Add("WritebackProcessedTime",[datetime]);
$dt.Columns.Add("DiagStatus",[int]);
$dt.Columns.Add("AccountingPeriodCase_PK_ID",[guid]);
$dt = $ds.Tables[0];
$bcp = New-Object 'Data.SqlClient.SqlBulkCopy' $connection; #bulkcopy to the destination table.
$bcp.DestinationTableName = 'dbo.tempor';
#$bcp.ColumnMappings.Count;
$bcp.ColumnMappings.Clear();
$bcp.ColumnMappings.Add('StartTime','StartTime');
$bcp.ColumnMappings.Add('EndTime','EndTime');
$bcp.ColumnMappings.Add('Status','Status');
$bcp.ColumnMappings.Add('ProcessedTime','ProcessedTime');
$bcp.ColumnMappings.Add('ScheduleId','ScheduleId');
$bcp.ColumnMappings.Add('Model','Model');
$bcp.ColumnMappings.Add('ModelVersion','ModelVersion');
$bcp.ColumnMappings.Add('ModelVersionState','ModelVersionState');
$bcp.ColumnMappings.Add('ModifiedTime','ModifiedTime');
$bcp.ColumnMappings.Add('WriteBackLastRunTime','WriteBackLastRunTime');
$bcp.ColumnMappings.Add('ModifiedBy','ModifiedBy');
$bcp.ColumnMappings.Add('ModelType','ModelType');
$bcp.ColumnMappings.Add('IsTimeRange','IsTimeRange');
$bcp.ColumnMappings.Add('WriteBackStatus','WriteBackStatus');
$bcp.ColumnMappings.Add('RetryWriteBack','RetryWriteBack');
$bcp.ColumnMappings.Add('NumOfRetry','NumOfRetry');
$bcp.ColumnMappings.Add('FrequencyUnits','FrequencyUnits');
$bcp.ColumnMappings.Add('ScheduleType','ScheduleType');
#$bcp.ColumnMappings.Add('CustomType','CustomType');
#$bcp.ColumnMappings.Add('ShiftStartTime','ShiftStartTime');
#$bcp.ColumnMappings.Add('StartWeekDay','StartWeekDay');
#$bcp.ColumnMappings.Add('EndWeekDay','EndWeekDay');
$bcp.ColumnMappings.Add('WritebackProcessedTime','WritebackProcessedTime');
$bcp.ColumnMappings.Add('DiagStatus','DiagStatus');
$bcp.ColumnMappings.Add('AccountingPeriodCase_PK_ID','AccountingPeriodCase_PK_ID');
if ($connection.State -ne [Data.ConnectionState]::Open) {
'Connection to DB is not open.'
Exit
}
$bcp.WriteToServer($dt); #writing to server
$connection.Close();
The error I'm facing is:
Exception calling "WriteToServer" with "1" argument(s): "The given value of type String from the data source cannot be converted to type uniqueidentifier of the specified target column." At C:\Documents\DCT\SavedScripts\XMLtoDB\AccountingPeriodCases\sample.ps1:91 char:1
+ $bcp.WriteToServer($dt); #writing to server
+ ~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : InvalidOperationException
and the table I created has the columns with datatypes:
The thing is, I guess i need to convert the String that is in the datatable under the column ScheduleId and AccountingPeriodCase_PK_ID as they are not getting converted to uniqueidentifier.
I tried using
[System.Guid]::Parse($dt.Columns[4]);
[System.Guid]::Parse($dt.Columns[24]);
but the error repeats.
The xml content to be loaded under thr ScheduleId and AccountingPeriodCase_PK_ID looks like:
<ScheduleId>db6f3178-4702-456c-904b-2cd833b54efa</ScheduleId>
<AccountingPeriodCase_PK_ID>9544940d-8d1a-e711-80d3-0050569c2fb6</AccountingPeriodCase_PK_ID>
Could anyone help me resolve this issue?
Thank you
From a scenario, which I had faced earlier, it comes down to the source of your input.
The thing is if you are reading from an XML file, then you are bound to get an error, because while reading from a flat-file(txt, csv) or an XML, PowerShell will resolve the undefined types to String. The trick is to have some sort of control over the source. If your source is a PowerShell variable such as datatable or an array, then you can directly insert values from this variable to your destination table, since the original data types of the columns in a table are preserved.
This is the approach which I followed. In my case, the source was a result of the Invoke-Sqlcmd which preserved the original data types and hence made insertion error less.
Again as mentioned in the comment by OP, he was parsing values based on columns and not rows. This could also lead to an error. For example, there is one more way of inserting values in a table which is using the INSERT statement. Mind you that, the SQL Server INSERT statement inserts value on a row-level and not a column level. In such a scenario, parsing the values column wise may fail.

Troubleshooting DSC configuration

I'm looking for an advice on how to troubleshoot DSC configuration. I'm using Azure DSC Extension and the logs are:
VERBOSE: [2017-09-08 02:56:48Z] [VERBOSE] [sql-sql-0]:
[[xSqlServer]ConfigureSqlServerWithAlwaysOn] The file "MSDBLog" has been
modified in the system catalog. The new path will be used the next time the
database is started.
VERBOSE: [2017-09-08 02:56:50Z] [ERROR] Exception setting "StartupParameters":
"STARTUPPARAMETERS: unknown property."
VERBOSE: [2017-09-08 02:56:50Z] [VERBOSE] [sql-sql-0]:
[[xSqlServer]ConfigureSqlServerWithAlwaysOn] Stopping SQL server instance
'MSSQLSERVER' ...
VERBOSE: [2017-09-08 02:57:07Z] [VERBOSE] [sql-sql-0]:
[[xSqlServer]ConfigureSqlServerWithAlwaysOn] Starting SQL server instance
'MSSQLSERVER' ...
From there it just freezes then times out.
This is the only part of my DSC configuration that uses xSQLServer:
xSqlServer ConfigureSqlServerWithAlwaysOn
{
InstanceName = $env:COMPUTERNAME
SqlAdministratorCredential = $Admincreds
ServiceCredential = $SQLCreds
MaxDegreeOfParallelism = 1
FilePath = "F:\DATA"
LogPath = "G:\LOG"
DomainAdministratorCredential = $DomainFQDNCreds
DependsOn = "[xSqlLogin]AddSqlServerServiceAccountToSysadminServerRole"
}
This is the part of MicrosoftAzure_xSqlServer.psm1 that contains mentions of "startup":
function Alter-SystemDatabaseLocation([string]$FilePath, [string]$LogPath,[PSCredential]$ServiceCredential )
{
$permissionString = $ServiceCredential.UserName+":(OI)(CI)(F)"
icacls $FilePath /grant $permissionString
icacls $LogPath /grant $permissionString
Invoke-Sqlcmd "Use master"
Invoke-sqlCmd "ALTER DATABASE tempdb MODIFY FILE (NAME = tempdev, FILENAME = '$FilePath\tempdb.mdf');"
Invoke-sqlCmd "ALTER DATABASE tempdb MODIFY FILE (NAME = templog, FILENAME = '$LogPath\templog.ldf');"
Invoke-sqlCmd "ALTER DATABASE model MODIFY FILE (NAME = modeldev, FILENAME = '$FilePath\model.mdf');"
Invoke-sqlCmd "ALTER DATABASE model MODIFY FILE (NAME = modellog, FILENAME = '$LogPath\modellog.ldf');"
Invoke-sqlCmd "ALTER DATABASE msdb MODIFY FILE (NAME = MSDBData, FILENAME = '$FilePath\msdbdata.mdf');"
Invoke-sqlCmd "ALTER DATABASE msdb MODIFY FILE (NAME = MSDBLog, FILENAME = '$LogPath\msdblog.ldf');"
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SqlWmiManagement')| Out-Null
$smowmi = New-Object Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer
$sqlsvc = $smowmi.Services | Where-Object {$_.Name -like 'MSSQL*'}
$OldStartupParameters = $sqlsvc.StartupParameters
$params = '-d'+$FilePath+'\master.mdf;-e'+$LogPath+'\ERRORLOG;-l'+$LogPath+'\mastlog.ldf'
$sqlsvc[1].StartupParameters = $params
$sqlsvc[1].Alter()
}
What should be my next steps to understand the problem?
If it makes any difference, I'm trying to create SQL Server Always On Availability group on Windows Server 2016 and SQL Server 2016 SP1, by using templates and DSC code that work with Windows Server 2012 R2 and SQL Server 2014.
my advice is to use existing working code ;) I don't think it makes sense to copy paste the blog post. But I'll leave links to relevant files here.
Links:
1. DSC modules
2. ARM Template. Note: Arm template that deploys sql needs to have vnet and domain already in place. Also, you will get a hard time trying to deploy this template outside of that framework, so probably just copy out dsc extension\dsc scripts and use in your deployments.
3. Parameters file example
The xSqlServer (the new one) is not capable of moving sql to another disk, so you are stuck with custom script or this old xSql module. Also, please note, DSC modules inside packages are modified. These DSC configurations won't work with not modified versions of modules (xSqlCreateVirtualDataDisk, xDatabase, xSQLServerAlwaysOnAvailabilityGroupListener maybe some other modules, I can't recall at this point).
PS. working that configuration out and patching relevant parts wasn't exactly a pleasant journey...
PPS. that repo also contains dsc configuration for ADDS that can also run in parallel (compared to official example).
try this.
$sqlsvc = $smowmi.Services | Where-Object {$_.Name -eq 'MSSQLSERVER'}
$OldStartupParameters = $sqlsvc.StartupParameters
$params = '-d'+$FilePath+'\master.mdf;-e'+$LogPath+'\ERRORLOG;-l'+$LogPath+'\mastlog.ldf'
$sqlsvc[0].StartupParameters = $params
$sqlsvc[0].Alter()

Resources