Trying to Import an XML file into a MS SQL table using PowerShell - sql-server

I am trying to import an XML file into a MS SQL table using PowerShell, I have modified a PowerShell script I already had that worked fine but could do with some help as the new XML file has multiple entries for some elements.( I have included a sample of the XML file)
Table layout is below and as is a sample of the file I am trying to import and also my script I am trying to modify to get working.
Table layout is:
------------------------------
:supplier_id : name : Symbol :
and here is an part of the XML file itself
<SupplierMapping supplier_id="4536" name="Joby">
<Symbol>Joby</Symbol>
</SupplierMapping>
<SupplierMapping supplier_id="4537" name="ACT">
<Symbol>ACT</Symbol>
<Symbol>ADVANCED CABLE TECH</Symbol>
<Symbol>ADVANCED CABLE TECHNOLOGY</Symbol>
<Symbol>IEC LOCK</Symbol>
</SupplierMapping>
As you can see some supplier id's and names will have multiple <Symbol> names so I would like to have multiple entries in my table for those, however the script I have modified only seems to pull in the supplier_id and name elements and misses the <Symbol> part entirely. any help or guidance would be much appreciated.
Set-ExecutionPolicy Unrestricted -Scope LocalMachine
[String]$global:connectionString = "Data Source=Apps2\Apps2;Initial Catalog=DTEDATA;Integrated Security=SSPI";
[System.Data.DataTable]$global:dt = New-Object System.Data.DataTable;
[System.Xml.XmlTextReader]$global:xmlReader = New-Object System.Xml.XmlTextReader("C:\Scripts\icecat\supplier_mapping.xml");
[Int32]$global:batchSize = 100;
function Add-FileRow() {
$newRow = $dt.NewRow();
$null = $dt.Rows.Add($newRow);
$newRow["supplier_id"] = $global:xmlReader.GetAttribute("supplier_id");
$newRow["name"] = $global:xmlReader.GetAttribute("name");
$newRow["Symbol"] = $global:xmlReader.GetAttribute("Symbol");
}
# init data table schema
$da = New-Object System.Data.SqlClient.SqlDataAdapter("SELECT * FROM Supplier_Mapping_Test WHERE 0 = 1", $global:connectionString);
$null = $da.Fill($global:dt);
$bcp = New-Object System.Data.SqlClient.SqlBulkCopy($global:connectionString);
$bcp.DestinationTableName = "dbo.Supplier_Mapping_Test";
$recordCount = 0;
while ($xmlReader.Read() -eq $true) {
if (($xmlReader.NodeType -eq [System.Xml.XmlNodeType]::Element) -and ($xmlReader.Name -eq "SupplierMapping"))
Add-FileRow -xmlReader $xmlReader;
$recordCount += 1;
if (($recordCount % $global:batchSize) -eq 0) {
$bcp.WriteToServer($dt);
$dt.Rows.Clear();
Write-Host "$recordCount file elements processed so far";
}
}
}
if ($dt.Rows.Count -gt 0) {
$bcp.WriteToServer($dt);
}
$bcp.Close();
$xmlReader.Close();
Write-Host "$recordCount file elements imported ";
catch {
throw;
}

Related

How to remove encryption from all objects in SQL Server?

I have more than a hundred encrypted procedures and functions that I want to decrypt (I am trying a bacpac file export but it fails due to procedures being encrypted). I tried using dbforge sql decryptor decryption wizard for in place alter but I get the error:
Definition is invalid. Can't find CREATE keyword.
When I try to see the DDL script of a stored procedure(using dbforge sql decryptor), I get the error:
A definition for the object dbo.pt_blocks cannot be shown because it is encrypted by a third party tool
I can not find a resolution to this. Are there any solutions or other tools available for this?
Edit: I found this resource which mentions
take the source code and issue an ALTER command without the encryption option. Just take the source code and remove the WITH ENCRYPTION
How could I achieve this?
EDIT: I have enabled remote DAC. How can I decrypt everything? The accepted answer from this question has a broken link.
Edit: The problem has been solved by uninstalling a third party tool which was creating encrypted procedures.
Below is a PowerShell example that creates a script file of all encrypted objects, gleaned from Paul White's The Internals of WITH ENCRYPTION article. Change the data source and initial catalog in the 2 connection strings to the desired server and database as well as script file path.
A DAC connection is used to retrieve values from system tables so sysadmin server role membership is required. If run remotely, the SQL Server remote admin connections option must be enabled and TCP port 1434 allowed through the firewall.
The script can be run from the PowerShell ISE or from a command prompt after customization. Example command-line invocation, assuming script was saved to file "Decrypt-Objects.ps1".
powershell -ExecutionPolicy RemoteSigned -File C:\PowershellScripts\Decrypt-Objects.ps1
PowerShell script:
# PowerShell implementation of T-SQL code from https://sqlperformance.com/2016/05/sql-performance/the-internals-of-with-encryption
Function Get-DecryptedString($pwd, $data) {
$key = [System.Array]::CreateInstance([int], 256)
$box = [System.Array]::CreateInstance([int], 256)
$cipher = [System.Array]::CreateInstance([byte], $data.Length)
for ($i = 0; $i -lt 256; ++$i) {
$key[$i] = $pwd[$i % $pwd.Length]
$box[$i] = $i
}
for ($j = $i = 0; $i -lt 256; ++$i) {
$j = ($j + $box[$i] + $key[$i]) % 256
$tmp = $box[$i]
$box[$i] = $box[$j]
$box[$j] = $tmp
}
for ($a = $j = $i = 0; $i -lt $data.Length; ++$i) {
++$a
$a %= 256
$j += $box[$a]
$j %= 256
$tmp = $box[$a]
$box[$a] = $box[$j]
$box[$j] = $tmp
$k = $box[(($box[$a] + $box[$j]) % 256)]
$cipher[$i] = ($data[$i] -bxor $k)
}
$decryptedString = [System.Text.Encoding]::Unicode.GetString($cipher)
return $decryptedString
}
Function Get-ClearObjectText($connectionString, $objectName) {
$getRc4KeyQuery = #"
DECLARE
#objectid integer = OBJECT_ID(#ObjectName),
#family_guid binary(16),
#objid binary(4),
#subobjid binary(2);
-- Find the database family GUID
SELECT #family_guid = CONVERT(binary(16), DRS.family_guid)
FROM sys.database_recovery_status AS DRS
WHERE DRS.database_id = DB_ID();
-- Convert object ID to little-endian binary(4)
SET #objid = CONVERT(binary(4), REVERSE(CONVERT(binary(4), #objectid)));
SELECT
-- Read the encrypted value
#imageval = SOV.imageval,
-- Get the subobjid and convert to little-endian binary
#subobjid = CONVERT(binary(2), REVERSE(CONVERT(binary(2), SOV.subobjid)))
FROM sys.sysobjvalues AS SOV
WHERE
SOV.[objid] = #objectid
AND SOV.valclass = 1;
-- Compute the RC4 initialization key
SELECT #RC4key = HASHBYTES('SHA1', #family_guid + #objid + #subobjid);
"#
$connection = New-Object System.Data.SqlClient.SqlConnection($dacConnectionString)
$connection.Open()
$command = New-Object System.Data.SqlClient.SqlCommand($getRc4KeyQuery, $connection)
($command.Parameters.Add("#ObjectName", [System.Data.SqlDbType]::NVarChar, 261)).Value = $objectName
($command.Parameters.Add("#imageval", [System.Data.SqlDbType]::VarBinary, -1)).Direction = [System.Data.ParameterDirection]::Output
($command.Parameters.Add("#RC4key", [System.Data.SqlDbType]::Binary, 20)).Direction = [System.Data.ParameterDirection]::Output
[void]$command.ExecuteNonQuery()
$imageval = $command.Parameters["#imageval"].Value
$RC4key = $command.Parameters["#RC4key"].Value
$connection.Close()
$decryptedString = Get-DecryptedString -pwd $RC4key -data $imageval
Return $decryptedString
}
# ############
# ### MAIN ###
# ############
# DAC connection string for decryption
$dacConnectionString = "Data Source=admin:YourServer;Initial Catalog=YourDatabase;Integrated Security=SSPI"
# normal connection string for encrypted object list
$connectionString = "Data Source=YourServer;Initial Catalog=YourDatabase;Integrated Security=SSPI"
# target file path for clear encrypted objects DDL
$scriptFilePath = "C:\Scripts\EncryptedObjects.sql"
[void](New-Item -Path "C:\Scripts\EncryptedObjects.sql" -ItemType file -Force) # create directory (if needed) and empty script file
$EncryptedObjectQuery = #"
SELECT
QUOTENAME(OBJECT_SCHEMA_NAME(object_id)) + '.' + QUOTENAME(name) AS QualifiedObjectName
FROM sys.objects
WHERE OBJECTPROPERTY(object_id, 'IsEncrypted') = 1;
"#
try {
$connection = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$command = New-Object System.Data.SqlClient.SqlCommand($EncryptedObjectQuery, $connection)
$connection.Open()
$reader = $command.ExecuteReader()
while ($reader.Read()) {
$createObjectScript = Get-ClearObjectText -connectionString $dacConnectionString -objectName $reader["QualifiedObjectName"]
$createObjectScript | Out-File -FilePath $scriptFilePath -Append
"GO" | Out-File -FilePath $scriptFilePath -Append
}
$connection.Close()
}
catch {
throw
}

SMO to script just the database and it settings (not the objects in it)

I have put together a nice PowerShell script to script out the objects (tables, functions, sprocs etc) from a database, limiting it to the ones in a list.
But I am stuck trying to find a way to script the database itself. Each time I do that, it seems to try to script out the whole database (it is way to large for that to go well).
Assuming I have a $db variable that is a reference to my database, how can I use SMO to script out that database, creating it with the same Properties and DatabaseScopedConfigurations, but none of the actual objects in it?
Update:
For reference here is my current script. It takes a server and database name and will script out all the objects found in a file called DbObjectsList.txt (assuming they are in the database). But this does not actually make the database. The database I am running this on is a legacy one, and it has a bunch of odd options set. I would like to preserve those.
$serverName = "MyServerName"
$databaseName = "MyDbName"
$date_ = (date -f yyyyMMdd)
$path = ".\"+"$date_"
# Load the Sql Server Management Objects (SMO) and output to null so we don't show the dll details.
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') > $null
# Setup the scripting options
$scriptOptions = new-object ('Microsoft.SqlServer.Management.Smo.ScriptingOptions')
$scriptOptions.ExtendedProperties = $true
$scriptOptions.AnsiPadding = $true
$scriptOptions.ClusteredIndexes = $true
# Dri = Declarative Referential Integrity
$scriptOptions.DriAll = $true
$scriptOptions.Triggers = $true
$scriptOptions.NoCollation = $false
$scriptOptions.SchemaQualify = $true
$scriptOptions.ScriptSchema = $true
$scriptOptions.EnforceScriptingOptions = $true
$scriptOptions.SchemaQualifyForeignKeysReferences = $true
$scriptOptions.NonClusteredIndexes = $true
$scriptOptions.Statistics = $true
$scriptOptions.Permissions = $true
$scriptOptions.OptimizerData = $true
# get a reference to the database we are going to be scripting from
$serverInstance = New-Object ('Microsoft.SqlServer.Management.Smo.Server') $serverName
$db=$serverInstance.Databases | Where-Object {$_.Name -eq $databaseName}
$dbname = "$db".replace("[","").replace("]","")
$dbpath = "$path"+ "\"+"$dbname" + "\"
if ( !(Test-Path $dbpath))
{
$null=new-item -type directory -name "$dbname"-path "$path"
}
# Load the list of db objects we want to script.
$listPath = ".\DbObjectList.txt"
if ((Test-Path $listPath))
{
$dbListItems = Get-Content -Path $listPath
}
else
{
throw "Could not find DbObjectst.txt file (it should have a list of what to script)."
}
# Setup the output file, removing any existing one
$outFile = "$dbpath" + "FullScript.sql"
if ((Test-Path $outFile)){Remove-Item $outFile }
$typeDelimiter = "=========="
foreach ($dbListItem in $dbListItems)
{
# Let the caller know which one we are working on.
echo $dbListItem
if ($dbListItem.StartsWith($typeDelimiter))
{
# Pull the type out of the header
$startIndex = $typeDelimiter.Length;
$stopIndex = $dbListItem.LastIndexOf($typeDelimiter)
$type = $dbListItem.Substring($startIndex, $stopIndex - $startIndex).Trim()
continue;
}
if ($type -eq $null)
{
throw "Types not included DbObjectsList.txt. Add types before groups of objects, surrounded by " + $typeDelimiter
}
foreach ($dbObjectToScript in $db.$type)
{
$objName = "$dbObjectToScript".replace("[","").replace("]","")
$compareDbListItem = "$dbListItem".replace("[","").replace("]","")
if ($compareDbListItem -eq $objName)
{
"-- " + $dbListItem | out-File -Append $outFile
$dbObjectToScript.Script($scriptOptions)+"GO" | out-File -Append $outFile
}
}
}

Powershell function to import csv file to SQL Server database table

I have created a PowerShell function that bulk copies data from a .csv file (first row is the header), and inserts the data in to a SQL Server database table.
See my code:
function BulkCsvImport($sqlserver, $database, $table, $csvfile, $csvdelimiter, $firstrowcolumnnames) {
Write-Host "Bulk Import Started."
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
[void][Reflection.Assembly]::LoadWithPartialName("System.Data")
[void][Reflection.Assembly]::LoadWithPartialName("System.Data.SqlClient")
# 50k worked fastest and kept memory usage to a minimum
$batchsize = 50000
# Build the sqlbulkcopy connection, and set the timeout to infinite
$connectionstring = "Data Source=$sqlserver;Integrated Security=true;Initial Catalog=$database;"
# Wipe the bulk insert table first
Invoke-Sqlcmd -Query "TRUNCATE TABLE $table" -ServerInstance $sqlserver -Database $database
$bulkcopy = New-Object Data.SqlClient.SqlBulkCopy($connectionstring, [System.Data.SqlClient.SqlBulkCopyOptions]::TableLock)
$bulkcopy.DestinationTableName = $table
$bulkcopy.bulkcopyTimeout = 0
$bulkcopy.batchsize = $batchsize
# Create the datatable, and autogenerate the columns.
$datatable = New-Object System.Data.DataTable
# Open the text file from disk
$reader = New-Object System.IO.StreamReader($csvfile)
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
if ($firstrowcolumnnames -eq $true) { $null = $reader.readLine() }
foreach ($column in $columns) {
$null = $datatable.Columns.Add()
}
# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null) {
$null = $datatable.Rows.Add($line.Split($csvdelimiter))
$i++;
if (($i % $batchsize) -eq 0) {
$bulkcopy.WriteToServer($datatable)
Write-Host "$i rows have been inserted in $($elapsed.Elapsed.ToString())."
$datatable.Clear()
}
}
# Add in all the remaining rows since the last clear
if($datatable.Rows.Count -gt 0) {
$bulkcopy.WriteToServer($datatable)
$datatable.Clear()
}
# Clean Up
$reader.Close();
$reader.Dispose()
$bulkcopy.Close();
$bulkcopy.Dispose()
$datatable.Dispose()
Write-Host "Bulk Import Completed. $i rows have been inserted into the database."
# Write-Host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"
# Sometimes the Garbage Collector takes too long to clear the huge datatable.
$i = 0
[System.GC]::Collect()
}
I am looking to modify the above though so that the column names in the .csv file match up with the column names in the SQL Server database table. They should be identical. At the moment the data is being imported in to the incorrect database columns.
Could I get some assistance as what I need to do to modify the above function to achieve this?
I would use existing open source solution:
Import-DbaCsv - dbatools.io
Import-DbaCsv.ps1
Efficiently imports very large (and small) CSV files into SQL Server.
Import-DbaCsv takes advantage of .NET's super fast SqlBulkCopy class to import CSV files into SQL Server.
Parameters:
-ColumnMap
By default, the bulk copy tries to automap columns. When it doesn't
work as desired, this parameter will help.
PS C:\> $columns = #{
>> Text = 'FirstName'
>> Number = 'PhoneNumber'
>> }
PS C:\> Import-DbaCsv -Path c:\temp\supersmall.csv
-SqlInstance sql2016 -Database tempdb -ColumnMap $columns
-BatchSize 50000 -Table table_name -Truncate
The CSV column 'Text' is inserted into SQL column 'FirstName' and CSV column Number is inserted into the SQL Column 'PhoneNumber'. All other columns are ignored and therefore null or default values.

Using Powershell to Bulk Import Large CSV into SQL Server

I came across a post discussing how to use Powershell to bulk import massive data relatively fast. I have a typical csv file with about 5 million rows formatted in the usual way.
I keep getting the same error messages regardless if I choose to import a txt or csv file. Playing around with the csvdelimiter/firstcolumnnames section also created their own issues.
I've spent hours trying to figure out how to get it to work with MY csv files and I keep getting the same error messages no matter what I try. All field names accept Null and they are identical in every way between the table and csv file. I do not have a primary key for the database.
# Database variables
$sqlserver = "SERVERNAMEHERE"
$database = "autos"
$table = "AgedAutos"
# CSV variables
$csvfile = "C:\temp\aged.csv"
$csvdelimiter = "',"
$firstRowColumnNames = $true
################### No need to modify anything below ###################
Write-Host "Script started..."
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
[void][Reflection.Assembly]::LoadWithPartialName("System.Data")
[void][Reflection.Assembly]::LoadWithPartialName("System.Data.SqlClient")
# 50k worked fastest and kept memory usage to a minimum
$batchsize = 50000
# Build the sqlbulkcopy connection, and set the timeout to infinite
$connectionstring = "Data Source=$sqlserver;Integrated Security=true;Initial Catalog=$database;"
$bulkcopy = New-Object Data.SqlClient.SqlBulkCopy($connectionstring, [System.Data.SqlClient.SqlBulkCopyOptions]::TableLock)
$bulkcopy.DestinationTableName = $table
$bulkcopy.bulkcopyTimeout = 0
$bulkcopy.batchsize = $batchsize
# Create the datatable, and autogenerate the columns.
$datatable = New-Object System.Data.DataTable
# Open the text file from disk
$reader = New-Object System.IO.StreamReader($csvfile)
$columns = (Get-Content $csvfile -First 1).Split($csvdelimiter)
if ($firstRowColumnNames -eq $true) { $null = $reader.readLine() }
foreach ($column in $columns) {
$null = $datatable.Columns.Add()
}
# Read in the data, line by line
while (($line = $reader.ReadLine()) -ne $null) {
$null = $datatable.Rows.Add($line.Split($csvdelimiter))
$i++; if (($i % $batchsize) -eq 1) {
$bulkcopy.WriteToServer($datatable)
Write-Host "$i rows have been inserted in $($elapsed.Elapsed.ToString())."
$datatable.Clear()
}
}
# Add in all the remaining rows since the last clear
if($datatable.Rows.Count -gt 0) {
$bulkcopy.WriteToServer($datatable)
$datatable.Clear()
}
# Clean Up
$reader.Close(); $reader.Dispose()
$bulkcopy.Close(); $bulkcopy.Dispose()
$datatable.Dispose()
Write-Host "Script complete. $i rows have been inserted into the database."
Write-Host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"
# Sometimes the Garbage Collector takes too long to clear the huge datatable.
[System.GC]::Collect()
Error message listed below.
Exception calling "WriteToServer" with "1" argument(s): "The given value of type String from the data source cannot be converted to
type date of the specified target column."
At C:\powershell_scripts\batch_csv_import-code1-working-test for auto table.ps1:43 char:3
+ $bulkcopy.WriteToServer($datatable)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : InvalidOperationException
340000 rows have been inserted in 00:00:03.5156162
I have no idea what that error means since I cannot find anything useful on Google. I'm thinking one of the columns might be listed incorrectly in SQL Server, but I could be wrong.
Please help me figure out the problem. Thanks.
You are getting all the data in the first column because your value for $csvdelimiter is incorrect.
you have: $csvdelimiter = "',"
it should be: $csvdelimiter = ","

Powershell Write-Host showing only dataTable name instead of data

I'm trying to write a Powershell script that executes a SQL query contained in a .sql file
Function RunSQLScript ($connstring, $filePath)
{
$query = get-content $filePath;
$DTSet = New-Object System.Data.DataSet;
$Conn=New-Object System.Data.SQLClient.SQLConnection $connstring;
$Conn.Open();
try
{
$DataCmd = New-Object System.Data.SqlClient.SqlCommand;
$MyQuery = $query;
$DataCmd.CommandText = $MyQuery;
$DataCmd.Connection = $Conn;
$DAadapter = New-Object System.Data.SqlClient.SqlDataAdapter;
$DAadapter.SelectCommand = $DataCmd;
$DAadapter.Fill($DTSet) | Out-Null;
for ($i = 0; $i -lt $DTSet.Tables.Count; $i++) {
Write-Host $DTSet.Tables[$i];
}
}
finally
{
$Conn.Close();
$Conn.Dispose();
}
return $DTSet;
}
The internal Write-Host is showing the DataTable name instead of the DataRows.
If I manually create a DataSet with a DataTable in Powershell Console, Write-Host shows me the data in the DataTable rows, so I can't really figure out why it is not doing that in the previous script.
Can you give me some clues on how to show the data contained in the datatables instead of the table names?
Thank you
This piece of code was quite helpful for me, posting it here if anybody needs it.
for ($i = 0; $i -lt $DTSet.Tables.Count; $i++) {
$DTSet.Tables[$i] | format-table | out-host
}
That produces a nice table-like output on screen.

Resources