Sqlbulkcopy Excessive Memory Consumtion even with EnableStreaming and low BatchSize - sql-server

I try to bulk load data from Oracle to SqlServer through Powershell Sqlserver Module Sqlbulkcopy
On small Data, everything works fine, but on big Datasets, even if bachsize and streaming are set, sqlbulkcopy is taking all the memory available... until an out of memory
Also the notify function seems to give no answer, so I guess even with streaming=True, the process first load everything to memory...
What did I missed ?
$current = Get-Date
#copy table from Oracle table to SQL Server table
add-type -path "D:\oracle\product\12.1.0\client_1\odp.net\managed\common\Oracle.ManagedDataAccess.dll";
#define oracle connectin string
$conn_str = "cstr"
# query for oracle table
$qry = "
SELECT
ID,CREATEDT,MODIFIEDDT
FROM MYTABLE
WHERE source.ISSYNTHETIC=0 AND source.VALIDFROM >= TO_Date('2019-01-01','yyyy-mm-dd')
";
# key (on the left side) is the source column while value (on the right side) is the target column
[hashtable] $mapping = #{'ID'='ID';'CREATEDT'='CREATEDT';'MODIFIEDDT'};
$adapter = new-object Oracle.ManagedDataAccess.Client.OracleDataAdapter($qry, $conn_str);
#$info = new-object Oracle.ManagedDataAccess.Client;
#Write-Host ( $info | Format-Table | Out-String)
$dtbl = new-object System.Data.DataTable('MYTABLE');
#this Fill method will populate the $dtbl with the query $qry result
$adapter.Fill($dtbl);
#define sql server target instance
$sqlconn = "cstr";
$sqlbc = new-object system.data.sqlclient.Sqlbulkcopy($sqlconn)
$sqlbc.BatchSize = 1000;
$sqlbc.EnableStreaming = $true;
$sqlbc.NotifyAfter = 1000;
$sqlbc.DestinationTableName="DWHODS.MYTABLE";
#need to tell $sqlbc the column mapping info
foreach ($k in $mapping.keys)
{
$colMapping = new-object System.Data.SqlClient.SqlBulkCopyColumnMapping($k, $mapping[$k]);
$sqlbc.ColumnMappings.Add($colMapping) | out-null
}
$sqlbc.WriteToServer($dtbl);
$sqlbc.close;
$end= Get-Date
$diff= New-TimeSpan -Start $current -End $end
Write-Output "import needed : $diff"

Thanks to Jeroen, I changed the code like this, now its no more consuming memory :
$oraConn = New-Object Oracle.ManagedDataAccess.Client.OracleConnection($conn_str);
$oraConn.Open();
$command = $oraConn.CreateCommand();
$command.CommandText=$qry;
$reader = $command.ExecuteReader()
...
$sqlbc.WriteToServer($reader);

Related

How to remove encryption from all objects in SQL Server?

I have more than a hundred encrypted procedures and functions that I want to decrypt (I am trying a bacpac file export but it fails due to procedures being encrypted). I tried using dbforge sql decryptor decryption wizard for in place alter but I get the error:
Definition is invalid. Can't find CREATE keyword.
When I try to see the DDL script of a stored procedure(using dbforge sql decryptor), I get the error:
A definition for the object dbo.pt_blocks cannot be shown because it is encrypted by a third party tool
I can not find a resolution to this. Are there any solutions or other tools available for this?
Edit: I found this resource which mentions
take the source code and issue an ALTER command without the encryption option. Just take the source code and remove the WITH ENCRYPTION
How could I achieve this?
EDIT: I have enabled remote DAC. How can I decrypt everything? The accepted answer from this question has a broken link.
Edit: The problem has been solved by uninstalling a third party tool which was creating encrypted procedures.
Below is a PowerShell example that creates a script file of all encrypted objects, gleaned from Paul White's The Internals of WITH ENCRYPTION article. Change the data source and initial catalog in the 2 connection strings to the desired server and database as well as script file path.
A DAC connection is used to retrieve values from system tables so sysadmin server role membership is required. If run remotely, the SQL Server remote admin connections option must be enabled and TCP port 1434 allowed through the firewall.
The script can be run from the PowerShell ISE or from a command prompt after customization. Example command-line invocation, assuming script was saved to file "Decrypt-Objects.ps1".
powershell -ExecutionPolicy RemoteSigned -File C:\PowershellScripts\Decrypt-Objects.ps1
PowerShell script:
# PowerShell implementation of T-SQL code from https://sqlperformance.com/2016/05/sql-performance/the-internals-of-with-encryption
Function Get-DecryptedString($pwd, $data) {
$key = [System.Array]::CreateInstance([int], 256)
$box = [System.Array]::CreateInstance([int], 256)
$cipher = [System.Array]::CreateInstance([byte], $data.Length)
for ($i = 0; $i -lt 256; ++$i) {
$key[$i] = $pwd[$i % $pwd.Length]
$box[$i] = $i
}
for ($j = $i = 0; $i -lt 256; ++$i) {
$j = ($j + $box[$i] + $key[$i]) % 256
$tmp = $box[$i]
$box[$i] = $box[$j]
$box[$j] = $tmp
}
for ($a = $j = $i = 0; $i -lt $data.Length; ++$i) {
++$a
$a %= 256
$j += $box[$a]
$j %= 256
$tmp = $box[$a]
$box[$a] = $box[$j]
$box[$j] = $tmp
$k = $box[(($box[$a] + $box[$j]) % 256)]
$cipher[$i] = ($data[$i] -bxor $k)
}
$decryptedString = [System.Text.Encoding]::Unicode.GetString($cipher)
return $decryptedString
}
Function Get-ClearObjectText($connectionString, $objectName) {
$getRc4KeyQuery = #"
DECLARE
#objectid integer = OBJECT_ID(#ObjectName),
#family_guid binary(16),
#objid binary(4),
#subobjid binary(2);
-- Find the database family GUID
SELECT #family_guid = CONVERT(binary(16), DRS.family_guid)
FROM sys.database_recovery_status AS DRS
WHERE DRS.database_id = DB_ID();
-- Convert object ID to little-endian binary(4)
SET #objid = CONVERT(binary(4), REVERSE(CONVERT(binary(4), #objectid)));
SELECT
-- Read the encrypted value
#imageval = SOV.imageval,
-- Get the subobjid and convert to little-endian binary
#subobjid = CONVERT(binary(2), REVERSE(CONVERT(binary(2), SOV.subobjid)))
FROM sys.sysobjvalues AS SOV
WHERE
SOV.[objid] = #objectid
AND SOV.valclass = 1;
-- Compute the RC4 initialization key
SELECT #RC4key = HASHBYTES('SHA1', #family_guid + #objid + #subobjid);
"#
$connection = New-Object System.Data.SqlClient.SqlConnection($dacConnectionString)
$connection.Open()
$command = New-Object System.Data.SqlClient.SqlCommand($getRc4KeyQuery, $connection)
($command.Parameters.Add("#ObjectName", [System.Data.SqlDbType]::NVarChar, 261)).Value = $objectName
($command.Parameters.Add("#imageval", [System.Data.SqlDbType]::VarBinary, -1)).Direction = [System.Data.ParameterDirection]::Output
($command.Parameters.Add("#RC4key", [System.Data.SqlDbType]::Binary, 20)).Direction = [System.Data.ParameterDirection]::Output
[void]$command.ExecuteNonQuery()
$imageval = $command.Parameters["#imageval"].Value
$RC4key = $command.Parameters["#RC4key"].Value
$connection.Close()
$decryptedString = Get-DecryptedString -pwd $RC4key -data $imageval
Return $decryptedString
}
# ############
# ### MAIN ###
# ############
# DAC connection string for decryption
$dacConnectionString = "Data Source=admin:YourServer;Initial Catalog=YourDatabase;Integrated Security=SSPI"
# normal connection string for encrypted object list
$connectionString = "Data Source=YourServer;Initial Catalog=YourDatabase;Integrated Security=SSPI"
# target file path for clear encrypted objects DDL
$scriptFilePath = "C:\Scripts\EncryptedObjects.sql"
[void](New-Item -Path "C:\Scripts\EncryptedObjects.sql" -ItemType file -Force) # create directory (if needed) and empty script file
$EncryptedObjectQuery = #"
SELECT
QUOTENAME(OBJECT_SCHEMA_NAME(object_id)) + '.' + QUOTENAME(name) AS QualifiedObjectName
FROM sys.objects
WHERE OBJECTPROPERTY(object_id, 'IsEncrypted') = 1;
"#
try {
$connection = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$command = New-Object System.Data.SqlClient.SqlCommand($EncryptedObjectQuery, $connection)
$connection.Open()
$reader = $command.ExecuteReader()
while ($reader.Read()) {
$createObjectScript = Get-ClearObjectText -connectionString $dacConnectionString -objectName $reader["QualifiedObjectName"]
$createObjectScript | Out-File -FilePath $scriptFilePath -Append
"GO" | Out-File -FilePath $scriptFilePath -Append
}
$connection.Close()
}
catch {
throw
}

Looping through pscustomobject dataset and add to a SQL DB

I have a variable, $CiscoCMDB, which holds data for 2968 records like this (some fields are all filled, some are not like below):
SearchCode: D12345678911
Name1: 1212
Category: Office Phone
AssetTag:
Status: Stock
SerialNumber: FCH1549BBBB
Model: CISCO IP PHONE 7945G
MacAddress:
Location: SF
OwnerOrganization:
OwnerPerson:
I also have a SQL DB created, CiscoCMDB, with the same name for the columns that are present for the rows. I would like All records to be written to the SQL table. I've tried to use the Write-Datatable cmdlet, but it gave me errors. along with other CMDlets. I figured, if I am using SQL SELECT statements in my code, I should just use SQL INSERTs in my code. Only problem is, I'm not sure how to go about this other than knowing I will need a foreach loop to loop through.
Can someone help me on this as I'm getting confused and frustrated. I'm not exactly sure how to go about looping through correctly.
Pull data from another SQL DB
Organize data furthur (picking the columns I want)
Manipulate data to strip ^C from searchcode (all our search codes are prefaced with C)
Manipulate data to strip ^DN from Name1 field (again all Name1s are prefaced with DN)
Fill in my data
$CMDBCiscoQuery = #()
foreach ($row in $table) {
$hash = [ordered]#{
'SearchCode' = $row.SearchCode
'Category' = $row.Category
'Status' = $row.Status
'Name1' = $row.Name1
'SerialNumber' = $row.SerialNumber
'Model' = $row.Model
'MacAddress' = $row.MacAddress
'Location' = $row.Location
'OwnerOrg' = $row.OwnerOrganization
'Owner' = $row.OwnerPerson
}
$obj = New-Object -TypeName PSObject -Property $hash
$CMDBCiscoQuery+= $obj
$CMDBCisco = $CMDBCiscoQuery |
Select #{n='SearchCode';e={$_.Searchcode -replace "^C"}},
#{n='Name1';e={$_.DN -replace "^DN"}}, Category, AssetTag,
Status, SerialNumber, Model, MacAddress, Location,
OwnerOrganization, OwnerPerson
#Filling CMDB table
Write-Verbose "Filling CMDBCisco Table"
$ConnectionTimeout = 30
$ServerInstance = "myserver"
$Database = "Audits"
$conn = New-Object System.Data.SqlClient.SQLConnection
$ConnectionString = "Server={0};Database={1};Connect Timeout={2};Integrated Security = True;" -f $ServerInstance,$Database,$ConnectionTimeout
$conn.ConnectionString = $ConnectionString
$conn.Open()
$cmd = New-Object System.Data.SqlClient.SqlCommand($conn);
$conn.Close()

PowerShell pipe / SQL insert query data limits (and increasing them?)

Are there any limits to the size of a string you can assign to a variable in powershell or any limits to the size of the text sent within an SQL INSERT query?
I have a big CSV file coming in to PowerShell and through string construction in a foreach loop I am generating SQL INSERT queries for each row. The resulting INSERT query; INSERT query; is over about 4MB.
The SQL server has a perfect schema to receive the data, however, when sending the 4MB collection of INSERT queries (each seperated by ;) I get an error that looks to me like the long 4MB set of insert queries was truncated somehow. I guess I have hit some kind of limit.
Is there a way of getting around this (programatically in PowerShell) or a way of increasing the size limit of an acceptable collection of SQL INSERT queries?
My code is using System.Data.SqlClient.SqlConnection and System.Data.sqlclient.SqlCommand.
Smaller datasets work ok but the larger datasets give an error like the following example. Each different dataset gives off a different "Incorrect syntax near" indicator.
Exception calling "ExecuteNonQuery" with "0" argument(s): "Incorrect syntax
near '('."
At C:\Users\stuart\Desktop\git\ADStfL\WorkInProgress.ps1:211 char:3
+ $SQLCommand.executenonquery()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SqlException
In my experience, the best performing way to do this is to load the CSV into a DataTable and then use SQLBulkCopy.
$ErrorActionPreference = 'Stop';
$Csv = Import-Csv -Path $FileName;
$SqlServer = 'MyServer';
$SqlDatabase = 'MyDatabase';
$DestinationTableName = 'MyTable';
# Create Connection String
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase;
# Define your DataTable. The column order of the DataTable must either match the table in the database, or
# you must specify the column mapping in SqlBulkCopy.ColumnMapping. If you have an IDENTITY column, it's a
# bit more complicated
$DataTable = New-Object -TypeName System.Data.DataTable -ArgumentList $DestinationTableName;
$NewColumn = $DataTable.Columns.Add('Id',[System.Int32]);
$NewColumn.AllowDBNull = $false;
$NewColumn = $DataTable.Columns.Add('IntegerField',[System.Int32]);
$NewColumn.AllowDBNull = $false;
$NewColumn = $DataTable.Columns.Add('DecimalField',[System.Decimal]);
$NewColumn.AllowDBNull = $false;
$NewColumn = $DataTable.Columns.Add('VarCharField',[System.String]);
$NewColumn.MaxLength = 50;
$NewColumn = $DataTable.Columns.Add('DateTimeField',[System.DateTime]);
$NewColumn.AllowDBNull = $false;
# Populate your datatable from the CSV file
# You may find that you need to type cast some of the fields.
$Csv | ForEach-Object {
$NewRow = $DataTable.NewRow();
$NewRow['Id'] = $_.Id;
$NewRow['IntegerField'] = $_.IntegerField;
$NewRow['DecimalField'] = $_.DecimalFiled;
$NewRow['StringField'] = $_.StringField1;
$NewRow['DateTimeField'] = $_.DateTimeField1;
$DataTable.Rows.Add($NewRow);
}
# Create Connection
$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString;
# Open Connection
$SqlConnection.Open();
# Start Transaction
$SqlTransaction = $SqlConnection.BeginTransaction();
# Double check the possible options at https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopyoptions(v=vs.110).aspx
# If you need multiple then -bor them together
$SqlBulkCopyOptions = [System.Data.SqlClient.SqlBulkCopyOptions]::CheckConstraints;
# Create SqlBulkCopy class
$SqlBulkCopy = New-Object -TypeName System.Data.SqlClient.SqlBulkCopy -ArgumentList $SqlConnection, $SqlBulkCopyOptions, $SqlTransaction;
# Specify destination table
$SqlBulkCopy.DestinationTableName = $DestinationTableName;
# Do the insert; rollback on error
try {
$SqlBulkCopy.WriteToServer($DataTable);
$SqlTransaction.Commit();
}
catch {
# Roll back transaction and rethrow error
$SqlTransaction.Rollback();
throw ($_);
}
finally {
$SqlConnection.Close();
$SqlConnection.Dispose();
}
The other method is to use an SQLCommand and do it row by row:
$ErrorActionPreference = 'Stop';
$Csv = Import-Csv -Path $FileName;
$SqlServer = 'MyServer';
$SqlDatabase = 'MyDatabase';
# Create Connection String
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase;
# Create Connection
$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString;
# Create Command
$InsertCommandText = 'INSERT INTO DestinationTable (Id, IntegerField, DecimalField, StringField, DateTimeField) VALUES (#Id, #IntegerField, #DecimalField, #StringField, #DateTimeField)';
$InsertCommand = New-Object -TypeName System.Data.SqlClient.SqlCommand -ArgumentList $SqlConnection;
[void]$InsertCommand.Parameters.Add('#Id', [System.Data.SqlDbType]::Int);
[void]$InsertCommand.Parameters.Add('#IntegerField', [System.Data.SqlDbType]::Int);
[void]$InsertCommand.Parameters.Add('#DecimalField', [System.Data.SqlDbType]::Decimal);
[void]$InsertCommand.Parameters.Add('#StringField', [System.Data.SqlDbType]::VarChar,50);
[void]$InsertCommand.Parameters.Add('#DateTimeField', [System.Data.SqlDbType]::DateTime);
# Open connection and start transaction
$SqlConnection.Open()
$SqlTransaction = $SqlConnection.BeginTransaction();
$InsertCommand.Transaction = $SqlTransaction;
$RowsInserted = 0;
try {
$line = 0;
$Csv | ForEach-Object {
$line++;
# Specify parameter values
$InsertCommand.Parameters['#Id'].Value = $_.Id;
$InsertCommand.Parameters['#IntegerField'].Value = $_.IntegerField;
$InsertCommand.Parameters['#DecimalField'].Value = $_.DecimalField;
$InsertCommand.Parameters['#StringField'].Value = $_.StringField;
$InsertCommand.Parameters['#DateTimeField'].Value = $_.DateTimeField;
$RowsInserted += $InsertCommand.ExecuteNonQuery();
# Clear parameter values
$InsertCommand.Parameters | ForEach-Object { $_.Value = $null };
}
$SqlTransaction.Commit();
Write-Output "Rows affected: $RowsInserted";
}
catch {
# Roll back transaction and rethrow error
$SqlTransaction.Rollback();
Write-Error "Error on line $line" -ErrorAction Continue;
throw ($_);
}
finally {
$SqlConnection.Close();
$SqlConnection.Dispose();
}
Edit: Oh, I forgot one important point. If you need to set the value of a field to null in the database, you need to set it's value to [System.DBNull]::Value, not $null.

Powershell - proper way to execute SQL query with multiple select statements and result tables

I'm trying to execute an SQL query with few select statements, that returns multiple tables as a result. The problem is that I can't find a way to read and use the tables separately.
Expected results:
Actual results: (it is printed row by row)
Purpose: I've made a script that creates an empty excel file with multiple sheets and each of the sheets will be used to contain each resultset of the query.
The only thing left is to put the needed text into the sheets. Here is my code for that part only:
$ConnectionString = "Data Source=...;Initial Catalog=...;User Id=...;Password=..."
$DBServerName = $ConnectionString.split('=')[1].split(';')[0]
$DBName = $ConnectionString.split('=')[2].split(';')[0]
$DBUser = $ConnectionString.split('=')[3].split(';')[0]
$DBPassword = $ConnectionString.split('=')[4].split(';')[0]
$CurrentFilePath = "C:\SQLqueryWithManyResultsets.sql"
$query = Get-Content -literalPath $CurrentFilePath | Out-String #getting the query string from file
$resultTables = Invoke-Sqlcmd -Query $query -ServerInstance $DBServerName -Database $DBName -DisableVariables -Password $DBPassword -Username $DBUser -ErrorAction Stop
foreach ($result in $resultTables) {
$result | Format-Table #where the magic happens
}
I've made a lot of research, but I cannot find a proper way to store and read the tables the way i need.
Try this:
Clear-Host;
$objConnection = New-Object System.Data.SqlClient.SqlConnection;
$objConnection.ConnectionString = "...";
$ObjCmd = New-Object System.Data.SqlClient.SqlCommand;
$ObjCmd.CommandText = "...";
$ObjCmd.Connection = $objConnection;
$ObjCmd.CommandTimeout = 0;
$objAdapter = New-Object System.Data.SqlClient.SqlDataAdapter;
$objAdapter.SelectCommand = $ObjCmd;
$objDataSet = New-Object System.Data.DataSet;
$objAdapter.Fill($objDataSet) | Out-Null;
for ($i=0; $i -lt $objDataSet.Tables.Count; $i++) {
Write-Host ($objDataSet.Tables[$i] | Format-Table | Out-String);
}
$query = $null;
$objDataSet = $null;
$objConnection.Close();
$objConnection = $null;

Getting data from SQL Server 2008 with Powershell

I'm having some difficulties getting data from a simple stored procedure in SQL Server working. I have a Powershell script that needs to get variables filled from 3 columns (the procedure just returns 1 row)
Here's what I have that isn't working. Somehow I'm not referencing the column value correctly. I've tried various methods, but usually get the error "Cannot index into a null array". I don't want to iterate through the resultset, I just want to directly set some values from the one row returned into variables
$conn = new-object Microsoft.SqlServer.Management.Common.ServerConnection
$conn.ConnectionString = `
"Server=ServerName;Database=ShopDB;Integrated Security=True"
$sqlQuery = new-object System.Collections.Specialized.StringCollection
$sqlQuery.Add("JobSettingsGet")
$Resultset = $conn.ExecuteWithResults($sqlQuery)
# $UserName = $table.Rows(0).Columns(0)
# error - Method invocation failed because [System.Data.DataTable] doesn't
# contain a method named 'Rows'.
# $UserName = $table.Rows[0].Columns[0]
# error - cannot index null array
# $UserName = $table.Row(0).Column(0)
# error - Cannot index into a null array, also Method invocation failed
# because [System.Data.DataTable] doesn't contain a method named 'Row'.
# $UserName = $table.Rows[0].Columns[1].Value
# error - cannot index null array
I'd like to use the column name if possible, too.
Any pointers?
thanks,
Sylvia
You could use System.Data.SqlClient.SqlConnection instead of SqlServer.Management class
$conn = New-Object Data.Sqlclient.Sqlconnection`
("Data Source=DATABASE;Initial Catalog=master;Integrated Security=SSPI;")
$adapter = New-Object Data.Sqlclient.Sqldataadapter("exec sp_who2", $conn)
$set = new-object data.dataset
$adapter.fill($set)
$table = new-object data.datatable
$table = $set.tables[0]
In order to print the whole table, just use format-table
$table | ft -AutoSize
To get a column by name, use the foreach iterator and access note properties by name
$table | % {$_.login}
Ed: Here is the same query by using SqlServer.Management.
[void][Reflection.Assembly]::LoadWithPartialName`
("Microsoft.SqlServer.ConnectionInfo")
$conn = new-object Microsoft.SqlServer.Management.Common.ServerConnection
$conn.ConnectionString = `
"Server=ServerName;Database=ShopDB;Integrated Security=True"
$sqlQuery = new-object System.Collections.Specialized.StringCollection
$sqlQuery.Add("exec sp_who2")
$resultset = $conn.ExecuteWithResults($sqlQuery)
Access the results by picking the zeroeth resultset, zeroeth table, zeroeth row and item by name
$resultset[0].tables.Item(0).Rows[0].Item("Login")

Resources