MSSQL lets you do a multi-row UPDATE and INSERT from some source, like another table, or JSON & XML data, etc. Psuedo-example:
INSERT INTO TBL SELECT * FROM SOURCE
Is it possible to use a PowerShell hash table as the source?
#psuedo code
$carHashTable = [pscustomobject]#{
color = 'red'
make = 'geo'
model = 'metro'
year = 1996
insured = 0
}
#here-string
$sql = #"
insert into
SomeDumbTable
select
*
from $carHashTable
"#
Invoke-Sqlcmd -Query $sql
All the examples I've seen online use a foreach loop to do the actual insert or update. Bossmang says loops (in SQL proper) are bad.
The only other option I can think of is using PowerShell to create a temp table in SQL, then use the temp table as the source to do the the multi-insert.
Attempting to answer my own question. If I make an incorrect statement, please correct me.
1) YES. You can convert a hash table using this module created by James Brundage into an object that the SqlServer module can deal with.
#import module that converts PSObjects into a DataTable that SQL can handle (external author)
import-module -name C:\PowerShell\ConvertTo-DataTable.ps1 -Verbose
#import the csv, and convert it to a DataTable
$Jobs = import-csv -Path $Repository\$File -Delimiter '|' | ConvertTo-DataTable
2) For the larger question, can you UPDATE without a foreach loop?, I'm going to say NO, at least not with a SQL #temp table, which was my original intention. Here's what I've observed:
Write-SqlTableData does not appear to support #temp tables
Write-SqlTableData DOES allow me to use the converted PS $object
to write data; however, it only appends data to tables; I don't know
of a way to UPDATE on some key
The only way I was able to achieve this was by gleaning a lot on
StackOverflow, and bringing in the SQL .NET client libraries
System.Data.SqlClient.
Those libraries allowed me to open and maintain a session with SQL so that I could create a #temp table, bulk copy content into it, then finally UPDATEing the real target table.
The SqlServer Modules for PowerShell seems to immediately open and close the session, so any #temp tables created there are only avilable to that session
The session created by the .NET client libraries can be opened and maintained to run multiple queries, but it's separate from sessions created by the SqlServer modules -- PS can't gain access to #temp tables created in .Net sessions, and vice versa.
Related
I am updating an on-premises SQL Server database table with data from a csv file using a Copy Data activity. There is an int identity Id column on the sink table that gets generated when I do the Upsert. I would like to retrieve the Id value generated in that table to use later in the pipeline.
Is there a way to do this?
I can't use a data flow as I am using a self-hosted Integration Runtime.
Hi #Nick.McDermaid, I am loading about 7,000 rows from the file to the database. I want to store the identities in the database the file comes from.
Edit:
I have 2 databases (source/target). I want to upsert (using MERGE SQL below, with the OUTPUT clause) into the target db from the source db and then return the Ids (via the OUTPUT resultset) to the source db. The problem I have is that the upsert (MERGE) SQL gets it's SELECT statement from the same target db that the target table is in (when using a Copy Data activity), but I need to get the SELECT from the source db. Is there a way to do this, maybe using the Script activity?
Edit 2: To clarify, the 2 databases are on different servers.
Edit 3 (MERGE Update):
MERGE Product AS target
USING (SELECT [epProductDescription]
,[epProductPrimaryReference]
FROM [epProduct]
WHERE [epEndpointId] = '438E5150-8B7C-493C-9E79-AF4E990DEA04') AS source
ON target.[Sku] = source.[epProductPrimaryReference]
WHEN MATCHED THEN
UPDATE SET [Name] = source.[epProductDescription]
,[Sku] = source.[epProductPrimaryReference]
WHEN NOT MATCHED THEN
INSERT ([Name]
,[Sku]
VALUES (source.[epProductDescription]
,source.[epProductPrimaryReference]
OUTPUT $action, inserted.*, updated.*;
Edit 3 (sample data):
source sample:
target output
Is there a way to do this, maybe using the Script activity?
Yes, you can execute this script using Script activity in ADF
As your tables are on different SQL servers first you have to create Linked server with source database on target Database.
go to >> Server Objects >> Linked Server >> New Linked server and create linked server with source database on target Database as below.
While creating linked server make sure same user must exist on both databases.
then I wrote Merge Query using this linked sever source.
My Sample Query:
MERGE INTO PersonsTarget as trg
USING (SELECT [LastName],[FirstName],[State]
FROM [OP3].[sample1].[dbo].[Personssource]) AS src
ON trg.[State] = src.[State]
WHEN MATCHED THEN
UPDATE SET [LastName] = src.[LastName]
,[FirstName] = src.[FirstName]
WHEN NOT MATCHED THEN
INSERT ([LastName],[FirstName],[State])
VALUES (src.[FirstName],src.[LastName],src.[State])
OUTPUT $action, inserted.*;
Then In Script activity I provided the script
Note: In linked service for on premises target table use same user which you used in linked service
Executed successfully and returning Ids:
Objective: automate creating a table along with all its columns and data types given a SSIS source
My guess is:
1) Pointing Sources to a Destination to a SQL command
2) Using Select * into ... Problem is I don't know what the from equivalent of a source is
Alternative) Store results in Recordset and pass on to Execute SQL task. Problem then is how to access that result from execute sql task
I think you should use a Recordset Destination to store data into an System.Object Variable, Then use a Script Task (starts after that Data Flow Task is executed) in which you will select the System.Object Variable as ReadOnly Variable. and you will write your own code to insert the Recordset to SQL using System.Data.SqlClient.SQLCommand Object
You can refer to one of these links
Issues with SSIS Script Task and SQL Bulk Insert - C#
Insert DataTable into SQL Table in C#
If you need just the structure of table use this trick
select top 0 * into NewTable from YourTable
ETL Script to dynamically map multiple execute sql resultset to multiple tables (table name based on sql file provided)
I have a source folder with sql files ( I can put it up as stored procedures as well ) . I know how to loop and execute sql tasks in a foreach container. Now the part where I'm stuck is I need to use the final result set of each sql queries and shove it into a table with the same name as the sql file.
So, Folder -> script1.sql , script2.sql etc -> ETL -> goes to table script1, table script2 etc.
EDIT : Based on the comment made by Joe, I just want to say that I'm aware of using insert within a script but I need to insert it onto a table in a different server.And Linked servers are not the ideal solutions
Any psuedocode or link to tutorials will be extremely helpful . Thanks!
I would add the table creation to the script. It is probably the simplest way to do this. If your script is Select SomeField From Table1, you could change it to Select SomeField Into Table script1 From Table1. Then there is no need to map in SSIS which is not easy to do from my experience.
I experienced a very strange issue and can be repeated.
Basically, I use invoke-sqlcmd to call a script file by using -inputfile, but if the script file has some execution error (like insert into a table where a column should not be null), the script file will be executed twice. I can see the two executions from profiler as well.
Here is the way to reproduce the issue (My environment: Win 8.1 + SQL2014 + PS 5.0)
create two tables in a database
Use TestDB
create table dbo.s (id int identity primary key, b varchar(50));
create table dbo.t (id int primary key, s_id int, b varchar(50));
alter table dbo.t add constraint fk_t foreign key (s_id) references dbo.s(id)
Now create a sql file (let's call it, c:\temp\t.sql) with the following two lines
insert into dbo.s ( b) select 'hello world'
insert into dbo.t (s_id, b) -- purposely missing id to cause an error
select 1, 'good morning'
Run the following PS cmdlet
invoke-sqlcmd -Server "<my_local_server>" -database TestDB -inputfile "c:\temp\t.sql"
Your PS will return an error, now if you open an SSMS query window and do the following
select * from TestDB.dbo.s
You will see two records there instead of one.
On the other hand, if I run sqlcmd.exe, there is NO such issue, i.e. just one record in dbo.s.
Is there some configuration in SQLPS I missed?
I see you asked this same question on the MSDN Database Engine forum: https://social.msdn.microsoft.com/Forums/en-US/d4167226-2da7-49ec-a5c2-60e964785c2c/powershell-invokesqlcmd-calls-stored-procedure-second-time-after-query-timeout-is-expired. Below is the SMO workaround from that thread.
$SqlServerName = "YourServer";
$DatabaseName = "YourDatabase";
$ScriptFileName = "C:\Scripts\YourSqlScriptFile.sql";
Add-Type -Path "C:\Program Files\Microsoft SQL Server\110\SDK\Assemblies\Microsoft.SqlServer.Smo.dll";
$sr = New-Object System.IO.StreamReader($ScriptFileName);
$script = $sr.ReadToEnd();
$sr.Close();
$Server = New-Object Microsoft.SqlServer.Management.Smo.Server($SqlServerName);
$db = $Server.Databases[$DatabaseName];
$db.ExecuteNonQuery($script);
Updating this tread with the fix from Microsoft:
Cumulative Update 2 for SQL Server 2016 SP1
Cumulative Update 4 for SQL Server 2014 Service Pack 2
FIX: "Invoke-sqlcmd" cmdlet executes a query statement multiple times if an error occurs in SQL Server 2014 or 2016
Even i had a similar issue.
Fixed it by adding -QueryTimeout parameter to Invoke-Sqlcmd cmdlet.
Basically it seems that the query somehow times out and due to a bug in Invoke-Sqlcmd cmdlet it tries to insert it again. Strange but try this. Keep the query timeout big enough for the query to execute.
I know this is a old thread, but maybe it can help someone.
I found out if you move your line "select 1, 'good morning'" in your example before the insert statement, which has the exception, it works like intended.
I had a similar Issue with try catch, that the first return value will be ignored when it's an exception, so I had to make sure that the first line was Select 1. Strange bug.
I know it's possible to do a bulk insert from a file like this:
strSQL = "BULK INSERT Northwind.dbo.[Order Details]
FROM 'e:\My Documents\TextFiles\OrderDetails.txt' " & _
"WITH ( FIELDTERMINATOR = ',', ROWTERMINATOR = '\n' )"
But I can't seem to find a way to insert an object that's in memory instead. Is this possible?
The file must be visible from the SQL Server itself and its account must have access rights (if you are using SQL Server):
From http://technet.microsoft.com/en-us/library/ms188365.aspx
BULK INSERT can import data from a disk (including network, floppy disk, hard disk, and so on). 'data_file' must specify a valid path from the server on which SQL Server is running. If data_file is a remote file, specify the Universal Naming Convention (UNC) name. A UNC name has the form \Systemname\ShareName\Path\FileName. For example, \SystemX\DiskZ\Sales\update.txt.
You can use a table variable to populate a temporary table, then use a MERGE statement to process that temp table into the target database.
T-SQL Merge statement docs here http://msdn.microsoft.com/en-us/library/bb510625.aspx
The solution ended up being to build a COM object in C# that does the bulk insert and then leveraging that COM object in the VB6 project.