What is the best way to update a table from a List inside an SSIS Script Task?
We have a shared class library. I have used the dll in the script task to do most of the necessary work. The dll method then returns a List which contains data related to the processes that it ran. It is my job to write this list to a table.
I'm thinking I will loop through each item in the List and and run the update SQL statement.
For brevity, I did not paste the SQL statement, but it is actually an Upsert using MERGE.
Actually, I wish there were a way to output the List to the input of an Execute SQL Task, but I'm unsure if that is possible.
Here is what I have so far. As you can see it is unfinished.
private void UpdateEtlData(List<ProcessStatitics> statistics)
{
var connection = GetOhioConnectionString();
// will loop thru each item in statistics and run the
// following sequence. This code is unfinished, but
// I will use properties inside each statistic to form the
// query
foreach(statistic in statistics)
{
SqlCommand command = new SqlCommand();
command.Connection = connection;
command.CommandText = ""
}
}
The easiest thing for you to do here is to create your SQLCommand outside of the loop, and set it up with parameters to write your data. This blog post covers it well: http://csharp-station.com/Tutorial/AdoDotNet/Lesson06
Steps are:
// 1. declare command object with parameter
SqlCommand cmd = new SqlCommand(
"Insert into CityList (CityName) values (#City)", conn);
// 2. define parameters used in command object
SqlParameter param = new SqlParameter();
param.ParameterName = "#City";
// 3. add new parameter to command object
cmd.Parameters.Add(param);
// When you want to assign the value
param.Value = inputCity;
Then in your loop you can assign your value from the list to param.Value, and call command.ExecuteNonQuery
Related
I am using dependency injection with Entity Framework Core and the context is created through a scope factory:
using (var scope = this._scopeFactory.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<thecontext>();
//run code here
}
A transaction is started after all the objects are created:
using (var trans = context.Database.BeginTransaction())
{
try
{
This works well but during the process I need to insert records by running a stored procedure that is outside the transaction. In other words even if the transaction is aborted I still need the results of this stored procedure insert to persist. In addition any inserts from the 'isolated' stored procedure must be available to the process running under the transaction as well as outside the current scope.
The normal method of executing the stored procedure is by getting the connection and attaching the transaction to a new command. But I need to either use a new connection that is outside the current scope so it is not bound by the transaction or perhaps there is another way?
string sql = $"theProc";
var cnn = _context.Database.GetDbConnection();
if (cnn.State == ConnectionState.Closed)
cnn.Open();
DbCommand cmd = cnn.CreateCommand();
cmd.CommandText = sql;
cmd.CommandType = CommandType.StoredProcedure;
if (_context.Database.CurrentTransaction != null)
cmd.Transaction = _context.Database.CurrentTransaction.GetDbTransaction();
I ended up doing something like:
var cnnstr = _context.Database.GetConnectionString();
var isolatedCnn = new SqlConnection(cnnstr);
isolatedCnn.Open();
using (isolatedCnn)
{
This works but is ugly for a number of reasons so I'd like to find a better solution.
I have procedure in Snowflake and would like to call it from my Timer Triggered Azure Function App.
That procedure expects a parameter which is of type string. Following is my code snippet to connect to Snowflake and calling that procedure with parameter.
using (IDbConnection conn = new SnowflakeDbConnection())
{
//Connect to Snowflake
conn.ConnectionString = Environment.GetEnvironmentVariable("SnowflakeConnection");
conn.Open();
using (IDbCommand cmd = conn.CreateCommand())
{
if (conn.State == ConnectionState.Open)
{
cmd.CommandText = "SP_Snowflake_Procedure";
//cmd.CommandType = CommandType.StoredProcedure;
var date = cmd.CreateParameter();
date.ParameterName = "RUNDATE";
date.DbType = DbType.String;
date.Value = "2018-01-01";
cmd.Parameters.Add(date);
using (IDataReader dr = cmd.ExecuteReader())
{
/****************
Logic to work on data
received from SP
*****************/
}
}
}
}
When control comes to cmd.ExecuteReader(), it's failing with error:
Snowflake.Data: SQL compilation error: syntax error line 1 at position 0 unexpected 'SP_Snowflake_Procedure'.
I don't understand this Snowflake, how to call a procedure. I had a thought of, it is way similar to MS SQL. But I am wrong. I couldn't even find proper related documents.
I could use same code without procedure call but simple SELECT statement and worked fine.
Suggest me any changes here.
I can't tell from the code if you're using the ODBC driver for Snowflake or the .NET driver for Snowflake. The ODBC driver supports more features than the .NET driver, but I think executing SPs should be supported in both.
You'll need to make the call using a SQL statement that executes a query (as opposed to methods that execute non-query). It will return a table with a single row with the return from the SP. It will contain a single column with the name of the SP and the scalar value of the SP (basically what would be returned to the SQL worksheet if run in the web UI).
Here's a sample SP to test in case you need a simple one:
create or replace procedure EchoString(stringValue string)
returns VARCHAR
language JavaScript
as
$$
// Note that variables passed to Snowflake stored procedures
// muat be all CAPITAL letters when used in the body of the
// procedure code.
return STRINGVALUE
$$;
--Run the stored procedure to echo the value.
call EchoString('Echo this string.');
Here's how to call the SP from a C# project using an ODBC connection:
OdbcConnection DbConnection = new OdbcConnection("DSN=Snowflake;pwd=******");
OdbcCommand DbCommandSetup = DbConnection.CreateCommand();
DbConnection.Open();
// These two lines are only required if you get a message about no running warehouse.
// It will depend on how your calling user is set up in Snowflake.
DbCommandSetup.CommandText = "use warehouse TEST;";
DbCommandSetup.ExecuteNonQuery();
OdbcCommand DbCommand = DbConnection.CreateCommand();
DbCommand.CommandText = "call TEST.PUBLIC.ECHOSTRING('Echo this string.')";
OdbcDataReader DbReader = DbCommand.ExecuteReader();
// Note: If you define a Snowflake SP, DB, or schema in mixed case without double quoting
// the name, Snowflake will uppercase it in the catalog. You can call it from here without
// converting to upper case as long as it's not double quoted (escaped \") in the string.
I have for-each loop container in my SSIS master package
There is 'Execute Package Task'
under this 'Execute SQL Server Agent Job Task' is there.(It has 35 steps)
Problem:
SSIS master package is starting next iteration without completion of all the steps in 'Execute SQL Server Agent Job Task'.
Please suggest an approach to start the next iteration after completion of all the steps only.
Thanks in advance :)
I know I keep throwing C# solutions at SSIS problems but this is exactly what I do to prevent a job/process to run while a job is running.
I have a function that does this check and returns a boolean true/false. Use that result to determine whether to start the job or not.
public static bool isDatabaseUpdating()
{
List<string> jobs = new List<string>();
jobs.Add("[Insert name of Job here]");
jobs.Add("[Insert another job and so on here]");
string sql = "exec msdb.dbo.sp_help_job #execution_status = 1";
using (OleDbConnection conn = new OleDbConnection(cstr))
{
using (OleDbCommand cmd = new OleDbCommand(sql, conn))
{
cmd.CommandType = CommandType.Text;
cmd.Connection.Open();
OleDbDataReader rdr = cmd.ExecuteReader();
while (rdr.Read())
{
if (jobs.Contains(rdr["name"].ToString())) return true;
}
}
}
return false;
}
To use it set a SSIS Variable like this:
Dts.Variables["#variableName"].Value = isDatabaseUpdating();
And then in control flow set expression on path appropriately.
The real key to understanding this function is the SQL.
exec msdb.dbo.sp_help_job #execution_status = 1
That returns a dataset of jobs that are currently running.
Enhancement to your application
This is what your control flow will look like:
I have some general purpose code in my app for calling non-query commands like inserts:
DBComm.CommandText = SQL
DBComm.Connection = Cnn
Ans = DBComm.ExecuteNonQuery
I'm using this to call an SP that takes two strings and returns an int:
Dim SQL As String = "EXEC Import_Validation " & Code & "," & User
Return DbS.Execute(SQL)
This works fine, with the exception that the return value is not the value of the SP (is it a job id of some sort?). So I modified it slightly:
Param = New SqlParameter("#RETURN_VALUE", SqlDbType.Int)
Param.Direction = ParameterDirection.ReturnValue
DBComm.Parameters.Add(Param)
DBComm.CommandText = SQL
DBComm.Connection = Cnn
DBComm.ExecuteNonQuery
This runs, but always returns 0 for RETURN_VALUE. I suspect this is because I do not have the following line:
DBComm.CommandType = CommandType.StoredProcedure
Is that suspicion correct?
However, if I add this line, the SQL no longer works, complaining Could not find stored procedure "Import_Validation '1234', 'maury'. I assume that is because it thinks the string I passed in is just the name, and is being confused by the parameters. I could add the parameters as input parameters on DBComm, but then the method would be specific to a particular SP.
So is there a way I can call an SP using SQL I construct and still get a parameter back out?
If you are just returning an integer, I've used this:
Param.value=cmd.ExecuteScalar()
I Cannot guess nor explain why you had trouble executing the store procedure, since you didn't share the full code (full part of the code that have issue). However, generally speaking, when you use ExecuteNonQuery() you're supposed to use a valid T-SQL. By default, the commandType is Text. So, if you need to execute a store procedure, you'll need to change the command type to StoredProcedure before executing the query. So, I would say your guess is correct in this line :
DBComm.CommandType = CommandType.StoredProcedure
Also, in your code, you declared a #RETURN_VALUE parameter, but I couldn't see anything calling its value !? So, maybe this is the issue ?
anyhow, check this :
Using connection As New SqlConnection(ConnectionString)
command As New SqlCommand("dbo.Import_Validation", connection)
command.CommandType = CommandType.StoredProcedure
command.Parameters.Add("#Code", SqlDbType.VarChar, 250)
command.Parameters.Add("#User", SqlDbType.VarChar, 250)
command.Parameters.Add("#RETURN_VALUE", SqlDbType.Int).Direction = ParameterDirection.ReturnValue
command.Parameters("#Code").Value = Code
command.Parameters("#User").Value = User
connection.Open()
command.ExecuteNonQuery()
'IF returned value is more than one row, then use reader instead
Dim return_value As Integer = Convert.ToInt32(command.Parameters("#RETURN_VALUE").Value)
connection.Close()
End Using
Over the last few days, I was asked to move a company program over from an Access back-end, to SQL Server.
There are 2 copies of the program, the live data version, on the server, and the local version on my PCs C: Drive, to ensure if I make a mistake, it doesn't affect the live data.
So, I managed to migrate the Access database, tables and data over to SQL Server 2008, and the local version of the program now works.
The easiest way, or so I'm informed, to now do the same to the live version of the program, is to write an imports program, which wipes all of the data from each table in the SQL Server database, and then copies over the data from the live Access database. However, I've never done this before, so I'm not really even sure where to begin.
Could anybody point me in the right direction on how to begin or do this, so that I only have to change the connection path in the program, rather than go through the whole process again?
PS, I work in vb.net, so that's the language I would need any responses in!
Thanks.
Usually one uses the SQL Server Import and Export Wizard for this.
It's a separate tool that is installed with SQL Server Management Studio (SSMS).
ANSWER
Step 1;
I added a new path to the ini file for the database to read. This connected to the live database. Once this connection is open in the project, proceed to step 2.
Step 2;
Create a new class, where the imports and exports will happen.
Step 3;
Put a button, or some sort of control in the program to initiate the import/export. For example, I had a button which, when clicked, asked the user to confirm that they wanted to import a new database and overwrite the existing one. If yes, call the function which does this, in the newly made imports class.
Step 4;
Now that you know how to get this set up, the code would be something like
Public Function importdatabase(/connections go in here/)
Declare transaction
Create sql variable
Try
Begin the transaction here
sql to delete the data from one table
sql to select all data from database that is being imported
For loop to iterate over each record in the database table
Declare a variable for each field in the database
variable1 = ("fieldname1")
variable2 = ("fieldname2")
sql statement to insert the new values
call to the function which runs the sql query
Next
commit transaction
Catch ex As Exception
Throw
End Try
Step 5; Repeat the delete/insert process for each database table
Below this, I have other functions.
One function created a new datatable, this is referenced as
For each dr as datarow in /functionname(parameters).Rows
Next one is to execute the sql statement (not required, any command to execute it will do)
Next one is used for parameterising my SQL query
The rest are to replace null values in the database with empty strings, set dates, etc
You can use the following class to import table(s) in access to sql server.
You need:
- The connection string of the source (including access file name) and the target one.
- The Source Table ,target tatble (if null it is the same as the source table)
Class ImportHelper
'modify connectionstring as needed
Public Property SourceConnectionString() As String
Get
Return m_SourceConnectionString
End Get
Set
m_SourceConnectionString = Value
End Set
End Property
Private m_SourceConnectionString As String
Public Property DestinationConnectionString() As String
Get
Return m_DestinationConnectionString
End Get
Set
m_DestinationConnectionString = Value
End Set
End Property
Private m_DestinationConnectionString As String
Public Sub New(sourceConnectionString__1 As String, destinationConnectionString__2 As String)
SourceConnectionString = sourceConnectionString__1
DestinationConnectionString = destinationConnectionString__2
End Sub
Public Sub Import(sourceTable As String, Optional targetTable As String = Nothing)
Using sourceConnection = New OleDbConnection(SourceConnectionString)
If String.IsNullOrEmpty(targetTable) Then
targetTable = sourceTable
End If
sourceConnection.Open()
' Perform an initial count on the destination table.
Dim commandRowCount = New OleDbCommand(Convert.ToString("SELECT COUNT(*) FROM ") & sourceTable, sourceConnection)
Dim countStart As Long = Convert.ToInt32(commandRowCount.ExecuteScalar())
Console.WriteLine("Source Table [{0}] has {1} rows", sourceTable, countStart)
' Get data from the source table
Dim commandSourceData = New OleDbCommand(Convert.ToString("SELECT * FROM ") & sourceTable, sourceConnection)
Dim reader = commandSourceData.ExecuteReader()
'---------------
Using destinationConnection As New SqlConnection(DestinationConnectionString)
destinationConnection.Open()
Using bulkCopy As New SqlBulkCopy(destinationConnection)
bulkCopy.DestinationTableName = targetTable
Try
' Write from the source to the destination.
bulkCopy.WriteToServer(reader)
Console.WriteLine(Convert.ToString("Sucess Importing ") & sourceTable)
Catch ex As Exception
Console.WriteLine(ex.Message)
Finally
reader.Close()
End Try
'using
End Using
'using
End Using
End Using
'using
End Sub
End Class
How to use:
Private Sub Test()
'modify connectionstring as needed
'Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\mydatabase.mdb;User Id=admin;Password=; //access 97..2000
Dim SourceConnectionString As String = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\temp\database1.accdb;Persist Security Info=False;"
Dim DestinationConnectionString As String = "Data Source=xxxx;Initial Catalog=test;user=xxx;password=xxx;"
New ImportHelper(SourceConnectionString, DestinationConnectionString).Import("table1", "test1")
End Sub