I am using SQL Server 2016.
I have a stored procedure GET_RECORDS that takes input parameters for filter and outputs a CURSOR parameter
I want to get this cursor in my SSIS package
I had created data flow task, OleDb source and variables for parameter values. Then mapped parameters
Params mapping screen
but when I wanted to save the component - I got an error
error screen
I tried to add clause WITH RESULT SETS with some dummy columns, but my procedure doesn't return any result set
What am I doing wrong?
Any advices will be helpful.
Thank you.
With regards, Yuriy.
The source component is trying to determine what columns and types will be returned. Because you are using dynamic SQL the metadata can change each time you run it.
With result sets allows you to define the data being returned but should only be used if you are guaranteed to have those results every time you execute.
EDIT:
I create a connection and run the command so that it populates a data table. Then I put the column headers into a string array. There are plenty of examples out there.
Then I use the following function to create a destination table. Finally I create a datareader and pass that to the .Net SqlBulkCopy. Hope this helps.
private void CreateTable(string TableName, string[] Fields)
{
if (TableExists(TableName) && Overwrite)
{
SqlCommand = new SqlCommand($"Drop Table [{TableName}]", SqlConnection);
SqlCommand.ExecuteNonQuery();
}
string Sql = $"Create Table [{TableName}] (";
int ColumnNumber = 1;
foreach (string Field in Fields)
{
string FieldValue = Field;
if (! HasHeaders)
{
FieldValue = "Column" + ColumnNumber;
ColumnNumber++;
}
Sql += $"[{FieldValue}] Varchar(8000),";
}
Sql = Sql + "ImportFileID Int, ID Int Identity(1,1) Not Null, Constraint [PK_" + TableName + "] Primary Key Clustered ([ID] Asc))";
SqlCommand = new SqlCommand(Sql, SqlConnection);
SqlCommand.ExecuteNonQuery();
}
Use ado.net source instead of oledb source, define a simple select and get the columns you wish to return. Now you can define expresión in the dataflow properties.
Search ado.net source dynamic sql
:)
try to return the records and use foreach in ETL instead of cursor
https://www.simple-talk.com/sql/ssis/implementing-foreach-looping-logic-in-ssis/
I think you can do it from a simple way, but I don't know what you are you doing, exactly...
Related
I am not sure if this is possible but I have not been able to come across clear documentation for this use case. I am using F# 4 and the FSharp.Data.SqlClient library to connect to SQL Server 2016. I am wanting to call a stored procedure that returns multiple tables and turn those tables into the corresponding records. In this case the first table is made up of items and the second table is made up of customers.
My instinct is that it should look something like this:
let items, customers = cmd.Execute()
My gut is that items would be an IEnumerable<item> and customers would be an IEnumerable<customer> where item and customer are both Record types. What it appears is happening though is that FSharp.Data.SqlClient is only seeing the first returned table from the stored procedure. I am working on a SQL Server 2016 Developer instance. Here is the T-SQL to setup the example:
create table Item (
ItemID int identity(1, 1) primary key,
ItemName nvarchar(50)
)
go
create table Customer (
CustomerID int identity(1, 1) primary key,
CustomerName nvarchar(50)
)
go
insert into Item (ItemName) values ('A');
insert into Item (ItemName) values ('B');
insert into Item (ItemName) values ('C');
insert into Customer (CustomerName) values ('Gary');
insert into Customer (CustomerName) values ('Sergei');
insert into Customer (CustomerName) values ('Elise');
go
create procedure dbo.ExampleProcedure
as
begin
set nocount on;
select
ItemID,
ItemName
from Item
select
CustomerID,
CustomerName
from Customer
end;
And here is the F# script that I am testing with. It shows what I would like to be able to do but I get a compile error on the last line:
#r "../packages/FSharp.Data.SqlClient.1.8.2/lib/net40/FSharp.Data.SqlClient.dll"
#r "../packages/FSharp.Data.2.3.2/lib/net40/FSharp.Data.dll"
#r "System.Xml.Linq.dll"
open FSharp.Data
[<Literal>]
let connStr =
"Data Source=**connection string**;"
type queryExample = SqlProgrammabilityProvider<connStr>
do
use cmd = new queryExample.dbo.ExampleProcedure(connStr)
let items, customers = cmd.Execute()
I am wanting items to correspond to the first returned table and customers to correspond to the second returned table. The intellisense suggests that FSharp.Data.SqlClient is only seeing the first table. When I hover over cmd.Execute() the popup says "This expression was expected to have type 'a*'b but here has type System.Collections.Generic.IEnumerable<SqlProgrammabilityProvider<...>.dbo.ExampleProcedure.Record>". If I do the following I get access to the Items query in the stored procedure:
// Learn more about F# at http://fsharp.org. See the 'F# Tutorial' project
// for more guidance on F# programming.
#r "../packages/FSharp.Data.SqlClient.1.8.2/lib/net40/FSharp.Data.SqlClient.dll"
#r "../packages/FSharp.Data.2.3.2/lib/net40/FSharp.Data.dll"
#r "System.Xml.Linq.dll"
open FSharp.Data
[<Literal>]
let connStr =
"Data Source=**connection string**;"
type queryExample = SqlProgrammabilityProvider<connStr>
do
use cmd = new queryExample.dbo.ExampleProcedure(connStr)
for item in cmd.Execute() do
printfn "%A" item.ItemID
Is this even possible? Is my approach wrong? I could not find clear documentation on this use case but I thought it would be common enough it would be covered.
Update
Just to clarify what I am trying to achieve I am showing how I solve this in C#. In C# I create a DataSet object and populate it with the results of the Stored Procedure. From there I pick out the individual tables to work with. After extracting the tables I then use LINQ to transform the rows into the corresponding objects. It often looks something like the following:
using System.Data;
using System.Data.SqlClient;
var connStr = "**connection string**"
var sqlConnection = new SqlConnection(connStr );
var sqlCommand = new SqlCommand("ExampleProcedure", sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
var dataSet = new DataSet();
var adapter = new SqlDataAdapter(sqlCommand);
adapter.Fill(dataSet);
var itemsTable = dataSet.Tables[0];
// Turn the itemsTable into a List<Item> using LINQ here
var customersTable = dataSet.Tables[1];
// Turn the customersTable into List<Customer> using LINQ here
I find this to be overly verbose for such a simple thing as extracting the individual tables but perhaps I am too sensitive to code clutter. I know that F# must have a more elegant and terse way to express this.
I don't know F#, however this is a data access problem.
When a stored procedure returns multiple resultsets, you need to access they in sequence, one by one.
cmd.ExecuteReader() returns an instance of a datareader pointing to the first resultset. You need to process this resultset, may be filling a list with instances of a custom class, than you call the method "NextResult" and you will have access to the next resultset and so on.
A reference for the method "NextResult": https://msdn.microsoft.com/pt-br/library/system.data.sqlclient.sqldatareader.nextresult(v=vs.110).aspx
I have a customer setup that uses stored procedures to return data from its SQL Server database.
Those stored procedures are all built the same way - they take a bunch of input parameters, and they return:
the first is just a single row, single column with the result code (type INT) - 0 for success, some other values otherwise; if the value is 0, then there's a second result set that contains the actual data
the second result set can be anything - any number of columns and rows
I'm trying to create a "generic" way to interface with this system, and my attempt is this:
create a class that takes stored procedure name and input parameters
a return type that contains both an ErrorCode INT property as well as a DataTable results property
However, I'm having trouble getting this to work with ADO.NET and SQL Server 2008 R2.
My code boils down to this:
public MyResultType CallService(string procedureName, MyParameters parameters)
{
MyResultType result = new MyResultType { ErrorCode = 0 };
using (SqlConnection conn = new SqlConnection(_connectionString))
using (SqlCommand cmd = new SqlCommand(procedureName, conn))
{
cmd.CommandType = CommandType.StoredProcedure;
SetupParameters(cmd, parameters);
// open connection, execute the stored procedure
conn.Open();
using (SqlDataReader rdr = cmd.ExecuteReader())
{
// get the first result set - the status code, one row, one column of type INT
while (rdr.Read())
{
result.ErrorCode = rdr.GetInt32(0);
}
// if we got a "0" (success) -> go to the next result set and load it into the table
if(result.ErrorCode == 0 && rdr.NextResult())
{
result.ResultTable = new DataTable();
result.ResultTable.Load(rdr);
int colCount = result.ResultTable.Columns.Count;
int rowCount = result.ResultTable.Rows.Count;
}
rdr.Close();
}
conn.Close();
}
return result;
}
My issue is: the call to the stored procedure works just fine, the error code = 0 is picked up just fine, the data table is created and the number of columns is the expected value - but there are NO ROWS loaded...
I've been trying everything I can think of to get those rows into the DataTable - no luck. And of course - if I execute this very same stored procedure in SQL Server Management Studio, everything works just fine, I get my ErrorCode=0 and my result set of 18 columns and 5 rows - no problem....
What am I missing? Can anyone spot the problem in my code? Why aren't the rows of the second result set loaded (but the columns are being detected, it seems)?
The code as published works fine - there was a difference between how I call it from C# and in SQL Server Management Studio : NULL handling.
I had some input parameters set to int and thus provided a 0 value to the stored procedure, while it really expected to get a NULL if the value isn't defined......
Stupid rookie mistake..... SORRY GUYS! And thanks for your inputs!
The DataTable.Load method implicitly calls the NextResult method, so combining that with your explicit call, it is advancing to a resultset that isn't there. You should remove your own call to NextResult if you want to use DataTable.Load or loop through and fill a datatable yourself.
I am trying to use Dapper support my data access for my server app.
My server app has another application that drops records into my database at a rate of 400 per minute.
My app pulls them out in batches, processes them, and then deletes them from the database.
Since data continues to flow into the database while I am processing, I don't have a good way to say delete from myTable where allProcessed = true.
However, I do know the PK value of the rows to delete. So I want to do a delete from myTable where Id in #listToDelete
Problem is that if my server goes down for even 6 mintues, then I have over 2100 rows to delete.
Since Dapper takes my #listToDelete and turns each one into a parameter, my call to delete fails. (Causing my data purging to get even further behind.)
What is the best way to deal with this in Dapper?
NOTES:
I have looked at Tabled Valued Parameters but from what I can see, they are not very performant. This piece of my architecture is the bottle neck of my system and I need to be very very fast.
One option is to create a temp table on the server and then use the bulk load facility to upload all the IDs into that table at once. Then use a join, EXISTS or IN clause to delete only the records that you uploaded into your temp table.
Bulk loads are a well-optimized path in SQL Server and it should be very fast.
For example:
Execute the statement CREATE TABLE #RowsToDelete(ID INT PRIMARY KEY)
Use a bulk load to insert keys into #RowsToDelete
Execute DELETE FROM myTable where Id IN (SELECT ID FROM #RowsToDelete)
Execute DROP TABLE #RowsToDelte (the table will also be automatically dropped if you close the session)
(Assuming Dapper) code example:
conn.Open();
var columnName = "ID";
conn.Execute(string.Format("CREATE TABLE #{0}s({0} INT PRIMARY KEY)", columnName));
using (var bulkCopy = new SqlBulkCopy(conn))
{
bulkCopy.BatchSize = ids.Count;
bulkCopy.DestinationTableName = string.Format("#{0}s", columnName);
var table = new DataTable();
table.Columns.Add(columnName, typeof (int));
bulkCopy.ColumnMappings.Add(columnName, columnName);
foreach (var id in ids)
{
table.Rows.Add(id);
}
bulkCopy.WriteToServer(table);
}
//or do other things with your table instead of deleting here
conn.Execute(string.Format(#"DELETE FROM myTable where Id IN
(SELECT {0} FROM #{0}s", columnName));
conn.Execute(string.Format("DROP TABLE #{0}s", columnName));
To get this code working, I went dark side.
Since Dapper makes my list into parameters. And SQL Server can't handle a lot of parameters. (I have never needed even double digit parameters before). I had to go with Dynamic SQL.
So here was my solution:
string listOfIdsJoined = "("+String.Join(",", listOfIds.ToArray())+")";
connection.Execute("delete from myTable where Id in " + listOfIdsJoined);
Before everyone grabs the their torches and pitchforks, let me explain.
This code runs on a server whose only input is a data feed from a Mainframe system.
The list I am dynamically creating is a list of longs/bigints.
The longs/bigints are from an Identity column.
I know constructing dynamic SQL is bad juju, but in this case, I just can't see how it leads to a security risk.
Dapper request the List of object having parameter as a property so in above case a list of object having Id as property will work.
connection.Execute("delete from myTable where Id in (#Id)", listOfIds.AsEnumerable().Select(i=> new { Id = i }).ToList());
This will work.
I have a SQLite database in one system, I need to extract the data stored in SQLite to Oracle database. How do I do this?
Oracle provides product called the Oracle Database Mobile Server (previously called Oracle Database Lite) which allows you to synchronize between a SQLite and an Oracle database. It provides scalable bi-directional sync, schema mapping, security, etc. The Mobile Server supports both synchronous and asynchronous data sync. If this is more than a one-time export and you need to keep your SQLite and Oracle Databases in sync, this is a great tool!
Disclaimer: I'm one of the Product Managers for Oracle Database Mobile Server, so I'm a bit biased. However, the Mobile Server really is a great tool to use for keeping your SQLite (or Berkeley DB) and Oracle Databases in sync.
You'll have to convert the SQLite to a text file (not certain of the format) and then use Oracle to load the database from text (source is http://www.orafaq.com/wiki/SQLite). You can use the .dump command from the SQLite interactive shell to dump to a text file (see the docs for syntax).
SQL Loader is a utility that will read a delimited text file and import it into an oracle database. You will need to map out how each column from your flat file out of sqlite matches to the corresponding one in the Oracle database. Here is a good FAQ that should help you get started.
If you are a developer, you could develop an application to perform the sync. You would do
SELECT name FROM sqlite_master WHERE type='table'
to get the table names, then you could re-create them in Oracle (you can do DROP TABLE tablename in Oracle first, to avoid a conflict, assuming SQLite will be authoritative) with CREATE TABLE commands. Getting the columns for each one takes
SELECT sql FROM sqlite_master WHERE type='table' and name='MyTable'
And then you have to parse the result:
string columnNames = sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').replace(/ [^,]+/g, '').split(',');
string[] columnArray = columnNames.Split(',');
foreach (string s in columnArray)
{
// Add column to table using:
// ALTER TABLE MyTable ADD COLUMN s NVARCHAR(250)
}
A StringBuilder can be used to collect the table name with its columns to create your INSERT command. To add the values, it would just be a matter of doing SELECT * FROM MyTable for each of the tables during your loop through the table names you got back from the initial query. You would iterate the columns of the rows of the datatable you were returned and add the values to the StringBuilder:
INSERT INTO MyTable ( + columnA, columnB, etc. + ) VALUES ( datarow[0], datarow[1], etc. + ).
Not exactly like that, though - you fill in the data by appending the column name and its data as you run through the loops. You can get the column names by appending s in that foreach loop, above. Each column value is then set using a foreach loop that gives you each object obj in drData.ItemArray. If all you have are string fields, it's easy, you just add obj.ToString() to your StringBuilder for each column value in your query like I have below. Then you run the query after collecting all of the column values for each row. You use a new StringBuilder for each row - it needs to get reset to INSERT INTO MyTable ( + columnA, columnB, etc. + ) VALUES ( prior to each new row, so the new column values can be appended.
If you have mixed datatypes (i.e. DATE, BLOB, etc.), you'll need to determine the column types along the way, store it in a list or array, then use a counter to determine the index of that list/array slot and get the type, so you know how to translate your object into something Oracle can use - whether that means simply adding to_date() to the result, with formatting, for a date (since SQLite stores these as date strings with the format yyyy-MM-dd HH:mm:ss), or adding it to an OracleParameter for a BLOB and sending that along to a RunOracleCommand function. (I did not go into this, below.)
Putting all of this together yields this:
string[] columnArray = null;
DataTable dtTableNames = GetSQLiteTable("SELECT name FROM sqlite_master WHERE type='table'");
if (dtTableNames != null && dtTableNames.Rows != null)
{
if (dtTableNames.Rows.Count > 0)
{
// We have tables
foreach (DataRow dr in dtTableNames.Rows)
{
// Do everything about this table here
StringBuilder sb = new StringBuilder();
sb.Append("INSERT INTO " + tableName + " ("); // we will collect column names here
string tableName = dr["NAME"] != null ? dr["NAME"].ToString() : String.Empty;
if (!String.IsNullOrEmpty(tableName))
{
RunOracleCommand("DROP TABLE " + tableName);
RunOracleCommand("CREATE TABLE " + tableName);
}
DataTable dtColumnNames = GetSQLiteTable("SELECT sql FROM sqlite_master WHERE type='table' AND name='"+tableName+"'");
if (dtColumnNames != null && dtColumnNames.Rows != null)
{
if (dtColumnNames.Rows.Count > 0)
{
// We have columns
foreach (DataRow drCol in dtTableNames.Rows)
{
string sql = drCol["SQL"] != null ? drCol["SQL"].ToString() : String.Empty;
if (!String.IsNullOrEmpty(sql))
{
string columnNames = sql.replace(/^[^\(]+\(([^\)]+)\)/g, '$1').replace(/ [^,]+/g, '').split(',');
columnArray = columnNames.Split(',');
foreach (string s in columnArray)
{
// Add column to table using:
RunOracleCommand("ALTER TABLE " + tableName + " ADD COLUMN " + s + " NVARCHAR(250)"); // can hard-code like this or use logic to determine the datatype/column width
sb.Append("'" + s + "',");
}
sb.TrimEnd(",");
sb.Append(") VALUES (");
}
}
}
}
// Get SQLite Table data for insertion to Oracle
DataTable dtTableData = GetSQLiteTable("SELECT * FROM " + tableName);
if (dtTableData != null && dtTableData.Rows != null)
{
if (dtTableData.Rows.Count > 0)
{
// We have data
foreach (DataRow drData in dtTableData.Rows)
{
StringBuilder sbRow = sb; // resets to baseline for each row
foreach (object obj in drData.ItemArray)
{
// This is simplistic and assumes you have string data for an NVARCHAR field
sbRow.Append("'" + obj.ToString() + "',");
}
sbRow.TrimEnd(",");
sbRow.Append(")");
RunOracleCommand(sbRow.ToString());
}
}
}
}
}
}
All of this assumes you have a RunOracleCommand() void function that can take a SQL command and run it against an Oracle DB, and a GetSQLiteTable() function that can return a DataTable from your SQLite DB by passing it a SQL command.
Note that this code is untested, as I wrote it directly in this post, but it is based heavily on code I wrote to sync Oracle into SQLite, which has been tested and works.
i have a console app in c# that extracts 20 fields from an oracle DB witht he code below and i wanted an efficient way to insert them into SQL 2005.
i dotn want to insert each one of the 20,000 within the while loop, obviously. i was thinking to change the code to use a data set to cache all the records and then do a bulk insert...
thoughts?
pseudo code would be nice since i am new to oracle.
this is my code where i was testing getting a connection to oracle and seeing if i can view the data... now i can view it i want to get it out and into sql2005... what do i do from here?
static void getData()
{
string connectionString = GetConnectionString();
using (OracleConnection connection = new OracleConnection())
{
connection.ConnectionString = connectionString;
connection.Open();
OracleCommand command = connection.CreateCommand();
string sql = "SELECT * FROM BUG";
command.CommandText = sql;
OracleDataReader reader = command.ExecuteReader();
while (reader.Read())
{
//string myField = (string)reader["Project"];
string myField = reader[0].ToString();
Console.WriteLine(myField);
}
}
}
You can create a CSV file and then use BULK INSERT to insert the file into SQL Server. Have a look here for an example.
The "bulk" insert with the cached Dataset will work exactly like the while loop you are not wanting to write! The problem is that you'll lose control of the process if you try to use the "bulk" insert of the Dataset class. It is extraneous work in the end.
Maybe the best solution is to use a DataWriter so that you have complete control and no Dataset overhead.
You can actually do 100-1000 inserts per sql batch. Just generate multiple inserts, then submit. Pregenerate the next SELECT batch WHILE THE FIRST EXECUTES.