How to execute stored procedure on ODBC Snowflake Destinastion? - sql-server

I'am building new package for move data from aws sql server instance to snowflake odbc destination. If i found rows which was updated i must change them on snowflake as well. In common's i found only 'OLE DB Command' for execute procedure for update diffrent rows.
The problem is i need something like "ODBC Command" for execute procedure to update diffrent rows between SQL Server&Snowflake.

OK, I do it.
So if u need UPDATE rows on ODBC destination in SSIS u have only one way to do that u need to use Script Component. Before I thought it will be something like ODBC Command and we will need to write stored procedure to change rows in the destination. I link that for ppl who care in the future.
The OLE DB Command transformation runs an SQL statement for each row in a data flow. For example, you can run an SQL statement that inserts, updates, or deletes rows in a database table.
Microsoft OLE DB Command description
I wrote a simple code in c# to Update Rows and it works perfectly. U can simple rebuild it for execute procedure or do whatever u need.
public class ScriptMain : UserComponent
{
OdbcConnection odbcConn;
OdbcCommand odbcCmd;
OdbcParameter odbcParam;
public override void AcquireConnections(object Transaction)
{
/// Create a String base on that which u define on package for connection and
adding a password
string connectionString;
connectionString = this.Connections.SFConnection.ConnectionString;
odbcConn = new OdbcConnection(connectionString + "PWD=YOURPASSWORD");
odbcConn.Open();
}
public override void PreExecute()
{
///Create command which we wanna execute
base.PreExecute();
odbcCmd = new OdbcCommand("UPDATE klienci SET IMIE= ?,NAZWISKO= ? ,NUMER_TELEFONU= ? ,EMAIL= ? ,ULICA= ? ,MIASTO= ? ,STATE= ? ,ZIP_CODE = ? WHERE CUSTOMER_ID= ?", odbcConn);
}
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
///Adding parameters and connecting them with our input column from package
odbcCmd.Parameters.AddWithValue("#IMIE", Row.Sourcefirstname);
odbcCmd.Parameters.AddWithValue("#NAZWISKO", Row.Sourcelastname);
odbcCmd.Parameters.AddWithValue("#NUMER_TELEFONU", Row.Sourcephone);
odbcCmd.Parameters.AddWithValue("#EMAIL", Row.Sourceemail);
odbcCmd.Parameters.AddWithValue("#ULICA", Row.Sourcestreet);
odbcCmd.Parameters.AddWithValue("#MIASTO", Row.Sourcecity);
odbcCmd.Parameters.AddWithValue("#STATE", Row.Sourcestate);
odbcCmd.Parameters.AddWithValue("#ZIP_CODE", Row.Sourcezipcode);
odbcCmd.Parameters.AddWithValue("#CUSTOMER_ID", Row.Sourcecustomerid);
odbcCmd.ExecuteNonQuery();
}
}

Related

SQL Server : get messages from referenced entities procedure in code

I'm running big dependency scan on legacy db and see that some objects have obsolete ref links, if you run this code in SSMS for View that points to not existing table like in my case, you will get your output on Results tab AND error info in Messages . Like in my case below.
I tried to check all env things I know and output of this stored procedure, but didn't see any indication.
How I can capture this event as I'm running this in looped dynamic SQL script and capture output in my table for further processing?
Updated:
it just text in Message box ,on error, you still have output on
Results tab
this is sp, it loop thru object list I took from sys.object and run this string as my sample to get all dependencies, load all into table. This call to
sql_reference_entities is the only way to get inter database
dependency on column level. So I need stick to this 100$>
--
Select *
From sys.dm_sql_referenced_entities('dbo.v_View_Obs_Table','Object')
--
----update------
This behavior was fixed in SQL Server 2014 SP3 and SQL Server 2016 SP2:
Starting from Microsoft SQL Server 2012, errors raised by
sys.dm_sql_referenced_entities (such as when an object has undergone a
schema change) cannot be caught in a TRY...CATCH Transact-SQL block.
While this behavior is expected in SQL Server 2012 and above, this
improvement introduces a new column that's called is_incomplete to the
Dynamic Management View (DMV).
KB4038418 - Update adds a new column to DMV sys.dm_sql_referenced_entities in SQL Server 2014 and 2016
----update-------
The tldr is that you can't capture these on the server side, and must use a client program in C#, PowerShell or some other client that can process info messages.
That DMV is doing something strange that I don't fully understand. It's generating errors (which a normal UDF is not allowed to do), and those errors do not trigger a TRY/CATCH block or set ##error. EG
create table tempdb.dbo.foo(id int)
go
create view dbo.v_View_Obs_Table
as
select * from tempdb.dbo.foo
go
drop table tempdb.dbo.foo
go
begin try
Select * From sys.dm_sql_referenced_entities('dbo.v_View_Obs_Table','Object')
end try
begin catch
select ERROR_MESSAGE(); --<-- not hit
end catch
However these are real errors, as you can see running this from client code:
using System;
using System.Data.SqlClient;
namespace ConsoleApp6
{
class Program
{
static void Main(string[] args)
{
using (var con = new SqlConnection("Server=.;database=AdventureWorks;integrated security=true"))
{
con.Open();
con.FireInfoMessageEventOnUserErrors = true;
con.InfoMessage += (s, a) =>
{
Console.WriteLine($"{a.Message}");
foreach (SqlError e in a.Errors)
{
Console.WriteLine($"{e.Message} Number:{e.Number} Class:{e.Class} State:{e.State} at {e.Procedure}:{e.LineNumber}");
}
};
var cmd = con.CreateCommand();
cmd.CommandText = "Select * From sys.dm_sql_referenced_entities('dbo.v_View_Obs_Table','Object')";
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read() || (rdr.NextResult() && rdr.Read()))
{
Console.WriteLine(rdr[0]);
}
}
Console.ReadKey();
}
}
}
}
outputs
Invalid object name 'tempdb.dbo.foo'.
Invalid object name 'tempdb.dbo.foo'. Number:208 Class:16 State:3 at v_View_Obs_Table:4
0
The dependencies reported for entity "dbo.v_View_Obs_Table" might not include references to all columns. This is either because the entity references an object that does not exist or because of an error in one or more statements in the entity. Before rerunning the query, ensure that there are no errors in the entity and that all objects referenced by the entity exist.
The dependencies reported for entity "dbo.v_View_Obs_Table" might not include references to all columns. This is either because the entity references an object that does not exist or because of an error in one or more statements in the entity. Before rerunning the query, ensure that there are no errors in the entity and that all objects referenced by the entity exist. Number:2020 Class:16 State:1 at :1

SQL Server 2016 SSIS get cursor from stored procedure

I am using SQL Server 2016.
I have a stored procedure GET_RECORDS that takes input parameters for filter and outputs a CURSOR parameter
I want to get this cursor in my SSIS package
I had created data flow task, OleDb source and variables for parameter values. Then mapped parameters
Params mapping screen
but when I wanted to save the component - I got an error
error screen
I tried to add clause WITH RESULT SETS with some dummy columns, but my procedure doesn't return any result set
What am I doing wrong?
Any advices will be helpful.
Thank you.
With regards, Yuriy.
The source component is trying to determine what columns and types will be returned. Because you are using dynamic SQL the metadata can change each time you run it.
With result sets allows you to define the data being returned but should only be used if you are guaranteed to have those results every time you execute.
EDIT:
I create a connection and run the command so that it populates a data table. Then I put the column headers into a string array. There are plenty of examples out there.
Then I use the following function to create a destination table. Finally I create a datareader and pass that to the .Net SqlBulkCopy. Hope this helps.
private void CreateTable(string TableName, string[] Fields)
{
if (TableExists(TableName) && Overwrite)
{
SqlCommand = new SqlCommand($"Drop Table [{TableName}]", SqlConnection);
SqlCommand.ExecuteNonQuery();
}
string Sql = $"Create Table [{TableName}] (";
int ColumnNumber = 1;
foreach (string Field in Fields)
{
string FieldValue = Field;
if (! HasHeaders)
{
FieldValue = "Column" + ColumnNumber;
ColumnNumber++;
}
Sql += $"[{FieldValue}] Varchar(8000),";
}
Sql = Sql + "ImportFileID Int, ID Int Identity(1,1) Not Null, Constraint [PK_" + TableName + "] Primary Key Clustered ([ID] Asc))";
SqlCommand = new SqlCommand(Sql, SqlConnection);
SqlCommand.ExecuteNonQuery();
}
Use ado.net source instead of oledb source, define a simple select and get the columns you wish to return. Now you can define expresión in the dataflow properties.
Search ado.net source dynamic sql
:)
try to return the records and use foreach in ETL instead of cursor
https://www.simple-talk.com/sql/ssis/implementing-foreach-looping-logic-in-ssis/
I think you can do it from a simple way, but I don't know what you are you doing, exactly...

Set database collation in Entity Framework Code-First Initializer

I want to set the default collation for a database, when Entity Framework Code First creates it.
I've tried the following:
public class TestInitializer<T> : DropCreateDatabaseAlways<T> where T: DbContext
{
protected override void Seed(T context)
{
context.Database.ExecuteSqlCommand("ALTER DATABASE [Test] SET SINGLE_USER WITH ROLLBACK IMMEDIATE");
context.Database.ExecuteSqlCommand("ALTER DATABASE [Test] COLLATE Latin1_General_CI_AS");
context.Database.ExecuteSqlCommand("ALTER DATABASE [Test] SET MULTI_USER");
}
}
This appears to run OK when SQL Server is already set to the same default collation Latin1_General_CI_AS.
But if I specify a different collation, say SQL_Latin1_General_CP1_CI_AS this fails with the error,
System.Data.SqlClient.SqlException: Resetting the connection results in a different
state than the initial login. The login fails.
Can anyone advise how I can set the collation please?
Solution with a command interceptor
It is definitely possible, though it's a bit of a hack. You can alter the CREATE DATABASE command with a command interceptor. Il will intercept all the commands sent to the database, recognize the database creation command based on a regex expression, and alter the command text with your collation.
Before database creation
DbInterception.Add(new CreateDatabaseCollationInterceptor("SQL_Romanian_Cp1250_CI_AS_KI_WI"));
The interceptor
public class CreateDatabaseCollationInterceptor : IDbCommandInterceptor
{
private readonly string _collation;
public CreateDatabaseCollationInterceptor(string collation)
{
_collation = collation;
}
public void NonQueryExecuted(DbCommand command, DbCommandInterceptionContext<int> interceptionContext) { }
public void NonQueryExecuting(DbCommand command, DbCommandInterceptionContext<int> interceptionContext)
{
// Works for SQL Server
if (Regex.IsMatch(command.CommandText, #"^create database \[.*]$"))
{
command.CommandText += " COLLATE " + _collation;
}
}
public void ReaderExecuted(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext) { }
public void ReaderExecuting(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext) { }
public void ScalarExecuted(DbCommand command, DbCommandInterceptionContext<object> interceptionContext) { }
public void ScalarExecuting(DbCommand command, DbCommandInterceptionContext<object> interceptionContext) { }
}
Remarks
Since the database is created with the right collation from the start, all the columns will automatically inherit that collation and you wan't have to ALTER them afterwards.
Be aware that it will impact any later database creation occurring inside the application domain. So you might want to remove the interceptor after the database is created.
I was able to change collation with a custom migration (EF6). I have automatic migrations enabled. You need to delete your DB first.
Create the migration code by typing Add-Migration [YourCustomMigration] in Package Manager Console. (Code First Migrations)
First step should create your migration class with current model creation code in the Up() override. Add your ALTER DATABASE code BEFORE the table creation codes so they are created using the database collation you want. Also, note the suppressTransaction flag:
public override void Up()
{
Sql("ALTER DATABASE [YourDB] COLLATE [YourCollation]", suppressTransaction: true);
[...Your DB Objects Creation codes here...]
}
Each update-database command issued from then on creates a new migration class. All migration codes are executed in order.
My solution with EFCore 2.1 was to derive from the SqlServerMigrationsSqlGenerator and override Generate(SqlServerCreateDatabaseOperation, IModel, MigrationCommandListBuilder)
internal class CustomSqlServerMigrationsSqlGenerator : SqlServerMigrationsSqlGenerator
{
internal const string DatabaseCollationName = "SQL_Latin1_General_CP1_CI_AI";
public CustomSqlServerMigrationsSqlGenerator(
MigrationsSqlGeneratorDependencies dependencies,
IMigrationsAnnotationProvider migrationsAnnotations)
: base(dependencies, migrationsAnnotations)
{
}
protected override void Generate(
SqlServerCreateDatabaseOperation operation,
IModel model,
MigrationCommandListBuilder builder)
{
base.Generate(operation, model, builder);
if (DatabaseCollationName != null)
{
builder
.Append("ALTER DATABASE ")
.Append(Dependencies.SqlGenerationHelper.DelimitIdentifier(operation.Name))
.Append(" COLLATE ")
.Append(DatabaseCollationName)
.AppendLine(Dependencies.SqlGenerationHelper.StatementTerminator)
.EndCommand(suppressTransaction: true);
}
}
}
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
base.OnConfiguring(optionsBuilder);
optionsBuilder.ReplaceService<IMigrationsSqlGenerator, CustomSqlServerMigrationsSqlGenerator>();
}
then used it in the DbContext by replacing the IMigrationsSqlGenerator service
public class MyDbContext : DbContext
{
//...
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
base.OnConfiguring(optionsBuilder);
optionsBuilder.ReplaceService<IMigrationsSqlGenerator, CustomSqlServerMigrationsSqlGenerator>();
}
//...
}
I have had the same problem a while ago. Possible solutions:
It appears that EF creates the database using the server default
collation so one thing you could do is change that.
You cannot change the database collation within the Seed() method
but you can change the collation of individual columns for a table
(NOTE: there is no such thing as table collation, it does relate to
column in a table). You will have to change each column's collation
separately.
If you are using migrations, you could alter the table column
collations within your Up() method.
As you are using the Seed() method, I would suggest the following (modify as appropriate) within the Seed() method:
context.Database.ExecuteSqlCommand(
#"ALTER TABLE MyTable ALTER COLUMN MyColumn NVARCHAR(max) COLLATE MyCollation NOT NULL");
Hope that helps.
I would like to explain why you should not use the seed method for this. If you change your database collation after any columns have been added there is a large risk for collation conflicts like below
Cannot resolve the collation conflict between
"SQL_Latin1_General_CP1_CI_AS" and "Latin1_General_100_CI_AS" in the
equal to operation.
This is due to the fact that if you alter your database with ALTER DATABASE [YourDb] COLLATE [YourCollation] you will only change the databases collation and not previously created columns.
Example in T-SQL:
DECLARE #DBName nvarchar(50), #SQLString nvarchar(200)
SET #DBName = db_name();
SET #SQLString = 'ALTER DATABASE [' + #DBName + '] COLLATE Latin1_General_100_CI_AS'
EXEC(#SQLString)
/* Find Collation of SQL Server Database */
SELECT DATABASEPROPERTYEX(#DBName, 'Collation')
/* Find Collation of SQL Server Database Table Column */
SELECT name, collation_name
FROM sys.columns
WHERE OBJECT_ID IN (SELECT OBJECT_ID
FROM sys.objects
WHERE type = 'U'
AND name = 'AspNetUsers')
AND name = 'FirstName'
Due to this you need to change database collation before any columns are added or change every column separately. Possible solutions:
#MathieuRenda https://stackoverflow.com/a/42576705/3850405
I would put the DbInterception.Add in a class deriving from DbConfiguration or in Application_Start in Global.asax as recommended in the documentation. Note: Wherever you put this code, be careful not to execute DbInterception.Add for the same interceptor more than once, or you'll get additional interceptor instances.
public class ApplicationDbConfiguration: DbConfiguration
{
public ApplicationDbConfiguration()
{
DbInterception.Add(new CreateDatabaseCollationInterceptor("Latin1_General_100_CI_AS"));
}
}
I would also not inherit from the interface but instead use the implementation of DbCommandInterceptor as Microsoft does in their examples.
using System.Data.Common;
using System.Data.Entity.Infrastructure.Interception;
using System.Text.RegularExpressions;
namespace Application.Repositories.EntityFramework
{
public class CreateDatabaseCollationInterceptor : DbCommandInterceptor
{
private readonly string _collation;
public CreateDatabaseCollationInterceptor(string collation)
{
_collation = collation;
}
public override void NonQueryExecuting(DbCommand command, DbCommandInterceptionContext<int> interceptionContext)
{
// Works for SQL Server
if (Regex.IsMatch(command.CommandText, #"^create database \[.*]$"))
{
command.CommandText += " COLLATE " + _collation;
}
}
}
}
More information here: https://learn.microsoft.com/en-us/aspnet/mvc/overview/getting-started/getting-started-with-ef-using-mvc/connection-resiliency-and-command-interception-with-the-entity-framework-in-an-asp-net-mvc-application
#steliosalex: https://stackoverflow.com/a/22895703/3850405. Note that changing every column might not be enough either. You also need to handle metadata and parameters for stored procedure and similar get the collation that the database had when these where created. Changing collation completely requires a create database command with the right collation.
#RahmiAksu https://stackoverflow.com/a/31119371/3850405 NOTE: This is not a good solution in my opinion but if you use it edit the very first migration. Can't be used if the database is already in production. If you have a seed method the exception Resetting the connection results in a different state than the initial login will be thrown.
Your Seed SqlException can be solved by using a plain ADO.Net connection, so the context's connection won't be reset. However as mentioned above this will probably cause a lot of errors later.
using (var conn = new SqlConnection(context.Database.Connection.ConnectionString))
{
using (var cmd = conn.CreateCommand())
{
cmd.CommandText =
string.Format("ALTER DATABASE [{0}] COLLATE Latin1_General_100_CI_AS",
context.Database.Connection.Database));
conn.Open();
cmd.ExecuteNonQuery();
}
}
SqlException: Resetting the connection results in a different state
than the initial login. The login fails. Login failed for user ''.
Cannot continue the execution because the session is in the kill
state.
Source:
https://stackoverflow.com/a/50400609/3850405
It's simply not possible using current versions of EF (EF6). However, at least EF6+ can now work with already existent database. We've changed our deployment scenario such that the database is already created by our deployment script (incl. the default collation) and let EF6 work with the existing database (using the correct default collation).
If you absolutely have to create the database inside your code and cannot use anything else than EF (e.g. you are not able to create the database using ADO.NET) then you have to go for seliosalex answer. It's the only solution we came up, however, see my comment, it is a lot of work to do it right.
EF 5 now supports creating missing tables in an existing database with Code First, so you can create an empty database and set the collation correct, before running an CF on it.
1):
public class DataBaseContext : System.Data.Entity.DbContext
{
public DataBaseContext() : base("MyDB") { }
static DataBaseContext()
{
System.Data.Entity.Database.SetInitializer(new MyInitializer());
}
....
}
2):
public class MyInitializer : DropCreateDatabaseIfModelChanges<DataBaseContext>
{
public MyInitializer() { }
protected override void Seed(DataBaseContext context)
{
base.Seed(context);
using (var conn = new SqlConnection(context.Database.Connection.ConnectionString))
{
using (var cmd = conn.CreateCommand())
{
cmd.CommandText =
string.Format("ALTER DATABASE [{0}] COLLATE Persian_100_CI_AS",
context.Database.Connection.Database);
conn.Open();
cmd.ExecuteNonQuery();
conn.Close();
}
context.Database.Connection.Close();
}
}
}

Problems deleting data from database

I am using Hibernate to access my database. I would like to delete a set of fields on function of a criteria. My database is PostgreSQL and my Java code is:
public void deleteAttr(String parameter){
Configuration cfg = new Configuration();
cfg.configure(resource.getString("hibernate_config_file"));
SessionFactory sessionFactory = cfg.buildSessionFactory();
session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
tx.begin();
String sql = "delete from attribute where timestamp > to_date('"+parameter+"','YYYY-MM-DD')"
session.createSQLQuery(sql);
tx.commit();
}
The method runs, but it doesn't delete data from database. I have also checked the sql sentence in PgAdmin and it works, but not in code. Why? Does someone help me?
Thanks in advance!
It's because you're creating a query, but you don't execute it:
String sql = "delete from attribute where timestamp > to_date('"+parameter+"','YYYY-MM-DD')"
Query query = session.createSQLQuery(sql);
query.executeUpdate();
You should really use bound named parameters rather than string concatenation to pass parameters in your query: it's usually more efficient, it' much more robust, but above all, it doesn't open the door to SQL injection attacks.

C# 20,000 records selected from oracle need to be inserted into SQL2005

i have a console app in c# that extracts 20 fields from an oracle DB witht he code below and i wanted an efficient way to insert them into SQL 2005.
i dotn want to insert each one of the 20,000 within the while loop, obviously. i was thinking to change the code to use a data set to cache all the records and then do a bulk insert...
thoughts?
pseudo code would be nice since i am new to oracle.
this is my code where i was testing getting a connection to oracle and seeing if i can view the data... now i can view it i want to get it out and into sql2005... what do i do from here?
static void getData()
{
string connectionString = GetConnectionString();
using (OracleConnection connection = new OracleConnection())
{
connection.ConnectionString = connectionString;
connection.Open();
OracleCommand command = connection.CreateCommand();
string sql = "SELECT * FROM BUG";
command.CommandText = sql;
OracleDataReader reader = command.ExecuteReader();
while (reader.Read())
{
//string myField = (string)reader["Project"];
string myField = reader[0].ToString();
Console.WriteLine(myField);
}
}
}
You can create a CSV file and then use BULK INSERT to insert the file into SQL Server. Have a look here for an example.
The "bulk" insert with the cached Dataset will work exactly like the while loop you are not wanting to write! The problem is that you'll lose control of the process if you try to use the "bulk" insert of the Dataset class. It is extraneous work in the end.
Maybe the best solution is to use a DataWriter so that you have complete control and no Dataset overhead.
You can actually do 100-1000 inserts per sql batch. Just generate multiple inserts, then submit. Pregenerate the next SELECT batch WHILE THE FIRST EXECUTES.

Resources