I need to put a message into an Azure ServiceBus queue from an SSIS package running under SQL Server 2014. As suggested in this post: connecting to azure service bus queue from ssis,
I wrote a Script Task that references the "Azure SDK 2.9". This approach has worked for me with Azure Storage Accounts to work with blobs (referencing the Microsoft.WindowsAzure.Storage assembly), but it is NOT working for the Azure Storage Bus (referencing the Microsoft.ServiceBus assembly). Any calls I make into that assembly trigger a Run-time exception: "exception has been thrown by the target of an invocation: at System.RuntimeMethodHandle.InvokeMethod(...)" When I comment out all calls to the Microsoft.ServiceBus assembly it runs fine, so it is obviously something about the assembly reference (version 2.4). I tried updating to the latest version with NuGet (version 3.0) and that made no difference.
So my question is: has anybody been able to place a message in an Azure Service Bus queue from SSIS, and if so, how did you do it?
Since somebody will ask for my Script Task code, I'm posting it:
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion
#region CustomNamespaces
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
#endregion
namespace ST_dba6519c1eda4e0c968485a6eb7a6c29
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
try
{
// Create the message for the Queue
string ClientShortName = Dts.Variables["$Package::ClientShortName"].Value.ToString();
bool bExtendedForecast = (bool)Dts.Variables["$Package::ExtendedForecast"].Value;
var msg = new BrokeredMessage(ClientShortName + ": ExtendedForecast=" + bExtendedForecast.ToString()); // this statement throws the exception
// get Service Bus Connection Information from the Package Parameters
string SBAccessKey = Dts.Variables["$Package::ServiceBusAccessKey"].Value.ToString();
string SBNamespace = Dts.Variables["$Package::ServiceBusNamespace"].Value.ToString();
string SBQueue = Dts.Variables["$Package::ServiceBusQueueName"].Value.ToString();
String connStr = "Endpoint=sb://" + SBNamespace +
".servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=" + SBAccessKey;
// First Method tried
Uri SBUri = ServiceBusEnvironment.CreateServiceUri(String.Empty, SBNamespace, String.Empty); // this statement throws the exception
TokenProvider SBToken = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", SBAccessKey);
NamespaceManager nsMgr = new NamespaceManager(SBUri, SBToken);
MessagingFactory msgFactory = MessagingFactory.Create(nsMgr.Address, nsMgr.Settings.TokenProvider);
QueueClient queueClient2 = msgFactory.CreateQueueClient(SBQueue);
queueClient2.Send(msg);
// Second Method tried
MessagingFactory factory = MessagingFactory.CreateFromConnectionString(connStr); // this statement throws the exception
MessageSender queueSender = factory.CreateMessageSender(SBQueue);
queueSender.Send(msg);
// Third Method tried
QueueClient queueClient = QueueClient.CreateFromConnectionString(connStr, SBQueue); // this statement throws the exception
queueClient.Send(msg);
Dts.TaskResult = (int)ScriptResults.Success;
}
catch
{
Dts.TaskResult = (int)ScriptResults.Failure;
}
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
So, of course 10 minutes after I post the question, I hit upon the answer. I had to run GACUTIL -i Microsoft.ServiceBus. Once that was done, I chose to use the Third Method in the code (the simplest) to Send the message, and it worked fine.
Related
I am developing small POC application to test .NET7 support for distributed transactions since this is pretty important aspect in our workflow.
So far I've been unable to make it work and I'm not sure why. It seems to me either some kind of bug in .NET7 or im missing something.
In short POC is pretty simple, it runs WorkerService which does two things:
Saves into "bussiness database"
Publishes a message on NServiceBus queue which uses MSSQL Transport.
Without Transaction Scope this works fine however, when adding transaction scope I'm asked to turn on support for distributed transactions using:
TransactionManager.ImplicitDistributedTransactions = true;
Executable code in Worker service is as follows:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
int number = 0;
try
{
while (!stoppingToken.IsCancellationRequested)
{
number = number + 1;
using var transactionScope = TransactionUtils.CreateTransactionScope();
await SaveDummyDataIntoTable2Dapper($"saved {number}").ConfigureAwait(false);
await messageSession.Publish(new MyMessage { Number = number }, stoppingToken)
.ConfigureAwait(false);
_logger.LogInformation("Publishing message {number}", number);
_logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
transactionScope.Complete();
_logger.LogInformation("Transaction complete");
await Task.Delay(1000, stoppingToken);
}
}
catch (Exception e)
{
_logger.LogError("Exception: {ex}", e);
throw;
}
}
Transaction scope is created with the following parameters:
public class TransactionUtils
{
public static TransactionScope CreateTransactionScope()
{
var transactionOptions = new TransactionOptions();
transactionOptions.IsolationLevel = IsolationLevel.ReadCommitted;
transactionOptions.Timeout = TransactionManager.MaximumTimeout;
return new TransactionScope(TransactionScopeOption.Required, transactionOptions,TransactionScopeAsyncFlowOption.Enabled);
}
}
Code for saving into database uses simple dapper GenericRepository library:
private async Task SaveDummyDataIntoTable2Dapper(string data)
{
using var scope = ServiceProvider.CreateScope();
var mainTableRepository =
scope.ServiceProvider
.GetRequiredService<MainTableRepository>();
await mainTableRepository.InsertAsync(new MainTable()
{
Data = data,
UpdatedDate = DateTime.Now
});
}
I had to use scope here since repository is scoped and worker is singleton so It cannot be injected directly.
I've tried persistence with EF Core as well same results:
Transaction.Complete() line passes and then when trying to dispose of transaction scope it hangs(sometimes it manages to insert couple of rows then hangs).
Without transaction scope everything works fine
I'm not sure what(if anything) I'm missing here or simply this still does not work in .NET7?
Note that I have MSDTC enable on my machine and im executing this on Windows 10
We've been able to solve this by using the following code.
With this modification DTC is actually invoked correctly and works from within .NET7.
using var transactionScope = TransactionUtils.CreateTransactionScope().EnsureDistributed();
Extension method EnsureDistributed implementation is as follows:
public static TransactionScope EnsureDistributed(this TransactionScope ts)
{
Transaction.Current?.EnlistDurable(DummyEnlistmentNotification.Id, new DummyEnlistmentNotification(),
EnlistmentOptions.None);
return ts;
}
internal class DummyEnlistmentNotification : IEnlistmentNotification
{
internal static readonly Guid Id = new("8d952615-7f67-4579-94fa-5c36f0c61478");
public void Prepare(PreparingEnlistment preparingEnlistment)
{
preparingEnlistment.Prepared();
}
public void Commit(Enlistment enlistment)
{
enlistment.Done();
}
public void Rollback(Enlistment enlistment)
{
enlistment.Done();
}
public void InDoubt(Enlistment enlistment)
{
enlistment.Done();
}
This is 10year old code snippet yet it works(im guessing because .NET Core merely copied and refactored the code from .NET for DistributedTransactions, which also copied bugs).
What it does it creates Distributed transaction right away rather than creating LTM transaction then promoting it to DTC if required.
More details explanation can be found here:
https://www.davidboike.dev/2010/04/forcibly-creating-a-distributed-net-transaction/
https://github.com/davybrion/companysite-dotnet/blob/master/content/blog/2010-03-msdtc-woes-with-nservicebus-and-nhibernate.md
Ensure you're using Microsoft.Data.SqlClient +v5.1
Replace all "usings" System.Data.SqlClient > Microsoft.Data.SqlClient
Ensure ImplicitDistributedTransactions is set True:
TransactionManager.ImplicitDistributedTransactions = true;
using (var ts = new TransactionScope(your options))
{
TransactionInterop.GetTransmitterPropagationToken(Transaction.Current);
... your code ..
ts.Complete();
}
I'm new to SSIS and would like to send an email notification when a package fails. I'm using script task with the following code:
#region Namespaces
using System;
using System.Net;
using System.Net.Mail;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
#endregion
public void Main()
{
// TODO: Add your code here
String SendMailFrom = Dts.Variables["EmailFrom"].Value.ToString();
String SendMailTo = Dts.Variables["EmailTo"].Value.ToString();
String SendMailSubject = Dts.Variables["EmailSubject"].Value.ToString();
String SendMailBody = Dts.Variables["EmailBody"].Value.ToString();
try
{
MailMessage email = new MailMessage();
SmtpClient SmtpServer = new SmtpClient("smtp.office365.com");
// START
email.From = new MailAddress(SendMailFrom);
email.To.Add(SendMailTo);
email.Subject = SendMailSubject;
email.Body = SendMailBody;
//END
SmtpServer.Port = 587;
SmtpServer.Credentials = new System.Net.NetworkCredential(SendMailFrom, "Password");
SmtpServer.EnableSsl = true;
SmtpServer.Send(email);
MessageBox.Show("Email was Successfully Sent ");
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
Dts.TaskResult = (int)ScriptResults.Success;
Dts.TaskResult = (int)ScriptResults.Success;
}
My first issue is that I can not get this task to work with my own credentials, I get error "System.IOException: Unable to read data from the transport connection: net_io_connection closed."
But even beyond that, I know its unwise to hardcode my own credentials into this script task which I want run by a SQL Agent Job. Is there a way to send this email without any credentials? I don't care where the email is from, only where it is sent to.
SSIS Send Email task has lots of limitations.
It was created long time ago to work with Microsoft Exchange. It even doesn't support emails in HTML format.
Instead of the following line:
SmtpServer.Credentials = new System.Net.NetworkCredential(SendMailFrom, "Password");
You can try the following:
SmtpServer.UseDefaultCredentials = true;
SmtpServer.Credentials = CredentialCache.DefaultNetworkCredentials;
I found the solution to my initial problem here
I was able to add the following line of code to run the script using my own credentials:
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12
However still need to figure out a solution to running this script using sql agent job. Would there be credential issues if another user were to run the job?
I have a task to send query result in an email using SSIS 2017.
I have referred to previous posts related to that but i am not able to get my required answer.
Here's what i have done so far:
1. Create an executive sql task with my query and resultset which refers to a variable of object type
2. create a foreach container
3. placed a script task under foreach loop(i think my issue is with this task) and have placed the variable as readonlyvariable
4. send email task with bodytext as variable
My result set is just one column from the table.
I have referred to this url: How to send the records from a table in an e-mail body using SSIS package?
and please find attached the script task code.
Would be great if you guys could help me out
/*Microsoft SQL Server Integration Services Script Task
Write scripts using Microsoft Visual C# 2008.
The ScriptMain is the entry point class of the script.
*/
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
namespace ST_8015f41e93944f0e944089c73b520312
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
Variables varCollection = null;
string header = string.Empty;
string message = string.Empty;
Dts.VariableDispenser.LockForWrite("User::EmailMessage");
Dts.VariableDispenser.LockForWrite("User::Result");
Dts.VariableDispenser.GetVariables(ref varCollection);
//Set the header message for the query result
if (varCollection["User::EmailMessage"].Value == string.Empty)
{
header = "Execute SQL task output sent using Send Email Task in SSIS:\n\n";
header += string.Format("{0}\n", "Result");
//varCollection["User::EmailMessage"].Value = header;
}
//Format the query result with tab delimiters
message = string.Format("{0}",
varCollection["User::Result"].Value);
varCollection["User::EmailMessage"].Value = varCollection["User::EmailMessage"].Value + message;
Dts.TaskResult = (int)ScriptResults.Success;
}
}
}
I was able to accomplish this without using a script task or db mail. I needed to email myself notices of any reports that failed to refresh across multiple reporting systems. But I didn't want to save the results to a file and attach the file. I just wanted the results in the email body.
I wrote my query to output one concatenated column and added the query results to a recordset Object and used a foreach loop container to append each row to a string variable with "/n" to start a new line.
This created a string that I could use as the body of the email.
Data Flow
Control Flow
UPDATE
I could never make this work with a "Windows Authentication" (domain) user. But with a "SQL Server Authentication" user everything is working like it's supposed to.
ORIGINAL QUESTION
My connectionString: Server=ip;Database=dbname;User Id=xxx\user;Password=pass;
The connection string is located in appsettings.json like this:
{
"Logging": {
"IncludeScopes": false,
"LogLevel": {
"Default": "Warning"
}
},
"ConnectionStrings": {
"ConnectionString": "Server=ip;Database=dbname;User Id=xxx\user;Password=pass;"
}
}
Then i pass it to a static class from the "Startup.cs" file, like this:
public void ConfigureServices(IServiceCollection services)
{
// Add framework services.
services.AddMvc();
Orm.DatabaseConnection.ConnectionString = Configuration["ConnectionStrings:ConnectionString"];
}
This is where I initiate the connection:
using System.Data.SqlClient;
namespace MyProject.Orm
{
public static class DatabaseConnection
{
public static string ConnectionString { get; set; }
public static SqlConnection ConnectionFactory()
{
return new SqlConnection(ConnectionString);
}
}
}
And this is my controller:
public string Get()
{
using (var databaseConnection = Orm.DatabaseConnection.ConnectionFactory())
{
var sections = databaseConnection.Query("SELECT * FROM myTable").ToList();
return sections.ToString();
}
}
Where this line:
var databaseConnection = Orm.DatabaseConnection.ConnectionFactory();
returns:
ServerVersion: "'databaseConnection.ServerVersion' threw an exception of type 'System.InvalidOperationException'"
Message: "Invalid operation. The connection is closed."
Source: "System.Data.SqlClient"
StackTrace: "at
System.Data.SqlClient.SqlConnection.GetOpenTdsConnection()\n
at
System.Data.SqlClient.SqlConnection.get_ServerVersion()"
And i get this error on new SqlConnection: "error CS0119: 'SqlConnection' is a type, which is not valid in the given context".
But the program execution doesn't stop because of these errors.
The application then hangs on the following line:
var sections = databaseConnection.Query("SELECT * FROM myTable").ToList();
I'm using Dapper as my ORM (not EntityFramework). In "myTable" sql table are only 17 rows and 5 columns so it should load fast.
I tried all kinds of different connectionStrings but it always fails. If i try the same with .NET Framework 4.5, everything works fine. The problem is .NET Core 2.0.
Any idea about fixing it is welcome. Because i spent too many hours on this already.
Try to add databaseConnection.Open().
public string Get()
{
using (var databaseConnection = new SqlConnection(#"Server=ip;Database=dbname;User Id=xxx\user;Password=pass;Pooling=false;"))
{
databaseConnection.Open();
var sections = databaseConnection.Query("SELECT * FROM myTable").ToList();
return sections.ToString();
}
}
To avoid problems with connection pool that described in comments you add Pooling=false; to connection string:
Server=ip;Database=dbname;User Id=xxx\user;Password=pass;Pooling=false;
Edit:
I hardcoded connection string and removed factory to make example smaller
Try creating a self-contained deployment, this should eliminate and strange dependency stuff. If it works then at least you know that it's due to some assembly binding type stuff.
The exception "error CS0119: 'SqlConnection' is a type, which is not valid in the given context" smells like it is.
I have a webjob that calls a long running stored procedure that keeps timing out. Can anyone help please?
The web job is called using the following code:
static void Main()
{
ApplicationDbContext context = new ApplicationDbContext();
context.Database.CommandTimeout = 6000;
context.PopulateJobTypeDescendants();
}
The method on the context (ApplicationDbContext) is shown below:
public void PopulateJobTypeDescendants()
{
Database.ExecuteSqlCommand("PopulateJobTypeDescendants");
}
The following exception is raised when the web job is run. We have read that it could be related to the plan/DTUs on the server so we went from S1 -> S3, this still didn't solve the issue and the process bombs out after 45 seconds. The strange thing is that if I connect to azure sql db from SSMS and call the stored procedure it works fine.
[07/11/2016 22:25:02 > e2cf50: ERR ] Unhandled Exception:
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding. This failure occurred while attempting to connect to
the routing destination. The duration spent while attempting to
connect to the original server was - [Pre-Login] initialization=14;
handshake=26; [Login] initialization=0; authentication=0; [Post-Login]
complete=1; ---> System.ComponentModel.Win32Exception: The wait
operation timed out
The connection string is shown below:
<add name="TempsContext" connectionString="Server=tcp:[XXX],1433;Database=temps_testing;User ID=[XXX];Password=[XXX];Trusted_Connection=False;Encrypt=True;Connection Timeout=600;" providerName="System.Data.SqlClient" />
It is possible that this is caused by some inconsistencies in how EF propagates the value of CommandTimeout to the commands its creates, e.g. to do database initialization or to obtain versioning information from the server.
It should be possible to use command interception as a workaround, e.g.:
using System.Data.Common;
using System.Data.Entity;
using System.Data.Entity.Infrastructure.Interception;
namespace CommandTimeOutBug
{
class Program
{
static void Main(string[] args)
{
DbInterception.Add(new MyInterceptor());
using (var context = new ApplicationDbContext())
{
context.Database.CommandTimeout = 6000;
context.PopulateJobTypeDescendants();
}
}
}
public class ApplicationDbContext : DbContext
{
public void PopulateJobTypeDescendants()
{
Database.ExecuteSqlCommand("PopulateJobTypeDescendants");
}
}
public class MyInterceptor: DbCommandInterceptor
{
public override void NonQueryExecuting(DbCommand command,
DbCommandInterceptionContext<int> interceptionContext)
{
command.CommandTimeout = 6000;
base.NonQueryExecuting(command, interceptionContext);
}
public override void ReaderExecuting(DbCommand command,
DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
command.CommandTimeout = 6000;
base.ReaderExecuting(command, interceptionContext);
}
public override void ScalarExecuting(DbCommand command,
DbCommandInterceptionContext<object> interceptionContext)
{
command.CommandTimeout = 6000;
base.ScalarExecuting(command, interceptionContext);
}
}
}
I have created a bug at https://github.com/aspnet/EntityFramework6/issues/24 to track this.
Indeed it looks like a performance issue. Could you please reply me offline at mihaelab at micrososft dot com with server, database names details and full call stack of the exception?
Thanks,
Mihaela