I've just started using Dapper for a project, having mostly used ORMs like NHibernate and EF for the past few years.
Typically in our web applications we implement session per request, beginning a transaction at the start of the request and committing it at the end.
Should we do something similar when working directly with SqlConnection / System.Transactions?
How does StackOverflow do it?
Solution
Taking the advice of both #gbn and #Sam Safron I'm not using transactions. In my case I'm only doing read queries so it seems there is no real requirement to use transactions (contrary to what I've been told about implicit transactions).
I create a lightweight session interface so that I can use a connection per request. This is quite beneficial to me as with Dapper I often need to create a few different queries to build up an object and would rather share the same connection.
The work of scoping the connection per request and disposing it is done by my IoC container (StructureMap):
public interface ISession : IDisposable {
IDbConnection Connection { get; }
}
public class DbSession : ISession {
private static readonly object #lock = new object();
private readonly ILogger logger;
private readonly string connectionString;
private IDbConnection cn;
public DbSession(string connectionString, ILogger logger) {
this.connectionString = connectionString;
this.logger = logger;
}
public IDbConnection Connection { get { return GetConnection(); } }
private IDbConnection GetConnection() {
if (cn == null) {
lock (#lock) {
if (cn == null) {
logger.Debug("Creating Connection");
cn = new SqlConnection(connectionString);
cn.Open();
logger.Debug("Opened Connection");
}
}
}
return cn;
}
public void Dispose() {
if (cn != null) {
logger.Debug("Disposing connection (current state '{0}')", cn.State);
cn.Dispose();
}
}
}
This is what we do:
We define a static called DB on an object called Current
public static DBContext DB
{
var result = GetContextItem<T>(itemKey);
if (result == null)
{
result = InstantiateDB();
SetContextItem(itemKey, result);
}
return result;
}
public static T GetContextItem<T>(string itemKey, bool strict = true)
{
#if DEBUG // HttpContext is null for unit test calls, which are only done in DEBUG
if (Context == null)
{
var result = CallContext.GetData(itemKey);
return result != null ? (T)result : default(T);
}
else
{
#endif
var ctx = HttpContext.Current;
if (ctx == null)
{
if (strict) throw new InvalidOperationException("GetContextItem without a context");
return default(T);
}
else
{
var result = ctx.Items[itemKey];
return result != null ? (T)result : default(T);
}
#if DEBUG
}
#endif
}
public static void SetContextItem(string itemKey, object item)
{
#if DEBUG // HttpContext is null for unit test calls, which are only done in DEBUG
if (Context == null)
{
CallContext.SetData(itemKey, item);
}
else
{
#endif
HttpContext.Current.Items[itemKey] = item;
#if DEBUG
}
#endif
}
In our case InstantiateDB returns an L2S context, however in your case it could be an open SQLConnection or whatever.
On our application object we ensure that our connection is closed at the end of the request.
protected void Application_EndRequest(object sender, EventArgs e)
{
Current.DisposeDB(); // closes connection, clears context
}
Then anywhere in your code where you need access to the db you simple call Current.DB and stuff automatically works. This is also unit test friendly due to all the #if DEBUG stuff.
We do not start any transactions per session, if we did and had updates at the beginning of our session, we would get serious locking issues, as the locks would not be released till the end.
You'd only start a SQL Server Transaction when you need to with something like TransactionScope when you call the database with a "write" call.
See a random example in this recent question: Why is a nested transaction committed even if TransactionScope.Complete() is never called?
You would not open a connection and start a transaction per http request. Only on demand. I'm having difficulty understanding why some folk advocate opening a database transaction per session: sheer idiocy when you look at what a database transaction is
Note: I'm not against the pattern per se. I am against unnecessary, too long, client-side database transactions that invoke MSDTC
Related
Having a spout which on each tick goes to Postgre database and reads an additional row. The spout code looks as follows:
class RawDataLevelSpout extends BaseRichSpout implements Serializable {
private int counter;
SpoutOutputCollector collector;
#Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("col1", "col2"));
}
#Override
public void open(Map map, TopologyContext context, SpoutOutputCollector spoutOutputCollector) {
collector = spoutOutputCollector;
}
private Connection initializeDatabaseConnection() {
try {
Class.forName("org.postgresql.Driver");
Connection connection = null;
connection = DriverManager.getConnection(
DATABASE_URI,"root", "root");
return connection;
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
}
return null;
}
#Override
public void close() {
}
#Override
public void nextTuple() {
List<String> values = new ArrayList<>();
PreparedStatement statement = null;
try {
Connection connection = initializeDatabaseConnection();
statement = connection.prepareStatement("SELECT * FROM table1 ORDER BY col1 LIMIT 1 OFFSET ?");
statement.setInt(1, counter++);
ResultSet resultSet = statement.executeQuery();
resultSet.next();
ResultSetMetaData resultSetMetaData = resultSet.getMetaData();
int totalColumns = resultSetMetaData.getColumnCount();
for (int i = 1; i <= totalColumns; i++) {
String value = resultSet.getString(i);
values.add(value);
}
connection.close();
} catch (SQLException e) {
e.printStackTrace();
}
collector.emit(new Values(values.stream().toArray(String[]::new)));
}
}
What is the standard way how to approach connection pooling in Spouts in apache storm? Furthermore, is it possible to somehow synchronize the coutner variable accross multiple running instances within the cluster topology?
Regarding connection pooling, you could pool connections via static variable if you wanted, but since you aren't guaranteed to have all spout instances running in the same JVM, I don't think there's any point.
No, there is no way to synchronize the counter. The spout instances may be running on different JVMs, and you don't want them all blocking while the spouts agree what the counter value is. I don't think your spout implementation makes sense though. If you wanted to just read one row at a time, why would you not just run a single spout instance instead of trying to synchronize multiple spouts?
You seem to be trying to use your relational database as a queue system, which is probably a bad fit. Consider e.g. Kafka instead. I think you should be able to use either one of https://www.confluent.io/product/connectors/ or http://debezium.io/ to stream data from your Postgres to Kafka.
I have an winform application which consumes windows service, i user ChannelFactory
to connect to service, problem is when i call service method using channel the memory usage increase and after
method execute memory not go down(even after form close), i call GC.Collect but no change
channel Create class
public class Channel1
{
List<ChannelFactory> chanelList = new List<ChannelFactory>();
ISales salesObj;
public ISales Sales
{
get
{
if (salesObj == null)
{
ChannelFactory<ISales> saleschannel = new ChannelFactory<ISales>("SalesEndPoint");
chanelList.Add(saleschannel);
salesObj = saleschannel.CreateChannel();
}
return salesObj;
}
}
public void CloseAllChannels()
{
foreach (ChannelFactory chFac in chanelList)
{
chFac.Abort();
((IDisposable)chFac).Dispose();
}
salesObj = null;
}
}
base class
public class Base:Form
{
public Channel1 channelService = new Channel1();
public Channel1 CHANNEL
{
get
{
return channelService;
}
}
}
winform class
Form1:Base
private void btnView_Click(object sender, EventArgs e)
{
DataTable _dt = new DataTable();
try
{
gvAccounts.AutoGenerateColumns = false;
_dt = CHANNEL.Sales.GetDatatable();
gvAccounts.DataSource = _dt;
}
catch (Exception ex)
{
MessageBox.Show("Error Occurred while processing...\n" + ex.Message, "Warning", MessageBoxButtons.OK, MessageBoxIcon.Warning);
}
finally
{
CHANNEL.CloseAllChannels();
_dt.Dispose();
//GC.Collect();
}
}
You're on the right track in terms of using ChannelFactory<T>, but your implementation is a bit off.
ChannelFactory<T> creates a factory for generating channels of type T. This is a relatively expensive operation (as compared to just creating a channel from the existing factory), and is generally done once per life of the application (usually at start). You can then use that factory instance to create as many channels as your application needs.
Generally, once I've created the factory and cached it, when I need to make a call to the service I get a channel from the factory, make the call, and then close/abort the channel.
Using your posted code as a starting point, I would do something like this:
public class Channel1
{
ChannelFactory<ISales> salesChannel;
public ISales Sales
{
get
{
if (salesChannel == null)
{
salesChannel = new ChannelFactory<ISales>("SalesEndPoint");
}
return salesChannel.CreateChannel();
}
}
}
Note that I've replaced the salesObj with salesChannel (the factory). This will create the factory the first time it's called, and create a new channel from the factory every time.
Unless you have a particular requirement to do so, I wouldn't keep track of the different channels, especially if follow the open/do method/close approach.
In your form, it'd look something like this:
private void btnView_Click(object sender, EventArgs e)
{
DataTable _dt = new DataTable();
try
{
gvAccounts.AutoGenerateColumns = false;
ISales client = CHANNEL.Sales
_dt = client.GetDatatable();
gvAccounts.DataSource = _dt;
((ICommunicationObject)client).Close();
}
catch (Exception ex)
{
((ICommunicationObject)client).Abort();
MessageBox.Show("Error Occurred while processing...\n" + ex.Message, "Warning", MessageBoxButtons.OK, MessageBoxIcon.Warning);
}
}
The code above gets a new ISales channel from the factory in CHANNEL, executes the call, and then closes the channel. If an exception happens, the channel is aborted in the catch block.
I would avoid using Dispose() out of the box on the channels, as the implementation in the framework is flawed and will throw an error if the channel is in a faulted state. If you really want to use Dispose() and force the garbage collection, you can - but you'll have to work around the WCF dispose issue. Google will give you a number of workarounds (google WCF Using for a start).
Just getting my head around WCF, so forgive me for the inelegant coding.
The issue I'm having is I seem to be submitting data twice to my service (see screenshot), even though (I think) I'm only doing it once.
Could someone please let me know what I might be doing wrong? Or even just suggest a better way to do it if I'm doing it inefficiently.
Code follows:
public void EndOfLevel()
{
GlobalVariable.TotalQuestionsAsked = 10;
GlobalVariable.CorrectDecimal = GlobalVariable.Correct / GlobalVariable.TotalQuestionsAsked;
//Show loading screen
UploadingScreen.Visibility = Visibility.Visible;
//Submit this levels results.
Service1Client client = null;
client = new Service1Client();
//Gather the results and details
Result thislevel = new Result();
thislevel.Datetime = DateTime.Now;
thislevel.result = GlobalVariable.CorrectDecimal;
thislevel.TimesTable = GlobalVariable.NeedsHelpWith;
//submit them
try
{
client.SubmitResultAsync(thislevel);
}
catch
{
MessageBox.Show("Error uploading data");
}
finally
{
client.Close();
Results r3 = new Results();
this.NavigationService.Navigate(r3);
}
}
WCF Test Client:
Cheers,
Nick
If I may, here's a pattern for managing our asynchronous calls between our WPF applications and our WCF Services.
In this section we have a public accessor to our service client that ensures that the connection to the client is open prior to calling a service method:
public static MyServiceClient Client
{
get
{
return GetMyServiceClient();
}
}
private static MyServiceClient client;
private static MyService.MyServiceClient GetMyServiceClient()
{
VerifyClientConnection();
return client;
}
private static void VerifyClientConnection()
{
if (client == null || client.State == System.ServiceModel.CommunicationState.Closed)
{
client = new MyService.MyServiceClient();
}
}
And in this section is an example of our asynchronous call and callback pattern (this example shows the delegate and callback we're using for passing exception data to our service):
public delegate void LogExceptionCompletedEvent();
public static LogExceptionCompletedEvent LogExceptionCompleted;
public static void LogExceptionAsync(SilverlightException exception)
{
string json = JsonConvert.SerializeObject(exception);
Client.LogExceptionCompleted -= client_LogExceptionCompleted;
Client.LogExceptionCompleted += client_LogExceptionCompleted;
Client.LogExceptionAsync(json);
}
private static void client_LogExceptionCompleted(object sender, AsyncCompletedEventArgs e)
{
if (LogExceptionCompleted != null)
{
LogExceptionCompleted();
}
}
In this example, a view model could attach an event handler to the LogExceptionCompleted delegate and in turn receive the result of the callback when it returns from the service.
We basically repeat this pattern for the asynchronous WCF service calls we need to make from our application and it keeps them very organized as well as unit testable.
Is there any existing plumbing to run WCF calls in batches in a BackgroundWorker?
Obviously since all Silverlight WCF calls are async - if I run them all in a backgroundworker they will all return instantly.
I just don't want to implement a nasty hack if theres a nice way to run service calls and collect the results.
Doesnt matter what order they are done in
All operations are independent
I'd like to have no more than 5 items running at once
Edit: i've also noticed (when using Fiddler) that no more than about 7 calls are able to be sent at any one time. Even when running out-of-browser this limit applies. Is this due to my default browser settings - or configurable also. obviously its a poor man's solution (and not suitable for what i want) but something I'll probably need to take account of to make sure the rest of my app remains responsive if i'm running this as a background task and don't want it using up all my connections.
I think your best bet would be to have your main thread put service request items into a Queue that is shared with a BackgroundWorker thread. The BackgroundWorker can then read from the Queue, and when it detects a new item, initiate the async WCF service request, and setup to handle the AsyncCompletion event. Don't forget to lock the Queue before you call Enqueue() or Dequeue() from different threads.
Here is some code that suggests the beginning of a solution:
using System;
using System.Collections.Generic;
using System.ComponentModel;
namespace MyApplication
{
public class RequestItem
{
public string RequestItemData { get; set; }
}
public class ServiceHelper
{
private BackgroundWorker _Worker = new BackgroundWorker();
private Queue<RequestItem> _Queue = new Queue<RequestItem>();
private List<RequestItem> _ActiveRequests = new List<RequestItem>();
private const int _MaxRequests = 3;
public ServiceHelper()
{
_Worker.DoWork += DoWork;
_Worker.RunWorkerAsync();
}
private void DoWork(object sender, DoWorkEventArgs e)
{
while (!_Worker.CancellationPending)
{
// TBD: Add a N millisecond timer here
// so we are not constantly checking the Queue
// Don't bother checking the queue
// if we already have MaxRequests in process
int _NumRequests = 0;
lock (_ActiveRequests)
{
_NumRequests = _ActiveRequests.Count;
}
if (_NumRequests >= _MaxRequests)
continue;
// Check the queue for new request items
RequestItem item = null;
lock (_Queue)
{
RequestItem item = _Queue.Dequeue();
}
if (item == null)
continue;
// We found a new request item!
lock (_ActiveRequests)
{
_ActiveRequests.Add(item);
}
// TBD: Initiate an async service request,
// something like the following:
try
{
MyServiceRequestClient proxy = new MyServiceRequestClient();
proxy.RequestCompleted += OnRequestCompleted;
proxy.RequestAsync(item);
}
catch (Exception ex)
{
}
}
}
private void OnRequestCompleted(object sender, RequestCompletedEventArgs e)
{
try
{
if (e.Error != null || e.Cancelled)
return;
RequestItem item = e.Result;
lock (_ActiveRequests)
{
_ActiveRequests.Remove(item);
}
}
catch (Exception ex)
{
}
}
public void AddRequest(RequestItem item)
{
lock (_Queue)
{
_Queue.Enqueue(item);
}
}
}
}
Let me know if I can offer more help.
I get the above error sometimes during the read. The exception originates from ASP.NET SqlDataReader whenever you try to read data before calling the Read() method. Since EF does all these internally, I am wondering what else can cause this error. could it be network (or) db connectivity?
thanks
Additional Bounty Info (GenericTypeTea):
I've got the same error after upgrading to EF Code First RC (4.1):
"Invalid attempt to read when no data
is present"
This is the code in question:
using (var context = GetContext())
{
var query = from item in context.Preferences
where item.UserName == userName
where item.PrefName == "TreeState"
select item;
// Error on this line
Preference entity = query.FirstOrDefault();
return entity == null ? null : entity.Value;
}
The table structure is as follows:
Preference
{
Username [varchar(50)]
PrefName [varchar(50)]
Value [varchar(max)] Nullable
}
The table is standalone and has no relationships. This is the DbModelBuilder code:
private void ConfigurePreference(DbModelBuilder builder)
{
builder.Entity<Preference>().HasKey(x => new { x.UserName, x.PrefName });
builder.Entity<Preference>().ToTable("RP_Preference");
}
Exactly the same code works perfectly in CTP5. I'm guessing this is an RC bug, but any ideas of how to fix it would be appreciated.
This error occurs when there is a large amount of data in the RC release. The difference between the RC and CTP5 is that you need to specify the [MaxLength] property that contains a large amount of data.
Are you re-using contexts? I would guess this is happening as a result of something you are doing within GetContext
If GetContext() provides a stale context, in which the DataReader is closed/corrupted, I could see the above happening.
I cannot reproduce your problem on EF4.1 RC1.
POCO:
public class Preference
{
public string UserName { get; set; }
public string PrefName { get; set; }
public string Value { get; set; }
}
Context:
public class PreferenceContext : DbContext
{
public DbSet<Preference> Preferences {get;set;}
public PreferenceContext()
: base("Data Source=localhost;Initial Catalog=_so_question_ef41_rc;Integrated Security=SSPI;") {
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
ConfigurePreference(modelBuilder);
base.OnModelCreating(modelBuilder);
}
private void ConfigurePreference(DbModelBuilder builder)
{
builder.Entity<Preference>().HasKey(x => new { x.UserName, x.PrefName });
builder.Entity<Preference>().ToTable("RP_Preference");
}
}
My little Console App:
class Program
{
static void Main(string[] args)
{
string userName = "Anon";
for (int i = 0; i < 10000; i++)
{
var p = GetPreference(userName);
}
}
private static string GetPreference(string userName)
{
using (var context = new PreferenceContext())
{
var query = from item in context.Preferences
where item.UserName == userName
where item.PrefName == "TreeState"
select item;
// Error on this line
Preference entity = query.FirstOrDefault();
return entity == null ? null : entity.Value;
}
}
}
I do 10,000 reads, and no error. You will need to post more complete code to continue.
Increase the CommandTimeout on the context.
I had the same issue with EF4 - In my case I was (trying to) return the list of entities within the using{} section. This is the same as you are doing in your question:
return entity == null ? null : entity.Value;
} // end using
I moved the return to after the } and it worked.
I think I had the problem because the code was in a function which had already queried the database in another using block, I suspect the table was locking but not reporting the error, ending the using block before the return released the database lock.
Steve