I am new to dapper and plan to use it on my new project. After reading it, seems like the only problem I might have is ConcurrentDictionary.
Dapper caches information about every query it runs, this allow it to materialize objects
quickly and process parameters quickly. The current implementation
caches this information in a ConcurrentDictionary object. The objects
it stores are never flushed. If you are generating SQL strings on the
fly without using parameters it is possible you will hit memory
issues. We may convert the dictionaries to an LRU Cache.
How do I avoid this problem? Can someone please show me some code tell me how and when to flush it?
Per the comments up top, here's an example of on-the-fly:
var builder = new StringBuilder();
builder.AppendLine("SELECT Foo FROM Bar");
if (fisrtName != null || lastName != null)
builder.AppendLine("WHERE");
if (firstName != null)
builder.AppendLine(" Bar.FirstName = #Firstname");
if (firstName != null && lastName != null)
builder.Append(" AND");
if (lastName != null)
builder.AppendLine(" Bar.LastName = #LastName");
var sql = builder.ToString();
As you can see, the actual SQL that dapper will now run will be different based on whether or not firstName and/or lastName are null. If both are null, you get one SQL string. If only firstName is not null, you get another. If only lastName is not null, you get yet another. And finally, if both are not null you get a fourth permutation.
This is what is meant by "on-the-fly" -- dapper will cache based on these unique permutations, and given a more complex scenario, it is easy to see how you'll end up with a great many different permutations, all of which would need to be cached independently.
Related
I am trying to handle near-simultaneous input to my Entity Framework application. Members (users) can rate things, so I have a table for their ratings, where one column is the member's ID, one is the ID of the thing they're rating, one is the rating, and another is the time they rated it. The most recent rating is supposed to override the earlier ratings. When I receive input, I check to see if the member has already rated a thing or not, and if they have, I just update the rating using the existing row, or if they haven't, I add a new row. I noticed that when input comes in from the same user for the same item at nearly the same time, that I end up with two ratings for that user for the same thing.
Earlier I asked this question: How can I avoid duplicate rows from near-simultaneous SQL adds? and I followed the suggestion to add a SQL constraint requiring unique combinations of MemberID and ThingID, which makes sense, but I am having trouble getting this technique to work, probably because I don't know the syntax for doing what I want to do when an exception occurs. The exception comes up saying the constraint was violated, and what I would like to do then is forget the attemptd illegal addition of a row with the same MemberID and ThingID, and instead fetch the existing one and simply set the values to this slightly more recent data. However I have not been able to come up with a syntax that will do that. I have tried a few things and always I get an exception when I try to SaveChanges after getting the exception - either the unique constraint is still coming up, or I get a deadlock exception.
The latest version I tried was like this:
// Get the member's rating for the thing, or create it.
Member_Thing_Rating memPref = (from mip in _myEntities.Member_Thing_Rating
where mip.thingID == thingId
where mip.MemberID == memberId
select mip).FirstOrDefault();
bool RetryGet = false;
if (memPref == null)
{
using (TransactionScope txScope = new TransactionScope())
{
try
{
memPref = new Member_Thing_Rating();
memPref.MemberID = memberId;
memPref.thingID = thingId;
memPref.EffectiveDate = DateTime.Now;
_myEntities.Member_Thing_Rating.AddObject(memPref);
_myEntities.SaveChanges();
}
catch (Exception ex)
{
Thread.Sleep(750);
RetryGet = true;
}
}
if (RetryGet == true)
{
Member_Thing_Rating memPref = (from mip in _myEntities.Member_Thing_Rating
where mip.thingID == thingId
where mip.MemberID == memberId
select mip).FirstOrDefault();
}
}
After writing the above, I also tried wrapping the logic in a function call, because it seems like Entity Framework cleans up database transactions when leaving scope from where changes were submitted. So instead of using TransactionScope and managing the exception at the same level as above, I wrapped the whole thing inside a managing function, like this:
bool Succeeded = false;
while (Succeeded == false)
{
Thread.Sleep(750);
Exception Problem = AttemptToSaveMemberIngredientPreference(memberId, ingredientId, rating);
if (Problem == null)
Succeeded = true;
else
{
Exception BaseEx = Problem.GetBaseException();
}
}
But this only results in an unending string of exceptions on the unique constraint, being handled forever at the higher-level function. I have a 3/4 second delay between attempts, so I am surprised that there can be a reported conflict yet still there is nothing found when I query for a row. I suppose that indicates that all of the threads are failing because they are running at the same time and Entity Framework notices them all and fails them all before any succeed. So I suppose there should be a way to respond to the exception by looking at all the submissions and adjusting them? I don't know or see the syntax for that. So again, what is the way to handle this?
Update:
Paddy makes three good suggestions below. I expect his Stored Procedure technique would work around the problem, but I am still interested in the answer to the question. That is, surely one should be able to respond to this exception by manipulating the submission, but I haven't yet found the syntax to get it to insert one row and use the latest value.
To quote Eric Lippert, "if it hurts, stop doing it". If you are anticipating getting very high volumnes and you want to do an 'insert or update', then you may want to consider handling this within a stored procedure instead of using the methods outlined above.
Your problem is coming because there is a small gap between your call to the DB to check for existence and your insert/update.
The sproc could use a MERGE to do the insert or update in a single pass on the table, guaranteeing that you will only see a single row for a rating and that it will be the most recent update you receive.
Note - you can include the sproc in your EF model and call it using similar EF syntax.
Note 2 - Looking at your code, you don't rollback the transaction scope prior to sleeping your thread in the case of exception. This is a relatively long time to be holding a transaction open, particularly when you are expecting very high volumes. You may want to update your code something like this:
try
{
memPref = new Member_Thing_Rating();
memPref.MemberID = memberId;
memPref.thingID = thingId;
memPref.EffectiveDate = DateTime.Now;
_myEntities.Member_Thing_Rating.AddObject(memPref);
_myEntities.SaveChanges();
txScope.Complete();
}
catch (Exception ex)
{
txScope.Dispose();
Thread.Sleep(750);
RetryGet = true;
}
This may be why you seem to be suffering from deadlocks when you retry, particularly if you are getting rapid concurrent requests.
I want to retrieve data whose parent key is null
query = pm.newQuery(Question.class, "state==0");
query.setFilter("tier instanceof Tier");
query.setFilter("tier.id == null");
query.setClass(Question.class);
query.setOrdering("lastCustomerActivityTime");
questions = (List<Question>) query.execute();
I have written above code but it fetch all data including whose parent key is not null.
is there any solution?
Strongly suggest you read up on the JDO API and JDOQL. If you set the filter, it sets the filter (overwriting what was there). Nothing is magically appended to the end of what you had, and its illogical to expect it.
I have to retrieve the db entry. I know the query I have written returns only one row. Therefor I am using getSingleResult. However when the query returns a null, I am not able to catch it. How do I solve this?
Here is the piece of code I have
try {
result = em.createQuery("SELECT d FROM fields d WHERE d.fieldID = :fieldID", Field.class)
.setParameter("fieldID", fieldID)
.getSingleResult();
//manual null check only seems to work. But it seems tedious to check every DB column for null value :(
if((result.getValsText() == null)){
result = new Field();
result.setValText("empty");
}
} catch (NoResultException e) {
result = new Field();
result.setValText("empty");
}
Please advise.
Thanks
For non-Id attributes you can either
send a COUNT query before your regular query or better
call getResultList() on your Query
getResultList will return an empty list if no results have been found. To get your single result, check whether the list is empty and call get(0) on it.
If the attribute is the #Id attribute, call entityManager.find(MyEntity.class, id); - find returns null if no results have been found and a single instance of your entity if it has been found.
EDIT- the last option is preferable for reasons of perfomance and readability. Use the second option (the list) where you cannot use the last one.
I am trying to make an update query in Access right now. I have two teachers who have the same data tables in a different database except for the information that they have entered in themselves. Other than their individually entered info, everything is the same. I am trying to merge their two tblDemographics tables so that they can see what each other has done in case a students moves to one of the schools they cover. I was wondering if I have to basically enter in every field that has to be updated to, or is there some short hand that might make it to where I can basically say where Null, update to field of same name?
Sorry if this doesn't make sense. I just am trying to see if there is a more efficient way to do it.
UPDATE tbleDemographics LEFT JOIN tbleDemographics1 ON tbleDemographics.[Local ID] = tbleDemographics1.[Local ID] SET tbleDemographics.FName = [tbleDemographics1.Fname], tbleDemographics.LName = [tbleDemographics1.LName], tbleDemographics.MName = [tbleDemographics1.MName]
WHERE (((tbleDemographics.FName) Is Null) AND ((tbleDemographics.LName) Is Null) AND ((tbleDemographics.MName) Is Null) AND ((tbleDemographics.ClMgr) Is Null) AND ((tbleDemographics.School) Is Null) AND ((tbleDemographics.Grade) Is Null) AND ((tbleDemographics.[Prim Dis]) Is Null) AND ((tbleDemographics.[Sec Dis]) Is Null) AND ((tbleDemographics.[Third Dis]) Is Null) AND ((tbleDemographics.[Local ID]) Is Null) AND ((tbleDemographics.GTID) Is Null) AND ((tbleDemographics.Status) Is Null) AND ((tbleDemographics.[Homeroom Teacher]) Is Null) AND ((tbleDemographics.[GPS Math Teacher]) Is Null) AND ((tbleDemographics.[Number Worlds Teacher]) Is Null) AND ((tbleDemographics.IntervHMcCain) Is Null) AND ((tbleDemographics.InterMSmith) Is Null) AND ((tbleDemographics.InterALacey) Is Null) AND ((tbleDemographics.InterLDaughtry) Is Null) AND ((tbleDemographics.DelInclusion) Is Null) AND ((tbleDemographics.DelRegEd) Is Null) AND ((tbleDemographics.DelConsult) Is Null) AND ((tbleDemographics.DelRes) Is Null) AND ((tbleDemographics.DelPara) Is Null) AND ((tbleDemographics.[DelMIPull-out]) Is Null) AND ((tbleDemographics.DelMIInc) Is Null) AND ((tbleDemographics.OTServices) Is Null) AND ((tbleDemographics.PTServices) Is Null) AND ((tbleDemographics.OIServices) Is Null) AND ((tbleDemographics.SpServices) Is Null) AND ((tbleDemographics.Notes) Is Null));
That looks too messy for my taste, but I want to simply that if possible so that I don't actually have to qualify each entry.
After looking at it some more, I feel quite silly about the SQL that I did put up here. SO yes, I see that I put the Is Nulls in the wrong place and that the way I have it set up with basically overwrite everything... oops. I hope though that the general idea is understood.
Ok...so now that I am getting more and more into this database trying to figure this out myself... I see so many issues that it is not even funny. This database is set up horribly to the point to where it is a nightmare.... if anyone can still come up with a general idea to what I am getting up that would be greatly Appreciated.
Database merging is such a problem that you're probably best off dumping the data out of both databases, merging the data, and then importing it back into a new database. Note that you may need to renumber the keys of any records added to the database, as the keys will collide with entries in the other database. But since it's a database, the keys will also need to be changed in any records that are linked to that one.
If a record is missing in one database but not the other, is it a deletion or an insertion? Or a version problem?
In Access this is particularly difficult as there is no journalling to help you see the history of updates.
OutOfMemoryError caused when db4o databse has 15000+ objects
My question is in reference to my previous question (above). For the same PostedMessage model and same query.
With 100,000 PostedMessage objects, the query takes about 1243 ms to return first 20 PostedMessages.
Now, I have saved 1,000,000 PostedMessage objects in db4o. The same query took 342,132 ms. Which is non-linearly high.
How can I optimize the query speed?
FYR:
The timeSent and timeReceived are Indexed fields.
I am using SNAPSHOT query mode.
I am not using TA/TP.
Do you sort the result? Unfortunatly db4o doesn't use the index for sorting / orderBy. That means it will run a regular sort algorith, with O(n*log(n)). It won't scala liniearly.
Also db4o doesn't support a TOP operator. That means even without sorting it takes quite a bit of time to copy the ids to the results set, even when you never read the entities afterwards.
So, there's no real good solution for this, except trying to use some criteria which cut down the result size.
Some adventerous people might use a different query evaluation, but personally don't recommend that.
#Gamlor No, I am not sorting at all. The code is as follows:
public static ObjectSet<PostedMessage> getMessagesBetweenDates(
Calendar after,
Calendar before,
ObjectContainer db) {
if (after == null || before == null || db == null) {
return null;
}
Query q = db.query(); //db is pre-configured to use SNAPSHOT mode.
q.constrain(PostedMessage.class);
Constraint from = q.descend("timeRecieved").constrain(new Long(after.getTimeInMillis())).greater().equal();
q.descend("timeRecieved").constrain(new Long(before.getTimeInMillis())).smaller().equal().and(from);
ObjectSet<EmailMessage> results = q.execute();
return results;
}
The arguments to this method are as follows:
after = 13-09-2011 10:55:55
before = 13-09-2011 10:56:10
And I expect only 10 PostedMessages to be returned between "after" and "before". (I am generating dummy PostedMessage with timeReceived incremented by 1 sec each.)