I'm using Dapper with Dapper-Extensions. I'm currently deleting all the objects one-by-one:
dbConnection.Delete<MyObj>(data);
This is bad not only for performance, but also because if a delete fails I would like to rollback the entire operation. Is there a way to perform a "massive" delete, for example passing a list of objects instead of data?
You may pass IPredicate to delete multiple records based on condition (WHERE clause) in one go.
If you simply pass empty IPredicate, all records from the table will be deleted.
Following function handles both the cases:
protected void DeleteBy(IPredicate where)
{//If 'where' is null, this method will delete all rows from the table.
if(where == null)
where = new PredicateGroup { Operator = GroupOperator.And, Predicates = new List<IPredicate>() };//Send empty predicateGroup to delete all records.
var result = connection.Delete<TPoco>(predicate, ......);
}
In above code, TPoco is your POCO type which is mapped to database table you are talking about.
You can build the predicate something like below:
var predicateGroup = new PredicateGroup { Operator = GroupOperator.And, Predicates = new List<IPredicate>() };
if(!string.IsNullOrEmpty(filterValue))
predicateGroup.Predicates.Add(Predicates.Field<MyPoco>(x => x.MyProperty, Operator.Eq, PredicateGroup));
Transaction is different thing. You can put all your current code in transaction. You can put my code in transaction as well. With my code, transaction does not make much difference though; although it is recommended to always use transactions.
About passing list of objects, I do not see any way. Following are two extension methods of Dapper Extensions for deleting the record:
public static bool Delete<T>(this IDbConnection connection, object predicate, IDbTransaction transaction = null, int? commandTimeout = default(int?)) where T : class;
public static bool Delete<T>(this IDbConnection connection, T entity, IDbTransaction transaction = null, int? commandTimeout = default(int?)) where T : class;
None of it accepts list of objects. One accept predicate and other accepts single object.
Related
In the Dapper documentation, it says you can use an IEnumerable parameter to execute a command multiple times. It gives the following example:
connection.Execute(#"insert MyTable(colA, colB) values (#a, #b)",
new[] { new { a=1, b=1 }, new { a=2, b=2 }, new { a=3, b=3 } }
).IsEqualTo(3); // 3 rows inserted: "1,1", "2,2" and "3,3"
Will this result in multiple round-trips to the database (i.e. one for each T in the IEnumerable<T>)? Or is Dapper smart enough to transform the multiple queries into a batch and just do one round-trip? The documentation says an example usage is batch loading, so I suspect it only does one round-trip, but I want to be sure before I use it for performance-sensitive code.
As a follow-up question, and depending on the answer to the first, I'd be curious how transactions are handled? That is, is there one transaction for the whole set of Ts, or one transaction per T?
I finally got around to looking at this again. Looking at the source code (in \Dapper\SqlMapper.cs), I found the following snippet in method ExecuteImpl:
// ...
foreach (var obj in multiExec)
{
if (isFirst)
{
masterSql = cmd.CommandText;
isFirst = false;
identity = new Identity(command.CommandText, cmd.CommandType, cnn, null, obj.GetType(), null);
info = GetCacheInfo(identity, obj, command.AddToCache);
}
else
{
cmd.CommandText = masterSql; // because we do magic replaces on "in" etc
cmd.Parameters.Clear(); // current code is Add-tastic
}
info.ParamReader(cmd, obj);
total += cmd.ExecuteNonQuery();
}
// ...
The interesting part is on the second-last line where ExecuteNonQuery is called. That method is being called on each iteration of the for loop, so I guess it is not being batched in the sense of a set-based operation. Therefore, multiple round-trips are required. However, it is being batched in the sense that all operations are performed on the same connection, and within the same transaction if so specified.
The only way I can think of to do a set-based operation is to create a custom table-valued type (in the database) for the object of interest. Then, in the .NET code pass a DataTable object containing matching names and types as a command parameter. If there were a way to do this without having to create a table-valued type for every object, I'd love to hear about it.
I developing web application with play framework 2.3.8 and scala, with complex architecture on backend and front-end side. As backend we use MS SQL, with many stored procedures, and called it by anorm. And here one of the problems.
I need to update some fields in database. The front end calls play framework, and recive name of the field, and value. Then I parse, field name, and then I need to generate SQL Query for update field. I need assign null, for all parameters, except recived parameter. I try to do it like that:
def updateCensusPaperXX(name: String, value: String, user: User) = {
DB.withConnection { implicit c =>
try {
var sqlstring = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}"
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
for (p <- params){
sqlstring += ", "
if (name.endsWith(p))
sqlstring += value
else
sqlstring += "null"
}
SQL(sqlstring)
.on(
"login" -> user.login,
"domain" -> user.domain,
).execute()
} catch {
case e: Throwable => Logger.error("update CensusPaper04 error", e)
}
}
}
But actually that doesn't work in all cases. For example, when I try to save string, it give's me an error like:
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 'some phrase'
What is the best way to generate sql query using anorm with all nulls except one?
The reason this is happening is because when you write the string value directly into the SQL statement, it needs to be quoted. One way to solve this would be to determine which of the fields are strings and add conditional logic to determine whether to quote the value. This is probably not the best way to go about it. As a general rule, you should be using named parameters rather than building a string to with the parameter values. This has a few of benefits:
It will probably be easier for you to diagnose issues because you will get more sensible error messages back at runtime.
It protects against the possibility of SQL injection.
You get the usual performance benefit of reusing the prepared statement although this might not amount to much in the case of stored procedure invocation.
What this means is that you should treat your list of fields as named parameters as you do with user and domain. This can be accomplished with some minor changes to your code above. First, you can build your SQL statement as follows:
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
val sqlString = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}," +
params.map("{" + _ + "}").mkString{","}
What is happening above is that you don't need to insert the values directly, so you can just build the string by adding the list of parameters to the end of your query string.
Then you can go ahead and start building your parameter list. Note, the parameters to the on method of SQL is a vararg list of NamedParameter. Basically, we need to create Seq of NamedParameters that covers "login", "domain" and the list of fields you are populating. Something like the following should work:
val userDomainParams: Seq[NamedParameter] = (("login",user.login),("domain",user.domain))
val additionalParams = params.map(p =>
if (name.endsWith(p))
NamedParameter(p, value)
else
NamedParameter(p, None)
).toSeq
val fullParams = userDomainParams ++ additionalParams
// At this point you can execute as follows
SQL(sqlString).on(fullParams:_*).execute()
What is happening here is that you building the list of parameters and then using the splat operator :_* to expand the sequence into the varargs needed as arguments for the on method. Note that the None used in the NamedParameter above is converted into a jdbc NULL by Anorm.
This takes care of the issue related to strings because you are no longer writing the string directly into the query and it has the added benefit eliminating other issues related with writing the SQL string rather than using parameters.
I'm working with objectify.
In my app, i have Employee entities and Task entities. Every task is created for a given employee. Datastore is generating id for Task entities.
But i have a restriction, i can't save a task for an employee if that task overlaps with another already existing task.
I don't know how to implement that. I don't even know how to start. Should i use a transaction ?
#Entity
class Task {
#Id
private Long id;
#Index
private long startDate; // since epoch
#Index
private long endDate;
#Index
private Ref<User> createdFor;
public Task(String id, long startDate, long endDate, User assignedTo) {
this.id = null;
this.startDate = startDate;
this.endDate = endDate;
this.createdFor = Ref.create(assignedTo);
}
}
#Entity
class Employee {
#Id
private String id;
private String name;
public Employee(String id, String name) {
this.id = id;
this.name = name;
}
}
You can't do it with the entities you've set up, because between the time you queried for tasks, and inserted the new task, you can't guarantee that someone wouldn't have already inserted a new task that conflicts with the one you're inserting. Even if you use a transaction, any concurrently added, conflicting tasks won't be part of your transaction's entity group, so there's the potential for violating your constraint.
Can you modify your architecture so instead of each task having a ref to the employee its created for, every Employee contains a collection of tasks created for that Employee? That way when your querying the Employee's tasks for conflicts, the Employee would be timestamped in your transaction's Entity Group, and if someone else put a new task into it before you finished putting your new task, a concurrent modification exception would be thrown and you would then retry. But yes, have both your query and your put in the same Transaction.
Read here about Transactions, Entity Groups and Optimistic Concurrency: https://code.google.com/p/objectify-appengine/wiki/Concepts#Transactions
As far as ensuring your tasks don't overlap, you just need to check whether either of your new task's start of end date is within the date range of any previous Tasks for the same employee. You also need to check that your not setting a new task that starts before and ends after a previous task's date range. I suggest using a composite.and filter for for each of the tests, and then combining those three composite filters in a composite.or filter, which will be the one you finally apply. There may be a more succint way, but this is how I figure it:
Note these filters would not apply in the new architecture I'm suggesting. Maybe I'll delete them.
////////Limit to tasks assigned to the same employee
Filter sameEmployeeTask =
new FilterPredicate("createdFor", FilterOperator.EQUAL, thisCreatedFor);
/////////Check if new startDate is in range of the prior task
Filter newTaskStartBeforePriorTaskEnd =
new FilterPredicate("endDate", FilterOperator.GREATER_THAN, thisStartDate);
Filter newTaskStartAfterPriorTaskStart =
new FilterPredicate("startDate", FilterOperator.LESS_THAN, thisStartDate);
Filter newTaskStartInPriorTaskRange =
CompositeFilterOperator.and(sameEmployeeTask, newTaskStartBeforePriorTaskEnd, newTaskStartAfterPriorTaskStart);
/////////Check if new endDate is in range of the prior task
Filter newTaskEndBeforePriorTaskEnd =
new FilterPredicate("endDate", FilterOperator.GREATER_THAN, thisEndDate);
Filter newTaskEndAfterPriorTaskStart =
new FilterPredicate("startDate", FilterOperator.LESS_THAN, thisEndDate);
Filter newTaskEndInPriorTaskRange =
CompositeFilterOperator.and(sameEmployeeTask, newTaskEndBeforePriorTaskEnd, newTaskEndAfterPriorTaskStart);
/////////Check if this Task overlaps the prior one on both sides
Filter newTaskStartBeforePriorTaskStart =
new FilterPredicate("startDate", FilterOperator.GREATER_THAN_OR_EQUAL, thisStartDate);
Filter newTaskEndAfterPriorTaskEnd =
new FilterPredicate("endDate", FilterOperator.LESS_THAN_OR_EQUAL, thisEndDate);
Filter PriorTaskRangeWithinNewTaskStartEnd = CompositeFilterOperator.and(sameEmployeeTask ,newTaskStartBeforePriorTaskStart, newTaskEndAfterPriorTaskEnd);
/////////Combine them to test if any of the three returns one or more tasks
Filter newTaskOverlapPriorTask = CompositeFilterOperator.or(newTaskStartInPriorTaskRange,newTaskEndInPriorTaskRange,PriorTaskRangeWithinNewTaskStartEnd);
/////////Proceed
Query q = new Query("Task").setFilter(newTaskOverlapPriorTask);
PreparedQuery pq = datastore.prepare(q);
If you don't return any results, then you don't have any overlaps, so go ahead and save the new task.
Ok, I hope I can be a bit more helpful. I've attempted to edit your question and change your entities to the right architecture. I've added an embedded collection of tasks and an attemptAdd method to your Employee. I've added a detectOverlap method to both your Task and your Employee. With these in place, your could use something like the transaction below. You will need to deal with the cases where you task doesn't get added because there's a conflicting task, and also the case where the add fails due to a ConcurrentModificationException. But you could make another question out of those, and you should have the start you need in the meantime.
Adapted from: https://code.google.com/p/objectify-appengine/wiki/Transactions
Task myTask = new Task(startDate,endDate,description);
public boolean assignTaskToEmployee(EmployeeId, myTask) {
ofy().transact(new VoidWork() {
public void vrun() {
Employee assignee = ofy().load().key(EmployeeId).now());
boolean taskAdded = assignee.attemptAdd(myTask);
ofy().save().entity(assignee);
return taskAdded;
}
}
}
I am creating a task-scheduler on SQL Server 2008.
I have a table that I use to store tasks. Each task is a task name (e.g. ImportFile) and arguments. I store arguments in XML column, since different tasks have different signatures.
Table is as follows:
Id:integer(PK) | operation:nvarchar | Arguments:xml
Before queuing a task, I often need to verify that given task hasn't been scheduled yet. The lookup is done based on both operation and args.
Question: Using Linq-to-Sql how can I check if given operation+args is present in the queue already?
I am looking for something like:
var isTaskScheduled = db.Tasks.Any(t =>
t.Opearation == task.Operation &&
t.Arguments == task.ArgumentsAsXElement);
(which doesn't work because SQL Server can't compare XML type)
Any alternative implementation suggestions?
You might want to surface e.g. a string property that encapsultes your Arguments, or maybe it would be sufficient to have e.g. the length and a CRC of your Arguments as extra properties on your class:
public partial class Task
{
public int ArgumentLength
{ .... }
public int ArgumentCRC
{ .... }
}
That way, if you can compare length (of your XML) and the CRC and they match, you can be pretty sure and safe to assume the two XML's are identical. Your check would then be something like:
var isTaskScheduled =
db.Tasks.Any(t => t.Operation == task.Operation &&
t.ArgumentLength == task.ArgumentLength &&
t.ArgumentCRC == task.ArgumentCRC);
or something like that.
This may be a stretch, but you could use a "Hashcode" when saving the data to the database, then query on the hashcode value at a later date / time.
This assumes that you have a class that represents your task entity and that you have overridden the GetHashCode method of said class.
Now, when you go to query the database to see if the task is in the scheduled queue, you simply query on the hashcode, thus avoiding the need to do any xml poking at query time.
var t1 = new Task{Operation="Run", Arguments="someXElement.value"};
var t2 = new Task{Operation="Run", Arguments="someXElement.value"};
in the code above t1 == t2 because you are overriding GetHashCode and computing the hash for Operation+Arguments.Value. if you store the hashcode in the db, then you can easily tell if you have an object in the DB that equals the hash code that you are checking for.
This may be similar to what marc_s was talking about.
You can write a class which implements IComparable:
public class XMLArgument : IComparable
{
public XMLArgument(string argument)
{
}
public int CompareTo(object obj)
{
...
}
}
var isTaskScheduled = db.Tasks.Any(t =>
t.Opearation == task.Operation &&
(new XMLArgument(t.Arguments)).CompareTo(new XMLArgument(task.ArgumentsAsXElement)) == 0);
This works:
var i = (from x in db.Test
where x.Id == 1
select x).First();
db.Test.DeleteOnSubmit(i);
db.SubmitChanges();
I get a cast error for this (int/string):
var i = db.Test.Single(x => x.Id == 1);
db.Test.DeleteOnSubmit(i);
db.SubmitChanges();
I was also able to make an update using Single sucesssfully on the same table. Any clues?
Update
Here's more of the error message:
Here's more:
[InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.String'.]
System.Data.Linq.SingleKeyManager2.TryCreateKeyFromValues(Object[] values, V& v) +134
System.Data.Linq.IdentityCache2.Find(Object[] keyValues) +57
System.Data.Linq.StandardIdentityManager.Find(MetaType type, Object[] keyValues) +51
System.Data.Linq.CommonDataServices.GetCachedObject(MetaType type, Object[] keyValues) +113
System.Data.Linq.ChangeProcessor.GetOtherItem(MetaAssociation assoc, Object instance) +235
System.Data.Linq.ChangeProcessor.BuildEdgeMaps() +510
System.Data.Linq.ChangeProcessor.SubmitChanges(ConflictMode failureMode) +137
System.Data.Linq.DataContext.SubmitChanges(ConflictMode failureMode) +453
System.Data.Linq.DataContext.SubmitChanges() +38
What's the data type of your id field?
I just ran the following code against a dummy database that has data in it where my key field is of type int:
using (var db = new dbDataContext())
{
var x = db.items.Single(i => i.fld_key == 1);
db.items.DeleteOnSubmit(x);
db.SubmitChanges();
var y = (from i in db.items
where i.fld_key == 2
select i).First();
db.items.DeleteOnSubmit(y);
db.SubmitChanges();
}
I had no problems with either, and checking the database, items 1 and 2 were both deleted successfully.
Are you attempting to delete an item that doesn't exist in the second code block - as you deleted it with the first. This would mean that y is actually null and you're attempting to DeleteOnSubmit passing null which will give you an invalid cast as you can't cast null.
Try attempting to delete a different item in your second code block, I would imagine it would work in that case.
Change .Single to .SingleOrDefault and wrap your delete lines to check for null:
if(null != x)
{
db.items.DeleteOnSubmit(x);
db.SubmitChanges();
}
I would wager this will fix the problem you're seeing.
Edit Looking at the exception you've posted, I'm wondering if you changed the data type of your table to int after you generated your dbml. Reload the table into your dbml and see if that fixes the issue. Either that, or the issue is the other way around and you changed it to VarChar/NVarChar and haven't updated your dbml.
I would definitely take a look at your entities and make sure that the data types match those in your data table in your underlying database.
Have you changed the datatypes in your database lately? (Maybe those types currently differ from your generated classes)
Similarly, if you're using a stored procedure for deleting items, make sure the parameter types match up to your generated types.