I am doing bulk insert into a SQL table using Dapper plus. Here is the code
DapperPlusManager.Entity<HardwareComponentType>("WithInsertIfNotExists").Table("dbo.Components")
.UseBulkOptions(options => {
options.InsertIfNotExists = true;
}).Identity(x=>x.ComponentId).Map("Name", "Name").Map("Status", "Status").MapValue("getDate()", "ModifiedDateTime");
As you can see, for the column, "modifiedDateTime, I am trying to pass "getDate", but I am unable to do so. I am getting exception. Basically, it is taking that as a string.
How do I pass getDate() as parameter in this syntax, that is my question.
The Map method can take any value, such as:
.Map(x => DateTime.Now, "ModifiedDateTime")
That is important to use the x => part, as it means the DateTime.Now will be evaluated for every entity and every time this method is called.
The MapValue method should only be used if the ModifiedDateTime is a constant. For example, if you are creating a custom DapperContext only for this BulkInsert, so all entities will have the same value. But in your case, you look using the GlobalContextMapping, so the Map(x => method is the one you are looking for.
Related
I need to project some fields from jsonb column where few of them are optional
I'm using EF Core 3.1 and npgsl and so far I got this
var shipments = _dbContext.Shipments.Select(x => new
{
ShipmentNo= x.ShipmentNumber,
ReportNum = x.ShipmentData.RootElement.GetProperty("reportNumber"),
ShipmentValue= x.ShipmentData.RootElement.GetProperty("shipmentMetadata").GetProperty("value").GetString(),
}
However value is optional and this is throwing exception. I see .TryGetProperty(...) method but it requires output variable and, I presume, its evaluation on server side. I wonder if there is way to handle this so query runs completely in Postgres.
You've forgotten to add GetInt32 (or whatever the type is) for reportNumber, just like you have a GetString after the shipmentMetadata. In order for this to be translatable to SQL, you need to tell the provider which type you expect to come out of the JSON element.
Given the following code:
var dbRecords = _context.Alerts.AsNoTracking()
.Where(a => a.OrganizationId == _authorization.OrganizationId)
.ToList();
var dbRecords2 = _context.Alerts
.Where(a => a.OrganizationId == _authorization.OrganizationId)
.ToList();
foreach (var untrackedRecord in dbRecords) {
var trackedRecord = dbRecords2.First(a => a.Id == untrackedRecord.Id);
Assert.AreEqual(untrackedRecord.TimeStamp.Ticks, trackedRecord.TimeStamp.Ticks);
}
Where the TimeStamp data is stored in SQL Server 2012 in a column defined as datetime2(0).
The Assert fails, and the debugger demonstrates that the two Ticks values are always different.
Expected: 636179928520000000 But was: 636179928523681935
The untracked value will always be rounded off to the nearest second (which is expected, based on what SQL is storing). When creating the record, the value I'm saving comes from DateTime.Now.
Testing some more, this doesn't appear to be true (the inconsistent ticks) for every object I'm testing, only for records I've inserted recently. Looking at the code and given the way the column is defined, it's not obvious to my why that would matter.
For now, to get my tests to pass, I'm just comparing the DateTime values down to the second, which is all that's required. However, I'm just wanting to understand why this is happening: Why can I not reliably compare two DateTime values depending on whether or not the entities are being tracked?
I figured this out, so answering my own question; I found I left off what turns out to be a key piece of information here. I mentioned that this issue came up in testing. What I didn't mention is that we're inserting the records and then testing all within a single transaction, and within a single DbContext.
Because I use the same DbContext for all work, the Alert objects that are inserted for testing are cached. When I query the objects using AsNoTracking, the DbContext has to refresh the objects before giving them back to me (since their current state isn't being tracked, and therefore is unknown to EF), apparently without updating what's in the cache (since we told EF we don't want to track the objects).
Querying for the same objects without AsNoTracking results in a cache hit; those objects that were inserted are still in the cache, so the cached versions are returned.
Given that, it's clear why the Ticks aren't matching up. The non-cached objects are pulling the DateTime values from the database, where the precision is defined to only store the time down the nearest second. The cached objects have the original DateTime.Now values, which stores the time down to ms. This explains why the Ticks don't match between the two DateTimes, even though both objects represent the same underlying database record.
I'm using ExecuteStoreCommand to facilitate bulk deletes.
This Works
// ExecuteStoreCommand syntax via http://stackoverflow.com/a/13024320/740639
// Delete all of today's events
DateTime eventDate = DateTime.Now;
string sql = "DELETE FROM EventCalendar WHERE EventDate = {0}";
context.ObjectContext.ExecuteStoreCommand(sql, new object[] { eventDate });
I want to do this, but it does not work
I get an ArgumentException: "No mapping exists from object type System.DateTime[] to a known managed provider native type."
// Delete all of the events for the past 3 days
DateTime[] eventDates = new DateTime[] {
DateTime.Now.AddDay(-1),
DateTime.Now.AddDay(-2),
DateTime.Now.AddDay(-3)
};
string sql = "DELETE FROM EventCalendar WHERE EventDate IN ({0})";
context.ObjectContext.ExecuteStoreCommand(sql, new object[] { eventDates });
Proposed Solution
I could convert my DateTime[] into the string "'2013-06-24', '2013-06-23', '2013-06-22'" and pass that as the parameter, but that seems like a hack.
I know there are libraries that add bulk delete support to EF, but I don't want to go that route at this time.
Any suggestions?
If you manipulate the string, every time you change the dates you will get a slightly different SQL statement. It may look the same, but the string is different and will have a different hash. So this subverts caching of the execution plan and kills performance.
You can't really pass parameters in an IN clause the way you're thinking. There are some other ideas in this post that might work, but if you read the notes they all say they have lousy performance.
If you can be assured that you will have a fixed number of inputs, you could use the technique of passing them as separate parameters:
... WHERE EventDate IN (#dt1, #dt2, #dt3)
Otherwise, I would go with an approach like Tom suggested, pass them as a delimited string and parse it on the server.
Also - since you are doing an exact match, you probably want to snap to midnight using DateTime.Today rather than DateTime.Now, or use the .Date property of an existing DateTime.
Here's another suggestion, if you really are just trying to delete for the last three days, why not do this?
string sql = "DELETE FROM EventCalendar WHERE EventDate BETWEEN {0} AND {1}";
context.ObjectContext.ExecuteStoreCommand(sql,
new object[] { DateTime.Today.AddDays(-3), DateTime.Today.AddDays(-1) });
Or with a bit of thought, you could probably just use sql getdate() and similar functions instead of passing parameters at all.
You could combine the dates with a delimiter on the .net side, pass it through as a paramter, and then use a Table-Valued function to split a delimited string on the server side.
Pass in something like "2013-06-24|2013-06-23|2013-06-22"
http://ole.michelsen.dk/blog/split-string-to-table-using-transact-sql/
Lets say I have a model, Article that has a large amount of columns and the database contains more than 100,000 rows. If I do something like var articles = db.Articles.ToList() it is retrieving the entire article model for each article in the database and holding it in memory right?
So if I am populating a table that only shows the date of the entry and it's title is there a way to only retrieve just these columns from the database using the entity framework, and would it be more efficient?
According to this,
There is a cost required to track returned objects in the object
context. Detecting changes to objects and ensuring that multiple
requests for the same logical entity return the same object instance
requires that objects be attached to an ObjectContext instance. If you
do not plan to make updates or deletes to objects and do not require
identity management , consider using the NoTracking merge options when
you execute queries.
it looks like I should use NoTracking since the data isn't being changed or deleted, only displayed. So my query now becomes var articles = db.Articles.AsNoTracking().ToList(). Are there other things I should do to make this more efficient?
Another question I have is that according to this answer, using .Contains(...) will cause a large performance drop when dealing with a large database. What is the recommended method to use to search through the entries in a large database?
It's called a projection and just translates into a SELECT column1, column2, ... in SQL:
var result = db.Articles
.Select(a => new
{
Date = a.Date,
Title = a.Title
})
.ToList();
Instead of a => new { ... } (creates a list of "anonymous" objects) you can also use a named helper class (or "view model"): a => new MyViewModel { ... } that contains only the selected properties (but you can't use a => new Article { ... } as an entity itself).
For such a projection you don't need AsNoTracking() because projected data are not tracked anyway, only full entity objects are tracked.
Instead of using Contains the more common way is to use Where like:
var date = DateTime.Now.AddYears(-1);
var result = db.Articles
.Where(a => date <= a.Date)
.Select(a => new
{
Date = a.Date,
Title = a.Title
})
.ToList();
This would select only the articles that are not older than a year. The Where is just translated into a SQL WHERE statement and the filter is performed in the database (which is as fast as the SQL query is, depending on table size and proper indexing, etc.). Only the result of this filter is loaded into memory.
Edit
Refering to your comment below:
Don't confuse IEnumerable<T>.Contains(T t) with string.Contains(string subString). The answer you have linked in your question talks about the first version of Contains. If you want to search for articles that have the string "keyword" in the text body you need the second Contains version:
string keyword = "Entity Framework";
var result = db.Articles
.Where(a => a.Body.Contains(keyword))
.Select(a => new
{
Date = a.Date,
Title = a.Title
})
.ToList();
This will translate into something like WHERE Body like N'%Entity Framework%' in SQL. The answer about the poor performance of Contains doesn't apply to this version of Contains at all.
I need to find the highest value from the database that satisfies a certain formatting convention. Specifically, I would like to find the highest value that looks like
EU999999 ('9' being any digit)
select max(col) will return something like 'EUZ...' for instance that I want to exclude.
The following query does the trick, but I can't produce this via Linq-to-SQL. There seems to be no translation for the isnumeric() function in SQL Server.
select max(col) from table where col like 'EU%'
and 1=isnumeric(replace(col, 'EU', ''))
Writing a database function, stored procedure, or anything else of that nature is far down the list of my preferred solutions, because this table is central to my app and I cannot easily replace the table object with something else.
What's the next-best solution?
Although ISNUMERIC is missing, you could always try the nearly equivalent NOT LIKE '%[^0-9]%, i.e., there is no non-digit in the string, or alternatively, the string is empty or consists only of digits:
from x in table
where SqlMethods.Like(x.col, 'EU[0-9]%') // starts with EU and at least one digit
&& !SqlMethods.Like(x.col, '__%[^0-9]%') // and no non-digits
select x;
Of course, if you know that the number of digits is fixed, this can be simplified to
from x in table
where SqlMethods.Like(x.col, 'EU[0-9][0-9][0-9][0-9][0-9][0-9]')
select x;
You could make use of the ISNUMERIC function by adding a method to a partial class for the DataContext. It would be similar to using a UDF.
In your DataContext's partial class add this:
partial class MyDataContext
{
[Function(Name = "ISNUMERIC", IsComposable = true)]
public int IsNumeric(string input)
{
throw new NotImplementedException(); // this won't get called
}
}
Then your code would use it in this manner:
var query = dc.TableName
.Select(p => new { p.Col, ReplacedText = p.Col.Replace("EU", "") })
.Where(p => SqlMethods.Like(p.Col, "EU%")
&& dc.IsNumeric(p.ReplacedText) == 1)
.OrderByDescending(p => p.ReplacedText)
.First()
.Col;
Console.WriteLine(query);
Or you could make use MAX:
var query = dc.TableName
.Select(p => new { p.Col, ReplacedText = p.Col.Replace("EU", "") })
.Where(p => SqlMethods.Like(p.Col, "EU%")
&& dc.IsNumeric(p.ReplacedText) == 1);
var result = query.Where(p => p.ReplacedText == query.Max(p => p.ReplacedText))
.First()
.Col;
Console.WriteLine("Max: {0}, Result: {1}", max, result);
Depending on your final goal it might be possible to stop at the max variable and prepend it with the "EU" text to avoid the 2nd query that gets the column name.
EDIT: as mentioned in the comments, the shortcoming of this approach is that ordering is done on text rather than numeric values and there's currently no translation for Int32.Parse on SQL.
As you said, there is no translation for IsNumeric from LINQ to SQL. There are a few options, you already wrote database function and stored procedure down. I like to add two more.
Option 1: You can do this by mixing LINQ to SQL with LINQ to Objects, but when you've got a big database, don't expect great performance:
var cols = (from c in db.Table where c.StartsWith("EU") select c).ToList();
var stripped = from c in cols select int.Parse(c.Replace("EU", ""));
var max = stripped.Max();
Option 2: change your database schema :-)
My suggestion is to fall back to in-line SQL and use the DataContext.ExecuteQuery() method. You would use the SQL query you posted in the beginning.
This is what I have done in similar situations. Not ideal, granted, due to the lack of the type-checking and possible syntax errors, but simply make sure it is included in any unit tests. Not every possible query is covered by the Linq syntax, hence the existence of ExecuteQuery in the first place.