Since string.Join(...) is not translatable in the latest EF Core and a lot of our queries do use that method, I'm currently trying to implement it with user-defined functions. I can see that the function actually works in the ssms but I can't get it to work in c#.
The method needs to accept an array of strings as an input so I created a user-defined table type:
IF TYPE_ID('[system].[stringList]') IS NULL
BEGIN
CREATE TYPE system.stringList AS TABLE (val nvarchar(max))
END
GO
Then I created the function itself:
create function toCommaSeparatedString(#strings [system].[stringList] readonly)
returns nvarchar(max)
as
begin
DECLARE #values nvarchar(max)
SELECT #values = COALESCE(#values + ', ', '') + [val]
FROM #strings
return #values
end
I can verify that it works by simply executing the following sql in the ssms:
declare #input [system].[stringList]
insert into #input values ('1'), ('2'), ('3'), ('4')
select dbo.toCommaSeparatedString(#input)
In c# I declared the function on my DbContext (I tried both string[] and IEnumerable<string>):
[DbFunction("toCommaSeparatedString", "dbo")]
public static string ToCommaSeparatedString(string[] strings) => throw new NotImplementedException();
And used it:
...
var output = MyDbContext.ToCommaSeparatedString(new [] { "1", "2" });
...
but I'm getting an exception:
"The parameter 'strings' for the DbFunction 'BusinessUnitDbContext.ToCommaSeparatedString' has an invalid type 'string[]'. Ensure the parameter type can be mapped by the current provider.",
Is it possible to achieve what I'm trying to achieve here? Should I also configure that function on the DbContext?
EDIT:
Here is my custom projection that i want to use to create the viewmodel:
private static IQueryable<IView> ProjectToJobOverView(this IQueryable<Job> entities)
{
return entities.Select(j => new JobOverView
{
Id = j.EntityId,
Deleted = j.Deleted.HasValue,
Name = j.Name,
Status = j.Status.ToString(),
NumberOfInstructionSets = j.Tasks.Count(t => t.InstructionSet != null),
NumberOfCompletedInstructionSets = j.Tasks.Count(t => t.InstructionSet.IsCompleted),
NumberOfOrderLines = j.Tasks
.SelectMany(t => t.TaskOrderLines).Select(x => x.OrderLineId)
.Distinct()
.Count(),
Details = j.Tasks
.OfType<StockMovementTask>()
.Where(t => t.InstructionSet != null)
.Select(t => t.InstructionSet)
.Select(s => new JobOverviewDetail
{
Id = s.EntityId,
Deleted = s.Deleted.HasValue,
State = s.Tasks.First().OperationState.ToString(),
DeliveryMethod = BusinessUnitDbContext.ToCommaSeparatedString(
s.Tasks
.First()
.TaskOrderLines.Where(x => x.OrderLine.Order is OutgoingOrder && (x.OrderLine.Order as OutgoingOrder).DeliveryType != null)
.Select(x => (x.OrderLine.Order as OutgoingOrder).DeliveryType.Name).ToArray()),
}),
Categories = j.Tasks.Select(t => t.Process.ProcessDefinition.ProcessDefinitionGroup.ProcessDefinitionCategory.ToString()).Distinct().ToArray(),
OrderNumbers = j.Tasks.OfType<StockMovementTask>().SelectMany(t => t.TaskOrderLines).Select(j => j.OrderLine.Order.Number).Distinct().ToArray(),
});
}
Related
This CakePHP Query isn't using the conditional, $subQuery for some reason:
$subQuery = $this->queryFactory->newSelect('table_name')
->select(['id'])
->where(['id' => $id]);
$query = $this->queryFactory->newQuery()
->insert(
['id', 'machine', 'logfile', 'updated', 'time']
)
->into('table_name')
->values([
'id' => $id,
'machine' => $machine['id'],
'logfile' => $logFile,
'updated' => $updateDate,
'time' => $updateTime
])
->where(function (QueryExpression $exp) use ($subQuery) {
return $exp->notExists($subQuery);
});
$query->execute();
...it just inserts record even when it exists, but why?
The above code is only part of the required SQL that looks like this:
IF NOT EXISTS(
SELECT 1
FROM table_name
WHERE id = '$id'
)
INSERT INTO table_name (id, machine, logfile, updated, time)
VALUES (?,?,?,?,?)
ELSE
UPDATE table_name
SET updated = '$var1', time = ' $var2'
WHERE id = '$id';
There is no API that would allow to generate such a statement directly, the query builder isn't ment to generate (and execute) such SQL constructs, it can only compile SELECT, INSERT, UPDATE, and DELETE queries, and while the query expression builder can be used to stitch together arbitrary expressions, it will wrap itself and query objects into parentheses (as it is meant for use in query objects), which would be incompatible with what you're trying to build.
So if you want to run such constructs on SQL level, then you either have to write the SQL manually, or create custom expression classes that can build such constructs. In any case you would have to run the SQL manually then.
Here's a very basic quick & dirty example of such a custom expression class:
namespace App\Database\Expression;
use Cake\Database\ExpressionInterface;
use Cake\Database\ValueBinder;
use Closure;
class IfElseExpression implements ExpressionInterface
{
protected $_if;
protected $_then;
protected $_else;
public function if(ExpressionInterface $expression)
{
$this->_if = $expression;
return $this;
}
public function then(ExpressionInterface $expression)
{
$this->_then = $expression;
return $this;
}
public function else(ExpressionInterface $expression)
{
$this->_else = $expression;
return $this;
}
public function sql(ValueBinder $binder): string
{
$if = $this->_if->sql($binder);
$then = $this->_then->sql($binder);
$else = $this->_else->sql($binder);
return "IF $if $then ELSE $else";
}
public function traverse(Closure $callback)
{
$callback($this->_if);
$this->_if->traverse($callback);
$callback($this->_then);
$this->_then->traverse($callback);
$callback($this->_else);
$this->_else->traverse($callback);
return $this;
}
public function __clone()
{
$this->_if = clone $this->_if;
$this->_then = clone $this->_then;
$this->_else = clone $this->_else;
}
}
It could then be used something like this:
$notExists = (new \Cake\Database\Expression\QueryExpression())
->notExists($subQuery);
$insertQuery = $this->queryFactory->newQuery()
->insert(/* ... */)
//...
;
$updateQuery = $this->queryFactory->newQuery()
->update(/* ... */)
//...
;
$ifElse = (new \App\Database\Expression\IfElseExpression())
->if($notExists)
->then($insertQuery)
->else($updateQuery);
$binder = new \Cake\Database\ValueBinder();
$sql = $ifElse->sql($binder);
$statement = $connection->prepare($sql);
$binder->attachTo($statement);
$statement->execute();
See also
Cookbook > Database Access & ORM > Database Basics > Interacting with Statements
Yes, thanks. My own preference is to avoid the requirement to code the value binding explicitly. Using where(), I can do something like this:
$subQuery = $this->queryFactory->newSelect('table_name')
->select(['id'])
->where(['id' => $id])
->limit(1);
$find = $subQuery->execute()->fetchAll('assoc');
if (!empty($find)) {
$values = [
'id' => $id,
'machine' => $machine,
'logfile' => $logFile,
'updated' => $var1,
'time' => $var2
];
$query = $this->queryFactory->newInsert('table_name', $values);
} else {
$query = $this->queryFactory->newUpdate('table_name')
->set([
'updated' => $someVar,
'time' => $someVar2
])
->where(['id' => $id]);
}
$query->execute();
I have the following Entity Framework query:
var queryResponse = await db.DataPoints.GroupBy(x => x.Device).Select(x => new
{
lon = x.Key.DataPoints.OrderByDescending(y => y.DateTime).Select(y => y.Longitude).FirstOrDefault(),
lat = x.Key.DataPoints.OrderByDescending(y => y.DateTime).Select(y => y.Longitude).FirstOrDefault(),
date = x.Key.DataPoints.OrderByDescending(y => y.DateTime).Select(y => y.DateTime).FirstOrDefault(),
label = x.Key.Name,
})
.OrderByDescending(x => x.label)
.ToListAsync();
Now you can see in my select, I have to get lon,'lat', and 'date'. However the way im doing it, I have to orderby and select the first one 3 times. The 'DataPoints' is a very large table.
in C# i would normally do the orderBy once and just select the entire object, and then later on break it up into the 3 properties. However in this case I want SQL to return the exact fields.
is there a more efective way to write this query?
Try this:
var queryResponse =
(from g in db.DataPoints.GroupBy(x => x.Device)
let latestDataPoint = g.Key.DataPoints.OrderByDescending(p => p.DateTime)
.FirstOrDefault()
select new
{
lon = latestDataPoint.Longitude,
lat = latestDataPoint.Latitude,
date = latestDataPoint.DateTime,
label = g.Key.Name
})
.OrderByDescending(x => x.label)
.ToList();
I am using node-mssql
My query file is as below
BEGIN TRANSACTION
DECLARE #status NVARCHAR(30);
SET #status = 'create';
DECLARE #i UNIQUEIDENTIFIER;
SET #i = NEWID();
DECLARE #t DATETIME2;
SET #t = SYSUTCDATETIME();
IF NOT EXISTS(
SELECT * FROM user WHERE email = #email AND company_id= #company_id
) BEGIN
SET #i = NEWID();
INSERT INTO user (comapny_id, id, email, password) VALUES ( #company_id, #i, #email, #password);
INSERT INTO user_transaction( id, date, type) VALUES ( #i, #t, #status);
SELECT #i as 'id', #email as 'email';
END ELSE BEGIN
SELECT NULL as 'id', #email as 'email';
END
COMMIT TRANSACTION
And my createuserquery in query.js file is
datastore.getQueryFromSqlFile('create_user', (err: any, query: string) => {
if (err) {
done(err);
} else {
var request = new sql.Request(connectionOrTransaction);
request.input('email', sql.NVarChar(200), email);
request.input('password', sql.NVarChar(200), some_password);
request.input('company_id', sql.UniqueIdentifier, company_id);
request.query(query, function (err, data) {});
Now I need to modify these to insert bulk of user data imported from CSV file (>20000 entries)
I was thinking of doing something like
async.mapSeries(Object.keys(users), function (item, callback) {
query.createuser(email, company_id, function (err, data) {
callback(err, err ? 'Error message: ' + data : data);
});
}, function (err, results) {
})
But this is not efficient as I get connection timeout. Increasing connectionTimeout or requestTimeout in config file doesn't help much.
How could I make my query faster for bulk insert around 20000-40000 entries in per attempt?
For me it looks like a job for a prepared statement.
var ps = new sql.PreparedStatement();
ps.input('email', sql.VarChar);
ps.input('password', sql.VarChar);
ps.input('company_id', sql.Int);
ps.prepare(" ... your sql ... ", function(err) {
// ... error checks
// users must be an array of users
async.mapSeries(users, function(user, next) {
ps.execute({email: user.email, password: user.password, company_id: user.company_id}, next);
}, function(err) {
// ... error checks
ps.unprepare(function(err) {
// ... error checks
// done !
});
});
});
Every execute is called as a single request, so you should not be timeouted by requestTimeout. connectionTimeout is something that only affect connecting phase. Once you're connected to db, only requestTimeout matters.
I have a query which must sum the values from several tables and add the result. The system is simply an inventory system and I'm trying to get the stock level by calculating incomings (deliveries), outgoings (issues) and adjustments to items.
As the stock level is a calculated value (sum(deliveries) - sum(issues)) + sum(adjustments) I am trying to create a function that will get this value with a minimal number of queries.
At current I have linq that performs three separate queries to get each summed value and then perform the addition/subtraction in my function, however I am convinced there must be a better way to calculate the value without having to do three separate queries.
The current function is as follows:
public static int GetStockLevel(int itemId)
{
using (var db = EntityModel.Create())
{
var issueItemStock = db.IssueItems.Where(x => x.ItemId == itemId).Sum(x => x.QuantityFulfilled);
var deliveryItemStock = db.DeliveryItems.Where(x => x.ItemId == itemId).Sum(x => x.Quantity);
var adjustmentsStock = db.Adjustments.Where(x => x.ItemId == itemId).Sum(x => x.Quantity);
return (deliveryItemStock - issueItemStock) + adjustmentsStock;
}
}
In my mind the SQL query is quite simple, so I have considered a stored procedure, however I think there must be a way to do this with linq.
Many thanks
Edit: Answer
Taking the code from Ocelot20's answer, with a slight change. Each of the lets can return a null, and if it does then linq throws an exception. Using the DefaultIfEmpty command will negate this, and return a 0 for the final calculation. The actual code I have used is as follows:
from ii in db.Items
let issueItems = db.IssueItems.Where(x => x.ItemId == itemId).Select(t => t.QuantityFulfilled).DefaultIfEmpty(0).Sum()
let deliveryItemStock = db.DeliveryItems.Where(x => x.ItemId == itemId).Select(t => t.Quantity).DefaultIfEmpty(0).Sum()
let adjustmentsStock = db.Adjustments.Where(x => x.ItemId == itemId).Select(t => t.Quantity).DefaultIfEmpty(0).Sum()
select (deliveryItemStock - issueItems) + adjustmentsStock);
Without knowing what your entities look like, you could do something like this:
public static int GetStockLevel(int itemId)
{
using (var db = EntityModel.Create())
{
// Note: Won't work if there are no IssueItems found.
return (from ii in db.IssueItems
let issueItems = db.IssueItems.Where(x => x.ItemId == itemId)
.Sum(x => x.QuantityFulfilled)
let deliveryItemStock = db.DeliveryItems.Where(x => x.ItemId == itemId)
.Sum(x => x.Quantity)
let adjustmentsStock = db.Adjustments.Where(x => x.ItemId == itemId)
.Sum(x => x.Quantity)
select issueItems + deliveryItemStock + adjustmentsStock).FirstOrDefault() ?? 0;
}
}
I tested a similar query on my own db and it worked in a single query. I suspect that since they all have a common ItemId, that using entity relations could make this look something like:
// Ideal solution:
(from i in db.Items
where i.Id == itemId
let issueItems = i.IssueItems.Sum(x => x.QuantityFulfilled)
let deliveryItemStock = i.DeliveryItems.Sum(x => x.Quantity)
let adjustmentsStock = i.Adjustments.Sum(x => x.Quantity)
select issueItems + deliveryItemStock + adjustmentsStock).SingleOrDefault() ?? 0;
Have you considered adding a view to the database that performs the calculations that you can then just use a simple select query (or SP) to return the values that you need?
I reckon this should work and the SQL generated is not particularly complex. If you think there is something wrong with it let me know and I will update my answer.
public static int GetStockLevel(int itemId)
{
using (var db = EntityModel.Create())
{
return db.IssueItems.Where(x => x.ItemId == itemId).GroupBy(x => x.ItemId)
.GroupJoin(db.DeliveryItems, x => x.First().ItemId, y => y.ItemId, (x, y) => new
{ Issues = x, Deliveries = y})
.GroupJoin(db.Adjustments, x=> x.Issues.First().ItemId, y=> y.ItemId, (x, y) => new
{
IssuesSum = x.Issues.Sum(i => i.QuantityFullfilled),
DeliveriesSum = x.Deliveries.Sum(d => d.Quantity),
AdjustmentsSum = y.Sum(a => a.Quantity)})
.Select(x => x.IssuesSum - x.DeliverysSum + x.AdjustmentsSum);
}
}
Given the following tables:
InstrumentLogs
InstrumentLogId (PK, int, not null)
InstrumentId (FK, int, not null)
LogDate (datetime, not null)
Action (string, not null)
Instruments
InstrumentId (PK, int, not null)
CountyId (FK, int, not null)
Counties
CountyId (PK, int, not null)
StateFips (FK, int, not null)
Name (string, not null)
States
StateFips (PK, int, not null)
Name (string, not null)
I need to figure out how to write this SQL query using Entity Framework:
select s.StateName, count(*) as total
from instruments as i
join counties as c on i.CountyID = c.CountyID
join states as s on s.StateFIPS = c.StateFIPS
where i.InstrumentID in
(select i1.InstrumentId from InstrumentLogs as i1 where i1.action = 'review' and
i1.logdate in (select max(logdate) from instrumentlogs as i2 where i1.instrumentid
=i2.InstrumentID group by i2.instrumentid))
group by s.StateName
I tried something along the lines of:
_context.Instruments.Include(i => i.County.State)
.Where(i => _context.Logs.Where(l => l.Action == 'review'
&& _context.Logs.Where(l2 => l2.InstrumentId == l.InstrumentId).Max(l2 => l2.LogDate) == l.LogDate).GroupBy(i => i.County.State.Name)
.Select(g => new { State = g.Key.Name, Total = g.Count() });
However, EF doesn't like this. I wind up with the error stating that only primitive types or enumeration types are supported.
Thanks for your help.
I finally got it to work like this
var _query = _dataContext.IndexLogs.GroupBy(l => l.InstrumentId, l => l)
.Select(grp => new { Id = grp.Key, Date = grp.Max(g => g.ActionDate) });
//Prequery to find log records that match given type and action type
var _indexes = _dataContext.IndexLogs.Where(l => l.Action == type
&& _query.Contains(new { Id = l.InstrumentId, Date = l.ActionDate})).Select(i => i.InstrumentId);
var _states = _dataContext.AdvancedIndexes
.Include(i => i.County.State)
.Where(a => a.Id > 0 && _indexes.Contains(a.Id))
.GroupBy(i => new { i.County.State.Id, i.County.State.Name })
.Select(g => new { fips = g.Key.Id, name = g.Key.Name, count = g.Count() });