I have an existing SQL Server 2008 database which has a number of views, stored procedures and functions.
I want to be able to SELECT data from one of these SQL functions and limit the number of rows that it returns in a paging scenario.
I have tried using .Select with .Skip and .Take as follows:
public IEnumerable<Product> CallSqlFunction_dbo_Search_Products_View(int clientId,
string environmentCode,
int sessionId)
{
IEnumerable<Product> results;
using (var db = _dbConnectionFactory.Open())
{
results = db.Select<Product>(#"
SELECT
*
FROM
[dbo].[Search_Products_View]
(
#pClientID,
#pEnvironmentCode,
#pSessionId
)", new
{
pClientID = clientId,
pEnvironmentCode = environmentCode,
pSessionId = sessionId
})
.Skip(0)
.Take(1000);
db.Close();
}
return results;
}
This produces the following SQL which is executed on the SQL Server.
exec sp_executesql N'
SELECT
*
FROM
[dbo].[Search_Products_View]
(
#pClientID,
#pEnvironmentCode,
#pSessionId
)',N'#pClientID int,#pEnvironmentCode varchar(8000),#pSessionId int',#pClientID=0,#pEnvironmentCode='LIVE',#pSessionId=12345
It means that this query returns 134,000 products, not the first page of 1000 I was expecting. The paging happens on the API server once the SQL Server has returned 134,000 rows.
Is it possible to use ORMLite so that I can get it to generate the paging in the query similar to this:
exec sp_executesql N'
SELECT
[t1].*
FROM (
SELECT
ROW_NUMBER() OVER (ORDER BY [t0].[ProductId], [t0].[ProductName])
FROM
[dbo].[Search_Products_View](#pClientId, #pEnvironmentCode, #pSessionId) AS [t0]
WHERE
(LOWER([t0].[ProductStatus]) = #pProductStatus1) OR (LOWER([t0].[ProductStatus]) = #pProductStatus2) OR (LOWER([t0].[ProductStatus]) = #pProductStatus3)
) AS [t1]
WHERE
[t1].[ROW_NUMBER] BETWEEN #pPageNumber + 1 AND #pPageNumber + #pNumberOfRowsPerPage
ORDER BY [t1].[ROW_NUMBER]',
N'#pClientId decimal(9,0),#pEnvironmentCode char(3),#pSessionId decimal(9,0),#pProductStatus1 varchar(8000),#pProductStatus2 varchar(8000),#pProductStatus3 varchar(8000),#pPageNumber int,#pNumberOfRowsPerPage int',
#pClientId=0,#pEnvironmentCode='LIVE',#pSessionId=12345,#pProductStatus1='1',#pProductStatus2='2',#pProductStatus3='3',#pPageNumber=0,#pNumberOfRowsPerPage=1000
OrmLite will use the windowing function hack in <= SQL Server 2008 for its typed queries but not for Custom SQL. You'll need to include the entire SQL (inc. Windowing function) into your Custom SQL.
If you do this a lot I'd suggest wrapping the Windowing Function SQL Template in an extension method so you can easily make use of it in your custom queries, e.g:
results = db.Select<Product>(#"
SELECT
*
FROM
[dbo].[Search_Products_View]
(
#pClientID,
#pEnvironmentCode,
#pSessionId
)"
.InWindowingPage(0,1000), new
{
pClientID = clientId,
pEnvironmentCode = environmentCode,
pSessionId = sessionId
})
If you want to use DB Params for the offsets you'll need some coupling to use conventional param names:
results = db.Select<Product>(#"
SELECT
*
FROM
[dbo].[Search_Products_View]
(
#pClientID,
#pEnvironmentCode,
#pSessionId
)"
.InWindowingPage(), new
{
pClientID = clientId,
pEnvironmentCode = environmentCode,
pSessionId = sessionId,
pPageNumber = 0,
pNumberOfRowsPerPage = 100
})
Related
I'm trying to do a "database side" bulk copy (i.e. SELECT INTO/INSERT INTO) using linq2db. However, my code is trying to bring the dataset over the wire which is not possible given the size of the DB in question.
My code looks like this:
using (var db = new MyDb()) {
var list = db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
});
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.BulkCopy(new BulkCopyOptions {
BulkCopyType = BulkCopyType.MultipleRows
}, list);
}
The generated SQL looks like this:
BeforeExecute
-- DBNAME SqlServer.2017
TRUNCATE TABLE DESTINATION_TABLE
DataConnection
Query Execution Time (AfterExecute): 00:00:00.0361209. Records Affected: -1.
DataConnection
BeforeExecute
-- DBNAME SqlServer.2017
DECLARE #take Int -- Int32
SET #take = 1
DECLARE #take_1 Int -- Int32
SET #take_1 = 1
DECLARE #take_2 Int -- Int32
...
SELECT
(
SELECT TOP (#take)
[p].[YEAR]
FROM
[dbo].[SOURCE_TABLE] [p]
WHERE
(([p_16].[YEAR] = [p].[YEAR] OR [p_16].[YEAR] IS NULL AND [p].[YEAR] IS NULL) AND ...
...)
FROM SOURCE_TABLE p_16
WHERE p_16.YEAR > 2012
GROUP BY
...
DataConnection
That is all that is logged as the bulkcopy fails with a timeout, i.e. SqlException "Execution Timeout Expired".
Please note that running this query as an INSERT INTO statement takes less than 1 second directly in the DB.
PS: Anyone have any recommendations as to good code based ETL tools to do large DB (+ 1 TB) ETL. Given the DB size I need things to run in the database and not bring data over the wire. I've tried pyspark, python bonobo, c# etlbox and they all move too much data around. I thought linq2db had potential, i.e. basically just act like a C# to SQL transpiler but it is also trying to move data around.
I would suggest to rewrite your query because group by can not return first element. Also Truncate is a part of the library.
var sourceQuery =
from s in db.SourceTable
where s.Year > 2012
select new
{
Source = s,
Count = Sql.Ext.Count(s.Column3 == 'Y' ? 1 : null).Over()
.PartitionBy(s.Column1, s.Column2).ToValue()
RN = Sql.Ext.RowNumber().Over()
.PartitionBy(s.Column1, s.Column2).OrderByDesc(s.Year).ToValue()
};
db.DestinationTable.Truncate();
sourceQuery.Where(s => s.RN == 1)
.Insert(db.DestinationTable,
e => new DestinationTable
{
Property1 = 'Constant Value',
Property2 = e.Source.Column1,
Property3 = e.Source.Column2,
Property4 = e.Count
});
After some investigation I stumbled onto this issue. Which lead me to the solution. The code above needs to change to:
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
}).Insert(db.DestinationTable, e => e);
Documentation of the linq2db project leaves a bit to be desired however, in terms of functionality its looking like a great project for ETLs (without horrible 1000s of line copy/paste sql/ssis scripts).
I'm trying to convert this SQL query into a linq query but not having much luck.
SELECT DISTINCT gen.ID
, gen.Name
, Ssec.System
FROM dbo.Generic Gen
JOIN dbo.SystemsSelected SSel
ON Gen.RecordID = SSel.RecordID
JOIN dbo.Security SSec
ON (
SSel.SystemA = CASE WHEN Ssec.System = '1stSystem' THEN 1 ELSE NULL END
OR SSel.SystemB = CASE WHEN Ssec.System = '2ndSystem' THEN 1 ELSE NULL END
)
and SSec.Username = 'myUserName'
I've had a look at the following posts but cannot apply the examples to my code:
Link join with case condition
Linq query with table joins
Any help would be appreciated!
Thanks
I cannot answer my own question directly but I found a workaround.
I used linq to instead call a Stored Procedure and that worked fine.
To do this:
1) Save code as a Stored Procedure that accepts a parameter named UserName
2) add the SP into Entity Framework Model
3) call the SP using:
public JsonResult GetNewTeams(string userUserName)
{
using (YourDBNameEntities db = new YourDBNameEntities ())
{
var ret = db.SP_YourStoredProcedureName(userUserName).ToList();
return Json(ret, JsonRequestBehavior.AllowGet);
}
}
Thats it, done, no more scary linq!! :)
sauce: Using stored procedures in LINQ
Wanting to pass a single codeigniter query result to my controller using json.
my model function is:
function get_linked_loads_qty($q){
$sql = $this->db->query("
IF EXISTS(
select qtylinkedorders
from Linked_Order_Summary
join Linked_Order_lines
on Linked_Order_Summary.linkedorderid= Linked_Order_lines.linked_order_id
where load_number='$q'
)
begin
select qtylinkedorders
from Linked_Order_Summary
join Linked_Order_lines
on Linked_Order_Summary.linkedorderid= Linked_Order_lines.linked_order_id
where load_number='$q'
END
ELSE BEGIN
select 1 as qtylinkedorders
from customer_Order_lines
where CustomerOrderID='$q'
group by CustomerOrderID
END
");
$query = $this->db->get();
if($query->num_rows > 0){
foreach ($query->result_array() as $row){
$row_set[] = htmlentities(stripslashes($row[qtylinkedorders])); //build an array
}
return $row_set;
}
}
and my controller is:
function get_linked_loads_qty(){
$this->load->model('Sales_model');
if (isset($_POST['data'])){
$q = strtolower($_POST['data']);
$data = $this->Sales_model->get_linked_loads_qty($q);
$this->output->set_content_type('application/json')->set_output(json_encode($data));
}
}
The sql works 100% in mssql. the codeigniter error is:
<p>Error Number: 42000</p><p>[Microsoft][SQL Server Native Client 11.0][SQL Server]Must specify table to select from.</p><p>SELECT *</p>
as mentioned the query executes perfectly in mssql.
any assistance welcome. Thanks.
I believe the $this->db->get() is not needed, and is what causes the error. The $this->db->query("blah...") will run the query and return the result. See CodeIgniter DB Query Docs
I am looking for something to extract all the SQL queries present in my SSIS/DTSX package. But nothing is helping me till now.
I already had a look at Microsoft.SqlServer.DTS API's from Microsoft. But they are extracting some queries straight forward. But the queries that are present in DTS:variable TAG, they are not extracted.
I want something in .Net framework. As i need to use the output to perform some other task as well. I am using C#.
Sample code as follows. Does not address all the situations
// this function takes the list of task hosts as input
// and gives all the queries present in taskhosts.
public static string ExtractQueriesFromTasks(List<TaskHost> Tasks)
{
string src_query = "";
foreach (TaskHost executable in Tasks)
{
DtsContainer Seq_container = (DtsContainer)executable;
if (executable.InnerObject.GetType().Name == "ExecuteSQLTask")
{
ExecuteSQLTask sqlTask = (ExecuteSQLTask)executable.InnerObject;
string src_query2 = sqlTask.SqlStatementSource;
src_query = src_query + "\n" + src_query2.ToUpper();
}
if (executable.InnerObject.GetType().Name == "__ComObject")
{
IDTSPipeline100 sqlTask = (IDTSPipeline100)executable.InnerObject;
Console.WriteLine(Microsoft.VisualBasic.Information.TypeName(executable.InnerObject));
//ExecuteSQLTask sqlTask = (ExecuteSQLTask)executable.InnerObject;
//string src_query2 = sqlTask.SqlStatementSource;
//src_query = src_query + "\n" + src_query2.ToUpper();
}
if (executable.InnerObject.GetType().Name == "ScriptTask")
{
ExecuteSQLTask sqlTask = (ExecuteSQLTask)executable.InnerObject;
string src_query2 = sqlTask.SqlStatementSource;
src_query = src_query + "\n" + src_query2.ToUpper();
}
}
return src_query;
}
The following Query is helpful to retrieve all the sql statements inside the SSIS package
;WITH XMLNAMESPACES ('www.microsoft.com/SqlServer/Dts' AS
DTS,'www.microsoft.com/sqlserver/dts/tasks/sqltask' AS SQLTask)
-- Query to Extract SQL Tasks with Name and SQL Statement
SELECT Pkg.props.value('../../DTS:Property[#DTS:Name="ObjectName"]
[1]','varchar(MAX)') ObjectName,
Pkg.props.value('(#SQLTask:SqlStatementSource)[1]', 'NVARCHAR(MAX)') AS
SqlStatement FROM (select cast(pkgblob.BulkColumn as XML) pkgXML from
openrowset(bulk 'Your DTS package with name and location Path',single_blob)
as pkgblob) t CROSS APPLY pkgXML.nodes('//DTS:ObjectData//SQLTask:SqlTaskData') Pkg(props)
UNION
-- Query to Extract DTS Pipline task with Name and SqlCommand
SELECT Pkg.props.value('../../../../DTS:Property[#DTS:Name="ObjectName"]
[1]','varchar(MAX)') ObjectName,
Pkg.props.value('data(./properties/property[#name=''SqlCommand''])[1]',
'varchar(max)') SqlStatement FROM(select cast(pkgblob.BulkColumn as XML)
pkgXML from openrowset(bulk 'Your DTS package with name and location
Path',single_blob) as pkgblob) t CROSS APPLY
pkgXML.nodes('//DTS:Executable//pipeline//components//component') Pkg(props)
WHERE Pkg.props.value('data(./properties/property[#name=''SqlCommand''])
[1]', 'varchar(max)') <>''
There is another way.
You can create a custom log event. It is written about here:
enabling custom logging for ssis tasks
Then you just need to run the package and parse the log file that is created.
I'm not sure about DTS though but that should get all the SQL from expressions etc. in an SSIS package.
I have the following code which counts vehicles grouped by vehicle type.
using (var uow = new UnitOfWork(new NHibernateHelper().SessionFactory)) {
var repo = new Repository<Vehicle>(uow.Session);
var vtSummary= repo.ListAll()
.GroupBy(v => v.VehicleType.Name)
.Select(v => new NameCount {
EntityDescription = v.First().VehicleType.Name,
QtyCount = v.Count() })
.OrderByDescending(v => v.QtyCount).ToList();
uow.Commit();
return vtSummary;
}
The above produces the following sql code:
SELECT VehicleType.Name as col_0_0_,
CAST(COUNT(*) AS INT) as col_1_0_
FROM Vehicle vehicle0_
LEFT OUTER JOIN VehicleType vehicletype1_
ON vehicle0_.VehicleTypeId= VehicleType.Id
GROUP BY VehicleType.Name
ORDER BY CAST(COUNT(*) AS INT) DESC
The SQL code runs perfectly well under MS SQL Server but testing under SQl CE it produces the following error:
System.Data.SqlServerCe.SqlCeException : Expressions in the ORDER BY list cannot contain aggregate functions.
A solution from Sub query in Sql server CE is to specify an alias for the column and use the alias in the order by clause.
Is there any way to provide an alias in the LINQ expression that I am using to enable it run without raising errors?
You can perform the OrderBy in memory, with LINQ to objects:
var vtSummary= repo.ListAll()
.GroupBy(v => v.VehicleType.Name)
.Select(v => new NameCount {
EntityDescription = v.First().VehicleType.Name,
QtyCount = v.Count() })
.AsEnumerable() //SQL executes at this point
.OrderByDescending(v => v.QtyCount).ToList();