I´m trying to optimize some process in my application but I´m stuck with this problem. My application is working so the entity mapping is correct. Simplifying what I´m trying to do is this:
using (var offCtx = new CheckinOfflineEntities())
{
using (var trans = offCtx.Database.BeginTransaction(IsolationLevel.Snapshot))
{
DateTime purgePivot = DateTime.Now.AddDays(-2);
count = offCtx.Database.ExecuteSqlCommand(#"select L.* into #NewLegs from InventoryLeg L where L.STDUTC >= {0}", purgePivot);
long d = offCtx.Database.SqlQuery<long>("select count(*) from #NewLegs").FirstOrDefault();
}
}
I´m selecting some data I want to delete from one table, storing it in a temporary table so that I can use this temporary table in other queries to exclude related data.
The problem is, when I try to use the temporary table I´m receiving the exception SqlException: "Invalid object name '#NewLegs'."
Thank you for your time.
You can merge the query like this.
And count returns int, not long.
COUNT always returns an int data type value. - MSDN
var query = string.Format("{0} {1}",
#"select L.* into #NewLegs from InventoryLeg L where L.STDUTC >= #STDUTC",
#"select count(*) from #NewLegs")
var d = offCtx.Database.SqlQuery<int>(query, new SqlParameter("STDUTC", purgePivot))
.FirstOrDefault();
I realy don´t know why but removing the parameter and adding it in the query solved the problem.
The code below works fine:
using (var offCtx = new CheckinOfflineEntities())
{
using (var trans = offCtx.Database.BeginTransaction(IsolationLevel.Snapshot))
{
DateTime purgePivot = DateTime.Now.AddDays(-2);
count = offCtx.Database.ExecuteSqlCommand(#"select L.* into #NewLegs from InventoryLeg L where L.STDUTC >= " + purgePivot.toString("yyyy-mm-dd HH:mm:ss");
long d = offCtx.Database.SqlQuery<long>("select count(*) from #NewLegs").FirstOrDefault();
}
}
Related
I'm trying to do a "database side" bulk copy (i.e. SELECT INTO/INSERT INTO) using linq2db. However, my code is trying to bring the dataset over the wire which is not possible given the size of the DB in question.
My code looks like this:
using (var db = new MyDb()) {
var list = db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
});
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.BulkCopy(new BulkCopyOptions {
BulkCopyType = BulkCopyType.MultipleRows
}, list);
}
The generated SQL looks like this:
BeforeExecute
-- DBNAME SqlServer.2017
TRUNCATE TABLE DESTINATION_TABLE
DataConnection
Query Execution Time (AfterExecute): 00:00:00.0361209. Records Affected: -1.
DataConnection
BeforeExecute
-- DBNAME SqlServer.2017
DECLARE #take Int -- Int32
SET #take = 1
DECLARE #take_1 Int -- Int32
SET #take_1 = 1
DECLARE #take_2 Int -- Int32
...
SELECT
(
SELECT TOP (#take)
[p].[YEAR]
FROM
[dbo].[SOURCE_TABLE] [p]
WHERE
(([p_16].[YEAR] = [p].[YEAR] OR [p_16].[YEAR] IS NULL AND [p].[YEAR] IS NULL) AND ...
...)
FROM SOURCE_TABLE p_16
WHERE p_16.YEAR > 2012
GROUP BY
...
DataConnection
That is all that is logged as the bulkcopy fails with a timeout, i.e. SqlException "Execution Timeout Expired".
Please note that running this query as an INSERT INTO statement takes less than 1 second directly in the DB.
PS: Anyone have any recommendations as to good code based ETL tools to do large DB (+ 1 TB) ETL. Given the DB size I need things to run in the database and not bring data over the wire. I've tried pyspark, python bonobo, c# etlbox and they all move too much data around. I thought linq2db had potential, i.e. basically just act like a C# to SQL transpiler but it is also trying to move data around.
I would suggest to rewrite your query because group by can not return first element. Also Truncate is a part of the library.
var sourceQuery =
from s in db.SourceTable
where s.Year > 2012
select new
{
Source = s,
Count = Sql.Ext.Count(s.Column3 == 'Y' ? 1 : null).Over()
.PartitionBy(s.Column1, s.Column2).ToValue()
RN = Sql.Ext.RowNumber().Over()
.PartitionBy(s.Column1, s.Column2).OrderByDesc(s.Year).ToValue()
};
db.DestinationTable.Truncate();
sourceQuery.Where(s => s.RN == 1)
.Insert(db.DestinationTable,
e => new DestinationTable
{
Property1 = 'Constant Value',
Property2 = e.Source.Column1,
Property3 = e.Source.Column2,
Property4 = e.Count
});
After some investigation I stumbled onto this issue. Which lead me to the solution. The code above needs to change to:
db.Execute("TRUNCATE TABLE DESTINATION_TABLE");
db.SourceTable.
Where(s => s.Year > 2012).
GroupBy(s => new { s.Column1, s.Column2 }).
Select(g => new DestinationTable {
Property1 = 'Constant Value',
Property2 = g.First().Column1,
Property3 = g.First().Column2,
Property4 = g.Count(s => s.Column3 == 'Y')
}).Insert(db.DestinationTable, e => e);
Documentation of the linq2db project leaves a bit to be desired however, in terms of functionality its looking like a great project for ETLs (without horrible 1000s of line copy/paste sql/ssis scripts).
I'm getting strange results using trying to do a simple query against a date column using Linq and EF Core.
If I run the query using a date from a list of DateTime I get no results. If I substitute DateTime.Now and add a negative number of days so that if matches the date in the list of DateTimes then the query returns results as expected.
So what is the difference between DateTime.Now and another DateTime object?
In practice, why would this work (rewinding now by 30 days in the first example gives the same date as datesToCheck[0] in the second):
var reports = from r
in db.DailyReports
.Where(r => r.UserId.Equals(currentuser.Identity.Name)
&& r.Date > DateTime.Now.AddDays(-30))
select r;
But not this:
var reports = from r
in db.DailyReports
.Where(r => r.UserId.Equals(currentuser.Identity.Name)
&& r.Date > datesToCheck[0])
select r;
The database is SQL Server 2017, the column is a non-nullable smalldatetime
The datesToCheck list is generated thus:
var datesToCheck = new List<DateTime>();
var startDate = DateTime.Now;
//get Monday date three weeks ago
if (DateTime.Now.DayOfWeek != DayOfWeek.Monday)
{
while (startDate.DayOfWeek != DayOfWeek.Monday)
{
startDate = startDate.AddDays(-1);
}
}
startDate = startDate.AddDays(-21);
while (startDate < DateTime.Now)
{
if (startDate.DayOfWeek != DayOfWeek.Saturday || startDate.DayOfWeek != DayOfWeek.Sunday)
{
datesToCheck.Add(startDate);
startDate = startDate.AddDays(1);
}
}
The same behavior exists in EF6 and, as far as I know, all versions of EF. Basically, the compiler isn't clever enough to decide if datesToCheck[0] should be evaluated or converted to SQL. The query will work if you store the value in a variable and then use the variable in the LINQ query. See also: Why can't we use arrays in Entity Framework queries?
You probably have some datatype issue, Try:
DateTime datesToCheck = DateTime.Now.AddDays(-30);
var reports = from r
in db.DailyReports
.Where(r => r.UserId.Equals(currentuser.Identity.Name)
&& r.Date > datesToCheck )
select r;
Below is the function that inserts my data.
using (SqlCommand insSwipeDataCommand = connection.CreateCommand())
{
insSwipeDataCommand.Transaction = transaction;
insSwipeDataCommand.CommandType = CommandType.StoredProcedure;
insSwipeDataCommand.CommandText = "dbo.insSwipeData_sp";
SqlParameter attendeeTableParam = insSwipeDataCommand.Parameters.Add("#AttendeeTable", SqlDbType.Structured);
attendeeTableParam.Value = this.dataTable;
attendeeTableParam.TypeName = "AttendeeTableType";
// add orgid parameter
insSwipeDataCommand.Parameters.Add("#orgId", SqlDbType.UniqueIdentifier).Value = this.organizationId;
insSwipeDataCommand.ExecuteNonQuery();
}
insSwipeData_sp
create PROC dbo.insSwipeData_sp
(#AttendeeTable AttendeeTableType READONLY
,#orgId UNIQUEIDENTIFIER
)
AS
BEGIN
SET NOCOUNT ON
DECLARE #enteredUserId UNIQUEIDENTIFIER
SET #enteredUserId = 'xxxxxxxxx-xxxx-xxx-xxxx-xxxxxxxxxx'
-- Delete old Swipe records
DELETE FROM dbo.swipeData_tbl
WHERE orgId = #orgId
-- CREATE Swipe Records
INSERT INTO dbo.swipeData_tbl
(orgId, sdid, rawData, enteredUserId, enteredUtc, manualEntry)
SELECT #orgId, attendeeId, barcode
,#enteredUserId, GETUTCDATE(), 0 -- Consider ( datepart , date ) if date here is needed as NVARCHAR
FROM #AttendeeTable
WHERE barcode IS NOT NULL and LTRIM(RTRIM(barcode)) <> '';
END
Here is an image of my AttendeeTableType schema.
and here is an image of my this.datatable that i am using for my attendeeTableParam
On the insSwipeDataCommand.ExecuteNonQuery(); line i get the following error.
The data for table-valued parameter "#AttendeeTable" doesn't conform to the table type of the parameter.
Per the error, your data does not conform to the table type exactly. Note "exactly" -- if you do not specify types for the columns, they will be inferred, and they can easily be inferred incorrectly. The best approach here is to create a table that you know matches the table type definition:
var dt = new DataTable();
dt.Columns.Add("firstName", typeof(string)).MaxLength = 100;
dt.Columns.Add("lastName", typeof(string)).MaxLength = 100;
dt.Columns.Add("studentId", typeof(string)).MaxLength = 10;
dt.Columns.Add("email", typeof(string)).MaxLength = 100;
dt.Columns.Add("barcode", typeof(string)).MaxLength = 100;
dt.Columns.Add("dob", typeof(string)).MaxLength = 200;
dt.Columns.Add("major", typeof(string)).MaxLength = 200;
dt.Columns.Add("gender", typeof(string)).MaxLength = 200;
dt.Columns.Add("classCode", typeof(string)).MaxLength = 15;
dt.Columns.Add("currentclassCode", typeof(string)).MaxLength = 15;
dt.Columns.Add("entranceCode", typeof(string)).MaxLength = 15;
dt.Columns.Add("attendeeId", typeof(Guid));
And then use .Clone() to create a new table with the correct schema when you need to insert data. This way, if you have a type or length mismatch, it will be caught on the client end.
There is another approach you can take that does not rely on embedding the table definition into the application, which is fetching it from the database. There are pros and cons to this -- it requires an extra roundtrip to the database and it's not as easy to spot mistakes in the application logic if the types or columns don't match, but it does give you additional flexibility to change the type without having to change the application (adding a new, nullable column, for example).
var dt = new DataTable();
using (var connection = new SqlConnection(...)) {
connection.Open();
using (var command = new SqlCommand()) {
command.Connection = connection;
command.CommandText = "DECLARE #t dbo.AttendeeTableType; SELECT * FROM #t;"
using (var reader = command.ExecuteReader()) {
dt.Load(reader);
}
}
}
Obviously you probably want to cache the results of this and .Clone(), rather than doing it for every command involving the table type parameter.
I reached this page while searching for similar issue that I was experiencing, but none of the replies did help me. After some head beating I found that the error is being generated in case when the data table being passed from the code has some data that does not match the TVP type specification.
For example if you defined the following type:
CREATE TYPE EmployeeType AS TABLE
(
EmpID BigInt, EmpName VARCHAR(100)
)
and suppose the data table that you are passing has (say) one EmpName that has more than 100 characters then "*** does not conform to table type" error is generated.
This solved my issue. Hope it will help others as well.
Even though Jeroen Mostert answer helped me solve one piece of the puzzle, this error was still popping up.
Here is what I learned;
Datatable created and passed as table value param should
correspond to the sequence / order of field in table value type defined
in sql.
So if your table type looks like below :
CREATE TYPE [dbo].[tvp_mytabletype] AS TABLE(
[ID] BIGINT,
[AddressCode] BIGINT,
[Address] NVARCHAR(200) NOT NULL
)
then define your DT with columns in exact same order/sequence like below :
DataTable dataTable = new DataTable();
//DataTable definition
dataTable.Columns.Add("ID", typeof(long));
dataTable.Columns.Add("AddressCode", typeof(long));
dataTable.Columns.Add("Address", typeof(string)).MaxLength = 200;
Your attendeeId is looks strange. It must be Guid in C# side.
I am trying to write a dapper query for IN clause, but it's not working throwing casting error saying "Conversion failed when converting the nvarchar value 'A8B08B50-2930-42DC-9DAA-776AC7810A0A' to data type int." . In below query fleetAsset is Guid converted into string.
public IQueryable<MarketTransaction> GetMarketTransactions(int fleetId, int userId, int rowCount)
{
//Original EF queries which I am trying to convert to Dapper
//var fleetAsset = (from logicalFleetNode in _context.LogicalFleetNodes
// where logicalFleetNode.LogicalFleetId == fleetId
// select logicalFleetNode.AssetID).ToList();
////This query fetches guid of assetprofiles for which user having permissions based on the assets user looking onto fleet
//var assetProfileIds = (from ap in _context.AssetProfileJoinAccounts
// where fleetAsset.Contains(ap.AssetProfile.AssetID) && ap.AccountId == userId
// select ap.AssetProfileId).ToList();
var fleetAsset = _context.Database.Connection.Query<string>("SELECT CONVERT(varchar(36),AssetID) from LogicalFleetNodes Where LogicalFleetId=#Fleetid",
new { fleetId }).AsEnumerable();
//This query fetches guid of assetprofiles for which user having permissions based on the assets user looking onto fleet
var sql = String.Format("SELECT TOP(#RowCount) AssetProfileId FROM [AssetProfileJoinAccounts] AS APJA WHERE ( EXISTS (SELECT " +
"1 AS [C1] FROM [dbo].[LogicalFleetNodes] AS LFN " +
"INNER JOIN [dbo].[AssetProfile] AS AP ON [LFN].[AssetID] = [AP].[AssetID]" +
" WHERE ([APJA].[AssetProfileId] = [AP].[ID]) " +
" AND ([APJA].[AccountId] = #AccountId AND LogicalFleetId IN #FleetId)))");
var assetProfileIds = _context.Database.Connection.Query<Guid>(sql, new { AccountId = userId, FleetId = fleetAsset, RowCount=rowCount });
Dapper performs expansion, so if the data types match, you should just need to do:
LogicalFleetId IN #FleetId
(note no parentheses)
Passing in a FleetId (typically via an anonymous type like in the question) that is an obvious array or list or similar.
If it isn't working when you remove the parentheses, then there are two questions to ask:
what is the column type of LocalFleetId?
what is the declared type of the local variable fleetAsset (that you are passing in as FleetId)?
Update: test case showing it working fine:
public void GuidIn_SO_24177902()
{
// invent and populate
Guid a = Guid.NewGuid(), b = Guid.NewGuid(),
c = Guid.NewGuid(), d = Guid.NewGuid();
connection.Execute("create table #foo (i int, g uniqueidentifier)");
connection.Execute("insert #foo(i,g) values(#i,#g)",
new[] { new { i = 1, g = a }, new { i = 2, g = b },
new { i = 3, g = c },new { i = 4, g = d }});
// check that rows 2&3 yield guids b&c
var guids = connection.Query<Guid>("select g from #foo where i in (2,3)")
.ToArray();
guids.Length.Equals(2);
guids.Contains(a).Equals(false);
guids.Contains(b).Equals(true);
guids.Contains(c).Equals(true);
guids.Contains(d).Equals(false);
// in query on the guids
var rows = connection.Query(
"select * from #foo where g in #guids order by i", new { guids })
.Select(row => new { i = (int)row.i, g = (Guid)row.g }).ToArray();
rows.Length.Equals(2);
rows[0].i.Equals(2);
rows[0].g.Equals(b);
rows[1].i.Equals(3);
rows[1].g.Equals(c);
}
I have a Code column of type varchar in database, which includes numbers only. I use the method below to get the next code, and it works fine. But I prefer all the calculations and type conversions to be done on SQL side, to have less overhead. Using the code below, I retrieve all data, which is not needed. Any ideas is really appreciated.
public string GetNextCode(int type)
{
var codeList = (from o in ObjectQuery
where o.Type == Type
select o.Code).ToList<string>();
List<int> list = codeList.Select(int.Parse).ToList();
int nextDL = list.Max() + 1;
return nextDL.ToString();
}
You could try something like the below, it orders the result set descending by the Code and then grabs the first result. Please note I haven't tested this but I think it should work.
var codeList = (from o in ObjectQuery
where o.Type == Type
select o.Code).OrderByDescending(
x => x.Code).First()
Instead of OrderByDescending().First() you could also use OrderBy().Last()
This is a solution I came up with:
var max = (from o in ObjectQuery
let num = ObjectQuery.Where(i => i.Type == Type).Select(i => i.Code).Cast<int>().Max()
select num).FirstOrDefault() + 1;
return max.ToString();