I need to find the highest value from the database that satisfies a certain formatting convention. Specifically, I would like to find the highest value that looks like
EU999999 ('9' being any digit)
select max(col) will return something like 'EUZ...' for instance that I want to exclude.
The following query does the trick, but I can't produce this via Linq-to-SQL. There seems to be no translation for the isnumeric() function in SQL Server.
select max(col) from table where col like 'EU%'
and 1=isnumeric(replace(col, 'EU', ''))
Writing a database function, stored procedure, or anything else of that nature is far down the list of my preferred solutions, because this table is central to my app and I cannot easily replace the table object with something else.
What's the next-best solution?
Although ISNUMERIC is missing, you could always try the nearly equivalent NOT LIKE '%[^0-9]%, i.e., there is no non-digit in the string, or alternatively, the string is empty or consists only of digits:
from x in table
where SqlMethods.Like(x.col, 'EU[0-9]%') // starts with EU and at least one digit
&& !SqlMethods.Like(x.col, '__%[^0-9]%') // and no non-digits
select x;
Of course, if you know that the number of digits is fixed, this can be simplified to
from x in table
where SqlMethods.Like(x.col, 'EU[0-9][0-9][0-9][0-9][0-9][0-9]')
select x;
You could make use of the ISNUMERIC function by adding a method to a partial class for the DataContext. It would be similar to using a UDF.
In your DataContext's partial class add this:
partial class MyDataContext
{
[Function(Name = "ISNUMERIC", IsComposable = true)]
public int IsNumeric(string input)
{
throw new NotImplementedException(); // this won't get called
}
}
Then your code would use it in this manner:
var query = dc.TableName
.Select(p => new { p.Col, ReplacedText = p.Col.Replace("EU", "") })
.Where(p => SqlMethods.Like(p.Col, "EU%")
&& dc.IsNumeric(p.ReplacedText) == 1)
.OrderByDescending(p => p.ReplacedText)
.First()
.Col;
Console.WriteLine(query);
Or you could make use MAX:
var query = dc.TableName
.Select(p => new { p.Col, ReplacedText = p.Col.Replace("EU", "") })
.Where(p => SqlMethods.Like(p.Col, "EU%")
&& dc.IsNumeric(p.ReplacedText) == 1);
var result = query.Where(p => p.ReplacedText == query.Max(p => p.ReplacedText))
.First()
.Col;
Console.WriteLine("Max: {0}, Result: {1}", max, result);
Depending on your final goal it might be possible to stop at the max variable and prepend it with the "EU" text to avoid the 2nd query that gets the column name.
EDIT: as mentioned in the comments, the shortcoming of this approach is that ordering is done on text rather than numeric values and there's currently no translation for Int32.Parse on SQL.
As you said, there is no translation for IsNumeric from LINQ to SQL. There are a few options, you already wrote database function and stored procedure down. I like to add two more.
Option 1: You can do this by mixing LINQ to SQL with LINQ to Objects, but when you've got a big database, don't expect great performance:
var cols = (from c in db.Table where c.StartsWith("EU") select c).ToList();
var stripped = from c in cols select int.Parse(c.Replace("EU", ""));
var max = stripped.Max();
Option 2: change your database schema :-)
My suggestion is to fall back to in-line SQL and use the DataContext.ExecuteQuery() method. You would use the SQL query you posted in the beginning.
This is what I have done in similar situations. Not ideal, granted, due to the lack of the type-checking and possible syntax errors, but simply make sure it is included in any unit tests. Not every possible query is covered by the Linq syntax, hence the existence of ExecuteQuery in the first place.
Related
I developing web application with play framework 2.3.8 and scala, with complex architecture on backend and front-end side. As backend we use MS SQL, with many stored procedures, and called it by anorm. And here one of the problems.
I need to update some fields in database. The front end calls play framework, and recive name of the field, and value. Then I parse, field name, and then I need to generate SQL Query for update field. I need assign null, for all parameters, except recived parameter. I try to do it like that:
def updateCensusPaperXX(name: String, value: String, user: User) = {
DB.withConnection { implicit c =>
try {
var sqlstring = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}"
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
for (p <- params){
sqlstring += ", "
if (name.endsWith(p))
sqlstring += value
else
sqlstring += "null"
}
SQL(sqlstring)
.on(
"login" -> user.login,
"domain" -> user.domain,
).execute()
} catch {
case e: Throwable => Logger.error("update CensusPaper04 error", e)
}
}
}
But actually that doesn't work in all cases. For example, when I try to save string, it give's me an error like:
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 'some phrase'
What is the best way to generate sql query using anorm with all nulls except one?
The reason this is happening is because when you write the string value directly into the SQL statement, it needs to be quoted. One way to solve this would be to determine which of the fields are strings and add conditional logic to determine whether to quote the value. This is probably not the best way to go about it. As a general rule, you should be using named parameters rather than building a string to with the parameter values. This has a few of benefits:
It will probably be easier for you to diagnose issues because you will get more sensible error messages back at runtime.
It protects against the possibility of SQL injection.
You get the usual performance benefit of reusing the prepared statement although this might not amount to much in the case of stored procedure invocation.
What this means is that you should treat your list of fields as named parameters as you do with user and domain. This can be accomplished with some minor changes to your code above. First, you can build your SQL statement as follows:
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
val sqlString = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}," +
params.map("{" + _ + "}").mkString{","}
What is happening above is that you don't need to insert the values directly, so you can just build the string by adding the list of parameters to the end of your query string.
Then you can go ahead and start building your parameter list. Note, the parameters to the on method of SQL is a vararg list of NamedParameter. Basically, we need to create Seq of NamedParameters that covers "login", "domain" and the list of fields you are populating. Something like the following should work:
val userDomainParams: Seq[NamedParameter] = (("login",user.login),("domain",user.domain))
val additionalParams = params.map(p =>
if (name.endsWith(p))
NamedParameter(p, value)
else
NamedParameter(p, None)
).toSeq
val fullParams = userDomainParams ++ additionalParams
// At this point you can execute as follows
SQL(sqlString).on(fullParams:_*).execute()
What is happening here is that you building the list of parameters and then using the splat operator :_* to expand the sequence into the varargs needed as arguments for the on method. Note that the None used in the NamedParameter above is converted into a jdbc NULL by Anorm.
This takes care of the issue related to strings because you are no longer writing the string directly into the query and it has the added benefit eliminating other issues related with writing the SQL string rather than using parameters.
In the development of an Orchard module, how do I retrieve ContentParts case insensitively filtered by a field? I have tried
var name = viewModel.Name.ToUpper();
var samples = _contentManager.Query<SamplePart, SamplePartRecord>()
.Where(x => x.Name.ToUpper() == name)
.List();
and I'm getting an error
Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
but when I tried to retrieve without bothering if it's case sensitive
var name = viewModel.Name;
var samples = _contentManager.Query<SamplePart, SamplePartRecord>()
.Where(x => x.Name == name)
.List();
No errors reported.
What gives?
Be aware that the expression inside the Where clause is being translated by NHibernate to an SQL query at some point. Hence you're pretty restricted to what you can do there. In this case it seems like the ToUpper method is not supported.
Another thing - the string comparison behavior in SQL Server depends on the actual collation set on your database. By default it's case insensitive, so any string comparison will ignore case. So, if you stick to defaults you're good with just an ordinary == on two strings, like in your last example.
I have a followup to another Slick question I recently asked (Slick table Query: Trouble with recognizing values) here. Please bear with me!! I'm new to databasing and Slick seems especially poor on documentation. Anyway, I have this table:
object Users extends Table[(Int, String)]("Users") {
def userId = column[Int]("UserId", O.PrimaryKey, O.AutoInc)
def userName = column[String]("UserName")
def * = userId ~ userName
}
Part I
I'm attempting to query with this function:
def findByQuery(where: List[(String, String)]) = SlickInit.dbSlave withSession {
val q = for {
x <- Users if foo((x.userId, x.userName), where)
} yield x
q.firstOption.map { case(userId, userName) =>
User(userId, userName)}
}
where "where" is a list of search queries //ex. ("userId", "1"),("userName", "Alex")
"foo" is a helper function that tests equality. I'm running into a type error.
x.userId is of type Column[Int]. How can one manipulate this as an Int? I tried casting, ex:
foo(x.userId.asInstanceOf[Int]...)
but am also experiencing trouble with that. How does one deal with Slick return types?
Part II
Is anyone familiar with the casting function:
def * = userId ~ userName <> (User, User.unapply _)
? I know there have been some excellent answers to this question, most notably here: scala slick method I can not understand so far and a very similar question here: mapped projection with companion object in SLICK. But can anyone explain why the compiler responds with
<> method overloaded
for that simple line of code?
Let's start with the problem:
val q = for {
x <- Users if foo((x.userId, x.userName), where)
} yield x
See, Slick transforms Scala expressions into SQL. To be able to transform conditions, as you want, into SQL statement, Slick requires some special types to be used. The way these types works are actually part of the transformation Slick performs.
For example, when you write List(1,2,3) filter { x => x == 2 } the filter predicate is executed for each element in the list. But Slick can't do that! So Query[ATable] filter { arow => arow.id === 2 } actually means "make a select with the condition id = 2" (I am skipping details here).
I wrote a mock of your foo function and asked Slick to generate the SQL for the query q:
select x2."UserId", x2."UserName" from "Users" x2 where false
See the false? That's because foo is a simple predicate that Scala evaluates into the Boolean false. A similar predicate done in a Query, instead of a list, evaluates into a description of a what needs to be done in the SQL generation. Compare the difference between the filters in List and in Slick:
List[A].filter(A => Boolean):List[A]
Query[E,U].filter[T](f: E => T)(implicit wt: CanBeQueryCondition[T]):Query[E,U]
List filter evaluates to a list of As, while Query.filter evaluates into new Query!
Now, a step towards a solution.
It seems that what you want is actually the in operator of SQL. The in operator returns true if there is an element in a list, eg: 4 in (1,2,3,4) is true. Notice that (1,2,3,4) is a SQL list, not Tuple like in Scala.
For this use case of the in SQL operator Slick uses the operator inSet.
Now comes the second part of the problem. (I renamed the where variable to list, because where is a Slick method)
You could try:
val q = for {
x <- Users if (x.userId,x.userName) inSet list
} yield x
But that won't compile! That's because SQL doesn't have Tuples the way Scala has. In SQL you can't do (1,"Alfred") in ((1,"Alfred"),(2, "Mary")) (remember, the (x,y,z) is the SQL syntax for lists, I am abusing the syntax here only to show that it's invalid -- also there are many dialects of SQL out there, it is possible some of them do support tuples and lists in a similar way.)
One possible solution is to use only the userId field:
val q = for {
x <- Users if x.userId inSet list2
} yield x
This generates select x2."UserId", x2."UserName" from "Users" x2 where x2."UserId" in (1, 2, 3)
But since you are explicitly using user id and user name, it's reasonable to assume that user id doesn't uniquely identify a user. So, to amend that we can concatenate both values. Of course, we need to do the same in the list.
val list2 = list map { t => t._1 + t._2 }
val q2 = for {
x <- Users if (x.userId.asColumnOf[String] ++ x.userName) inSet list2
} yield x
Look the generated SQL:
select x2."UserId", x2."UserName" from "Users" x2
where (cast(x2."UserId" as VARCHAR)||x2."UserName") in ('1a', '3c', '2b')
See the above ||? It's the string concatenation operator used in H2Db. H2Db is the Slick driver I am using to run your example. This query and the others can vary slightly depending on the database you are using.
Hope that it clarifies how slick works and solve your problem. At least the first one. :)
Part I:
Slick uses Column[...]-types instead of ordinary Scala types. You also need to define Slick helper functions using Column types. You could implement foo like this:
def foo( columns: (Column[Int],Column[String]), values: List[(Int,String)] ) : Column[Boolean] = values.map( value => columns._1 === value._1 && columns._2 === value._2 ).reduce( _ || _ )
Also read pedrofurla's answer to better understand how Slick works.
Part II:
Method <> is indeed overloaded and when types don't work out the Scala compiler can easily become uncertain which overload it should use. (We should get rid of the overloading in Slick I think.) Writing
def * = userId ~ userName <> (User.tupled _, User.unapply _)
may slightly improve the error message you get. To solve the problem make sure that the Column types of userId and userName exactly correspond to the member types of your User case class, which should look something like case class User( id:Int, name:String ). Also make sure, that you extend Table[User] (not Table[(Int,String)]) when mapping to User.
Lets say I have a model, Article that has a large amount of columns and the database contains more than 100,000 rows. If I do something like var articles = db.Articles.ToList() it is retrieving the entire article model for each article in the database and holding it in memory right?
So if I am populating a table that only shows the date of the entry and it's title is there a way to only retrieve just these columns from the database using the entity framework, and would it be more efficient?
According to this,
There is a cost required to track returned objects in the object
context. Detecting changes to objects and ensuring that multiple
requests for the same logical entity return the same object instance
requires that objects be attached to an ObjectContext instance. If you
do not plan to make updates or deletes to objects and do not require
identity management , consider using the NoTracking merge options when
you execute queries.
it looks like I should use NoTracking since the data isn't being changed or deleted, only displayed. So my query now becomes var articles = db.Articles.AsNoTracking().ToList(). Are there other things I should do to make this more efficient?
Another question I have is that according to this answer, using .Contains(...) will cause a large performance drop when dealing with a large database. What is the recommended method to use to search through the entries in a large database?
It's called a projection and just translates into a SELECT column1, column2, ... in SQL:
var result = db.Articles
.Select(a => new
{
Date = a.Date,
Title = a.Title
})
.ToList();
Instead of a => new { ... } (creates a list of "anonymous" objects) you can also use a named helper class (or "view model"): a => new MyViewModel { ... } that contains only the selected properties (but you can't use a => new Article { ... } as an entity itself).
For such a projection you don't need AsNoTracking() because projected data are not tracked anyway, only full entity objects are tracked.
Instead of using Contains the more common way is to use Where like:
var date = DateTime.Now.AddYears(-1);
var result = db.Articles
.Where(a => date <= a.Date)
.Select(a => new
{
Date = a.Date,
Title = a.Title
})
.ToList();
This would select only the articles that are not older than a year. The Where is just translated into a SQL WHERE statement and the filter is performed in the database (which is as fast as the SQL query is, depending on table size and proper indexing, etc.). Only the result of this filter is loaded into memory.
Edit
Refering to your comment below:
Don't confuse IEnumerable<T>.Contains(T t) with string.Contains(string subString). The answer you have linked in your question talks about the first version of Contains. If you want to search for articles that have the string "keyword" in the text body you need the second Contains version:
string keyword = "Entity Framework";
var result = db.Articles
.Where(a => a.Body.Contains(keyword))
.Select(a => new
{
Date = a.Date,
Title = a.Title
})
.ToList();
This will translate into something like WHERE Body like N'%Entity Framework%' in SQL. The answer about the poor performance of Contains doesn't apply to this version of Contains at all.
I am creating a task-scheduler on SQL Server 2008.
I have a table that I use to store tasks. Each task is a task name (e.g. ImportFile) and arguments. I store arguments in XML column, since different tasks have different signatures.
Table is as follows:
Id:integer(PK) | operation:nvarchar | Arguments:xml
Before queuing a task, I often need to verify that given task hasn't been scheduled yet. The lookup is done based on both operation and args.
Question: Using Linq-to-Sql how can I check if given operation+args is present in the queue already?
I am looking for something like:
var isTaskScheduled = db.Tasks.Any(t =>
t.Opearation == task.Operation &&
t.Arguments == task.ArgumentsAsXElement);
(which doesn't work because SQL Server can't compare XML type)
Any alternative implementation suggestions?
You might want to surface e.g. a string property that encapsultes your Arguments, or maybe it would be sufficient to have e.g. the length and a CRC of your Arguments as extra properties on your class:
public partial class Task
{
public int ArgumentLength
{ .... }
public int ArgumentCRC
{ .... }
}
That way, if you can compare length (of your XML) and the CRC and they match, you can be pretty sure and safe to assume the two XML's are identical. Your check would then be something like:
var isTaskScheduled =
db.Tasks.Any(t => t.Operation == task.Operation &&
t.ArgumentLength == task.ArgumentLength &&
t.ArgumentCRC == task.ArgumentCRC);
or something like that.
This may be a stretch, but you could use a "Hashcode" when saving the data to the database, then query on the hashcode value at a later date / time.
This assumes that you have a class that represents your task entity and that you have overridden the GetHashCode method of said class.
Now, when you go to query the database to see if the task is in the scheduled queue, you simply query on the hashcode, thus avoiding the need to do any xml poking at query time.
var t1 = new Task{Operation="Run", Arguments="someXElement.value"};
var t2 = new Task{Operation="Run", Arguments="someXElement.value"};
in the code above t1 == t2 because you are overriding GetHashCode and computing the hash for Operation+Arguments.Value. if you store the hashcode in the db, then you can easily tell if you have an object in the DB that equals the hash code that you are checking for.
This may be similar to what marc_s was talking about.
You can write a class which implements IComparable:
public class XMLArgument : IComparable
{
public XMLArgument(string argument)
{
}
public int CompareTo(object obj)
{
...
}
}
var isTaskScheduled = db.Tasks.Any(t =>
t.Opearation == task.Operation &&
(new XMLArgument(t.Arguments)).CompareTo(new XMLArgument(task.ArgumentsAsXElement)) == 0);