Generate sql query by anorm, with all nulls except one - sql-server

I developing web application with play framework 2.3.8 and scala, with complex architecture on backend and front-end side. As backend we use MS SQL, with many stored procedures, and called it by anorm. And here one of the problems.
I need to update some fields in database. The front end calls play framework, and recive name of the field, and value. Then I parse, field name, and then I need to generate SQL Query for update field. I need assign null, for all parameters, except recived parameter. I try to do it like that:
def updateCensusPaperXX(name: String, value: String, user: User) = {
DB.withConnection { implicit c =>
try {
var sqlstring = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}"
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
for (p <- params){
sqlstring += ", "
if (name.endsWith(p))
sqlstring += value
else
sqlstring += "null"
}
SQL(sqlstring)
.on(
"login" -> user.login,
"domain" -> user.domain,
).execute()
} catch {
case e: Throwable => Logger.error("update CensusPaper04 error", e)
}
}
}
But actually that doesn't work in all cases. For example, when I try to save string, it give's me an error like:
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near 'some phrase'
What is the best way to generate sql query using anorm with all nulls except one?

The reason this is happening is because when you write the string value directly into the SQL statement, it needs to be quoted. One way to solve this would be to determine which of the fields are strings and add conditional logic to determine whether to quote the value. This is probably not the best way to go about it. As a general rule, you should be using named parameters rather than building a string to with the parameter values. This has a few of benefits:
It will probably be easier for you to diagnose issues because you will get more sensible error messages back at runtime.
It protects against the possibility of SQL injection.
You get the usual performance benefit of reusing the prepared statement although this might not amount to much in the case of stored procedure invocation.
What this means is that you should treat your list of fields as named parameters as you do with user and domain. This can be accomplished with some minor changes to your code above. First, you can build your SQL statement as follows:
val params = List(
"fieldName1",
"fieldName2",
...,
"fieldNameXX"
)
val sqlString = "Execute [ScXX].[updateCensusPaperXX] {login}, {domain}," +
params.map("{" + _ + "}").mkString{","}
What is happening above is that you don't need to insert the values directly, so you can just build the string by adding the list of parameters to the end of your query string.
Then you can go ahead and start building your parameter list. Note, the parameters to the on method of SQL is a vararg list of NamedParameter. Basically, we need to create Seq of NamedParameters that covers "login", "domain" and the list of fields you are populating. Something like the following should work:
val userDomainParams: Seq[NamedParameter] = (("login",user.login),("domain",user.domain))
val additionalParams = params.map(p =>
if (name.endsWith(p))
NamedParameter(p, value)
else
NamedParameter(p, None)
).toSeq
val fullParams = userDomainParams ++ additionalParams
// At this point you can execute as follows
SQL(sqlString).on(fullParams:_*).execute()
What is happening here is that you building the list of parameters and then using the splat operator :_* to expand the sequence into the varargs needed as arguments for the on method. Note that the None used in the NamedParameter above is converted into a jdbc NULL by Anorm.
This takes care of the issue related to strings because you are no longer writing the string directly into the query and it has the added benefit eliminating other issues related with writing the SQL string rather than using parameters.

Related

Fetching ElasticSearch Results into SQL Server by calling Web Service using SQL CLR

Code Migration due to Performance Issues :-
SQL Server LIKE Condition ( BEFORE )
SQL Server Full Text Search --> CONTAINS ( BEFORE )
Elastic Search ( CURRENTLY )
Achieved So Far :-
We have a web page created in ASP.Net Core which has a Auto Complete Drop Down of 2.5+ Million Companies Indexed in Elastic Search https://www.99corporates.com/
Due to performance issues we have successfully shifted our code from SQL Server Full Text Search to Elastic Search and using NEST v7.2.1 and Elasticsearch.Net v7.2.1 in our .Net Code.
Still looking for a solution :-
If the user does not select a company from the Auto Complete List and simply enters a few characters and clicks on go then a list should be displayed which we had done earlier by using the SQL Server Full Text Search --> CONTAINS
Can we call the ASP.Net Web Service which we have created using SQL CLR and code like SELECT * FROM dbo.Table WHERE Name IN( dbo.SQLWebRequest('') )
[System.Web.Script.Services.ScriptMethod()]
[System.Web.Services.WebMethod]
public static List<string> SearchCompany(string prefixText, int count)
{
}
Any better or alternate option
While that solution (i.e. the SQL-APIConsumer SQLCLR project) "works", it is not scalable. It also requires setting the database to TRUSTWORTHY ON (a security risk), and loads a few assemblies as UNSAFE, such as Json.NET, which is risky if any of them use static variables for caching, expecting each caller to be isolated / have their own App Domain, because SQLCLR is a single, shared App Domain, hence static variables are shared across all callers, and multiple concurrent threads can cause race-conditions (this is not to say that this is something that is definitely happening since I haven't seen the code, but if you haven't either reviewed the code or conducted testing with multiple concurrent threads to ensure that it doesn't pose a problem, then it's definitely a gamble with regards to stability and ensuring predictable, expected behavior).
To a slight degree I am biased given that I do sell a SQLCLR library, SQL#, in which the Full version contains a stored procedure that also does this but a) handles security properly via signatures (it does not enable TRUSTWORTHY), b) allows for handling scalability, c) does not require any UNSAFE assemblies, and d) handles more scenarios (better header handling, etc). It doesn't handle any JSON, it just returns the web service response and you can unpack that using OPENJSON or something else if you prefer. (yes, there is a Free version of SQL#, but it does not contain INET_GetWebPages).
HOWEVER, I don't think SQLCLR is a good fit for this scenario in the first place. In your first two versions of this project (using LIKE and then CONTAINS) it made sense to send the user input directly into the query. But now that you are using a web service to get a list of matching values from that user input, you are no longer confined to that approach. You can, and should, handle the web service / Elastic Search portion of this separately, in the app layer.
Rather than passing the user input into the query, only to have the query pause to get that list of 0 or more matching values, you should do the following:
Before executing any query, get the list of matching values directly in the app layer.
If no matching values are returned, you can skip the database call entirely as you already have your answer, and respond immediately to the user (much faster response time when no matches return)
If there are matches, then execute the search stored procedure, sending that list of matches as-is via Table-Valued Parameter (TVP) which becomes a table variable in the stored procedure. Use that table variable to INNER JOIN against the table rather than doing an IN list since IN lists do not scale well. Also, be sure to send the TVP values to SQL Server using the IEnumerable<SqlDataRecord> method, not the DataTable approach as that merely wastes CPU / time and memory.
For example code on how to accomplish this correctly, please see my answer to Pass Dictionary to Stored Procedure T-SQL
In C#-style pseudo-code, this would be something along the lines of the following:
List<string> = companies;
companies = SearchCompany(PrefixText, Count);
if (companies.Length == 0)
{
Response.Write("Nope");
}
else
{
using(SqlConnection db = new SqlConnection(connectionString))
{
using(SqlCommand batch = db.CreateCommand())
{
batch.CommandType = CommandType.StoredProcedure;
batch.CommandText = "ProcName";
SqlParameter tvp = new SqlParameter("ParamName", SqlDbType.Structured);
tvp.Value = MethodThatYieldReturnsList(companies);
batch.Paramaters.Add(tvp);
db.Open();
using(SqlDataReader results = db.ExecuteReader())
{
if (results.HasRows)
{
// deal with results
Response.Write(results....);
}
}
}
}
}
Done. Got the solution.
Used SQL CLR https://github.com/geral2/SQL-APIConsumer
exec [dbo].[APICaller_POST]
#URL = 'https://www.-----/SearchCompany'
,#JsonBody = '{"searchText":"GOOG","count":10}'
Let me know if there is any other / better options to achieve this.

Does the feature "execute command multiple times" result in multiple round-trips to database?

In the Dapper documentation, it says you can use an IEnumerable parameter to execute a command multiple times. It gives the following example:
connection.Execute(#"insert MyTable(colA, colB) values (#a, #b)",
new[] { new { a=1, b=1 }, new { a=2, b=2 }, new { a=3, b=3 } }
).IsEqualTo(3); // 3 rows inserted: "1,1", "2,2" and "3,3"
Will this result in multiple round-trips to the database (i.e. one for each T in the IEnumerable<T>)? Or is Dapper smart enough to transform the multiple queries into a batch and just do one round-trip? The documentation says an example usage is batch loading, so I suspect it only does one round-trip, but I want to be sure before I use it for performance-sensitive code.
As a follow-up question, and depending on the answer to the first, I'd be curious how transactions are handled? That is, is there one transaction for the whole set of Ts, or one transaction per T?
I finally got around to looking at this again. Looking at the source code (in \Dapper\SqlMapper.cs), I found the following snippet in method ExecuteImpl:
// ...
foreach (var obj in multiExec)
{
if (isFirst)
{
masterSql = cmd.CommandText;
isFirst = false;
identity = new Identity(command.CommandText, cmd.CommandType, cnn, null, obj.GetType(), null);
info = GetCacheInfo(identity, obj, command.AddToCache);
}
else
{
cmd.CommandText = masterSql; // because we do magic replaces on "in" etc
cmd.Parameters.Clear(); // current code is Add-tastic
}
info.ParamReader(cmd, obj);
total += cmd.ExecuteNonQuery();
}
// ...
The interesting part is on the second-last line where ExecuteNonQuery is called. That method is being called on each iteration of the for loop, so I guess it is not being batched in the sense of a set-based operation. Therefore, multiple round-trips are required. However, it is being batched in the sense that all operations are performed on the same connection, and within the same transaction if so specified.
The only way I can think of to do a set-based operation is to create a custom table-valued type (in the database) for the object of interest. Then, in the .NET code pass a DataTable object containing matching names and types as a command parameter. If there were a way to do this without having to create a table-valued type for every object, I'd love to hear about it.

SQL Injection-safe call of polymorphic function

Several times I've found myself refactoring web application code and end up wanting to do something like this (Groovy in this case, but could be anything):
Map getData(String relationName, Integer rowId) {
def sql = Sql.newInstance([...])
def result = sql.firstRow('SELECT getRelationRow(?,?)', relationName, rowId)
sql.close()
return new HashMap(result)
}
where the stored procedure getRelationRow(relname text, rowid integer) executes dynamic sql to retrieve the row of the specified rowid in the requested relation. The best example I've seen of such a function is this polymorphic function using anyelement type, and is called as
SELECT * FROM data_of(NULL::pcdmet, 17);
However to call this in the above code would require
def result = sql.firstRow("SELECT * FROM data_of(NULL::${relationName},?)", rowId)
that is, it required the relation name to be pasted into the query, which risks SQL Injection. So is there away to keep the polymorphic goodness of the stored procedure but allow it to be called with generic relation names?
I don't think it can be done this way. I assume Groovy is using prepared statements here, which requires that input and return types are known at prepare time, while my function derives the return type from the polymorphic input type.
I am pretty sure you need string concatenation. But don't fret, there are functions like pg_escape() to sanitize table names and make SQLi impossible. Don't know Groovy, but it should have that, too.
Or does it?
Based on the function data_of(..) at the end of this related answer:
Refactor a PL/pgSQL function to return the output of various SELECT queries
With PREPARE I can make this work by declaring the return type explicitly:
PREPARE fooplan ("relationName") AS -- table name works as row type
SELECT * FROM data_of($1, $2);
Then I can hand in NULL, which is cast to "relationName" from the prepared context:
EXECUTE fooplan(NULL, 1);
So this might work after all, if your interface supports this. But you still have to concatenate the table name as return data type (and therefore defend against SQLi). Catch 22 I guess.

Pass a DateTime[] as a SQL parameter for ObjectContext.ExecuteStoreCommand

I'm using ExecuteStoreCommand to facilitate bulk deletes.
This Works
// ExecuteStoreCommand syntax via http://stackoverflow.com/a/13024320/740639
// Delete all of today's events
DateTime eventDate = DateTime.Now;
string sql = "DELETE FROM EventCalendar WHERE EventDate = {0}";
context.ObjectContext.ExecuteStoreCommand(sql, new object[] { eventDate });
I want to do this, but it does not work
I get an ArgumentException: "No mapping exists from object type System.DateTime[] to a known managed provider native type."
// Delete all of the events for the past 3 days
DateTime[] eventDates = new DateTime[] {
DateTime.Now.AddDay(-1),
DateTime.Now.AddDay(-2),
DateTime.Now.AddDay(-3)
};
string sql = "DELETE FROM EventCalendar WHERE EventDate IN ({0})";
context.ObjectContext.ExecuteStoreCommand(sql, new object[] { eventDates });
Proposed Solution
I could convert my DateTime[] into the string "'2013-06-24', '2013-06-23', '2013-06-22'" and pass that as the parameter, but that seems like a hack.
I know there are libraries that add bulk delete support to EF, but I don't want to go that route at this time.
Any suggestions?
If you manipulate the string, every time you change the dates you will get a slightly different SQL statement. It may look the same, but the string is different and will have a different hash. So this subverts caching of the execution plan and kills performance.
You can't really pass parameters in an IN clause the way you're thinking. There are some other ideas in this post that might work, but if you read the notes they all say they have lousy performance.
If you can be assured that you will have a fixed number of inputs, you could use the technique of passing them as separate parameters:
... WHERE EventDate IN (#dt1, #dt2, #dt3)
Otherwise, I would go with an approach like Tom suggested, pass them as a delimited string and parse it on the server.
Also - since you are doing an exact match, you probably want to snap to midnight using DateTime.Today rather than DateTime.Now, or use the .Date property of an existing DateTime.
Here's another suggestion, if you really are just trying to delete for the last three days, why not do this?
string sql = "DELETE FROM EventCalendar WHERE EventDate BETWEEN {0} AND {1}";
context.ObjectContext.ExecuteStoreCommand(sql,
new object[] { DateTime.Today.AddDays(-3), DateTime.Today.AddDays(-1) });
Or with a bit of thought, you could probably just use sql getdate() and similar functions instead of passing parameters at all.
You could combine the dates with a delimiter on the .net side, pass it through as a paramter, and then use a Table-Valued function to split a delimited string on the server side.
Pass in something like "2013-06-24|2013-06-23|2013-06-22"
http://ole.michelsen.dk/blog/split-string-to-table-using-transact-sql/

T-SQL IsNumeric() and Linq-to-SQL

I need to find the highest value from the database that satisfies a certain formatting convention. Specifically, I would like to find the highest value that looks like
EU999999 ('9' being any digit)
select max(col) will return something like 'EUZ...' for instance that I want to exclude.
The following query does the trick, but I can't produce this via Linq-to-SQL. There seems to be no translation for the isnumeric() function in SQL Server.
select max(col) from table where col like 'EU%'
and 1=isnumeric(replace(col, 'EU', ''))
Writing a database function, stored procedure, or anything else of that nature is far down the list of my preferred solutions, because this table is central to my app and I cannot easily replace the table object with something else.
What's the next-best solution?
Although ISNUMERIC is missing, you could always try the nearly equivalent NOT LIKE '%[^0-9]%, i.e., there is no non-digit in the string, or alternatively, the string is empty or consists only of digits:
from x in table
where SqlMethods.Like(x.col, 'EU[0-9]%') // starts with EU and at least one digit
&& !SqlMethods.Like(x.col, '__%[^0-9]%') // and no non-digits
select x;
Of course, if you know that the number of digits is fixed, this can be simplified to
from x in table
where SqlMethods.Like(x.col, 'EU[0-9][0-9][0-9][0-9][0-9][0-9]')
select x;
You could make use of the ISNUMERIC function by adding a method to a partial class for the DataContext. It would be similar to using a UDF.
In your DataContext's partial class add this:
partial class MyDataContext
{
[Function(Name = "ISNUMERIC", IsComposable = true)]
public int IsNumeric(string input)
{
throw new NotImplementedException(); // this won't get called
}
}
Then your code would use it in this manner:
var query = dc.TableName
.Select(p => new { p.Col, ReplacedText = p.Col.Replace("EU", "") })
.Where(p => SqlMethods.Like(p.Col, "EU%")
&& dc.IsNumeric(p.ReplacedText) == 1)
.OrderByDescending(p => p.ReplacedText)
.First()
.Col;
Console.WriteLine(query);
Or you could make use MAX:
var query = dc.TableName
.Select(p => new { p.Col, ReplacedText = p.Col.Replace("EU", "") })
.Where(p => SqlMethods.Like(p.Col, "EU%")
&& dc.IsNumeric(p.ReplacedText) == 1);
var result = query.Where(p => p.ReplacedText == query.Max(p => p.ReplacedText))
.First()
.Col;
Console.WriteLine("Max: {0}, Result: {1}", max, result);
Depending on your final goal it might be possible to stop at the max variable and prepend it with the "EU" text to avoid the 2nd query that gets the column name.
EDIT: as mentioned in the comments, the shortcoming of this approach is that ordering is done on text rather than numeric values and there's currently no translation for Int32.Parse on SQL.
As you said, there is no translation for IsNumeric from LINQ to SQL. There are a few options, you already wrote database function and stored procedure down. I like to add two more.
Option 1: You can do this by mixing LINQ to SQL with LINQ to Objects, but when you've got a big database, don't expect great performance:
var cols = (from c in db.Table where c.StartsWith("EU") select c).ToList();
var stripped = from c in cols select int.Parse(c.Replace("EU", ""));
var max = stripped.Max();
Option 2: change your database schema :-)
My suggestion is to fall back to in-line SQL and use the DataContext.ExecuteQuery() method. You would use the SQL query you posted in the beginning.
This is what I have done in similar situations. Not ideal, granted, due to the lack of the type-checking and possible syntax errors, but simply make sure it is included in any unit tests. Not every possible query is covered by the Linq syntax, hence the existence of ExecuteQuery in the first place.

Resources