Just ran into a major headache when concatenating several #varchar(max) variables together to build an email based on several different queries.
For efficiencies sake, I was using several varchars to build the email at once, rather than going through roughly the same query two or three or more times to build it using only one varchar.
This worked, right up until my varchars got to longer than 8000 characters. Then the concatenation of them all into one varchar (which I could shove into the #body parameter of msdb.dbo.sp_send_dbmail) returned "", and even LEN() wouldn't actually give me a length.
Anyhow, I've gotten around this by doing roughly the same queries several times and building the email with only one varchar(max).
TL;DR
I'm not happy with the solution. How could I have appended these varchar(max) variables to each other?
One thing I've hit in the past which may or may not help here: SQL seems to "forget" what datatype its working with when you concatenate varchar(max). Instead of maintaining the MAX, it devolves to conventional varcharnitude, meaning truncation at 8000 characters or so. To get around this, we use the following trick:
Start with
SET #MyMaxVarchar = #aVarcharMaxValue + #SomeString + #SomeOtherString + #etc
and revise like so:
SET #MyMaxVarchar = cast(#aVarcharMaxValue as varchar(max)) + #SomeString + #SomeOtherString + #etc
Again, this may not help with your particular problem, but remembering it might save you major headaches down the road some day.
This may not have happened in your case, but there's a "gotcha" embedded in SQL Management Studio involving VARCHAR(MAX): SQL Studio will only output so many characters in the results grid. You can test this:
SELECT #MyLongVar, LEN(#MyLongVar)
You may find that the length of the actual data returned (most text editors can give you this) is less than the length of the data stored in the variable.
The fix is in Tools | Options | Query Results | SQL Server | Results to Grid; increase Maximum Characters Retrieved | Non XML data to some very large number. Unfortunately the maximum is 65,535, which may not be enough.
If your problem does not involve outputting the variable's value in SQL Studio, please disregard.
I have found that MS SQL silently does NOTHING when attempting to concatentate a string to a NULL value. therefore this solution always works for me:
UPDATE myTable
SET isNull(myCol, '') += 'my text'
WHERE myColumnID = 9999
Related
I keep getting errors when using the REPLACE function in SQL and i don't know why?
I execute this query:
UPDATE [ShellPlus].[dbo].[Views]
SET [ShellPlus].[dbo].[Views].SerializedProperties = REPLACE(SerializedProperties,'|EKZ PSG met verkort EEG','|EKZ PSG met verkort EEG|EEG kort op EEG3')
WHERE [ShellPlus].[dbo].[Views].InternalViewID = '3F4C1E8E-DA0C-4829-B447-F6BDAD9CD505'
And I keep getting this message:
Msg 8116, Level 16, State 1, Line 6
Argument data type ntext is invalid for argument 1 of replace function.
At UPDATE I give the correct table
At SET I give the correct column
At REPLACE I give the: (column name, 'old string', 'new string')
What am I doing wrong?
The real solution is fix the data type:
USE ShellPlus;
ALTER TABLE [dbo].[Views] ALTER COLUMN SerializedProperties nvarchar(MAX);
Then your existing query will work. But you should also normalise that data.
Try this:
UPDATE [ShellPlus].[dbo].[Views]
SET [ShellPlus].[dbo].[Views].SerializedProperties = CAST(REPLACE(CAST(SerializedPropertiesas NVarchar(MAX)),'|EKZ PSG met verkort EEG','|EKZ PSG met verkort EEG|EEG kort op EEG3') ) AS NText)
WHERE [ShellPlus].[dbo].[Views].InternalViewID = '3F4C1E8E-DA0C-4829-B447-F6BDAD9CD505'
Your doing at least three things wrong:
It seems like you're storing delimited data in your column - which is a mistake. For more information, read Is storing a delimited list in a database column really that bad?, where you will see a lot of reasons why the answer to this question is Absolutely yes!
You're using the Text data type, which is deprecated since SQL Server 2008 introduced varchar(max) to replace it. Given the fact that we're in 2019 and the 2008 version just ended it's extended support this July, its high time to change that Text data type to varchar(max) (and if you're using the 2008 or 2008 r2 version, upgrade your SQL Server).
You're using four-parts identifiers for your column names (Thanks #Larnu for pointing that out in the comments). Best practice is to use two-parts identifiers for column names. Read my answer here for a details explanation.
The solution to your problem involves refactoring the database structure - normalize what needs to be normalized, and replace of all Text, NText and Image with their modern replacement data types: varchar(max), nvarchar(max) and varbinary(max). Also, this would be a good time to figure out if you really need to support values longer than 8000 chars (4000 for unicode values) and if not, use a more appropriate value (max length columns have poor performance compared to "regular" length columns).
I have a spreadsheet that gets all values loaded into SQL Server. One of the fields in the spreadsheet happens to be money. Now in order for everything to be displayed correcctly - i added a field in my tbl with Money as DataType.
When i read the value from spreadsheet I pretty much store it as a String, such as this... "94259.4". When it get's inserted in sql server it looks like this "94259.4000". Is there a way for me to basically get rid of the 0's in the sql server value when I grab it from DB - because the issue I'm running across is that - even though these two values are the same - because they are both compared as Strings - it thinks that there not the same values.
I'm foreseeing another issue when the value might look like this...94,259.40 I think what might work is limiting the numbers to 2 after the period. So as long as I select the value from Server with this format 94,259.40 - I thin I should be okay.
EDIT:
For Column = 1 To 34
Select Case Column
Case 1 'Field 1
If Not ([String].IsNullOrEmpty(CStr(excel.Cells(Row, Column).Value)) Or CStr(excel.Cells(Row, Column).Value) = "") Then
strField1 = CStr(excel.Cells(Row, Column).Value)
End If
Case 2 'Field 2
' and so on
I go through each field and store the value as a string. Then I compare it against the DB and see if there is a record that has the same values. The only field in my way is the Money field.
You can use the Format() to compare strings, or even Float For example:
Declare #YourTable table (value money)
Insert Into #YourTable values
(94259.4000),
(94259.4500),
(94259.0000)
Select Original = value
,AsFloat = cast(value as float)
,Formatted = format(value,'0.####')
From #YourTable
Returns
Original AsFloat Formatted
94259.40 94259.4 94259.4
94259.45 94259.45 94259.45
94259.00 94259 94259
I should note that Format() has some great functionality, but it is NOT known for its performance
The core issue is that string data is being used to represent numeric information, hence the problems comparing "123.400" to "123.4" and getting mismatches. They should mismatch. They're strings.
The solution is to store the data in the spreadsheet in its proper form - numeric, and then select a proper format for the database - which is NOT the "Money" datatype (insert shudders and visions of vultures circling overhead). Otherwise, you are going to have an expanding kluge of conversions between types as you go back and forth between two improperly designed solutions, and finding more and more edge cases that "don't quite work," and require more special cases...and so on.
Dear Friends,
I've faced with a problem never thought of ever. My problem seems too simple but I can't find a solution to it.
I have a sql server database column that is of type NVarchar and is filled with standard persian characters. when I'm trying to run a very simple query on it which incorporates the LIKE operator, the resultset becomes empty although I know the query term is present in the table. Here is the very smiple example query which doesn't act corectly:
SELECT * FROM T_Contacts WHERE C_ContactName LIKE '%ف%'
ف is a persian character and the ContactName coulmn contains multiple entries which contain that character.
Please tell me how should I rewrite the expression or what change should I apply. Note that my database's collation is SQL_Latin1_General_CP1_CI_AS.
Thank you very much
Also, if those values are stored as NVARCHAR (which I hope they are!!), you should always use the N'..' prefix for any string literals to make sure you don't get any unwanted conversions back to non-Unicode VARCHAR.
So you should be searching:
SELECT * FROM T_Contacts
WHERE C_ContactName COLLATE Persian_100_CI_AS LIKE N'%ف%'
Shouldn't it be:
SELECT * FROM T_Contacts WHERE C_ContactName LIKE N'%ف%'
ie, with the N in front of the comparing string, so it treats it like an nvarchar?
I know that dynamic SQL queries are bad due to the SQL Injection issues (as well as performance and other issues). I also know that parameterized queries are prefered to avoid injection issues, we all know that.
But my client is still very stubborn and thinks that just
var UserName=Request.Form["UserName"];
UserName=UserName.Replace("'","''");
SQL="SELECT * FROM Users where UserName='" + UserName + "'";
Is enought protection against SQL injection issues against (SQL Server (Only), not mysql).
Can anyone give me real SQL Injection attack example that still can get through the Replace case above? Guess there's some unicode character issues?
I want some real live examples of attacks that still can get through that simple replace.
My question is only for SQL Server and I know that MySQL has some issues with the \ character.
This will not work if you are using NUMBERs.
"SELECT * FROM data WHERE id = " + a_variable + ";"
using
1;DROP TABLE users
Gives you
SELECT * FROM DATA WHERE id=1;DROP TABLE users;
Have a look at
SQL injection
MSDN SQL Injection
EDIT
Have a look at this. It is very close to your question
Proving SQL Injection
Please input your age : 21; drop table users;
SELECT * FROM table where age = 21; drop table users;
ouchies
I have some trouble understanding the scope of replacement. Your original line is:
SQL=SQL.Replace("''","'");
Because you apply it to the variable name SQL, I would assume you are replacing all occurrences of '' with ' in the entire statement.
This can't be correct: consider this statement:
SELECT * FROM tab WHERE col = '<input value goes here>'
Now, if is the empty string, the statement will be:
SELECT * FROM tab WHERE col = ''
...and after SQL.Replace("''", "'") it will become:
SELECT * FROM tab WHERE col = '
As you can see, it will leave a dangling single quote, and yields a syntax error.
Now, let's suppose you intended to write SQL.Replace("'", "''") then the replaced statement would become:
SELECT * FROM tab WHERE col = ''''
Although syntactically correct, you are now comparing col to a literal single quote (as the '' inside the outer single quotes that delimit the literal string will evaluate to a literal single quote). So this can't be right either.
This leads me to believe that you might be doing something like this:
SQL = "SELECT * FROM tab WHERE col = '" & ParamValue.Replace("'", "''") & "'"
Now, as was already pointed out by the previous poster, this approach does not work for number. Or actually, this approach is only applicable in case you want to process the input inside a string literal in the SQL stament.
There is at least on case where this may be problematic. If MS SQL servers QUOTED_IDENTIFIER setting is disabled, then literal strings may also be enclosed by double quote characters. In this case, user values injecting a double quote will lead to the same problems as you have with single quote strings. In addition, the standard escape sequence for a single quote (two single quotes) doesn't work anymore!!
Just consider this snippet:
SET QUOTED_IDENTIFIER OFF
SELECT " "" '' "
This gives the result:
" ''
So at least, the escaping process must be different depending on whether you delimit strings with single or with double quotes. This may not seem a big problem as QUOTED_IDENTIFIER is ON by default, but still. See:
http://msdn.microsoft.com/en-us/library/ms174393.aspx
Please see this XKCD cartoon:
Little Bobby Tables
The answers so far have been targeting on condition query with numeric datatypes and not having single quote in the WHERE clause.
However in MSSQL *at least in ver 2005), this works even if id is say an integer type:
"SELECT * FROM data WHERE id = '" + a_variable + "';"
I hate to say this but unless stored procedure (code that calls EXECUTE, EXEC, or sp_executesql) is used or WHERE clauses do not use quotes for numeric types, using single quote replacement will almost prevent possibility of SQL Injection. I cannot be 100% certain, and I really hope someone can prove me wrong.
I mentioned stored procedure due to second level injection which I only recently read about. See an SO post here on What is second level SQL Injection.
To quote from the accepted answer of the SO question "Proving SQL Injection":
[...] there is nothing inherently unsafe in a properly-quoted SQL statement.
So, if
String data is properly escaped using Replace("'","''") (and your SQL uses single quotes around strings, see Roland's answer w.r.t. QUOTED_IDENTIFIER),
numeric data comes from numeric variables and is properly (i.e. culture-invariantly) converted to string, and
datetime data comes from datetime variables and is properly converted to string (i.e. into one of the culture-invariant formats accepted by SQL Server).
then I cannot think of any way that SQL injection could be done in SQL Server.
The Unicode thing you mentioned in your question was a MySQL bug. Accounting for such problems in your code provides an extra layer of security (which is usually a good thing). Primarily, it's the task of the database engine to make sure that a properly-quoted SQL statement is not a security risk.
Your client is correct.
SQL = SQL.Replace("'","''");
will stop all injection attacks.
The reason this is not considered safe is that it's easy to miss one string entirely.
I've been battling this one for a while now. I have a stored proc that takes in 3 parameters that are used to filter. If a specific value is passed in, I want to filter on that. If -1 is passed in, give me all.
I've tried it the following two ways:
First way:
SELECT field1, field2...etc
FROM my_view
WHERE
parm1 = CASE WHEN #PARM1= -1 THEN parm1 ELSE #PARM1 END
AND parm2 = CASE WHEN #PARM2 = -1 THEN parm2 ELSE #PARM2 END
AND parm3 = CASE WHEN #PARM3 = -1 THEN parm3 ELSE #PARM3 END
Second Way:
SELECT field1, field2...etc
FROM my_view
WHERE
(#PARM1 = -1 OR parm1 = #PARM1)
AND (#PARM2 = -1 OR parm2 = #PARM2)
AND (#PARM3 = -1 OR parm3 = #PARM3)
I read somewhere that the second way will short circuit and never eval the second part if true. My DBA said it forces a table scan. I have not verified this, but it seems to run slower on some cases.
The main table that this view selects from has somewhere around 1.5 million records, and the view proceeds to join on about 15 other tables to gather a bunch of other information.
Both of these methods are slow...taking me from instant to anywhere from 2-40 seconds, which in my situation is completely unacceptable.
Is there a better way that doesn't involve breaking it down into each separate case of specific vs -1 ?
Any help is appreciated. Thanks.
I read somewhere that the second way will short circuit and never eval the second part if true. My DBA said it forces a table scan.
You read wrong; it will not short circuit. Your DBA is right; it will not play well with the query optimizer and likely force a table scan.
The first option is about as good as it gets. Your options to improve things are dynamic sql or a long stored procedure with every possible combination of filter columns so you get independent query plans. You might also try using the "WITH RECOMPILE" option, but I don't think it will help you.
if you are running SQL Server 2005 or above you can use IFs to make multiple version of the query with the proper WHERE so an index can be used. Each query plan will be placed in the query cache.
also, here is a very comprehensive article on this topic:
Dynamic Search Conditions in T-SQL by Erland Sommarskog
it covers all the issues and methods of trying to write queries with multiple optional search conditions
here is the table of contents:
Introduction
The Case Study: Searching Orders
The Northgale Database
Dynamic SQL
Introduction
Using sp_executesql
Using the CLR
Using EXEC()
When Caching Is Not Really What You Want
Static SQL
Introduction
x = #x OR #x IS NULL
Using IF statements
Umachandar's Bag of Tricks
Using Temp Tables
x = #x AND #x IS NOT NULL
Handling Complex Conditions
Hybrid Solutions – Using both Static and Dynamic SQL
Using Views
Using Inline Table Functions
Conclusion
Feedback and Acknowledgements
Revision History
If you pass in a null value when you want everything, then you can write your where clause as
Where colName = IsNull(#Paramater, ColName)
This is basically same as your first method... it will work as long as the column itself is not nullable... Null values IN the column will mess it up slightly.
The only approach to speed it up is to add an index on the column being filtered on in the Where clause. Is there one already? If not, that will result in a dramatic improvement.
No other way I can think of then doing:
WHERE
(MyCase IS NULL OR MyCase = #MyCaseParameter)
AND ....
The second one is more simpler and readable to ther developers if you ask me.
SQL 2008 and later make some improvements to optimization for things like (MyCase IS NULL OR MyCase = #MyCaseParameter) AND ....
If you can upgrade, and if you add an OPTION (RECOMPILE) to get decent perf for all possible param combinations (this is a situation where there is no single plan that is good for all possible param combinations), you may find that this performs well.
http://blogs.msdn.com/b/bartd/archive/2009/05/03/sometimes-the-simplest-solution-isn-t-the-best-solution-the-all-in-one-search-query.aspx