I load excel file into sql as varchar(max) and got that Scientific e value which now I try to convert into numeric as I need to do compare that value, and here I'm running into problem.
This is main question: How and to what type I can convert this to compare with whole integer value ?
On the pic You can see how this seen in Excel, even formatted to text it somehow still loaded into varchar(max) not like char string. This can be seen from my test code.
DECLARE #C VARCHAR(MAX) = '1.1001562717e+011', #Nc VARCHAR(MAX) = '110015627174';
SELECT #c, LEN(#c) LenC ,
ISNUMERIC(#c) NumYN
---, CAST(#c AS DECIMAL(38,2)) cDec ---CAST(#c AS NUMERIC) cNum --, CAST(#c AS BIGINT) cInt
WHERE #c LIKE '%[^0-9]%'
AND ISNUMERIC(#c) = 1
To start, ISNUMERIC is a terrible function, it does not give good results; it is often wrong. If you try ISNUMERIC('1.1001562717e+011') you'll notice that you get the value 1, however, CONVERT(numeric(13,1),'1.1001562717e+011') will produce an error. A far better function is TRY_CONVERT (or TRY_CAST), which returns NULL if the conversion fails for the specific data type: TRY_CONVERT(numeric(13,1),'1.1001562717e+011').
Being specific on the data type is actually important here, as ISNUMERIC could be (incorrectly) suggesting that the value could be converted to at least 1 of the numeric data types; but that doesn't mean all of them. For scientific data types the only data type you can convert to is a float/real:
SELECT TRY_CONVERT(numeric(13,1),'1.1001562717e+011') AS Numeric,
TRY_CONVERT(bigint,'1.1001562717e+011') AS int,
TRY_CONVERT(float,'1.1001562717e+011') AS float,
TRY_CONVERT(money,'1.1001562717e+011') AS money;
Notice that only float has a value here. As you want a numeric as the final value, then you'll need to CONVERT the value twice:
CONVERT(numeric(13,1),TRY_CONVERT(float,'1.1001562717e+011'))
Related
Here is my question:
This is a table named HB06,the data type of "WRTime" is datatime.I want to convert all WRTime to int. For example 2012-11-09 10:52:38.000 will be converted to 20121109105238.
Thank you!
You can't. The value is too large for the int data type. But it does fit into a bigint.
What you can do is to convert it to a string with the desired format, and then cast that ti the bigint type. Using the FORMAT function is IMO more straight forward:
DECLARE #a datetime = '20120304 23:34:12'
SELECT #a
SELECT CAST(FORMAT(#a, 'yyyyMMddhhmmss') AS bigint)
The alternative is to use CONVERT function, which uses less CPU. But there's not direct style that matches that format, so you would then REPLACE() various "litter" characters with nothing. I wouldn't bother with the CONVERT() option unless you work over large data sets.
select
cast(replace(replace(replace(convert(varchar(19), WRTime, 121),':',''),'-',''),' ','') as bigint)
FROM HB06
You can try with below one
select concat(convert(varchar,WRTime,112),datepart(HH,WRTime),
datepart(MINUTE,WRTime),datepart(SS,WRTime)) from HB06
Use below Convert function:
SELECT CONVERT(VARCHAR(100),WRTime,112)+REPLACE(CONVERT(VARCHAR(100),WRTime,108),':','')
FROM HB06
SQL Version 2012 or higher you can use the FORMAT function to get just year and month, then cast it as an int.
On versions prior to 2012 you can do the formatting with the convert function, then cast as int.
declare #WRTime datetime
set #WRTime = '2012-11-09 10:52:38.000'
select cast(format(#WRTime,'yyyyMM') as int) --2012 or higher
OR You can use:
SELECT YEAR(#WRTime)*100 + MONTH(#WRTime);
2012-11-09 10:52:38.000 will be converted to 20121109105238??
Int can't convert this so use "BIGINT"
declare #WRTime datetime
set #WRTime = '2012-11-09 10:52:38.000'
select cast(format(#WRTime,'yyyyMMddHHmmssfff') as bigint) --2012 or higher
I have observed an peculiar issue in the ISNUMERIC function. Below is the same code:
declare #variable nvarchar(max) = '123,456,789,147,258,369'
if (ISNUMERIC(#variable) = 1 )
select 'numeric' as output
else
select 'char' as output`
Below is the link for demo of the out put and different scenarios
Demo Here
It gives output as Character, though it numeric. Any suggestions for the behavior.
I have observed this behavior only when we copy the numbers and paste it in variable declaration, instead if we just type the number then it works fine.
Demo Here
Your CSV string is just that, a string, despite that it consists of CSV numbers. If you want to verify that you have an all-numeric CSV string you could try removing all commas and then asserting that what remains is a number:
declare #variable nvarchar(max) = '123,456,789,147,258,369'
if (ISNUMERIC(REPLACE(#variable, ',', ''l) = 1 )
select 'numeric' as output
else select 'char' as output
The string you defined, '123,456,789,147,258,369' is not a number, a number cannot have multiple commas.
If you have specific criteria on what do you consider a number, your should build your a user defined function to determine to sort strings as 'numeric' or 'char' according to your criteria.
Best regards,
Sergio
The reason for the behavior you are seeing appears to be due to the length of the string.
After reading this on MSDN:
ISNUMERIC returns 1 when the input expression evaluates to a valid
numeric data type
I ran some tests:
select ISNUMERIC('1,234')
1
select ISNUMERIC('1,234,456')
1
proves that you can have commas in the string
select isnumeric('123,456,789,147,258,369')
0
confirms the behavior you are seeing.
Your string is too big to be an int, but it could be a bigint, so:
select CONVERT(bigint,'123,456,789,147,258,369')
Error converting data type varchar to bigint.
select CONVERT(int,'1,234')
Conversion failed when converting the varchar value '1,234' to data type int.
proves that strings with commas can't be converted to int. What about decimals?
select CONVERT(decimal(10,0),'1,234')
Error converting data type varchar to numeric.
select CONVERT(float,'1,234')
Error converting data type varchar to float.
nope. that leaves money:
select CONVERT(money,'2,234')
2234.00
select CONVERT(money,'2,234,000')
2234000.00
so strings with (multiple) commas can be converted to money. But if the number would be too big to fit in a money datatype:
select CONVERT(money,'123,456,789,147,258,369')
Arithmetic overflow error converting expression to data type money.
then it can't be converted, and if it can't be converted, then ISNUMERIC()=0.
Thanks for your responses .Below is the final solution I got by eliminating the comma, NEXT line char and Space character as below:
declare #variable nvarchar(max) = '21386, 21385, 20178, 20176, 19958, 20026, 19976, 19933, 20029, 19921, 4552, 19784, 4700, 19730, 14655, 4749, 4998, 19604,'
if (ISNUMERIC(REPLACE( replace(REPLACE(REPLACE(#variable, ',' ,'1'), CHAR(10), ''), CHAR(9), ''), CHAR(13), '')) = 1 )
select 'numeric' as output
else select 'char' as output
I've been toying around with T-SQL in an attempt to understand the implicit conversion and overload resolution rules, but somehow it seems to work a bit strange...
Context:
Data type conversion: https://msdn.microsoft.com/en-us/library/ms191530.aspx
Data type precedence: https://msdn.microsoft.com/en-us/library/ms190309.aspx
Abs: https://msdn.microsoft.com/en-us/library/ms189800.aspx
Basically the latter tells you that abs will work on int, float, decimal, etc. So let's see how that works:
declare #foo2 sql_variant;
set #foo2 = abs(4);
select sql_variant_property(#foo2, 'BaseType')
-- result: int. OK, apparently we have an int overload. As expected.
declare #foo2 sql_variant;
set #foo2 = abs(cast(4.0 as float));
select sql_variant_property(#foo2, 'BaseType')
-- result: float. OK, apparently we have a float overload. As expected.
Now, according to the implicit type conversion table, we are allowed to implicitly convert stuff. We're going to check this by converting a varbinary to int, which should happen according to the type precedence rules:
declare #foo varbinary(4);
set #foo = cast(4 as varbinary(4));
select #foo + 2;
-- result: int. OK, as expected.
From this result I would expect the following to work as well:
declare #foo varbinary(4);
set #foo = cast(4 as varbinary(4));
select abs(#foo);
-- result: error: Operand type clash: varbinary is incompatible with float
Stuff I don't understand here (the question):
Why does the implicit conversion pick the 'float' overload of the 'abs'? Is this just a random overload? Or perhaps the overload that's highest in the precedence list (which happens to be 'float')?
Why isn't implicit conversion applied from varbinary -> int? After all, it's a perfectly valid conversion.
From MSDN for Abs:
Arguments
numeric_expression
Is an expression of the exact numeric or approximate numeric data type category.
I guess that calling Abs( varbinary ) would try to convert the varbinary to an exact numeric or approximate numeric data type.
Float is at the top of Data Type Precedence for these types so there is the problem I guess.
UPDATE with my logic
declare #foo varbinary(4);
Declare a varbinary variable.
set #foo = cast( 4 as varbinary(4));
Set to this variable a value by performing an explicit cast from int to varbinary which is perfectly doable.
select sql_variant_property(#foo, 'BaseType')
This shows that the variable is of type varbinary.
select abs(#foo);
Try to run abs on varbinary.
The error message is clear:
Operand type clash: varbinary is incompatible with float
So my guess is that abs is trying to implicit convert varbinary to the first of the Data Type Precedence for exact numeric or approximate numeric data types which is float.
This conversion fails according to CAST and CONVERT chart.
I hope I've got this right! Comments appreciated.
You've found the conversion and precedence MSDN pages so let's take your example apart.
declare #foo varbinary(4);
set #foo = cast(4 as varbinary(4));
Now, what type is foo?
SELECT SQL_VARIANT_PROPERTY(#foo,'BaseType')
--------
varbinary
Good so far so good. But what's 'in' #foo ?
SELECT #foo
----------
0x00000004
Now, ABS() accepts a numeric - a floating point number:
What does 4.00 look like compared to 4? Let's try:
SELECT CONVERT(VARBINARY(MAX), 4.00) ,CONVERT(VARBINARY(MAX), 4);
------------------ -----------------------
0x0302000190010000 0x00000004
See the internal representation of 4.00 is different to 4? It's storing precision, scale and the value whereas your 'int in a varbinary' isn't.
And that's why the conversion fails. ABS() accepts a numeric, your 'wrong format' varbinary cannot be coerced to a numeric as it's not one, it's some other representation - we know it represents an int but SQL Server doesn't.
If you want to add a floating point number to an int that's 'inside' a varbinary, you have to do something like
select CONVERT(INT, #foo) + 2.0
Back to your example:
declare #foo varbinary(max);
set #foo = cast(4.00 as varbinary(max));
SELECT ABS(#foo+2.0)
----------
6.0
4.00 is converted into a varbinary in the correct internal format, and happily coerced into a numeric by the ABS() call. Happy days.
I'm finding this to be truly bizarre behaviour.
Here is my T-SQL:
declare #testText nvarchar(1000);
set #testText = '17D4,A,1';
select txt_value from fn_ParseText2Table (#testText , ',' )
where fn_ParseText2Table is a function that parses the text into a table where you can get the txt, int, and floating point values, if they work.
The 17D4 is a product code that I'm trying to extract within a larger query, and all other 3817 records work fine.
select (
select txt_value
from fn_ParseText2Table(t.primaryKeyValues , ',' ) as pk
where position = 1) as product_NBR
from database.dbo.tablesToParse as t
where t.tableName = 'ProductData'
I found the function here.
What I've found is that if the string starts with some numbers (I've tested anywhere from 1-4 ) followed by 'D', it fails with the 'Error converting data type varchar to numeric.' message.
All other combinations of text work. Lower case d is fine. C is fine, E, F, etc. So '17F5,A,1' is fine. Also 'asdf 17D5,A,1' is fine. '1D,A,1' is not fine.
I'm very confused. Is there a special escape character in T-SQL for 'D'?
Update:
I should clarify that the error occurs inside fn_ParseText2Table()
Update 2
It's SQL server 10 - 64 bit, running on a windows 2008 server.
As well, I've tested this in a sql mgr query window:
declare #testText nvarchar(1000);
set #testText = '17D4';
select isnumeric( #testText )
The IsNumeric() call returns 1, which is why the fn_ParseText2Table() function tries to cast it to an in and it fails. I could add an extra check to that function to lower the text first and see if that's also numeric.
I figured it was related to floating-point literals but I was surprised it wouldn't cast to a numeric type. I guess it only works when float is the target type. You'll find the explanation here:
http://www.sqlservercentral.com/Forums/Topic202581-8-1.aspx
This behavior doesn't match up with other SQL Server literals, ie, constants:
select 1d -- literal 1 with d treated as the column alias
select 1e -- literal 1.0 as a float
select cast('1d' as float), cast('1e' as float) -- neither of these will cast without an exponent value
select cast('1d0' as float), cast('1e0' as float) -- these work though
I have some dirty input data that is being imported into a raw source table within SQL Server (2008 R2). Fields that are defined as decimal(9,2) or decimal(4,2) by the input provider are coming in as strings, however, the strings do not always conform to the data definition (go figure!).
We import the data from flat files into the raw tables,then apply some conversion scripts to insert the 'cleaned' data into tables with the proper data types assigned to columns.
For instance:
raw_table
TotalAmount varchar(12)
clean_table
TotalAmount decimal(9,2)
Now, my question is this. If I want to do some 'basic' cleanup on this, I would want to do it in a function along the lines of:
CREATE FUNCTION [dbo].[StringToDecimal]
(
#conversionString VARCHAR(12)
)
RETURNS DECIMAL(9,2)
AS
BEGIN
DECLARE #rsp DECIMAL(9,2)
IF ISNUMERIC( LTRIM(RTRIM(REPLACE(#conversionString,' ',''))) ) = 1
BEGIN
SET #rsp = ISNULL( CONVERT( decimal(17,6), NULLIF( LTRIM(RTRIM(REPLACE(#conversionString,' ',''))),'') ), 0 )
END
ELSE
BEGIN
SET #rsp = 0 -- or we can return NULL here
END
RETURN #rsp
END
However, how could one go about supporting various sized decimals in this mix? Is there a way to parametrize the response type? I considered just returning a decimal of the largest size we generally see, then converting it again on the other end, however, you run into arithmetic overflow issues.
Would appreciate any thoughts/insight into solving this one!
Is there a way to parametrize the response type?
It's simpler than you think. Just return as a VARCHAR and do the casting to decimal(x,y) from the VARCHAR. You don't even need to cast - you can directly assign a VARCHAR (as long as it holds valid decimal data) to a decimal column/variable.
I will create 2 functions instead. StringToDecimal2 does the actual conversion, but returns one of 6 "error codes". You can use it to check why a string is invalid. Or use the wrapper dbo.StringToDecimal which just turns the invalid codes into NULL.
CREATE FUNCTION [dbo].[StringToDecimal2]
(
#conversionString VARCHAR(12),
#precision int, -- total digits
#scale int -- after decimal point
)
RETURNS VARCHAR(100)
AS
BEGIN
-- remove spaces, we'll allow this error. no need to trim
set #conversionString = REPLACE(#conversionString,' ','')
-- note: 1,234.56 (thousands separated) will be invalid, so will 1,234,56 (European decimals)
-- well, ok, let's clean up the thousands separators. BUT! It will incorrectly scale European decimals
set #conversionString = REPLACE(#conversionString,',','')
-- we don't support scientific notation either, so 1e4 (10,000) is out
if #conversionString like '%[^0-9.+-]%' return 'INVALID1' -- only digits and decimal are valid (plus +-)
if #conversionString like '%.%.%' return 'INVALID2' -- too many decimals
if #conversionString like '_%[+-]%' return 'INVALID3' -- +- symbol not in the first position
if #conversionString like '[.+-]' return 'INVALID4' -- a single character from "+-."
if #conversionString like '[+-].' return 'INVALID5' -- symbol and decimal only
-- add a decimal place so it is easier to work with below
if #conversionString not like '%.%'
set #conversionString = #conversionString + '.'
-- allow decimal places to go only as far as scale
set #conversionString = left(#conversionString, charindex('.', #conversionString)+#scale)
-- ensure the data is within precision number of digits in total
if charindex('.', #conversionString) > #precision - #scale + 1
return 'INVALID6' -- too many digits before decimal
RETURN #conversionString
END
GO
CREATE FUNCTION [dbo].[StringToDecimal]
(
#conversionString VARCHAR(12),
#precision int, -- total digits
#scale int -- after decimal point
)
RETURNS VARCHAR(100)
AS
BEGIN
RETURN case when [dbo].[StringToDecimal2](#conversionString, #precision, #scale) like 'INVALID%'
then null else [dbo].[StringToDecimal2](#conversionString, #precision, #scale) end
END
GO
Some tests:
select [dbo].[StringToDecimal2]('12342342', 9,2)
select convert(decimal(9,2),[dbo].[StringToDecimal]('1234234', 9,2))
select convert(decimal(9,2),[dbo].[StringToDecimal]('12342342', 9,2))
select convert(decimal(9,2),[dbo].[StringToDecimal]('123423.3333', 9,2))
select convert(decimal(20,10),[dbo].[StringToDecimal]('123423sd.3333', 20,10))
select convert(decimal(20,10),[dbo].[StringToDecimal]('123423sd..3333', 20,10))
select convert(decimal(20,10),[dbo].[StringToDecimal]('-123423.3333', 20,10))
select convert(decimal(20,10),[dbo].[StringToDecimal]('+123423..3333', 20,10))
Thanks for the extra information. It sounds like you have three steps:
Remove all characters from the string that are not digits or a decimal point (do you ever get multiple points in one string?)
Convert to (9,5) or (4,1) as appropriate (how do you decide this? is there rounding? does 10X.781 become 10.78100 or 10.7 or 10.8?)
Insert/update the final value somewhere
Based on point 1 alone, I would immediately avoid TSQL and think about an external script or CLR procedure. A CLR function could do the parsing, but you still have the problem of returning different data types.
Since this appears to be some kind of ETL task, in my environment I would probably implement it as a script component in an SSIS package. The component would do the parsing and send the clean data to different outputs for further processing. If it was a one-time task I would use a Python script to parse the input data and generate INSERT or UPDATE statements.
I don't know if any of those solutions are suitable for you, but maybe it'll give you some ideas. And you should probably avoid the ISNUMERIC() function; search this site or Google to find some of the 'strange' input that it considers to be numeric.