I'm working with a table that contains numeric data with decimals stored as VARCHAR. These numbers have an unpredictable number of decimals (up to 10 I believe) but they can also come in scientific notation ('1231E-03')
I've been trying different workarounds to deal with them (like what is explained here https://stackoverflow.com/a/18452603/5866637)
I have no issues converting regular decimal numbers, but when dealing with the exponential notation I'm unable to get rid of the floating point precision issues.
DECLARE #Value VARCHAR(MAX) = '123456789.123e-5'
SELECT CONVERT(Decimal(38,18), CONVERT(float, #Value))
Will be resolved as
1234.567891229999986535
Is it possible to do any transformation so I can convert the number above to the decimal
1234.567891230000000000
I've tried concatenating a bunch of zeroes to the number but when I have to do the casting to float, it eats all those extra zeroes.
The other thing I've tried is to reduce the precision of my decimal values. If I use Decimal(38,13) I get the expected value but, could that cause issues? If I know that I will never get a value in the database with more than 13 decimal points, is it safe to use convert to (38,13) or shall I pursue a different workaround?
Any help would be appreciated and sorry if this is considered duplicate, but the closest match I found is the link above and it was formulated 6 years ago.
You can use ROUND
DECLARE #Value VARCHAR(MAX) = '123456789.123e-5'
SELECT round(CONVERT(Decimal(38,18), CONVERT(float, #Value)), 8)
Related
Even though DECIMAL is an exact numeric type (unlike FLOAT, which is approximate), it behaves rather strangely in the following example:
DECLARE #DECIMAL_VALUE1 DECIMAL(20,9) = 504.70 / 0.151562
DECLARE #DECIMAL_VALUE2 DECIMAL(20,0) = 504.70 / 0.151562
DECLARE #INTEGER_VALUE INT = 504.70 / 0.151562
SELECT
#DECIMAL_VALUE1 AS DECIMAL_VALUE1, -- 3329.990366978
#DECIMAL_VALUE2 AS DECIMAL_VALUE2, -- 3330
#INTEGER_VALUE AS INTEGER_VALUE -- 3329
A value other than 3329 causes a bug in our application. Making the variable type an INTEGER solved our issue, but I cannot get my head around as to why it was caused in the first place.
You asked, "Why it was caused in the first place":
To know why you need to understand the nature of each datatype and how it operates within SQL Server.
Integer math truncates decimals (no rounding, same as "FLOOR" function (which is why you get 3329)).
Decimal with 0 places rounds (which is why you get 3330 as 3329.99 rounds up)
Decimal with precision/scale rounds to Nth scale (which is why you get 3329.990366978...).
So this isn't unexpected behavior, it's expected given the datatypes involved. It just may have been unanticipated behavior. The nuances of each datatype can be problematic until one runs into them.
I'll choose to ignore the float comment as it is not germane to the question.
I have a problem converting float to string but t-sql returns it with scientific format.
Example:
declare #amount float
set #amount=5223421341.23
print cast(#amount as nchar(20))
returns "5.22342e+009"
Well I tried the function STR but there is the point: I don't know how many decimals could have the float and I don't want to round it.
There is a way to return the float as nchar with the same precision as float is declared?
Thanks a lot.
There is a problem with the decimals places. The below code shows that the decimal value is distorted and I cant find a way around it. If there was a way to determine the precision of the float then the result could be rounded to a correct value.
With an unknown precision, I have yet to find a solution.
Declare #Amount float
Set #Amount=5223421341.23
Select LTrim(RTrim(Str(#Amount, 1000, 1000)))
Produces : 5223421341.2299995000000000
Based on what I am reading floats are only approximations by definition so this may be the best you can get.
https://msdn.microsoft.com/en-us/library/ms173773.aspx
http://www.webopedia.com/TERM/F/floating_point_number.html
Note that most floating-point numbers a computer can represent are
just approximations. One of the challenges in programming with
floating-point values is ensuring that the approximations lead to
reasonable results. If the programmeris not careful, small
discrepancies in the approximations can snowball to the point where
the final results become meaningless.
According to Microsoft, the best method in SQL is likely to use the STR function.
declare #amount float
set #amount=5223421341.23
print str(#amount,20,6)
print convert(VARCHAR(50),rtrim(ltrim(str(#amount,20,6))))
This seems like it would cover most scenarios, but you need to find out the max and min values in your data set and also the max precision.
According to Microsoft SQL Server documentation, (https://msdn.microsoft.com/en-us/library/ms173773.aspx) float is a synonym for float(53), and a float(53) has a precision of 15 digits.
But the 15 digits of precision say nothing about the exponent. The exponent is about 300. Which means that floats are capable of representing numbers containing about 300 zeros in them. That's why scientific notation is indispensable.
If scientific notation was not a problem for you, 21 characters would be enough to hold 15 digits with a period among them, plus the trailing e+999.
But since scientific notation is an issue for you, then you have to have a text field large enough to hold numbers of the form
0.0000000000000000000000000000000...00000000000000000000000000723237435435644
(that's almost 300 zeros followed by 15 digits)
and
377733453453453000000000000000000...00000000000000000000000000000000000000.0
(that's 15 digits followed by almost 300 zeros.)
So clearly, you are going to need some pretty huge text fields for what you are trying to do.
So, bottom line is, instead of looking for a solution to the problem as stated, I would recommend revising what you are trying to accomplish here.
I was performing some simple financial calculations in SQL Server when I discovered some odd behavior. I was trying to convert a string of numbers to a decimal type. While the string did not contain a decimal point, I knew from my specifications that the last 3 positions in the string were supposed to be behind the decimal point.
My first approach was flawed, but went something like this:
select convert(decimal(11,3),89456123/1000) as TotalUnits
This resulted in 89456.000. Performing the division before the cast resulted in the decimal parts being truncated.
So I moved the division operation outside the cast, like this:
select convert(decimal(11,3),89456123)/1000 as TotalUnits
This resulted in an explosion of positions after the decimal point. It returned 89456.12300000
According to my decimal specification, I wanted 11 digits, with 3 of them behind the decimal point. Now I have 13 total digits, with 8 behind the decimal. What happened?
To get what I want, I guess I have to double cast, like this:
select convert(decimal(11,3), convert(decimal(11,3),89456123)/1000)
which gives 89456.123.
It turns out no matter what I divide by, the resulting decimal point explosion is the same. Is the division converting the datatype into a double or something?
My question is this:
Why is this happening, and is there a more elegant way to compensate for it, instead of double-casting to decimal.
EDIT
I found this similar question on SO, but it looks like they are again double-casting.
SQL server does integer arithmetic, to force it to use numeric, you can multiply it by 1.0
No need of using convert twice. This gives 89456.123 with out double convert.
select convert(decimal(11,3),89456123*1.0/1000) as TotalUnits
Why does convert(decimal(11,3),89456123)/1000 end up with 6 decimal places? The rules demand it. numeric division has rather complicated rules about the resulting type.
When you say 1.0 you end up with a numeric with the least scale factors possible to represent this value:
SELECT SQL_VARIANT_PROPERTY(1.11, 'BaseType')
SELECT SQL_VARIANT_PROPERTY(1.11, 'Precision')
SELECT SQL_VARIANT_PROPERTY(1.11, 'Scale')
SELECT SQL_VARIANT_PROPERTY(1.11, 'TotalBytes')
What should you do? I think there is no really elegant solution because of the complicated rules. Any solution I can think of involves rather crazy type inference of intermediate results. I recommend pretty much the same solution that RADAR already gave:
select convert(decimal(11,3), convert(decimal(11, 3), 89456123)/1000) as TotalUnits
The main difference is that I think the *1.0 "trick" used as a short hand for a cast is obfuscating the meaning of the code. If you happen to like it feel free to use it, though.
select convert(decimal(11,3),89456123/CONVERT(decimal(11,3),1000))
I'm seeing some strange behavior when rounding in SQL Server 2008. Given the following code:
DECLARE #Value REAL
SELECT #Value = .35
SELECT ROUND(#Value, 1)
I would expect the value to be .4, however it outputs .3. I must assume this is because the value stored is actually less than .35, something like .34999999999999. Is this the case, or am I doing something wrong? Is there a way to ensure this behaves as expected, at least from the visible value?
When you are using floating-point values like REAL and FLOAT (same thing), the SQL Server ROUND() function follows IEEE Standard 754 and uses the "round up" algorithm.
But that means different things for different floating-point values. Some ".5" values end up getting stored as an approximation of ".49999999999", others as ".500000001", etc. It rounds up the value that is actually stored, not the value you gave it to begin with.
http://msdn.microsoft.com/en-us/library/ms187912.aspx
If exact decimal math matters to you, use DECIMAL, not FLOAT or REAL.
It's too bad the data is stored as REAL, but not all hope is lost. Convert the REAL to DECIMAL(10,2) before rounding it. That way the 0.3499999999999 (or whatever inaccurate value is being stored) will be rounded to .35, and then you'll round that to 0.4. You can even convert he result to DECIMAL(10,1) if you want it to be displayed as 0.4:
DECLARE #Value REAL
SELECT #Value = .35
SELECT CONVERT(DECIMAL(10,1), ROUND(CONVERT(DECIMAL(10,2), #Value), 1))
What datatype is the most efficient to store a number such as 11.111.
The numbers will have up 2 digits before the point and up to three after.
Currently the column is a 'bigint', I am guessing that this will need to be changed to a decimal or a float.
Any help much appreciated.
thanks
If you need exact representation, the smallmoney datatype uses 4 bytes.
This is equivalent to using an int (4 bytes), and performing multiplication/division to take care of the fractional digits.
I would think that you have two options: Multiply by 1000 and store as an int (and divide on the way out), or just store as a double. With memory and drive space being so cheap, it's probably not worth the hassle of optimizing any further.
You should use the DECIMAL data type. It stores exact floating point numbers
DECLARE #d decimal(5, 3)
SET #d=12.671
SELECT #d, DATALENGTH(#d)
As #Mitch pointed out, this would be 5 bytes in size. You pay an extra byte for easier calculations with the value.
Bigint is an integer datatype, so yeah, that won't work without some preprocessing footwork. Do you need to store the values precisely, or only approximately? The binary floating point types (float, real) are efficient and compact, but don't represent many decimal values exactly, and so comparisons won't always give you what you expect, and errors accumulate. decimal is a good bet. A decimal of precision 5 and scale 3 (meeting your requirements) consumes five bytes.
See documentation on the decimal datatype here, and datatypes in general here.
Use the decimal data type to set exact precision:
decimal(5,3)