As an example I found a simple calculation like:
select cast(200.00 as float) + 1908.30 + 170.00 + (-1150.00) + (-1128.30)
As a normal addition this results in 0.00 but SQL Server shows the result as 2.27373675443232E-13.
Why is this and how can I avoid this?
This error is inherent in the float datatype, and is the reason for the existence of decimal type. Never do money calculations as float values!
You can take a look at What Every Computer Scientist Should Know About Floating-Point Arithmetic, or any Google result for "float rounding error".
select cast(200.00 as decimal(10,2)) + 1908.30 + 170.00 + (-1150.00) + (-1128.30)
This is not unique to sql.
For example, in python:
'%.50f' % (200.+1908.3+170.-1150.-1128.30)
Will result in
'0.00000000000022737367544323205947875976562500000000'
Which is the same you got.
This is inherent to the inability to represent every number that can be expressed as a finite sum of powers of 10 (such as 1908.3) as a finite sum of positive and negative powers of 2, which is what representing it as a float does. An inevitable approximation occurs, and as you perform operations on said approximations, the final result may have a small error. The link provided by Amadan explains it.
Related
When I run this simple operation in SQL server:
Select 800.0 /30.0
I get the value 26.666666, where even if it rounds for 6 digits it should be 26.666667.
How can I get the calculation to be accurate? I tried to search about it online and I found a solution where I cast each operand to a high precision decimal before the operation, but this will not be convenient for me because I have many long complex calculations. think there must be a better solution.
When a using division, in SQL Server, any digits after the resulting scale are truncated, not rounded. For your expression you have a decimal(4,1) and a decimal(3,1), which results in a decimal(10,6):
Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
Scale = max(6, s1 + p2 + 1)
As a result, 26.66666666666666~ is truncated to 26.666666.
You can get around this by can increasing the size of the precision and scale, and then CONVERT back to your required precision and scale. For example, increase the precision and scale of the decimal(3,1) to decimal(5,2) and convert back to a decimal(10,6):
SELECT CONVERT(decimal(10,6),800.0 / CONVERT(decimal(5,3),30.0));
This returns 26.666667.
This might helpful:
Use ROUND (Transact-SQL)
SELECT ROUND(800.0 /30.0, 5) AS RoundValue;
Result:
RoundValue
26.666670
I believe it's because SQL Server takes your numbers as decimal values (which are exact e.g., 6.6666 and 6.6667 means exactly those values, not 6 and two-thirds) rather than float values (which can work with approximate numbers).
If you explicity cast/convert it to a float at the start, you should get your calculations running smoothly.
Here's some examples to demonstrate the difference between int, decimal, and float calculations
Dividing 20 by 3
Dividing 20 by 3, then multiplying by 3 again (which mathematically should be 20).
SELECT (20/3) AS int_calc,
(20/3) * 3 AS int_calc_x3,
(CAST(20 AS decimal(10,3)) /3) AS dec_calc,
(CAST(20 AS decimal(10,3)) /3) * 3 AS dec_calc_x3,
(CAST(20 AS float) /3) AS float_calc,
(CAST(20 AS float) /3) * 3 AS float_calc_x3
with the following results
int_calc int_calc_x3 dec_calc dec_calc_x3 float_calc float_calc_x3
6 18 6.666666 19.999998 6.66666666666667 20
In your case, you can use
Select CAST(800.0 AS float) /30.0
which results in 26.6666666666667
Note if you then multiply back by 30, it gets the correct result e.g.,
Select (CAST(800.0 AS float) /30.0) * 30
results in 800. Solutions dealing with decimals will not have this.
Note also that once you have it as a float, then it should stay a float until converted back to a decimal or an int somehow (e.g., saved in a table as an int). So...
SELECT A.Num / 30
FROM (Select ((CAST(800.0 AS float) /30.0) * 30) AS Num) AS A
will still result in 26.6666666666667
This will hopefully help you in your long complex calculations.
I'm trying to multiply two values. One is a decimal and one is numeric.
Example - Total is what I want:
Number Decimal Total
900 1.111 999.9
800 1.25 1000
460 4.25 1955
In my Sql query, I've tried the following:
(ISNUMERIC(UpgradeEmptyNodesPercentageLimitForAllocation) * RawTotalNodes) as ExpectedEmptyNodeCountForUpgrade
However, it always returns Number. How do the above?
Thanks...
Check the scale on your data types, if they are zero then SQL will remove the decimals
DECLARE #d1 DECIMAL(18,0) = 1.111
DECLARE #d2 DECIMAL(18,10) = 1.111
SELECT #d1,#d2
ISNUMERIC is really for evaluating strings, not number types -- and it only returns 0 or 1. I think what you want to do is a CAST of your numeric and/or decimal values to one with more precision, and then multiply. What is affecting the operation is the precision and scale of both factors to your operation, as-stored.
These types, by name, are actually interchangeable - but precision and scale are not.
Try casting both to a NUMERIC of some acceptabe precision and scale, and then multiplying. Or, if you don't have a precision and scale that will always work, then cast to REAL, if that's an option.
Read more on MSDN.
I am looking for a division result that is extremely accurate.
This SQL returns the following results:
SELECT (CAST(297282.26 AS DECIMAL(38, 30)) / CAST(495470.44 AS DECIMAL(38, 30))) AS ResultDecimal
SELECT (CAST(297282.26 AS FLOAT) / CAST(495470.44 AS FLOAT)) AS ResultFloat
Here is the accurate result from WolframAlpha:
http://www.wolframalpha.com/input/?i=297282.26%2F495470.44
I was under the impression that DECIMAL would be more accurate than FLOAT:
"Because of the approximate nature of the float and real data types, do not use these data types when exact numeric behavior is required, such as in financial applications, in operations involving rounding, or in equality checks. Instead, use the integer, decimal, money, or smallmoney data types."
https://technet.microsoft.com/en-us/library/ms187912(v=sql.105).aspx
Why does the FLOAT calculation give me a result more accurate than when using DECIMAL?
I found the best precision to be when you use:
SELECT (CAST(297282.26 AS DECIMAL(15, 9)) / CAST(495470.44 AS DECIMAL(24, 2))) AS ResultDecimal
This gives a result of
0.599999991926864496699338915153
I think the actual value (to 100 digits) is:
0.5999999919268644966993389151530412187657451370862810705720405842980259326873264124495499670979362562...
Please bear in mind SQL Server defines the maximum precision and scale for division as:
max precision = (p1 - s1 + s2) + MAX(6, s1 + p2 + 1) -- up to 38
max scale = MAX(6, s1 + p2 + 1)
Where p1 & p2 are the precision of the two numbers and s1 & s2 are the scale of the numbers.
In this case the maximum precision is (15-9+2) + MAX(6, 9+24+1) = 8 + 34 = 42.
However SQL Server only allows a maximum precision of 38.
The maximum scale = MAX(6, 9+24+1) = 34
Hopefully you already understand that just because the FLOAT version presents more numbers after the decimal point, doesn't necessarily mean that those are the true numbers. This is about precision, not accuracy.
It is the CAST function itself that causes this loss of precision, not the difference between the FLOAT and DECIMAL data types.
To demonstrate this, compare your previous results to the result of this:
SELECT 297282.26 / 495470.44 AS ResultNoCast
In my version of the query, the presence of a decimal point in the literal numbers tells SQL Server to treat the values as DECIMAL datatype, with appropriate length and precision as determined by the server. The result is more precise than when you CAST explicitly to DECIMAL.
A clue to the reason for this can be found hidden in the official documentation of the CAST function, under Truncating and Rounding Results:
When you convert data types that differ in decimal places, sometimes the result value is truncated and at other times it is rounded. The following table shows the behavior.
From | To | Behavior
numeric | numeric | Round
So the fact that each separate literal value is treated as a NUMERIC (same thing as DECIMAL) on the way in, and is being casted to NUMERIC, causes rounding.
Anticipating your next question a little, if you want a more precise result from the NUMERIC/DECIMAL datatype, you just need to tell SQL Server that each component of the calculation is more precise:
SELECT 297282.26000000 / 495470.44000000 AS ResultSuperPrecise
This appears (from experimentation) to be the most precise I can get: either adding or removing a 0 from either the numerator or denominator makes the result less precise. I'm at a loss to explain why that is, because the result is only 23 digits to the right of the decimal point.
It doesn't give you a more accurate result. I say that because the value is an approximate and not all values will be available to stored in a float. On the other side of that coin though is that float has the possibility of a lot more precision. The maximum precision of a decimal/numeric is 38. https://msdn.microsoft.com/en-us/library/ms187746.aspx
When you look at float though the maximum precision is 53. https://msdn.microsoft.com/en-us/library/ms173773.aspx
Okay, here is what I think is going on.
#philosophicles - I think you are right in that the CAST is causing the problem, but not because I am trying to "convert data types that differ in decimal places".
When I execute the following statement
SELECT CAST((297282.26 / 495470.44) AS DECIMAL(38, 30)) AS ResultDecimal
The accurate result for the calculation is
This has way more than 30 digits after the decimal point, and my data type has scale set to 30. So the CAST rounds the value, then just adds zeros to the end until there are 30 digits. We end up with this:
So the interesting thing is how does the CAST determine up to how many decimals to round or truncate the output? I am not sure, but as #philosophicles pointed out, the scale of the input effects the rounding applied on the output.
SELECT CAST(((297282.26/10000) / (495470.44/10000)) AS DECIMAL(38, 30)) AS ResultDecimal
Thoughts?
Also interesting:
However, in simple terms, precision is lost when the input scales are
high because the result scales need to be dropped to 38 with a
matching precision drop.
https://dba.stackexchange.com/questions/41743/automatic-decimal-rounding-issue
The precision and scale of the numeric data types besides decimal are fixed.
https://dba.stackexchange.com/questions/41743/automatic-decimal-rounding-issue
It is really strange how auto convert between numeric data behaves in T-Sql
Declare #fdays as float(12)
Declare #mAmount As Money
Declare #fDaysi as float(12)
Set #fdays =3
Set #fdaysi =1
Set #mAmount=527228.52
Set #mAmount = #fdaysi * #mAmount/#fDays
Select #mAmount, 527228.52/3
The result of this computation is
175742.8281 175742.840000
Does this occur because money and float are not actually the same kind of numeric data? Float is Approximate Numeric and Money is Exact Numeric
Money and Decimal are fixed numeric datatypes while Float is an
approximate numeric datatype. Results of mathematical operations on
floating point numbers can seem unpredictable, especially when
rounding is involved. Be sure you understand the significance of the
difference before you use Float!
Also, Money doesn't provide any advantages over Decimal. If fractional
units up to 5 decimal places are not valid in your currency or
database schema, just use Decimal with the appropriate precision and
scale.
ref link : http://www.sqlservercentral.com/Forums/Topic1408159-391-1.aspx
Should you choose the MONEY or DECIMAL(x,y) datatypes in SQL Server?
https://dba.stackexchange.com/questions/12916/datatypes-in-sql-server-difference-between-similar-dataypes-numeric-money
float [ (n) ]
Where n is the number of bits that are used to store the mantissa of the float number in scientific notation and, therefore, dictates the precision and storage size. If n is specified, it must be a value between 1 and 53. The default value of n is 53.
When n in 1-24 then precision is 7 digits.
When n in 25-53 then precision is 15 digits.
So in your example precision is 7 digits, thus first part #fdaysi * #mAmount
rounds result to 7 digits 527228.5. The second part returns 527228.5/3=175742.828 and casting 175742.828 to Money results in 175742.8281. So FLOAT and REAL are approximate data types and sometimes you get such surprises.
DECLARE #f AS FLOAT = '29545428.022495';
SELECT CAST(#f AS NUMERIC(28, 14)) AS value;
The result of this is 29545428.02249500200000 with just a casting.
I have really simply question about DECIMAL (and maybe NUMERIC) type in SQL Server 2008 R2.
MSDN said:
(scale)
The maximum number of decimal digits that can be stored to the right of the decimal point. Scale must be a value from 0 through p.
I understand this following way:
if I have DECIMAL(10, 5) - I am able to store 12345.12345 or 12345678.91.
if I have DECIMAL(5, 5) - I can have 12345 or 1234.5 or 1.2345, etc...
Is it clear?
But I got this error message:
SELECT CAST(2.8514 AS DECIMAL(5,5))
Arithmetic overflow error converting numeric to data type numeric.
I thought 5,5 means I can have up to 5 digits and up to 5 CAN BE right of the decimal point.
As I tried:
SELECT CAST(12.851 AS DECIMAL(6,5)) - overflows too
however
SELECT CAST(1.23456 AS DECIMAL(6,5)) - is OK.
So what's the truth?
DECIMAL(a,b) says that I can have up to a digits and JUST b of them are right to the decimal point (and there rest a-b to the left to the dec. point)?
I'm really confused about statement in doc which is copied everywhere. Please take a while and explain me this simple thing.
Lot of thanks!
The easiest way to think of it (for me) is that precision is the total number of digits, of which scale is the number of digits to the right of the decimal point. So DECIMAL(p,s) means p-s digits to the left of the point, and s digits to the right of the point.
That explains all the conversion errors you're seeing: the 2.8514 cannot be decimal(5,5) because p-s = 0; 12.851 cannot be decimal(6,5) because p-s = 1 and so on.