I'm puzzled by the lack of precision in this result:
select convert(numeric(11, 10), 1 / 3.0)
0.3333330000
Why only six decimal places when I asked for ten? Here I wrote 3.0 instead of 3 to force floating point division instead of integer division (the value of 1/3 is zero).
But what type is that constant 3.0? It doesn't seem to be any of the native SQL Server floating point types, which all give a different result:
select convert(numeric(11, 10), 1 / convert(real, 3))
0.3333333433
select convert(numeric(11, 10), 1 / convert(double precision, 3))
0.3333333333
select convert(numeric(11, 10), 1 / convert(float, 3))
0.3333333333
Until now I have tried to write 3.0 to get a floating point constant, as would happen in programming languages like C. But that isn't the interpretation in SQL. What's happening?
The answer is that 3.0 does not give a constant of floating point type, but one of numeric type, specifically numeric(2,1). You can see this using:
select sql_variant_property(3.0, 'basetype') as b,
sql_variant_property(3.0, 'precision') as p,
sql_variant_property(3.0, 'scale') as s
b: numeric
p: 2
s: 1
If you specify this type explictly you get the same odd limited-precision result:
select convert(numeric(11, 10), 1 / convert(numeric(2, 1), 3))
0.3333330000
It's still odd -- how did we end up with six decimal places, when the original numeric constant had only one and the output type wants ten? By the rules of numeric type promotion we can see that the division expression gets type numeric(8, 6):
select sql_variant_property(1 / convert(numeric(2, 1), 3), 'basetype') as b,
sql_variant_property(1 / convert(numeric(2, 1), 3), 'precision') as p,
sql_variant_property(1 / convert(numeric(2, 1), 3), 'scale') as s
b: numeric
p: 8
s: 6
Why exactly the reciprocal of something with one d.p. should get six d.p., rather than five or seven, is not clear to me, but I guess SQL has to pick something.
The moral of the story is not to write numeric constants like 1.0 in SQL expecting to get floating point semantics. You will get numeric semantics, but probably with too few decimal places to be useful. Write convert(1, float) or 1e instead if you want floating point, or else use numeric type and check that each step of the calculation has the number of d.p. you want.
Thanks to colleagues who pointed me to https://bertwagner.com/posts/data-type-precedence-and-implicit-conversions/ which explains some of this.
Related
I am experiencing a very strange behavior here with Microsoft SQL Server 2016 (SP2-CU15):
select convert(datetime, max(TS) + 1.0/24) as A
from table;
yields 2021-01-16 11:59:00.000
while
select convert(datetime, max(TS) + 1.0/24) as A
, dateadd(hour, 1, max(TS)) as B
from table;
gives me 2021-01-16 11:58:59.943 for A (and 2021-01-16 11:59:00.000 for B). So, it seems to me that adding the second column changes the result for the first?!
I can force the two-column version to work by casting 1.0 to real, btw: convert(datetime, max(TS) + cast(1.0 as real)/24), but I can not force the one-column version to fail by writing convert(datetime, max(TS) + cast(1.0 as float)/24).
Any ideas what's happening here?
Thanks!
Hendrik.
Update: As requested, here is a minimal example:
CREATE TABLE TestTS (TS FLOAT);
INSERT INTO TestTS (TS) VALUES (44210.4993055556);
SELECT convert(datetime, max(TS) + 1.0/24) as A
, dateadd(hour, 1, max(TS)) as B
from TestTS
As described, if you comment out the B-column, the value of A changes.
There's nothing wrong with DATEADD. The problem is the rest of the question.
First, there's a critical bug. Dates are stored as floats. An appropriate type should be used instead, eg datetime2, datetime or datetimeoffset. The best options are datetime2(0) or datetimeoffset(0), assuming no millisecond precision is needed.
datetime is essentially a legacy type, whose internal storage format is ... a float in the OADate format. That doesn't mean floats should be instead of the correct type though, no more than varbinary should be used instead of int or bigint.
Then, there's an attempt to add one hour to the OADate value, by calculating the floating point value of 1 hour in that format, 1/24. That's an irrational number though (0.04166666666....) which means that rounding errors always result in an inaccurate value.
Solution
The real solution is to use the correct type and DATEADD, eg :
CREATE TABLE TestTS (TS datetime2(0));
INSERT INTO TestTS (TS) VALUES ('2021-01-16 10:59:00.000');
SELECT dateadd(hour, 1, max(TS)) as B
from TestTS
If you want millisecond precision, use datetime2(3).
Getting the hack to work.
If you used datetime you wouldn't need to convert to datetime in the end, but the result would still be imprecise. This :
declare #TestTS table (TS datetime);
INSERT INTO #TestTS (TS) VALUES ('2021-01-16 10:59:00.000');
SELECT max(ts)+ (1.0/24)
from #TestTS
Produces 2021-01-16 11:58:59.943. The only reason the hack looked to be working in the first place was probably due to rounding errors during conversion.
The only way to get a correct result by adding floating point numbers is to increase precision to 8 fractional digits :
declare #TestTS table (TS datetime);
INSERT INTO #TestTS (TS) VALUES ('2021-01-16 10:59:00.000');
SELECT max(ts)+ (1.00000/24)--, dateadd(hour, 1, max(TS)) as B
from #TestTS
That produces 2021-01-16 11:59:00.000.
1.0 is a decimal(2,1). T-SQL calculates the fractional digits of decimal division based on the functional digits of the operands. If the operands have up to 4 fractional digits, the result will have 6 fractional digits, which isn't enough. 1 digit is added for any fractional digit above 4. 1.00000 results in 8 fractional digits 0.04166666
Don't do this though.
Cause
Thanks to #MartinSmith for the clue.
The cause is query auto-parameterization and the data types being chosen to store values.
Query 1 is auto-parameterized:
StatementText="SELECT CONVERT([datetime],MAX([TS])+#1/#2)
....
<ColumnReference Column="#2" ParameterCompiledValue="(24)" ParameterRuntimeValue="(24)" />
<ColumnReference Column="#1" ParameterCompiledValue="(1.0)" ParameterRuntimeValue="(1.0)" />
Query 2 is not auto-parameterized:
StatementText="SELECT convert(datetime, max(TS) + 1.0/24) as A...."
Why it happens is the first query and not the second query is a bit of a black magic.
From SQL Server data types page:
When you use the +, -, *, /, or % arithmetic operators to perform
implicit or explicit conversion of int, smallint, tinyint, or bigint
constant values to the float, real, decimal or numeric data types, the
rules that SQL Server applies when it calculates the data type and
precision of the expression results differ depending on whether the
query is autoparameterized or not.
Therefore, similar expressions in queries can sometimes produce
different results. When a query is not autoparameterized, the constant
value is first converted to numeric, whose precision is just large
enough to hold the value of the constant, before converting to the
specified data type. For example, the constant value 1 is converted to
numeric (1, 0), and the constant value 250 is converted to numeric (3, 0).
When a query is autoparameterized, the constant value is always
converted to numeric (10, 0) before converting to the final data
type. When the / operator is involved, not only can the result type's
precision differ among similar queries, but the result value can
differ also. For example, the result value of an autoparameterized
query that includes the expression SELECT CAST (1.0 / 7 AS float)
will differ from the result value of the same query that is not
autoparameterized, because the results of the autoparameterized query
will be truncated to fit into the numeric (10, 0) data type.
Effect
Based on the above, the following data types are used (refer to See: Precision, scale, and Length (Transact-SQL) for explanation of how result types are calculated):
Query 1 gives higher precision:
NUMERIC( 2, 1 ) / NUMERIC( 10, 0 ) = NUMERIC( 13, 12 )
Query 2:
NUMERIC( 2, 1 ) / NUMERIC( 2, 0 ) = NUMERIC( 7, 6 )
Solution
Cast your literals and / or intermediate results to the desired type to avoid surprises.
In your specific case, best solution is not to use number arithmetic to manipulate dates as Panagiotis Kanavos explains in his answer.
Alternatively, forcing float data types (per Dan Guzman comment) convert(datetime, max(TS) + 1e/24) would do the trick as well.
This question deals with the same issue.
When I run this simple operation in SQL server:
Select 800.0 /30.0
I get the value 26.666666, where even if it rounds for 6 digits it should be 26.666667.
How can I get the calculation to be accurate? I tried to search about it online and I found a solution where I cast each operand to a high precision decimal before the operation, but this will not be convenient for me because I have many long complex calculations. think there must be a better solution.
When a using division, in SQL Server, any digits after the resulting scale are truncated, not rounded. For your expression you have a decimal(4,1) and a decimal(3,1), which results in a decimal(10,6):
Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
Scale = max(6, s1 + p2 + 1)
As a result, 26.66666666666666~ is truncated to 26.666666.
You can get around this by can increasing the size of the precision and scale, and then CONVERT back to your required precision and scale. For example, increase the precision and scale of the decimal(3,1) to decimal(5,2) and convert back to a decimal(10,6):
SELECT CONVERT(decimal(10,6),800.0 / CONVERT(decimal(5,3),30.0));
This returns 26.666667.
This might helpful:
Use ROUND (Transact-SQL)
SELECT ROUND(800.0 /30.0, 5) AS RoundValue;
Result:
RoundValue
26.666670
I believe it's because SQL Server takes your numbers as decimal values (which are exact e.g., 6.6666 and 6.6667 means exactly those values, not 6 and two-thirds) rather than float values (which can work with approximate numbers).
If you explicity cast/convert it to a float at the start, you should get your calculations running smoothly.
Here's some examples to demonstrate the difference between int, decimal, and float calculations
Dividing 20 by 3
Dividing 20 by 3, then multiplying by 3 again (which mathematically should be 20).
SELECT (20/3) AS int_calc,
(20/3) * 3 AS int_calc_x3,
(CAST(20 AS decimal(10,3)) /3) AS dec_calc,
(CAST(20 AS decimal(10,3)) /3) * 3 AS dec_calc_x3,
(CAST(20 AS float) /3) AS float_calc,
(CAST(20 AS float) /3) * 3 AS float_calc_x3
with the following results
int_calc int_calc_x3 dec_calc dec_calc_x3 float_calc float_calc_x3
6 18 6.666666 19.999998 6.66666666666667 20
In your case, you can use
Select CAST(800.0 AS float) /30.0
which results in 26.6666666666667
Note if you then multiply back by 30, it gets the correct result e.g.,
Select (CAST(800.0 AS float) /30.0) * 30
results in 800. Solutions dealing with decimals will not have this.
Note also that once you have it as a float, then it should stay a float until converted back to a decimal or an int somehow (e.g., saved in a table as an int). So...
SELECT A.Num / 30
FROM (Select ((CAST(800.0 AS float) /30.0) * 30) AS Num) AS A
will still result in 26.6666666666667
This will hopefully help you in your long complex calculations.
Round functions has two behaviors: With the value cours is equal to "3.1235", round(cours, 3) = 3.123. Although, when we replace cours by its value (3.1235) in this round formula, round(3.1235, 3) = 3.1240.
You shouldn't be using the float datatype for specific decimal values like this as it's not designed for that purpose. Would need to see more of your code to get a better context of what you're trying to do, but if it needs to be a float initially, potentially you could cast #cours as decimal?
round(cast(#cours as decimal(5,4)), 3)
Your FLOAT does not really contain 3.1235, that is only what is printed or shown in a grid. Internally the FLOAT is 3.1234999999999999, which is obviously rounded down to 3.123.
The literal 3.1235 becomes a NUMERIC with enough precision to be totally exact, and so it is rounded up to 3.124, as one would expect.
Proof:
SELECT CAST('3.1235' as FLOAT),
CAST( 3.1235 as FLOAT)
-- misleading output: both print 3.1235
SELECT CAST(CAST('3.1235' as FLOAT) as NUMERIC(24,23)),
CAST(CAST( 3.1235 as FLOAT) as NUMERIC(24,23))
-- both print 3.12349999999999990000000
SELECT CAST('3.1235' as NUMERIC(24,23)),
CAST( 3.1235 as NUMERIC(24,23))
-- both print 3.12350000000000000000000
I am looking for a division result that is extremely accurate.
This SQL returns the following results:
SELECT (CAST(297282.26 AS DECIMAL(38, 30)) / CAST(495470.44 AS DECIMAL(38, 30))) AS ResultDecimal
SELECT (CAST(297282.26 AS FLOAT) / CAST(495470.44 AS FLOAT)) AS ResultFloat
Here is the accurate result from WolframAlpha:
http://www.wolframalpha.com/input/?i=297282.26%2F495470.44
I was under the impression that DECIMAL would be more accurate than FLOAT:
"Because of the approximate nature of the float and real data types, do not use these data types when exact numeric behavior is required, such as in financial applications, in operations involving rounding, or in equality checks. Instead, use the integer, decimal, money, or smallmoney data types."
https://technet.microsoft.com/en-us/library/ms187912(v=sql.105).aspx
Why does the FLOAT calculation give me a result more accurate than when using DECIMAL?
I found the best precision to be when you use:
SELECT (CAST(297282.26 AS DECIMAL(15, 9)) / CAST(495470.44 AS DECIMAL(24, 2))) AS ResultDecimal
This gives a result of
0.599999991926864496699338915153
I think the actual value (to 100 digits) is:
0.5999999919268644966993389151530412187657451370862810705720405842980259326873264124495499670979362562...
Please bear in mind SQL Server defines the maximum precision and scale for division as:
max precision = (p1 - s1 + s2) + MAX(6, s1 + p2 + 1) -- up to 38
max scale = MAX(6, s1 + p2 + 1)
Where p1 & p2 are the precision of the two numbers and s1 & s2 are the scale of the numbers.
In this case the maximum precision is (15-9+2) + MAX(6, 9+24+1) = 8 + 34 = 42.
However SQL Server only allows a maximum precision of 38.
The maximum scale = MAX(6, 9+24+1) = 34
Hopefully you already understand that just because the FLOAT version presents more numbers after the decimal point, doesn't necessarily mean that those are the true numbers. This is about precision, not accuracy.
It is the CAST function itself that causes this loss of precision, not the difference between the FLOAT and DECIMAL data types.
To demonstrate this, compare your previous results to the result of this:
SELECT 297282.26 / 495470.44 AS ResultNoCast
In my version of the query, the presence of a decimal point in the literal numbers tells SQL Server to treat the values as DECIMAL datatype, with appropriate length and precision as determined by the server. The result is more precise than when you CAST explicitly to DECIMAL.
A clue to the reason for this can be found hidden in the official documentation of the CAST function, under Truncating and Rounding Results:
When you convert data types that differ in decimal places, sometimes the result value is truncated and at other times it is rounded. The following table shows the behavior.
From | To | Behavior
numeric | numeric | Round
So the fact that each separate literal value is treated as a NUMERIC (same thing as DECIMAL) on the way in, and is being casted to NUMERIC, causes rounding.
Anticipating your next question a little, if you want a more precise result from the NUMERIC/DECIMAL datatype, you just need to tell SQL Server that each component of the calculation is more precise:
SELECT 297282.26000000 / 495470.44000000 AS ResultSuperPrecise
This appears (from experimentation) to be the most precise I can get: either adding or removing a 0 from either the numerator or denominator makes the result less precise. I'm at a loss to explain why that is, because the result is only 23 digits to the right of the decimal point.
It doesn't give you a more accurate result. I say that because the value is an approximate and not all values will be available to stored in a float. On the other side of that coin though is that float has the possibility of a lot more precision. The maximum precision of a decimal/numeric is 38. https://msdn.microsoft.com/en-us/library/ms187746.aspx
When you look at float though the maximum precision is 53. https://msdn.microsoft.com/en-us/library/ms173773.aspx
Okay, here is what I think is going on.
#philosophicles - I think you are right in that the CAST is causing the problem, but not because I am trying to "convert data types that differ in decimal places".
When I execute the following statement
SELECT CAST((297282.26 / 495470.44) AS DECIMAL(38, 30)) AS ResultDecimal
The accurate result for the calculation is
This has way more than 30 digits after the decimal point, and my data type has scale set to 30. So the CAST rounds the value, then just adds zeros to the end until there are 30 digits. We end up with this:
So the interesting thing is how does the CAST determine up to how many decimals to round or truncate the output? I am not sure, but as #philosophicles pointed out, the scale of the input effects the rounding applied on the output.
SELECT CAST(((297282.26/10000) / (495470.44/10000)) AS DECIMAL(38, 30)) AS ResultDecimal
Thoughts?
Also interesting:
However, in simple terms, precision is lost when the input scales are
high because the result scales need to be dropped to 38 with a
matching precision drop.
https://dba.stackexchange.com/questions/41743/automatic-decimal-rounding-issue
The precision and scale of the numeric data types besides decimal are fixed.
https://dba.stackexchange.com/questions/41743/automatic-decimal-rounding-issue
Using T-SQL and Microsoft SQL Server I would like to specify the number of decimal digits when I do a division between 2 integer numbers like:
select 1/3
That currently returns 0. I would like it to return 0,33.
Something like:
select round(1/3, -2)
But that doesn't work. How can I achieve the desired result?
The suggestions from stb and xiowl are fine if you're looking for a constant. If you need to use existing fields or parameters which are integers, you can cast them to be floats first:
SELECT CAST(1 AS float) / CAST(3 AS float)
or
SELECT CAST(MyIntField1 AS float) / CAST(MyIntField2 AS float)
Because SQL Server performs integer division. Try this:
select 1 * 1.0 / 3
This is helpful when you pass integers as params.
select x * 1.0 / y
It's not necessary to cast both of them. Result datatype for a division is always the one with the higher data type precedence. Thus the solution must be:
SELECT CAST(1 AS float) / 3
or
SELECT 1 / CAST(3 AS float)
use
select 1/3.0
This will do the job.
I understand that CASTing to FLOAT is not allowed in MySQL and will raise an error when you attempt to CAST(1 AS float) as stated at MySQL dev.
The workaround to this is a simple one. Just do
(1 + 0.0)
Then use ROUND to achieve a specific number of decimal places like
ROUND((1+0.0)/(2+0.0), 3)
The above SQL divides 1 by 2 and returns a float to 3 decimal places, as in it would be 0.500.
One can CAST to the following types: binary, char, date, datetime, decimal, json, nchar, signed, time, and unsigned.
Looks like this trick works in SQL Server and is shorter (based in previous answers)
SELECT 1.0*MyInt1/MyInt2
Or:
SELECT (1.0*MyInt1)/MyInt2
Use this
select cast((1*1.00)/3 AS DECIMAL(16,2)) as Result
Here in this sql first convert to float or multiply by 1.00 .Which output will be a float number.Here i consider 2 decimal places. You can choose what you need.
If you came here (just like me) to find the solution for integer value, here is the answer:
CAST(9/2 AS UNSIGNED)
returns 5
I was surprised to see select 0.7/0.9 returning 0.8 in Teradata given they're already as floats/decimal numbers! I had to do cast(0.7 as float) to get the output that I was after.
When using literals, the best way is to "tell" SQL
which type you mean.
if you want a decimal result, add decimal point ".0" to your numbers:
SELECT 1.0 / 3.0
Result
0.333333
if you want a float (real) result, add "e0" to your numbers:
SELECT 1e0 / 3e0
Result
0.333333333333333