Following batch statement if i execute in SSMS giving me Result like given below
DECLARE #M_TUBE_VOLUME NUMERIC(38,10),
#M_TUBE_OD NUMERIC(38,10)=12.50000,
#M_TUBE_ID NUMERIC(38,10)=12.50000,
#M_TUBE_LEN NUMERIC(38,10)=4000.00000,
#M_TUBE_COUNT NUMERIC(38,10)=212.4215000,
#M_S_TUBE_LEN NUMERIC(38,10)=0.0000,
#M_S_TUBE_COUNT NUMERIC(38,10)=3587.000
SET #M_TUBE_VOLUME=(SELECT 3.141592 / 4 * #M_TUBE_OD * #M_TUBE_ID * ((#M_TUBE_LEN * #M_TUBE_COUNT) + (#M_S_TUBE_LEN * #M_S_TUBE_COUNT)));
SELECT #M_TUBE_VOLUME
RESULT -- 104272138.7104680000
and if i execute same thing in SSMS using select statement
SELECT 3.141592 / 4 * 12.50000 * 12.50000 * ((4000.00000 * 212.4215000) + (0.0000 * 3587.000))
RESULT -- 104272138.285625000000000000000
Why two result are different any reason please help me
it has something to do with rounding. As your computation involved numbers with decimal places.
Please refer to documentation Precision, scale, and Length (Transact-SQL)
I have extracted the relevant portion here
Operation Result precision Result scale *
e1 * e2 p1 + p2 + 1 s1 + s2
e1 / e2 p1 - s1 + s2 + max(6, s1 + p2 + 1) max(6, s1 + p2 + 1)
The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, it's reduced to 38, and the
corresponding scale is reduced to try to prevent truncating the
integral part of a result. In some cases such as multiplication or
division, scale factor won't be reduced, to maintain decimal
precision, although the overflow error can be raised.
In your second query, numbers like 3.141592 is taking as numeric(7,6) and 12.50000 is numeric(7,5) and in your first query, all those are numeric(38,10)
The rounding occurs on the first 4 expression.
I use SELECT INTO a temp table and then query the data type of the column in the temp table
SELECT f1 = 3.141592 / 4 * #M_TUBE_OD * #M_TUBE_OD,
f2 = 3.141592 / 4 * 12.50000 * 12.50000
INTO #t
select c.name, t.name, c.precision, c.scale
from tempdb.sys.columns c
inner join master.sys.systypes t on c.system_type_id = t.xtype
where object_id = object_id('tempdb..#t')
name name percision scale
f1 numeric 38 6
f2 numeric 25 18
You can see that for your first query, the result scale is reduced to only 6. So for first query, the result of the first 4 expression is 122.718438 (rounded to 38) rather than 122.7184375.
You over define the precision of your numeric. Reduce it and try
In the second version, you haven't explicitly defined your data type and it has defaulted to FLOAT.
You can verify this by changing your query to
DECLARE #M_TUBE_VOLUME FLOAT,
#M_TUBE_OD FLOAT=12.50000,
#M_TUBE_ID FLOAT=12.50000,
#M_TUBE_LEN FLOAT=4000.00000,
#M_TUBE_COUNT FLOAT=212.4215000,
#M_S_TUBE_LEN FLOAT=0.0000,
#M_S_TUBE_COUNT FLOAT=3587.000
SET #M_TUBE_VOLUME=(SELECT 3.141592 / 4 * #M_TUBE_OD * #M_TUBE_ID * ((#M_TUBE_LEN * #M_TUBE_COUNT) + (#M_S_TUBE_LEN * #M_S_TUBE_COUNT)));
SELECT #M_TUBE_VOLUME
Related
When I run this simple operation in SQL server:
Select 800.0 /30.0
I get the value 26.666666, where even if it rounds for 6 digits it should be 26.666667.
How can I get the calculation to be accurate? I tried to search about it online and I found a solution where I cast each operand to a high precision decimal before the operation, but this will not be convenient for me because I have many long complex calculations. think there must be a better solution.
When a using division, in SQL Server, any digits after the resulting scale are truncated, not rounded. For your expression you have a decimal(4,1) and a decimal(3,1), which results in a decimal(10,6):
Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
Scale = max(6, s1 + p2 + 1)
As a result, 26.66666666666666~ is truncated to 26.666666.
You can get around this by can increasing the size of the precision and scale, and then CONVERT back to your required precision and scale. For example, increase the precision and scale of the decimal(3,1) to decimal(5,2) and convert back to a decimal(10,6):
SELECT CONVERT(decimal(10,6),800.0 / CONVERT(decimal(5,3),30.0));
This returns 26.666667.
This might helpful:
Use ROUND (Transact-SQL)
SELECT ROUND(800.0 /30.0, 5) AS RoundValue;
Result:
RoundValue
26.666670
I believe it's because SQL Server takes your numbers as decimal values (which are exact e.g., 6.6666 and 6.6667 means exactly those values, not 6 and two-thirds) rather than float values (which can work with approximate numbers).
If you explicity cast/convert it to a float at the start, you should get your calculations running smoothly.
Here's some examples to demonstrate the difference between int, decimal, and float calculations
Dividing 20 by 3
Dividing 20 by 3, then multiplying by 3 again (which mathematically should be 20).
SELECT (20/3) AS int_calc,
(20/3) * 3 AS int_calc_x3,
(CAST(20 AS decimal(10,3)) /3) AS dec_calc,
(CAST(20 AS decimal(10,3)) /3) * 3 AS dec_calc_x3,
(CAST(20 AS float) /3) AS float_calc,
(CAST(20 AS float) /3) * 3 AS float_calc_x3
with the following results
int_calc int_calc_x3 dec_calc dec_calc_x3 float_calc float_calc_x3
6 18 6.666666 19.999998 6.66666666666667 20
In your case, you can use
Select CAST(800.0 AS float) /30.0
which results in 26.6666666666667
Note if you then multiply back by 30, it gets the correct result e.g.,
Select (CAST(800.0 AS float) /30.0) * 30
results in 800. Solutions dealing with decimals will not have this.
Note also that once you have it as a float, then it should stay a float until converted back to a decimal or an int somehow (e.g., saved in a table as an int). So...
SELECT A.Num / 30
FROM (Select ((CAST(800.0 AS float) /30.0) * 30) AS Num) AS A
will still result in 26.6666666666667
This will hopefully help you in your long complex calculations.
This question already has an answer here:
Create a computed column and round
(1 answer)
Closed 3 years ago.
We used to have the following field in database:
FieldA decimal(19, 2)
We have changed the field to a computed field:
ALTER TABLE MyTable
ADD FieldA AS FieldB * FieldC
Where:
FieldB decimal(19, 2)
FieldC decimal(19, 5)
The resulting new FieldA:
FieldA (Computed, decimal(38,6))
The legacy code uses FieldAwith the assumption that it has two decimals. Thus, the string translations do not apply formatting. As a result, with the new definition, FieldA is shown with six decimals, which is unacceptable.
Is it possible to modify the precision of the computed column (say, to keep the original type decimal(19, 2)) or do we need to add appropriate formatting to all the places that are displaying the value?
Rounding does not work:
ADD FieldA AS ROUND(FieldB * FieldC, 2)
Try this:
CREATE TABLE [dbo].[TEST]
(
[A] DECIMAL(19,2)
,[B] DECIMAL(19,5)
,[C] AS [A] * [B]
,[D] AS CAST([A] * [B] AS DECIMAL(19,2))
);
GO
SELECT *
FROM [sys].[columns]
WHERE [object_id] = OBJECT_ID('[dbo].[TEST]');
Why? Check the formula of how decimal precision is set when operations are perform over decimal values.
So, we have:
[A] DECIMAL(19,2)
[B] DECIMAL(19,5)
[C] AS [A] * [B]
and the formula is:
p1 + p2 + 1 => 2 + 5 + 1 => 7
s1 + s2 => 19 + 19 = > 38
but:
In multiplication and division operations, we need precision - scale
places to store the integral part of the result. The scale might be
reduced using the following rules:
The resulting scale is reduced to min(scale, 38 - (precision-scale)) if the integral part is less than 32, because it
can't be greater than 38 - (precision-scale). Result might be rounded
in this case.
The scale won't be changed if it's less than 6 and if the integral part is greater than 32. In this case, overflow error might be raised
if it can't fit into decimal(38, scale)
The scale will be set to 6 if it's greater than 6 and if the integral part is greater than 32. In this case, both integral part and
scale would be reduced and resulting type is decimal(38,6). Result
might be rounded to 6 decimal places or the overflow error will be
thrown if the integral part can't fit into 32 digits.
and from rule 3 the scale is capped to 6 and you get DECIMAL(38,6). That's why you need to explicitly cast the result to your target decimal type.
Why doesn't this
select round(17 * .235, 2), round(17 * (1-.235), 2)
return
4.00 and 13.00
instead of
4.00 and 13.01
Also,
select CONVERT(Decimal(19,2), 17 * .235), CONVERT(Decimal(19,2), 17 * (1-.235))
returns
4.00 and 13.01
How can I force the two calculations to add up to the starting number (17.00)?
select cast(17 * .235 as float)
,cast(17 * (1-.235) as float)
,cast(17 * .235 as float)+cast(17 * (1-.235) as float)
Returns
(No column name) (No column name) (No column name)
3.995 13.005 17
If you don't like float, try decimal (some folks hate float, me not so much)
select cast(17 * .235 as decimal(18,9))
,cast(17 * (1-.235) as decimal(18,9))
,cast(17 * .235 as decimal(18,9))+cast(17 * (1-.235) as decimal(18,9))
I think the answer is that SQL rounds AWAY FROM ZERO. Which is different from VB's rounding (ROUND TOWARDS EVEN). At least that's the way I read this:
http://blogs.lessthandot.com/index.php/DataMgmt/DataDesign/sql-server-rounding-methods/
I have the following query:
DECLARE #A as numeric(36,14) = 480
DECLARE #B as numeric(36,14) = 1
select #B/#A
select cast(#B as decimal)/cast(#A as decimal)
Why does the first calculation returns 0.002083 and the second one returns 0.00208333333333333?
Isn´t numeric(36,14) good enough to have a good precision (just as the second query)?
If I use only numeric, instead of numeric(36,14), I have a good precision again:
select cast(#B as numeric)/cast(#A as numeric)
You can calculate precision and scale by yourself using this documentation from SQL Server Books online.
I tried to calculate precision and scale for your case (operation=division, p=36, s=14) and I got a pretty strange results...
precision of the result: [p1 - s1 + s2 + max(6, s1 + p2 + 1)] -> 36-14+14+max(6,14+36+1)=36+51=87
scale of the result : [max(6, s1 + p2 + 1)] -> max(6,14+36+1)=51
In this situation precision is greater than 38 and in this case (as stated in the documentation)
*The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is
reduced to prevent the integral part of a result from being truncated.
scale must be reduced by (87-38=) 49, that is (51-49=) 2 ...
I think that minimum scale length is 6 (because of expression scale=[max(6, s1 + p2 + 1)]) and it can't be reduced lower than 6 - that we have as a result (0.002083).
Just contributing for the understanding of the problem (going deeper on #Andrey answer), the things could be tricky, depending on the order of calculations.
Consider the variables:^
DECLARE #A as NUMERIC(36,19) = 100
DECLARE #B as NUMERIC(36,19) = 480
DECLARE #C as NUMERIC(36,19) = 100
Calculating A/B*C
If you want to calculate A/B*C, using the formulaes, we have:
A/B is of type NUMERIC(38,6) --> as calculated by #Andrey
The result will be 0.208333 (with scale of 6)
Multiplying by 100, we will get 20.833300
Calculating A*C/B
The result of A*C is 10000 of type NUMERIC(38,6). Diving by C, the result will be 20.833333 of type NUMERIC(38,6)
Then, the result may vary depending on the order of calculation (the same problem was pointed in https://dba.stackexchange.com/questions/77664/how-does-sql-server-determine-precision-scale).
I have a column that is varchar, and I am trying to convert it to numeric(8,2) to return it like this 1.00, but its returning it like this instead 1.000
Example:
Select
Convert(numeric(8,2),WholesaleCost) * 2.2)
FROM Table
This Returns 50.900
Seems like when I reduce the scale to numeric(8,1) instead of numeric(8,2) it returns how I want it, which is 50.90
Does anyone know why its doing this? Why its adding an additional scale digit to the result? Shouldnt it be 50.9 when i do numeric(8,1)?
Because you are doing numeric(8,2) * numeric(2,1) and the resultant datatype is numeric(11,3)
When multiplying two numeric numbers e1 * e2 the result has precision p1 + p2 + 1 and scale s1 + s2 as per Precision, Scale, and Length