I am just reading on the MSDN about Precision Handling.
Taken out from the table on this site:
Operation: e1 / e2
Result precision: p1 - s1 + s2 + max(6, s1 + p2 + 1)
Result scale: max(6, s1 + p2 + 1)
For the explanation of the used expressions:
The operand expressions are denoted as expression e1, with precision p1 and scale s1, and expression e2, with precision p2 and scale s2.
What I do not understand (more like I am not 100% sure I understand it) is this expression
max(6, s1 + p2 + 1)
Can someone explain it to me?
Many thanks :)
See my worked example here T-SQL Decimal Division Accuracy
It means maximum of 6 or (scale1 + precision2 + 1) for the scale of result
Related
Can someone explain the following results. I have read about the SQL decimal type and the precision and scale amounts when doing multiplication and division, but I still can't understand this:
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) = 105.0000000000000
select cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) = 1.3636363636363
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) = 9.545454
select cast(cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) as decimal(25,13))/cast(11 as decimal(25,13)) = 9.5454545454545
So multiplication and division yield 13 decimals, but when chained they suddenly yield 6 decimals. Only when the multiplication is first cast as decimal(25,13) and only then divided, does it again yield 13 decimals. Does SQL Server cast the multiplication into the default decimal(38,0) and then divide this by the decimal(25,13)?
I am working on bom calculations for very low-priced products, and I want all the decimals I can get. Do I need to do this casting on every step or can I somehow set a default for a single query to use this decimal(25,13) on all decimals if not stated otherwise?
Below are relevant excerpts from the precision, scale, and length documentation for context of decimal multiplication and division behavior.
+--------------------------------------+-------------------------------------+---------------------+
| Operation | Result precision | Result scale * |
+--------------------------------------+-------------------------------------+---------------------+
| e1 * e2 | p1 + p2 + 1 | s1 + s2 |
| e1 / e2 | p1 - s1 + s2 + max(6, s1 + p2 + 1) | max(6, s1 + p2 + 1) |
+--------------------------------------+-------------------------------------+---------------------+
* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, it's reduced to 38, and the corresponding scale is reduced to try to prevent truncating the integral part of a result. In some cases such as multiplication or division, scale factor won't be reduced, to maintain decimal precision, although the overflow error can be raised.
In multiplication and division operations, we need precision - scale places to store the integral part of the result. The scale might be reduced using the following rules:
1. The resulting scale is reduced to min(scale, 38 - (precision-scale)) if the integral part is less than 32, because it can't be greater than 38 - (precision-scale). Result might be rounded in this case.
2. The scale won't be changed if it's less than 6 and if the integral part is greater than 32. In this case, overflow error might be raised if it can't fit into decimal(38, scale)
3. The scale will be set to 6 if it's greater than 6 and if the integral part is greater than 32. In this case, both integral part and scale would be reduced and resulting type is decimal(38,6). Result might be rounded to 6 decimal places or the overflow error will be thrown if the integral part can't fit into 32 digits.
Item #3 applies to the third query in your question, which can be observed with sp_describe_first_result_set:
EXEC sp_describe_first_result_set N'
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) --= 9.545454
';
The system_type_name from the above query shows decimal(38,6). Since the expression result scale of 6 is insufficient for your need, an explict CAST of the multiplication expression to a decimal type with a lower precision (like you did in the last query in your question) will provide a greater scale for the result type but with an increased risk of overflow.
The explict CAST to decimal(25, 13) results in a result type of decimal(38, 13):
EXEC sp_describe_first_result_set N'
select cast(
cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) as decimal(25,13))/cast(11 as decimal(25,13)) --= 9.5454545454545
';
I try to do multiplication and then convert to decimal
select cast(7*15 as decimal(25,13))/cast(11 as decimal(25,13))
select 'foo ' + str(1.0/3.0, 15, 12) returns foo 0.333333000000. Is there a possibility to increase the decimal precision so that every digit after the decimal point is a 3?
You need to increase the accuracy of your input values. To quote the documentation when performing a division on a decimal the resulting precision is p1 - s1 + s2 + max(6, s1 + p2 + 1) and the resulting scale is max(6, s1 + p2 + 1).
In your expression, both 1.0 and 3.0 are a decimal(2,1). This means your resulting precision is 2 - 1 + 1 + max(6, 1 + 2 + 1) = 2 + max(6,5) = 2 + 6 = 8. For your scale, the result is max(6, 1 + 2 + 1) = max(6,5) = 6. Thus you're new datatype is a decimal(8,6). This results in the expression 1.0 / 3.0 = 0.333333.
You are then casting this value to a string, with a precision of 15 and a scale of 12. 0.333333 as a decimal(15,12) is 0.333333000000, as the precision has already been lost; SQL server doesn't remember that the value is technically 0.3~.
Thus, to get the answer you want, you need to add more decimal places to your intial values. For example:
SELECT 'foo ' + CONVERT(varchar(20),CONVERT(decimal(15,12),1.000000/3.0000000));
or, use a conversion
SELECT 'foo ' + CONVERT(varchar(20),CONVERT(decimal(15,12),CONVERT(decimal(15,12),1.0)/3.0));
Any questions, please do ask.
Is precision the reason I should cast both num and denominator as decimal so that it returns a decimal? And why does the first & second statement bring different precisions? Both only cast one part.
And instead of casting both to decimal(12,4), why just not cast the denominator to a higher precision?
For example:
select 3/cast(2 as decimal(12,4))
select cast(3 as decimal(12,4))/2
select cast(3 as decimal(12,4))/cast(2 as decimal(12,4))
select 3/cast(2 as decimal(16,4))
RETURNS
1.5000000000000
1.500000
1.50000000000000000
1.50000000000000000
This is related to Precision, Scale, and Length
Precision is the number of digits in a number.
Scale is the number of digits to the right of the decimal point in a number.
For example, the number 123.45 has a precision of 5 and a scale of 2.
The following table defines how the precision and scale of the result are calculated when the result of an operation is of type decimal. The result is decimal when either of the following is true:
Both expressions are decimal.
One expression is decimal and the other is a data type with a lower precedence than decimal.
The operand expressions are denoted as expression e1, with precision p1 and scale s1, and expression e2, with precision p2 and scale s2.
The precision and scale for any expression that is not decimal is the precision and scale defined for the data type of the expression.
Operation || Result precision || Result scale
e1 + e2 || max(s1, s2) + max(p1-s1, p2-s2) + 1 || max(s1, s2)
e1 - e2 || max(s1, s2) + max(p1-s1, p2-s2) + 1 || max(s1, s2)
e1 * e2 || p1 + p2 + 1 || s1 + s2
e1 / e2 || p1 - s1 + s2 + max(6, s1 + p2 + 1) || max(6, s1 + p2 + 1)
e1 % e2 || min(p1-s1, p2 -s2) + max( s1,s2 ) || max(s1, s2)
You can read more in this MSDN article
I have value A of type DECIMAL(19,8) - the scale is 8, so the number of decimal digits that will be stored to the right of the decimal point is 8.
Now, I am dividing A on B, where B is BIGINT. For, example:
SELECT CAST(3 AS DECIMAL(19, 8)) / CAST(27 AS BIGINT) -- 0.111111111111111111111111111
,CAST(300 AS DECIMAL(19, 8)) / CAST(27 AS BIGINT) -- 11.111111111111111111111111111
,CAST(75003 AS DECIMAL(19, 8)) / CAST(13664400 AS BIGINT) -- 0.005488934750153684025643277
the output values are with length: 29, 30, 29 respectively.
Could anyone tell why the length of the value for the three divisions is not 30? How the SQL Server is calculating the scale of the final result?
Argument 1: 3 AS DECIMAL(19, 8)
Argument 2: 27 AS DECIMAL (18, 0) -- default precision is 18, default scale is 0 (BIGINT was converted to DECIMAL due to type precedence)
p1 = 19
p2 = 18
s1 = 8
s2 = 0
max precision = (p1 - s1 + s2) + MAX(6, s1 + p2 + 1) -- up to 38
max scale = MAX(6, s1 + p2 + 1)
Let's calculate for example 1:
precision: (19 - 8 + 0) + MAX(6, 8 + 18 + 1) = 38
scale: MAX(6, 8 + 18 + 1) = 27
For all your examples you will get always max 27 scale.
0.111111111111111111111111111 (27)
11.111111111111111111111111111 (27)
0.005488934750153684025643277 (27)
The whole part takes only necessary digits (1), (2), (1).
For me everything is perfectly valid.
This answer is based on work of #Paul White from Decimal Truncation In division.
This is call Data Type Precedence.
When a query do something between different but yet compatible types, one of them has to be casted to the other type, eitheir with an explicit or implicit conversion.
If you look at Data Type Conversion (Database Engine)
, you will see that there is an implicit conversion between Decimal and Bigint.
Therefore you query does not requiere an explicit cast.
If you look at Data Type Precedence (Transact-SQL) on MSDN, you will see:
decimal
bigint
This means that decimal has a higher precedence than bigint and the bigint value will converted to decimal.
In the end, you calculation will be:
3,000... / 27,000...
300,000... / 27,000...
75003,000... / 27,000...
If you want it to be 3 / 27, you must do an explicit cast on the Decimal value.
I have the following code:
SELECT -701385.10 -- -701385.10
SELECT SUM(-701385.10) -- -701385.10
SELECT -701385.10/2889991754.89 -- -0.000242694498630
SELECT SUM(-701385.10)/2889991754.89 -- -0.000242
In the last SELECT the result is truncated to 6 decimal places. I've read through the Precision, Scale, and Length article and unless my working is wrong, I can't understand why the truncation is occurring. The type of the expression SUM(-701385.10) should be DECIMAL(38,2) - see SUM - so the type resulting from the division should have:
Precision:
p1 - s1 + s2 + max(6, s1 + p2 + 1)
38 - 2 + 2 + max(6, 2 + 10 + 1)
38 - max(6,13)
38 - 13
25
Scale:
max(6, s1 + p2 + 1)
max(6, 2 + 10 + 1)
max(6, 13)
13
So why are the decimal places being truncated?
Your working is wrong
Precision: p1 - s1 + s2 + max(6, s1 + p2 + 1)
Scale: max(6, s1 + p2 + 1)
Gives
Precision: 38 - 2 + 2 + max(6, 2 + 12 + 1) = 53
Scale: max(6, 2 + 12 + 1) = 15
Which is greater than 38 so you are getting truncation as covered here