I have value A of type DECIMAL(19,8) - the scale is 8, so the number of decimal digits that will be stored to the right of the decimal point is 8.
Now, I am dividing A on B, where B is BIGINT. For, example:
SELECT CAST(3 AS DECIMAL(19, 8)) / CAST(27 AS BIGINT) -- 0.111111111111111111111111111
,CAST(300 AS DECIMAL(19, 8)) / CAST(27 AS BIGINT) -- 11.111111111111111111111111111
,CAST(75003 AS DECIMAL(19, 8)) / CAST(13664400 AS BIGINT) -- 0.005488934750153684025643277
the output values are with length: 29, 30, 29 respectively.
Could anyone tell why the length of the value for the three divisions is not 30? How the SQL Server is calculating the scale of the final result?
Argument 1: 3 AS DECIMAL(19, 8)
Argument 2: 27 AS DECIMAL (18, 0) -- default precision is 18, default scale is 0 (BIGINT was converted to DECIMAL due to type precedence)
p1 = 19
p2 = 18
s1 = 8
s2 = 0
max precision = (p1 - s1 + s2) + MAX(6, s1 + p2 + 1) -- up to 38
max scale = MAX(6, s1 + p2 + 1)
Let's calculate for example 1:
precision: (19 - 8 + 0) + MAX(6, 8 + 18 + 1) = 38
scale: MAX(6, 8 + 18 + 1) = 27
For all your examples you will get always max 27 scale.
0.111111111111111111111111111 (27)
11.111111111111111111111111111 (27)
0.005488934750153684025643277 (27)
The whole part takes only necessary digits (1), (2), (1).
For me everything is perfectly valid.
This answer is based on work of #Paul White from Decimal Truncation In division.
This is call Data Type Precedence.
When a query do something between different but yet compatible types, one of them has to be casted to the other type, eitheir with an explicit or implicit conversion.
If you look at Data Type Conversion (Database Engine)
, you will see that there is an implicit conversion between Decimal and Bigint.
Therefore you query does not requiere an explicit cast.
If you look at Data Type Precedence (Transact-SQL) on MSDN, you will see:
decimal
bigint
This means that decimal has a higher precedence than bigint and the bigint value will converted to decimal.
In the end, you calculation will be:
3,000... / 27,000...
300,000... / 27,000...
75003,000... / 27,000...
If you want it to be 3 / 27, you must do an explicit cast on the Decimal value.
Related
Can someone explain the following results. I have read about the SQL decimal type and the precision and scale amounts when doing multiplication and division, but I still can't understand this:
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) = 105.0000000000000
select cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) = 1.3636363636363
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) = 9.545454
select cast(cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) as decimal(25,13))/cast(11 as decimal(25,13)) = 9.5454545454545
So multiplication and division yield 13 decimals, but when chained they suddenly yield 6 decimals. Only when the multiplication is first cast as decimal(25,13) and only then divided, does it again yield 13 decimals. Does SQL Server cast the multiplication into the default decimal(38,0) and then divide this by the decimal(25,13)?
I am working on bom calculations for very low-priced products, and I want all the decimals I can get. Do I need to do this casting on every step or can I somehow set a default for a single query to use this decimal(25,13) on all decimals if not stated otherwise?
Below are relevant excerpts from the precision, scale, and length documentation for context of decimal multiplication and division behavior.
+--------------------------------------+-------------------------------------+---------------------+
| Operation | Result precision | Result scale * |
+--------------------------------------+-------------------------------------+---------------------+
| e1 * e2 | p1 + p2 + 1 | s1 + s2 |
| e1 / e2 | p1 - s1 + s2 + max(6, s1 + p2 + 1) | max(6, s1 + p2 + 1) |
+--------------------------------------+-------------------------------------+---------------------+
* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, it's reduced to 38, and the corresponding scale is reduced to try to prevent truncating the integral part of a result. In some cases such as multiplication or division, scale factor won't be reduced, to maintain decimal precision, although the overflow error can be raised.
In multiplication and division operations, we need precision - scale places to store the integral part of the result. The scale might be reduced using the following rules:
1. The resulting scale is reduced to min(scale, 38 - (precision-scale)) if the integral part is less than 32, because it can't be greater than 38 - (precision-scale). Result might be rounded in this case.
2. The scale won't be changed if it's less than 6 and if the integral part is greater than 32. In this case, overflow error might be raised if it can't fit into decimal(38, scale)
3. The scale will be set to 6 if it's greater than 6 and if the integral part is greater than 32. In this case, both integral part and scale would be reduced and resulting type is decimal(38,6). Result might be rounded to 6 decimal places or the overflow error will be thrown if the integral part can't fit into 32 digits.
Item #3 applies to the third query in your question, which can be observed with sp_describe_first_result_set:
EXEC sp_describe_first_result_set N'
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) --= 9.545454
';
The system_type_name from the above query shows decimal(38,6). Since the expression result scale of 6 is insufficient for your need, an explict CAST of the multiplication expression to a decimal type with a lower precision (like you did in the last query in your question) will provide a greater scale for the result type but with an increased risk of overflow.
The explict CAST to decimal(25, 13) results in a result type of decimal(38, 13):
EXEC sp_describe_first_result_set N'
select cast(
cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) as decimal(25,13))/cast(11 as decimal(25,13)) --= 9.5454545454545
';
I try to do multiplication and then convert to decimal
select cast(7*15 as decimal(25,13))/cast(11 as decimal(25,13))
A coworker came up with an internal ID that combines multiple different identifiers, all bitshifted together.
I'm trying to get out the "TotalSeconds" from this BIGINT using T-SQL. The only way I can think of offhand to do is to do an AND against the number that would be bits 21(?) to 50, and convert that back to an INT, and then use dateadd to get the time.
Specifically, this part:
StartUTC = new DateTime(BaseYear, 1, 1).ToUniversalTime();
TimeSpan timeSpan = DateTime.UtcNow - StartUTC;
...(one of the identifiers)
+ (((ulong) timeSpan.TotalSeconds % 1073741824L) << 20) + // 30 bits.
...(another of the identifiers)
Note: SQL Server does not cope gracefully with unsigned integers.
Division by a power of two is equivalent to right-shifting, e.g. Foo >> 3 in C# would be Foo / Power( 2, 3 ) in TSQL. It works with integer types up to 64-bits.
Bitwise AND (&) and modulus (%) can be used with powers of two, but they are not interchangeable. To extract the five least-significant-bits you could use either Foo % Power( 2, 5 ) or Foo & ( Power( 2, 5 ) - 1 ). The first returns the remainder after division by 0x20 and the second masks off the unwanted bits using a mask of 0x1F. There are two important differences to keep in mind: (1) bitwise AND is limited to 32-bit values while modulus works up to 64-bits and (2) if the original value is negative then the signs of the results differ:
select 10 % 4 as ModPos, 10 & 3 as AndPos, -10 % 4 as ModNeg, -10 & 3 as AndNeg;
Putting it all together, you can extract a bitfield of FieldWidth bits starting FieldOffset bits from the LSB of a BigInt value by dividing the original value to move the bits of interest to the least significant bits, then masking off any unwanted bits:
( OriginalValue / Power( 2, FieldOffset) ) % Power( 2, FieldWidth )
The original question involves converting the resulting NumberOfSeconds into a TimeSpan. The closest TSQL datatype is Time and it is limited to values less than 24 hours, i.e. a time-of-day. If the value is within the range of Time you can convert it thusly:
Cast( DateAdd( second, NumberOfSeconds, 0 ) as Time )
select 'foo ' + str(1.0/3.0, 15, 12) returns foo 0.333333000000. Is there a possibility to increase the decimal precision so that every digit after the decimal point is a 3?
You need to increase the accuracy of your input values. To quote the documentation when performing a division on a decimal the resulting precision is p1 - s1 + s2 + max(6, s1 + p2 + 1) and the resulting scale is max(6, s1 + p2 + 1).
In your expression, both 1.0 and 3.0 are a decimal(2,1). This means your resulting precision is 2 - 1 + 1 + max(6, 1 + 2 + 1) = 2 + max(6,5) = 2 + 6 = 8. For your scale, the result is max(6, 1 + 2 + 1) = max(6,5) = 6. Thus you're new datatype is a decimal(8,6). This results in the expression 1.0 / 3.0 = 0.333333.
You are then casting this value to a string, with a precision of 15 and a scale of 12. 0.333333 as a decimal(15,12) is 0.333333000000, as the precision has already been lost; SQL server doesn't remember that the value is technically 0.3~.
Thus, to get the answer you want, you need to add more decimal places to your intial values. For example:
SELECT 'foo ' + CONVERT(varchar(20),CONVERT(decimal(15,12),1.000000/3.0000000));
or, use a conversion
SELECT 'foo ' + CONVERT(varchar(20),CONVERT(decimal(15,12),CONVERT(decimal(15,12),1.0)/3.0));
Any questions, please do ask.
Is precision the reason I should cast both num and denominator as decimal so that it returns a decimal? And why does the first & second statement bring different precisions? Both only cast one part.
And instead of casting both to decimal(12,4), why just not cast the denominator to a higher precision?
For example:
select 3/cast(2 as decimal(12,4))
select cast(3 as decimal(12,4))/2
select cast(3 as decimal(12,4))/cast(2 as decimal(12,4))
select 3/cast(2 as decimal(16,4))
RETURNS
1.5000000000000
1.500000
1.50000000000000000
1.50000000000000000
This is related to Precision, Scale, and Length
Precision is the number of digits in a number.
Scale is the number of digits to the right of the decimal point in a number.
For example, the number 123.45 has a precision of 5 and a scale of 2.
The following table defines how the precision and scale of the result are calculated when the result of an operation is of type decimal. The result is decimal when either of the following is true:
Both expressions are decimal.
One expression is decimal and the other is a data type with a lower precedence than decimal.
The operand expressions are denoted as expression e1, with precision p1 and scale s1, and expression e2, with precision p2 and scale s2.
The precision and scale for any expression that is not decimal is the precision and scale defined for the data type of the expression.
Operation || Result precision || Result scale
e1 + e2 || max(s1, s2) + max(p1-s1, p2-s2) + 1 || max(s1, s2)
e1 - e2 || max(s1, s2) + max(p1-s1, p2-s2) + 1 || max(s1, s2)
e1 * e2 || p1 + p2 + 1 || s1 + s2
e1 / e2 || p1 - s1 + s2 + max(6, s1 + p2 + 1) || max(6, s1 + p2 + 1)
e1 % e2 || min(p1-s1, p2 -s2) + max( s1,s2 ) || max(s1, s2)
You can read more in this MSDN article
I read all rounding functions of T-SQL like Round, Floor, and Ceil, but none of them has rounded down decimal numbers correctly for me.
I have 2 questions:
How to round down a decimal number (3.69 ==> 3.5)?
How to round up the last 3 digits of an integer (e.g. 142600 ==> 143000)?
1) select CAST(FLOOR(2 * 3.69) / 2 AS decimal(2, 1)) handles the first case - courtesy of an answer to a similar question on SQL Server Forums, which I adapted and quickly checked.
Note that if the numbers you are rounding to the nearest 0.5 could be bigger (e.g. 333.69 => 333.5), be sure to specify more decimal precision when you cast (e.g. select CAST(FLOOR(2 * 3.69) / 2 AS decimal(10, 1))), or you could get an overflow error:
Msg 8115, Level 16, State 8, Line 1
Arithmetic overflow error converting numeric to data type numeric.
Extra precision will not affect the bottom-line result (i.e. select CAST(FLOOR(2 * 3.69) / 2 AS decimal(10, 1)) and select CAST(FLOOR(2 * 3.69) / 2 AS decimal(2, 1)) both yield 3.5); but it is wasteful if the numbers you are rounding will always be smaller.
Online references with examples are available for T-SQL FLOOR, CAST, and decimal to help.
2) select ROUND(142600, -3) handles the second case.
A similar online reference is available for T-SQL ROUND.
As per #J0e3gan 's anwser, Sql Server's Round allows rounding to the nearest powers of 10 using the length parameter, where length is 10^(-length), e.g.
length = 0 : 10 ^ 0 = nearest 1
length = 3 : 10 ^ -3 = nearest .001
length = -3 : 10 ^ 3 = nearest 1000
etc
However, in general, with a simple 1-based rounding function - e.g. (Sql Round with Length=0) to round to an arbitrary value of "nearest N" - with the formula:
round(X / N) * N
e.g. nearest 100
select round(12345 / 100.0, 0) * 100.0 -- 12300
select round(-9876 / 100.0, 0) * 100.0 -- -9900
select round(-9849 / 100.0, 0) * 100.0 -- -9800
... Nearest 0.5
select round(5.123 / 0.5, 0) * 0.5 -- 5.000
select round(6.499 / 0.5, 0) * 0.5 -- 6.500
select round(-4.499 / 0.5, 0) * 0.5 -- -4.50
... Nearest 0.02
select round(5.123 / .02, 0) * .02 -- 5.12
select round(-9.871 / .02, 0) * .02 -- -9.88
etc
Remember that the type used for the divisors must be numeric / decimal or float.
The Oracle/PLSQL FLOOR function returns the largest integer value that is equal to or less than a number.
eg:
FLOOR(7.9)
Result: 7
FLOOR(30.29)
Result: 30
FLOOR(-7.9)
Result: -8