Can someone explain the following results. I have read about the SQL decimal type and the precision and scale amounts when doing multiplication and division, but I still can't understand this:
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) = 105.0000000000000
select cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) = 1.3636363636363
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) = 9.545454
select cast(cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) as decimal(25,13))/cast(11 as decimal(25,13)) = 9.5454545454545
So multiplication and division yield 13 decimals, but when chained they suddenly yield 6 decimals. Only when the multiplication is first cast as decimal(25,13) and only then divided, does it again yield 13 decimals. Does SQL Server cast the multiplication into the default decimal(38,0) and then divide this by the decimal(25,13)?
I am working on bom calculations for very low-priced products, and I want all the decimals I can get. Do I need to do this casting on every step or can I somehow set a default for a single query to use this decimal(25,13) on all decimals if not stated otherwise?
Below are relevant excerpts from the precision, scale, and length documentation for context of decimal multiplication and division behavior.
+--------------------------------------+-------------------------------------+---------------------+
| Operation | Result precision | Result scale * |
+--------------------------------------+-------------------------------------+---------------------+
| e1 * e2 | p1 + p2 + 1 | s1 + s2 |
| e1 / e2 | p1 - s1 + s2 + max(6, s1 + p2 + 1) | max(6, s1 + p2 + 1) |
+--------------------------------------+-------------------------------------+---------------------+
* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, it's reduced to 38, and the corresponding scale is reduced to try to prevent truncating the integral part of a result. In some cases such as multiplication or division, scale factor won't be reduced, to maintain decimal precision, although the overflow error can be raised.
In multiplication and division operations, we need precision - scale places to store the integral part of the result. The scale might be reduced using the following rules:
1. The resulting scale is reduced to min(scale, 38 - (precision-scale)) if the integral part is less than 32, because it can't be greater than 38 - (precision-scale). Result might be rounded in this case.
2. The scale won't be changed if it's less than 6 and if the integral part is greater than 32. In this case, overflow error might be raised if it can't fit into decimal(38, scale)
3. The scale will be set to 6 if it's greater than 6 and if the integral part is greater than 32. In this case, both integral part and scale would be reduced and resulting type is decimal(38,6). Result might be rounded to 6 decimal places or the overflow error will be thrown if the integral part can't fit into 32 digits.
Item #3 applies to the third query in your question, which can be observed with sp_describe_first_result_set:
EXEC sp_describe_first_result_set N'
select cast(7 as decimal(25,13))*cast(15 as decimal(25,13))/cast(11 as decimal(25,13)) --= 9.545454
';
The system_type_name from the above query shows decimal(38,6). Since the expression result scale of 6 is insufficient for your need, an explict CAST of the multiplication expression to a decimal type with a lower precision (like you did in the last query in your question) will provide a greater scale for the result type but with an increased risk of overflow.
The explict CAST to decimal(25, 13) results in a result type of decimal(38, 13):
EXEC sp_describe_first_result_set N'
select cast(
cast(7 as decimal(25,13))*cast(15 as decimal(25,13)) as decimal(25,13))/cast(11 as decimal(25,13)) --= 9.5454545454545
';
I try to do multiplication and then convert to decimal
select cast(7*15 as decimal(25,13))/cast(11 as decimal(25,13))
Related
A coworker came up with an internal ID that combines multiple different identifiers, all bitshifted together.
I'm trying to get out the "TotalSeconds" from this BIGINT using T-SQL. The only way I can think of offhand to do is to do an AND against the number that would be bits 21(?) to 50, and convert that back to an INT, and then use dateadd to get the time.
Specifically, this part:
StartUTC = new DateTime(BaseYear, 1, 1).ToUniversalTime();
TimeSpan timeSpan = DateTime.UtcNow - StartUTC;
...(one of the identifiers)
+ (((ulong) timeSpan.TotalSeconds % 1073741824L) << 20) + // 30 bits.
...(another of the identifiers)
Note: SQL Server does not cope gracefully with unsigned integers.
Division by a power of two is equivalent to right-shifting, e.g. Foo >> 3 in C# would be Foo / Power( 2, 3 ) in TSQL. It works with integer types up to 64-bits.
Bitwise AND (&) and modulus (%) can be used with powers of two, but they are not interchangeable. To extract the five least-significant-bits you could use either Foo % Power( 2, 5 ) or Foo & ( Power( 2, 5 ) - 1 ). The first returns the remainder after division by 0x20 and the second masks off the unwanted bits using a mask of 0x1F. There are two important differences to keep in mind: (1) bitwise AND is limited to 32-bit values while modulus works up to 64-bits and (2) if the original value is negative then the signs of the results differ:
select 10 % 4 as ModPos, 10 & 3 as AndPos, -10 % 4 as ModNeg, -10 & 3 as AndNeg;
Putting it all together, you can extract a bitfield of FieldWidth bits starting FieldOffset bits from the LSB of a BigInt value by dividing the original value to move the bits of interest to the least significant bits, then masking off any unwanted bits:
( OriginalValue / Power( 2, FieldOffset) ) % Power( 2, FieldWidth )
The original question involves converting the resulting NumberOfSeconds into a TimeSpan. The closest TSQL datatype is Time and it is limited to values less than 24 hours, i.e. a time-of-day. If the value is within the range of Time you can convert it thusly:
Cast( DateAdd( second, NumberOfSeconds, 0 ) as Time )
I have value A of type DECIMAL(19,8) - the scale is 8, so the number of decimal digits that will be stored to the right of the decimal point is 8.
Now, I am dividing A on B, where B is BIGINT. For, example:
SELECT CAST(3 AS DECIMAL(19, 8)) / CAST(27 AS BIGINT) -- 0.111111111111111111111111111
,CAST(300 AS DECIMAL(19, 8)) / CAST(27 AS BIGINT) -- 11.111111111111111111111111111
,CAST(75003 AS DECIMAL(19, 8)) / CAST(13664400 AS BIGINT) -- 0.005488934750153684025643277
the output values are with length: 29, 30, 29 respectively.
Could anyone tell why the length of the value for the three divisions is not 30? How the SQL Server is calculating the scale of the final result?
Argument 1: 3 AS DECIMAL(19, 8)
Argument 2: 27 AS DECIMAL (18, 0) -- default precision is 18, default scale is 0 (BIGINT was converted to DECIMAL due to type precedence)
p1 = 19
p2 = 18
s1 = 8
s2 = 0
max precision = (p1 - s1 + s2) + MAX(6, s1 + p2 + 1) -- up to 38
max scale = MAX(6, s1 + p2 + 1)
Let's calculate for example 1:
precision: (19 - 8 + 0) + MAX(6, 8 + 18 + 1) = 38
scale: MAX(6, 8 + 18 + 1) = 27
For all your examples you will get always max 27 scale.
0.111111111111111111111111111 (27)
11.111111111111111111111111111 (27)
0.005488934750153684025643277 (27)
The whole part takes only necessary digits (1), (2), (1).
For me everything is perfectly valid.
This answer is based on work of #Paul White from Decimal Truncation In division.
This is call Data Type Precedence.
When a query do something between different but yet compatible types, one of them has to be casted to the other type, eitheir with an explicit or implicit conversion.
If you look at Data Type Conversion (Database Engine)
, you will see that there is an implicit conversion between Decimal and Bigint.
Therefore you query does not requiere an explicit cast.
If you look at Data Type Precedence (Transact-SQL) on MSDN, you will see:
decimal
bigint
This means that decimal has a higher precedence than bigint and the bigint value will converted to decimal.
In the end, you calculation will be:
3,000... / 27,000...
300,000... / 27,000...
75003,000... / 27,000...
If you want it to be 3 / 27, you must do an explicit cast on the Decimal value.
Here's the number I'm working on
1 01110 001 = ____
1 sign bit, 5 exp bits, 3 fraction bits
bias = 15
Here's my current process, hopefully you can tell me where I'm missing something
Convert binary exponent to decimal
01110 = 14
Subtract bias
14 - 15 = -1
Multiply fraction bits by result
0.001 * 2^-1 = 0.0001
Convert to decimal
.0001 = 1/16
The sign bit is 1 so my result is -1/16, however the given answer is -9/16. Would anyone mind explaining where the extra 8 in the fraction is coming from?
You seem to have the correct concept, including an understanding of the excess-N representation, but you're missing a crucial point.
The 3 bits used to encode the fractional part of the magnitude are 001, but there is an implicit 1. preceding the fraction bits, so the full magnitude is actually 1.001, which can be represented as an improper fraction as 1+1/8 => 9/8.
2^(-1) is the same as 1/(2^1), or 1/2.
9/8 * 1/2 = 9/16. Take the sign bit into account, and you arrive at the answer -9/16.
For normalized floating point representation, the Mantissa (fractional bits) = 1 + f. This is sometimes called an implied leading 1 representation. This is a trick for getting an additional bit of precision for free since we can always adjust the exponent E so that significant M is in the range 1<=M < 2 ...
You are almost correct but must take into consideration the implied 1. If it is denormalized (meaning the exponent bits are all 0s) you do not add an implied 1.
I would solve this problem as such...
1 01110 001
bias = 2^(k-1) -1 = 14
Exponent = e - bias
14 - 15 = -1
Take the fractional bits ->> 001
Add the implied 1 ->> 1.001
Shift it by the exponent, which is -1. Becomes .1001
Count up the values, 1(1/2) + 0(1/4) + 0(1/8) + 1(1/16) = 9/16
With the a negative sign bit it becomes -9/16
hope that helps!
I read all rounding functions of T-SQL like Round, Floor, and Ceil, but none of them has rounded down decimal numbers correctly for me.
I have 2 questions:
How to round down a decimal number (3.69 ==> 3.5)?
How to round up the last 3 digits of an integer (e.g. 142600 ==> 143000)?
1) select CAST(FLOOR(2 * 3.69) / 2 AS decimal(2, 1)) handles the first case - courtesy of an answer to a similar question on SQL Server Forums, which I adapted and quickly checked.
Note that if the numbers you are rounding to the nearest 0.5 could be bigger (e.g. 333.69 => 333.5), be sure to specify more decimal precision when you cast (e.g. select CAST(FLOOR(2 * 3.69) / 2 AS decimal(10, 1))), or you could get an overflow error:
Msg 8115, Level 16, State 8, Line 1
Arithmetic overflow error converting numeric to data type numeric.
Extra precision will not affect the bottom-line result (i.e. select CAST(FLOOR(2 * 3.69) / 2 AS decimal(10, 1)) and select CAST(FLOOR(2 * 3.69) / 2 AS decimal(2, 1)) both yield 3.5); but it is wasteful if the numbers you are rounding will always be smaller.
Online references with examples are available for T-SQL FLOOR, CAST, and decimal to help.
2) select ROUND(142600, -3) handles the second case.
A similar online reference is available for T-SQL ROUND.
As per #J0e3gan 's anwser, Sql Server's Round allows rounding to the nearest powers of 10 using the length parameter, where length is 10^(-length), e.g.
length = 0 : 10 ^ 0 = nearest 1
length = 3 : 10 ^ -3 = nearest .001
length = -3 : 10 ^ 3 = nearest 1000
etc
However, in general, with a simple 1-based rounding function - e.g. (Sql Round with Length=0) to round to an arbitrary value of "nearest N" - with the formula:
round(X / N) * N
e.g. nearest 100
select round(12345 / 100.0, 0) * 100.0 -- 12300
select round(-9876 / 100.0, 0) * 100.0 -- -9900
select round(-9849 / 100.0, 0) * 100.0 -- -9800
... Nearest 0.5
select round(5.123 / 0.5, 0) * 0.5 -- 5.000
select round(6.499 / 0.5, 0) * 0.5 -- 6.500
select round(-4.499 / 0.5, 0) * 0.5 -- -4.50
... Nearest 0.02
select round(5.123 / .02, 0) * .02 -- 5.12
select round(-9.871 / .02, 0) * .02 -- -9.88
etc
Remember that the type used for the divisors must be numeric / decimal or float.
The Oracle/PLSQL FLOOR function returns the largest integer value that is equal to or less than a number.
eg:
FLOOR(7.9)
Result: 7
FLOOR(30.29)
Result: 30
FLOOR(-7.9)
Result: -8
According to the entry for decimal and numeric data types in SQL Server 2008 Books Online, precision is:
p (precision)
The maximum total number of decimal digits that can be stored, both to the left and to the right of the decimal point. The precision must be a value from 1 through the maximum precision of 38. The default precision is 18.
However, the second select below fails with "Arithmetic overflow error converting int to data type numeric."
SELECT CAST(123456789 as decimal(9,0))
SELECT CAST(123456789 as decimal(9,1))
see here: http://msdn.microsoft.com/en-us/library/aa258832(SQL.80).aspx
decimal[(p[, s])]
p (precision) Specifies the maximum total number of decimal digits
that can be stored, both to the left
and to the right of the decimal point.
The precision must be a value from 1
through the maximum precision. The
maximum precision is 38. The default
precision is 18.
s (scale) Specifies the maximum number of decimal digits that can be
stored to the right of the decimal
point. Scale must be a value from 0
through p. Scale can be specified only
if precision is specified. The default
scale is 0; therefore, 0 <= s <= p.
Maximum storage sizes vary, based on
the precision.
when using: decimal(p,s), think of p as how many total digits (regardless of left or right of the decimal point) you want to store, and s as how many of those p digits should be to the right of the decimal point.
DECIMAL(10,5)= 12345.12345
DECIMAL(10,2)= 12345678.12
DECIMAL(10,10)= .1234567891
DECIMAL(11,10)= 1.1234567891
your sample code fails:
SELECT CAST(123456789 as decimal(9,1))
because:
9=precision (total number of digits to left and right of decimal)
1=scale (total number of digits to the right of the decimal)
(9-1)=8 (total digits to the left of the decimal)
and your value 123456789 requires 9 digits to the left of the decimal. you will need decimal(10,1) or just decimal(9,0)
Correct. Since you're doing decimal(9,1) that means you have 9 total digits, but the ,1 is reserving one of them for the right of the decimal place, so you can do at most 8 to the left and 1 to the right.
try
SELECT CAST(123456789 as decimal(10,1))