This question already has an answer here:
Create a computed column and round
(1 answer)
Closed 3 years ago.
We used to have the following field in database:
FieldA decimal(19, 2)
We have changed the field to a computed field:
ALTER TABLE MyTable
ADD FieldA AS FieldB * FieldC
Where:
FieldB decimal(19, 2)
FieldC decimal(19, 5)
The resulting new FieldA:
FieldA (Computed, decimal(38,6))
The legacy code uses FieldAwith the assumption that it has two decimals. Thus, the string translations do not apply formatting. As a result, with the new definition, FieldA is shown with six decimals, which is unacceptable.
Is it possible to modify the precision of the computed column (say, to keep the original type decimal(19, 2)) or do we need to add appropriate formatting to all the places that are displaying the value?
Rounding does not work:
ADD FieldA AS ROUND(FieldB * FieldC, 2)
Try this:
CREATE TABLE [dbo].[TEST]
(
[A] DECIMAL(19,2)
,[B] DECIMAL(19,5)
,[C] AS [A] * [B]
,[D] AS CAST([A] * [B] AS DECIMAL(19,2))
);
GO
SELECT *
FROM [sys].[columns]
WHERE [object_id] = OBJECT_ID('[dbo].[TEST]');
Why? Check the formula of how decimal precision is set when operations are perform over decimal values.
So, we have:
[A] DECIMAL(19,2)
[B] DECIMAL(19,5)
[C] AS [A] * [B]
and the formula is:
p1 + p2 + 1 => 2 + 5 + 1 => 7
s1 + s2 => 19 + 19 = > 38
but:
In multiplication and division operations, we need precision - scale
places to store the integral part of the result. The scale might be
reduced using the following rules:
The resulting scale is reduced to min(scale, 38 - (precision-scale)) if the integral part is less than 32, because it
can't be greater than 38 - (precision-scale). Result might be rounded
in this case.
The scale won't be changed if it's less than 6 and if the integral part is greater than 32. In this case, overflow error might be raised
if it can't fit into decimal(38, scale)
The scale will be set to 6 if it's greater than 6 and if the integral part is greater than 32. In this case, both integral part and
scale would be reduced and resulting type is decimal(38,6). Result
might be rounded to 6 decimal places or the overflow error will be
thrown if the integral part can't fit into 32 digits.
and from rule 3 the scale is capped to 6 and you get DECIMAL(38,6). That's why you need to explicitly cast the result to your target decimal type.
Related
When I run this simple operation in SQL server:
Select 800.0 /30.0
I get the value 26.666666, where even if it rounds for 6 digits it should be 26.666667.
How can I get the calculation to be accurate? I tried to search about it online and I found a solution where I cast each operand to a high precision decimal before the operation, but this will not be convenient for me because I have many long complex calculations. think there must be a better solution.
When a using division, in SQL Server, any digits after the resulting scale are truncated, not rounded. For your expression you have a decimal(4,1) and a decimal(3,1), which results in a decimal(10,6):
Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
Scale = max(6, s1 + p2 + 1)
As a result, 26.66666666666666~ is truncated to 26.666666.
You can get around this by can increasing the size of the precision and scale, and then CONVERT back to your required precision and scale. For example, increase the precision and scale of the decimal(3,1) to decimal(5,2) and convert back to a decimal(10,6):
SELECT CONVERT(decimal(10,6),800.0 / CONVERT(decimal(5,3),30.0));
This returns 26.666667.
This might helpful:
Use ROUND (Transact-SQL)
SELECT ROUND(800.0 /30.0, 5) AS RoundValue;
Result:
RoundValue
26.666670
I believe it's because SQL Server takes your numbers as decimal values (which are exact e.g., 6.6666 and 6.6667 means exactly those values, not 6 and two-thirds) rather than float values (which can work with approximate numbers).
If you explicity cast/convert it to a float at the start, you should get your calculations running smoothly.
Here's some examples to demonstrate the difference between int, decimal, and float calculations
Dividing 20 by 3
Dividing 20 by 3, then multiplying by 3 again (which mathematically should be 20).
SELECT (20/3) AS int_calc,
(20/3) * 3 AS int_calc_x3,
(CAST(20 AS decimal(10,3)) /3) AS dec_calc,
(CAST(20 AS decimal(10,3)) /3) * 3 AS dec_calc_x3,
(CAST(20 AS float) /3) AS float_calc,
(CAST(20 AS float) /3) * 3 AS float_calc_x3
with the following results
int_calc int_calc_x3 dec_calc dec_calc_x3 float_calc float_calc_x3
6 18 6.666666 19.999998 6.66666666666667 20
In your case, you can use
Select CAST(800.0 AS float) /30.0
which results in 26.6666666666667
Note if you then multiply back by 30, it gets the correct result e.g.,
Select (CAST(800.0 AS float) /30.0) * 30
results in 800. Solutions dealing with decimals will not have this.
Note also that once you have it as a float, then it should stay a float until converted back to a decimal or an int somehow (e.g., saved in a table as an int). So...
SELECT A.Num / 30
FROM (Select ((CAST(800.0 AS float) /30.0) * 30) AS Num) AS A
will still result in 26.6666666666667
This will hopefully help you in your long complex calculations.
I am trying to filter out some query results to where it only shows items with 6 decimal places. I don't need it to round up or add 0's to the answer, just filter out anything that is 5 decimal places or below. My current query looks like this: (ex. if item is 199.54215 i dont want to see it but if it is 145.253146 i need it returned)
select
TRA_CODPLANTA,
TRA_WO,
TRA_IMASTER,
tra_codtipotransaccion,
tra_Correlativo,
TRA_INGRESOFECHA,
abs(tra_cantidadparcial) as QTY
from mw_tra_transaccion
where FLOOR (Tra_cantidadparcial*100000) !=tra_cantidadparcial*100000
and substring(tra_imaster,1,2) not in ('CP','SG','PI','MR')
and TRA_CODPLANTA not in ('4Q' , '5C' , '5V' , '8H' , '7W' , 'BD', 'DP')
AND tra_INGRESOFECHA > #from_date
and abs(tra_cantidadparcial) > 0.00000
Any assistance would be greatly appreciated!
Here is an example with ROUND, which seems to be the ideal function to use, since it remains in the realms of numbers. If you have at most 5 decimal places, then rounding to 5 decimal places will leave the value unchanged.
create table #test (Tra_cantidadparcial decimal(20,10));
INSERT #test (Tra_cantidadparcial) VALUES (1),(99999.999999), (1.000001), (45.000001), (45.00001);
SELECT * FROM #test WHERE ROUND(Tra_cantidadparcial,5) != Tra_cantidadparcial;
drop table #test
If your database values are VARCHAR and exist in the DB like so:
100.123456
100.1
100.100
You can achieve this using a wildcard LIKE statement example
WHERE YOUR_COLUMN_NAME LIKE '%.[0-9][0-9][0-9][0-9][0-9][0-9]%'
This will being anything containing a decimal place followed by AT LEAST 6 numeric values
Here is an example using a conversion to varchar and using the LEN - the CHARINDEX of the decimal point, I'm not saying this is the best way, but you did ask for an example in syntax, so here you go:
--Temp Decimal value holding up to 10 decimal places and 10 whole number places
DECLARE #temp DECIMAL(20, 10) = 123.4565432135
--LEN returns an integer number of characters in the converted varchar
--CHARINDEX returns the integer location of the decimal where it is found in the varchar
--IF the number of characters left after subtracting the index of the decimal from the length of the varchar is greater than 5,
--you have more than 6 decimal places
IF LEN(CAST(#temp AS varchar(20))) - CHARINDEX('.', CAST(#temp AS varchar(20)), 0) > 5
SELECT 1
ELSE
SELECT 0
Here is a shorthand way.
WHERE (LEN(CONVERT(DOUBLE PRECISION, FieldName % 1)) - 2) >=5
One way would be to convert / cast that column to a lower precision. Doing this would cause automatic rounding, but that would show you if it is 6 decimals or not based on the last digit. If the last digit of the converted value is 0, then it's false, otherwise it's true.
declare #table table (v decimal(11,10))
insert into #table
values
(1.123456789),
(1.123456),
(1.123),
(1.123405678)
select
v
,cast(v as decimal(11,5)) --here, we are changing the value to have a precision of 5. Notice the rounding.
,right(cast(v as decimal(11,5)),1) --this is taking the last digit. If it's 0, we don't want it
from #table
Thus, your where clause would simply be.
where right(cast(tra_cantidadparcial as decimal(11,5)),1) > 0
I have a query:
use [Some_Database];
go
select toip 10 IDEVENT,
COUNT(*)
from [AL_PROTOCOL]
group by IDEVENT
order by Quanity desc
go
In output I get results as
IDEVENT Quanity
664 4,037787E+07
566 2,124254E+07
438 1,248467E+07
294 9926404
564 9777449
436 5784661
310 5709771
428 5161083
432 5154893
434 5150308
So, then I try to calculate second (*60 / 1,000,000) column using CONVERT (real (100), COUNT(*)*60/1000000, 2) as 'Size, Mb'
next way:
use [Some_Database];
go
select top 10 IDEVENT,
COUNT(*),
CONVERT (real (100), COUNT(*)*60/1000000, 2) as 'Size, Mb'
from [AL_PROTOCOL]
group by IDEVENT
order by Quanity desc
go
After this, I get this error
8115 'Arithmetic overflow error converting expression to int type
numeric'.
Technet explains, that this kind of errors occurs when a value in the column is overgrowing the maximal size of integer type. But, why doesn't it convert in REAL type?
Also, this question solved by replacing '*60/1,000,000' on '/1,000,000*60', but my interest is not satisfied.
What am I doing wrong? thanks
Look carefully at the order of operations you have asked the DBMS to perform. CONVERT (real (100), COUNT(*)*60/1000000, 2) means:
COUNT(*)
result * 60
result / 1000000
CONVERT (real (100), result, 2)
Until step 4, the value is still an integer, so you get an overflow error if COUNT(*) * 60 is higher than the maximum representable integer.
Because * and / have the same precedence, the workaround you found is CONVERT (real (100), COUNT(*)/1000000*60, 2), which means:
COUNT(*)
result / 1000000
result * 60
CONVERT (real (100), result, 2)
We're still doing maths on integers, but now we never overflow because we divide before multiplying.
But what you actually wanted was to do all the maths on the floating point value:
COUNT(*)
CONVERT (real (100), result, 2)
result * 60
result / 1000000
For that, you just need to swap the nesting around so that you convert the COUNT(*) result directly, and then apply the maths: CONVERT (real (100), COUNT(*), 2) * 60 / 1000000
If you multiply a really large number by 60, you're going to get an even bigger number, increasing the chances of an arithmetic overflow. The fact that you divide it by 1,000,000 after is too late, you've already tried to generate a number that is too big to proceed with.
However, when you divide it by 1,000,000 first before multiplying by 60, then you are always going to get a smaller number, so you should never get an arithmetic overflow.
Also, with your CONVERT, you are doing it on the entire result, and all of the numbers in the calculation are of INT types, so it will generate an INT before the conversion.
COUNT(*)*60/1000000 -- COUNT(*) is an INT, as well as the other numbers
If you change the ordering of the CONVERT \ CAST it should work.
DECLARE #number AS INT
SET #number = 123456789
-- this doesn't work
SELECT CAST((#number * 60) AS REAL(100))
-- this does
SELECT CAST(#number AS REAL(100)) * 60
Using SQL Server 2012...
I have two columns:
Price [decimal(28,12)]
OustandingShares [decimal(38,3)] -- The 38 is overkill but alas, not my call.
When I do an ALTER TABLE I get a resulting computed column as a [decimal(38,6)]. I need the datatype to be [decimal(28,12)].
ALTER TABLE [xyz].MyTable
ADD Mv AS OustandingShares * Price
How can I effectively get 12 decimals of scale on this computed column? I've tried doing convert on the OutstandingShares to 12 decimal places as well as wrapping a convert around the OutstandingShares * Price. The only thing I get is a computed field at [decimal(28,12)] with six trailing zeros.
Thoughts?
The Fix
This does what you want:
CONVERT(DECIMAL(28,12), (
CONVERT(DECIMAL(15, 3), [OustandingShares])
* CONVERT(DECIMAL(24, 12), [Price])
)
)
Test with this:
SELECT CONVERT(DECIMAL(28,12),
(CONVERT(DECIMAL(24,12), 5304.987781883689)
* CONVERT(DECIMAL(15,3), 3510.88)));
Result:
18625175.503659806036
The Reason
The computation is being truncated due to SQL Server's rules for how to handle Precision and Scale across various operations. These rules are detailed in the MSDN page for Precision, Scale, and Length. The details we are interested in for this case are:
Operation: e1 * e2
Result precision: p1 + p2 + 1
Result scale *: s1 + s2
Here the datatypes in play are:
DECIMAL(28, 12)
DECIMAL(38, 3)
This should result in:
Precision = (28 + 38 + 1) = 67
Scale = 15
But the max length of the DECIMAL type is 38. So what gives? We now need to notice that there was a footnote attached to the "Result scale" calculation, being:
* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.
So it seems that in order to get the Precision back down to 38 it chopped off 9 decimal places.
And this is why my proposed fix works. I kept the "Scale" values the same as we don't want to truncate going in and expanding them serves no purpose as SQL Server will expand the Scale as appropriate. The key is in reducing the Precision so that the truncation would be non-existent or at least minimal.
With DECIMAL(15, 3) and DECIMAL(24, 12) we should get:
Precision = (15 + 24 + 1) = 40
Scale = 15
40 is over the limit so reduce by 2 to get down to 38, which means reduce the Scale by 2 leaving us with a true "Result Scale" of 13, which is 1 more than we need and will even be seeing.
Use cast() or convert(). Something like:
ALTER TABLE [xyz].MyTable ADD Mv AS cast(OustandingShares * Price as decimal(12, 6)
or whatever type you want it to be.
EDIT:
Oh, I think I'm getting the idea. The problem is the calculation itself. In that case, do the conversion before the multiplication, so you don't have to depend on SQL Server's (arcane) rules for conforming decimal types.
ALTER TABLE [xyz].MyTable
ADD Mv AS cast(OustandingShares as decimal(28, 12) * cast(Price as decimal(28, 12))
I believe what is happening in your case is that the maximum precision on the calculated result exceeds the allowed thresholds, so the scale is reduced accordingly. This is explained at the bottom of this page.
I have the following query:
DECLARE #A as numeric(36,14) = 480
DECLARE #B as numeric(36,14) = 1
select #B/#A
select cast(#B as decimal)/cast(#A as decimal)
Why does the first calculation returns 0.002083 and the second one returns 0.00208333333333333?
Isn´t numeric(36,14) good enough to have a good precision (just as the second query)?
If I use only numeric, instead of numeric(36,14), I have a good precision again:
select cast(#B as numeric)/cast(#A as numeric)
You can calculate precision and scale by yourself using this documentation from SQL Server Books online.
I tried to calculate precision and scale for your case (operation=division, p=36, s=14) and I got a pretty strange results...
precision of the result: [p1 - s1 + s2 + max(6, s1 + p2 + 1)] -> 36-14+14+max(6,14+36+1)=36+51=87
scale of the result : [max(6, s1 + p2 + 1)] -> max(6,14+36+1)=51
In this situation precision is greater than 38 and in this case (as stated in the documentation)
*The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is
reduced to prevent the integral part of a result from being truncated.
scale must be reduced by (87-38=) 49, that is (51-49=) 2 ...
I think that minimum scale length is 6 (because of expression scale=[max(6, s1 + p2 + 1)]) and it can't be reduced lower than 6 - that we have as a result (0.002083).
Just contributing for the understanding of the problem (going deeper on #Andrey answer), the things could be tricky, depending on the order of calculations.
Consider the variables:^
DECLARE #A as NUMERIC(36,19) = 100
DECLARE #B as NUMERIC(36,19) = 480
DECLARE #C as NUMERIC(36,19) = 100
Calculating A/B*C
If you want to calculate A/B*C, using the formulaes, we have:
A/B is of type NUMERIC(38,6) --> as calculated by #Andrey
The result will be 0.208333 (with scale of 6)
Multiplying by 100, we will get 20.833300
Calculating A*C/B
The result of A*C is 10000 of type NUMERIC(38,6). Diving by C, the result will be 20.833333 of type NUMERIC(38,6)
Then, the result may vary depending on the order of calculation (the same problem was pointed in https://dba.stackexchange.com/questions/77664/how-does-sql-server-determine-precision-scale).