Basically, two queries that do a similar averaging, are giving me different results; the "actual" value expected is 81.25, so with normal rounding, you would think that CAST-ing it to DECIMAL(10,1) would result in 81.3, but one of the queries (the first one) gives 81.2
Here's the examples:
CAST( AVG( t1.NumCorrect / 7.0 * 100.0 ) AS decimal(10,1))
vs.
CAST( AVG( t2.PercentCorrect ) AS decimal(10,1))
The only difference, as far as I can tell, is that PercentCorrect is type REAL, and NumCorrect is type FLOAT.. but those are both floating-point types, so why would casting them behave differently?
In the first line, NumCorrect is like a "# correct out of 7 possible", so I'm calculating a "Percent" on-the-fly, which is then AVG'd; whereas in the 2nd example, the Percent is pre-calculated, and I just need it AVG'd. Again, first line is the one that gives 81.2 instead of 81.3
If I need to provide more context with the surrounding queries or source-data, I can try... just let me know.
Execute this on SQL and you'll get your answer.
select 1425554.3/5457.251
select convert(real, (1425554.3/5457.251))
Basically, t2.PercentCorrect is already casted/converted to real which truncates the decimal point values. t1.NumCorrect is divided on the fly and all decimal points are processed on the avg function.
Hope this helps!
The easiest way is to divide by 1.0 and it will produce 6 digit precision.
Example:
select avg(incomeamount/1.0),vendorid from dailyincome group by vendorid
Related
I was performing some simple financial calculations in SQL Server when I discovered some odd behavior. I was trying to convert a string of numbers to a decimal type. While the string did not contain a decimal point, I knew from my specifications that the last 3 positions in the string were supposed to be behind the decimal point.
My first approach was flawed, but went something like this:
select convert(decimal(11,3),89456123/1000) as TotalUnits
This resulted in 89456.000. Performing the division before the cast resulted in the decimal parts being truncated.
So I moved the division operation outside the cast, like this:
select convert(decimal(11,3),89456123)/1000 as TotalUnits
This resulted in an explosion of positions after the decimal point. It returned 89456.12300000
According to my decimal specification, I wanted 11 digits, with 3 of them behind the decimal point. Now I have 13 total digits, with 8 behind the decimal. What happened?
To get what I want, I guess I have to double cast, like this:
select convert(decimal(11,3), convert(decimal(11,3),89456123)/1000)
which gives 89456.123.
It turns out no matter what I divide by, the resulting decimal point explosion is the same. Is the division converting the datatype into a double or something?
My question is this:
Why is this happening, and is there a more elegant way to compensate for it, instead of double-casting to decimal.
EDIT
I found this similar question on SO, but it looks like they are again double-casting.
SQL server does integer arithmetic, to force it to use numeric, you can multiply it by 1.0
No need of using convert twice. This gives 89456.123 with out double convert.
select convert(decimal(11,3),89456123*1.0/1000) as TotalUnits
Why does convert(decimal(11,3),89456123)/1000 end up with 6 decimal places? The rules demand it. numeric division has rather complicated rules about the resulting type.
When you say 1.0 you end up with a numeric with the least scale factors possible to represent this value:
SELECT SQL_VARIANT_PROPERTY(1.11, 'BaseType')
SELECT SQL_VARIANT_PROPERTY(1.11, 'Precision')
SELECT SQL_VARIANT_PROPERTY(1.11, 'Scale')
SELECT SQL_VARIANT_PROPERTY(1.11, 'TotalBytes')
What should you do? I think there is no really elegant solution because of the complicated rules. Any solution I can think of involves rather crazy type inference of intermediate results. I recommend pretty much the same solution that RADAR already gave:
select convert(decimal(11,3), convert(decimal(11, 3), 89456123)/1000) as TotalUnits
The main difference is that I think the *1.0 "trick" used as a short hand for a cast is obfuscating the meaning of the code. If you happen to like it feel free to use it, though.
select convert(decimal(11,3),89456123/CONVERT(decimal(11,3),1000))
Recently, I came across an anomaly that while dividing two integers, I am getting only the quotients and reminders are simply ignored.
SELECT 12/8,12%8
The above query gives 1 and 4 respectively, which is similar to Java/C programming. Again applying Java/C programming methods used below calculations to obtain the expected value.
SELECT 12.0/8,12/8.0
The answer is 1.5000 and 1.5000 respectively.
Working on my task I got a situation to obtain percentage value across two counted values (integers) and I stuck up with the results similar to the former query. Then I worked out through the same by multiplying one of the value with 1.0 . This solved my issue.
But later on, going through few scripts, used in my project (developed long back), I noticed in certain cases the decimal values are returned from the query even though two counted values (whole numbers) are divided.
I first noticed this in Netezza. But same holds true in SQL Server as well.
Please advise on what basis the datatypes of returned values are decided.
When dividing both integers, it will perform integer division, which returns an integer. To perform floating point division, you must either cast one or both of the operands to float/decimal/double.
SELECT cast(12 as float)/8
SELECT 12/cast(8 as float)
SELECT cast(12 as float)/cast(8 as float)
SELECT cast(12/8 as float)
Note that the last query is different since the integer division is performed first before casting to float,that is why the decimal value was already lost.
I've function in T-SQL:
sum(ar.tothandlingtime)/(60*60*24)
and in my result set I've all 0, because the result of this part of the day. Always is below 0.
I want to continue to work on the results, so I need an accurate result in a form and in a view. How?
It is doing integer division, and thus truncating the decimal.
To get your desired result, try converting one side to a decimal:
CONVERT(decimal(19, 18), SUM(ar.tothandlingtime))/(60*60*24)
Using this lets SQL know to perform decimal-based division.
If you need to, you can also play with the precision and scale of the decimal (read more here: http://msdn.microsoft.com/en-us/library/ms187746.aspx)
Of course, if you don't care about the precision, you can also achieve this by putting .0 after each hard-coded number:
(60.0*60.0*24.0)
For example,
select 5/(60.0*60.0*24.0) -- Result: 0.000057870370
select 5/(60*60*24) -- Result: 0
In my experience, this is generally the quickest way to get it to register as decimal division without explicitly using a CAST or CONVERT. If you were strictly using integer-based column values or aggregate functions, though, you would need to convert it, like in the first example.
You are dividing by an int trying converting that to a decimal. Change it like this
sum(ar.tothandlingtime)/CAST((60*60*24) AS DECIMAL ))
Has anyone encountered the following where when you divide a number in SQL a random number of trailing zeros are appended?...
SELECT 8.95/10 ... results in 0.895000
If you have encountered this, what is the reason for the addition of the zeros?
UPDATE: I am aware that casting the result to FLOAT will remove the 0's
First of all, seeing trailing zeros or anything when querying in SSMS is not because it's something special with the DB engine, but it's always the result of the internal query result formating used for displaying. After all, all numbers are just binary values in some representation that at some point gets translated to strings for displaying.
Anyway, the real reason is because of the datatypes involved, and how SSMS decides to display them. When doing those calculations, SQL Server must decide what datatype the result will be, based on the types of the inputs, and in that particular case it was numeric(7,6). You can easily see the result types by saving the result to a temp table and running sp_columns on that:
SELECT 8.95 AS dividend,10 AS divider,8.95/10 AS result INTO #temp ;
EXEC tempdb..sp_columns '#temp' ;
SELECT * FROM #temp ;
DROP TABLE #temp ;
In my case it returned this (among other uninteresting things for now):
COLUMN_NAME TYPE_NAME PRECISION LENGTH SCALE
dividend numeric 3 5 2
divided int 10 4 0
result numeric 7 9 6
Playing with castings in various places in the division will only change the resulting data types. The interesting fact is the Scale for the result column, note that it's a 6. That's exactly the number of decimal places that SSMS decides to display for the NUMERIC data type, regardless of the actual value. FLOAT don't have this formating from SSMS, which is why the casting eliminates the trailing zeros. Of course, when using the DB from outside SSMS, the formating will depend on the calling application and will not be subject to all this.
As another example of this behavior, just try SELECT CAST(1 AS NUMERIC(18,10)) and see that it shows 1.0000000000.
I run this example in SQL Server Management Studio:
SELECT CONVERT(REAL, -2101.12) n INTO #t
SELECT * FROM #t
SELECT SUM(n) FROM #t
The first SELECT creates a temp table #t with 1 column n of type real, and it puts 1 row in it with the value -2101.12.
The second SELECT confirms that the table is created with the intended content and the result is:
n
---------
-2101.12
The third SELECT sums the only number that is there, but the result is:
-2101.1201171875
So the question is: Where the 0.0001171875 comes from?
EDIT: I know the lack of precision for the real and float data types, unfortunately I cannot change the database schema because of this. What surprise me though, is that I would expect to see also the extra decimals in the second select since it is supposed to be stored with that lack of precision. Since it does not happens on the second select, then why the sum function picks it up?
You've just discovered real (aka floating point) data is approximate.
Use decimal datatype instead.
The FLOAT and REAL data types are known as approximate data types. The behavior of FLOAT and REAL follows the IEEE 754 specification on approximate numeric data types.
Approximate numeric data types do not store the exact values specified for many numbers; They store an extremely close approximation of the value. For many applications, the tiny difference between the specified value and the stored approximation is not noticeable. At times, though, the difference becomes noticeable. Because of the approximate nature of the FLOAT and REAL data types, do not use these data types when exact numeric behavior is required, such as in financial applications, in operations involving rounding, or in equality checks. Instead, use the integer, decimal, money, or smallmoney data types.
Avoid using FLOAT or REAL columns in WHERE clause search conditions, especially with the = or <> operators. It is best to limit FLOAT and REAL columns with > or < comparisons.
Source of above statement