REAL column holding values outside documented range - sql-server

According to MSDN, the range for REAL values is - 3.40E + 38 to -1.18E - 38, 0 and 1.18E - 38 to 3.40E + 38. However, I have quite a few values beyond that range in my table.
The following query returns lots of very small values and no very large ones:
SELECT MyColumn ,
*
FROM data.MyTable
WHERE MyColumn <> 0
AND ( MyColumn < CONVERT(REAL, 1.18E-38)
OR MyColumn > CONVERT(REAL, 3.40E+38)
)
AND ( MyColumn < CONVERT(REAL, -3.40E+38)
OR MyColumn > CONVERT(REAL, -1.18E-38)
)
It is easy to show how these values end up in the table. I cannot insert them directly:
CREATE TABLE a(r REAL NULL);
GO
INSERT INTO a(r) VALUES(4.330473E-39);
GO
SELECT r FROM a
GO
DROP TABLE a;
----
0.0
But I can divide two columns and get and outside of range value:
CREATE TABLE a
(
r1 REAL NULL ,
r2 REAL NULL ,
r3 REAL NULL
) ;
GO
INSERT INTO a
( r1, r2 )
VALUES ( 4.330473E-38, 1000 ) ;
GO
UPDATE a
SET r3 = r1 / r2 ;
SELECT r1 ,
r2 ,
r3
FROM a
r1 r2 r3
------------- ------------- -------------
4.330473E-38 1000 4.330433E-41
So I guess MSDN gives wrong ranges of valid data, correct?
Am I missing anything?
Several people suggested that this is a bug.
What part of this behavior exactly is a bug. is it:
Wrong constants documented in MSDN and used in DBCC, as well as wrong threshold for rounding down.
Update being able to save wrong values

Books Online documents only the normal range for single- and double-precision floating point numbers. The IEEE 754 rules also specify floating-point numbers closer to zero than the smallest non-zero normal value, known variously as denormalized, denormal, and subnormal numbers. From that last link:
Denormal numbers provide the guarantee that addition and subtraction of floating-point numbers never underflows; two nearby floating-point
numbers always have a representable non-zero difference. Without
gradual underflow, the subtraction a−b can underflow and produce zero
even though the values are not equal. This can, in turn, lead to
division by zero errors that cannot occur when gradual underflow is
used.
SQL Server is following the rules for single-precision floating point calculations in the examples posted. The bug may be that DBCC checks only for normal values, and throws an incorrect error message when it encounters a stored denormal value.
Example producing a denormal single-precision value:
DECLARE
#v1 real = 14e-39,
#v2 real = 1e+07;
-- 1.4013e-045
SELECT #v1 / #v2;
Example showing a stored float denormal passes DBCC checks:
CREATE TABLE dbo.b (v1 float PRIMARY KEY);
INSERT b VALUES (POWER(2e0, -1075));
SELECT v1 FROM b; -- 4.94065645841247E-324
DBCC CHECKTABLE(b) WITH DATA_PURITY; -- No errors or warnings
DROP TABLE dbo.b;

This is a bug in SQL Server. The last script you post is a nice repro. Add one line to it at the end:
DBCC CHECKDB WITH data_purity
This fails with:
Msg 2570, Level 16, State 3, Line 1 Page (1:313), slot 0 in object ID
357576312, index ID 0, partition ID 1801439851932155904, alloc unit ID
2017612634169999360 (type "In-row data"). Column "r3" value is out of
range for data type "real". Update column to a legal value.
This proves it is a bug. I suggest you file a bug with Microsoft Connect for SQL Server.

Related

How to convert VARCHAR columns to DECIMAL without rounding in SQL Server?

In my SQL class, I'm working with a table that is all VARCHAR. I'm trying to convert each column to a more correct data type.
For example. I have a column called Item_Cost that has a value like:
1.25000000000000000000
I tried to run this query:
ALTER TABLE <table>
ALTER COLUMN Item_Cost DECIMAL
This query does run successfully, but it turns it into 1 instead of 1.25.
How do I prevent the rounding?
Check out the documentation for the data type decimal. The type is defined by optional parameters p (precision) and s (scale). The latter determines the numbers to the right of the decimal point.
Extract from the documentation (I highlighted the important bit in bold):
s (scale)
The number of decimal digits that are stored to the right of
the decimal point. This number is subtracted from p to determine the
maximum number of digits to the left of the decimal point. Scale must
be a value from 0 through p, and can only be specified if precision is
specified. The default scale is 0 and so 0 <= s <= p. Maximum storage
sizes vary, based on the precision.
Defining a suitable precision and scale fixes your issue.
Sample data
create table MyData
(
Item_Cost nvarchar(100)
);
insert into MyData (Item_Cost) values ('1.25000000000000000000');
Solution
ALTER TABLE MyData Alter Column Item_Cost DECIMAL(10, 3);
Result
Item_Cost
---------
1.250
Fiddle

netezza Double Precision Output Truncates Vaules

I've noticed that the nzsql and 'nzunload' just truncates double precision column's mantissa values. Here is the issue:
select tot_amt from table1;
tot_amt
~~~~~~~
123.124
567.678
while when I use other clients like Aginity for Data analytics - the output I get is
tot_amt
~~~~~~~
123.1240535
567.6780122
Also I've found the 'truncation' happens when netezza encounters 0 after 3 mantissa digits.
We are trying to migrate this db to oracle and due to this issue the entire project is messed and the client doesn't trust our migration scripts. Has anyone encountered this issue? The only workaround, even frmo IBM engineer is to cast it TO_CHAR( '999,999.999', col ) This will kill the unload scripts if I have to do it for billions of rows.
I can reproduce this issue where I have a table created with column as FLOAT(6) such as:
USERDB.USER(USER)=> create table ZZ (
USERDB.USER(USER)(> YY FLOAT(6)
USERDB.USER(USER)(> );
CREATE TABLE
USERDB.USER(USER)=> insert into ZZ (yy) values (123.123456789);
INSERT 0 1
USERDB.USER(USER)=> insert into ZZ (yy) values (12.123456789);
INSERT 0 1
USERDB.USER(USER)=> select * from ZZ;
YY
---------
123.123
12.1234
(2 rows)
USERDB.USER(USER)=> select CAST ( YY as FLOAT(15) ) from ZZ;
?COLUMN?
----------------
123.1234588623
12.123399734497
(2 rows)
USERDB.USER(USER)=>
I can cast the column values to a wider type, however the problem I see is that the value I inserted is not the same as the value returned. And the same is true if I use Aginity also to query, the values are incorrect.
Check the precision (and scale) of the 'tot_amt' column in table1, I guess the data type used to store values is quite small (FLOAT(6) maybe?), and NZSQL is telling you the correct values as enforced by the data type.

How do I control the datatype of a computed column?

Using SQL Server 2012...
I have two columns:
Price [decimal(28,12)]
OustandingShares [decimal(38,3)] -- The 38 is overkill but alas, not my call.
When I do an ALTER TABLE I get a resulting computed column as a [decimal(38,6)]. I need the datatype to be [decimal(28,12)].
ALTER TABLE [xyz].MyTable
ADD Mv AS OustandingShares * Price
How can I effectively get 12 decimals of scale on this computed column? I've tried doing convert on the OutstandingShares to 12 decimal places as well as wrapping a convert around the OutstandingShares * Price. The only thing I get is a computed field at [decimal(28,12)] with six trailing zeros.
Thoughts?
The Fix
This does what you want:
CONVERT(DECIMAL(28,12), (
CONVERT(DECIMAL(15, 3), [OustandingShares])
* CONVERT(DECIMAL(24, 12), [Price])
)
)
Test with this:
SELECT CONVERT(DECIMAL(28,12),
(CONVERT(DECIMAL(24,12), 5304.987781883689)
* CONVERT(DECIMAL(15,3), 3510.88)));
Result:
18625175.503659806036
The Reason
The computation is being truncated due to SQL Server's rules for how to handle Precision and Scale across various operations. These rules are detailed in the MSDN page for Precision, Scale, and Length. The details we are interested in for this case are:
Operation: e1 * e2
Result precision: p1 + p2 + 1
Result scale *: s1 + s2
Here the datatypes in play are:
DECIMAL(28, 12)
DECIMAL(38, 3)
This should result in:
Precision = (28 + 38 + 1) = 67
Scale = 15
But the max length of the DECIMAL type is 38. So what gives? We now need to notice that there was a footnote attached to the "Result scale" calculation, being:
* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.
So it seems that in order to get the Precision back down to 38 it chopped off 9 decimal places.
And this is why my proposed fix works. I kept the "Scale" values the same as we don't want to truncate going in and expanding them serves no purpose as SQL Server will expand the Scale as appropriate. The key is in reducing the Precision so that the truncation would be non-existent or at least minimal.
With DECIMAL(15, 3) and DECIMAL(24, 12) we should get:
Precision = (15 + 24 + 1) = 40
Scale = 15
40 is over the limit so reduce by 2 to get down to 38, which means reduce the Scale by 2 leaving us with a true "Result Scale" of 13, which is 1 more than we need and will even be seeing.
Use cast() or convert(). Something like:
ALTER TABLE [xyz].MyTable ADD Mv AS cast(OustandingShares * Price as decimal(12, 6)
or whatever type you want it to be.
EDIT:
Oh, I think I'm getting the idea. The problem is the calculation itself. In that case, do the conversion before the multiplication, so you don't have to depend on SQL Server's (arcane) rules for conforming decimal types.
ALTER TABLE [xyz].MyTable
ADD Mv AS cast(OustandingShares as decimal(28, 12) * cast(Price as decimal(28, 12))
I believe what is happening in your case is that the maximum precision on the calculated result exceeds the allowed thresholds, so the scale is reduced accordingly. This is explained at the bottom of this page.

How to get the count of digits after the decimal point in a float column in ms sql?

I have to count the digits after the decimal point in a database hosted by a MS Sql Server (2005 or 2008 does not matter), in order to correct some errors made by users.
I have the same problem on an Oracle database, but there things are less complicated.
Bottom line is on Oracle the select is:
select length( substr(to_char(MY_FIELD), instr(to_char(MY_FILED),'.',1,1)+1, length(to_char(MY_FILED)))) as digits_length
from MY_TABLE
where the filed My_filed is float(38).
On Ms Sql server I try to use:
select LEN(SUBSTRING(CAST(MY_FIELD AS VARCHAR), CHARINDEX('.',CAST(MY_FILED AS VARCHAR),1)+1, LEN(CAST(MY_FIELD AS VARCHAR)))) as digits_length
from MY_TABLE
The problem is that on MS Sql Server, when i cast MY_FIELD as varchar the float number is truncated by only 2 decimals and the count of the digits is wrong.
Can someone give me any hints?
Best regards.
SELECT
LEN(CAST(REVERSE(SUBSTRING(STR(MY_FIELD, 13, 11), CHARINDEX('.', STR(MY_FIELD, 13, 11)) + 1, 20)) AS decimal))
from TABLE
I have received from my friend a very simple solution which is just great. So I will post the workaround in order to help others in the same position as me.
First, make function:
create FUNCTION dbo.countDigits(#A float) RETURNS tinyint AS
BEGIN
declare #R tinyint
IF #A IS NULL
RETURN NULL
set #R = 0
while #A - str(#A, 18 + #R, #r) <> 0
begin
SET #R = #R + 1
end
RETURN #R
END
GO
Second:
select MY_FIELD,
dbo.countDigits(MY_FIELD)
from MY_TABLE
Using the function will get you the exact number of digits after the decimal point.
The first thing is to switch to using CONVERT rather than CAST. The difference is, with CONVERT, you can specify a format code. CAST uses whatever the default format code is:
When expression is float or real, style can be one of the values shown in the following table. Other values are processed as 0.
None of the formats are particularly appealing, but I think the best for you to use would be 2. So it would be:
CONVERT(varchar(25),MY_FIELD,2)
This will, unfortunately, give you the value in scientific notation and always with 16 digits e.g. 1.234567890123456e+000. To get the number of "real" digits, you need to split this number apart, work out the number of digits in the decimal portion, and offset it by the number provided in the exponent.
And, of course, insert usual caveats/warnings about trying to talk about digits when dealing with a number which has a defined binary representation. The number of "digits" of a particular float may vary depending on how it was calculated.
I'm not sure about speed. etc or the elegance of this code. it was for some ad-hoc testing to find the first decimal value . but this code could be changed to loop through all the decimals and find the last time a value was greater than zero easily.
DECLARE #NoOfDecimals int = 0
Declare #ROUNDINGPRECISION numeric(32,16) = -.00001000
select #ROUNDINGPRECISION = ABS(#ROUNDINGPRECISION)
select #ROUNDINGPRECISION = #ROUNDINGPRECISION - floor(#ROUNDINGPRECISION)
while #ROUNDINGPRECISION < 1
Begin
select #NoOfDecimals = #NoOfDecimals +1
select #ROUNDINGPRECISION = #ROUNDINGPRECISION * 10
end;
select #NoOfDecimals

Appropriate datatype for holding percent values?

What is the best datatype for holding percent values ranging from 0.00% to 100.00%?
Assuming two decimal places on your percentages, the data type you use depends on how you plan to store your percentages:
If you are going to store their fractional equivalent (e.g. 100.00% stored as 1.0000), I would store the data in a decimal(5,4) data type with a CHECK constraint that ensures that the values never exceed 1.0000 (assuming that is the cap) and never go below 0 (assuming that is the floor).
If you are going to store their face value (e.g. 100.00% is stored as 100.00), then you should use decimal(5,2) with an appropriate CHECK constraint.
Combined with a good column name, it makes it clear to other developers what the data is and how the data is stored in the column.
Hold as a decimal.
Add check constraints if you want to limit the range (e.g. between 0 to 100%; in some cases there may be valid reasons to go beyond 100% or potentially even into the negatives).
Treat value 1 as 100%, 0.5 as 50%, etc. This will allow any math operations to function as expected (i.e. as opposed to using value 100 as 100%).
Amend precision and scale as required (these are the two values in brackets columnName decimal(precision, scale). Precision says the total number of digits that can be held in the number, scale says how many of those are after the decimal place, so decimal(3,2) is a number which can be represented as #.##; decimal(5,3) would be ##.###.
decimal and numeric are essentially the same thing. However decimal is ANSI compliant, so always use that unless told otherwise (e.g. by your company's coding standards).
Example Scenarios
For your case (0.00% to 100.00%) you'd want decimal(5,4).
For the most common case (0% to 100%) you'd want decimal(3,2).
In both of the above, the check constraints would be the same
Example:
if object_id('Demo') is null
create table Demo
(
Id bigint not null identity(1,1) constraint pk_Demo primary key
, Name nvarchar(256) not null constraint uk_Demo unique
, SomePercentValue decimal(3,2) constraint chk_Demo_SomePercentValue check (SomePercentValue between 0 and 1)
, SomePrecisionPercentValue decimal(5,2) constraint chk_Demo_SomePrecisionPercentValue check (SomePrecisionPercentValue between 0 and 1)
)
Further Reading:
Decimal Scale & Precision: http://msdn.microsoft.com/en-us/library/aa258832%28SQL.80%29.aspx
0 to 1 vs 0 to 100: C#: Storing percentages, 50 or 0.50?
Decimal vs Numeric: Is there any difference between DECIMAL and NUMERIC in SQL Server?
I agree with Thomas and I would choose the DECIMAL(5,4) solution at least for WPF applications.
Have a look to the MSDN Numeric Format String to know why :
http://msdn.microsoft.com/en-us/library/dwhawy9k#PFormatString
The percent ("P") format specifier multiplies a number by 100 and converts it to a string that represents a percentage.
Then you would be able to use this in your XAML code:
DataFormatString="{}{0:P}"
If 2 decimal places is your level of precision, then a "smallint" would handle this in the smallest space (2-bytes). You store the percent multiplied by 100.
EDIT: The decimal type is probably a better match. Then you don't need to manually scale. It takes 5 bytes per value.
Use numeric(n,n) where n has enough resolution to round to 1.00. For instance:
declare #discount numeric(9,9)
, #quantity int
select #discount = 0.999999999
, #quantity = 10000
select convert(money, #discount * #quantity)

Resources