I have a staging table loaded with data from a SAS dataset containing 5M records. All the columns are varchar. I am trying to convert a couple of columns to decimal(32,10). But it generates an error. I tried cast, I tried convert and even splitting the data up before and after decimal - same result.
I looked at the IsNumeric flag of the column and there are 0 records <> 1 meaning the data is numeric.
case
when wtd_count = '.' THEN NULL
when wtd_count = '' THEN NULL
else convert(decimal(32, 10), wtd_count)
end
Error:
Msg 8114, Level 16, State 5, Line 99
Error converting data type varchar to numeric.
So I'm wondering what else I can do to convert the data to decimal? Any idea?
Any help will greatly be appreciated.
If you are in SQL Server 2012 and above try to use try_parse or try_convert
ISNUMERIC is not reliable for what you're doing. It will flag values with things like monetary symbols and commas in them as valid.
It seems quite likely that there is some non-numeric data present. TRY_CONVERT or TRY_PARSE are your friend here. As an FYI, SQL Server version 11.0.x is SQL Server 2012, so you should be able to use these.
I also find it hard to believe that converting to numeric works, but not decimal. I can find no information that suggests the actual implementation of these two data types is different, and as such neither should work.
I would do some more in depth analysis of your data to make sure it looks like you're expecting it to.
After read your case statement i suppose that you have coma separated values. I'm pretty sure that you should use: CONVERT(DECIMAL(32,10),REPLACE(wtd_count,',','.'))
Related
From my understanding, and I am probably wrong here, but doesn't the ISNUMERIC() return a 1 if whatever we are looking at is a numeric data type? And since most zip codes are saved as varchar data types, shouldn't it then return a 0? I looked up the documentation and that's what it says there to me, what am I missing here? I get zip codes are numbers, but because they are saved as a string shouldn't that make a difference? Thanks in advance.
ISNUMERIC() is specifically to look at strings, not at things already stored as numbers.
However, I don't recommend using it. For instance, '3.1e5' is considered numeric.
Instead, use try_convert():
try_convert(int, zip)
This returns NULL if the column cannot be converted.
I transferred an Oracle database to SQL Server and all seems to have went well. The various ID columns are large numbers so I had to use Decimal as they were too large for BigInt.
I am now trying to read the data using pandas.read_sql using pyodbc connection with ODBC Driver 17 for SQL Server. df = pandas.read_sql("SELECT * FROM table1"),con)
The numbers are coming out as float64 and when I try to print them our use them in SQL statements they come out in scientific notation and when I try to use '{:.0f}'.format(df.loc[i,'Id']) It turns several numbers into the same number such as 90300111000003078520832. It is like precision is lost when it goes to scientific notation.
I also tried pd.options.display.float_format = '{:.0f}'.format before the read_sql but this did not help.
Clearly I must be doing something wrong as the Ids in the database are correct.
Any help is appreciated Thanks
pandas' read_sql method has an option named coerce_float which defaults to True and it …
Attempts to convert values of non-string, non-numeric objects (like decimal.Decimal) to floating point, useful for SQL result sets.
However, in your case it is not useful, so simply specify coerce_float=False.
I've had this problem too, especially working with long ids: read_sql works fine for the primary key, but not for other columns (like the retweeted_status_id from Twitter API calls). Setting coerce_float to false does nothing for me, so instead I cast retweeted_status_id to a character format in my sql query.
Using psql, I do:
df = pandas.read_sql("SELECT *, Id::text FROM table1"),con)
But in SQL server it'd be something like
df = pandas.read_sql("SELECT *, CONVERT(text, Id) FROM table1"),con)
or
df = pandas.read_sql("SELECT *, CAST(Id AS varchar) FROM table1"),con)
Obviously there's a cost here if you're asking to cast many rows, and a more efficient option might be to pull from SQL server without using pandas (as a nested list or JSON or something else) which will also preserve your long integer formats.
I keep getting errors when using the REPLACE function in SQL and i don't know why?
I execute this query:
UPDATE [ShellPlus].[dbo].[Views]
SET [ShellPlus].[dbo].[Views].SerializedProperties = REPLACE(SerializedProperties,'|EKZ PSG met verkort EEG','|EKZ PSG met verkort EEG|EEG kort op EEG3')
WHERE [ShellPlus].[dbo].[Views].InternalViewID = '3F4C1E8E-DA0C-4829-B447-F6BDAD9CD505'
And I keep getting this message:
Msg 8116, Level 16, State 1, Line 6
Argument data type ntext is invalid for argument 1 of replace function.
At UPDATE I give the correct table
At SET I give the correct column
At REPLACE I give the: (column name, 'old string', 'new string')
What am I doing wrong?
The real solution is fix the data type:
USE ShellPlus;
ALTER TABLE [dbo].[Views] ALTER COLUMN SerializedProperties nvarchar(MAX);
Then your existing query will work. But you should also normalise that data.
Try this:
UPDATE [ShellPlus].[dbo].[Views]
SET [ShellPlus].[dbo].[Views].SerializedProperties = CAST(REPLACE(CAST(SerializedPropertiesas NVarchar(MAX)),'|EKZ PSG met verkort EEG','|EKZ PSG met verkort EEG|EEG kort op EEG3') ) AS NText)
WHERE [ShellPlus].[dbo].[Views].InternalViewID = '3F4C1E8E-DA0C-4829-B447-F6BDAD9CD505'
Your doing at least three things wrong:
It seems like you're storing delimited data in your column - which is a mistake. For more information, read Is storing a delimited list in a database column really that bad?, where you will see a lot of reasons why the answer to this question is Absolutely yes!
You're using the Text data type, which is deprecated since SQL Server 2008 introduced varchar(max) to replace it. Given the fact that we're in 2019 and the 2008 version just ended it's extended support this July, its high time to change that Text data type to varchar(max) (and if you're using the 2008 or 2008 r2 version, upgrade your SQL Server).
You're using four-parts identifiers for your column names (Thanks #Larnu for pointing that out in the comments). Best practice is to use two-parts identifiers for column names. Read my answer here for a details explanation.
The solution to your problem involves refactoring the database structure - normalize what needs to be normalized, and replace of all Text, NText and Image with their modern replacement data types: varchar(max), nvarchar(max) and varbinary(max). Also, this would be a good time to figure out if you really need to support values longer than 8000 chars (4000 for unicode values) and if not, use a more appropriate value (max length columns have poor performance compared to "regular" length columns).
I've the following problem:
I need to calculate a date using dateadd function.
dateadd(d,delay),CONVERT(DATE,Started,105))
delay is taken from a different table than started, and its in numeric format
the problem is I receive this type of error:
Messaggio 241, livello 16, stato 1, riga 14
Conversion failed when converting date and/or time from character string.
I then tried to do something like:
dateadd(d,CAST(delay AS NUMERIC(8)),CONVERT(DATE,Started,105))
resulting in the same error :(
Anyone can please help me?
The error message is very clear. There are values in the Started column that cannot be converted to date using the style you provided. Such is the path your table designer chose when choosing to store dates as strings. Now you need to either clean up the data (better yet - clean and convert the column to the correct datatype) or you write your queries to avoid the bad rows. So how do you do that?
You could (and should) get into the habit of searching the internet for answers. This question is posted daily somewhere. Assuming you have a current version of sql server, you can use the following to find the "bad" rows.
select * from dbo.mytable as tbl where try_convert(date, tbl.Started, 105) is null
order by tbl.Started;
And you can adapt that logic to avoid bad rows very easily.
I am new to SQL Server so please accept my apologies if my question seems too easy. I tried finding a solution, but so far I couldn't see anything that I could use in my query.
I am trying to find length of the biggest columns in a table and I'm using the following query.
SELECT
SUM(DATALENGTH([INSDetails])) as [INSDetails]
FROM
dbo.Contractors
The table Contractors is slightly over 8GB and has over 30mln rows. INSDetails column is varchar(2048)
The query above works perfectly well for all the other columns of all the other tables in my database, but when I run it on Contractors table it returns an error
Msg 8115, Level 16, State 2, Line 26
Arithmetic overflow error converting expression to data type int.
I understand that this error message appears when you try to convert a value of a certain datatype to another datatype, but that value is too large for the second datatype.
Could you please help me to rewrite the query or suggest alternative approach to get the output?
I read that someone suggested using CAST AS big int to solve this issue, but I'm not sure how it can be incorporated in my query.
Any suggestions will be appreciated. Thank you in advance.
select sum(cast(datalength([INSDetails]) as bigint))