I am trying to insert a XML request data into a table where the size of the XML message is more than 4KB and I am receiving the error: 'ORA-01461: can bind a LONG value only for insert into a LONG column'. I am receiving the XML request data as part of a transaction as a string data and due to the server used (webMethods Integration server 9.0) I have a limitation of converting the data into another format. I have tried the following methods:
insert into testtable values (xmltype())
insert into testtable values (xmltype().getClobVal())
insert into testtable values (TO_CLOB())
The table schema is as given below:
desc testtable
Name Null Type
------- -------- ---------
XMLDATA NOT NULL XMLTYPE()
The insert to the table should also happen within a short time as the operation is executed for every request message that is coming to the server. Please help.
This "anonymous PL/SQL block"-approach might work, but I'm not sure, since I don't have webmethods of my own to test it ...
Instead of
insert into testtable
values (xmltype(:1))
try running
declare
l_long long;
l_string varchar2(32767);
begin
l_long := :1;
l_string := l_long;
insert into testtable values (xmltype(l_string));
end;
There might, however, be an issue with processing XML documents larger than 32767 (maybe even larger than 4000) chars.
I modified the above code a bit and it is working now, but I am not really sure why:
declare
l_long long;
l_clob clob;
begin
l_long := :1;
l_clob := TO_CLOB(l_long);
insert into testtable values (xmltype(l_clob));
dbms_lob.freetemporary(l_clob);
end;
This is working for XML type data above 4KB as well. #nop77svk - thanks for your guidance.
By default everything in wM is defined as a string. But while using the adapter connection you need to check what is the input type. So for the field which has been defined as Long in DB you need to either pass a long object from wM or you need to make the input type for the field as string in adapter service. Also you need to check the size defined in the DB. If you suspect large size then set the field type in DB as CLOB instead.
Related
We recently had an identifier column move from int values to bigint values. An ETL process which loads these values was not updated. That process is using SQL bulk insert, and we are seeing incorrect values in the destination table. I would have expected a hard failure.
Please note this process has been running successfully for a long time before this issue.
Can anyone tell me what the heck SQL Server is doing here!? I know how to fix the situation, but I'm trying to better understand it for the data cleanup effort that I'll need to complete, as well as just for the fact that it looks like black magic!
I've been able to re-create this issue in SQL Server 2017 and 2019.
Simplified Example CSV file contents:
310067463717
310067463718
310067463719
Example SQL:
create table #t (t int)
bulk insert #t from 'c:\temp\test.csv'
with (datafiletype = 'char',
fieldterminator = '|'
)
select * from #t
Resulting data:
829818405
829818406
829818407
Interestingly, I tried with smaller values, and I do see an error:
Example CSV file contents (2147483647 is the largest int value for SQL Server):
310067463717
310067463718
310067463719
2147483647
2147483648
Running the same SQL code as above, I get an error for one row:
Msg 4867, Level 16, State 1, Line 4
Bulk load data conversion error (overflow) for row 5, column 1 (t).
The resulting data looks like this:
829818405
829818406
829818407
2147483647
I also tried just now with a much higher value, 31006746371945654, and that threw the same overflow error as 2147483648.
And last, I did confirm that if I create the table with the column defined as bigint, the data inserted is correct.
create table #t (t bigint)
bulk insert #t from 'c:\temp\test.csv'
with (datafiletype = 'char',
fieldterminator = '|'
)
select * from #t
Resulting data:
2147483647
2147483648
310067463717
310067463718
310067463719
I'm working on SQL Server 2008.
I delete all data from a table and then I try to insert value to the table. Here's the code:
TRUNCATE TABLE [dbo].[STRAT_tmp_StratMain]
INSERT INTO [dbo].[STRAT_tmp_StratMain] ([FileNum])
SELECT [dbo].[STRAT_tmp_Customer].[NumericFileNumber]
FROM [dbo].[STRAT_tmp_Customer];
The FileNum in STRAT_tmp_StratMain is float number and is also index and can't be null.
NumericFileNumber is float and can be null but is never null and there are no duplicates in it (each row is unique number).
The table STRAT_tmp_StratMain contain much more fields but all can be null and also has a defualt values.
When I try to run this query I get the error:
Msg 8152, Level 16, State 4, Line 1 String or binary data would be
truncated. The statement has been terminated.
I tried also to do simply:
INSERT INTO [dbo].[STRAT_tmp_StratMain] ([FileNum]) Values (1);
Still get the same error.
Any ideas?
Thanks,
Ilan
I am not able to reproduce your issue. When I run this code on SQL Server 2008, I get no error:
DECLARE #tt TABLE (FileNum float NOT NULL);
INSERT INTO #tt (FileNum) VALUES (1);
Check the Default constraints on all the columns in your target table and make sure none of them would try to insert a string value that would truncated by the datatype limitations of the column.
example: SomeColumn varchar(1) DEFAULT 'Hello'
This due to the data you are trying to insert does not fit in the field: if you have a defined length of (say) 10 or 50 characters but the data you are trying to insert is longer than that.
I have a SQL table in which some columns, when viewed in SQL Server Manager, contain <Unable to read data>. Does anyone know how to query for <Unable to read data>? I can individually modify the data in this column with update table set column = NULL where key = 'value', but how can I find whether additional rows exist with this bad data?
I would recommend against replacing the data. There is nothing wrong with it, is just that SSMs cannot display it properly in the Edit panel. The data in the database itself is perfectly fine, from your description.
This script shows the problem:
create table test (id int not null identity(1,1) primary key,
large_value numeric(38,0));
go
insert into test (large_value) values (1);
insert into test (large_value) values (12345678901234567890123456789012345678);
insert into test (large_value) values (1234567890123456789012345678901234567);
insert into test (large_value) values (123456789012345678901234567890123456);
insert into test (large_value) values (12345678901234567890123456789012345);
insert into test (large_value) values (1234567890123456789012345678901234);
insert into test (large_value) values (123456789012345678901234567890123);
insert into test (large_value) values (12345678901234567890123456789012);
insert into test (large_value) values (1234567890123456789012345678901);
insert into test (large_value) values (123456789012345678901234567890);
insert into test (large_value) values (12345678901234567890123456789);
insert into test (large_value) values (NULL);
go
select * from test;
go
The SELECT will work fine, but showing the Edit Top 200 Rows in object explorer will not:
There is a Connect Item for this issue. SSMS 2012 still exhibits the same problem.
If we look at the Numeric and Decimal details we'll see that the problem occurs at a weird boundary, at precision 29 which is actually not a SQL Server boundary (precision 28 is):
Precision Storage bytes
1 - 9 5
10-19 9
20-28 13
29-38 17
If we check the .Net (SSMS is a managed application) decimal precision table we can see quickly where the crux of the issue is: Precision is 28-29 significant digits. So the .Net decimal type cannot map high precision (>29) SQL Server numeric/decimal types.
This will affect not only SSMS display, but your applications as well. Specialized applications like SSIS will use high precisions representation like DT_NUMERIC:
DT_NUMERIC An exact numeric value with a fixed precision and scale.
This data type is a 16-byte unsigned integer with a separate sign, a
scale of 0 - 38, and a maximum precision of 38.
Now back to your problem: you can discover invalid entries by simply looking at the value. Knowing that the C# representation range can accommodate values between approximate (-7.9 x 1028 to 7.9 x 1028) / (100 to 28)` (the range depends on the scale) you can search for values outside the range on each column (the actual values to search between will depend on the column scale). But that begs the question 'what to replace the data with?'.
I would recommend instead using dedicated tools for import export, tools that are capable of handling high precision numeric values. SSIS is the obvious candidate. But even the modest bcp.exe would also fit the bill.
BTW if your values are actually incorrect (ie. true corruption) then I would recommend running DBCC CHECKTABLE (...) WITH DATA_PURITY:
DATA_PURITY
Causes DBCC CHECKDB to check the database for column values that are not valid or out-of-range. For example, DBCC CHECKDB detects
columns with date and time values that are larger than or less than
the acceptable range for the datetime data type; or decimal or
approximate-numeric data type columns with scale or precision values
that are not valid.
For databases created in SQL Server 2005 and later, column-value integrity checks are enabled by default and do not require the
DATA_PURITY option. For databases upgraded from earlier versions of
SQL Server, column-value checks are not enabled by default until DBCC
CHECKDB WITH DATA_PURITY has been run error free on the database.
After this, DBCC CHECKDB checks column-value integrity by default.
Q: How can this issue arise for a datetime column?
use tempdb;
go
create table test(d datetime)
insert into test (d) values (getdate())
select %%physloc%%, * from test;
-- Row is on page 0x9100000001000000
dbcc traceon(3604,-1);
dbcc page(2,1,145,3);
Memory Dump #0x000000003FA1A060
0000000000000000: 10000c00 75f9ff00 6aa00000 010000 ....uùÿ.j .....
Slot 0 Column 1 Offset 0x4 Length 8 Length (physical) 8
dbcc writepage(2,1,145, 100, 8, 0xFFFFFFFFFFFFFFFF)
dbcc checktable('test') with data_purity;
Msg 2570, Level 16, State 3, Line 2 Page (1:145), slot 0 in object ID
837578022, index ID 0, partition ID 2882303763115671552, alloc unit ID
2882303763120062464 (type "In-row data"). Column "d" value is out of
range for data type "datetime". Update column to a legal value.
As suggested above ,these errors usually occurs when Precision and scale are not preserved .If your comfortable with SSIS then you can achieve to get those rows which are corrupt .Taking the values which Martin Smith created
CREATE TABLE T(ID int ,C DECIMAL(38,0));
INSERT INTO T VALUES(1,9999999999999999999999999999999999999)
The above table reproduces the error . Here the first column represents the primary key . I inserted around 1000 rows out of which few were corrupted values . Below is the SSIS package design
In the Data Conversion ,i took the column C which had errors and tried to cast it to Decimal(38,0) .Since a conversion or truncation error will occur ,therefore i redirected the error rows to an OLEDB command which basically updates the table and sets the column to NULL
Update T
Set C=NULL
where ID=?
The value of C and ID will be directed to oledb command .In case if there is no error then i'm just inserting into a table ( Actually no need to do this ).This will work if you have a primary key column in your table .
In case if there is any error in date time column a sql query can be written to verify the format of datetime values .Please go through the MSDN link for valid date time value
Select * from YourTable where ISDATE(Col)!=1
I think you can fetch data with cursor. please try again with cursor query such as below query :
DECLARE VerifyCursor CURSOR FOR
SELECT *
FROM MyTable
WHILE 1=1 BEGIN
BEGIN Try
FETCH FIRST FROM VerifyCursor INTO #Column1, #Column2, ...
INSERT INTO #MyTable2(Column1, Column2,...)
VALUES (#Column1, #Column2, ...)
END TRY
BEGIN CATCH
END CATCH
IF (##FETCH_STATUS<>0) BREAK
End
OPEN VerifyCursor
CLOSE VerifyCursor
DEALLOCATE VerifyCursor
Replacing the bad data is simple with an update:
UPDATE table SET column = NULL WHERE key_column = 'Some value'
I'm in the process of importing data from a legacy MySQL database into SQL Server 2005.
I have one table in particular that's causing me grief. I've imported it from MySQL using a linked server and the MySQL ODBC driver, and I end up with this:
Col Name Datatype MaxLen
OrderItem_ID bigint 8
PDM_Structure_ID int 4
LastModifiedDate datetime 8
LastModifiedUser varchar 20
CreationDate datetime 8
CreationUser varchar 20
XMLData text -1
OrderHeader_ID bigint 8
Contract_Action varchar 10
ContractItem int 4
My main focus is on the XMLData column - I need to clean it up and make it so that I can convert it to an XML datatype to use XQuery on it.
So I set the table option "large data out of row" to 1:
EXEC sp_tableoption 'OrderItem', 'large value types out of row', 1
and then I go ahead and convert XMLData to VARCHAR(MAX) and do some cleanup of the XML stored in that field. All fine so far.
But when I now try to convert that column to XML datatype:
ALTER TABLE dbo.OrderItem
ALTER COLUMN XMLData XML
I get this message here:
Msg 511, Level 16, State 1, Line 1
Cannot create a row of size 8077 which
is greater than the allowable maximum
row size of 8060. The statement has
been terminated.
which is rather surprising, seeing that the columns besides the XMLData only make up roughly 90 bytes, and I specifically instructed SQL Server to store all "large data" off-row....
So why on earth does SQL Server refuse to convert that column to XML data??? Any ideas?? Thoughts?? Things I can check / change in my approach??
Update: I don't know what changed, but on a second attempt to import the raw data from MySQL into SQL Server, I was successfully able to convert that NTEXT -> VARCHAR(MAX) column to XML in the end..... odd..... anyhoo - works now - thanks guys for all your input and recommendations! Highly appreciated !
If you have sufficient storage space, you could try selecting from the VARCHAR(MAX) version of the table into a new table with the same schema but with XMLData set up as XML - either using SELECT INTO or by explicitly creating the table before you begin.
PS - it's a side issue unrelated to your problem, but you might want to check that you're not losing Unicode characters in the original MySQL XMLData field by this conversion since the text/varchar data types won't support them.
Can you ADD a new column of type xml?
If so, add the new xml column, update the table to set the new column equal to the XmlData column and then drop the XmlData column.
Edit
I have a table "TestTable" with a "nvarchar(max)" column.
select * from sys.tables where name = 'TestTable'
This gives a result containing:
[lob_data_space_id] [text_in_row_limit] [large_value_types_out_of_row]
1 0 0
yet I can happily save 500k characters in my nvarchar(max) field.
What do you get if you query sys.tables for your OrderItems table?
If your [text_in_row_limit] is not zero, try this, which should convert any existing in-row strings into BLOBs:
exec sp_tableoption 'OrderItems', 'text in row', 0
and then try to switch from nvarchar(max) to xml.
From BOL,
Disabling the text in row option or
reducing the limit of the option will
require the conversion of all BLOBs;
therefore, the process can be long,
depending on the number of BLOB
strings that must be converted. The
table is locked during the conversion
process.
SQL Server 2005 Question:
I'm working on a data conversion project where I'm taking 80k+ rows and moving them from one table to another. When I run the TSQL, it bombs with various errors having to do with converting types, or whatever. Is there a way to find out what row caused the error?
=====================
UPDATE:
I'm performing an INSERT INTO TABLE1 (...) SELECT ... FROM TABLE2
Table2 is just a bunch of varchar fields where TABLE1 has the right types.
This script will be put into a sproc and executed from an SSIS package. The SSIS package first imports 5 large flat files into TABLE2.
Here is a sample error message: "The conversion of a char data type to a datetime data type resulted in an out-of-range datetime value."
There are many date fields. In TABLE2, there are data values like '02/05/1075' for Birthdate. I want to examine each row that is causing the error, so I can report to the department responsible for the bad data so they can correct it.
This is not the way to do it with SSIS. You should have the data flow from your source, to your destination, with whatever transformations you need in the middle. You'll be able to get error details, and in fact, error rows by using the error output of the destination.
I often send the error output of a destination to another destination - a text file, or a table set up to permit everything, including data that would not have been valid in the real destination.
Actually, if you do this the standard way in SSIS, then data type mismatches should be detected at design time.
What I do is split the rowset in half with a WHERE clause:
INSERT MyTable(id, datecol) SELECT id, datecol FROM OtherTable WHERE ID BETWEEN 0 AND 40,000
and then keep changing the values on the between part of the where clause. I've done this by hand many times, but it occurs to me that you could automate the splitting with a little .Net code in a loop, trapping exceptions and then narrowing it down to just the row throwing the exception, little by little.
I assume you do the update with the INSERT INTO ...
Instead try to do the update with the cursor, use exception handling to catch the error and log all you need: the row number it failed on etc.
Not exactly a cursor, but as effective - I had over 4 million rows to examine with multiple conversion failrues. Here is what I used, and it resulted in a two temp tables one with all my values and assigned rows and one that simply contained a list of rows in the first temp table that failed to convert.
select row_number() over (order by TimeID) as rownum,timeID into #TestingTable from MyTableWithBadData
set nocount on
declare #row as int
declare #last as int
set #row=0
select #last = count(*) from #TestingTable
declare #timeid as decimal(24,0)
create table #fails (rownum int)
while #row<=#last
begin
Begin Try
select #timeid=cast(timeID as decimal(24,0)) from #TestingTable where rownum = #row
end try
begin catch
print cast(#row as varchar(25)) + ' : failed'
insert into #fails(rownum) values(#row)
end catch
set #row = #row+1
end
if you are looping, add prints in the loop.
if you are using set based operations, add a restrictive WHERE condition and run it. Keep running it (each time making it more and more restrictive) until you can find the row in the data. if you could run it for blocks of N rows, then just select out those rows and look at them.
ADD CASE statements to catch the problems (converting that bad value to NULL or whetever) and put a value in a new FlagColumn telling you the type of problem:
CASE WHEN ISNUMERIC(x)!=1 then NULL ELSE x END as x
,CASE WHEN ISNUMERIC(x)!=1 then 'not numeric' else NULL END AS FlagColumn
then select out the new converted data where FlagColumn IS NOT NULL
you could try using select statements with isnumeric() or isdate() functions on the various columns of the source data
EDIT
There are many date fields. In TABLE2,
there are data values like
'02/05/1075' for Birthdate. I want to
examine each row that is causing the
error, so I can report to the
department responsible for the bad
data so they can correct it.
Use this to return all bad date rows:
SELECT * FROM YourTable WHERE ISDATE(YourDateColumn)!=1
If you are working with cursors, yes and is trivial. If you are not working with cursors, I don't think so because SQL operations are ACID, or transactions per se.
John Sauders has the right idea, there are better ways to do this kind of processing using SSIS. However, learning SSIS and redoing your package to completely change the process may not be an option at this time, so I offer this advice. You appear to be having trouble with the dates being incorrect. So first run a query to identify those records which are bad and insert them into an execptions table. Then do you insert only of those records that are left. Something like:
insert exceptiontable (field1, field2)
select field1, field2 from table2 where isdate(field2) = 0
insert table1 (field1, field2)
select field1, field2 from table2 where isdate(field2) = 1
Then of course you can send the contents of the exception table to the people who provide the bad data.