I'm using Delphi-XE2 Enterprise, SQLServer 2008 R2 on Windows Server 2008 R2. When a SQL error is raised, the DataSnap server captures it and returns it to the client application. This all works, except the error message it truncated to 256 characters.
A stack track of the DataSnap server when the error is raised shows the SQL error message is truncated when the SQL DB Error is retrieved through the DBX Framework using the [file:Data.DBXDynalink]TDBXMethodTable.RaiseError() method. The following line of code from the RaiseError() method sets the MessageLength to 256, even when the error message is longer. Thus the error message returned to the client is never longer than 256 characters.
Status := FDBXBase_GetErrorMessageLength(DBXHandle, DBXResult, MessageLength);
FDBXBase_GetErrorMessageLength is of type TDBXCommon_GetErrorMessageLength, which is defined as:
TDBXCommon_GetErrorMessageLength = function(Handle: TDBXCommonHandle; LastErrorCode: TDBXErrorCode; out ErrorLen: TInt32): TDBXErrorCode; stdcall;
Is there a way to increase the buffer size to allow more of the database error to be sent to the client (e.g., 512 characters), or is this just a limitation of the DBX Framework?
#Sertac found a very similar problem that Carlos had with a driver truncating the error messages. Carlos found that changing the midas.dll to increase the error buffersize fixed his problem. I'm sure the solution is the same for my issue.
Thanks #Sertac for providing the link to Carlos' question:
Reconcile Error: Has anyone had problems with truncated error messages?
Related
I run some R code after querying 100M records and get the following error after the process runs for over 6 hours:
Msg 39004, Level 16, State 19, Line 300
A 'R' script error occurred during execution of 'sp_execute_external_script'
with HRESULT 0x80004005.
HRESULT 0x80004005 appears to be associated in Windows with Connectivity, Permissions or an "Unspecified" error.
I know from logging in my R code that the process never reaches the R script at all. I also know that the entire procedure completes after 4 minutes on a smaller number of records, for example, 1M. This leads me to believe that this is a scaling problem or some issue with the data, rather than a bug in my R code. I have not included the R code or the full query for proprietary reasons.
However, I would expect a disk or memory error to display a 0x80004004 Out of memory error if that were the case.
One clue I noticed in the SQL ERRORLOG is the following:
SQL Server received abort message and abort execution for major error : 18
and minor error : 42
However the time of this log line does not coincide with the interruption of the process, although it does occur after it started. Unfortunately, there is precious little on the web about "major error 18".
A SQL Trace when running from SSMS shows the client logging in and logging out every 6 minutes or so, but I can only assume this is normal keepalive behaviour.
The sanitized sp_execute_external_script call:
EXEC sp_execute_external_script
#language = N'R'
, #script = N'#We never get here
#returns name of output data file'
, #input_data_1 = N'SELECT TOP 100000000 FROM DATA'
, #input_data_1_name = N'x'
, #output_data_1_name = N'output_file_df'
WITH RESULT SETS ((output_file varchar(100) not null))
Server Specs:
8 cores
256 GB RAM
SQL Server 2016 CTP 3
Any ideas, suggestions or debugging hints would be greatly appreciated!
UPDATE:
Set TRACE_LEVEL=3 in rlauncher.config to turn on a higher level of logging and re-ran the process. The log reveals a cleanup process that ran, removing session files, at the time the entire process failed after 6.5 hours.
[2016-05-30 01:35:34.419][00002070][00001EC4][Info] SQLSatellite_LaunchSatellite(1, A187BC64-C349-410B-861E-BFDC714C8017, 1, 49232, nullptr) completed: 00000000
[2016-05-30 01:35:34.420][00002070][00001EC4][Info] < SQLSatellite_LaunchSatellite, dllmain.cpp, 223
[2016-05-30 08:04:02.443][00002070][00001EC4][Info] > SQLSatellite_LauncherCleanUp, dllmain.cpp, 309
[2016-05-30 08:04:07.443][00002070][00001EC4][Warning] Session A187BC64-C349-410B-861E-BFDC714C8017 cleanup wait failed with 258 and error 0
[2016-05-30 08:04:07.444][00002070][00001EC4][Info] Session(A187BC64-C349-410B-861E-BFDC714C8017) logged 2 output files
[2016-05-30 08:04:07.444][00002070][00001EC4][Warning] TryDeleteSingleFile(C:\PROGRA~1\MICROS~1\MSSQL1~1.MSS\MSSQL\EXTENS~1\MSSQLSERVER06\A187BC64-C349-410B-861E-BFDC714C8017\Rscript1878455a2528) failed with 32
[2016-05-30 08:04:07.445][00002070][00001EC4][Warning] TryDeleteSingleDirectory(C:\PROGRA~1\MICROS~1\MSSQL1~1.MSS\MSSQL\EXTENS~1\MSSQLSERVER06\A187BC64-C349-410B-861E-BFDC714C8017) failed with 32
[2016-05-30 08:04:08.446][00002070][00001EC4][Info] Session A187BC64-C349-410B-861E-BFDC714C8017 removed from MSSQLSERVER06 user
[2016-05-30 08:04:08.447][00002070][00001EC4][Info] SQLSatellite_LauncherCleanUp(A187BC64-C349-410B-861E-BFDC714C8017) completed: 00000000
It appears the only way to allow my long-running process to continue is to:
a) Extend the Job Cleanup wait time to allow the job to finish
b) Disable the Job Cleanup process
I have thus far been unable to find the value that sets the Job Cleanup wait time in the MSSQLLaunchpad service.
While a JOB_CLEANUP_ON_EXIT flag exists in rlauncher.config, setting it to 0 has no effect. The service seems to reset it to 1 when it is restarted.
Again, any suggestions or assistance would be much appreciated!
By default, SQL Server reads all data into R memory as a Data Frame before starting execution of R script. Based on the fact that the script works with 1M rows and fails to start with 100M rows, this could potentially be an Out of Memory error. To resolve memory issues, (other than increasing memory on machine/reducing data size) you can try one of these solutions
Increase memory allocation for R process execution using sys.resource_governor_external_resource_pools max_memory_percent setting. By default, SQL Server limits R process execution to 20% of memory.
Streaming execution for R script instead of loading all data into memory. Note that this parameter can only be used in cases where the output of the R script doesn’t depend on reading or looking at the entire set of rows.
The Warnings in RLauncher.log about data cleanup happened after the R script execution can be safely ignored and probably not the root cause for the failures you are seeing.
Unable to resolve this issue in SQL, I simply avoided the SQL Server Launchpad service which was interrupting the processing and pulled the data from SQL using the R RODBC library. The pull took just over 3 hours (instead of 6+ using sp_execute_external_procedure).
This might implicate the SQL Launchpad service, and suggests that memory was not the issue.
Please try your scenario in SQL Server 2016 RTM. There have been many functional and performance fixes made since CTP3.
For more information on how to get the SQL Server 2016 RTM checkout SQL Server 2016 is generally available today blogpost.
I had almost the same issue with SQL Server 2016 RTM-CU1. My Query failed with error 0x80004004 instead of 0x80004005. And it failed beginning with 10,000,000 records, but that could be related to only having 16 GB memory and/or different data.
I got around it by using a field list instead of "*". Even if the field list contains all the fields from the data source (a rather complicated view in my case), a query featuring a field list is always successful, while "SELECT TOP x * FROM ..." always fails for some large x.
I've had the a similar error (0x80004004), and the problem was that one of the rows in one of the columns contained a "very" special character (I'm saying "very" because other special characters did not cause this error).
So that when I replaced 'Folkelånet Telefinans' with 'Folkelanet Telefinans', the problem went away.
In your case, maybe at least one of the values in the last 99M rows contains something like that character, and you just have to replace it. I hope that Microsoft will resolve this issue at some point.
I set up the system first by
[old_path]=which('rdsamp');if(~isempty(old_path)) rmpath(old_path(1:end-8)); end
wfdb_url='http://physionet.org/physiotools/matlab/wfdb-app-matlab/wfdb-app-toolbox-0-9-3.zip';
[filestr,status] = urlwrite(wfdb_url,'wfdb-app-toolbox-0-9-3.zip');
unzip('wfdb-app-toolbox-0-9-3.zip');
cd mcode
addpath(pwd);savepath
I am trying to read databases from Physionet.
I have successfully reached one database mitdb by
[tm,sig]=rdsamp('mitdb/100',1)
but I want to reach the database ptbdb unsuccessfully by
[tm,sig]=rdsamp('ptbdb/100',1)
and get the error
Warning: Could not get signal information. Attempting to read signal without buffering.
> In rdsamp at 107
Error: Cannot convert to double:
init: can't open header for record ptbdb/100
Error using rdsamp (line 145)
Java exception occurred:
java.lang.NumberFormatException: Cannot convert
at org.physionet.wfdb.Wfdbexec.execToDoubleArray(Unknown Source)
The first error message refers to these lines in rdsamp.m:
if(isempty(N))
[siginfo,~]=wfdbdesc(recordName);
if(~isempty(siginfo))
N=siginfo(1).LengthSamples;
else
warning('Could not get signal information. Attempting to read signal without buffering.')
end
end
This line if(~isempty(siginfo)) is false means that the siginfo is empty that is there is no signal. Why? No access to the database, I think.
I think other errors follow from it.
So the error must follow from this line
[siginfo,~]=wfdbdesc(recordName);
What does the snake mean here in the brackets?
How can you get data from ptbdb by Matlab?
So
Does this error mean that the connection cannot be established to the database?
or
that there does not exists such data in the database?
It would be very nice to know how you can check if you have connection to the database like in Postrgres. It would be much easier to debug.
If you run physionetdb("ptdb",1) it will download the files to your computer. You will then be able to see the available records in the <current-dir>/ptdb/
Source: physionetdb function documentation. You are interested in the DoBatchDownload parameter.
After downloading it, I believe every command from the toolbox will check if you have the files locally before fetching from the server (as long as you give the function the correct path to the local files).
The problem is that the data unit "100" does not exist in the database ptbdb.
I run finally successfully after waiting 35 minutes with 100Mb cable broadband:
db_list = physionetdb('ptbdb')
and get not complete data finally to the patient 54 - there should be 294 patients.
'ptbdb/patient001/s0014lre' 'ptbdb/patient001/s0014lre' ... cut ...
The main developer, Ikaro's' answer helped me to wait so long:
The WFDB Toolbox connects to PhysioNet's file server. The databases
accessible through the WFDB Toolbox are not SQL database, they consist
of flat files. The error message that you are getting regarding the
ptdb/100 database is because you are attempting to get a record that
does not exist on the database.
For more information on a particular database or record in PhysioNet
please type:
help physionetdb
and
physionetdb('ptdb')
This flat file system is really a bottle neck in the system.
It would be a good time to change to SQL.
SqlException has the property Number.
Then there is this: http://msdn.microsoft.com/en-us/library/cc645603.aspx
and this: http://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx
And it seems to be one or the other
QUESTION:
How is it decided which?
REASON FOR ASKING:
I need to catch certain SqlExceptions and decide how to deal with them based on the Number property but I don't know which list I should look at when it seems like the system is using messages from both, and I don't know what criteria is used for choosing.
For example:
Number 53 - from server error message list (exists on both)
Number 10054 - from system error message list (exists on both)
Number -1 - from server error message list (exists only on server list)
Number 121 - from system error message list (exists on both)
......
The theory goes that it's the SQL error number, eg. the server side ERROR_NUMBER(). In other words, the first list.
However there are a number of exceptions reported by SqlClient that occur on the client side, not on the server side. A typical example would be an error like failure to connect to the server, since you did not connect there is no server side error to speak of. For example a bad server name (does not resolve in DNS), in such cases the InnerException will point toward a Win32Exception with NativeErrorCode value of ERROR_BAD_NETPATH. In this case 53, the OS system error code, will be reported as SqlException error number.
Other cases the error reported by the SqlClient is a socket error, like an abrupt disconnect. Again, there is no 'server side' error to speak of, and the SqlException will wrap an InnerException of type SocketException (a subclass of Win32Error) with the NativeErrorCode of one of the well known WSA error numbers, like WSAECONNRESET. In this case the SqlException.ErrorNumber will be 10054, but it's the 10054 from the WSA range, not the 10054 from the SQL Server errors range. I know...
So what are you supposed to do? Make sure you check the InnerException, if it's a Win32Exception then the ErrorNumber is coming from a system error code (OS error). Otherwise it should be a SQL Server error number.
Oh, and then there is -1... I think that is reported by SqlClient itself (eg. some internal state errors) but I'm not sure.
I would look at the documentation for the SqlException.Number property.
This is what it says
This is a wrapper for the Number property of the first SqlError in the
Errors property. For more information on SQL Server engine errors, see
SQL Server Books Online.
I seem to be experiencing a strange issue when using the SqlErrText from a DataWindow in our application.
The DataWindow executes a stored procedure, lets call it vp_ut_storedProc, and in my case it throws an error when the DataWindow is updated.
When I go into the function to build the error message for our application, the SqlErrText is passed in as follows (as parameter asErrText):
SQLSTATE = 42000
Microsoft SQL Server Native Client 10.0
TimestampNV|Someone Has Updated the Record. Please Refresh.
No changes made to the database
execute dbo.vp_ut_storedProc
I know that PowerBuilder receives the entire error text which is a good thing. But, when we try to save the error text into a local variable
isErrText = asErrText
The value of isErrText is: SQLSTATE = 42000
So, for some reason, it completely ignores everything after the second line. I figure it has something to do with the way the PowerBuilder is reading in the string, but I don't know why this happens.
I should also note that it's not just limited to this one DataWindow. It happens on quite a few of them.
put this in your dberror event of your datawindow
string s_temp
long l_start, l_end
l_start = 42 +Pos(sqlerrtext,'Microsoft SQL Server Native Client 10.0')
l_end = Pos(sqlerrtext,sqlsyntax) - l_start - 3
IF l_end <=0 THEN l_end = Len(sqlerrtext) - l_start
s_temp = Mid(sqlerrtext,l_start,l_end)
fw_msg(s_temp) //can write this at messagebox(this.title,s_temp) // instead
RETURN 1
if its insert or select error. it ignores the syntax and outputs the error. if its a different error. it shows the rest of the errcode
I've managed to connect from SBCL running on debian to an SQL Server 2000 instance over the network using FreeTDS/unixODBC.
I can actually get data back from the server, so all is working.
However, many of the columns trigger what seem to be unsupported data types a-la:
The value 2147483647 is not of type FIXNUM.
and
-11 fell through ECASE expression.
Wanted one of (-7 -6 -2 -3 -4 93 92 91 11 10 ...).
Anyone have experience using CLSQL with SQL Server would be able to help out?
This (error with 2147483647) occurs because the FreeTDS driver doesn't handle OLEDB BLOBs so well.
You have to issue the following SQL command to make it work:
set textsize 102400
You can see the FreeTDS FAQ entry here. Excerpt:
The text data type is different from char and varchar types. The maximum data length of a text column is governed by the textsize connection option. Microsoft claims in their documentation to use a default textsize of 4000 characters, but in fact their implementation is inconsistent. Sometimes text columns are returned with a size of 4 GB!
The best solution is to make sure you set the textsize option to a reasonable value when establishing a connection.
As for the ECASE expression, I haven't really solved it but I have hacked it away by doing a data conversion of timestamp into a binary value, and a uniqueidentifier into a varchar(36).