Perl ADO thinks printed output in stored procedure is an error! - sql-server

First of all (in case this is important) I'm using ActiveState's Perl (v5.8.7 built for MSWin32-x86-multi-thread).
I've just emerged from a three hour long debugging session, trying to find the source of an error. I found there was simply no error, but for some reason ADO's connection object was getting the Errors.Count increased with each printed message in my stored procedure's output.
Consider following Transact SQL code:
CREATE PROCEDURE dbo.My_Sample() AS
BEGIN TRAN my_tran
-- Does something useful
if ##error <> 0 BEGIN
ROLLBACK TRAN my_tran
RAISERROR( 'SP My_Sample failed', 16, 1)
END ELSE BEGIN
COMMIT TRAN my_tran
PRINT 'SP My_Sample succeeded'
END
Now imagine a Perl sub more or less like:
sub execute_SQL {
# $conn is an already opened ADO connection object
# pointing to my SQL Server
# $sql is the T-SQL statement to be executed
my($conn, $sql) = #_;
$conn->Execute($sql);
my $error_collection = $conn->Errors();
my $ecount = $error_collection->Count;
if ($ecount == 0 ) { return 0; }
print "\n" . $ecount . " errors found\n";
print "Executed SQL Code:\n$sql\n\n";
print "Errors while executing:\n";
foreach my $error (in $error_collection){
print "Error: [" . $error->{Number} . "] " . $error->{Description} . "\n";
}
return 1;
}
Somewhere else, in the main Perl code, I'm calling the above sub as:
execute_SQL( $conn, 'EXEC dbo.My_Sample' );
In the end I got it that every PRINT statement causes a new pseudo-error to be appended to the ADO Errors collection. The quick fix I implemented was to change that PRINT in the SP into a SELECT, to bypass this.
The questions I'd like to ask are:
Is this behaviour normal?
Is there a way to avoid/bypass it?

This is to be expected as it's what ADO does and the Win32::ADO is quite a thin layer above it.
ref: knowledge base note that the RAISERROR and PRINT statements are returned through the ADO errors collection

OK, after a lot of testing and reading, I came to found it explained in the BOLs' article "Using PRINT" (my emphasis):
The PRINT statement is used to return messages to applications. PRINT takes either a character or Unicode string expression as a parameter and returns the string as a message to the application. The message is returned as an informational error to applications using the SQLClient namespace or the ActiveX Data Objects (ADO), OLE DB, and Open Database Connectivity (ODBC) application programming interfaces (APIs). SQLSTATE is set to 01000, the native error is set to 0, and the error message string is set to the character string specified in the PRINT statement. The string is returned to the message handler callback function in DB-Library applications.
Armed with this knowledge I adapted this VB6 from this DevX article until I got this:
sub execute_SQL {
# $conn is an already opened ADO connection object
# pointing to my SQL Server
# $sql is the T-SQL statement to be executed
# Returns 0 if no error found, 1 otherwise
my($conn, $sql) = #_;
$conn->Execute($sql);
my $error_collection = $conn->Errors();
my $ecount = $error_collection->Count;
if ($ecount == 0 ) { return 0; }
my ($is_message, $real_error_found);
foreach my $error (in $error_collection){
$is_message = ($error->{SQLState} eq "01000" && $error->{NativeError}==0);
$real_error_found=1 unless $is_message;
if( $is_message) {
print "Message # " . $error->{Number}
. "\n Text: " . $error->{Description} ."\n";
} else {
print "Error # " . $error->{Number}
. "\n Description: " . $error->{Description}
. "\nSource: " . $error->{Source} . "\n";
}
}
print $message_to_print;
return $real_error_found;
}
So now my Perl sub correctly sorts out real errors (emitted from SQL Server via a RaisError) and a common message outputted via "PRINT".
Thanks to Richard Harrison for his answer which lead me to the way of success.

Related

Perl performance is slow, file I/O issue or due to while loop

I have the following code in my while loop and it is significantly slow, any suggestions on how to improve this?
open IN, "<$FileDir/$file" || Err( "Failed to open $file at location: $FileDir" );
my $linenum = 0;
while ( $line = <IN> ) {
if ( $linenum == 0 ) {
Log(" This is header line : $line");
$linenum++;
} else {
$linenum++;
my $csv = Text::CSV_XS->new();
my $status = $csv->parse($line);
my #val = $csv->fields();
$index = 0;
Log("number of parameters for this file is: $sth->{NUM_OF_PARAMS}");
for ( $index = 0; $index <= $#val; $index++ ) {
if ( $index < $sth->{NUM_OF_PARAMS} ) {
$sth->bind_param( $index + 1, $val[$index] );
}
}
if ( $sth->execute() ) {
$ifa_dbh->commit();
} else {
Log("line $linenum insert failed");
$ifa_dbh->rollback();
exit(1);
}
}
}
By far the most expensive operation there is accessing the database server; it's a network trip, hundreds of milliseconds or some such, each time.
Are those DB operations inserts, as they appear? If so, instead of inserting row by row construct a string for an insert statement with multiple rows, in principle as many as there are, in that loop. Then run that one transaction.
Test and scale down as needed, if that adds up to too many rows. Can keep adding rows to the string for the insert statement up to a decided maximum number, insert that, then keep going.†
A few more readily seen inefficiencies
Don't construct an object every time through the loop. Build it once befor the loop, and then use/repopulate as needed in the loop. Then, there is no need for parse+fields here, while getline is also a bit faster
Don't need that if statement for every read. First read one line of data, and that's your header. Then enter the loop, without ifs
Altogether, without placeholders which now may not be needed, something like
my $csv = Text::CSV_XS->new({ binary => 1, auto_diag => 1 });
# There's a $table earlier, with its #fields to populate
my $qry = "INSERT into $table (", join(',', #fields), ") VALUES ";
open my $IN, '<', "$FileDir/$file"
or Err( "Failed to open $file at location: $FileDir" );
my $header_arrayref = $csv->getline($IN);
Log( "This is header line : #$header_arrayref" );
my #sql_values;
while ( my $row = $csv->getline($IN) ) {
# Use as many elements in the row (#$row) as there are #fields
push #sql_values, '(' .
join(',', map { $dbh->quote($_) } #$row[0..$#fields]) . ')';
# May want to do more to sanitize input further
}
$qry .= join ', ', #sql_values;
# Now $qry is readye. It is
# INSERT into table_name (f1,f2,...) VALUES (v11,v12...), (v21,v22...),...
$dbh->do($qry) or die $DBI::errstr;
I've also corrected the error handling when opening the file, since that || in the question binds too tightly in this case, and there's effectively open IN, ( "<$FileDir/$file" || Err(...) ). We need or instead of || there. Then, the three-argument open is better. See perlopentut
If you do need the placeholders, perhaps because you can't have a single insert but it must be broken into many or for security reasons, then you need to generate the exact ?-tuples for each row to be inserted, and later supply the right number of values for them.
Can assemble data first and then build the ?-tuples based on it
my $qry = "INSERT into $table (", join(',', #fields), ") VALUES ";
...
my #data;
while ( my $row = $csv->getline($IN) ) {
push #data, [ #$row[0..$#fields] ];
}
# Append the right number of (?,?...),... with the right number of ? in each
$qry .= join ', ', map { '(' . join(',', ('?')x#$_) . ')' } #data;
# Now $qry is ready to bind and execute
# INSERT into table_name (f1,f2,...) VALUES (?,?,...), (?,?,...), ...
$dbh->do($qry, undef, map { #$_ } #data) or die $DBI::errstr;
This may generate a very large string, what may push the limits of your RDBMS or some other resource. In that case break #data into smaller batches. Then prepare the statement with the right number of (?,?,...) row-values for a batch, and execute in the loop over the batches.‡
Finally, another way altogether is to directly load data from a file using the database's tool for that particular purpose. This will be far faster than going through DBI, probably even including the need to process your input CSV into another one which will have only the needed data.
Since you don't need all data from your input CSV file, first read and process the file as above and write out a file with only the needed data (#data above). Then, there's two possible ways
Either use an SQL command for this – COPY in PostgreSQL, LOAD DATA [LOCAL] INFILE in MySQL and Oracle (etc); or,
Use a dedicated tool for importing/loading files from your RDBMS – mysqlimport (MySQL), SQL*Loader/sqlldr (Oracle), etc. I'd expect this to be the fastest way
The second of these options can also be done out of a program, by running the appropriate tool as an external command via system (or better yet via the suitable libraries).
† In one application I've put together as much as millions of rows in the initial insert -- the string itself for that statement was in high tens of MB -- and that keeps running with ~100k rows inserted in a single statement daily, for a few years by now. This is postgresql on good servers, and of course ymmv.
‡
Some RDBMS do not support a multi-row (batch) insert query like the one used here; in particular Oracle seems not to. (We were informed in the end that that's the database used here.) But there are other ways to do it in Oracle, please see links in comments, and search for more. Then the script will need to construct a different query but the principle of operation is the same.

What causes DBD::ODBC::st fetchrow_array failed: st_fetch/SQLFetch and how to avoid it?

EDIT1:
I don't actually get an empty array as stated below. Instead, I get an empty response body because of the following exception:
DBD::ODBC::st fetchrow_array failed: st_fetch/SQLFetch (long truncated DBI attribute LongTruncOk not set and/or LongReadLen too small) (SQL-HY000) [state was HY000 now 01004]
Which I can see there are posts about. I will look at those to see if I can fix this on my on. Will edit if not successful.
First let me start by saying, I do not know Perl well at all. This could be a careless error — I hope it is. I am building a hash from an array that is returned from SQL or from JavaScript on the front-end and one of the keys in the hash, "short-desc" needs to have the value which in the code below will be coming from a SQL database.
BFHHOTH 15x24S/S +2 UP-HNDWHL-UNSPOKED-GALV 15"x24" flush escape hatch w/hinge, internal handwheel, T-handle on top steel cover and ring
However with the code (removed unnecessary cases from switch):
#!perl
use Switch;
use DBI;
use JSON;
use CGI qw /param/;
use CGI::Carp qw(fatalsToBrowser);
use IO::Compress::Gzip qw(gzip $GzipError);
use URI::Encode;
use URI::Escape;
my $gzip_ok;
my $accept_encoding = $ENV{HTTP_ACCEPT_ENCODING};
if ( $accept_encoding && $accept_encoding =~ /\bgzip\b/ ) {
# $gzip_ok = 1;
}
print "Content-Type: application/json\n";
if ($gzip_ok) {
print "Content-Encoding: gzip\n";
}
print "\n";
my $action = param('ACTION');
my %jsonData;
my #jsonArray;
my $azDSN = DBI->connect('dbi:ODBC:Driver={SQL Server Native Client 10.0};Server=myServer;Database=myDB;Uid=me;Pwd={myPass};Encrypt=yes;Connection Timeout=30;');
switch ($action) {
case "GETINFO" {
my $paramID = param('ID');
getInfo($paramID);
my $json_text = JSON->new->pretty->utf8->encode( \#jsonArray );
if ($gzip_ok) {
my $zipText;
gzip \$json_text, \$zipText,
or die "gzip failed: $GzipError\n";
print $zipText;
}
else {
print $json_text;
}
}
}
sub getInfo {
my $myID = $_[0];
my $statement = <<"SQL";
SELECT
trefQuoteItemsID,
quote_position,
description,
comments
FROM
myDB.dbo.myTable where tID = $myID;
SQL
my $sti = $azDSN->prepare($statement) or die $statement;
$sti->execute() or die $DBI::errstr;
while ( my #row = $sti->fetchrow_array ) {
my %tempData;
%tempData = (
"tref" => $row[0],
"position" => $row[1],
"short_desc" => $row[2],
"comments" => $row[3]
);
$jsonArray[$count] = {%tempData};
$count++;
}
}
An empty array is returned to me on the front-end.
Oddly, if the string is:
BFHHOTH 15x24S/S +2 UP-HNDWHL-UNSPOKED-
the array contains the correct object.
But empty again if the string is:
BFHHOTH 15x24S/S +2 UP-HNDWHL-UNSPOKED-G
Have also tried with strings:
qwertyuiopasdfghjklzxcvbnm1234567890qwe #length is 39
which lets the hash gets built and:
qwertyuiopasdfghjklzxcvbnm1234567890qwer #length is 40
which will return an empty array so hash doesn't get built.
Are there any Perl gurus who have any suggestions?
From what I can tell, it's related to "long object" columns.
DBD::ODBC says:
You can retrieve a lob in chunks like this:
$sth->bind_col($column, undef, {TreatAsLOB=>1});
while(my $retrieved = $sth->odbc_lob_read($column, \my $data, $length)) {
print "retrieved=$retrieved lob_data=$data\n";
}
NOTE: to retrieve a lob like this you must first bind the lob column specifying BindAsLOB or DBD::ODBC will 1) bind the column as normal and it will be subject to LongReadLen and b) fail odbc_lob_read.
NOTE: Some database engines and ODBC drivers do not allow you to retrieve columns out of order (e.g., MS SQL Server unless you are using cursors). In those cases you must ensure the lob retrieved is the last (or only) column in your select list.
NOTE: You can retrieve only part of a lob but you will probably have to call finish on the statement handle before you do anything else with that statement. When only retrieving part of a large lob you could see a small delay when you call finish as some protocols used by ODBC drivers send the lob down the socket synchronously and there is no way to stop it (this means the ODBC driver needs to read all the lob from the socket even though you never retrieved it all yourself).
NOTE: If your select contains multiple lobs you cannot read part of the first lob, the second lob then return to the first lob. You must read all lobs in order and completely or read part of a lob and then do no further calls to odbc_lob_read.
There's no mention if you can retrieve a lob any other way, so I don't know if you have to do it this way, but at least you know one way. But I believe you can work around the problem by increasing the connection's LongReadLen attribute.
You should be able to set the attribute as follows:
my $dbh = DBI->connect($dsn, $user, $passwd, {
...,
LongReadLen => ...,
});
You should also be able to set the attribute as follows:
$dbh->{LongReadLen} = ...;
Hopefully, someone can give you a better answer.

PHP - execute Sql Server stored procedure with 2 output parameters

I'm working with a third-party company in order to integrate an web application with their Windows application.
To write the necessary data into their database they gave me the following procedure:
Declare #SavedOK int, #Message varchar(8000)
Exec pbProcessTransactions
#SavedOK output,
#Message output
I have to execute it with PHP, this is how I've written my application, and for this I used something that I've found on Micrsoft's documentations.
My code which executes the stored procedure with 2 output parameters is:
$tsql_callSP = "{call pbProcessTransactions(?, ? )}";
$SavedOK = '';
settype($SavedOK, "integer");
$Message = '';
settype($Message, "string");
$raspuns = '';
$params = array(
array($SavedOK, SQLSRV_PARAM_INOUT),
array($Message, SQLSRV_PARAM_INOUT)
);
/* Execute the query. */
$stmt3 = sqlsrv_query( $conn, $tsql_callSP, $params);
if( $stmt3 === false ) {
echo "Error in executing statement 3.\n";
die( print_r( sqlsrv_errors(), true)); }
/* Display the value of the output parameters. */
while ($obj=sqlsrv_fetch_object($stmt3)) {
// SET PARAMETERS - SET TERMS
echo $obj->SavedOK;
echo $obj->Message;
}
/*Free the statement and connection resources. */
sqlsrv_free_stmt( $stmt3);
/*WHERE SavedOK and Message are the 2 output parameters which will display various info depending if the procedure executes normally or not, SavedOK will display 1 or 0 , 1 if it's ok 0 if not and Message will display the error message if SavedOK is 0 */
My problem is that the statement is always false, it always displays "Error in executing statement 3" and I cannot see the output, if SavedOK is 1 or 0, which is the message.

"Cursor type changed" error on Perl OLE32 MSSQL dateadd function results

The following sql "select DATEADD(day, DATEDIFF(day, 2, GETDATE()), '20:00:00') as A" runs perfectly fine in Microsoft sql query.
However in perl, it complains with the following error "Description: [Microsoft][ODBC SQL Server Driver]Cursor type changed".
I double checked and my code can run select and update statements with no issues so I am a bit stumbled about the cursor error.
Note that I have not included my connection string but have illustrated that I am using win32::OLE.
use Win32::OLE;
$conn->{ConnectionString} = "...";
$conn->open; # open connection to the DB
$state = $conn ->state; #1 means connected
if($state ne "1"){...
$mssql_select = "select DATEADD(day, DATEDIFF(day, 2, GETDATE()), '20:00:00')
as A";
$rs->Open( $mssql_select, $conn);
my $error_collection = $conn->Errors();
my $ecount = $error_collection->Count;
my ($is_message, $real_error_found);
foreach my $error (in $error_collection)
{
#output error statements
$is_message = ($error->{SQLState} eq "01000" && $error->{NativeError}==0);
$real_error_found=1 unless $is_message;
$status = "ERROR # " . $error->{Number}
. "\n Description: " . $error->{Description}
. "\nSource: " . $error->{Source} . "\n";
}
Results in "Description: [Microsoft][ODBC SQL Server Driver]Cursor type changed".
Any brainstormnig ideas the group can provide would be appreciated.
I figured it out, see below
Solved it, turns out this is just a warning, sql server does not know what type of cursor will be returned, the results actually get returned. See http://social.msdn.microsoft.com/Forums/sqlserver/en-US/e15141e7-3084-487d-a60f-47afac046a55/odbcsql-cursor-error

Invalid parameter number error in Codeigniter

I am working on a process in Codeigniter to take a user-uploaded image (managed using the CI upload library) and insert it into a varbinary(max) field in a SQLServer database. My controller and model code are as follows.
if($this->upload->do_upload($upload_name)) {
//get temp image
$tmpName = $config['upload_path'] . $config['file_name'];
// Read it into $data variable
$fp = fopen($tmpName, 'rb');
$data = fread($fp, filesize($tmpName));
fclose($fp);
//insert into DB
$this->the_model->storeImage($data, $user_id);
//delete temp image
unlink($config['upload_path'] . $config['file_name']);
}
/***** Function from the_model ************/
function storePropertyImage($image_data, $user_id) {
$my_db = $this->load->database('admin');
$stmt = "INSERT INTO my_table (UserID, ImageData) VALUES (" . $my_db->escape($user_id) . ", " . $my_db->escape($image_data) . ")";
$insert = $my_db->query($stmt);
return $insert;
}
This all seems like it should be OK but when I run the code, I get the error:
Fatal error: Uncaught exception 'PDOException' with message
'SQLSTATE[HY093]: Invalid parameter number: mixed named and positional parameters'
in {my app path}\helpers\mssql_helper.php on line 213
I've done some googling on this error message and the results seem to indicate this is the result of there being a colon character in the $data value being sent to the model, making the DB think that I am trying to pass a named parameter when I am not. However I haven't been able to find any reports that match my specific use case or that have much info on how to correct the error.
I'd appreciate any insight on where I might be tripping up.
$image_data is a binary string. ->escape may not work on it, since it may escape random bytes in it, thus leaving you with a corrupted image. Also the binary string may contain quote characters (or other characters) that is making your query invalid.
Try to encode the binary string as hex before inserting into MySQL. You can use PHP's bin2hex for this.
$escaped_user_id = $my_db->escape($user_id);
$hex_image = bin2hex($image_data);
$stmt = "INSERT INTO my_table (UserID, ImageData) VALUES ({$escaped_user_id}, X'{$hex_image}')";
The X in X{$hex_image} is how MySQL handles literal hex strings: http://dev.mysql.com/doc/refman/5.1/en/hexadecimal-literals.html
If that doesn't work, you can also try UNHEX().
$escaped_user_id = $my_db->escape($user_id);
$hex_image = bin2hex($image_data);
$stmt = "INSERT INTO my_table (UserID, ImageData) VALUES ({$escaped_user_id}, UNHEX('{$hex_image}'))";
EDIT: I didn't notice you were using MSSQL and not MySQL. My bad. In MSSQL, you can insert literal hex strings with 0x.
$escaped_user_id = $my_db->escape($user_id);
$hex_image = bin2hex($image_data);
$stmt = "INSERT INTO my_table (UserID, ImageData) VALUES ({$escaped_user_id}, 0x{$hex_image})";

Resources