is it possible to see stdout (or log) T-SQL PRINT messages embedded in stored procedures when executed using sqlalchemy?
--edit--
It looks like pyodbc added a way to retrieve messages from a cursor for this very reason and it has been merged to master. I still cant figure out how to get this to bubble up to sqlalchemy
https://github.com/mkleehammer/pyodbc/pull/765#
--endedit--
I tried this adding echo=True to engine and setting a log level to debug but I can not see anything
this is how I am executing the usp
import sqlalchemy as sa
import logging
logging.basicConfig()
logger = logging.getLogger("sqlalchemy.engine")
logger.setLevel(logging.DEBUG)
engine = sa.create_engine(target_uri, echo=True)
statement = sa.sql.text(f"EXEC {usp_schema}.{usp_name}")
with engine.begin() as cnxn:
cnxn.execute(statement)
CREATE PROCEDURE dbo.usp_do_stuff
BEGIN
BEGIN TRAN
PRINT 'DELETE STAGE'
PRINT 'LOAD STAGE'
PRINT 'CLEAN STAGE'
PRINT 'TRANSPORT STAGE'
END TRAN
END;
https://github.com/mkleehammer/pyodbc/pull/765
has only recently been merged (2021-01-21) and SQLAlchemy's mssql+pyodbc:// dialect does not know anything about it yet. If you would like to see support for that feature added to SQLAlchemy then please open a feature request here.
In the meantime, the workaround is to install pyodbc from the current master branch …
pip install git+https://github.com/mkleehammer/pyodbc.git
… and then use a .raw_connection() to execute the SP and retrieve the messages:
"""existing stored procedure:
CREATE PROCEDURE [dbo].[usp_do_stuff]
AS
BEGIN
BEGIN TRANSACTION
PRINT 'DELETE STAGE'
PRINT 'LOAD STAGE'
PRINT 'CLEAN STAGE'
PRINT 'TRANSPORT STAGE'
COMMIT TRANSACTION
END
"""
cnxn = engine.raw_connection()
crsr = cnxn.cursor()
crsr.execute("EXEC usp_do_stuff")
print(crsr.messages)
while crsr.nextset():
print(crsr.messages)
"""
[('[01000] (0)', '[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]DELETE STAGE')]
[('[01000] (0)', '[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]LOAD STAGE')]
[('[01000] (0)', '[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]CLEAN STAGE')]
[('[01000] (0)', '[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]TRANSPORT STAGE')]
"""
Related
What I am using
Ubuntu 16.04
Python 3.6
FreeTDS, TDS Version 7.3
SQLAlchemy 1.2.5
Windows server 2012
SQL Server 2008 Enterprise
My purpose
I write code in Python on Ubuntu machine to insert and execute stored procedure on MS SQL Server 2008. I create an order for customer. An order may have many main ingredients, toppings. When finish order, I run a stored procedure to process data to user_order and employee_order.
The stored procedure
In stored procedure, when select data from source tables and process data, if any error is happened, transaction is rolled back.
My code snippet
def process():
engine = get_engine() # my method get engine by connection string
session_maker = sessionmaker(bind=engine.execution_options(isolation_level='SERIALIZABLE'))
session = session_maker()
ref = 'REF0000001'
try:
# Create order
order = Order(id=1, ref=ref)
# Add main ingredients
main1 = Main(order=1, name='coffee')
main2 = Main(order=1, name='milk')
# Topup
topup1 = TopUp(order=1, name='cookies')
topup2 = TopUp(order=1, name='chocolate')
session.add(order)
session.flush()
session.add_all([main1, main2])
session.flush()
session.add_all([topup1, topup2])
session.flush()
session.commit()
except:
session.rollback()
reraise
finally:
session.close()
del session
time.sleep(1)
session = session_maker()
session.execute('EXEC finish_order %a' % ref)
session.commit()
session.close()
del session
And result is
There is no error, but there is no data in user_order and employee_order even though stored procedure finish_order is run.
But, if I run the stored procedure again as a simple query in terminal or SQL Studio Management, the data is imported to destination tables.
Doubts
Is there any chance that data has not been finished inserting into origin tables yet when stored procedure is called?
Please help me with this case.
Thank you!
I am trying to use GO to get R to pull a multipart query from a SQL Server database but R keeps erroring out on me when I try this. Does anyone know a workaround to get RODBC to run multipart queries?
Example query:
query2 = "IF OBJECT_ID('tempdb..#ATTTempTable') IS NOT NULL
DROP TABLE #ATTTempTable
GO
SELECT
* INTO #ATTTempTable
FROM ETL.ATT.fact_responses fr
WHERE fr.ResponseDateTime > '2015-07-06'
"
channel <- odbcConnect("<host name>", uid="<uid>", pwd="<pwd>")
raw = sqlQuery(channel, query2)
close(channel)
and result
> raw
[1] "42000 102 [Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Incorrect syntax near 'GO'."
[2] "[RODBC] ERROR: Could not SQLExecDirect 'IF OBJECT_ID('tempdb..#ATTTempTable') IS NOT NULL\n DROP TABLE #ATTTempTable\n\nGO\n\nSELECT\n\t* INTO #ATTTempTable\nFROM ETL.ATT.fact_responses fr\nWHERE fr.ResponseDateTime > '2015-07-06'\n'"
>
Because your query contains multiple line with conditional logic it resembles a stored procedure.
Simply save that stored procedure in SQL Server:
CREATE PROCEDURE sqlServerSp #ResponseDateTime nvarchar(10)
AS
IF OBJECT_ID('tempdb..#ATTTempTable') IS NOT NULL
DROP TABLE #ATTTempTable;
GO
-- suppresses affected rows message so RODBC returns a dataset
SET NO COUNT ON;
GO
-- runs make-table action query
SELECT * INTO #ATTTempTable
FROM ETL.ATT.fact_responses fr
WHERE fr.ResponseDateTime > #ResponseDateTime;
GO
And then run the stored procedure in R. You can even pass parameters like the date:
channel <- odbcConnect("<host name>", uid="<uid>", pwd="<pwd>")
raw = sqlQuery(channel, "EXEC sqlServerSp #ResponseDateTime='2015-07-06'")
close(channel)
You can't. See https://msdn.microsoft.com/en-us/library/ms188037.aspx
you have to divide your query into two statements and run them separately.
I'm trying to run the following UPDATE query from a python script (note I've removed the database info):
print 'Connecting to db for update query...'
db = pyodbc.connect('DRIVER={FreeTDS};SERVER=<removed>;DATABASE=<removed>;UID=<removed>;PWD=<removed>')
cursor = db.cursor()
print ' Executing SQL queries...'
for i in range(len(data)):
sql = '''
UPDATE product.sanction
SET action_summary = '{action_summary}'
WHERE sanction_id = {sanction_id};
'''.format(sanction_id=data[i][0], action_summary=data[i][1])
cursor.execute(sql)
cursor.close()
db.commit()
db.close()
However, it hangs indefinitely, no error.
I'm new to pyodbc, but it should be setup correctly considering I'm having no problems performing SELECT queries. I did have to call CAST for SELECT queries (I've cast sanction_id AS INT [int identity on the database] and action_summary AS TEXT [nvarchar on the database]) to properly populate data, so perhaps the problem lies somewhere there, but I don't know where to start debugging. Converting the text to NVARCHAR didn't do anything either.
Here's an example of one of the rows in data:
(2861357, 'Exclusion Program: NonProcurement; Excluding Agency: HHS; CT Code: Z; Exclusion Type: Prohibition/Restriction; SAM Number: S4MR3Q9FL;')
I was unable to find my issue, but I ended up using QuerySets rather than running an UPDATE query.
The output of sql commands that is visible to users who interactively run SQL commands from SQL Server Management studio, is different than the output you get back from executing an ADO command or ADO query object.
USE [DBNAME]
BACKUP DATABASE [DBNAME] TO
DISK = 'C:\SqlBackup\Backup.mdf'
The successful completion output is like this:
Processed 465200 pages for database 'DBNAME', file 'filename' on file 2.
Processed 2 pages for database 'DBNAME', file 'filename_log' on file 2.
BACKUP DATABASE successfully processed 465202 pages in 90.595 seconds (40.116 MB/sec).
When I execute either a TADOCommand or TADOQuery with the CommandText or SQL set as above, I do not get any such output. How do I read this "secondary output" from the execution of an SQL command? I'm hoping that perhaps via some raw ADO operations I might be able to execute a command and get back the information above, for success, as well as any errors in performing an Sql backup.
Update: The answer below works better for me than my naive attempt, which did not work, using plain Delphi TADOCommand and TADOConnection classes:
create TADOCommand and TADOConnection.
execute command.
get info-messages back.
The problem I experienced in my own coding attempts, is that my first command is "use dbname" and the only recordset I traversed in my code, was the results of the "use dbname" command, not the second command I was executing. The accepted answer below traverses all recordsets that come back from executing the ADO command, and thus it works much better. Since I'm doing all this in a background thread, I actually think it's better to create the raw Com Objects anyways, and avoid any VCL entanglement in my thread. The code below could be a nice component if anybody is interested, let me know and I might make an open source "SQL Backup for Delphi" component.
Here is an example. I've tested it with D7 and MSSQL2000. And it adds to Memo1 all messages from server:
29 percent backed up.
58 percent backed up.
82 percent backed up.
98 percent backed up.
Processed 408 pages for database 'NorthWind', file 'Northwind' on file 1.
100 percent backed up.
Processed 1 pages for database 'NorthWind', file 'Northwind_log' on file 1.
BACKUP DATABASE successfully processed 409 pages in 0.124 seconds (26.962 MB/sec).
Also if it takes a long time consider to implement a WHILE loop not in the main thread.
uses AdoInt,ComObj;
.....
procedure TForm1.Button1Click(Sender: TObject);
var cmd : _Command;
Conn : _Connection;
RA : OleVariant;
rs :_RecordSet;
n : Integer;
begin
Memo1.Clear;
Conn := CreateComObject(CLASS_Connection) as _Connection;
Conn.ConnectionString := 'Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=NorthWind;Data Source=SQL_Server';
Conn.Open(Conn.ConnectionString,'','',Integer(adConnectUnspecified));
cmd := CreateComObject(CLASS_Command) as _Command;
cmd.CommandType := adCmdText;
cmd.Set_ActiveConnection(Conn);
cmd.CommandText := 'BACKUP DATABASE [NorthWind] TO DISK = N''c:\sql_backup\NorthWind'' WITH INIT , NOUNLOAD , NAME = N''NortWind backup'', NOSKIP , STATS = 10, NOFORMAT;';
rs:=cmd.Execute(RA,0,Integer(adCmdText));
while (rs<>nil) do
begin
for n:=0 to(Conn.Errors.Count-1)do begin
Memo1.Lines.Add(Conn.Errors.Item[n].Description);
end;
rs:=rs.NextRecordset(RA);
end;
cmd.Set_ActiveConnection(nil);
Conn.Close;
cmd := nil;
Conn := nil;
end;
I've found this thread (Russian) for stored procedure and correct it for BACKUP command.
I'm trying to automate my db restores during development, using TSQL on SQL Server 2008, using sqlalchemy with pyodbc as a transport.
The command I'm executing is:
"""CREATE DATABASE dbname
restore database dbname FROM DISK='C:\Backups\dbname.bak' WITH REPLACE,MOVE 'dbname_data' TO 'C:\Databases\dbname_data.mdf',MOVE 'dbname_log' TO 'C:\Databases\dbname_log.ldf'"""
Unfortunately, the in SQL Management Studio, after the code has run, I see that the DB remains in state "Restoring...".
If I restore through management studio, it works. If I use subprocess to call "sqlcmd", it works. pymssql has problems with authentication and doesnt even get that far.
What might be going wrong?
The BACKUP and RESTORE statements run asynchronously so they don't terminate before moving on to the rest of the code.
Using a while statement as described at http://ryepup.unwashedmeme.com/blog/2010/08/26/making-sql-server-backups-using-python-and-pyodbc/ solved this for me:
# setup your DB connection, cursor, etc
cur.execute('BACKUP DATABASE ? TO DISK=?',
['test', r'd:\temp\test.bak'])
while cur.nextset():
pass
Unable to reproduce the problem restoring directly from pyodbc (without sqlalchemy) doing the following:
connection = pyodbc.connect(connection_string) # ensure autocommit is set to `True` in connection string
cursor = connection.cursor()
affected = cursor.execute("""CREATE DATABASE test
RESTORE DATABASE test FROM DISK = 'D:\\test.bak' WITH REPLACE, MOVE 'test_data' TO 'D:\\test_data.mdf', MOVE 'test_log' to 'D:\\test_log.ldf' """)
while cursor.nextset():
pass
Some questions that need clarification:
What is the code in use to do the restore using sqlalchemy?
What version of the SQL Server ODBC driver is in use?
Are there any messages in the SQL Server log related to the restore?
Thanks to geographika for the Cursor.nextset() example!
For SQL Alchemy users, and thanks to geographika for the answer: I ended up using the “raw” DBAPI connection from the connection pool.
It is exactly as geographika's solution but with a few additional pieces:
import sqlalchemy as sa
driver = 'SQL+Server'
name = 'servername'
sql_engine_str = 'mssql+pyodbc://'\
+ name\
+ '/'\
+ 'master'\
+ '?driver='\
+ driver
engine = sa.create_engine(sql_engine_str, connect_args={'autocommit': True})
connection = engine.raw_connection()
try:
cursor = connection.cursor()
sql_cmd = """
RESTORE DATABASE [test]
FROM DISK = N'...\\test.bak'
WITH FILE = 1,
MOVE N'test'
TO N'...\\test_Primary.mdf',
MOVE N'test_log'
TO N'...\\test_log.ldf',
RECOVERY,
NOUNLOAD,
STATS = 5,
REPLACE
"""
cursor.execute(sql_cmd)
while cursor.nextset():
pass
except Exception as e:
logger.error(str(e), exc_info=True)
Five things fixed my problem with identical symptoms.
Found that my test.bak file contained the wrong mdf and ldf files:
>>> cursor.execute(r"RESTORE FILELISTONLY FROM DISK = 'test.bak'").fetchall()
[(u'WRONGNAME', u'C:\\Program Files\\Microsoft SQL ...),
(u'WRONGNAME_log', u'C:\\Program Files\\Microsoft SQL ...)]
Created a new bak file and made sure to set the copy-only backup option
Set the autocommit option for my connection.
connection = pyodbc.connect(connection_string, autocommit=True)
Used the connection.cursor only for a single RESTORE command and nothing else
Corrected the test_data MOVE to test in my RESTORE command (courtesy of #beargle).
affected = cursor.execute("""RESTORE DATABASE test FROM DISK = 'test.bak' WITH REPLACE, MOVE 'test' TO 'C:\\test.mdf', MOVE 'test_log' to 'C:\\test_log.ldf' """)