I try to run my stored procedure in Airflow. Simply, I imported mssql operator and tried to execute following:
sql_command = """ EXEC [spAirflowTest] """
t3 = MsSqlOperator( task_id = 'run_test_proc',
mssql_conn_id = 'FIConnection',
sql = sql_command,
dag = dag,
database = 'RDW')
It completes this task as successful. However, task is not even executed. Because I get no error from system, I also cannot identify the error. To identify whether it arrived to my microsoft sql server, I checked with data profiling and it seems like server gets the command but does not execute it. Indeed, I can see sql command in data profiling tool.
When I run command for reading something, like :
select *
from sys.tables
it returns successful, also, with result. How can I solve this problem? Is there anyone who encountered with this issue?
sql_command = """ EXEC [spAirflowTest] """
t3 = MsSqlOperator( task_id = 'run_test_proc',
mssql_conn_id = 'FIConnection',
sql = sql_command,
dag = dag,
database = 'RDW',
autocommit = True)
adding autocommit as above solved the issue
Related
Initially I had used the following command to rename SQL tables:
Q = """sp_rename {}, {}""".format(OLD_TABLE_NAME,NEW_TABLE NAME)
However, this caused an "Lock request time out period exceeded" error, which I believe was due to the lack of "commit" at the end of the query (although I am not confident on this).
So instead, I adopted a new query (adapted from this question).
Q2 = """BEGIN TRANSACTION
GO
EXECUTE sp_rename N'{}', N'{}', 'OBJECT'
GO
ALTER TABLE {} SET (LOCK_ESCALATION = TABLE)
GO
COMMIT""".format(OLD_TABLE_NAME,NEW_TABLE NAME,NEW_TABLE NAME)
However, I'm now getting a ProgrammingError saying "Incorrect syntax near 'GO'."
Do I need to remove some parts of Q2 for the query to work? Or is some other part wrong?
Below are the two functions I use to connect to my SQL server:
from sqlalchemy import create_engine
import pypyodbc as pp
server1 = {
'drivername': 'mssql+pyodbc',
'servername': 'SERVERNAME',
#'port': '5432',
'username': 'WebAccess',
'password': ':|Ax-*6_6!5H',
'driver': 'SQL Server Native Client 11.0',
'trusted_connection': 'yes',
'legacy_schema_aliasing': False
}
def getEngine(servername, database):
DB = server1
#Create connection to SQL database
DB['database'] = database
servername1 = servername.lower()
engine = create_engine('mssql+pyodbc://' + DB['username'] + ':' + DB['password'] + '#' + DB['servername'] + '/' + DB['database'] + '?' + 'driver=' + DB['driver'])#, echo=True)
return engine
def SQLcommand(query,servername,database):
connection = pp.connect("""Driver={SQL Server};Server=""" + servername + """;Database=""" + database + """;uid=USERNAME;pwd=PASSWORD""")
cursor = connection.cursor()
cursor.execute(query)
connection.commit()
connection.close()
This works for me based on the note in the SQLAlchemy Docs:
sql_stmt = f"""EXECUTE sp_rename '{table2}', '{table1}';"""
with connection.begin() as conn:
conn.execute(text(sql_stmt))
Alright dude, you got a few problems here.
pypyodbc is something you should move away from. That library was
all the rage a while ago, but to my knowledge it's not getting many
commits anymore, google is moving away from it, in my build we moved
away from it for a number of reasons. It was great while it lasted
but I think its on the out.
You cannot use 'GO' in a query. 'GO' is not a tsql statement, its a sqlcmd command - thus it doesn't work with odbc. 'GO' essentially just breaks the code into batches, so to do this not using 'GO' you need to run multiple batches, something like:
conn = engine.connect()
tran = conn.transaction()
conn.execute(f"EXECUTE sp_rename N'{OLD_TABLE_NAME}', N'{NEW_TABLE_NAME}', 'OBJECT'")
conn.execute(f"ALTER TABLE {NEW_TABLE_NAME} SET LOCK_ESCALATION = TABLE")
tran.commit()
I'm sure there's more going on here as well, but hopefully this'll get you going.
For anyone still wondering how to solve the problem, I fixed Jamie Marshall's answer in order to adapt it to new transaction api:
conn = engine.connect()
def rename(cnxn):
rename_query = f"EXECUTE sp_rename N'{OLD_TABLE_NAME}', N'{NEW_TABLE_NAME}', 'OBJECT'"
cnxn.execute(rename_query)
conn.transaction(rename)
What I am using
Ubuntu 16.04
Python 3.6
FreeTDS, TDS Version 7.3
SQLAlchemy 1.2.5
Windows server 2012
SQL Server 2008 Enterprise
My purpose
I write code in Python on Ubuntu machine to insert and execute stored procedure on MS SQL Server 2008. I create an order for customer. An order may have many main ingredients, toppings. When finish order, I run a stored procedure to process data to user_order and employee_order.
The stored procedure
In stored procedure, when select data from source tables and process data, if any error is happened, transaction is rolled back.
My code snippet
def process():
engine = get_engine() # my method get engine by connection string
session_maker = sessionmaker(bind=engine.execution_options(isolation_level='SERIALIZABLE'))
session = session_maker()
ref = 'REF0000001'
try:
# Create order
order = Order(id=1, ref=ref)
# Add main ingredients
main1 = Main(order=1, name='coffee')
main2 = Main(order=1, name='milk')
# Topup
topup1 = TopUp(order=1, name='cookies')
topup2 = TopUp(order=1, name='chocolate')
session.add(order)
session.flush()
session.add_all([main1, main2])
session.flush()
session.add_all([topup1, topup2])
session.flush()
session.commit()
except:
session.rollback()
reraise
finally:
session.close()
del session
time.sleep(1)
session = session_maker()
session.execute('EXEC finish_order %a' % ref)
session.commit()
session.close()
del session
And result is
There is no error, but there is no data in user_order and employee_order even though stored procedure finish_order is run.
But, if I run the stored procedure again as a simple query in terminal or SQL Studio Management, the data is imported to destination tables.
Doubts
Is there any chance that data has not been finished inserting into origin tables yet when stored procedure is called?
Please help me with this case.
Thank you!
Using SQL Server 2008 R2 I am getting deadlocks when the same update statement (with different parameters) is running concurrently. Here is the deadlock graph (sorry cannot post images on here yet):
http://i.stack.imgur.com/E6JBK.png
And here is the actual execution plan:
http://i.stack.imgur.com/emm9i.png
The update is like this:
exec sp_executesql N'UPDATE mapping.IssuerAlternateName
SET
UseCount = UseCount + 1,
MostRecentlyAppeared = GETDATE(),
MostRecentlyAppearedUnderlyingAssetName = #p1
WHERE ID = #p0
',N'#p0 int,#p1 nvarchar(4000)',#p0=1234,#p1=N'blah blah blah'
If I have understood things correctly we are trying to read and write from the same index (PK_IssuerAlternateName_1).
Is there any way to resolve this? I was wondering if adding an additional index to the primary key and using WITH INDEX might fix it by stopping the read of PK_IssuerAlternateName_1 (sorry the full name is truncated in the execution plan screenshot).
Or is the best option just to live with this and retry the transaction, which is how the error is currently handled in .NET client. It is certainly successful on retry, but it would be good to avoid the deadlock if possible.
Thanks
In situations similar to this, I have used the UPDLOCK hint to let the database know I intend to update this row. It is not implied by the UPDATE statement. Without the lock hint, it will first obtain a "shared" lock, and then try to escalate. However, this causes deadlocks in certain scenarios.
You will need to do this within your own TransactionScope to ensure everything works correctly.
var sql = #"UPDATE mapping.IssuerAlternateName with (UPDLOCK)
SET
UseCount = UseCount + 1,
MostRecentlyAppeared = GETDATE(),
MostRecentlyAppearedUnderlyingAssetName = #p1
WHERE ID = #p0";
var options = new TransactionOptions()
{
IsolationLevel = IsolationLevel.ReadCommitted // don't use Serializable!
};
using (var scope = new TransactionScope(TransactionScopeOption.RequiresNew, options))
{
using (var context = new YourDbContext())
{
// execute your command here
}
}
When I execute the code i got wrong sysdate for my oracle database.But when I execute that code on my terminal it updated real sysdate.
con = apprEvent.getConnection();
PreparedStatement pst1 = con.prepareStatement("UPDATE EVENTS SET E_STATUS='Approved',
E_AUTH_TIME=SYSDATE WHERE E_ID=" + eid);
queryRes = pst1.executeUpdate();
I'm trying to run the following UPDATE query from a python script (note I've removed the database info):
print 'Connecting to db for update query...'
db = pyodbc.connect('DRIVER={FreeTDS};SERVER=<removed>;DATABASE=<removed>;UID=<removed>;PWD=<removed>')
cursor = db.cursor()
print ' Executing SQL queries...'
for i in range(len(data)):
sql = '''
UPDATE product.sanction
SET action_summary = '{action_summary}'
WHERE sanction_id = {sanction_id};
'''.format(sanction_id=data[i][0], action_summary=data[i][1])
cursor.execute(sql)
cursor.close()
db.commit()
db.close()
However, it hangs indefinitely, no error.
I'm new to pyodbc, but it should be setup correctly considering I'm having no problems performing SELECT queries. I did have to call CAST for SELECT queries (I've cast sanction_id AS INT [int identity on the database] and action_summary AS TEXT [nvarchar on the database]) to properly populate data, so perhaps the problem lies somewhere there, but I don't know where to start debugging. Converting the text to NVARCHAR didn't do anything either.
Here's an example of one of the rows in data:
(2861357, 'Exclusion Program: NonProcurement; Excluding Agency: HHS; CT Code: Z; Exclusion Type: Prohibition/Restriction; SAM Number: S4MR3Q9FL;')
I was unable to find my issue, but I ended up using QuerySets rather than running an UPDATE query.