Execute sp_rename SQL command using pypyodbc - sql-server

Initially I had used the following command to rename SQL tables:
Q = """sp_rename {}, {}""".format(OLD_TABLE_NAME,NEW_TABLE NAME)
However, this caused an "Lock request time out period exceeded" error, which I believe was due to the lack of "commit" at the end of the query (although I am not confident on this).
So instead, I adopted a new query (adapted from this question).
Q2 = """BEGIN TRANSACTION
GO
EXECUTE sp_rename N'{}', N'{}', 'OBJECT'
GO
ALTER TABLE {} SET (LOCK_ESCALATION = TABLE)
GO
COMMIT""".format(OLD_TABLE_NAME,NEW_TABLE NAME,NEW_TABLE NAME)
However, I'm now getting a ProgrammingError saying "Incorrect syntax near 'GO'."
Do I need to remove some parts of Q2 for the query to work? Or is some other part wrong?
Below are the two functions I use to connect to my SQL server:
from sqlalchemy import create_engine
import pypyodbc as pp
server1 = {
'drivername': 'mssql+pyodbc',
'servername': 'SERVERNAME',
#'port': '5432',
'username': 'WebAccess',
'password': ':|Ax-*6_6!5H',
'driver': 'SQL Server Native Client 11.0',
'trusted_connection': 'yes',
'legacy_schema_aliasing': False
}
def getEngine(servername, database):
DB = server1
#Create connection to SQL database
DB['database'] = database
servername1 = servername.lower()
engine = create_engine('mssql+pyodbc://' + DB['username'] + ':' + DB['password'] + '#' + DB['servername'] + '/' + DB['database'] + '?' + 'driver=' + DB['driver'])#, echo=True)
return engine
def SQLcommand(query,servername,database):
connection = pp.connect("""Driver={SQL Server};Server=""" + servername + """;Database=""" + database + """;uid=USERNAME;pwd=PASSWORD""")
cursor = connection.cursor()
cursor.execute(query)
connection.commit()
connection.close()

This works for me based on the note in the SQLAlchemy Docs:
sql_stmt = f"""EXECUTE sp_rename '{table2}', '{table1}';"""
with connection.begin() as conn:
conn.execute(text(sql_stmt))

Alright dude, you got a few problems here.
pypyodbc is something you should move away from. That library was
all the rage a while ago, but to my knowledge it's not getting many
commits anymore, google is moving away from it, in my build we moved
away from it for a number of reasons. It was great while it lasted
but I think its on the out.
You cannot use 'GO' in a query. 'GO' is not a tsql statement, its a sqlcmd command - thus it doesn't work with odbc. 'GO' essentially just breaks the code into batches, so to do this not using 'GO' you need to run multiple batches, something like:
conn = engine.connect()
tran = conn.transaction()
conn.execute(f"EXECUTE sp_rename N'{OLD_TABLE_NAME}', N'{NEW_TABLE_NAME}', 'OBJECT'")
conn.execute(f"ALTER TABLE {NEW_TABLE_NAME} SET LOCK_ESCALATION = TABLE")
tran.commit()
I'm sure there's more going on here as well, but hopefully this'll get you going.

For anyone still wondering how to solve the problem, I fixed Jamie Marshall's answer in order to adapt it to new transaction api:
conn = engine.connect()
def rename(cnxn):
rename_query = f"EXECUTE sp_rename N'{OLD_TABLE_NAME}', N'{NEW_TABLE_NAME}', 'OBJECT'"
cnxn.execute(rename_query)
conn.transaction(rename)

Related

Run Stored Procedure in Airflow

I try to run my stored procedure in Airflow. Simply, I imported mssql operator and tried to execute following:
sql_command = """ EXEC [spAirflowTest] """
t3 = MsSqlOperator( task_id = 'run_test_proc',
mssql_conn_id = 'FIConnection',
sql = sql_command,
dag = dag,
database = 'RDW')
It completes this task as successful. However, task is not even executed. Because I get no error from system, I also cannot identify the error. To identify whether it arrived to my microsoft sql server, I checked with data profiling and it seems like server gets the command but does not execute it. Indeed, I can see sql command in data profiling tool.
When I run command for reading something, like :
select *
from sys.tables
it returns successful, also, with result. How can I solve this problem? Is there anyone who encountered with this issue?
sql_command = """ EXEC [spAirflowTest] """
t3 = MsSqlOperator( task_id = 'run_test_proc',
mssql_conn_id = 'FIConnection',
sql = sql_command,
dag = dag,
database = 'RDW',
autocommit = True)
adding autocommit as above solved the issue

Groovy, safe way to create MSSQL database

I'm trying to use groovy.sql.Sql to create databases in an MSSQL (Microsoft SQL Server) server. It seem like the prepared statement adds additional quotes around the last parameter breaking the query.
This test code:
import groovy.sql.Sql
import com.microsoft.sqlserver.jdbc.SQLServerDataSource
def host = 'myhost'
def port = '1433'
def database = 'mydatabasename'
def usernameName = 'myusername'
def password = 'mypassword'
def dataSource = new SQLServerDataSource()
dataSource.setURL("jdbc:sqlserver://$host:$port")
dataSource.setUser(username)
dataSource.setPassword(password)
def connection new Sql(dataSource)
connection.execute(
'IF EXISTS (SELECT * FROM master.dbo.sysdatabases WHERE name = ?) DROP DATABASE ?',
[ databaseName, databaseName ]
)
Gives the error:
Failed to execute: IF EXISTS (SELECT * FROM master.dbo.sysdatabases WHERE name = ?) DROP DATABASE ? because: Incorrect syntax near '#P1'.
How can I use prepared statements without having it add single quotes around parameter one (DROP DATABASE ? seem to be rewritten as DROP DATABASE '?') or can I write the query in a different way so that the added single quotes does not produce a syntax error?
I would also be fine with other frameworks, if anyone could give me a working example.
Can you try:
connection.execute(
"IF EXISTS (SELECT * FROM master.dbo.sysdatabases WHERE name = $databaseName) DROP DATABASE ${Sql.expand(databseName)}"
)

RODBC::sqlSave - problems creating/appending to a table

Related to several other questions on the RODBC package, I'm having problems using RODBC::sqlSave to write to a table on a SQL Server database. I'm using MS SQL Server 2008 and 64-bit R on a Windows RDP.
The solution in the 3rd link (questions) does work [sqlSave(ch, df)]. But in this case, it writes to the wrong data base. That is, my default DB is "C2G" but I want to write to "BI_Sandbox". And it doesn't allow for options such as rownames, etc. So there still seems to be a problem in the package.
Obviously, a possible solution would be to change my ODBC solution to the specified database, but it seems there should be a better method. And this wouldn't solve the problem of unusable parameters in the sqlSave command--such as rownames, varTypes, etc.
I have the following ODBC- System DSN connnection:
Microsoft SQL Server Native Client Version 11.00.3000
Data Source Name: c2g
Data Source Description: c2g
Server: DC01-WIN-SQLEDW\BISQL01,29537
Use Integrated Security: Yes
Database: C2G
Language: (Default)
Data Encryption: No
Trust Server Certificate: No
Multiple Active Result Sets(MARS): No
Mirror Server:
Translate Character Data: Yes
Log Long Running Queries: No
Log Driver Statistics: No
Use Regional Settings: No
Use ANSI Quoted Identifiers: Yes
Use ANSI Null, Paddings and Warnings: Yes
R code:
R> ch <- odbcConnect("c2g")
R> sqlSave(ch, zinq_scores, tablename = "[bi_sandbox].[dbo].[table1]",
append= FALSE, rownames= FALSE, colnames= FALSE)
Error in sqlColumns(channel, tablename) :
‘[bi_sandbox].[dbo].[table1]’: table not found on channel
# after error, try again:
R> sqlDrop(ch, "[bi_sandbox].[dbo].[table1]", errors = FALSE)
R> sqlSave(ch, zinq_scores, tablename = "[bi_sandbox].[dbo].[table1]",
append= FALSE, rownames= FALSE, colnames= FALSE)
Error in sqlSave(ch, zinq_scores, tablename = "[bi_sandbox].[dbo].[table1]", :
42S01 2714 [Microsoft][SQL Server Native Client 11.0][SQL Server]There is already an object named 'table1' in the database.
[RODBC] ERROR: Could not SQLExecDirect 'CREATE TABLE [bi_sandbox].[dbo].[table1] ("credibility_review" float, "creditbuilder" float, "no_product" float, "duns" varchar(255), "pos_credrev" varchar(5), "pos_credbuild" varchar(5))'
In the past, I've gotten around this by running the supremely inefficient sqlQuery with insert into row-by-row to get around this. But I tried this time and no data was written. Although the sqlQuery statement did not have an error or warning message.
temp <-"INSERT INTO [bi_sandbox].[dbo].[table1]
+ (credibility_review, creditbuilder, no_product, duns, pos_credrev, pos_credbuild) VALUES ("
>
> for(i in 1:nrow(zinq_scores)) {
+ sqlQuery(ch, paste(temp, "'", zinq_scores[i, 1], "'",",", " ",
+ "'", zinq_scores[i, 2], "'", ",",
+ "'", zinq_scores[i, 3], "'", ",",
+ "'", zinq_scores[i, 4], "'", ",",
+ "'", zinq_scores[i, 5], "'", ",",
+ "'", zinq_scores[i, 6], "'", ")"))
+ }
> str(sqlQuery(ch, "select * from [bi_sandbox].[dbo].[table1]"))
'data.frame': 0 obs. of 6 variables:
$ credibility_review: chr
$ creditbuilder : chr
$ no_product : chr
$ duns : chr
$ pos_credrev : chr
$ pos_credbuild : chr
Any help would be greatly appreciated.
Also, if there is any missing detail, please let me know and I'll edit the question.
My apologies up front. This is not exactly a "simple example." It's pretty trivial, but there are a lot of parts. And by the end, you'll probably think I'm crazy for doing it this way.
Starting in SQL Server Management Studio
First, I've created a database on SQL Server called mtcars with default schema dbo. I've also added myself as a user. Under my own user name, I am the database owner, so I can do anything I want to the database, but from R, I will connect using a generic account that only has EXECUTE privileges.
The predefined table in the database that we are going to write to is called mtcars. (So the full path to the table is mtcars.dbo.mtcars; it's lazy, I know). The code to define the table is
USE [mtcars]
GO
/****** Object: Table [dbo].[mtcars] Script Date: 2/22/2016 11:56:53 AM ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[mtcars](
[OID] [int] IDENTITY(1,1) NOT NULL,
[mpg] [numeric](18, 0) NULL,
[cyl] [numeric](18, 0) NULL,
[disp] [numeric](18, 0) NULL,
[hp] [numeric](18, 0) NULL
) ON [PRIMARY]
GO
Stored Procedures
I'm going to use two stored procedures. The first is an "UPSERT" procedure, that will first try to update a row in a table. If that fails, it will insert the row into the table.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE dbo.sample_procedure
#OID int = 0,
#mpg numeric(18,0) = 0,
#cyl numeric(18,0) = 0,
#disp numeric(18,0) = 0,
#hp numeric(18,0) = 0
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
-- TRANSACTION code borrowed from
-- http://stackoverflow.com/a/21209131/1017276
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN TRANSACTION;
UPDATE dbo.mtcars
SET mpg = #mpg,
cyl = #cyl,
disp = #disp,
hp = #hp
WHERE OID = #OID;
IF ##ROWCOUNT = 0
BEGIN
INSERT dbo.mtcars (mpg, cyl, disp, hp)
VALUES (#mpg, #cyl, #disp, #hp)
END
COMMIT TRANSACTION;
END
GO
Another stored procedure I will use is just the equivalent of RODBC::sqlFetch. As far as I can tell, sqlFetch depends on SQL injection, and I'm not allowed to use it. Just to be on the safe side of our data security policies, I write little procedures like this (Data security is pretty tight here, you may or may not need this)
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE dbo.get_mtcars
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
SELECT * FROM dbo.mtcars
END
GO
Now, from R
I have a utility function I use to help me manage inputting data into the stored procedures. sqlSave would do a lot of this automatically, so I'm kind of reinventing the wheel. The gist of the utility function is to determine if the value I'm pushing to the database needs to be nested in quotes or not.
#* Utility function. This does a couple helpful things like
#* Convert NA and NULL into a SQL NULL
#* wrap character strings and dates in single quotes
sqlNullString <- function(value, numeric=FALSE)
{
if (is.null(value)) value <- "NULL"
if (is.na(value)) value <- "NULL"
if (inherits(value, "Date")) value <- format(x = value, format = "%Y-%m-%d")
if (value == "NULL") return(value)
else if (numeric) return(value)
else return(paste0("'", value, "'"))
}
This next step isn't strictly necessary, but I'm going to do it just so that my R table is similar to my SQL table. This is organizational strategy on my part.
mtcars$OID <- NA
Now let's establish our connection:
server <- "[server_name]"
uid <- "[generic_user_name]"
pwd <- "[password]"
library(RODBC)
channel <- odbcDriverConnect(paste0("driver=SQL Server;",
"server=", server, ";",
"database=mtcars;",
"uid=", uid, ";",
"pwd=", pwd))
Now this next part is pure laziness. I'm going to use a for loop to push each row of the data frame the to SQL table one at a time. As noted in the original question, this is kind of inefficient. I'm sure I could write a stored procedure to accept several vectors of data, compile them into a temporary table, and do the UPSERT in SQL, but I don't work with large data sets when I'm doing this, and so it hasn't yet been worth it to me to write such a procedure. Instead, I prefer to stick with the code that is a little easier for me to reason with on my limited SQL skills.
Here, we're just going to push the first 5 rows of mtcars
#* Insert the first 5 rows into the SQL Table
for (i in 1:5)
{
sqlQuery(channel = channel,
query = paste0("EXECUTE dbo.sample_procedure ",
"#OID = ", sqlNullString(mtcars$OID[i]), ", ",
"#mpg = ", mtcars$mpg[i], ", ",
"#cyl = ", mtcars$cyl[i], ", ",
"#disp = ", mtcars$disp[i], ", ",
"#hp = ", mtcars$hp[i]))
}
And now we'll take a look at the table from SQL
sqlQuery(channel = channel,
query = "EXECUTE dbo.get_mtcars")
This next line is just to match up the OIDs in R and SQL for illustration purposes. Normally, I would do this manually.
mtcars$OID[1:5] <- 1:5
This next for loop will UPSERT all 32 rows. We already have 5, we're UPSERTing 32, and the SQL table at the end should have 32 if we've done it correctly. (That is, SQL will recognize the 5 rows that already exist)
#* Update/Insert (UPSERT) the entire table
for (i in 1:nrow(mtcars))
{
sqlQuery(channel = channel,
query = paste0("EXECUTE dbo.sample_procedure ",
"#OID = ", sqlNullString(mtcars$OID[i]), ", ",
"#mpg = ", mtcars$mpg[i], ", ",
"#cyl = ", mtcars$cyl[i], ", ",
"#disp = ", mtcars$disp[i], ", ",
"#hp = ", mtcars$hp[i]))
}
#* Notice that the first 5 rows were unchanged (though they would have changed
#* if we had changed the data...the point being that the stored procedure
#* correctly identified that these records already existed)
sqlQuery(channel = channel,
query = "EXECUTE dbo.get_mtcars")
Recap
The stored procedure approach has a major disadvantage in that it is blatantly reinventing the wheel. It also requires that you learn SQL. SQL is pretty easy to learn for simple tasks, but some of the code I've written for more complex tasks is pretty difficult to interpret. Some of my procedures have taken me the better part of a day to get right. (once they are done, however, they work incredibly well)
The other big disadvantage to the stored procedure is, I've noticed, it does require a little bit more code work and organization. I'd say it's probably been about 10% more code work and documentation than if I were just using SQL Injection.
The chief advantages of the stored procedures approach are
you have massive flexibility for what you want to do
You can store your SQL code into the database and not pollute your R code with potentially huge strings of SQL code
Avoiding SQL injection (again, this is a data security thing, and may not be an issue depending on your employer's policies. I'm strictly forbidden from using SQL injection, so stored procedures are my only option)
It should also be noted that I've not yet explored using Table-Valued parameters in my stored procedures, which might simplify things for me a bit.
In the past, I've gotten around this by running the supremely inefficient sqlQuery with insert into row-by-row to get around this. But I tried this time and no data was written. Although the sqlQuery statement did not have an error or warning message.
Faced it yesterday: in my case the issue was in scheme. The table was actually created but in my user own scheme.
First time you can create it and than you have this error (that object already exists)
After the investigation I found that some packages does not work correctly with schemes.
In the end I used "insert by line" solution. The solution is available here and here

SQl Server deadlock with simple update statement

Using SQL Server 2008 R2 I am getting deadlocks when the same update statement (with different parameters) is running concurrently. Here is the deadlock graph (sorry cannot post images on here yet):
http://i.stack.imgur.com/E6JBK.png
And here is the actual execution plan:
http://i.stack.imgur.com/emm9i.png
The update is like this:
exec sp_executesql N'UPDATE mapping.IssuerAlternateName
SET
UseCount = UseCount + 1,
MostRecentlyAppeared = GETDATE(),
MostRecentlyAppearedUnderlyingAssetName = #p1
WHERE ID = #p0
',N'#p0 int,#p1 nvarchar(4000)',#p0=1234,#p1=N'blah blah blah'
If I have understood things correctly we are trying to read and write from the same index (PK_IssuerAlternateName_1).
Is there any way to resolve this? I was wondering if adding an additional index to the primary key and using WITH INDEX might fix it by stopping the read of PK_IssuerAlternateName_1 (sorry the full name is truncated in the execution plan screenshot).
Or is the best option just to live with this and retry the transaction, which is how the error is currently handled in .NET client. It is certainly successful on retry, but it would be good to avoid the deadlock if possible.
Thanks
In situations similar to this, I have used the UPDLOCK hint to let the database know I intend to update this row. It is not implied by the UPDATE statement. Without the lock hint, it will first obtain a "shared" lock, and then try to escalate. However, this causes deadlocks in certain scenarios.
You will need to do this within your own TransactionScope to ensure everything works correctly.
var sql = #"UPDATE mapping.IssuerAlternateName with (UPDLOCK)
SET
UseCount = UseCount + 1,
MostRecentlyAppeared = GETDATE(),
MostRecentlyAppearedUnderlyingAssetName = #p1
WHERE ID = #p0";
var options = new TransactionOptions()
{
IsolationLevel = IsolationLevel.ReadCommitted // don't use Serializable!
};
using (var scope = new TransactionScope(TransactionScopeOption.RequiresNew, options))
{
using (var context = new YourDbContext())
{
// execute your command here
}
}

FreeTDS / SQL Server UPDATE Query Hangs Indefinitely

I'm trying to run the following UPDATE query from a python script (note I've removed the database info):
print 'Connecting to db for update query...'
db = pyodbc.connect('DRIVER={FreeTDS};SERVER=<removed>;DATABASE=<removed>;UID=<removed>;PWD=<removed>')
cursor = db.cursor()
print ' Executing SQL queries...'
for i in range(len(data)):
sql = '''
UPDATE product.sanction
SET action_summary = '{action_summary}'
WHERE sanction_id = {sanction_id};
'''.format(sanction_id=data[i][0], action_summary=data[i][1])
cursor.execute(sql)
cursor.close()
db.commit()
db.close()
However, it hangs indefinitely, no error.
I'm new to pyodbc, but it should be setup correctly considering I'm having no problems performing SELECT queries. I did have to call CAST for SELECT queries (I've cast sanction_id AS INT [int identity on the database] and action_summary AS TEXT [nvarchar on the database]) to properly populate data, so perhaps the problem lies somewhere there, but I don't know where to start debugging. Converting the text to NVARCHAR didn't do anything either.
Here's an example of one of the rows in data:
(2861357, 'Exclusion Program: NonProcurement; Excluding Agency: HHS; CT Code: Z; Exclusion Type: Prohibition/Restriction; SAM Number: S4MR3Q9FL;')
I was unable to find my issue, but I ended up using QuerySets rather than running an UPDATE query.

Resources