Creating flask-sqlalchemy table object of a table on a linked server - sql-server

I am creating a small flask website with a Microsoft SQL database server.
The web server is on a linux machine and I connecting to the database via odbc -> pyodbc & flask-sqlalchemy.
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy.engine import URL
connection_url = URL.create(
"mssql+pyodbc",
username="user",
password="password",
host="192.168.0.128",
port=1433,
database="Print",
query={"driver": "ODBC Driver 17 for SQL Server"})
app = Flask(__name__)
app.config['SECRET_KEY'] = 'hard to guess string'
app.config['SQLALCHEMY_DATABASE_URI'] = connection_url
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
The exsisting local database tables are no problem to connect to.
class Print(db.Model):
__tablename__ = 'Printers'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(24))
def __repr__(self):
return '<Name %r>' % self.name
But when I try to access a table from a linked server the query gets mangled somewhere in the translation.
class Collo(db.Model):
__tablename__ = 'collo'
__table_args__ = {'schema': 'sprod..sdba'}
type = db.Column(db.String(8), primary_key=True)
description = db.Column(db.String(30))
def __repr__(self):
return '<collo %r>' % self.description
The error:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42S02', "[42S02] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid object name 'sprod..sdba.collo'. (208) (SQLExecDirectW)")
[SQL: SELECT TOP 1 [sprod.].sdba.collo.type AS sprod__sdba_collo_type, [sprod.].sdba.collo.description AS sprod__sdba_collo_description
FROM [sprod.].sdba.collo
WHERE [sprod.].sdba.collo.type = ?]
[parameters: ('GOOT2',)]
Under sql management studio the query is:
select * from [SPROD]..[sdba].[collo]
selecting in pyodbc:
cursor.execute("select * from sprod..sdba.collo where type = 'GOOT2'" )
works fine.
How can I prevent the inclusion of the square brackets in flask-sqlalchemy?

This is a known issue with SQLAlchemy and SQL Server "linked servers" as discussed in this GitHub issue. The current workaround is to create a SYNONYM on the local server for the table on the linked server as described here, e.g.,
CREATE SYNONYM [dbo].[hello_199] FOR [DOCKER199].[myDb].[dbo].[hello]

Related

"SQLSTATE[42S02]: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server] Invalid 'users' object name

I migrated my MySQL database to SQL Server 2019.
I made the required configurations:
php.ini
extension=php_pdo_sqlsrv_80_ts_x64
extension=php_sqlsrv_80_ts_x64
.env
DB_CONNECTION=sqlsrv
DB_HOST=servername
DB_PORT=null
DB_DATABASE=dbname
DB_USERNAME=sa
DB_PASSWORD=password
The code below is working fine. O test this in web.php and i get db connect successfully and all (2) tables.
try {
DB::connection()->getPdo();
echo DB::connection()->getDatabaseName()." db connect successfully";
} catch (Exception $e) {
dd($e->getMessage());
}
$sql = DB::select("SELECT TOP (2) id, email, email_verified_at, password, remember_token, image, userable_id, userable_type, created_at, updated_at, deleted_at FROM soraeir.users", [1]);
die(print_r($sql));
Message when I try to login I get:
SQLSTATE[42S02]:
[Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
Invalid 'users' object name. (SQL: select top 1 * from [users] where [email] = admin and [users].[deleted_at] is null)"
I think eloquent must add an instance_name to all queries...I don't know how to proceed...?
Perhaps, instead of
select top 1 *
from [users]
where [email] = admin
and [users].[deleted_at] is null
We could use
select top 1 *
from [instance_name].[users]
where [email] = admin
and [users].[deleted_at] is null
I solved my problem
my problem was when I was doing the migration I misconfigured at the level of schema mapping with sql server migration assistant for mysql.
all my tables in sql server db was : mysql_db_name.table_name
Solution
Source Schema: mysql_db_name
Target Schema : sqlsrv_db_name.dbo
dbo is required
now all tables are as follows : dbo.table_name

How to set the CONNECTION_OPTIONS = 'ApplicationIntent=ReadOnly' for elastic queries on Azure SQL?

I am accessing the other database using elastic queries. The data source was created like this:
CREATE EXTERNAL DATA SOURCE TheCompanyQueryDataSrc WITH (
TYPE = RDBMS,
--CONNECTION_OPTIONS = 'ApplicationIntent=ReadOnly',
CREDENTIAL = ElasticDBQueryCred,
LOCATION = 'thecompanysql.database.windows.net',
DATABASE_NAME = 'TheCompanyProd'
);
To reduce the database load, the read-only replica was created and should be used. As far as I understand it, I should add the CONNECTION_OPTIONS = 'ApplicationIntent=ReadOnly' (commented out in the above code). However, I get only the Incorrect syntax near 'CONNECTION_OPTIONS'
Both databases (the one that sets the connection + external tables, and the other to-be-read-only are at the same server (thecompanysql.database.windows.net). Both are set the compatibility lever SQL Server 2019 (150).
What else should I set to make it work?
The CREATE EXTERNAL DATA SOURCE Syntax doesn't support the option CONNECTION_OPTIONS = 'ApplicationIntent=ReadOnly'. We can't use that in the statements.
If you want achieve that readonly request, the way is that please use the user account which only has the readonly(db_reader) permission to login the external database.
For example:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<password>' ;
CREATE DATABASE SCOPED CREDENTIAL SQL_Credential
WITH
IDENTITY = '<username>' -- readonly user account,
SECRET = '<password>' ;
CREATE EXTERNAL DATA SOURCE MyElasticDBQueryDataSrc
WITH
( TYPE = RDBMS ,
LOCATION = '<server_name>.database.windows.net' ,
DATABASE_NAME = 'Customers' ,
CREDENTIAL = SQL_Credential
) ;
Since the option is not supported, then we can't use it with elastic query. The only way to connect to the Azure SQL data with SSMS is like this:
HTH.

Pandas dataframe insert into SQL Server taking too long with execute and executemany

I have a pandas dataframe with 27 columns and ~45k rows that I need to insert into a SQL Server table.
I am currently using with the below code and it takes 90 mins to insert:
conn = pyodbc.connect('Driver={ODBC Driver 17 for SQL Server};\
Server=#servername;\
Database=dbtest;\
Trusted_Connection=yes;')
cursor = conn.cursor() #Create cursor
for index, row in t6.iterrows():
cursor.execute("insert into dbtest.dbo.test( col1, col2, col3, col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,,col27)\
values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
row['col1'],row['col2'], row['col3'],,row['col27'])
I have also tried to load using executemany and that takes even longer to complete, at nearly 120mins.
I am really looking for a faster load time since I need to run this daily.
You can set fast_executemany in pyodbc itself for versions>=4.0.19. It is off by default.
import pyodbc
server_name = 'localhost'
database_name = 'AdventureWorks2019'
table_name = 'MyTable'
driver = 'ODBC Driver 17 for SQL Server'
connection = pyodbc.connect(driver='{'+driver+'}', server=server_name, database=database_name, trusted_connection='yes')
cursor = connection.cursor()
cursor.fast_executemany = True # reduce number of calls to server on inserts
# form SQL statement
columns = ", ".join(df.columns)
values = '('+', '.join(['?']*len(df.columns))+')'
statement = "INSERT INTO "+table_name+" ("+columns+") VALUES "+values
# extract values from DataFrame into list of tuples
insert = [tuple(x) for x in df.values]
cursor.executemany(statement, insert)
Or if you prefer sqlalchemy and dataframes directly.
import sqlalchemy as db
engine = db.create_engine('mssql+pyodbc://#'+server_name+'/'+database_name+'?trusted_connection=yes&driver='+driver, fast_executemany=True)
df.to_sql(table_name, engine, if_exists='append', index=False)
See fast_executemany in this link.
https://github.com/mkleehammer/pyodbc/wiki/Features-beyond-the-DB-API
I have worked through this in the past, and this was the fastest that I could get it to work using sqlalchemy.
import sqlalchemy as sa
engine = (sa.create_engine(f'mssql://#{server}/{database}
?trusted_connection=yes&driver={driver_name}', fast_executemany=True)) #windows authentication
df.to_sql('Daily_Report', con=engine, if_exists='append', index=False)
If the engine is not working for you, then you may have a different setup so please see: https://docs.sqlalchemy.org/en/13/core/engines.html
You should be able to create the variables needed above, but here is how I get the driver:
driver_name = ''
driver_names = [x for x in pyodbc.drivers() if x.endswith(' for SQL Server')]
if driver_names:
driver_name = driver_names[-1] #You may need to change the [-1] if wrong driver to [-2] or a different option in the driver_names list.
if driver_name:
conn_str = f'''DRIVER={driver_name};SERVER='''
else:
print('(No suitable driver found. Cannot connect.)')
You can try to use the method 'multi' built in pandas to_sql.
df.to_sql('table_name', con=engine, if_exists='replace', index=False, method='multi')
The multi method allows you to 'Pass multiple values in a single INSERT clause.' per documentation.
I found it to be pretty efficient.

pandas to_sql for MS SQL

I'm trying to save a dataframe to MS SQL that uses Windows authentication. I've tried using engine, engine.connect(), engine.raw_connection() and they all throw up errors:
'Engine' object has no attribute 'cursor', 'Connection' object has no attribute 'cursor', and Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ... respectively.
params = urllib.parse.quote('DRIVER={ODBC Driver 13 for SQL Server};'
'SERVER=server;'
'DATABASE=db;'
'TRUSTED_CONNECTION=Yes;')
engine = create_engine('mssql+pyodbc:///?odbc_connect=%s' % params)
df.to_sql(table_name,engine, index=False)
This will do exactly what you want.
# Insert from dataframe to table in SQL Server
import time
import pandas as pd
import pyodbc
# create timer
start_time = time.time()
from sqlalchemy import create_engine
df = pd.read_csv("C:\\your_path\\CSV1.csv")
conn_str = (
r'DRIVER={SQL Server Native Client 11.0};'
r'SERVER=name_of_your_server;'
r'DATABASE=name_of_your_database;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str)
cursor = cnxn.cursor()
for index,row in df.iterrows():
cursor.execute('INSERT INTO dbo.Table_1([Name],[Address],[Age],[Work]) values (?,?,?,?)',
row['Name'],
row['Address'],
row['Age'],
row['Work'])
cnxn.commit()
cursor.close()
cnxn.close()
# see total time to do insert
print("%s seconds ---" % (time.time() - start_time))
Here is an update to my original answer. Basically, this is the old-school way of doing things (INSERT INTO). I recently stumbled upon a super-easy, scalable, and controllable, way of pushing data from Python to SQL Server. Try the sample code and post back if you have additional questions.
import pyodbc
import pandas as pd
engine = "mssql+pyodbc://your_server_name/your_database_name?driver=SQL Server Native Client 11.0?trusted_connection=yes"
... dataframe here...
dataframe.to_sql(x, engine, if_exists='append', index=True)
dataframe is pretty self explanatory.
x = the name yo uwant your table to be in SQL Server.

Django SQL Server ForeignKey that allows NULL

I've been struggling with this issue for some time. I'm switching one of my apps from Postgres to SQL Server database and I'm facing an issue with ForeignKey field. I'm running latest SQL Server version with Django 1.11 and using django-pyodbc-azure app.
class Owner(models.Model):
name = models.TextField()
dog = models.ForeignKey('Dog', related_name='+')
class Dog(models.Model):
name = models.TextField()
owner = models.ForeignKey('Owner', null=True, related_name='+')
When I try to insert a new record I get the following message:
dog = Dog.objects.create(name='Rex')
owner = Owner.objects.create(name='Mike', dog=dog)
dog.owner = owner
dog.save()
('23000', u"[23000] [Microsoft][ODBC Driver 13 for SQL Server][SQL
Server]Violation of UNIQUE KEY constraint
'UQ__owner__EF6DECB9214EF1D9'. Cannot insert duplicate key in object
'dbo.owner'. The duplicate key value is (NULL). (2627)
(SQLExecDirectW)")
The label owner is the problem, try:
dog.owner_id = owner.id

Resources