I am trying to use python's sqlalchemy library for connecting to microsoft azure data warehouse. and receiving the following error:
[ODBC Driver 17 for SQL Server][SQL Server]111214;An attempt to complete a transaction has failed. No corresponding transaction found. (111214)
The current code that I have tried:
import sqlalchemy
import urllib
params = urllib.quote_plus("Driver={ODBC Driver 17 for SQL Server};Server=<server-host>.database.windows.net,1433;Database=<database>;Uid=<user>#<server-host>;Pwd=<password>;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;")
engine = sqlalchemy.engine.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
engine.connect()
I can't seem to understand where the error is...is it db side or is my ODBC connection not right? I pulled it out of the Azure portal so I doubt that could be the case.
Ended up using this for the engine instead. Had to add connect_args.
engine = create_engine('mssql+pyodbc:///?odbc_connect=%s' % params, echo=True, connect_args={'autocommit': True})
Related
I am trying to export a Pandas dataframe to SQL Server using the following code:
import pyodbc
import sqlalchemy
from sqlalchemy import engine
DB={'servername':'NAME', 'database':'dbname','driver':'driver={SQL Native Client 11.0}'
}
engine=create_engine('mssql+pyodbc://'+'DB['servername']+'/'+['database']+'/'+['driver'])
df.to_sql(con=engine,'test')
The error that it shows is
Datasource name not specified and no default driver specified
The df has 1 column and 90k-100k rows
How can I overcome this issue in order to export the dataframe (df) into SQL Server?
Pandas uses SQLAlchemy to connect to databases, which in turn can use PyODBC. Microsoft recommends using PyODBC to connect to SQL Server.
The SQLAlchemy docs for SQL Server show that the connection string should have the form:
"mssql+pyodbc://user:password#myhost:port/databasename?driver=ODBC+Driver+17+for+SQL+Server"
ie
engine = create_engine("mssql+pyodbc://user:password#myhost:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
You can also use an ODBC connection string:
from sqlalchemy.engine import URL
connection_string = "DRIVER={SQL Server Native Client 10.0};SERVER=dagger;DATABASE=test;UID=user;PWD=password"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = create_engine(connection_url)
You'll have to install PyODBC and the correct SQL Server ODBC driver for your operating system. The SQL Server docs on Python link to the appropriate drivers for Windows, various Linux distros and MacOS
If you have trouble saving to SQL Server, try the SQL Server PyODBC Getting Started tutorial to ensure you can connect to the database
I am having trouble with a MS SQL connection when using pyinstaller. When run in interactive mode, everything works as expected. After compiling to an exe, the MS SQL database connection times out on the first query with the following error:
(pyodbc.OperationalError) ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]Named Pipes Provider: Could not open a connection to SQL Server [53]. (53) (SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expire (0); ...
My connection string is similar to the following:
create_engine(
"mssql+pyodbc://USER:PASSWORD#SERVERIP/DB_NAME?driver=ODBC+Driver+17+for+SQL+Server"
)
In attempting to diagnose the issue, I am printing out the drivers available to pyodbc with pyodbc.drivers() (which shows a large disparity between available drivers in compiled vs interactive) as well as the driver in use using
print(session.bind.dialect.name)
> pyodbc
print(session.bind.dialect.driver)
> mssql
It returns the upper level python modules which are being used but not the .dll that is handling it at a lower level. Is there any way to find which exact driver is being used? Any tips on what could be causing the error in the compiled version in the firstplace would be appreciated as well.
The issue may be in your connection string.
To create a proper connection string to connect to MSSQL Server ODBC driver with sqlAlchemy use the following:
import urllib
from sqlalchemy import create_engine
server = 'serverName\instanceName,port' # to specify an alternate port
database = 'mydb'
username = 'myusername'
password = 'mypassword'
params = urllib.parse.quote_plus('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+password)
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
Also, you can check the following article Connecting to Microsoft SQL Server using SQLAlchemy and PyODBC
Is there any way to find which exact driver [.dll] is being used?
import pyodbc
cnxn = engine.raw_connection()
print(cnxn.getinfo(pyodbc.SQL_DRIVER_NAME)) # msodbcsql17.dll
print(cnxn.getinfo(pyodbc.SQL_DRIVER_VER)) # 17.08.0001
I am trying to connect SQL Server using pymssql package with active directory authentication. Below is the code.
db = pymssql.connect(server=url, user=r'Domain\user', password=password, database=database)
cursor = db.cursor()
But it fails with below error:
Error (18452, b'Login failed. The login is from an untrusted domain and cannot be used with Windows authentication.DB-Lib error message 20018, severity 14:\nGeneral SQL Server error: Check messages from the SQL Server\nDB-Lib error message 20002, severity 9:\nAdaptive Server connection failed ()\n')
Python version: Python 3.8.6
PYMSSQL version: pymssql 2.1.5
For most client libraries you can't specify the username/password with Windows Integrated Auth. It just connects as the identity of the current process.
To do otherwise would require the driver to implement (or us a library that implements) the NTLM or Kerberos authentication protocols, and AFAIK only the Microsoft JDBC Driver for SQL Server does that.
Not sure that will work. Try it like this.
# connect to SQL Server
import pyodbc
from sqlalchemy import create_engine
driver= '{SQL Server Native Client 11.0}'
conn_str = (
r'DRIVER={SQL Server};'
r'SERVER=your_server_name;'
r'DATABASE=your_db_name;'
r'Trusted_Connection=yes;'
)
cnxn = pyodbc.connect(conn_str)
cursor = cnxn.cursor()
cursor.execute('SELECT * FROM Employee')
# print contents of table
for row in cursor:
print('row = %r' % (row,))
I am using pymssql=2.1.1 to connect Azure database from python. Due to idle connection for few minutes, i am getting error (Write to the server failed) and not able to fetch the data.
I am using connect method of pymssql to create connection to Azure DB.
conn = pymssql.connect(server=v_host, user=v_user, password=v_passwd, database=v_db)
cursor = self.conn.cursor(as_dict=True)
cursor.execute(query)
The error looks like
(20006, b'DB-Lib error message 20006, severity 9:\nWrite to the server failed\nNet-Lib error during Connection reset by peer (104)\n')
If you want to us pymssql connect to Azure SQL database, make sure the following requirements:
Examples:
import pymssql
conn=pymssql.connect("xxx.database.windows.net", "username#xxx", "password", "db_name")
cursor = conn.cursor()
cursor.execute(query)
For more details, please see: Connecting to Azure SQL Database. Starting with version 2.1.1 pymssql can be used to connect to Microsoft Azure SQL Database. And you can troubleshoot the error by pymssql Frequently asked questions.
Another way, you also can try the pyodbc example:
import pyodbc
server = '<server>.database.windows.net'
database = '<database>'
username = '<username>'
password = '<password>'
driver= '{ODBC Driver 17 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';SERVER='+server+';PORT=1433;DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute(query)
Here is the Azure document: Quickstart: Use Python to query an Azure SQL database.
Hope this helps.
Problem
I am trying to use python to read my csv file and put it into Microsoft SQL Server 2016 as a new table. Simply put, I don't want to create a table on SQL and import the csv, I want to write a script in python that can read the csv and create a new table in SQL for me.
UPDATE
I may have to rethink my approach. I corrected the driver, but I am getting the following error from to_sql. I am thinking that there is something wrong with my authentication scheme. Sadly, the to_sql documentation and sql_alchemy is not shedding much light. Starting to consider alternatives.
sqlalchemy.exc.DBAPIError was unhandled by user code
Message: (pyodbc.Error) ('08001', '[08001] [Microsoft][SQL Server Native Client 11.0]Named Pipes Provider: Could not open a connection to SQL Server [53]. (53) (SQLDriverConnect)')
Code
import pandas as pd
import sqlalchemy
#Read the file
data = pd.read_csv(file.csv)
#Connect to database and write the table
server = "DERPSERVER"
database = "HERPDB"
username = "DBUser"
password = "password"
tablename = "HerpDerpTable"
driver = "SQL+Server+Native+Client+11.0"
#Connect to SQL using SQL Server Driver
print("Connect to SQL Server")
cnxn = sqlalchemy.create_engine("mssql+pyodbc://"+username+":"+password+"#"+server +"/"+database + "?driver="+driver)
UPDATE
I rewrote the string as follows, but it doesn't work:
sqlalchemy.create_engine('mssql+pymssql://'+username+':'+ password + '#' + server + '/' + database + '/?charset=utf8')
data.to_sql(tablename, cnxn);
Attempts
These are some important things to note in my approach. Pay special attention to the second bullet point I share below. I think my connection string for create_engine is somehow or maybe wrong, but don't know what is wrong because I followed the documentation.
I believe I am in a DSN-less situation. Thus, was attempting to connect by other means as described by the documentation.
I was using this link to help me create the connection string part in create_engine.
I tried to_sql to write the to the database, but think my connection string might still be messed up? I consulted this question on stackoverflow.
Update I added the driver specification as MaxU and the documentation for sqlalchemy specified. However, I am getting an error saying my data source name was not found and no default driver is specified with to_sql. Do I need to feed to_sql the driver as well? If so, where is the documentation or a sample code that shows me where I am going wrong?
I am making good effort to pick up python and to use it as a scripting language because of future goals and needs. I would appreciate any assistance, help, mentorship rendered.
You should explicitly specify the MS SQL Server driver if you use SQLAlchemy version 1.0.0+:
Example:
engine = create_engine("mssql+pyodbc://scott:tiger#myhost:port/databasename?driver=SQL+Server+Native+Client+10.0")
Changed in version 1.0.0: Hostname-based PyODBC connections now require the SQL Server driver name specified explicitly. SQLAlchemy
cannot choose an optimal default here as it varies based on platform
and installed drivers.
List of Native SQL Server Client Names
SQL Server 2005:
SQL Server Native Client
SQL Server 2008:
SQL Server Native Client 10.0
SQL Server 2016:
SQL Server Native Client 11.0
Alternatively you can use pymssql:
engine = create_engine('mssql+pymssql://<username>:<password>#<freetds_name>/?charset=utf8')