I am trying to export a Pandas dataframe to SQL Server using the following code:
import pyodbc
import sqlalchemy
from sqlalchemy import engine
DB={'servername':'NAME', 'database':'dbname','driver':'driver={SQL Native Client 11.0}'
}
engine=create_engine('mssql+pyodbc://'+'DB['servername']+'/'+['database']+'/'+['driver'])
df.to_sql(con=engine,'test')
The error that it shows is
Datasource name not specified and no default driver specified
The df has 1 column and 90k-100k rows
How can I overcome this issue in order to export the dataframe (df) into SQL Server?
Pandas uses SQLAlchemy to connect to databases, which in turn can use PyODBC. Microsoft recommends using PyODBC to connect to SQL Server.
The SQLAlchemy docs for SQL Server show that the connection string should have the form:
"mssql+pyodbc://user:password#myhost:port/databasename?driver=ODBC+Driver+17+for+SQL+Server"
ie
engine = create_engine("mssql+pyodbc://user:password#myhost:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
You can also use an ODBC connection string:
from sqlalchemy.engine import URL
connection_string = "DRIVER={SQL Server Native Client 10.0};SERVER=dagger;DATABASE=test;UID=user;PWD=password"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = create_engine(connection_url)
You'll have to install PyODBC and the correct SQL Server ODBC driver for your operating system. The SQL Server docs on Python link to the appropriate drivers for Windows, various Linux distros and MacOS
If you have trouble saving to SQL Server, try the SQL Server PyODBC Getting Started tutorial to ensure you can connect to the database
Related
I have configured a new SQL Server 2017.XXX instance in my Windows 10. Then, created a new database called CUSTOMER_DEVICES. I tried to connect to database using pyodbc. I could not and got below error:
Error - pyodbc.OperationalError: ('08001', u'[08001] [Microsoft][ODBC SQL Server Driver]
When I use the same code in existing production server it works. Code below
import pyodbc
conn = pyodbc.connect(r'Driver={SQL Server};Server=localhost;Database=CUSTOMER_DEVICES;Trusted_Connection=yes;')
cursor = conn.cursor()
print(cursor)
Check if SQL server Configuration Management is configured. Enable TCP, IP. Check below link Check here
I am trying to use python's sqlalchemy library for connecting to microsoft azure data warehouse. and receiving the following error:
[ODBC Driver 17 for SQL Server][SQL Server]111214;An attempt to complete a transaction has failed. No corresponding transaction found. (111214)
The current code that I have tried:
import sqlalchemy
import urllib
params = urllib.quote_plus("Driver={ODBC Driver 17 for SQL Server};Server=<server-host>.database.windows.net,1433;Database=<database>;Uid=<user>#<server-host>;Pwd=<password>;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;")
engine = sqlalchemy.engine.create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
engine.connect()
I can't seem to understand where the error is...is it db side or is my ODBC connection not right? I pulled it out of the Azure portal so I doubt that could be the case.
Ended up using this for the engine instead. Had to add connect_args.
engine = create_engine('mssql+pyodbc:///?odbc_connect=%s' % params, echo=True, connect_args={'autocommit': True})
How can I connect with SQL server using Kerberos with SQLAlchemy URI within Superset. I am not able to find how to create kerberoised SQLAlchemy URI, which I will feed to Superset to connect with SQL Server. No Luck so far. Anyone have any idea?
I am trying to export data from PostgreSQL to MSSQL Server.
Currently, I am exploring the SQL Server Import and Export Wizard.
I have installed the native postgres driver and got the .Net Data Provider for PostgreSQL. Everything seemed fine until I tried to use it.
I got the following: Cannot get the supported data types from the database connection
Is this a driver problem or am I missing something?
Update:
Like #Panagiotis Kanavos suggested I have installed OLEDB driver
There seems to be a problem with it.
I fill the required data, Test the connection, it says that it is successfully established, but when I click Next to continue with the Wizard I get:
===================================
Cannot get string literals from the database connection "Provider=PostgreSQL OLE DB Provider;User ID=postgres;Data Source=localhost;Location=MyDb;Extended Properties=". (SQL Server Import and Export Wizard)
===================================
Object reference not set to an instance of an object. (DTSWizard)
Here are some screenshots:
After clicking Next:
There isn't any useful information about this problem. If anyone has any idea?
I was successful using the ODBC drivers (64 bit). 32bit drivers had some memory issues on large tables. See this post:
Migrating from PostgreSQL to SQL Server 2008
Problem
I am trying to use python to read my csv file and put it into Microsoft SQL Server 2016 as a new table. Simply put, I don't want to create a table on SQL and import the csv, I want to write a script in python that can read the csv and create a new table in SQL for me.
UPDATE
I may have to rethink my approach. I corrected the driver, but I am getting the following error from to_sql. I am thinking that there is something wrong with my authentication scheme. Sadly, the to_sql documentation and sql_alchemy is not shedding much light. Starting to consider alternatives.
sqlalchemy.exc.DBAPIError was unhandled by user code
Message: (pyodbc.Error) ('08001', '[08001] [Microsoft][SQL Server Native Client 11.0]Named Pipes Provider: Could not open a connection to SQL Server [53]. (53) (SQLDriverConnect)')
Code
import pandas as pd
import sqlalchemy
#Read the file
data = pd.read_csv(file.csv)
#Connect to database and write the table
server = "DERPSERVER"
database = "HERPDB"
username = "DBUser"
password = "password"
tablename = "HerpDerpTable"
driver = "SQL+Server+Native+Client+11.0"
#Connect to SQL using SQL Server Driver
print("Connect to SQL Server")
cnxn = sqlalchemy.create_engine("mssql+pyodbc://"+username+":"+password+"#"+server +"/"+database + "?driver="+driver)
UPDATE
I rewrote the string as follows, but it doesn't work:
sqlalchemy.create_engine('mssql+pymssql://'+username+':'+ password + '#' + server + '/' + database + '/?charset=utf8')
data.to_sql(tablename, cnxn);
Attempts
These are some important things to note in my approach. Pay special attention to the second bullet point I share below. I think my connection string for create_engine is somehow or maybe wrong, but don't know what is wrong because I followed the documentation.
I believe I am in a DSN-less situation. Thus, was attempting to connect by other means as described by the documentation.
I was using this link to help me create the connection string part in create_engine.
I tried to_sql to write the to the database, but think my connection string might still be messed up? I consulted this question on stackoverflow.
Update I added the driver specification as MaxU and the documentation for sqlalchemy specified. However, I am getting an error saying my data source name was not found and no default driver is specified with to_sql. Do I need to feed to_sql the driver as well? If so, where is the documentation or a sample code that shows me where I am going wrong?
I am making good effort to pick up python and to use it as a scripting language because of future goals and needs. I would appreciate any assistance, help, mentorship rendered.
You should explicitly specify the MS SQL Server driver if you use SQLAlchemy version 1.0.0+:
Example:
engine = create_engine("mssql+pyodbc://scott:tiger#myhost:port/databasename?driver=SQL+Server+Native+Client+10.0")
Changed in version 1.0.0: Hostname-based PyODBC connections now require the SQL Server driver name specified explicitly. SQLAlchemy
cannot choose an optimal default here as it varies based on platform
and installed drivers.
List of Native SQL Server Client Names
SQL Server 2005:
SQL Server Native Client
SQL Server 2008:
SQL Server Native Client 10.0
SQL Server 2016:
SQL Server Native Client 11.0
Alternatively you can use pymssql:
engine = create_engine('mssql+pymssql://<username>:<password>#<freetds_name>/?charset=utf8')