I want to create a connection to a SQL Server Database using a python script that you will see below. In fact, I managed to connect to a single database but I want to make a loop on a database list so that every time I point to a database I can connect directly.
My question is how can I make the name of the database as a variable. Could you help me please?! Thank you in advance
from os import listdir
from os.path import isfile, join
import pyodbc
import pandas as pd
mypath = 'C:\\Users\\DataBaseList'
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
for i in range (len(onlyfiles)) :
print (onlyfiles[i])
connection = pyodbc.connect("Driver={ODBC Driver 11 for SQL Server};"
"Server=FR0010APP31;"
"Database= Base;"
"uid=*****;pwd=*****")
cursor = connection.cursor()
sql='Select * from Activity'
df= pd.read_sql(sql, connection)
df.to_csv('DatabaseFile')
Related
I have a SQL Server on which I have databases that I want to use pandas to alter that data. I know how to get the data using pyodbc into a DataFrame, but then I have no clue how to get that DataFrame back into my SQL Server.
I have tried to create an engine with sqlalchemy and use the to_sql command, but I can not get that to work because my engine is never able to connect correctly to my database.
import pyodbc
import pandas
server = "server"
db = "db"
conn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+db+';Trusted_Connection=yes')
cursor = conn.cursor()
df = cursor.fetchall()
data = pandas.DataFrame(df)
conn.commit()
You can use pandas.DataFrame.to_sql to insert your dataframe into SQL server. Databases supported by SQLAlchemy are supported by this method.
Here is a example how you can achieve this:
from sqlalchemy import create_engine, event
from urllib.parse import quote_plus
import logging
import sys
import numpy as np
from datetime import datetime, timedelta
# setup logging
logging.basicConfig(stream=sys.stdout,
filemode='a',
format='%(asctime)s.%(msecs)3d %(levelname)s:%(name)s: %(message)s',
datefmt='%m-%d-%Y %H:%M:%S',
level=logging.DEBUG)
logger = logging.getLogger(__name__) # get the name of the module
def write_to_db(df, database_name, table_name):
"""
Creates a sqlalchemy engine and write the dataframe to database
"""
# replacing infinity by nan
df = df.replace([np.inf, -np.inf], np.nan)
user_name = 'USERNAME'
pwd = 'PASSWORD'
db_addr = '10.00.000.10'
chunk_size = 40
conn = "DRIVER={SQL Server};SERVER="+db_addr+";DATABASE="+database_name+";UID="+user_name+";PWD="+pwd+""
quoted = quote_plus(conn)
new_con = 'mssql+pyodbc:///?odbc_connect={}'.format(quoted)
# create sqlalchemy engine
engine = create_engine(new_con)
# Write to DB
logger.info("Writing to database ...")
st = datetime.now() # start time
# WARNING!! -- overwrites the table using if_exists='replace'
df.to_sql(table_name, engine, if_exists='replace', index=False, chunksize=chunk_size)
logger.info("Database updated...")
logger.info("Data written to '{}' databsae into '{}' table ...".format(database_name, table_name))
logger.info("Time taken to write to DB: {}".format((datetime.now()-st).total_seconds()))
Calling this method should write your dataframe to the database, note that it will replace the table if there is already a table in the database with the same name.
Hey so I have the below code I'm running where I'm pulling a table from a MS Sql Server table, running a bit of code and then trying to reimport it into another table within the same database. Running this in spyder
It runs all the way through, but when I
select * from pythontest
on the SQL Server, the table comes out blank. Is there anything that is standing out as not working?
## From SQL to DataFrame Pandas
import pandas as pd
import pyodbc
import mysql.connector
from sqlalchemy import create_engine
sql_conn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=njsrvnav1;"
"Database=cornerstone;"
"Trusted_Connection=yes;")
query = "SELECT [c1], [c2], [c3] from projectmaster"
df = pd.read_sql(query, sql_conn)
df = df[:100]
con = create_engine('mssql+pyodbc://username:pword#serverName:1433/cornerstone?driver=SQL+Server+Native+Client+11.0')
df.to_sql('dbo.pythontest', con, if_exists='replace')
con.dispose()
Try with pymysql :
conn = pymysql.connect(
host='',
port=
user='',
passwd='',
db='',
charset='utf8mb4')
df = pd.read_sql_query("SELECT * FROM table ",
conn)
df.head(2)
I am trying to connect to local MS SQL Express Edition. I am using canopy for Python editing.
Code:
import pymssql
conn = pymssql.connect(server='******\SQLEXPRESS',user = 'MEA\*****',password='*****',database='BSEG')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Table')
print(cursor.fetchone())
conn.close()
Error::
pymssql.pyx in pymssql.connect (pymssql.c:10734)()
_mssql.pyx in _mssql.connect (_mssql.c:21821)()
_mssql.pyx in _mssql.MSSQLConnection.init (_mssql.c:5917)()
ValueError: too many values to unpack (expected 2)
user = 'MEA\*****',password='*****'
MEA\***** seems to be Windows login, in this case you shouldn't pass in any password, your user name is enough, but you also should use Integrated security or Trusted parameter in your connection string
It should be smth like this:
server='******\SQLEXPRESS',Trusted_Connection=yes,database='BSEG'
Can we access SQL Server data from Iron Python script file if so how I m trying this following code ?
import sys
import clr
clr.AddReference("System.Windows")
from System.Windows import Application
import pypyodbc
constr="DRIVER={SQL Server Native Client 10.0};Data Source=SERVERName;Initial Catalog=DBName;Integrated Security=True"
cnxn=pypyodbc.connect(constr)
da=cnxn.cursor("Select Ename from tbl_Empployee where Emp_id=1",cn)
da.execute();
for row in da.fetchall():
for field in row:
Application.Current.RootVisual.FindName("textBox2").Text = field
You can use the .net native types, see: What's the simplest way to access mssql with python or ironpython?
your code could look like this:
import clr
clr.AddReference('System.Data')
from System.Data.SqlClient import SqlConnection, SqlParameter
conn_string = 'data source=SRG3SERVER; initial catalog=vijaykumarDB; trusted_connection=True'
connection = SqlConnection(conn_string)
connection.Open()
command = connection.CreateCommand()
command.CommandText = 'Select Ename from tbl_Empployee where Emp_id=#employeeId'
command.Parameters.Add(SqlParameter('#employeeId', 1))
reader = command.ExecuteReader()
while reader.Read():
print reader['Ename']
connection.Close()
This is probably a simple question:
How do I connect to Microsoft SQL Server 2008 R2 from Matlab?
How do I read a table into a matrix given some SQL query?
Update
I'd prefer a method that doesn't require use of manual setup using ODBC.
I present below a review of the different approaches for accessing databases in MATLAB. Here is a list of Stack Overflow questions where some of them were discussed:
How can I access a postgresql database from matlab with without matlabs database toolbox?
connection of MATLAB 7.0 and MYSQL
communicate MATLAB SQL Server
Getting names of Access database tables with Matlab
Invoking ADO.NET from MATLAB
Java
MATLAB have an embedded Java JVM, allowing you to directly call the JDBC drivers from MATLAB. You first need to make them available on the Java classpth in MATLAB:
javaclasspath('sqljdbc4.jar');
%# load driver and create connection
driver = com.microsoft.sqlserver.jdbc.SQLServerDriver;
conn = driver.connect('jdbc:sqlserver://<HOST>:<PORT>;databaseName=<DB>');
%# query database
q = conn.prepareStatement('select * from <TABLE>');
rs = q.executeQuery();
while rs.next()
char(rs.getString(0))
end
rs.close();
conn.close();
Database Toolbox
If you have access to the Database Toolbox, it can simplify the above as it acts as a wrapper around JDBC/ODBC stuff:
conn = database('<DB>', '<USER>','<PASS>', ...
'com.microsoft.sqlserver.jdbc.SQLServerDriver', ...
'jdbc:sqlserver://<HOST>:<PORT>;database=<DB>');
curs = exec(conn, 'select * from <TABLE>');
curs = fetch(curs);
curs.Data
close(curs)
close(conn)
You can also access the database through ODBC. First create a DSN to MSSQL Server (Control Panel > ODBC Data Sources), then use it from the Database Toolbox:
conn = database('myDB', '', ''); %# User/System DSN
%...
close(conn)
COM
You can directly use the ADO OLEDB component from MATLAB. One way is to specify a connection string (DNS-less):
conn = actxserver('ADODB.Connection');
conn.Open('Provider=sqloledb;Data Source=<HOST>;Initial Catalog=<DB>;User Id=<USER>;Password=<PASS>;');
conn.Execute('select * from <TABLE>').GetRows
conn.Close()
.NET
Finally, recent versions of MATLAB added the ability to call .NET from MATLAB. So you can use the ADO.NET data providers:
import System.Data.SqlClient.*
NET.addAssembly('System.Data');
conn = SqlConnection('Data Source=<HOST>;Initial Catalog=<DB>');
conn.Open();
q = SqlCommand('select * from <TABLE>', conn);
r = q.ExecuteReader();
while r.Read()
char(r.GetString(0))
end
r.Close()
conn.Close()