Azure SQL Databricks Integration - sql-server

These are my JDBC connection details:
jdbcHostname = "ss-owaisde.database.windows.net"
jdbcPort = 1433
jdbcDatabase = "database-owaisde"
jdbcUsername = "owaisde"
jdbcPassword = "******"
jdbcDriver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbcUrl = f"jdbc:sqlserver://{jdbcHostname}:{jdbcPort};databaseName=
{jdbcDatabase};user{jdbcUsername};password={jdbcPassword};driver={jdbcDriver}"
Executing the spark read
df1 = spark.read.format("jdbc").option("url",jdbcUrl).option("dbtable",
"SalesLT.Product").load()
Getting the following error on databricks
java.sql.SQLException: No suitable driver

I tried to replicate your issue with your code:
I got same error:
As per my knowledge in your case URL is not build in correct format. I tried with below code:
jdbcHostname = "<servername>.database.windows.net"
jdbcPort = 1433
jdbcDatabase = "<dbname>"
jdbcUsername = "<username>"
jdbcPassword = "<password>"
jdbcDriver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
#url = s"jdbc:sqlserver://${database_host}:${database_port}/${database_name}"
table = "Student"
jdbcUrl = f"jdbc:sqlserver://{jdbcHostname}:{jdbcPort};databaseName={jdbcDatabase}"
df1 = spark.read.format("jdbc").option("driver", jdbcDriver).option("url", jdbcUrl).option("dbtable", table).option("user", jdbcUsername).option("password", jdbcPassword).load()
Dataframe created successfully.
It worked fine for me, check from your end.

Related

postgresSQL terminating abnormally

I am trying to update postgresSQL table with psycopg2 (python package) sometimes it is failing with below error.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Here is the code
from psycopg2 import pool
now = datetime.now()
logoff_time = datetime(now.year, now.month, now.day, 15, 0, 0)
while True:
time.sleep(1)
try:
status = 'EXECUTED'
exec_type1 = 'CANCELLED'
exec_type2 = 'COMPLETED'
try:
postgreSQL_pool = pool.SimpleConnectionPool(1, 20, host = db_host,
database = db_name,
port = db_port,
user = db_user,
password = db_pwd)
if postgreSQL_pool:
print("Connection pool created successfully")
conn = postgreSQL_pool.getconn()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
sql = """ UPDATE orders SET status = %s, executed_type = %s WHERE order_id = %s"""
updated_rows = 0
try:
cur = conn.cursor()
cur.execute(sql, (status, exec_type1, order_id,))
conn.commit()
updated_rows = cur.rowcount
cur.close()
break
except (Exception, psycopg2.DatabaseError) as error:
print(error)
print(updated_rows)
except Exception as e:
print(e)
psycopg2 version: '2.8.6 (dt dec pq3 ext lo64)'
Postgres: PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
it is pretty much simple task but facing challenges. suggestions please
The server is crashing for some reason that you might be able to read in the server's logs.

Asyncio Run Loop Errors in Python 3.7

I am trying to use the asyncio packages to execute concurrent calls from one SQL Server to another in order to extract data. I'm hitting an issue of at the portion of myLoop.run_until_complete(cors) where it is telling me that the event loop is already running. I will admit that I am new to this package and may be overlooking something simple.
import pyodbc
import sqlalchemy
import pandas
import asyncio
import time
async def getEngine(startString):
sourceList = str.split(startString,'=')
server = str.split(sourceList[1],';')[0]
database = str.split(sourceList[2],';')[0]
user = str.split(sourceList[3],';')[0]
password = str.split(sourceList[4],';')[0]
returnEngine = sqlalchemy.create_engine("mssql+pyodbc://"+user+":"+password+"#"+server+"/"+database+"?driver=SQL+Server+Native+Client+11.0")
return returnEngine
async def getConnString(startString):
sourceList = str.split(startString,'=')
server = str.split(sourceList[1],';')[0]
database = str.split(sourceList[2],';')[0]
user = str.split(sourceList[3],';')[0]
password = str.split(sourceList[4],';')[0]
return "Driver={SQL Server Native Client 11.0};Server="+server+";Database="+database+";Uid="+user+";Pwd="+password+";"
async def executePackage(source,destination,query,sourceTable,destTable,lastmodifiedDate,basedOnStation):
sourceConnString = getConnString(source)
destEngine = getEngine(destination)
sourceConn = pyodbc.connect(sourceConnString)
newQuery = str.replace(query,'dateTest',str(lastmodifiedDate))
df = pandas.read_sql(newQuery,sourceConn)
print('Started '+sourceTable+'->'+destTable)
tic = time.perf_counter()
await df.to_sql(destTable,destEngine,index=False,if_exists="append")
toc = time.perf_counter()
secondsToFinish = toc - tic
print('Finished '+sourceTable+'->'+destTable+' in '+ str(secondsToFinish) +' seconds')
async def main():
connString = "Driver={SQL Server Native Client 11.0};Server=myServer;Trusted_Connection=yes;"
myConn = pyodbc.connect(connString)
cursor = myConn.cursor()
df = pandas.read_sql('exec mySql_stored_proc',myConn)
if len(df.index) > 0:
tasks = [executePackage(df.iloc[i,10],df.iloc[i,11],df.iloc[i,7],df.iloc[i,8],df.iloc[i,9],df.iloc[i,5],df.iloc[i,17])for i in range(len(df))]
myLoop = asyncio.get_event_loop()
cors = asyncio.wait(tasks)
myLoop.run_until_complete(cors)
if __name__ =="__main__":
asyncio.run(main())

SOAPUI database connection error[local port 127.0.0.1:4321 cannot be bound]

local port 127.0.0.1:4321 cannot be bound, please see code below for details, note that i am using the lport - 4321. Do i need to use something different? If yes, how can i find one?
package mypackage
import groovy.sql.Sql
import java.sql.*
import com.jcraft.jsch.JSch
import com.jcraft.jsch.Session
// ssh login
String sshHost = 'abc.com'
String sshUser = 'test'
String sshPass = 'test'
int sshPort = 22
String boundaddress ="0.0.0.0";
// database login
targetHost = 'localhost'
targetUser = 'test'
targetPass = 'test'
targetPort = 3306
lport = 4321
JSch jsch = new JSch();
Session session = jsch.getSession(sshUser, sshHost, sshPort);
session.setPassword(sshPass);
session.setConfig("StrictHostKeyChecking", "no");
System.out.println("Establishing Connection...");
session.connect();
int assinged_port=session.setPortForwardingL(lport, targetHost, targetPort);
Connection con = null;
String driver = "org.mariadb.jdbc.Driver";
String connectionString = "jdbc:mariadb://" + targetHost +":" + lport + "/";
con = DriverManager.getConnection(connectionString, targetUser, targetPass);
Statement st = con.createStatement();
String sql = "select * company "
st.execute(sql);

"Data source name not found" when using unixODBC connection

I am getting this error from views.py:
('IM002', '[IM002] [unixODBC][Driver Manager]Data source name not
found, and no default driver specified (0) (SQLDriverConnectW)')
Here is views.py to save the database data to a json file
connstr = 'DRIVER={SQL Server};SERVER=dev_appserver;DATABASE=DemoApp;'
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
cursor.execute("""
SELECT book_id, book_name, author_name, publisher_name, email, bookref
FROM Book
""")
rows = cursor.fetchall()
rowarray_list = []
for row in rows:
t = (row.book_id, row.book_name, row.author_name, row.publisher_name,
row.email, row.bookref)
rowarray_list.append(t)
j = json.dumps(rowarray_list)
rowarrays_file = 'student_rowarrays.json'
f = open(rowarrays_file,'w')
print >> f, j
objects_list = []
for row in rows:
d = collections.OrderedDict()
d['book_id'] = row.book_id
d['book_name'] = row.book_name
d['author_name'] = row.author_name
d['publisher_name'] = row.publisher_name
d['email'] = row.email
d['bookref'] = row.bookref
objects_list.append(d)
j = json.dumps(objects_list)
objects_file = 'student_objects.json'
f = open(objects_file,'w')
print >> f, j
conn.close()
This is coded to write the database data to a json file
Thanks
You don't appear to be passing a username or password when you connect to the database.
Try something like this:
connstr = 'DRIVER={SQL Server};SERVER=dev_appserver;DATABASE=DemoApp;UID=username;PWD=password'
conn = pyodbc.connect(connstr)

Sometimes isql doesn't connect and tsql does

I am stuck with a problem that happens sometimes around unixODBC and freeTDS. I have a CentOS webserver where I have settled configuration files as:
odbc.ini:
[XYZ]
Driver = FreeTDS
Server = X.X.X.X
Port = 1433
Database = mydatabase
TDS_Version = 8.0
odbcinst.ini
[PostgreSQL]
Description = ODBC for PostgreSQL
Driver = /usr/lib/libodbcpsql.so
Setup = /usr/lib/libodbcpsqlS.so
FileUsage = 1
[FreeTDS]
Description = v0.82 with protocol v8.0
Driver = /usr/local/lib/libtdsodbc.so
Setup = /usr/local/lib/libtdsodbc.so
UsageCount = 1
Trace = Yes
TraceFile = /tmp/freetds.log
ForceTrace = Yes
FileUsage = 1
[ODBC]
;Trace = Yes
;TraceFile = /tmp/freetds.log
;ForceTrace = Yes
;Pooling = No
freetds.conf
# $Id: freetds.conf,v 1.12 2007/12/25 06:02:36 jklowden Exp $
#
# This file is installed by FreeTDS if no file by the same
# name is found in the installation directory.
#
# For information about the layout of this file and its settings,
# see the freetds.conf manpage "man freetds.conf".
# Global settings are overridden by those in a database
# server specific section
[global]
# TDS protocol version
tds version = 8.0
# Whether to write a TDSDUMP file for diagnostic purposes
# (setting this to /tmp is insecure on a multi-user system)
; dump file = /tmp/freetds.log
; debug flags = 0xffff
# Command and connection timeouts
; timeout = 10
; connect timeout = 10
# If you get out-of-memory errors, it may mean that your client
# is trying to allocate a huge buffer for a TEXT field.
# Try setting 'text size' to a more reasonable limit
text size = 64512
# A typical Sybase server
[egServer50]
host = symachine.domain.com
port = 5000
tds version = 5.0
# A typical Microsoft server
[egServer70]
host = ntmachine.domain.com
port = 1433
tds version = 7.0
[XYZ]
host = X.X.X.X
port = 1433
tds version = 8.0
and doing isql -v XYZ username password when it won't connect it gives out this trace log:
[ODBC][22870][__handles.c][444]
Exit:[SQL_SUCCESS]
Environment = 0x938ab58
[ODBC][22870][SQLAllocHandle.c][345]
Entry:
Handle Type = 2
Input Handle = 0x938ab58
[ODBC][22870][SQLAllocHandle.c][463]
Exit:[SQL_SUCCESS]
Output Handle = 0x938b130
[ODBC][22870][SQLConnect.c][3549]
Entry:
Connection = 0x938b130
Server Name = [XYZ][length = 14 (SQL_NTS)]
User Name = [username][length = 11 (SQL_NTS)]
Authentication = [*************][length = 13 (SQL_NTS)]
UNICODE Using encoding ASCII 'ISO8859-1' and UNICODE 'UCS-2LE'
DIAG [42000] [FreeTDS][SQL Server]Login failed for user 'username'.
DIAG [42000] [FreeTDS][SQL Server]Cannot open database "mydatabase" requested by the login. The login failed.
DIAG [S1000] [FreeTDS][SQL Server]Unable to connect to data source
[ODBC][22870][SQLConnect.c][3917]
Exit:[SQL_ERROR]
[ODBC][22870][SQLError.c][424]
Entry:
Connection = 0x938b130
SQLState = 0xbf8ba54e
Native = 0xbf8ba350
Message Text = 0xbf8ba359
Buffer Length = 500
Text Len Ptr = 0xbf8ba356
[ODBC][22870][SQLError.c][461]
Exit:[SQL_SUCCESS]
SQLState = S1000
Native = 0xbf8ba350 -> 0
Message Text = [[unixODBC][FreeTDS][SQL Server]Unable to connect to data source]
[ODBC][22870][SQLError.c][424]
Entry:
Connection = 0x938b130
SQLState = 0xbf8ba54e
Native = 0xbf8ba350
Message Text = 0xbf8ba359
Buffer Length = 500
Text Len Ptr = 0xbf8ba356
[ODBC][22870][SQLError.c][461]
Exit:[SQL_SUCCESS]
SQLState = 37000
Native = 0xbf8ba350 -> 4060
Message Text = [[unixODBC][FreeTDS][SQL Server]Cannot open database "mydatabase" requested by the login. The login failed.]
[ODBC][22870][SQLError.c][424]
Entry:
Connection = 0x938b130
SQLState = 0xbf8ba54e
Native = 0xbf8ba350
Message Text = 0xbf8ba359
Buffer Length = 500
Text Len Ptr = 0xbf8ba356
[ODBC][22870][SQLError.c][461]
Exit:[SQL_SUCCESS]
SQLState = 37000
Native = 0xbf8ba350 -> 18456
Message Text = [[unixODBC][FreeTDS][SQL Server]Login failed for user 'username'.]
[ODBC][22870][SQLError.c][424]
Entry:
Connection = 0x938b130
SQLState = 0xbf8ba54e
Native = 0xbf8ba350
Message Text = 0xbf8ba359
Buffer Length = 500
Text Len Ptr = 0xbf8ba356
[ODBC][22870][SQLError.c][461]
Exit:[SQL_NO_DATA]
[ODBC][22870][SQLError.c][504]
Entry:
Environment = 0x938ab58
SQLState = 0xbf8ba54e
Native = 0xbf8ba350
Message Text = 0xbf8ba359
Buffer Length = 500
Text Len Ptr = 0xbf8ba356
[ODBC][22870][SQLError.c][541]
Exit:[SQL_NO_DATA]
[ODBC][22870][SQLFreeHandle.c][268]
Entry:
Handle Type = 2
Input Handle = 0x938b130
[ODBC][22870][SQLFreeHandle.c][317]
Exit:[SQL_SUCCESS]
[ODBC][22870][SQLFreeHandle.c][203]
Entry:
Handle Type = 1
Input Handle = 0x938ab58
when tsql command works... what could be?
unixODBC version is 2.2.11-7.1 and freeTDS doesn't appear as installed but it there is since there are present libtdsodbc.so in /usr/local/lib..
I have also to say only with a complete reboot of CentOS machine isql goes back working well...
What I can do or check? Thanks a lot in advance!
Cheers,
Luigi
you should fix odbc.ini to have a servername as configured at freetds.conf
instead of "Server = x.x.x.x" put "servername = XYZ"

Resources