I am trying to update postgresSQL table with psycopg2 (python package) sometimes it is failing with below error.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Here is the code
from psycopg2 import pool
now = datetime.now()
logoff_time = datetime(now.year, now.month, now.day, 15, 0, 0)
while True:
time.sleep(1)
try:
status = 'EXECUTED'
exec_type1 = 'CANCELLED'
exec_type2 = 'COMPLETED'
try:
postgreSQL_pool = pool.SimpleConnectionPool(1, 20, host = db_host,
database = db_name,
port = db_port,
user = db_user,
password = db_pwd)
if postgreSQL_pool:
print("Connection pool created successfully")
conn = postgreSQL_pool.getconn()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
sql = """ UPDATE orders SET status = %s, executed_type = %s WHERE order_id = %s"""
updated_rows = 0
try:
cur = conn.cursor()
cur.execute(sql, (status, exec_type1, order_id,))
conn.commit()
updated_rows = cur.rowcount
cur.close()
break
except (Exception, psycopg2.DatabaseError) as error:
print(error)
print(updated_rows)
except Exception as e:
print(e)
psycopg2 version: '2.8.6 (dt dec pq3 ext lo64)'
Postgres: PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-9), 64-bit
it is pretty much simple task but facing challenges. suggestions please
The server is crashing for some reason that you might be able to read in the server's logs.
Related
These are my JDBC connection details:
jdbcHostname = "ss-owaisde.database.windows.net"
jdbcPort = 1433
jdbcDatabase = "database-owaisde"
jdbcUsername = "owaisde"
jdbcPassword = "******"
jdbcDriver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbcUrl = f"jdbc:sqlserver://{jdbcHostname}:{jdbcPort};databaseName=
{jdbcDatabase};user{jdbcUsername};password={jdbcPassword};driver={jdbcDriver}"
Executing the spark read
df1 = spark.read.format("jdbc").option("url",jdbcUrl).option("dbtable",
"SalesLT.Product").load()
Getting the following error on databricks
java.sql.SQLException: No suitable driver
I tried to replicate your issue with your code:
I got same error:
As per my knowledge in your case URL is not build in correct format. I tried with below code:
jdbcHostname = "<servername>.database.windows.net"
jdbcPort = 1433
jdbcDatabase = "<dbname>"
jdbcUsername = "<username>"
jdbcPassword = "<password>"
jdbcDriver = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
#url = s"jdbc:sqlserver://${database_host}:${database_port}/${database_name}"
table = "Student"
jdbcUrl = f"jdbc:sqlserver://{jdbcHostname}:{jdbcPort};databaseName={jdbcDatabase}"
df1 = spark.read.format("jdbc").option("driver", jdbcDriver).option("url", jdbcUrl).option("dbtable", table).option("user", jdbcUsername).option("password", jdbcPassword).load()
Dataframe created successfully.
It worked fine for me, check from your end.
I am having trouble connecting to Amazon Keyspaces, both with my application code and cqlsh:
cqlsh cassandra.eu-west-2.amazonaws.com 9142 -u "xxxxxxxxxxxxxxx" -p "xxxxxxxxxxxxxxxxxxxxxx" --ssl
Connection error: ('Unable to connect to any servers', {'3.10.201.209': error(1, u"Tried connecting to [('3.10.201.209', 9142)]. Last error: [SSL] internal error (_ssl.c:727)")})
What is particularly confusing is that my setup worked in the past.
My cqlshrc:
[connection]
port = 9142
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
validate = true
certfile = /home/abc/.cassandra/AmazonRootCA1.pem
I fetched the certificate like this:
wget -c https://www.amazontrust.com/repository/AmazonRootCA1.pem
DNS seems fine:
nslookup cassandra.eu-west-2.amazonaws.com
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: cassandra.eu-west-2.amazonaws.com
Address: 3.10.201.209
I recently upgraded to Ubuntu 20.04 from 18.04, which may be causing issues.
Update: Yes, it probably changed the default SSL protocol
I figured it out for cqlsh; you need to set the SSL version:
[connection]
port = 9142
factory = cqlshlib.ssl.ssl_transport_factory
[cql]
version = 3.4.4
[ssl]
validate = true
certfile = /home/abc/.cassandra/AmazonRootCA1.pem
version = TLSv1_2
The fix for .NET solution is similar; you must set the SslProtocols correctly.
Here is an F# script that works:
#load "../.paket/load/netcoreapp3.1/CassandraCSharpDriver.fsx"
open System
open System.Net.Security
open System.Security
open System.Security.Authentication
open System.Security.Cryptography
open System.Security.Cryptography.X509Certificates
open Cassandra
let private getEnvVar (name : string) =
let x = Environment.GetEnvironmentVariable name
if String.IsNullOrWhiteSpace x
then
failwithf "The environment variable %s must be set" name
else
x
let region = getEnvVar "AWS_REGION"
let keyspace = getEnvVar "AWS_KEYSPACES_KEYSPACE"
let keyspacesUsername = getEnvVar "AWS_KEYSPACES_USERNAME"
let keyspacesPassword = getEnvVar "AWS_KEYSPACES_PASSWORD"
async {
let certCollection = X509Certificate2Collection ()
use cert = new X509Certificate2 (#"./AmazonRootCA1.pem", "amazon")
certCollection.Add (cert) |> ignore
let sslOptions =
SSLOptions
(
SslProtocols.Tls12,
true,
(fun sender certificate chain sslPolicyErrors ->
if sslPolicyErrors = SslPolicyErrors.None
then
true
else
printfn "Cassandra node SSL certificate validation error(s): {%A}" sslPolicyErrors
false)
)
|> (fun x -> x.SetCertificateCollection(certCollection))
let contactPoints = [| sprintf "cassandra.%s.amazonaws.com" region |]
let cluster =
Cluster.Builder()
.AddContactPoints(contactPoints)
.WithPort(9142)
.WithAuthProvider(PlainTextAuthProvider (keyspacesUsername, keyspacesPassword))
.WithSSL(sslOptions)
.Build()
use! cassandra =
cluster.ConnectAsync keyspace
|> Async.AwaitTask
printfn "Connected. "
}
|> Async.RunSynchronously
It should be easy to translate to C# :)
I am trying to insert data into my SQLite database, it goes fine until I get the error while creating a function for DB. It sends OperationalError at cursor.
I couldn't find solution for my problem.
Code I'm using:
import sqlite3
from sqlite3 import *
SQL_CREATE_STATEMENT = '''CREATE TABLE password
(id integer PRIMARY KEY NOT NULL,username text, password text, source text)'''
SQL_INSERT_STATEMENT = '''INSERT INTO password (username, password, source)VALUES({},{},{});'''
DATABASE_PATH = 'home/taha/lessons/projects/passStorage/passDB.db'
DATA = dict()
def create_connection(db_file):
try:
conn = sqlite3.connect(db_file)
return conn
except Error as e:
return e
def create_table(connection, sql_commands):
c = connection.cursor()
c.execute(sql_commands)
print('done')
def get_input():
USERNAME = input('username: ')
PASSWORD = input('password: ')
SOURCE = input('source: ')
return USERNAME,PASSWORD,SOURCE
def insert_date(connection, data):
c = connection.cursor()
c.execute(SQL_INSERT_STATEMENT.format(data.values))
def main():
conn = create_connection(DATABASE_PATH)
create_table(conn, SQL_CREATE_STATEMENT)
user_info = get_input()
DATA['username'], DATA['password'], DATA['SOURCE'] = user_info
insert_date(conn, DATA)
if __name__ == '__main__':
main()
I expect no error but it sends this:
c = connection.cursor()
AttributeError: 'OperationalError' object has no attribute 'cursor'
def create_connection(db_file):
try:
conn = sqlite3.connect(db_file)
return conn
except Error as e:
return e # <-- here you return OperationalError instance
AttributeError: 'OperationalError' object has no attribute 'cursor'
Shows that OperationalError has no attribute cursor
Add additional logic that check connection here.
I believe the core of your problem is wrong file path:
DATABASE_PATH = 'home/taha/lessons/projects/passStorage/passDB.db'
But I believe should be
DATABASE_PATH = '/home/taha/lessons/projects/passStorage/passDB.db'
I want to create a model in R using a connection to data stored in SQL Server datawarehouse.
I tried to use RevoScaleR library which returned
package RevoScaleR is not available (for R version 3.4.1)
so, I edited the connection string (given on the code below) for ODBC library:
install.packages("RevoScaleR")
#require("RevoScaleR")
if (!require("RODBC"))
install.packages("RODBC")
conn <- odbcDriverConnect(connection="Driver={SQL Server Native Client 11.0}; Server=CZPHADDWH01/DEV; Database=DWH_Staging; trusted_connection=true")
sqlWait <- TRUE;
sqlConsoleOutput <- FALSE;
cc <- RxInSqlServer(connectionString = conn, wait = sqlWait)
rxSetComputeContext(cc)
train_query <- "SELECT TOP(10000) * FROM dim.Contract"
formula <- as.formula("Cosi ~ ContractID + ApprovedLoanAmount + ApprovedLoadDuration")
forest_model <- rxDForest(formula = formula,
data = train_query,
nTree = 20,
maxDepth = 32,
mTry = 3,
seed = 5,
verbose = 1,
reportProgress = 1)
rxDForest_model <- as.raw(serialize(forest_model, connection = conn))
lenght(rxDForest_model)
However:
package 'RODBC' successfully unpacked and MD5 sums checked
The downloaded binary packages are in
C:\Users\sjirak\AppData\Local\Temp\Rtmpqa9iKN\downloaded_packages
Error in odbcDriverConnect(connection = "Driver={SQL Server Native
Client 11.0}; Server=CZPHADDWH01/DEV; Database=DWH_Staging;
trusted_connection=true") : could not find function
"odbcDriverConnect" In library(package, lib.loc = lib.loc,
character.only = TRUE, logical.return = TRUE, : there is no
package called 'RODBC'
Any help would be appreciated.
Looking at the documentation of the ODBC, I see the following functions
odbc-package
dbConnect,OdbcDriver-method
dbUnQuoteIdentifier
odbc
odbc-tables
OdbcConnection
odbcConnectionActions
odbcConnectionIcon
odbcDataType
OdbcDriver
odbcListColumns
odbcListDataSources
odbcListDrivers
odbcListObjects
odbcListObjectTypes
odbcPreviewObject
OdbcResult
odbcSetTransactionIsolationLevel
test_roundtrip
hence I dont see your function in this list. This could be the reason why...
Hence, check the documentation for the proper function.
I met a really weird issue: I'm able to connect to my SQL Server with tsql but I'm not able to do the same with pymssql. Let me give you more details.
With this kind of tsql command everything seems to be ok:
$ tsql -H '10.10.10.2' -U 'DOMAIN\myuser' -p 63849
Password:
locale is "fr_FR.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1>
But with the following basic python script I got an issue:
$ cat test_mssql.py
# -*- coding: utf8 -*-
import pymssql
import pandas as pd
conn = pymssql.connect(server=r'10.10.10.2', user=r'DOMAIN\myuser', password='mypassword', port='63849')
$ python test_mssql.py
Traceback (most recent call last):
File "api-vinci-rh/current/test_mssql.py", line 60, in <module>
conn = pymssql.connect(server=r'10.10.10.2', user=r'DOMAIN\myuser', password='mypassword', port='63849')
File "pymssql.pyx", line 641, in pymssql.connect (pymssql.c:10788)
pymssql.OperationalError: (20002, 'DB-Lib error message 20002, severity 9:\nAdaptive Server connection failed\n')
By activating TDSDUMPCONFIG env variable, I can see a difference between the tls command and my python script:
TDSDUMPCONFIG tls trace
config.c:224:Final connection parameters:
config.c:225: server_name = 10.10.10.2
config.c:226: server_host_name = 10.10.10.2
config.c:227: ip_addr = 10.10.10.2
config.c:228: instance_name =
config.c:229: port = 63849
config.c:230: major_version = 7
config.c:231: minor_version = 1
config.c:232: block_size = 0
config.c:233: language = us_english
config.c:234: server_charset =
config.c:235: connect_timeout = 0
config.c:236: client_host_name = myhost
config.c:237: client_charset = UTF-8
config.c:238: app_name = TSQL
config.c:239: user_name = DOMAIN\myuser
config.c:242: library = TDS-Library
config.c:243: bulk_copy = 0
config.c:244: suppress_language = 0
config.c:245: encrypt level = 0
config.c:246: query_timeout = 0
config.c:249: database =
config.c:250: dump_file = /tmp/freetds.log
config.c:251: debug_flags = 0
TDSDUMPCONFIG pymssql trace
config.c:224:Final connection parameters:
config.c:225: server_name = 10.10.10.2:63849
config.c:226: server_host_name = 10.10.10.2
config.c:227: ip_addr = 10.10.10.2
config.c:228: instance_name =
config.c:229: port = 63849
config.c:230: major_version = 7
config.c:231: minor_version = 1
config.c:232: block_size = 0
config.c:233: language = us_english
config.c:234: server_charset =
config.c:235: connect_timeout = 0
config.c:236: client_host_name = myhost
config.c:237: client_charset = UTF-8
config.c:238: app_name = pymssql=2.1.2
config.c:239: user_name =
config.c:242: library = DB-Library
config.c:243: bulk_copy = 0
config.c:244: suppress_language = 0
config.c:245: encrypt level = 0
config.c:246: query_timeout = 0
config.c:249: database =
config.c:250: dump_file = /tmp/freetds.log
config.c:251: debug_flags = 0
config.c:252: text_size = 64512
config.c:253: broken_dates = 0
config.c:254: emul_little_endian = 0
config.c:255: server_realm_name =
I think the 2 main differences is about the user_name which is empty with the python script and the library which is DB-Library instead of TDS-Library.
I currently find no way to force these parameters in the python script and I find no clue about my issue (the error message is a generic one).
I'm using python 2.7.9 and pymssql==2.1.2
More info about my env:
$ tsql -C
Compile-time settings (established with the "configure" script)
Version: freetds v0.91
freetds.conf directory: /etc/freetds
MS db-lib source compatibility: no
Sybase binary compatibility: yes
Thread safety: yes
iconv library: yes
TDS version: 4.2
iODBC: no
unixodbc: yes
SSPI "trusted" logins: no
Kerberos: yes
$ grep -v '^#' /etc/freetds/freetds.conf
[global]
tds version = 7.1
dump file = /tmp/freetds.log
text size = 64512
enable gssapi delegation = off
If you have any clue to help let me know.
Edit: I have another SQL Server instance for my test and my python script works... the username in the TDSDUMPCONFIG is set. So Iguess there is something strange on my env but don't know what
I find a clue and I'm quite surprised byt it... If I use a username with less than 32 characters, I have no more issue with pymssql ! I didn't find anything about this weird bug (?) but I promess it fixes my issue.
I need to see the code of pymssql and see if I find any lenght restriction or issue about the lenght of the username