The following code is running into a Docker container. I have a connection specified as follow which is working as I could make a simple query based on it.
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://username:pw#hostname?driver_name")
con_xpf = engine.connect()
con_xpf.execute("use db_name;")
After that I create a sqlite3 DB and connect to it:
DBNAME = "data/NEWDB.db"
con = sqlite3.connect(DBNAME)
tbllist = ["AAAAA","BBBBBBBB","CCCCCCCCC","DDDDDDDDD","EEEEEEEEE"
"FFFFFFFFF","GGGGGGGGG","HHHHHH","IIIIIIIII"]
chunksizes = [500000,800000,500000,1000000,500000,500000,500000,900000,500000]
The problem start from here when I try to run the following code to query from the first DB to write into the sqlite DB:
def loads_data_from_xpf(tbllist,chunksizes) :
for tbl,chunksize in zip(tbllist,chunksizes) :
cnt = 0
maxcnt = math.ceil(pd.read_sql("SELECT COUNT(*) CNT FROM x.{}".format(tbl),con_xpf).CNT[0]/chunksize)
with tqdm(range(maxcnt)) as pbar:
for chunk in pd.read_sql("SELECT * FROM x.{} A".format(tbl),con_xpf,chunksize=chunksize) :
chunk.to_sql(tbl,con,if_exists="append")
pbar.update()
loads_data_from_xpf(tbllist,chunksizes)
I got the following error:
OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x71 (113) (SQLGetData)')
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Could the problem be because of the size of the query? Do you see any error somewhere? I am stuck for a while. Any help would be appreciated.
EDIT
I am now able to see the sqlite DB file updating meaning data are pushed into it. But I have the following error which is because of the Jupyter Kernel. Help please
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/nbclient/client.py", line 841, in async_execute_cell
exec_reply = await self.task_poll_for_reply
concurrent.futures._base.CancelledError
During handling of the above exception, another exception occurred:
nbclient.exceptions.DeadKernelError: Kernel died
Related
New to SQL-Server. I'm attempting to load a pdf to a SQL-Server table (data type varbinary(max)) via PERL/MS ODBC driver/DBD::ODBC using the following (simplified) code:
use DBI qw(:sql_types);
open my $pdfFH, "test.pdf";
my #pdf = <$pdfFH>; close $pdfFH;
my $pdfStr = join('', #pdf);
my $dbh = <...valid db-handle ...>;
my $sth = $dbh->prepare(qq(
insert into
TestTable(Report)
values
(?)));
$sth->bind_param(1,$pdfStr,DBI::SQL_VARBINARY);
$sth->execute;
Error:
DBD::ODBC::st bind_param failed: [Microsoft][ODBC Driver 17 for SQL Server]Invalid precision value (SQL-HY104) at ./t_sqlserver.pl line 37.
DBD::ODBC::st execute failed: [Microsoft][ODBC Driver 17 for SQL Server]COUNT field incorrect or syntax error (SQL-07002) at ./t_sqlserver.pl line 38.
I am able to successfully load other data types. An alternative is to load the pdf locally from the file system using OPENROWSET(BULK...) but I would prefer to load directly to avoid moving the file from Linux to Windows.
The driver should be clever enough to guess the correct type most of the times. Try binding the parameter without specifying the type at all.
I am working off of a server housing various SQL databases (accessed via Microsoft SQL Server Management Studio) and am going to use R to perform analyses and explore a specific database within the server. I have network security that permits communication between machines, drivers installed on the R server, and RODBC installed.
When I attempt to establish a Windows ODBC connection in the Control panel>Administrative>Data Sources, I can only add a data source for the entirety of the SQL server, not just for the specifc database I want to look at. I pasted the code I have been experimenting with below.
library(RODBC)
channel <- odbcConnect("Example", uid="xxx", pwd=****");
sqlTables(channel)
sqlTables(ch, tableType = "TABLE")
res <- sqlFetch(ch, "samp.le", max = 15) #not recognizing as a table
library(RODBC)
ch <- odbcDriverConnect('driver={"SQL Server"}; server=Example; database=dbasesample; uid="xxxx", pwd = "****"')
Response: Warning messages:
1: In odbcDriverConnect("driver={\"SQL Server\"}; server=sample; database=dbasesample; uid=\"xxxx", pwd = \"xxxx\"") :
[RODBC] ERROR: state IM002, code 0, message [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
2: In odbcDriverConnect("driver={\"SQL Server\"}; server=sample; database=dbasesample; uid=\"xxxx\", pwd = \"xxxx!\"") :
ODBC connection failed
Any insight into this issue would be much appreciated.
Although while querying with the sqlQuery() function you can specify database, schema and table, e.g.
library(RODBC)
con = odbcConnect(dsn = 'local')
sample_query = sqlQuery(con,'select * from db.dbo.table')
I have not found a way to define the database from within the function parameters while using sqlFetch() or sqlSave(). An indirect way would be to define the default database in the dsn (as written in the comments). But then, you would need a different dsn for each database you would like to use.
A better solution would be to use the odbc and DBI packages instead of RODBC, and define the database in the connection statement e.g.
library(dplyr)
library(DBI)
library(odbc)
con <- dbConnect(dsn = 'local',database = 'db')
copy_to(con, rr2, temporary = F)
By the way, I found copy_to to be much faster than the equivalent sqlSave of RODBC.
Brief introduction of the problem:
the main problem is not in the connection procedure, i could connect to database successfully, and insert some rows in my database(firs code block shows this),but after closing the connection if someone tries to insert a row in the database ,matlab will terminate suddenly without any clear error message, (i expect to have a function to check if the connection is open or close or to get an error message to handle the error but non of these happened and just matlab closed because of a fatal error)
i wrote the following code to connect to MS SQL SERVER database in matlab:
conn=database.ODBCConnection('MS SQL SERVER','','');
insert(conn,'trace',{'obj_id','obj_type_id','time_step','pos_x','pos_y','vel_x','vel_y'},[1,1,1,0,0,0,0]);
close(conn);
and every thing was ok.
then i tried to insert another row (to check what is the error message) then Matlab closed (due to fatal error) without showing any error message.
i tried to use following functions to get status of database connection before inserting new raws:
isconnection(conn);
ping(conn);
but it says
Undefined function 'ping' for input arguments of type
'database.ODBCConnection'.
Undefined function 'isconnection' for input arguments of type
'database.ODBCConnection'
even i tried to use try-catch block but it didn't work and Matlab Closed for fatal error.
so i want to know is there any way to chek status of native ODBC to prevent sudden close of matlab in case of a closed connection??
Update:
>> conn=database.ODBCConnection('MS SQL SERVER','','')
conn =
ODBCConnection with properties:
Instance: 'MS SQL SERVER'
UserName: ''
Message: []
Handle: [1x1 database.internal.ODBCConnectHandle]
TimeOut: 0
AutoCommit: 0
Type: 'ODBCConnection Object'
>> close(conn)
>> conn
conn =
ODBCConnection with properties:
Instance: 'MS SQL SERVER'
UserName: ''
Message: []
Handle: [1x1 database.internal.ODBCConnectHandle]
TimeOut: 0
AutoCommit: 0
Type: 'ODBCConnection Object'
no properties or message changed before and after closing the connection,
the problem is that i don't know how to check that if a connection is still open or closed in other parts of a program!
in this case if i use an insert command when a connection was closed before,
matlab suddenly terminates (and show the message MATLAB(R2013B) has stopped working),
so i want to know is there any way to check if a native odbc connection has closed before?
Further update
>> conn=database('MS SQL SERVER','','')
conn =
Instance: 'MS SQL SERVER'
UserName: ''
Driver: []
URL: []
Constructor: [1x1 com.mathworks.toolbox.database.databaseConnect]
Message: []
Handle: [1x1 sun.jdbc.odbc.JdbcOdbcConnection]
TimeOut: 0
AutoCommit: 'on'
Type: 'Database Object'
>> isconnection(conn)
ans =
1
>> close(conn)
>> isconnection(conn)
ans =
0
i mean a function like "isconnection" in the example above for jdbc connection which returns 1 if a connection is open and 0 if the connection closed before.
I request you to check the database connection with toolstrip functionality of Matlab. You can find complete guide from here...
You can perform the testing first so that you can ruled out of any problem with server..
Once it is connected successfully..you can check the code.connection settings and apply it in your code accordingly.
Regards,
As per the documentation you can check status of an existing database.ODBCConnection or database.ODBCCursor in the Database Toolbox by checking the value of the Message property in the database.ODBCConnection object and the database.ODBCCursor Object.
You may need to set Error handling to store using setdbprefs('ErrorHandling','store'). Use setdbprefs('ErrorHandling','report') to switch it back again.
ping and isconnection only work on database connection object and not on database.ODBCConnection objects.
Working through a tutorial to pull database data with:
install.packages('RODBC')
require(RODBC)
myNewDB=odbcConnect("QV Training")
And I get the error:
In odbcDriverConnect("DSN=QV Training")
Data source name not found and no default driver specified
In odbcDriverConnect("DSN=QV Training") : ODBC connection failed
Is 'QV Training' meant to be the name of a database that may no longer be present?
How does R know where to look for the database anyway?
Thank you!
In Windows (unsure of other OSes) you need to go into the ODBC Data Source Administrator, and add the data source. The ODBC Data Source Administrator is accessed via the 'Administrative Tools' section of Control Panel (in Windows 10 at least).
The connection command is then simply
conn <- odbcConnect("QV Training")
library(RODBC)
con <- odbcConnect("Oracle", uid="system", pwd="root", rows_at_time = 500)
sqlQuery(con, "select file_name,sum(bytes)/1024/1024 AS MB from dba_data_files group by file_name")
d <- sqlQuery(con, "select * from dba_data_files")
close(con)
Hi There is a way to use django-pyodbc with pervasive database ? I try different settings but when I try to inspectdb to create a model from the database it shows me this error but I can create querys in pervasive with python pyodbc.
`('42000', '[42000] [Pervasive][ODBC Client Interface][LNA][Pervasive][ODBC
Engine Interface]Invalid SET statement. (0) (SQLExecDirectW)')
Request Method: GET
Request URL:
Django Version: 1.6.6
Exception Type: ProgrammingError
Exception Value:
('42000', '[42000] [Pervasive][ODBC Client Interface][LNA][Pervasive][ODBC Engine Interface]Invalid SET statement.
(0) (SQLExecDirectW)')
Exception Location: C:\Python27\lib\site-packages\django_pyodbc\base.py in _cursor, line 296
Python Executable: C:\Python27\python.exe
Python Version: 2.7.8
cursor.execute("SET DATEFORMAT Ymd; SET DATEFIRST %s" % self.datefirst)
pyodbc.ProgrammingError: ('42000', '[42000] [Pervasive][ODBC Client Interface][LNA][Pervasive] [ODBC Engine Interface]Invalid SET statement. (0) (SQLExecDirectW)')`
and this is the piece of the error when I try to run inspectdb on the database.
According to the main problem is in this file base.py line 296 is some problem with a set statement on the odbc driver this is piece of the code on base.py I try to comment but after another thing shows up.
`cursor.execute("SET DATEFORMAT Ymd; SET DATEFIRST %s" % self.datefirst)
if self.ops.sql_server_ver < 2005:
self.creation.data_types['TextField'] = 'ntext'
self.features.can_return_id_from_insert = False
ms_sqlncli = re.compile('^((LIB)?SQLN?CLI|LIBMSODBCSQL)')
self.drv_name = self.connection.getinfo(Database.SQL_DRIVER_NAME).upper()
# http://msdn.microsoft.com/en-us/library/ms131686.aspx
if self.ops.sql_server_ver >= 2005 and ms_sqlncli.match(self.drv_name) and self.MARS_Connection:
# How to to activate it: Add 'MARS_Connection': True
# to the DATABASE_OPTIONS dictionary setting
self.features.can_use_chunked_reads = True`
Try put single quote around %s:
cursor.execute("SET DATEFORMAT Ymd; SET DATEFIRST '%s'" % self.datefirst)
Check your access rights. The error message indicates a syntax error but according to mssql support site, error code 42000 is either "Syntax error or access violation".