Brief introduction of the problem:
the main problem is not in the connection procedure, i could connect to database successfully, and insert some rows in my database(firs code block shows this),but after closing the connection if someone tries to insert a row in the database ,matlab will terminate suddenly without any clear error message, (i expect to have a function to check if the connection is open or close or to get an error message to handle the error but non of these happened and just matlab closed because of a fatal error)
i wrote the following code to connect to MS SQL SERVER database in matlab:
conn=database.ODBCConnection('MS SQL SERVER','','');
insert(conn,'trace',{'obj_id','obj_type_id','time_step','pos_x','pos_y','vel_x','vel_y'},[1,1,1,0,0,0,0]);
close(conn);
and every thing was ok.
then i tried to insert another row (to check what is the error message) then Matlab closed (due to fatal error) without showing any error message.
i tried to use following functions to get status of database connection before inserting new raws:
isconnection(conn);
ping(conn);
but it says
Undefined function 'ping' for input arguments of type
'database.ODBCConnection'.
Undefined function 'isconnection' for input arguments of type
'database.ODBCConnection'
even i tried to use try-catch block but it didn't work and Matlab Closed for fatal error.
so i want to know is there any way to chek status of native ODBC to prevent sudden close of matlab in case of a closed connection??
Update:
>> conn=database.ODBCConnection('MS SQL SERVER','','')
conn =
ODBCConnection with properties:
Instance: 'MS SQL SERVER'
UserName: ''
Message: []
Handle: [1x1 database.internal.ODBCConnectHandle]
TimeOut: 0
AutoCommit: 0
Type: 'ODBCConnection Object'
>> close(conn)
>> conn
conn =
ODBCConnection with properties:
Instance: 'MS SQL SERVER'
UserName: ''
Message: []
Handle: [1x1 database.internal.ODBCConnectHandle]
TimeOut: 0
AutoCommit: 0
Type: 'ODBCConnection Object'
no properties or message changed before and after closing the connection,
the problem is that i don't know how to check that if a connection is still open or closed in other parts of a program!
in this case if i use an insert command when a connection was closed before,
matlab suddenly terminates (and show the message MATLAB(R2013B) has stopped working),
so i want to know is there any way to check if a native odbc connection has closed before?
Further update
>> conn=database('MS SQL SERVER','','')
conn =
Instance: 'MS SQL SERVER'
UserName: ''
Driver: []
URL: []
Constructor: [1x1 com.mathworks.toolbox.database.databaseConnect]
Message: []
Handle: [1x1 sun.jdbc.odbc.JdbcOdbcConnection]
TimeOut: 0
AutoCommit: 'on'
Type: 'Database Object'
>> isconnection(conn)
ans =
1
>> close(conn)
>> isconnection(conn)
ans =
0
i mean a function like "isconnection" in the example above for jdbc connection which returns 1 if a connection is open and 0 if the connection closed before.
I request you to check the database connection with toolstrip functionality of Matlab. You can find complete guide from here...
You can perform the testing first so that you can ruled out of any problem with server..
Once it is connected successfully..you can check the code.connection settings and apply it in your code accordingly.
Regards,
As per the documentation you can check status of an existing database.ODBCConnection or database.ODBCCursor in the Database Toolbox by checking the value of the Message property in the database.ODBCConnection object and the database.ODBCCursor Object.
You may need to set Error handling to store using setdbprefs('ErrorHandling','store'). Use setdbprefs('ErrorHandling','report') to switch it back again.
ping and isconnection only work on database connection object and not on database.ODBCConnection objects.
Related
The following code is running into a Docker container. I have a connection specified as follow which is working as I could make a simple query based on it.
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://username:pw#hostname?driver_name")
con_xpf = engine.connect()
con_xpf.execute("use db_name;")
After that I create a sqlite3 DB and connect to it:
DBNAME = "data/NEWDB.db"
con = sqlite3.connect(DBNAME)
tbllist = ["AAAAA","BBBBBBBB","CCCCCCCCC","DDDDDDDDD","EEEEEEEEE"
"FFFFFFFFF","GGGGGGGGG","HHHHHH","IIIIIIIII"]
chunksizes = [500000,800000,500000,1000000,500000,500000,500000,900000,500000]
The problem start from here when I try to run the following code to query from the first DB to write into the sqlite DB:
def loads_data_from_xpf(tbllist,chunksizes) :
for tbl,chunksize in zip(tbllist,chunksizes) :
cnt = 0
maxcnt = math.ceil(pd.read_sql("SELECT COUNT(*) CNT FROM x.{}".format(tbl),con_xpf).CNT[0]/chunksize)
with tqdm(range(maxcnt)) as pbar:
for chunk in pd.read_sql("SELECT * FROM x.{} A".format(tbl),con_xpf,chunksize=chunksize) :
chunk.to_sql(tbl,con,if_exists="append")
pbar.update()
loads_data_from_xpf(tbllist,chunksizes)
I got the following error:
OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x71 (113) (SQLGetData)')
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Could the problem be because of the size of the query? Do you see any error somewhere? I am stuck for a while. Any help would be appreciated.
EDIT
I am now able to see the sqlite DB file updating meaning data are pushed into it. But I have the following error which is because of the Jupyter Kernel. Help please
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/nbclient/client.py", line 841, in async_execute_cell
exec_reply = await self.task_poll_for_reply
concurrent.futures._base.CancelledError
During handling of the above exception, another exception occurred:
nbclient.exceptions.DeadKernelError: Kernel died
Apologies if this has been answered elsewhere, but I could not find it. With the following code I am connecting to an MS SQL database using the RJDBC mechanism using pooling from the pool package:
library(RJDBC)
library(DBI)
library(pool)
library(dplyr)
drv <-
JDBC(
"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"C:/R/RJDBC/Microsoft JDBC Driver 6.0 for SQL Server/sqljdbc_6.0/enu/jre8/sqljdbc42.jar"
)
pool_instance <- dbPool(
drv = drv,
dbname = "dbasename",
url = "jdbc:sqlserver://sql01",
user = "user",
password = "password"
)
mydata <- dbGetQuery(pool_instance, "select * from my.Table")
src_pool(pool_instance) %>% tbl("my.Table") %>% head(5)
When I run this code, I make a successful connection to my SQL Server database and the dbGetQuery function call retrieves the data as expected.
However, when I call the src_pool function I get the following error message:
Error in UseMethod("tbl") : no applicable method for 'tbl' applied
to an object of class "c('src_', 'src_sql', 'src')"
If I call the function src_pool(pool_instance) separately, without piping to the tbl function, the error message is similar:
Error in UseMethod("src_desc") : no applicable method for
'src_desc' applied to an object of class "c('src_', 'src_sql', 'src')"
I expected that either dplyr or pool would provide for these methods? Do I need to write code for these methods? What am I missing?
Note that I am a newby to SQL Server database connectivity.
In sqlserver I have a function which generates a complex xml of all products with several tables joined: location, suppliers, orders etc.
No problem in that, it runs in 68 sec and produces around 450MB.
It should only be called occationally during migration to another server, so it doesn't matter it takes some time.
I want to make this available for download over webserver.
I've tried some variations of this in classic asp:
Response.Buffer = false
set rs=conn.execute("select cast(dbo.exportXML() as varchar(max)) as res")
response.write rs("res")
But I just get a standard
An error occurred on the server when processing the URL. Please contact the system administrator.
If you are the system administrator please click here to find out more about this error.
Not my usual custom 500-errorhandler, so I'm not sure how to find the error.
The problem is in response.write rs("res"), if i just do
temp = rs("res")
the script runs, but displays nothing of cause; if I then
response.write temp
I get the same failure.
So the problem is writing such a ling string.
Can I save the file from tsql directly; and run the job periodically from sql agent?
I found that there seems to be a limit on how much data can be written at once using Response.Write. The workaround I used was to break the data into chunks like this:
Dim Data, Done
Done = False
Do While Not Done
Data = RecordSet(0).GetChunk(8192)
If Not Len(Data) = 0 Then
Response.Write Data
Else
Done = True
End If
Loop
Try this:
Response.ContentType = "text/xml"
rs.CursorLocation = 3
rs.Open "select cast(dbo.exportXML() as varchar(max)) as res",conn
'Persist the Recordset in XML format to the ASP Response object.
'The constant value for adPersistXML is 1.
rs.Save Response, 1
Working through a tutorial to pull database data with:
install.packages('RODBC')
require(RODBC)
myNewDB=odbcConnect("QV Training")
And I get the error:
In odbcDriverConnect("DSN=QV Training")
Data source name not found and no default driver specified
In odbcDriverConnect("DSN=QV Training") : ODBC connection failed
Is 'QV Training' meant to be the name of a database that may no longer be present?
How does R know where to look for the database anyway?
Thank you!
In Windows (unsure of other OSes) you need to go into the ODBC Data Source Administrator, and add the data source. The ODBC Data Source Administrator is accessed via the 'Administrative Tools' section of Control Panel (in Windows 10 at least).
The connection command is then simply
conn <- odbcConnect("QV Training")
library(RODBC)
con <- odbcConnect("Oracle", uid="system", pwd="root", rows_at_time = 500)
sqlQuery(con, "select file_name,sum(bytes)/1024/1024 AS MB from dba_data_files group by file_name")
d <- sqlQuery(con, "select * from dba_data_files")
close(con)
We are in the process to migrate our SQL 2000 box to SQL 2008. But we ran into an issue; when a result set (rows or not) is returned by using a query that has a UNION. Later in the code we try to add a new row and assign field to it but because a UNION was used, when we try to assign a value to the field it gives us a Multiple-step operation generated errors. Check each status value. error. We tried the following code on a Windows XP & Windows 7 and got the same result. But when we change our connection string to point back to our SQL 2000 box we don't get that error any more.
The following example show the problem we are having.
var c = new ADODB.Connection();
var cmd = new ADODB.Command();
var rs = new ADODB.Recordset();
object recordsAffected;
c.Open("Provider=SQLOLEDB;Server=*****;Database=*****;User Id=*****;Password=*****;");
cmd.ActiveConnection = c;
cmd.CommandType = ADODB.CommandTypeEnum.adCmdText;
cmd.CommandText = "create table testing2008 (id int)";
cmd.Execute(out recordsAffected);
try {
cmd.CommandText = "select * from testing2008 union select * from testing2008";
rs.CursorLocation = ADODB.CursorLocationEnum.adUseClient;
rs.Open(cmd, Type.Missing, ADODB.CursorTypeEnum.adOpenDynamic, ADODB.LockTypeEnum.adLockBatchOptimistic, -1);
rs.AddNew();
rs.Fields["id"].Value = 0; //throws exception
rs.Save();
}
catch (Exception ex) {
MessageBox.Show(ex.ToString());
}
finally {
cmd.CommandText = "drop table testing2008";
cmd.Execute(out recordsAffected);
c.Close();
}
The link below is an article that gives a great breakdown of the 6 scenarios this error message can occur:
Scenario 1 - Error occurs when trying to insert data into a database
Scenario 2 - Error occurs when trying to open an ADO connection
Scenario 3 - Error occurs inserting data into Access, where a fieldname has a space
Scenario 4 - Error occurs inserting data into Access, when using adLockBatchOptimistic
Scenario 5 - Error occurs inserting data into Access, when using Jet.OLEDB.3.51 or ODBC driver (not Jet.OLEDB.4.0)
Scenario 6 - Error occurs when using a Command object and Parameters
http://www.adopenstatic.com/faq/80040e21.asp
Hope it may help others that may be facing the same issue.
It is type mismatch, try
rs.Fields["id"].Value = "0";
or make sure you assign a Variant to the value.
Since I posted this problem, we figured out that the problem was when you do a union the attributes on the fields are not bound (i.e. the attributes: basecatalog, basetable & basecolumn are empty) to remedy our problem we had to force the values of those attributes, by saving the recordset to xml (adPersistXML), change the xml and reopen the recordset from the xml. This rebound the fields and we were able to continue. We know this may not be the most efficient solution, but it was for an older app and we didn't want to rewrite the sql statements. It looks like the main error Multiple-step operation generated errors. Check each status value. is related to when an error occurs when a value is assigned to a field.
Two things I can think of... Make sure your "ID" column will accept a zero (0). Also - I've stopped this issue on one occasion by not using the adUseClient cursor (try server).
Many times this is a type mismatch, trying to stuff a NULL into a non-null column, or attempting to write more characters into a column than it's designed to take.
Hope this helps. - Freddo
Same issue occurred to me the problem was that i violated an object property , in my case it was size the error came out as
"IntegrationException: Problem (Multiple-step operation generated errors. Check each status value.)"
Imports ADODB
Dim _RecordSet As Recordset
_rs.Fields.Append("Field_Name", DataTypeEnum.adVarChar, 50)
_Recordset("Field_Name").Value = _RecordDetails.Field_NameValue
_RecordDetails.Field_NameValue length was more than 50 chars , so this property was violated , hence the error occurred .
I found another scenario:
When I was trying to set the value of a adLongVarChar field in a new record for a memory-only adodb.recordset. In my case, the error was triggered because the string I was passing had a buried unicode character.
I found this error when our legacy application was trying to parse 1/1/0001 12AM date and time. Looks like VB6 recordsets doesn't like that value.
To get rid of the errors, I had to set all the offending dates to null.
I was getting this error when trying to insert/update the field with a value that did not match the table>field type.
For example, the database table > field was
char(1)
however, I was trying to insert/update
"apple"
into the record.
Once I change the inputted value to "a" and it worked.