I currently have an MS Access file that is about 1GB in size. I run an SQL query inside of it, with a bunch of joins, in order to obtain the exact data that i need for my reports. That bit works just fine, despite being quite slow.
In an effort to diminish the time it takes to perform these tasks, i am trying to do this very same thing using the PYODBC library, in Python. I have been able to connect to the MS Access file just fine, i can view all the tables and the queries that i have, but that exact same query is unable to run while using PYODBC because it "exceeds the maximum file size of 2GB". I have no idea why this happens when connecting to the database via Python but not in Access itself.
The code i am using is this:
import pyodbc
pyodbc.drivers()
conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\Desktop\filename.accdb;')
cursor = conn.cursor()
for i in cursor.tables():
print(i)
autocommit=True # i read this shrinks the temp file size, but have tried 'False' as well without success
sql = "{CALL QUERY_NAME}" # the query that runs just fine in MSAccess
cursor.execute(sql)
The error message i receive is 'Size of the query result in larger than the maximum size of a database (2GB), or there is not enough temporary storage space on the disk to store the query result'.
Shouldn't this work just as it does inside MSAccess itself? Anyone has any clue?
Thank you!
Related
I have a postgres table containing nearly 700,000 records, I import that table into MS access (via an ODBC data source) and end up with only 250,000 records.
I start with an empty MS access database (520 kbytes). Select (external data)/(New data source)/(from other sources)/(ODBC database)/(Import the source data)/(Machine data source) I pick my ODBC postgres database, and select the table I want, wait for 30 seconds, then I get a message box saying all objects have been successfully imported followed by being asked if I want to save the import steps.
There are no error messages, but the number of rows in my MS access version of the table is around 250,000.
Other info...
I'm using MS office 365 version 1710
I'm using postgres 9.5.7
I'm using the PostgrSQL ANSI ODBC driver (not sure which version)
There are no signs of any error messages (or warnings).
After the import the Access database is still only 375 Mbytes, well short of the 2 Gbyte limit.
I've checked the 'ODBC data sources' app to check how the postgres ODBC link is configured, there's no obvious problem with it.
The final message that MS access gives me after the import includes 'all objects imported without errors'
There is no obvious difference between the records that are getting through and those that aren't.
Why am I losing records, and what can I do to cure it?
Thanks
If you attempt to "slurp" all records from the database at one time, the ODBC driver will stop fetching at some point and just return what it has without warning. It's annoying. As far as I know this has nothing to do with the 32-bit limit.
The way to solve this is to not fetch all records at once, but use the declare/fetch option on the driver. You can do this permanently on the ODBC settings by going to your ODBC properties, selecting "Datasource" and then on "Page 2" checking the "Use Declare/Fetch" and setting your cache (# of rows) size. I recommend a number somewhere between 5,000 and 50,000. Each batch represents a hit to the database, so you want it to be reasonably large to begin with.
From all practical purposes, the use of declare/fetch will be totally transparent to your application. You won't even notice. You will on the database admin side, but if your fetch size is sufficiently large, it won't be an issue.
You can also do one-time edits to your connection string from your particular query. You would add the following to make this work:
UseDeclareFetch=1;Fetch=25000;
I have been given an Access Database that I have to try to decipher what it is doing.
As a start I see that there is a Pass Through query with a command like:
Exec RefreshGLTableLatestEntries
#sourceDB = 'DB_NAME' ,
#tablePrefix = 'TableName$' ,
#logFile = 'C:\logDB.txt'
When I run it I will get something like:
Result
Success... 108 rows inserted with a total amount of $0.000000
What I don't understand is where are the rows being copied from or copied to.
In the MSSQL database I don't see a table, query, standard procedure or function called 'TableName$'. There are quite a few tables & queries called 'TableName$SomethingElse'. Is there a way to see more details on where is the data coming from?
Similarly, how can I see where are the rows being inserted to? I can not find any file named 'logDB.txt' in my hard disk to see the log. I would suspect that it might not say much more that '...108 rows insterted...'
I'm using:
Access 2016 from Office 365, Version 1609
MS SQL Server Management Studio v17.1
Any ideas on how to get more information on how to get more information on what the Pass Through do?
A Pass-Through query in Access is equivalent to running its SQL code in SQL Server Management Studio.
(In the database that is designated by the connection string of the Pass-Through query.)
The SQL is sent as-is to MSSQL and run there.
RefreshGLTableLatestEntries is the stored procedure that is executed here. You need to locate and analyze it in SQL Server.
I have N number of tables in my database, which holds around 0.6 million records. I've created a SQL script which copies this data into same tables (basically it's a script to generate more data). I've tested the script it runs fine for small data (10k records). When I tried it to copy all data, it throws an error:
An error occurred while executing batch. Error message is: Error creating window handle.
1.What is the meaning of this error in SQL Server?
2.Does it has to do anything with my SQL in script, or is this cause of other component of SQL Server?
Handles are Windows tools to manage OS resources. When some app on your machine have memory leaks - you can run out of handles and this error occurs. Current state of handles can be seen in Task Manager (Handle Count)
As said in comments - it's a client side issue. For example large resultsets/query output to grid may end up to this error.
Solution: Reboot your PC, minimize the output of query. Also you can try to launch script via SQLCMD.
You can read more about it here.
Some explanation here.
I connect from ms access (frontend) to MS SQL server (database) using ODBC and file datasource. I've created MS Access pass-trough query collecting data from SQL Server (directly calls MS SQL Server function returning table).
When I run this query in MS SQL Management console, it runs about 1s. When I run it from MS Access, using ODBC connection string, the same query runs about 5 minutes (!!). But when I clear connection string from query properties, it runs between 1 and 2 seconds (so acceptable).
Other queries and linked tables using the same connection string works correctly (so the string is rather OK)
Why the hell this one makes problem? Any idea hat to check?
I know this is an old post but... I've been googling to no avail so I wanted to put this out there...
I have a drill down SP that returns.... 200 rows max
The MS Access app uses ALL pass through queries to Azure. All work great except for this one. In SSMS it comes back almost instantly. ODBC string is the same on ALL Queries so the connection is fine.
After setting the max rows property to 1000, the form came back almost instantly. Cleared the property and it went back to slow...
This property definitely made the difference for me. Setting it to a number that is well beyond the upper limit really got this working as it should be.
hth,
..bob
I use windows authentication (trust) only.
"clear" means to remove all except "ODBC;" phrase. When I leave this notice in query properties, the Access asks for DSN, I select one, and then I got the results after 2 sek.
But When I put full connection string
ODBC;Description=My Database;DRIVER=SQL Server;SERVER=MYSERVER;APP=2007 Microsoft Office system;DATABASE=MYDB_Prod;LANGUAGE=polski;Trusted_Connection=Yes
The query I call is:
SELECT * FROM dbo.fn_MyFunction()
Function fn_MyFunction() is table valued one, returns two rows and about 50 columns ( suspected large number of column might be the issue, because other queries have less one), and it is used in access report (that's why I use pass through query - the same happens when I test with MS SQL server view linked to MS Access tablle via odbc)
The same connection string works well with other queries
It seems I found solution (or workaround rather?) I've made the tests using VPN over internet link. When I move to office - it works well and return almost immediatelly with the string I wrote above.
It is strange for me that link type have such impact: other queries works well on both (LAN an VPN over internet), but this one - only on LAN ( works over internet too but with unacceptable performance). I could understand if all works slow.. by why this one? Any idea?
This is probably going to be an underspecified question, as I'm not looking for a specific fix:
I want to run a machine learning algorithm on some data in a SQL Server database. I'd like to use R to do the calculations -- which would involve using R to connect to the database, process the data, and write a table of results back to the database.
Is this possible? My guess is yes. Shouldn't be a problem using a client...
however, would it be possible to set this up on a linux box as a cron job?
Yes to all!
Your choices for scripting are either Rscript or littler as discussed in this previous post.
Having struggled with connecting to MSSQL databases from Linux, my recommendation is to use RJDBC for database connections to MSSQL. I used RODBC to connect from Windows but I was never able to get it working properly in Linux. To get RJDBC working you will need to have Java installed properly on your Linux box and may need to change some environment variables (seems I always have SOMETHING mis-configured with rJava). You will also need to download and install the JDBC drivers for Linux which you can get directly from Microsoft.
Once you get RJDBC installed and the drivers installed, the code for pulling data from the database will look something like the following template:
require(RJDBC)
drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver",
"/etc/sqljdbc_2.0/sqljdbc4.jar")
conn <- dbConnect(drv, "jdbc:sqlserver://mySqlServer", "userId", "Password")
sqlText <- paste("
SELECT *
FROM SomeTable
;")
myData <- dbGetQuery(conn, sqlText)
You can write a table with something like
dbWriteTable(conn, "myData", SomeTable, overwrite=TRUE)
When I do updates to my DB I generally use dbWriteTable() to create a temporary table on my database server then I issue a dbSendUpdate() that appends the temp table to my main table then a second dbSendUpdate() that drops the temporary table. You might find that pattern useful.
The only "gotcha" I ran into was that I could never get a Windows domain/username to work in the connection sequence. I had to set up an individual SQL Server account (like sa).
You may just write a script containing R code and put this in the first line:
#!/usr/bin/env Rscript
change the file permissions to allow execution and put in crontab as it would be a bash script.