(My environment: Windows 7 x64 and Server 2008, EclipseLink 2.5.2, I've tried the following with JTDS and the MS JDBC driver, and with MS SQL Server Express 2008 and 2012.)
I'm doing a paginated query in a table with anywhere from 5-50 million rows, with about 5 million rows matching the filter criteria. The following is the paginated query I'm using to display this data in the UI 25 rows at a time. The pagination is working properly - the list contains just 25 rows for each page. However, the query takes 24 seconds to return 25 rows, which seems long.
My goal is to log the generated SQL, so I can see exactly how JPA is accomplishing the pagination in SQL Server 2008 vs 2012. But the generated SQL doesn't include anything to do with pagination, which makes me wonder what else I'm not seeing in the generated SQL.
The query:
CriteriaBuilder cb = JPA.em().getCriteriaBuilder();
CriteriaQuery<RGHICarrierPull> cq = cb.createQuery(RGHICarrierPull.class);
Root<RGHICarrierPull> from = cq.from(RGHICarrierPull.class);
CriteriaQuery<RGHICarrierPull> select = cq.select(from);
// Add filter/sort predicates to "predicates"
...
select.where(predicates);
// Apply pagination
records.setFirstResult((page-1)*limit);
records.setMaxResults(limit);
// Get data
List<RGHICarrierPull> lst = records.getResultList();
To log the generated SQL programatically:
// Log the sql for this query
Session session = JPA.em().unwrap(JpaEntityManager.class).getActiveSession();
DatabaseQuery databaseQuery = ((EJBQueryImpl)records).getDatabaseQuery();
databaseQuery.prepareCall(session, new DatabaseRecord());
System.out.println(databaseQuery.getSQLString());
The logged SQL:
SELECT t1.SHIPTO_ZIP, t1.WHSE, t1.ANYNBR1, t1.ANYTEXT1, t1.CREATE_DATE_TIME, t1.
MOD_DATE_TIME, t1.PULL_TIME, t1.PULL_TIME_AMPM, t1.PULL_TRLR_CODE, t1.USER_ID,
1.SHIP_VIA FROM Ship_Via t0, RGHI_Carrier_Pull t1 WHERE ((t1.WHSE = 'WHSE1') AND
(t0.SHIP_VIA = t1.SHIP_VIA)) ORDER BY t0.SHIP_VIA ASC, t1.SHIPTO_ZIP ASC
Obviously, this is not a paginated query, so if I run this query directly, it runs for over a minute and returns all 5 million rows. I get the same results if I use persistence.xml settings to log all JPA queries, and also if I log the SQL from MS SQL Server.
Is this the actual generated SQL? I see two possibilities:
This is the full generated SQL, but EclipseLink is doing something else to accomplish the pagination.
EclipseLink is logging this generated SQL prior to the pagination stuff being added to it
Try setting the log level in EclipseLink to Finest, and check the database platform that is being used. EclipseLink logging will also show what is sending to the database. This should log the same SQL as what you get from the getSQLString(), but allows you to validate you are executing the correct api, and the initial start up logging will show if the platform being used matches your database, otherwise it will need to be specified using the target-database property: http://www.eclipse.org/eclipselink/documentation/2.4/jpa/extensions/p_target_database.htm
EclipseLink will use pagination within the generated SQL as described here http://wiki.eclipse.org/EclipseLink/Examples/JPA/Pagination if the plaform supports it, otherwise it resorts to using JDBC api to limit the results sent across, and then jump to the first result in the returned resultset, which is less efficient then if pagination is done entirely in the database.
EclipseLink (at least up to the current version 2.6.1) supports neither SQL Server 2012 pagination in the form of OFFSET-FETCH syntax nor older SQL Server TOP syntax, check out EclipseLink Database Support. Instead EclipseLink internally uses JDBC feature Statement.setMaxRows() which essentially discards excessive rows from the returning ResultSet. And there is no plans to support that in the future versions so far.
You can try to implement this manually extending the SQLServerPlatform and overriding method printSQLSelectStatement() similarly to how it is done in PostgreSQLPlatform.printSQLSelectStatement(). The working prototype is here: https://github.com/roman-sinyakov/eclipselink/blob/master/SQLServer2012Platform.java.
Related
In our project we are using two databases, one postgres and one mssql. We are using Jooq in order to query these DBs and with postgres everything was pretty straight forward!
But with mssql we are facing some troubles. The task is to select the top 10 values and let's say that we have the following java code:
DSL.using(conn)
.select(USE_CASE.asterisk())
.from(USE_CASE)
.where(USE_CASE.RECORD_ACTIVE.eq(true))
.orderBy(USE_CASE.CREATED_ON.desc())
.limit(10)
.offset(0)
.fetch(new UseCaseMapper()))
This works like a charm with postgres but on mssql we get the following error:
Execution exception[[CompletionException: org.jooq.exception.DataAccessException:
SQL [select "park"."dbo"."use_case".* from "park"."dbo"."use_case" where
"park"."dbo"."use_case"."record_active" = ?
order by "park"."dbo"."use_case"."created_on" desc limit ?];
Incorrect syntax near 'limit'.]]
I know that for mssql the equivalent query would be something like,
select top 10 *
from use_case
where record_active = true
order by created_on desc;
How can I change my java code to get the limit records in mssql?
Correct SQLDialect
The exception message hints at the fact that you're still using the PostgreSQL (or some other non-SQL Server) dialect as your catalogs/schemas/tables/columns are quoted using "double_quotes", instead of [brackets].
Just use SQLDialect.SQLSERVER instead, when running your query on SQL Server. The LIMIT 10 syntax will then be correctly translated to TOP 10.
jOOQ Professional Edition vs jOOQ Open Source Edition
From our posted error messages, it is not quite clear how you really configured your jOOQ integration, but if you're using DSL.using(Connection) without explicitly specifying the SQLDialect, and jOOQ doesn't correctly "guess" the appropriate SQLDialect, this can mostly be because of one of two reasons:
You're using some non-standard JDBC URL, which jOOQ doesn't recognise. It thus uses SQLDialect.DEFAULT, instead, which produces the wrong SQL syntax for your.
You're using the jOOQ Open Source Edition (Maven groupId org.jooq) instead of the jOOQ Professional Edition (Maven groupId org.jooq.pro, to be installed manually, as it is not distributed via Maven Central), which would also lead to jOOQ using SQLDialect.DEFAULT.
Notice: this can also happen by accident, e.g. as a transitive dependency that is being pulled in via Spring Boot or some other framework that already depends on the jOOQ Open Source Edition. In that case, you have to make sure to exclude that transitive dependency from ending up on your classpath. In Maven, you can display all your dependencies using mvn dependency:tree
I have a Microsoft SQL database that we have been using for several years. Starting this morning a single table in the database is throwing a time-out error whenever we attempt to insert or update any records.
I have tried to insert and update through:
Microsoft Access ODBC
a .Net Program via Entity Framework
a stored procedure run as an automatic job -- that runs each morning
a custom query written this morning to test the database and executed through SQL Server Management Studio
Opening the table directly via 'Edit Top 200 Rows' and typing in the appropriate values
We have restarted the service, then restarted the entire server and continue to get the same problems. The remainder of the database appears to be working fine. All data can be read even from the affected table, and other tables allow updates and inserts to be run just fine.
Looking through the data in the table, I have not found anything that appears out of the ordinary.
I am at a loss as to the next steps on finding the cause or solution.
Its not a space issue is it ? try ...
SELECT volume_mount_point Drive,
cast(sum(available_bytes)*100 / sum(total_bytes) as int) as [Free%],
avg(available_bytes/1024/1024/1024) FreeGB
from sys.master_files f
cross apply sys.dm_os_volume_stats(f.database_id, f.[file_id])
group by volume_mount_point
order by volume_mount_point;
Our ColdFusion 2016 Enterprise server (Windows Server 2012 R2) is not maintaining connections to an Azure Data Warehouse. The first Azure query on a page takes a second or more to run. Subsequent Azure queries on the same page take a fraction of a second, e.g.:
test1 (Datasource=azureDev, **Time=3485ms**, Records=1) in D:\DW\dwtest\CF2016\bob\azureAdhoc.cfm # 12:10:12.012
select count(*) cnt from dimpatient where name like 'smith%' and birthdate >'2014-02-01'
test2 (Datasource=AzureDev, **Time=125ms**, Records=3) in D:\DW\dwtest\CF2016\bob\azureAdhoc.cfm # 12:10:12.012
select * from dbo.dimPatientMergeStatus
test3 (Datasource=azureDev, **Time=281ms**, Records=1) in D:\DW\dwtest\CF2016\bob\azureAdhoc.cfm # 12:10:13.013
select count(*) cnt from dimpatient where name like 'jones%' and birthdate >'2004-02-01'
It seems apparent that CF is taking extra time to actually make the connection while running the first query on the page. We've tried with various queries and re-arranging their order and always end up with the same result.
We are connecting to Azure using the latest MS jdbc driver (mssql-jdbc-6.2.2.jre8.jar) and 'Maintain Connections' is checked. We first attempted to connect using the built in Microsoft SQL Server driver but kept getting this error:
Connection verification failed for data source: AzureDev2
java.sql.SQLException: [Macromedia][SQLServer JDBC Driver]Error
fetching requested database meta-data info.
We do not see this issue when we run the queries in SSMS.
Any idea what might be wrong?
How does ColdFusion Server manage database connections when there are client variables?
With ColdFusion Server version 4.5.1 SP1 and higher, when you store
your client variables in a database, your code connects to the
database only when a variable is set. This prevents unnecessary
database connections, for instance, in a case where you are using
client management, but no client variables are present in a particular
request.
https://helpx.adobe.com/coldfusion/kb/database-connections-handled-coldfusion.html
I connect from ms access (frontend) to MS SQL server (database) using ODBC and file datasource. I've created MS Access pass-trough query collecting data from SQL Server (directly calls MS SQL Server function returning table).
When I run this query in MS SQL Management console, it runs about 1s. When I run it from MS Access, using ODBC connection string, the same query runs about 5 minutes (!!). But when I clear connection string from query properties, it runs between 1 and 2 seconds (so acceptable).
Other queries and linked tables using the same connection string works correctly (so the string is rather OK)
Why the hell this one makes problem? Any idea hat to check?
I know this is an old post but... I've been googling to no avail so I wanted to put this out there...
I have a drill down SP that returns.... 200 rows max
The MS Access app uses ALL pass through queries to Azure. All work great except for this one. In SSMS it comes back almost instantly. ODBC string is the same on ALL Queries so the connection is fine.
After setting the max rows property to 1000, the form came back almost instantly. Cleared the property and it went back to slow...
This property definitely made the difference for me. Setting it to a number that is well beyond the upper limit really got this working as it should be.
hth,
..bob
I use windows authentication (trust) only.
"clear" means to remove all except "ODBC;" phrase. When I leave this notice in query properties, the Access asks for DSN, I select one, and then I got the results after 2 sek.
But When I put full connection string
ODBC;Description=My Database;DRIVER=SQL Server;SERVER=MYSERVER;APP=2007 Microsoft Office system;DATABASE=MYDB_Prod;LANGUAGE=polski;Trusted_Connection=Yes
The query I call is:
SELECT * FROM dbo.fn_MyFunction()
Function fn_MyFunction() is table valued one, returns two rows and about 50 columns ( suspected large number of column might be the issue, because other queries have less one), and it is used in access report (that's why I use pass through query - the same happens when I test with MS SQL server view linked to MS Access tablle via odbc)
The same connection string works well with other queries
It seems I found solution (or workaround rather?) I've made the tests using VPN over internet link. When I move to office - it works well and return almost immediatelly with the string I wrote above.
It is strange for me that link type have such impact: other queries works well on both (LAN an VPN over internet), but this one - only on LAN ( works over internet too but with unacceptable performance). I could understand if all works slow.. by why this one? Any idea?
I got this error when trying to run a query in SQL server query analyzer.
What is the reason?
If you're using cursors, then you've probably closed the cursor. Check out MSDN for more information (http://support.microsoft.com/kb/151693)
If you're not using cursors, then it could be an issue with the server, check the log.
And before it's lost to the sands of time, the KB article:
Article ID: 151693 - Last Review: October 3, 2003 - Revision: 3.0
FIX: Err Msg "Unknown Token Received from SQL Server"
Symptoms
When you set the statement options to use a server-side cursor and prepare a select statement on a SQL Server system table, the first execution of the select creates the cursor successfully. After you close this cursor, if you execute the prepared statement again, the following error message appears:
unknown token received from SQL Server.
Workaround
Use a forward-only cursor instead of a static, keyset, or dynamic cursor on the system tables. Note that the problem does not occur if a server-side cursor is created on a user-defined table or view.
If a forward-only cursor is unacceptable, prepare the select statement again on the system table and execute it to create a server-side cursor. Note that the first execution works fine and applications typically do not need to create a cursor on a system table repeatedly.
Status
Microsoft has confirmed this to be a problem in Microsoft SQL Server version 6.50.0201. This problem has been corrected in U.S. Service Pack 1 for Microsoft SQL Server version 6.5. For more information, contact your primary support provider.