Asynchronous cursor execution in Snowflake - snowflake-cloud-data-platform

(Submitting on behalf of a Snowflake user)
At the time of query execution on Snowflake, I need its query id. So I am using following code snippet:
cursor.execute(query, _no_results=True)
query_id = cursor.sfqid
cursor.query_result(query_id)
This code snippet working fine for small running queries. But for query which takes more than 40-45 seconds to execute, query_result function fails with KeyError u'rowtype'.
Stack trace:
File "snowflake/connector/cursor.py", line 631, in query_result
self._init_result_and_meta(data, _use_ijson)
File "snowflake/connector/cursor.py", line 591, in _init_result_and_meta
for column in data[u'rowtype']:
KeyError: u'rowtype'
Why would this error occur? How to solve this problem?
Any recommendations? Thanks!

The Snowflake Python Connector allows for async SQL execution by using ​cur.execute(sql, _no_results=True)​
This ​"fire and forget"​ style of SQL execution allows for the parent process to continue without waiting for the SQL command to complete (think long-running SQL that may time-out).
If this is used, many developers will write code that captures the unique Snowflake Query ID (like you have in your code) and then use that Query ID to ​"check back on the query status later"​, in some sort of looping process. When you check back to get the status, you can then get the results from that query_id using the result_scan( ) function.
https://docs.snowflake.net/manuals/sql-reference/functions/result_scan.html
I hope this helps...Rich

Related

using servicestack ormlite, is there a way to get an execution plan?

Using servicestack ormlite 6,4 and azure SQL server - using SQLServerDialect2012, we have an issue with an enums causing excessive stalling and timeouts.
If we just convert it to a string its quick as it should be.
var results = db.Select(q => q.SomeColumn == enum.value); -> 3,5 seconds
var results2 = db.Select(q => q.SomeColumn.tostring() == enum.value.tostring()); -> 0,08
we are using default settings so the enum in the db is defined as a varchar(255)
both queries give the same result.
to track the issue we wanted to see what its actually firing, but all we get is a query with some #1 #2 etc with no indication of what parameters where used or how they are defined.
All our attempts to get a 1:1 SQL string we can use to manually test the query and see the results have failed... mini profiler was the closest as it shows the parameter values...
but it does not contain the details necessary to recreate the used query and recreate the issue we have. (manually recreating the query gives 80ms as above)
Trying to get the execution plan with the query also fail.
db.ExecuteSql("SET STATISTICS PROFILE ON;");
var results = db.Select(q => q.SomeColumn == enum.value);
db.ExecuteSql("SET STATISTICS PROFILE OFF;");
only returns data, not any extra info i was hoping for.
I have not been able to find any sites or threads that explain how others get any kind of debug info.
What is the correct next step here?
OrmLite's Logging & Introspection page shows how you can view the SQL generated by OrmLite.
E.g. Configuring a debug logger should log the generated SQL and params:
LogManager.LogFactory = new ConsoleLogFactory(debugEnabled:true);

Power BI SFDC Object Connection - Task - Navigation Query Error

I am hoping to get some help with the following error when I try to connect to the Task object in Salesforce. It is showing at the Navigation query (Task1 below) and I am unclear as to the nature of the error or resolution.
DataSource.Error: exceeded 100000 distinct who/what's
Details:
List
Query:
let
Source = Salesforce.Data(),
Task1 = Source{[Name="Task"]}[Data]
in
Task1
The task volume in our SFDC instance is excessive, though I have previously been applying a date filter after the Navigation query (Task1 above) which is where I am currently experiencing the error, thus precluding me from using a date filter as I had been with other objects.
Thanks,
Rich
It is a permission problem/limitation. You can find the solution here.

SOQL requesting VersionData from ContentVersion causes drama

Executing the following query
"Select C.Title FROM ContentVersion C WHERE ContentDocumentId IN (SELECT ContentDocumentId FROM ContentWorkspaceDoc WHERE ContentWorkSpaceId='".LIBRARY_ID."')"
which gives me a big list of files in the library with Id LIBRARY_ID
As soon as I add
"Select C.Title,C.VersionData FROM..."
I only get one record. Only one of 8 records is a link, so what foolishness am I performing to get this undesired behavior?
Does VersionData require some additional privileges?
When using the API, you may receive fewer than the default 500 records in a QueryResult if the rows are wide, which they will be when retrieving the base64-encoded content stored in VersionData. You should check the done property, and call queryMore with the queryLocator to get more rows. See http://bit.ly/KEEo7M.
I had the same issue. I'd suggest you to run this same query in Anonymous window in developer console and you'll surely run into heap size error.
This happens when you exceed the governor limit ,which is 6mb for synchronous process. VersionData consists binary representation of file available in contentversion.
If you are using this query in your apex code, I'd suggest you to use batch apex for this one, only if the file size in versionData is < 12mb.

SOQL - Convert Date To Owner Locale

We use the DBAmp for integrating Salesforce.com with SQL Server (which basically adds a linked server), and are running queries against our SF data using OPENQUERY.
I'm trying to do some reporting against opportunities and want to return the created date of the opportunity in the opportunity owners local date time (i.e. the date time the user will see in salesforce).
Our dbamp configuration forces the dates to be UTC.
I stumbled across a date function (in the Salesforce documentation) that I thought might be some help, but I get an error when I try an use it so can't prove it, below is the example useage for the convertTimezone function:
SELECT HOUR_IN_DAY(convertTimezone(CreatedDate)), SUM(Amount)
FROM Opportunity
GROUP BY HOUR_IN_DAY(convertTimezone(CreatedDate))
Below is the error returned:
OLE DB provider "DBAmp.DBAmp" for linked server "SALESFORCE" returned message "Error 13005 : Error translating SQL statement: line 1:37: expecting "from", found '('".
Msg 7350, Level 16, State 2, Line 1
Cannot get the column information from OLE DB provider "DBAmp.DBAmp" for linked server "SALESFORCE".
Can you not use SOQL functions in OPENQUERY as below?
SELECT
*
FROM
OPENQUERY(SALESFORCE,'
SELECT HOUR_IN_DAY(convertTimezone(CreatedDate)), SUM(Amount)
FROM Opportunity
GROUP BY HOUR_IN_DAY(convertTimezone(CreatedDate))')
UPDATE:
I've just had some correspondence with Bill Emerson (I believe he is the creator of the DBAmp Integration Tool):
You should be able to use SOQL functions so I am not sure why you are
getting the parsing failure. I'll setup a test case and report back.
I'll update the post again when I hear back. Thanks
A new version of DBAmp (2.14.4) has just been released that fixes the issue with using ConvertTimezone in openquery.
Version 2.14.4
Code modified for better memory utilization
Added support for API 24.0 (SPRING 12)
Fixed issue with embedded question marks in string literals
Fixed issue with using ConvertTimezone in openquery
Fixed issue with "Invalid Numeric" when using aggregate functions in openquery
I'm fairly sure that because DBAmp uses SQL and not SOQL, SOQL functions would not be available, sorry.
You would need to expose this data some other way. Perhaps it's possible with a Salesforce report, web-service, or compiling the data through the program you are using to access the (DBAmp) SQL Server.
If you were to create a Salesforce web service, the following example might be helpful.
global class MyWebService
{
webservice static AggregateResult MyWebServiceMethod()
{
AggregateResult ar = [
SELECT
HOUR_IN_DAY(convertTimezone(CreatedDate)) Hour,
SUM(Amount) Amount
FROM Opportunity
GROUP BY HOUR_IN_DAY(convertTimezone(CreatedDate))];
system.debug(ar);
return ar;
}
}

List of SQL Server errors that should be retried?

Is there a concise list of SQL Server stored procedure errors that make sense to automatically retry? Obviously, retrying a "login failed" error doesn't make sense, but retrying "timeout" does. I'm thinking it might be easier to specify which errors to retry than to specify which errors not to retry.
So, besides "timeout" errors, what other errors would be good candidates for automatic retrying?
Thanks!
You should retry (re-run) the entire transaction, not just a single query/SP.
As for the errors to retry, I've been using the following list:
DeadlockVictim = 1205,
SnapshotUpdateConflict = 3960,
// I haven't encountered the following 4 errors in practice
// so I've removed these from my own code:
LockRequestTimeout = 1222,
OutOfMemory = 701,
OutOfLocks = 1204,
TimeoutWaitingForMemoryResource = 8645,
The most important one is of course the "deadlock victim" error 1205.
I would extend that list, if you want absolutely complete list use the query and filter the result.
select * from master.dbo.sysmessages where description like '%memory%'
int[] errorNums = new int[]
{
701, // Out of Memory
1204, // Lock Issue
1205, // Deadlock Victim
1222, // Lock request time out period exceeded.
7214, // Remote procedure time out of %d seconds exceeded. Remote procedure '%.*ls' is canceled.
7604, // Full-text operation failed due to a time out.
7618, // %d is not a valid value for a full-text connection time out.
8628, // A time out occurred while waiting to optimize the query. Rerun the query.
8645, // A time out occurred while waiting for memory resources to execute the query. Rerun the query.
8651, // Low memory condition
};
You can use a SQL query to look for errors explicitly requesting a retry (trying to exclude those that require another action too).
SELECT error, description
FROM master.dbo.sysmessages
WHERE msglangid = 1033
AND (description LIKE '%try%later.' OR description LIKE '%. rerun the%')
AND description NOT LIKE '%resolve%'
AND description NOT LIKE '%and try%'
AND description NOT LIKE '%and retry%'
Here's the list of error codes:
539,
617,
952,
956,
983,
1205,
1807,
3055,
5034,
5059,
5061,
5065,
8628,
8645,
8675,
10922,
14258,
20689,
25003,
27118,
30024,
30026,
30085,
33115,
33116,
40602,
40642,
40648
You can tweak the query to look for other conditions like timeouts or memory problems, but I'd recommend configuring your timeout length correctly up front, and then backing off slightly in these scenarios.
I'm not sure about a full listing of these errors, but I can warn you to be VERY careful about retrying queries. Often there's a larger problem afoot when you get errors from SQL, and simply re-running queries will only further compact the issue. For instance, with the timeout error, you typically will have either a network bottleneck, poorly indexed tables, or something on those lines, and re-running the same query will add to the latency of other queries already obviously struggling to execute.
The one sql server error that you should always catch on inserts and updates (and it is quite often missed), is the deadlock error no. 1205
Appropriate action is to retry the INSERT/UPDATE a small number of times.

Resources