I have been looking through the documentation for both the General Statistics and Advanced Statistics - but it seems only aggregated statistics are available. From what I am able to find, the most detailed information is available for a single day within a single category. But is it possible to retrieve statistics for a single marketing email, using e.g. it's name as parameter? Or is it necessary to use the Event Webhook, to store all events (opens, clicks, etc.) in my end, and do all the calculations myself?
Thank you
brgds
Lukas
At this time, it's necessary to use the Event Webhook. Using Event Webhook also allows you to implement unique_args, for more detailed & granular stats. You can use as many args per message as you'd like, naming the keys and values to whatever makes sense for you.
Related
In Snowflake SnowSight UI, in the Query Profile view, there is a section called Profile Overview where you can see the breakdown of the total execution time. It contains statistics like Processing, Local Disk I/O, Remote Disk I/O, Synchronization etc.
Full list here
https://docs.snowflake.com/en/user-guide/ui-snowsight-activity.html#profile-overview
I want to access those statistics programmatically instead of having to navigate to that section for each query that I want to analyze. The only system view I know that provides query statistics is the QUERY_HISTORY however it doesn't contain those stats.
https://docs.snowflake.com/en/sql-reference/account-usage/query_history.html
Question is, can I get those stats in any of the system views? If so, where and how?
It is possible to programmatically access query profile using GET_QUERY_OPERATOR_STATS
Returns statistics about individual query operators within a query. You can run this function for any query that was executed in the past 14 days.
For example, you can use this information to determine which operators are consuming the most resources. As another example, you can use this function to identify joins that have more output rows than input rows, which can be a sign of an “exploding” join (e.g. an unintended Cartesian product).
These statistics are also available in the query profile tab in Snowsight. The GET_QUERY_OPERATOR_STATS() function makes the same information available via a programmatic interface.
The GET_QUERY_OPERATOR_STATS function is a table function. It returns rows with statistics about each query operator in the query
set query_id = '<query_ud>';
select *
from table(get_query_operator_stats($query_id));
2023 update: GET_QUERY_OPERATOR_STATS()
See https://stackoverflow.com/a/74824120/132438 with Lukasz answer.
https://docs.snowflake.com/en/sql-reference/functions/get_query_operator_stats.html
Bad news: There's no programmatic way to get this.
Good news: This is a frequent request, so we might eventually have news.
In the internal tracker I left a note to update this answer once there is progress we can report.
You can do it via https://github.com/Snowflake-Labs/sfsnowsightextensions#get-sfqueryprofile. Doing it at scale (scraping-style) will likely yield ~60%-80% success rate. Please don't abuse it.
Inspired by a clever customer who did that to get what is now offered by https://docs.snowflake.com/en/sql-reference/account-usage/access_history.html
Completely unsupported as it says so on the repo homepage.
Just FYI, there is an upcoming feature called GET_QUERY_STATS (currently in private preview) https://docs.snowflake.com/en/LIMITEDACCESS/get_query_stats.html that will do just this and obviate the reason for Get-SFQueryProfile once it ships.
I wish to create a generic component which can save the Object Name and field Names with old and new values in a BigObject.
The brute force algo says, on every update of each object, get field API names using describe and check old and new value of those fields. If it gets modified insert it into new BigObject.
But it will consume a lot of CPU time and I am looking for an optimum solution to handle this.
Any suggestions are appreciated.
Well, do you have any code written already? Maybe benchmark it and then see what you can optimise instead of overdesigning it from the start... Keep it simple, write test harness and then try to optimise (without breaking unit tests).
Couple random ideas:
You'd be doing that in a trigger? So your "describe" could happen only once. You don't need to describe every single field, you need only one operation outside of trigger's main loop.
Set<String> fieldNames = Account.sObjectType.getDescribe().fields.getMap().keyset();
System.debug(fieldNames);
This will get you "only" field names but that's enough. You don't care whether they're picklists or dates or what. Use that with generic sObject.get('fieldNameHere') and it's a good start.
or maybe without describe at all. sObject's getPopulatedFieldsAsMap() will give you cool Map which you can easily iterate & compare.
or JSON.serialize the old & new version of the object and if they aren't identical - you know what to do. No idea if they'll always serialise with same field order though so checking if the maps are identical might be better
do you really need to hand-craft this field history tracking like that? You have 1M records free storage but it could explode really easily in busier SF org. Especially if you have workflows, processes, other triggers that would translate to multiple updates (= multiple trigger runs) in same transaction. Perhaps normal field history tracking + chatter feed tracking + even salesforce shield (it comes with 60 more fields tracked I think) would be more sensible for your business needs.
I have a table with 1000 rows. Each row represents a prompt text for an application. For the start I only want to translate the most used 20% of the promts. In the daily use some dialogs appear more often than others. So the prompt texts for the most displayed dialogs get fetched more often than the others.
However, it looks to me like there is no built-in mechanism to analyse the data by their select rates.
There are no triggers on select. There is no way to filter the data in the profiler. There is no way to filter data in an Audit. Is that true?
Are there any options to do that inside the SQL Server?
No. There is no way to track the frequency of how often data is selected.
This sounds like application metrics. You will have to write metrics logic yourself.
For example, you might create a table of MissingTranslations that tracks the frequency of requests. If your application detects a missing translation, insert a row into this table with a frequency of 1, or increment the counter if it already exists in the table.
You could then write another application that sorts the missing translations by frequency descending. When a user enters the translation, the translation app removes the entry from the list of missing translations or marks it as complete.
All that being said, you could abuse some SQL Server features to get some information. For example, a stored procedure that returns these translations could generate a user-configurable trace event with the translation info. A SQL Profiler session could listen for these events and write them to a table. This would get you a basic frequency.
It might be possible to get the same information from implementing auditing and then calling sys.fn_get_audit_file, but that sounds cumbersome at best.
In my opinion, it sounds easier and more stable to me to write this logic yourself.
#TabAlleman: "no, there's nothing you can do"
I want to write a calendar application. It is really recurring items that throw a wrench in the works for the DB schema. I would love some input on how to organize this.
What if a user creates an event, and inputs that it repeats everyone Monday, forever? How could I store all that in the database? I can't create infinite events. Do I simply put a table in there that holds the relevant info so I can calculate where all the events go? If so, I would have to calculate them every time the user views a new part of the calendar. What if they page through the months, but they have a ton of recurring items?
Also, the schema needs to handle when a user clicks an item and says "Edit this one in the sequence" not all items in the sequence. Do I then split the one item off of the sequence?
Update 1
I have not looked at iCal at all. To be clear, I think saving the info that allows you to calculate the recurring items, and splitting off any that differ from the sequence is a great way to store it to be able to transfer it. But I think that in an application, this would be too slow, to do the date math all over the place.
I have been struggling with the same problem, and I was actually toying with the "cache table" idea suggested above, but then I came across an alternative (suggested here) that doesn't seem to have been represented yet.
Build a table containing all events
EventID (primary key)
Description
StartDate
PeriodType - days, weeks, months, years
PeriodFreq - # of days, weeks, etc between events
EndDate
... other attributes that can be modified
Then add a table for exceptions to these events. This table uses a composite key, made up of the EventID that maps to the event table, and an instance ID to pick the particular event in the series.
EventID (key)
InstanceID (key)
InstanceDate - the modified date of the exception
IsCancelled - a flag to skip this date when traversing the series
... other attributes that can be modified
It seems to keep the event table normalised, and avoids splitting up series to handle exceptions.
I recently created a calendar application and this was one of the many challenges that I faced.
I eventually came up with a semi hack-ish solution. I created an event_type column. In that column, I had either: daily, weekly, monthly, or yearly. I also had a start_date and an end_date columns. Everything else was handled in the actual backend code.
I never tried to split an event if a user edited only one event. It wasn't necessary in the situation. However, you could split an event by changing the end_date of the first, creating a new event with a new start_date and the end_date of the original, and finally, a new event for the one you just chose to edit. This process would end up creating 3 events.
Hack-ish, I know. I couldn't think of a clever way to handle this problem at the time.
Hold the recurring item in the events table as normal, but flagged as recurring with the appropriate start/ end dates.
If the user modifies a single instance of the appointment, just create a new event, perhaps with a 'parentId' equal to the recurring event's id.
Build logic that causes the calendar to override any recurring events on a particular day with events with matching parent IDs.
Your question about performance is basically the old speed vs. storage issue. I really don't think the calculation required would exceed the space requirement for storing so many appointments. Just read up on database optimization- indexing etc.
Could you bridge the two worlds with a "cache" table, in which you pre-compute the next X days worth of events?
So three tables:
recurring_event_specs
one_time_events
cached_recurring_events
For any part of the calendar within X days of today, your query will UNION one_time_events and cached_recurring_events.
Then you would only have to do on-the-fly date calculations if the user tried to look at a part of the calendar more than X days in the future. I imagine you could find a sane X that would cover the majority of normal use.
The cached_recurring_events table would need to be updated whenever a user adds a new recurring event -- and possibly once a day offline, by a cron-job/scheduled-task. But only on days when no new recurring event has been created.
Why not use Google Calendar as a database for this calendar application by relying on Google Calendar's API for storing and retrieving calendar events?
The Calendar API is a REST API that can be accessed through explicit HTTP calls; the API exposes most of the features available in the Google Calendar Web interface, so your calendar application can as much functionality as Google Calendar does (a lot of functionality!!!).
Your application need only implement OAuth 2.0 for Google APIs, which can be made easy using a single sign-on service like Auth0 to provide the appropriate access tokens. Then, your calendar application can use these tokens in conjunction with the Calendar API to provide seamless storage and retrieval of calendar events in a JSON format.
Users create events within their own "New Calendar." This calendar is shared with you in the form of a gmail account dedicated to this application - the application's gmail account.
Basically, Google Calendar becomes your database, whereby you can have the application's gmail account not only store all of your application's events, but also allow you to view and edit these events with an intuitive interface.
The best way to do this is to store a standards based recurrence pattern string (iCal).. and leave blank if it's a single event. There are a few APIs that can parse the recurrence pattern and create event objects that you can bind to UI elements.... none of the occurrences need ever be stored in the database, only the initial event (occurrence)..
Couldn't you store the events per day with start and end time? It will generate a lot of data for events that happen everyday (maybe go non-relational for this) but it will make querying easier and it will be possible to make exceptions (f.e. the event place burned down, or employees are striking). To generate the days for the event I would suggest to implement that in the front-end derived on some ICal-ish pattern.
I'm working on a notification feed for my mobile app and am looking for some help on an issue.
The app is a Twitter/Facebook like app where users can post statuses and other users can like, comment, or subscribe to them.
One thing I want to have in my app is to have a notifications feed where users can see who liked/comment on their post or subscribed to them.
The first part of this system I have figured out, when a user likes/comments/subscribes, a Notification entity will be written to the datastore with details about the event. To show a users Notification's all I have to do is query for all Notification's for that user, sort by date created desc and we have a nice little feed of actions other users took on a specific users account.
The issue I have is what to do when someone unlikes a post, unsubscribes or deletes a comment. Currently, if I were to query for that specific notification, it is possible that nothing would return from the datastore because of eventual consistency. We could imagine someone liking, then immediate unliking a post (b/c who hasn't done that? =P). The query to find that Notification might return null and nothing would get deleted when calling ofy().delete().entity(notification).now(); And now the user has a notification in their feed saying Sally liked his post when in reality she liked then quickly unliked it!
A wrench in this whole system is that I cannot delete by Key<Notification>, because I don't really have a way to know id of the Notification when trying to delete it.
A potential solution I am experimenting with is to not delete any Notifications. Instead I would always write Notification's and simply indicate if the notification was positive or negative. Then in my query to display notifications to a specific user, I could somehow only display the sum-positive Notification's. This would save some money on datastore too because deleting entities is expensive.
There are three main ways I've solved this problem before:
deterministic key
for example
{user-Id}-{post-id}-{liked-by} for likes
{user-id}-{post-id}-{comment-by}-{comment-index} for comments
This will work for most basic use cases for the problem you defined, but you'll have some hairy edge cases to figure out (like managing indexes of comments as they get edited and deleted). This will allow get and delete by key
parallel data structures
The idea here is to create more than one entity at a time in a transaction, but to make sure they have related keys. For example, when someone comments on a feed item, create a Comment entity, then create a CommentedOn entity which has the same ID, but make it have a parent key of the commenter user.
Then, you can make a strongly consistent query for the CommentedOn, and use the same id to do a get by key on the Comment. You can also just store a key, rather than having matching IDs if that's too hard. Having matching IDs in practice was easier each time I did this.
The main limitation of this approach is that you're effectively creating an index yourself out of entities, and while this can give you strongly consistent queries where you need them the throughput limitations of transactional writes can become harder to understand. You also need to manage state changes (like deletes) carefully.
State flags on entities
Assuming the Notification object just shows the user that something happened but links to another entity for the actual data, you could store a state flag (deleted, hidden, private etc) on that entity. Then listing your notifications would be a matter of loading the entities server side and filtering in code (or possibly subsequent filtered queries).
At the end of the day, the complexity of the solution should mirror the complexity of the problem. I would start with approach 3 then migrate to approach 2 when the fuller set of requirements is understood. It is a more robust and flexible approach, but complexity of XG transaction limitations will rear its head - but ultimately a distributed feed like this is a hard problem.
What I ended up doing and what worked for my specific model was that before creating a Notification Entity I would first allocate and ID for it:
// Allocate an ID for a Notification
final Key<Notification> notificationKey = factory().allocateId(Notification.class);
final Long notificationId = notificationKey.getId();
Then when creating my Like or Follow Entity, I would set the property Like.notificationId = notificationId; or Follow.notificationId = notificationId;
Then I would save both Entities.
Later, when I want to delete the Like or Follow I can do so and at the same time get the Id of the Notification, load the Notification by key (which is strongly consistent to do so), and delete it too.
Just another approach that may help someone =D