how can I set warehouse RM notification for every 10% of credit used in snowflake. I have checked the settings which are already there but I did not found specific settings to configure alert notification which can give notifications on every 10% of total credit.
You can only get five notification triggers, so you'd probably want to set at 20% increments like this:
use role ACCOUNTADMIN;
CREATE RESOURCE MONITOR "MY_RESOURCE_MONITOR" WITH CREDIT_QUOTA = 1000
TRIGGERS
ON 20 PERCENT DO NOTIFY
ON 40 PERCENT DO NOTIFY
ON 60 PERCENT DO NOTIFY
ON 80 PERCENT DO NOTIFY
ON 100 PERCENT DO NOTIFY
;
ALTER WAREHOUSE "TEST" SET RESOURCE_MONITOR = "MY_RESOURCE_MONITOR";
use role SYSADMIN;
Related
We want to enable our users to get insights from the data. We have been using Tableau as a self-service BI platform. Our dashboard users have a specific request; they want to see data within a particular time period
Below is how my dataset looks like (dates are mm/dd/yy).
User request - To see how many customers were subscribed within a time period regardless of their current status. i.e. even if their current status is cancelled as long as they were active during the user-provided time period they should be counted
Example - User selects time range to be 01/01/2020 - 03/31/2020. Running a query on below data set should return count of 3. [CUST1 as they cancelled after 03/31/2020, CUST3 as they signed up before 01/01/2020 but are still active, CUST5 as they were active for some point during that period]
Problem - While I can write a SQL query with abundant where clauses to achieve this, we want a self-service automated way i.e. we want users to just provide us the time range and get the number. How do we achieve this in a BI tool like Tableau? I am also open to other tools, changing the data model design and other options. The goal is to just make this automated rather than having a person manually update and run a SQL query
Customer ID
Subscription Start Date
Subscription End Date
Subscription Status
CUST1
10/11/2019
04/12/2020
Cancelled
CUST2
01/12/2020
Active
CUST3
05/01/2019
Active
CUST4
06/07/2012
07/08/2012
Cancelled
CUST5
01/12/2020
03/14/2020
Cancelled
CUST6
04/12/2020
Active
I am using Flink-SQL 1.13. The target is to calculate number of new users in real-time.
Due to some constraints, I cannot directly use register events because the accounts created there like a platform pass. One account can login into multiple games and for each game, the user is new when it firstly enter that game. So I can only calculate this by checking whether this account has logined to this game before from the login log. The format of login log is like:
user_id game_id login_time
111 game1 2021-05-13 01:01:01
111 game3 2021-05-23 02:02:02
The question is the amount of login log increases significantly every day. Although I can save the log into HBase, one day it will still be too large...
Is there any other way to do this? Maybe I can put historical users into redis hyperloglog, but it seems Flink-SQL does not have a redis connector yet...Thanks for your help in advance...
INSERT INTO first_login_stream (user_id, first_login_time)
SELECT
user_id,
FIRST_VALUE(login_time) first_login_time
FROM login_log
GROUP BY user_id
Which goes back into your event system / kafka. Which you can read back in windows for some hourly stats (which you can save in HBase):
INSERT INTO hbase_stats
SELECT
window_start,
window_end,
count(user_id) user_count
FROM TABLE(
TUMBLE(
TABLE first_login_stream,
DESCRIPTOR(<kafka_ingestion_time>),
INTERVAL '1' HOUR
)
)
GROUP BY
window_start,
window_end
It has to be checkpointed/saved (otherwise you'll incur the full log processing on restart). The state size will only grow by the number or users and not logins (I think. You should validate that.).
How to alert long running queries, to multiple users in snowflake ?
Right now the alert is sent only to the account admin role user.
Is there any way to notify the long query alert to "the user running the query OR notify to
multiple users belong to the particular warehouse/database" ?
Is there any way to leverage Snowflake Notification Integration for the above alerts?
Thanks In Advance
Sundar
It is possible to fulfill such requirement by using alerts and email notifications.
Setting Up Alerts Based on Data in Snowflake:
In some cases, you might want to be notified or take action when data in Snowflake meets certain conditions. For example, you might want to receive a notification when:
The warehouse credit usage increases by a specified percentage of your current quota.
The resource consumption for your pipelines, tasks, materialized views, etc. increases beyond a specified amount.
A data access request is received from an unauthorized user.
Your data fails to comply with a particular business rule that you have set up.
To do this, you can set up a Snowflake alert. A Snowflake alert is a schema-level object that specifies:
A condition that triggers the alert (e.g. the presence of queries that take longer than a second to complete).
The action to perform when the condition is met (e.g. send an email notification, capture some data in a table, etc.).
When and how often the condition should be evaluated (e.g. every 24 hours, every Sunday at midnight, etc.).
Sample:
CREATE OR REPLACE ALERT alert_long_queries
WAREHOUSE = my_warehouse_name
SCHEDULE = '5 MINUTE'
IF (EXISTS (
SELECT *
FROM TABLE(SNOWFLAKE.INFORMATION_SCHEMA.QUERY_HISTORY())
WHERE EXECUTION_STATUS ILIKE 'RUNNING'
AND start_time < current_timestamp() - INTERVAL '5 MINUTES'
))
THEN CALL SYSTEM$SEND_EMAIL(...);
The only notification available out-of-the-box in Snowflake is the Resource Monitor whereby AccountAdmin members only can subscribe for notifications.
https://docs.snowflake.com/en/user-guide/resource-monitors.html#resource-monitor-properties
We know that we have "show transactions" to see the transactions currently connected to database.
But I am interested
- To get the count of active users for each warehouse?
-History of connections count for each warehouse?
Is there a way to get above information using the sql commands (not the web ui)
If I understood correctly, you want to see the warehouse and active user mapping. There is no direct views as per my knowledge but you can leverage provided query where by keeping warehouse size !='0' you can tied warehouse and user together. You can check the below link
https://docs.snowflake.com/en/sql-reference/account-usage/query_history.html
Before that
Snowflake Sessions are not tagged with user name or account , those are system
generated ID.
User and warehouse relationship is zero or many (An active user can use multiple warehouse in parallel , also a warehouse can be used by multiple users at same point of time)
A user can have active session without a running warehouse
It is not mandatory to have an active user to keep your warehouse running
Finally, queries can also be executed without turning the warehouse up
SELECT TO_CHAR(DATE_TRUNC('minute', query_history.START_TIME ),'YYYY-MM-DD
HH24:MI') AS "query_history.start_time",
query_history.WAREHOUSE_NAME AS "query_history.warehouse_name",
query_history.USER_NAME AS "query_history.user_name"
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY AS query_history
WHERE (query_history.WAREHOUSE_SIZE != '0')
GROUP BY DATE_TRUNC('minute', query_history.START_TIME ),2,3
ORDER BY 1 DESC
Note : Above SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY view refresh has latency of 45 minutes
I am trying to set up a Data Share with an expiration date, I am not sure how to do that. So far here is how I set it up:
//create resource monitor
CREATE RESOURCE MONITOR GPDATA WITH CREDIT_QUOTA = 100
triggers on 75 percent do notify
on 100 percent do suspend
on 110 percent do suspend_immediate;
//Create Warehouse
create warehouse "MarketshareWH"
AUTO_SUSPEND = TRUE
RESOURCE_MONITOR = GPDATA
COMMENT = 'Shared market data lies here';
create database market_shared_db from share bd56789.share;
The Warehouse auto-suspends and has a resource monitor, can I set an expiration on the share or should I create a task to remove grants on a share?
Thank you.
You have a few options, but the closest thing you can do to expiring a data share is dropping it. You can set up a task for when it's supposed to expire and just
DROP SHARE <name>
The Resource Monitor does not have an option to expire; however, even if it did that's not where you'd want to set an expiration. The reason is the users would still have access to the shared account, even though they couldn't run anything requiring a running warehouse.
If you do drop the share, you'll lose any user logins set up there. If you'd rather keep that for potential future use, your best option is to revoke the grants.