I need to perform series of action after the data load is completed from snowpipe to the landing area and want to make it run on its own. Please suggest if you see any other option here if not tasks.
You can, but not directly. You may want to look at external functions.
Snowflake supports external functions which can call a cloud api and therefore run any code that you are able to expose via a supported API.
The hyperlinked article gives a walk-through of creating an external function that you can adapt to suit your needs:
This walk-through is specific to AWS, but there are options for Azure and GCP
Platforms that Support Calling an External Function
In general, an external function can be called from a Snowflake account on any cloud platform that Snowflake supports:
Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform (GCP)
The steps you have to take to ensure appropriate security won't fit here:
however in summary you do the following:
Create an AWS IAM service role to execute all of the Lambda
functions
Create AWS Lambda functions for our examples
Create an AWS
API gateway to expose the Lambda functions to Snowflake
Create an
AWS IAM role to access the API gateway and configure it to access
both the API gateway and a Snowflake API integration object
Create
and test some external function objects in Snowflake
Sample Code
CREATE OR REPLACE external function external_functions.lambda.sum(filename varchar, rowcount number)
returns variant
api_integration = aws_lambda
as '<https://xxxxxxxxxx.execute-api.eu-west-2.amazonaws.com/snowflake-external-function-stage/snowflake-sum>'
If you decide to leverage external functions:
You may create a task, which runs a stored procedure.
This stored procedure could for read from your load history:
SELECT
*
FROM
(
select * from table(information_schema.copy_history(table_name=>'abc.mytable', start_time=> dateadd(hours, -1, current_timestamp())))
) q
WHERE q.PIPE_NAME ilike 'mypipe'
;
And use the output from the copy history to call your external function.
This external function could trigger any action that you can program on your cloud provider. This would be a "serverless" option.
If you are running on-premise / on a cloud vm. You can write a python script that runs in airflow / cron, that processes your copy history and run the next set of actions.
Related
I was wondering if anyone has sent data from Snowflake to an API (POST Request).
What is the best way to do that?
Thinking of using Snowflake to unload (COPY INTO) Azure blob storage then creating a script to send that data to an API. Or I could just use the Snowflake API directly all within a script and avoid blob storage.
Curious about what other people have done.
To send data to an API you will need to have a script running outside of Snowflake.
Then with Snowflake external functions you can trigger that script from within Snowflake - and you can send parameters to it too.
I did something similar here:
https://towardsdatascience.com/forecasts-in-snowflake-facebook-prophet-on-cloud-run-with-sql-71c6f7fdc4e3
The basic steps on that post are:
Have a script that runs Facebook Prophet inside a container that runs on Cloud Run.
Set up a Snowflake external function that calls the GCP proxy that calls that script with the parameters I need.
In your case I would look for a similar setup, with the script running within Azure.
https://docs.snowflake.com/en/sql-reference/external-functions-creating-azure.html
In my Logic Apps ARM template I have a "create record" to a Common Data Service connector.
"path": "/v2/datasets/#{encodeURIComponent(encodeURIComponent('orgd16*****.crm11'))}/tables/#{encodeURIComponent(encodeURIComponent('contacts'))}/items"
I know 'orgd16*****.crm11' is a CDS environment name, so I'm guessing the external API connection doesn't hold the environment ?
No. The external API connection does not hold the environment. The external API connection just authenticates and connects to the service.
Calls to an environment would pass through authorization on the common data service and if the user/service principal has access to the specified environment, it would be allowed.
I'm trying to write a custom connector Swagger file for Logic Apps and am having problems. The API I want to connect to only accepts OData queries so all my parameters are asking for $filter and the user has to type in Name eq 'Name' and Id eq 1. Is there a way to make this prettier and just ask them for the parameters directly?
I tried just adding them in (Name, Id, Active) but it puts them in the url like ?Name=. Not in the OData syntax. Is there any way to do what I want to do?
The custom connectors are designed to work as interfaces to existing REST APIs and the UI is more of a 1-1 mapping of their specification.
AFAIK, there is no way to direct customize how the connectors work but you could achieve it by proxy request through your own service.
You simply need a service which accepts requests the way you want and translate them accordingly for the actual service.
Azure API Management is probably the best candidate for this. As a bonus, once you have the APIs you need designed, you get an OpenAPI spec that you could use for the custom connector.
Depending on your expected load, you might have to use its Consumption Tier but do note that its currently in preview.
The alternatives could be having your own API hosted on Azure App Service or Azure Functions instead (or even Functions Proxies), again depending on your expected load.
PS: The downside of doing this is the obvious maintenance that you would have to uptake in case your requirements change and/or the backend API changes.
In OSB Layer when the endpoint uri is changed, I need to alert the core group that the endpoint has changed and to review it. I tried SLA Alert rules but it does not have options for it. My question is, the endpoint uri should be saved somewhere in the underlying database. If so what is the schema and the table name to query it.
URI or in fact any other part of OSB artifact is not stored in relational database but rather kept in memory in it's original XML structure. It can be only accessed thru dedicated session management API. Interfaces you will need to use are part o com.bea.wli.sb.management.configuration and com.bea.wli.sb.management.query packages. Unfortunately it is not as straightforward as it sounds, in short, to extract URI information you will need to:
Create session instance(SessionManagementMBean)
Obtain ALSBConfigurationMBean instance that operates on SessionManagementMBean
Create Query object instance(BusinessServiceQuery) an run it on ALSBConfigurationMBean to get ref object to osb artifact of your interest
Invoke getServiceDefinition on your ref object to get XML service
definition
Extract URI from XML service definition with XPath
Downside of this approach is that you are basically pooling configuration each time you want to check if anything has changed.
More information including JAVA/WLST examples can be found in Oracle Fusion Middleware Java API Reference for Oracle Service Bus
There is also a good blog post describing OSB customization with WLST ALSB/OSB customization using WLST
The information about services and all its properties can be obtained via Java API. The API documentation contains sample code, so you can get it up and running quite quickly, see the Querying resources paragraph when following the given link.
We use the API to read the service (both proxy and business) configuration and for simple management.
As long as you only read the properties you do not need to handle management sessions. Once you change the values, you need to start a session and activate it once you are done -- a very similar approach to Service bus console.
I am investigating ways to move data from SQL Server into system exposed via a RESTful HTTP API.
If I were to use SSIS would I have to write a custom connector to push the data to the HTTP API after the transform step, or is there a built in feature that supports pushing to an HTTP API?
If you only want to move a very small amount of data, you could use the Web Services Task
...but note that pushing data out of SQL Server is not what this task is intended for...
The Web Service task executes a Web service method. You can use the
Web Service task for the following purposes:
Writing to a variable the values that a Web service method returns.
For example, you could obtain the highest temperature of the day from
a Web service method, and then use that value to update a variable
that is used in an expression that sets a column value.
Writing to a file the values that a Web service method returns. For
example, a list of potential customers can be written to a file and
the file then used as a data source in a package that cleans the data
before it is written to a database.
For more control, you'll want to look at using the Script Component in a data flow. Much more flexibility/control.