How to process Azure Log events from Event hub and filter based on criteria.
We are trying to filter specific critical or security related Diagnostic and Activity logs before feeding into our Onprem SIEM solution.
Can someone please guide me how to filter the data from Event hub and then re-ingest into another event hub. Whether this is possible or any other alternatives available out there.
At a high level , the flow is shown below. Source from Diagnostic Logs (Monitor) -> Event Hub -> Filter/Query -> Event Hubenter image description here
The answer could be very bare - but in our case we are using Azure Functions to perform similar operations.
As you know the shape of data - the function can decide if event is forwarded to other EH or it is dropped.
Related
I want to build a real time ordering system where users can see what other people have also ordered in near real time. Will DynamoDB with streams allow me to do this?
As a customer
Given there are multiple people ordering
When another customer submits an order
Then I should see that order appear on my screen in near real time
Therefore, I need some way for each front end client to subscribe to changes, ideally without having to poll for changes every second.
As far as I can tell, streams allows me to trigger an action such as running a Lambda function on any udpate event but will it allow me to emit changes to any client / user that is listening for changes as well?
Some alternative options I have already tried is web sockets and GCP Firebase. Both work for my use case but I want to know if there is a way to acheive this within DynamoDB.
Best thing to do here is set up a pub/sub system. There are various ways to achieve this:
DDB -> Streams -> Lambda -> SNS -> Listener
Or via websocket API
DDB -> Streams -> Lambda -> APIGW WebSocket -> Listener
And you can also use Amplify/AppSync PubSub model that allows you to subscribe when you use GraphQL APIs to interact with DynamoDB.
https://docs.aws.amazon.com/appsync/latest/devguide/aws-appsync-real-time-data.html
I want to register a callback so my code can receive and process all new event logs for Microsoft-Windows-WMI-Activity/Trace.
To clarify, the log path for Microsoft-Windows-WMI-Activity/Trace is %SystemRoot%\System32\Winevt\Logs\Microsoft-Windows-WMI-Activity%4Trace.etl, and in the Windows Event Viewer, these events can be found in the tree under Application and Services Logs > Microsoft > Windows > WMI-Activity > Trace.
I know EvtSubscribe() can be used to subscribe to events on channels, but I want to know if this function can also be used for subscribing to events on Microsoft-Windows-WMI-Activity/Trace.
I know for the ChannelPath parameter, it says on the function's docs page that
you cannot subscribe to Analytic or Debug channels
When I view the properties for Microsoft-Windows-WMI-Activity/Trace in Event Viewer, I see the log is of type Analytic, so it seems I cannot subscribe with this function:
My questions:
Can I use this function to subscribe to Microsoft-Windows-WMI-Activity/Trace?
If I cannot with this function, is there some other function in the Windows API that I can use instead that will allow me to subscribe?
As you can see in the article
you cannot subscribe to Analytic or Debug channels
EvtSubscribe() cannot subscribe to Microsoft-Windows-WMI-Activity/Trace.
Explained in the introduction to the Event Types
-Direct channel
You cannot subscribe to a direct channel, but you can query a direct
channel.
A direct channel is performance-oriented. Events are not processed in
any way by the eventing system. This allows the direct channel to
support high volumes of events.
Direct channels have the following types:
Analytic
Analytic events are published in high volume. They describe program
operation and indicate problems that cannot be handled by user
intervention.
At present, in the officially provided method, seems to only be viewable through the Event Viewer.
https://learn.microsoft.com/en-us/windows/win32/wmisdk/tracing-wmi-activity#obtaining-wmi-events-through-event-viewer
I want to get notified (through email of slack channel) for every Exception occurence in my GAP services.
When I'm trying to create a notification through GCP Error Reporting, it seems like I'm being able to get one notification per incident (and not per occurrence), and that it is also being queried only once a day/hour/month.
I've also tried to create a policy in GCP Logs Monitoring based on log severity, but of course I don't get notified only for exceptions, and the slack notifications just announce that a threshold is being passed, without the actual data I want to get by push.
Any way I can make Error Reporting notify me per each occurance?
Is there any other internal tool by GCP that notify when event occurs? or should I use an external tool such as Epsagon/Operations (formally Stackdriver)?
So it seems like there is no availability for getting a pushed event on each exception occurrence, and since I didn't want to query it by pull (with ereport) or to get only new exceptions (through Stackdriver), I had to go with an external service such as Datadog.
I want to implement a connected OAuth app in Salesforce which should trigger push events in case some entities changed, for example an opportunity was closed.
Zapier implemented something similar
https://zapier.com/apps/salesforce/integrations/webhook
Could not find something I need which is a simple way to subscribe to entity changes using the OAuth client's token and passing a webhook endpoint. I read about apex callouts, streaming API and outbound messages.
Yeah, we solved this exact problem at Fusebit and I can help you understand the process as well.
Typically speaking here's what you need to do:
Create triggers on the Salesforce Objects you want to get updates for
Upload Apex class that will send an outgoing message to a pre-determined URL
Enable Remote Site Setting for the Domain you want to send the message to
Add in Secret Verification (or other auth method) to prevent spamming of your external URL
If you're leveraging javascript, then you can use the jsforce sdk & salesforce tooling API to push the code into the salesforce instance AFTER the Auth flow has occurred AND on Salesforce Instances that have API access enabled (typically - this is enterprise and above OR professional with API enabled).
This will be helpful for you to look through: https://jamesward.com/2014/06/30/create-webhooks-on-salesforce-com/
FYI - Zapier's webhooks implementation is actually polling every 15 minutes, instead of real-time incoming events.
In which programming language?
For consuming outbound messages you just need to be able to accept an XML message and send back "Ack" message to acknowledge receiving, otherwise SF will keep trying to resend it for 24h.
For consuming platform events / streaming API / Change Data Capture (CDC) you'll need to raise the event in SF (Platform Event you could raise from code, flow, process builder, CDC would happen automatically, you just tell it which objects it should track).
And then in client app you'd need to login to SF (SOAP or REST API), subscribe to channel (any library that supports cometd should be fine). Have you seen "EMP Connector", mentioned for example in https://trailhead.salesforce.com/en/content/learn/modules/change-data-capture/subscribe-to-events?trail_id=architect-solutions-with-the-right-api ?
Picking right messaging way is an art, there's free course that can help: https://trailhead.salesforce.com/en/content/learn/trails/architect-solutions-with-the-right-api
And pretty awesome PDF if you want to study for certification: https://resources.docs.salesforce.com/sfdc/pdf/integration_patterns_and_practices.pdf
I have data in Salesforce and run another application that works with the same data. The current workflow is that when data is entered into the custom application, it sends the information to Salesforce via SOAP. I want to establish the reverse link; when a value is changed on the Salesforce side, I want Salesforce to ping my application with the changes. Does Salesforce have a feature to do this? Something equivalent to a trigger maybe?
My current solution is mindless iteration through all Salesforce records. This is slow, hits the API limit often, and keeps data stale too long.
You can do this using Streaming API
Introduction:
Use Streaming API to receive notifications for changes to Salesforce data.
Use to push relevant data in realtime, instead of having to refresh the screen to get new information. Protocols Use for Connection:
The Bayeux protocol and CometD both use long polling.
Bayeux is a protocol for transporting asynchronous messages, primarily over HTTP.
CometD is a scalable HTTP-based event routing bus that uses an AJAX push technology pattern known as Comet. It implements the Bayeux
protocol. The Salesforce servers use version 2.0 of CometD.
How it Works:
Create a PushTopic based on a SOQL query. This defines the channel. (PushTopic is a standard object).
Clients subscribe to the channel.
A record is created, updated, deleted, or undeleted (an event occurs). The changes to that record are evaluated.
If the record changes match the criteria of the PushTopic query, a notification is generated by the server and received by the subscribed
clients.
Please check this link : http://www.salesforce.com/developer/docs/api_streaming/