Passing variables from predecessor task to Current task - snowflake-cloud-data-platform

Is there a way to set variables in Task 1 and pass them to the child task? It looks like every task has a different session so value of variable doesn't transfer in a task tree.

Check out SYSTEM$SET_RETURN_VALUE:
Explicitly sets the return value for a task.
In a tree of tasks, a task can call this function to set a return
value. Another task that identifies this task as the predecessor task
(using the AFTER keyword in the task definition) can retrieve the
return value set by the predecessor task.
There is an example in the link.

Related

Is Angular's foreach loop over an object asynchronous?

I read somewhere in the past that angular.foreach is asynchronous unlike looping over arrays which is synchronous. For a long time I was taking into account this and doing the necessary to avoid executing the code which comes after the loop before it's finishes all its iterations (by wrapping the angular.foreach inside an anonymous JavaScript function which calls a callback which will be executed once the loop finishes all iterations).
(function(callback){
angular.foreach(..)
callback();
})(callback)
But I had a conversation with a collegue who didn't agree that angular.foreach is asynchronous and I also couldn't find that information again which makes me confused now.
no. Take a look at the docs
Furthermore your code wouldn't work if foreach would be asynchronous.
If foreach would be async, the callback would be called immediately after calling foreach and foreach would be put onto the eventqueue which would execute it some time in the future.
Javascripts concurrency model does not have threads but instead uses an eventloop. This means every async operation is pushed onto the eventqueue and executed later.
Have a look into the MDN
There may be a scenario where you want to make code behave asynchronously.
I had a scenario where I used local storage to store an ad-hoc user selected collection of jobs that I wanted to perform the same operation on.
I had a web service call to convert a list of job names into a returned a collection of job objects. I initially tried using a
foreach loop inside the subscribe pf the service layer, that operated on the results.
Then I tried calling another method within the foreach loop that as it performed the operations removed the job name from local storage when the operation posted to the web service correctly.
The problem was on the second iteration I read the collection of names from local storage again - before the set to remove had completed.
There was a lot of manipulation of the job and object properties to create the parameters passed on the function call, so I ended up refactoring the code, creating a value object interface and stored the information in a value object array for the whole job collection I had returned. I included the index of the job too in the value object.
I introduced a BehaviourSubject property to the class.
During the restructuring, I just added an entry to the value object array collection within the forEach loop instead. At the end of the loop. I sent next(0) to the BehaviourSubject to start the ball rolling.
Each time a job name was removed from local storage, I converted service to return a Promise.
Then in the code after the service was called I put this code in the then part, behaviour subject.next(index from value object +1)
In the initialisation I set the behaviour subject up with a -1 value..
Then in the subscription to the BehaviourSubject class I ignored -1,
And when the index +1 was > length of value object collection called completion routine - which bounce app back to prior page.
When the index was between 0 and 1 less than collection size, I just called the method that had originally been in the forEach loop with the value object entry with the value object match the index of the behaviour subject.
By doing this I had converted the behaviour of the forEach into something asynchronous.

how to use jump to if I need to evaluate a condition of context variable for two different nodes at same time

I have one parent node ,based on the user input Iam setting a context variable at my application level eligibility:yes or no and passing back.And for my parent node I have two child nodes for conditions $eligibility=="yes" and $eligibility=="no".So once users input from parent node validation is done and context variable is passed back ,then I need to jump and look for condition of eligibility.If yes I need to go one node ,if no then to other.How can I do?
I tried putting true to node and added these two nodes to this and jump to true..But didnt worked..How can we achieve this?
what #data_henrik has mentioned is a good way to set context value and then switch to different flows depending upon the set value. But when you need to perform some logic before setting that value in the context from your application, it won't be a suitable way.
I had a requirement like this, so we used to send a dummy text from our application after we were done with setting the value in context after the parent node execution. Check out the images and explanation after that.
We didn't use Jump because we had to do some validation in the Conversation service after parent node before moving forward. Using jumps would've allowed the Conversation to move to next node before we could set value in context.
Use case flow - once user enters text for the parent node intent, for my case "#send-mail" intent, I show the parent response and do some functional validation in my app after that and add a value to the context. Now we send a dummy text "valid" which satisfies the intent "#Valid" and hence move to the next node in flow. In this node we check for the value in context (which is already set by now) and show appropriate response to user.
You can set within your first two test nodes, $testMe==true and $testMe==false a temp output variable within the output json packet, i.e. output{"temp":"true"} or "false". Then you can jump to a new set of nodes and test for the output.temp value, i.e. output.temp == 'true' then do something, or output.temp == 'false' then do something.
The nice side effect of this action is that the output.temp variable only has a life of that current conversation input. Unlike context variables which need to be removed / deleted.

Flink trigger on a custom window

I'm trying to evaluate Apache Flink for the use case we're currently running in production using custom code.
So let's say there's a stream of events each containing a specific attribute X which is a continuously increasing integer. That is a bunch of contiguous events have this attributes set to N, then the next batch has it set to N+1 etc.
I want to break the stream into windows of events with the same value of X and then do some computations on each separately.
So I define a GlobalWindow and a custom Trigger where in onElement method I check the attribute of any given element against the saved value of the current X (from state variable) and if they differ I conclude that we've accumulated all the events with X=CURRENT and it's time to do computation and increase the X value in the state.
The problem with this approach is that the element from the next logical batch (with X=CURRENT+1) has been already consumed but it's not a part of the previous batch.
Is there a way to put it back somehow into the stream so that it is properly accounted for the next batch?
Or maybe my approach is entirely wrong and there's an easier way to achieve what I need?
Thank you.
I think you are on a right track.
Trigger specifies when a window can be processed and results for a window can be emitted.
The WindowAssigner is the part which says to which window element will be assigned. So I would say you also need to provide a custom implementation of WindowAssigner that will assign same window to all elements with equal value of X.
A more idiomatic way to do this with Flink would be to use stream.keyBy(X).window(...). The keyBy(X) takes care of grouping elements by their particular value for X. You then apply any sort of window you like. In your case a SessionWindow may be a good choice. It will fire for each key after that key hasn't been seen for some configurable period of time.
This approach will be much more robust with regard to unordered data which you must always assume in a stream processing system.

How change task priority in cloudsim?

I change task priorities based my algorithm now I want gives this tasks to vms , How I can do it in cloudsim?
you can set the 'classType' parameter of cloudLet object from your algorithm to priorotize them.
There's a method bindCloudletToVm of the class DatacenterBroker. You can refer this

Apache Giraph : Number of vertices processed by each partition

I am a newbie trying to understand the working of Giraph 1.2.0. with hadoop 1.2.1.
Is there any way to figure out the number of vertices processed by each mapper?
The call method of org.apache.giraph.graph.ComputeCallable class is executed once per superstep. Inside this function, for each partition owned by this map task, the computePartition function is called. So, you can easily define an integer (counter) to this class. Then, in computePartition, if compute method of the vertex is called, increment the counter. Finally, at the end of call method print your counter. So, for each superstep of each mapper, it prints the number of vertices processed.

Resources