Why we need queueSizeRejectionThreshold in Hystrix apart from maxQueueSize?
By definition, queueSizeRejectionThreshold <= maxQueueSize. But I am not getting why not to reject thread when maxQueueSize becomes full, why to introduce the term queueSizeRejectionThreshold?
The documentation explains the reason why you may need queueSizeRejectionThreshold:
This property exists because the maxQueueSize of a BlockingQueue cannot be dynamically changed and we want to allow you to dynamically change the queue size that affects rejections.
If you don't want to change the queue size dynamically (in run time) simply set maxQueueSize = queueSizeRejectionThreshold.
Related
I'm trying to evaluate Apache Flink for the use case we're currently running in production using custom code.
So let's say there's a stream of events each containing a specific attribute X which is a continuously increasing integer. That is a bunch of contiguous events have this attributes set to N, then the next batch has it set to N+1 etc.
I want to break the stream into windows of events with the same value of X and then do some computations on each separately.
So I define a GlobalWindow and a custom Trigger where in onElement method I check the attribute of any given element against the saved value of the current X (from state variable) and if they differ I conclude that we've accumulated all the events with X=CURRENT and it's time to do computation and increase the X value in the state.
The problem with this approach is that the element from the next logical batch (with X=CURRENT+1) has been already consumed but it's not a part of the previous batch.
Is there a way to put it back somehow into the stream so that it is properly accounted for the next batch?
Or maybe my approach is entirely wrong and there's an easier way to achieve what I need?
Thank you.
I think you are on a right track.
Trigger specifies when a window can be processed and results for a window can be emitted.
The WindowAssigner is the part which says to which window element will be assigned. So I would say you also need to provide a custom implementation of WindowAssigner that will assign same window to all elements with equal value of X.
A more idiomatic way to do this with Flink would be to use stream.keyBy(X).window(...). The keyBy(X) takes care of grouping elements by their particular value for X. You then apply any sort of window you like. In your case a SessionWindow may be a good choice. It will fire for each key after that key hasn't been seen for some configurable period of time.
This approach will be much more robust with regard to unordered data which you must always assume in a stream processing system.
I've got a question about "rwlocks", especial about "rwlockattr".
I've got a linked list with several threads working with. Every Member on this list has got a "rwlock". So now I want to set up a rule to secure that threads who want access a write-lock have a higher priority. My intention is to use
int pthread_rwlockattr_setkind_np(pthread_rwlockattr_t *attr,int pref);
So now my Question; Do I need to initialize a "rwlockattr" for every "rwlock" in my linked List or is it enough to set up a global "rwlockattr", initialize it and set up the "PTHREAD_RWLOCK_PREFER_WRITER_NP" rule ?
regards
With every rwlock there is some by default attributes associated with it. For the pthread_rwlock_init() go through this link which will give you more information that how to use rwlock.
You can assign single attribute to your rwlocks. You globally create a single attribute and assign to your rwlocks having the same nature.
Go through this to understand the use of pthread_rwlock.
In general Attributes are to decide the nature of your rwlock.
I have a class whose objects are extensively used using shared_pointers. However, I want to track the usage of these objects and when the refcount goes to a particular value I want to delete the object. How can we do this ? I was thinking of overriding the shared_ptr's destructor so that I can decrement the refcount when every shared_ptr reference goes away. However, looks like that is not possible. What are the alternatives ?
You really wouldn't want to do that because if the refcount is greater than zero it means there are still pointers pointing to the object out there, probably intending to access it.
If you really wanted to do something like that, you'd have to make your own shared_ptr class, but I'd also add functionality for checking if the pointer is still valid since it might disappear on people.
I have two loops:
One loop gets data from a device and processes it. Scales received variables, calculates extra data.
Second loop visualizes the data and stores it.
There are lots of different variables that need to passed between those two loops - about 50 variables. I need the second loop to have access only to the newest values of the data. It needs to be able to read those variables any time they are needed to be visualized.
What is the best way to share such vector between two loops?
There are various ways of sharing data.
The fastest and simplest is a local variable, however that is rather uncontrolled, and you need to make sure to write them at one place (plus you need an indicator).
One of the most advanced options is creating a class for your data, and use an instance (if you create a by-ref class, otherwise it won't matter), and create a public 'GET' method.
In between you have sevaral other options:
queues
semaphores
property nodes
global variables
shared variables
notifiers
events
TCP-IP
In short there is no best way, it all depends on your skills and application.
As long as you're considering loops within the SAME application, there ARE good and bad ideas, though:
queues (OK, has most features)
notifiers (OK)
events (OK)
FGVs (OK, but keep an eye on massively parallel access hindering exec)
semaphores (that's not data comms)
property nodes (very inefficient, prone to race cond.)
global variables (prone to race cond.)
shared variables (badly implemented by NI, prone to race cond.)
TCP-IP (slow, awkward, affected by firewall config)
The quick and dirty way to do this is to write each value to an indicator in the producer loop - these indicators can be hidden offscreen, or in a page of a tab control, if you don't want to see them - and read a local variable of each one in the consumer loop. However if you have 50 different values it may become hard to maintain this code if you need to change or extend it.
As Ton says there are many different options but my suggestion would be:
Create a cluster control, with named elements, containing all your data
Save this cluster as a typedef
Create a notifier using this cluster as the data type
Bundle the data into the cluster (by name) and write this to the notifier in the producer loop
Read the cluster from the notifier in the consumer loop, unbundle it by name and do what you want with each element.
Using a cluster means you can easily pass it to different subVIs to process different elements if you like, and saving as a typedef means you can add, rename or alter the elements and your code will update to match. In your consumer loop you can use the timeout setting of the notifier read to control the loop timing, if you want. You can also use the notifier to tell the loops when to exit, by force-destroying it and trapping the error.
Two ways:
Use a display loop with SEQ (Single Element Queue)
Use a event structure with User Event. (Do not put two event structures in same loop!! Use another)
Use an enum with case structure and variant to cast the data to expected type.
(Notifier isn't reliable to stream data, because is a lossy scheme. Leave this only to trigger small actions)
If all of your variables can be bundled together in a single cluster to send at once, then you should use a single element queue. If your requirements change later such that the transmission cannot be lossy, then it's a matter of changing the input to the Obtain Queue VI (with a notifier you'd have to swap out all of the VIs). Setting up individual indicators and local variables would be pretty darn tedious. Also, not good style.
If the loops are inside of the same VI then:
The simplest solution would be local variables.
Little bit better to use shared variables.
Better is to use functional global variables (FGVs)
The best solution would be using SEQ (Single Element Queue).
Anyway for better understanding please go trough this paper.
Consider following function:
int get_something (int* array, int size);
It's purpose is to fill the passed array[size] with data from external resource (queryes to resource are expensive). The question is what to do, if resource has more elements, than provided array can handle? What is the best approach?
Edit: Current solution added:
Our approach, at the moment is following:
When user calls get_something() first time, with null argument we perform a full Query, allocate data in a cache (which is just a key-value storage) and return a number of items.
When user calls get_something() next time, with properly initialized buffer, we return him data from cache and clear a cache entry.
If user does not call get_something(), timeout occurs and cache for that item gets freed.
If user calls get_something() too late, and data has been cleared, we generate error state, so user knows that he has to repeat the request.
One option is to not modify the array at all and instead return the needed size as the return result. The caller must then call your function again with an array of at least this size.
Ok, your basic requirement is to Query a resource, and cache the returned data in memory, to avoid multiple accesses.
That means you will have to allocate memory within your program to store all of the data.
Problem #1 is to populate that cache. I will assume that you have that figured out, and there is some function get_resource();
problem #2 is how to design an api to allow client/user code to interact with that data.
In your example you you are using an array allocated by the client, as the cache, hoping to solve both problems with 1 buffer, but this doesn't solve the problem in all cases ( hence your posting ). So you really need to separate the 2 problems.
Model number #1 is to provide iterator / cursor functionality
iterator = get_something(); // Triggers caching of data from Resource
data = get_next_single_something( iterator );
status = release_something( iterator );
// The logic to release the data could be done automagically in get_next,
// after returning the last single_something, if that is always the use case.
Model #2 is to return the Whole object in a malloced buffer, and let the client manage the whole thing
data_type *pData=NULL;
unsigned size = get_something( &pData ); // Triggers caching of data from Resource
process( size, pData );
free( pData );
pData=NULL;
Model #3. If you are married to the client array, you can use Model #1 to return multiple values at once, but if there are more values, then get_something() will have to build a cache, and the client will still have to iterate.
Use realloc .
Reference link .
My choice would be to use the same model as fread() and turn the interface into a stream of sorts.
i.e.
either fill the buffer up or put all the items in it and return the number of items actually read
maintain some sort of state so that subsequent calls only get unread items
return 0 once all the items have been read
return a negative number if an error occurs.
allocate array dynamically i.e using malloc() and then, in the function, either use realloc() or free the previous list and allocate another, fill it and return the new size. For the second approach you can use the return value for returning new size but to update the callers address of array you will need to change the function to accept int** instead of int*
Can you do a check on how many elements the resource has? If so I'd do that then copy the array to an array as large as the resource.
or perhaps copying the array to an array double its size when you're reaching near the end?
http://www.devx.com/tips/Tip/13291
That depends on how your program should handle that situation, I guess.
One approach could be to fill the array to it's maximum, and return the total number of elements which are available. That way the caller could check if he needs to call the function again (see Mark Byers answer).
Logic behind that:
- Creates array with 100 items
- Calls your function and gets 150 returned
- Increases the array size or creates a second one
and calls your function again with that array
- Repeats that unless the returned item count is
equal or less the array size