I'm going to ask and answer this question because it took me forever to figure out and I wish the answer had been here to begin with.
The problem: One long-running unixODBC query blocks all others from the same application.
The question: How does one stop this from happening.
The answer, in the form of a cut-and-paste comment from __handles.c -- I know, why doesn't everyone think to look there for documentation to begin with, right?
/*
* use just one mutex for all the lists, this avoids any issues
* with deadlocks, the performance issue should be minimal, if it
* turns out to be a problem, we can readdress this
*
* We also have a mutex to protect the connection pooling code
*
* If compiled with thread support the DM allows four different
* thread strategies
*
* Level 0 - Only the DM internal structures are protected
* the driver is assumed to take care of it's self
*
* Level 1 - The driver is protected down to the statement level
* each statement will be protected, and the same for the connect
* level for connect functions, note that descriptors are considered
* equal to statements when it comes to thread protection.
*
* Level 2 - The driver is protected at the connection level. only
* one thread can be in a particular driver at one time
*
* Level 3 - The driver is protected at the env level, only one thing
* at a time.
*
* By default the driver open connections with a lock level of 3,
* this can be changed by adding the line
*
* Threading = N
*
* to the driver entry in odbcinst.ini, where N is the locking level
* (0-3)
*
*/
Just an addition to that answer. The current release of unixODBC 2.3.0 defaults to Threading = 0, so the default now is to assume that driver are thread safe. This was a risky assumption in years past, not so much now.
If your driver supports asynchronous functions, you can enabled it and execute time consuming functions in async mode.
No threads required at application side.
Related
I am having some trouble understanding the way windowing is implemented internally in Flink and could not find any article which explain this in depth. In my mind, there are two ways this can be done. Consider a simple window wordcount code as below
env.socketTextStream("localhost", 9999)
.flatMap(new Splitter())
.groupBy(0)
.window(Time.of(500, TimeUnit.SECONDS)).sum(1)
Method 1: Store all events for 500 seconds and at the end of the window, process all of them by applying the sum operation on the stored events.
Method 2: We use a counter to store a rolling sum for every window. As each event in a window comes, we do not store the individual events but keep adding 1 to previously stored counter and output the result at the end of the window.
Could someone kindly help to understand which of the above methods (or maybe a different approach) is used by Flink in reality. The reason is, there are pros and cons to both approach and is important to understand in order configure the resources for the cluster correctly.
eg: The Method 1 seems very close to batch processing and might potentially have issues related to spike in processing at every 500 sec interval while sitting idle otherwise etc while Method2 would need to maintain a common counter between all task managers.
sum is a reducing function as mentioned here(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#reducefunction). Internally, Flink will apply reduce function to each input element, and simply save the reduced result in ReduceState.
For other windows functions, like windows.apply(WindowFunction). There is no aggregation so all input elements will be saved in the ListState.
This document(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#window-functions) about windows stream mentions about how the internal elements are handled in Flink.
Starting with Windows Vista, every application automatically gets a thread pool (the "default thread pool").
My Question is: is there a way to configure the min and max number of threads for this default thread pool?
SetThreadpoolThreadMaximum seems to work only for a non-default thread pool (a pool created with CreateThreadpool):
SetThreadpoolThreadMaximum(NULL, 4); - throws 0xC000000D: An invalid parameter was passed to a service or function
According to Joe Duffy from Microsoft, this is not possible.
Here's a quote from his book, Concurrent Programming on Windows, chapter 7. Thread Pools, page 345:
Note that it is not possible to alter the default thread pool's
minimum and maximum count; instead, you must specify a pointer to a
custom TP_POOL object. Prior to Vista, you could change the
process-wide default pool's maximum (as we see later). The reason this
capability has been removed is because it depends on races: the last
component to call the API would win. This can cause conflicts between
components in the same process that are unaware of each other but want
different maximum or minimum values.
The "old way" of increasing the pool size the author is referring to is the macro WT_SET_MAX_THREADPOOL_THREADS, used in QueueUserWorkItem. That setting seems to be ignored in the new pool architecture.
So the default pool's minimum is fixed at 0 and the maximum is 500.
I'm implementing NTP Server and client (for the first time). I've few questions for which I couldn't found detailed explanation. Please help me with the below topics.
1. What is root dispersion ,precision and poll fields in the NTP packet format
2. Can i assign any values to them or shall i need to calculate?
Please suggest me....
So a good place to start would be the Official NTP Docs
rootdisp - total dispersion to the primary reference clock
precision - precision (log2 s)
poll - poll messages sent (every n sec)
You don't directly assign any of those values - you need to read up about the ntp.conf file and which directives you should use there for your specific environment.
You may also want to look at an answer I gave on Superuser
You didn't say which OS you are using, or which ntp client so this is all generic stuff based on linux. If you would like specific help then please expand your question with some examples and some of the reference material you have already looked at else it will be very hard to help.
I'm not an expert in Network Programming. I basically have two kinds of clients who have different time-outs. I am supposed to use UDP with connected sockets for client-server communication.
The problem is twofold:
a) I need to mark as died whichever client (alternatively, socket) does not respond for t1 seconds. Using select would time out if none of the sockets in read_fd_set have anything to read within the timeout value. So, how do I time-out any one socket which is not having data to read for quite some time?
Currently, whenever select returns, I myself keep track of which sockets are responding and which not. And I add t1.tu_sec to the individual time elapsed of each client (socket). Then, I manually close and exclude from FD_SET the socket which does not respond for (n) * (t1.tu_sec) time. Is this a good enough approach?
b) The main problem is that there are two kinds of clients which have different time-outs, t1 and t2. How do I handle this?
Can I have two select()s for the two kinds of clients in the same loop? Would it cause starvation without threads? Is using threads advisable (or even required) in this case?
I've been roaming around the web for ages!
Any help is much appreciated.
This is just a special case of a very common pattern, where a select/poll loop is associated with a collection of timers.
You can use a priority queue of tasks, ordered on next (absolute) firing time; the select timeout is always then just the absolute time at the front of the queue.
when select times out (and just before the next iteration, if your tasks may take a long time to complete), get the current time, pull every task that should already have executed off the queue, and execute it
(some) tasks will need to be re-scheduled, so make sure they can mutate the priority queue while you do this
Then your logic is trivial:
on read, mark the socket busy
on timer execution, mark the socket idle
if it was already idle, that means nothing was received since the last timer expiry: it's dead
A quick solution that comes to my mind, is to keep the sockets in a collection sorted by the time remaining until the nearest timeout.
Use select with the timeout set to the smallest time remaining, remove/close/delete the timed-out socket from the collection, and repeat.
So, in pseudo-code it might look like this:
C = collection of structs ( socket, timeout, time_remaining := timeout )
while (true) {
sort_the_collection_by_time_remaining
next_timeout = min(time_remaining in C)
select ( sockets in C, next_timeout )
update_all_time_remaining_values
remove_from_C_if_required //if timeout occured
}
It can easily be solved with a single select call. For each socket have two values related to the timeout: The actual timeout; And the amount of time until timeout. Then count down the "time until timeout" every 0.1 second (or similar), and when it reaches zero close the socket. If the socket receives traffic before the timeout simply reset the "time until timeout" to the timeout value and start the down-count again.
i need a locking in memcache. Since all operations are atomic that should be an easy task. My idea is to use a basic spin-lock mechanism. So every object that needs locking in memcache gets a lock object, which will be polled for access.
// pseudo code
// try to get a lock
int lock;
do
{
lock = Memcache.increment("lock", 1);
}
while(lock != 1)
// ok we got the lock
// do something here
// and finally unlock
Memcache.put("lock", 0);
How does such a solution perform? Do you have a better idea how to lock a memcache object?
Best regards,
Friedrich Schick
Be careful. You could potentially burn through a lot of your quota in that loop.
Locking is generally a bad idea - and in your example, will result in a busy-wait loop that consumes huge amounts of quota.
What do you need locking for? Perhaps we can suggest a better alternative.
If you really do need a loop: don't busy-wait, but include a delay, possibly with exponential back-off:
int delay = 100;
do {
lock = Memcache.increment("lock", 1);
usleep(delay);
delay = min(delay * 2, 100000);
}
while (!lock);
All operations on memcache are atomic, as you said. To echo others' responses, do not use a naive spin lock on the app engine. You'll use up your daily quota in about 20 minutes. Now, to your solution:
I've done something like this. I created a task queue with a bucket size of 1 and a exec rate of 1/10s (one task per 10 seconds). I used this queue for "spinning", except that it has the advantage of only checking once per 10 seconds. I'm not sure what your use case is, but even executing a task once per second is far better than just spinning in a loop. So you implement a task servlet that checks the status of this lock and, if free, does whatever you want it to do.