Multithreaded c program design help - c

I don't have much experience with multithreading and I'm writing a c program which I believe is suited to running in two threads. The program will listen on the serial port for data, read and process new data when it's available, and publish the newest processed data to other (irrelevant) modules via a third party IPC api (it's confusingly named IPC) when requested.
In order to receive the request to publish data via IPC, the program must call IPC_listenwait(wait_time);. Then if a request to publish is received while "listenwaiting" a handler is invoked to publish the newest data.
One option is to do this in one thread like:
for(;;) {
read_serial(inputBuffer);
process_data(inputBuffer, processedData); //Process and store
IPC_listenwait(wait_time); //If a request to publish is received during this,
} //then a handler will be invoked and the newest piece of
//processedData will be published to other modules
publishRequestHandler() { //Invoked when a message is received during IPC_listenwait
IPC_publish(newest(processedData));
}
And this works, but for the application it is important that the program is very responsive to the request to publish new data, and that the data published is the newest available. These goals are not satisfied with the above because data may arrive after the process begins listenwaiting and before a request to publish message is received. Or the process may be reading/processing when a request to publish message is incoming, but won't be able to service it until the next IPC_listenwait call.
The only design I can think of is to have one thread to read, which will just do something like:
readThread() {
for(;;) { //pseudocode
select();
read(inputBuffer);
process(inputBuffer, processedData);
}
}
And have the main thread just listening for incoming messages:
mainThread() {
IPC_listenwait(forever);
}
publishRequestHandler() { //Invoked when a message is received during IPC_listenwait
IPC_publish(newest(processedData));
}
Is this the design you would use? If so, will I need to use a semaphore when accessing or writing processedData?
Will this give me good responsiveness?
Thanks

You're mostly on the right track.
The one thing you have to watch out for is concurrent access to the publishable data, because you don't want one thread clobbering it while another is trying to read it. To prevent that, use a pair of buffers and a mutex-protected pointer to whichever one is considered current. When process_data() has something ready, it should dump its results in the non-current buffer, lock the pointer mutex, repoint the pointer to the buffer containing the new data and then release the mutex. Similarly, the publisher should lock the pointer mutex while it reads the current data, which will force anything that might want to clobber it to wait. This is a bit more complex than having a single, mutex-protected buffer but will assure that you always have something current to publish while new data is being prepared.
If your processing step takes long enough that you could get multiple sets of data to read, you might split the read/process thread into two and let the reader make sure the processor only ever gets the latest and greatest so you don't end up processing stuff you won't ever publish.
Excellent first question, by the way. Have an upvote.

Related

How to make a GTK application that receives data from a TCP socket

I am building a GUI (using C) which receives data to display from another application sending the data over a TCP socket. How do I do this using GTK (just a general overview of the approach I should take)? I have done a lot of searching and came across stuff about multithreading, GIOchannel etc. now I'm more confused than ever. There doesn't seem to be any conclusive articles or guides about how to actually achieve this.
There is basically one important rule:
You must call all gtk_* functions from the main thread.
If you update any widgets from another thread, you might get inconsistent results.
Of course, you don't want to wait for TCP data in that thread.
Therefore I would suggest you create a separate thread for doing the communication. In this thread you can wait for data and if you got anything that should affect what you show in your GUI, you can tell the main thread to do the required work.
A simple way to do this is to use g_idle_add() to enqueue a callback function. That callback function is then executed in context of main thread and can update your widgets.
The information what needs to be updated can be stored in some newly allocated memory that is passed to this callback where you have to free it afterwards.

Reading two streams (main and configs) in sequential in Flink

I have two streams, one is main stream let's say in example of fraud detection I have transactions stream and then I have second stream which is configs, in our example it is rules. So I connect main stream to config stream in order to do processing. But when first time flink starts and we are adding job it starts consuming from transactions and configs stream parallel and when wants process transaction it sometimes see that there is no config and we have to send transaction to dead letter queue. However, what I want to achieve is, if there is patential config which I could get a bit later I want to get that config first then get transaction in order to process it rather then sending it to dead letter queue. I have the same key for transactions and configs.
long story short, is there a way telling flink when first time job starts try to consume one stream until there isn't new value then start processing main stream? How I can make them kind of sequential?
The recommended way to approach this is to connect the 2 streams and apply a RichCoFlatMap that will allow you to buffer events from main while you're waiting to receive the config events.
Check out this useful section of the Flink tutorials. The very last paragraph actually describes your problem.
It is important to recognize that you have no control over the order in which the flatMap1 and flatMap2 callbacks are called. These two input streams are racing against each other, and the Flink runtime will do what it wants to regarding consuming events from one stream or the other. In cases where timing and/or ordering matter, you may find it necessary to buffer events in managed Flink state until your application is ready to process them. (Note: if you are truly desperate, it is possible to exert some limited control over the order in which a two-input operator consumes its inputs by using a custom Operator that implements the InputSelectable interface.
So in a nutshell you should connect your 2 streams and have some kind of ListState where you can "buffer" your main elements while waiting to receive the rules. When you receive an element from the config stream, you check whether you had some pending elements "waiting" for that config in your ListState (your buffer). If you do, you can then process these elements and emit them through the collector of your flatmap.
Starting with version 1.16, you can use the hybrid source support in Flink to read all of once source (configs, in your case) before reading the second source. Though I imagine you'd have to map the events to an Either<config, transaction> so that the data stream has consistent record types.

How to send multiple diagRequest messages in Vector CAPL?

I'm currently writing some CAPL code that is executed when clicking a button. It shall send multiple Diagnostic Requests. But CANoe is always telling me, that it can only send one request at a time. So I need to delay the requests. The diagSetRequestInterval function did not work. And since it is NOT a testcase, the testWaitForDiagResponse doesn't work either.
You have to wait until the request has been handled (either by a response from the target or by a timeout).
Since you are not in a test node you have to give back the control to the system, i.e. your function which did diagSendRequest shall end and you wait for some events on the bus to occur before you continue (otherwise the simulation would stall).
Once the request has been handled on diagRequest ... is called. Inside this event procedure, you could send the next request and so on.
Example:
Instead of:
myFunction()
{
diagRequest ECU.ProgrammingSession req1;
diagRequest ECU.SecuritySeed req2:
diagSendRequest(req1);
diagSendRequest(req2);
}
You would do something like this:
myFunction()
{
diagRequest ECU.ProgrammingSession req1;
diagSendRequest(req1);
}
on diagResponse ECU.ProgrammingSession
{
diagRequest ECU.SecuritySeed req2:
diagSendRequest(req2);
}
Timeout handling is a different topic, and left as an exercise :-)
You practically want to implement multiple TP connection simultaneously in CANoe. I presume you have only one Diagnostic Description in the Diagnostic/ISO TP configuration, which lets you to use only 1 TP connection at a time.
You can implement multiple diag layers in Diagnostic ISO/TP on the same Communication channel, as much as you want, but with different namings.
In simulation node, you will only have to declare the request you want with a different namespace, corresponding to one of the diag layer name you earlier created.
This way you can virtualize the multiple TP connection in UDS for the CANoe environment.
OR, you do not use diagnostic layer support by CANoe, and you construct the whole message with UDS payload on your data link layer (CAN, FR).
Depends what kind of Data link layer (CAN,FR) and how many comm channels with diag layer you have set.
In Flexray, for example ,you can send multiple diag requests in the same frcycle, if your frschedule provides multiple frslots in dynamic segment which the Diaglayer (or you) can use.

Aggregate results of batch consumer in Camel (for example from SQS)

I'm consuming messages from SQS FIFO queue with maxMessagesPerPoll=5 set.
Currently I'm processing each message individually which is a total waste of resources.
In my case, as we are using FIFO queue and all of those 5 messages are related to the same object, I could process them all toghether.
I though this might be done by using aggregate pattern but I wasn't able to get any results.
My consumer route looks like this:
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.process(exchange -> {
// process the message
})
I believe it should be possible to do something like this
from("aws-sqs://my-queue?maxMessagesPerPoll=5&messageGroupIdStrategy=usePropertyValue")
.aggregate(const(true), new GroupedExchangeAggregationStrategy())
.completionFromBatchConsumer()
.process(exchange -> {
// process ALL messages together as I now have a list of all exchanges
})
but the processor is never invoked.
Second thing:
If I'm able to make this work, when does ACK is sent to SQS? When each individual message is processed or when the aggregate process finishes? I hope the latter
When the processor is not called, the aggregator probably still waits for new messages to aggregate.
You could try to use completionSize(5) instead of completionFromBatchConsumer() for a test. If this works, the batch completion definition is the problem.
For the ACK against the broker: unfortunately no. I think the message is commited when it arrives at the aggregator.
The Camel aggregator component is a "stateful" component and therefore it must end the current transaction.
For this reason you can equip such components with persistent repositories to avoid data loss when the process is killed. In such a scenario the already aggregated messages would obviously be lost if you don't have a persistent repository attached.
The problem lies in GroupedExchangeAggregationStrategy
When I use this strategy, the output is an "array" of all exchanges. This means that the exchange that comes to the completion predicate no longer has the initial properties. Instead it has CamelGroupedExchange and CamelAggregatedSize which makes no use for the completionFromBatchConsumer()
As I don't actually need all exchanges being aggregated, it's enough to use GroupedBodyAggregationStrategy. Then exchange properties will remain as in the original exchange and just the body will contain an "array"
Another solution would be to use completionSize(Predicate predicate) and use a custom predicate that extracts necessary value from groupped exchanges.

x number of threads sending data to Server for displaying output on GUI

I have developed a single server/multiple client TCP Application.
The client consists of x number of threads each thread doing processing on its own data and then sending the data over TCP socket to the Server for displaying.
The Server is basically a GUI having a window. Server receves data from the client and displays it.
Now, the problem is that since there are 40 threads inside the client and each thread wants to send data, how can I achieve this using one connected socket?
My Suggestion:
My approach was to create a data structure inside each of the 40 threads in which data to be sent will be maintained. A separate Send Thread with one connected socket on client side is then created. This thread will read data from data structure of first thread, send it over the socket and then read the data from second thread and so on.
Confusions:
but I am not sure how would this be implemented as I am new to all this? :( What if a thread is writing to data structure and the Send Thread tries to read the data at the same time. I am familiar with mutex, critical section etc but that sounds too complex for my simple application.
Any other suggestions/comments other than my own suggestion are welcome.
If you think my own approach is correct then please help me solving my confusions that I mentioned above.
Thanks a lot in advance :)
Edit:
Can I put I timer on Send Thread and after a specific time the Send Thread suspends thread#1(so that it can access its data structure without any synchronization issues), reads data from its data structure, sends it over the tcp Socket, and resumes Thread#1 back, then it suspends Thread#2, reads data from its data structure, sends it over the tcp Socket, and resumes Thread#2 back and so on.
A common approach is to have one thread dedicated to sending the data. The other threads post their data into a shared container (list, deque, etc) and signal the sender thread that data is available. The sender then wakes up and processes whatever data is available.
EDIT:
The gist of it is as follows:
HANDLE data_available_event; // manual reset event; set when queue has data, clear when queue is empty
CRITICAL_SECTION cs; // protect access to data queue
std::deque<std::string> data_to_send;
WorkerThread()
{
while(do_work)
{
std::string data = generate_data()
EnterCriticalSection(&cs);
data_to_send.push_back(data);
SetEvent(data_available_event); // signal sender thread that data is available
LeaveCriticalSection(&cs);
}
}
SenderThread()
{
while(do_work)
{
WaitForSingleObject(data_available_event);
EnterCriticalSection(&cs);
std::string data = data_to_send.front();
data_to_send.pop_front();
if(data_to_send.empty())
{
ResetEvent(data_available_event); // queue is empty; reset event and wait until more data is available
}
LeaveCriticalSection(&cs);
send_data(data);
}
}
This is of course assuming the data can be sent in any order. I use strings only for illustrative purposes; you probably want some kind of custom object that knows how to serialize the data it holds.
Suspending thread#1 so you can access its data strcuture does not avoid synchronization issues. When you suspend it thread#1 could be in the midst of an update to the data, so the socket thread gets part of old data, part of new. That is data corruption.
You need a shared data structure such as a FIFO queue. The worker threads add to the queue, the socket thread removes the oldest item from the queue. All access to this shared queue must be protected with a critical section unless you implement a lock-free queue. (A circular buffer.)
Depending on your application needs, if you implement this queue you might not need the socket thread at all. Just do the dequeueing in the display thread.
There are a couple of ways to achieving it; Luke's idea suffers from race conditions that will still create data corruption
You avoid that by using UDP instead of TCP as the transport protocol. It'd be especially a good choice if you don't mind missing an occasional packet (which is okay for displaying rapidly changing data); it's fantastic for ensuring real-time updates on data where exact history doesn't matter (missing a point in a relatively smooth curve while plotting graphs is okay);
If the data packets are are small and sort of represent a stream then UDP is a great choice. Its benefit increases if you have multiple senders on different systems all displaying on a single screen.

Resources