My Flow has following structure:
StartNode(InjectNode) -> LongRunningFunction(Custom) -> LogNode(File-Write set to Append)
StartNode: just a Inject node that I use to start the flow
LongRunningFunction: the function creates inside a longer for-loop a signal via
node.send(msg)
for(let i = 0; i < 5000000; i++){
node.send({payload: i});
}
this is the full code of the LongRunningNode. Later I want to calculate some message string based on the counter inside the loop, but problem can be reproduced by above code
The file stays empty and the script stops execution after some time. If I connect a debug node as second output to LongRunningFunction the last console output is around 3236831, because the heap-memory is full
Is there a general problem with the structure of my program?
i even tried abusing context by putting
flow.set("pause", true);
while(flow.get("pause")){
}
into the for loop of enter code hereLongRunningFunction and setting "pause" to false again, when a file watch node triggers.
theoretically, this way it should send only 1 signal at a time and wait until the logging finished.
tldr: it never writes anything to the file and heap crashes
Edit: This may possibly be related to the problem:
https://nodejs.org/es/docs/guides/backpressuring-in-streams/
the text is for node.js, so not sure how this is applies to node-red or how to fix it
The problem with your approach of a long running function is that is never yields back to the node.js event loop. That stalls all other activity in the runtime and nothing can make any progress.
The while loop approach is particularly bad; the only thing the runtime can do is sit in that while loop. Nothing else can happen outside of the while loop to change the context value you are testing.
There are lots of different ways to write long running functions using timers and callbacks that allow the node.js event loop to continue processing other events along side. It all depends on what you actually want to do in your flow.
Related
Sorry, I am not allowed to post my full program - company rules.
I've got a portion of my program that reads a script file and reads in a timer value to the file. I need that value to pause the reading in of the next line from the script file, and the only way I'm aware of doing that is like this:
where the scan from string function retrieves the time in milliseconds and inserts it into the delay function. The problem with this is, it delays my main loop, and I have other timed functions running simultaneously based on previous commands issued from the same script file that may take more time to run. IS there any way to instantiate this, within a case structure, that will pause the reading of the next command, and let the main loop continue?
My script reader has an event handler so that the rest of my program will run without it until called.
This feeds directly into the nested case statements that feeds data directly to the timer. Any help is appreciated!
If I understand correctly:
You have a loop running some tasks all the time without interruption
Sometimes, a button is pressed, which will cause a special task being executed between two iterations of the loop.
A special task is to disable the button for some seconds
This is accomplished by the following simplified code.
When the button is clicked and the command from the file is WAIT 30, the current time plus 30s is written to the shift register, and the button is disabled grayed out. That is, the user can't click it, and he even sees this!
When the button is not clicked within 100ms, the Timeout case is executed. (again and again and again...) There, all the tasks which should run always, live. That case also checks if the current time is greater than the time in the shift register, and re-enables the button again.
I am writing a graphical program for Linux, in C, using Xlib. The problem is simple.
I have an event, that occurs outside of the Xlib event loop, that needs to be handled and will cause some change to what is displayed to the screen. In my case, I am trying to make a cursor blink inside of a textbox. My event loop looks like this:
XEvent event;
while(window.loop_running == 1) {
XNextEvent(window.dis, &event);
event_handler(&window, &event); // This is a custom function to handle events
}
Solution 1a:
My first thought was to create a 2nd loop with pthreads. This 2nd loop could handle the asynchronous event and draw as needed to the screen. However, Xlib, doesn't play nice with multiple threads. Even if I used a mutex to block the event_handler() function, during asynchronous drawing in the 2nd loop, I still periodically had crashes from X. Additionally, if the event loop is not cycled, after about 10 calls from the pthread loop, the program locks up.
Solution 1b:
This could be solved by calling XInitThreads() at the start of my program. However, this causes valgrind to report a memory leak. It seems that there is some memory allocated that is not freed on exit. And calls to XCloseDisplay() just hang if I call XInitThreads(). I still haven't figured out how to destroy and clean-up windows in my program, but that might be better saved for a separate question. Additionally, calling XInitThreads() at the start of my program stops the program from freezing after 10 calls are made from the pthread loop without cycling of the event loop. However, calls to X start to block after about 10 calls from the pthread loop. Things briefly resume after the event loop is cycled, such as by mousing over the window. However, calls quickly start blocking again when in-loop event activity ceases. Interestingly, I noticed that I could replicate this issue in some other programs such as Bluefish. If I open Bluefish, start a cursor in the main textbox, and then mouseout, after about 10 seconds the cursor stops blinking. Clearly this isn't always an issue since things like a video player's display would freeze after some period of no X events being triggered.
Solution 1c:
I can stop the window from freezing by using XSendEvent() to cycle the event loop after drawing is completed from the pthread loop. However, this seems really Hacky. And I can't guarantee that it will work since I don't know at exactly what point, X will stop listening. I haven't been able to determine the root cause of this issue. As I said, it seems to happen after about 10 seconds, but this varies depending on how I change the cycle rate of the blinking cursor. I'm tempted to guess that it is a function of the actual calls to X being made. There are approximately 2 per pixel, per redraw. It has to 1) set the foreground color and 2) draw the pixel from the bitmap buffer to the screen. Currently, my window only supports a resolution of 640x480. Of course, I am just guessing that this can be used to determine the failure point since I really don't know the cause.
Solution 2:
I can drop all of this and re-implement the event loop by polling the event queue with XEventsQueued(), handling them as they come. But I'll be honest, I hate this solution. This is a really hacky solution that will increase the processing power required for this application and increase the the event response latency, since I would want to sleep the thread between polls to prevent just spinning the thread and pegging the CPU core. I am writing this program with the goal of a fast, stable, and lean program.
Does anyone have a solution? It's such a simple and fundamental problem, but I have only seen sample applications that use XNextEvent in an event loop. I haven't found any samples of how to handle out of event loop events. Thanks for the help. I am a brand new member to Stack Overflow. This is my first post. So I apologize if I make a mistake.
User: mosvy
Wrote this comment:
You should use poll() with the fd obtained with ConnectionNumber() and the fds your other events comes on. When the X11 fd is "ready", you process the events with while(XPending()){ XNextEvent(); ... }. Even then, X11 functions which are of the form request/reply (eg XQueryTree) may stall your event loop. The solution is to switch to xcb (where you could split those in their request/reply parts). IMHO xcb is just as ugly and not much better than Xlib, but it's the only thing readily available.
This works great! Thanks for pointing me in the right direction. I wish you would have written that as an answer. I would have accepted it for you.
EDIT: Deleted my previous edit because I later discovered that the problem was a mistake elsewhere in my full program. In fact, the edit was incorrect and the code didn't correctly read in events.
Here is how my event loop looks now. I'll probably reorganize it, to try to clean it up. But as a proof of concept, here:
// window is a custom struct defined and setup elsewhere in my program
// event_handler() is a custom function elsewhere in my program
window.fd = XConnectionNumber(window.dis);
struct pollfd fds;
fds.fd = window.fd;
fds.events = POLLIN;
int poll_ret;
XEvent event;
while(window.loop_running == 1) {
poll_ret = poll(&fds, 1, 10);
if (poll_ret < 0) {
window.loop_running = 0;
} else if(poll_ret > 0) {
while (XPending(window.dis) > 0) {
XNextEvent(window.dis, &event);
event_handler(&window, &event);
}
}
}
I can use poll() to listen for all events using kernel file descriptors, including X events. The file descriptor for X events is obtained with ConnectionNumber(). I also needed to set it to listen to the file descriptor events of type "POLLIN". Poll accepts an array of file descriptors. See the documentation for more information on the schema. When poll() returns, I can check if it timed out or if an event is enqued. I can then act that accordingly.
I believe this will work for custom events if needed. To do that, I will probably just need to set up my own file descriptors to interact with. I'll look into it if needed.
This solution means that I do not need to call XInitThreads() since I am not ever making simultaneous calls to X.
![enter image description here][1]I have a high voltage control VI and I'd like it to increase the output voltage by a user set increment every x number of seconds. At the moment I have a timed sequence outside the main while loop but it never starts. When it's inside the while loop it delays all other functions. I'm afraid I'm such a beginner at this that I can't post a picture yet. All that needs to happen is an increase in voltage by x amount every y seconds. Is there a way to fix this or a better way of doing it? I'm open to suggestions! Thanks!
Eric,
Without seeing the code I am guessing that you have the two loops in series (i.e. the starting of the while loop depends upon an output of the timed loop; this is the only way that one loop might block another). If this is the case, then decouple the two loops so that they are not directly dependent on each other.
If the while loop is dependent on user input, then use an event structure and then pass the new parameters via a queue (this would be your producer-consumer pattern).
Also, get rid of the timed loop and replace with a while loop. The timed loop is only simulated on non-real time machines and it can disrupt determinisitic features of a real-time system. Given that you are looking for sending out a a signal on the order of seconds, it is absolutely not necessary.
Anyways, if I am off base, please throw the code in question up so that we can review it.
Cheers, Matt
i wrote some code that monitors a directory DIR with inotify() and when a file gets moved in DIR i get a .txt output of that file(its an nfcapd file with flows of my network interface). This happens every 5 minutes.
After that, i used Snort's DPX starter kit, with which you can extend Snort by writing your own preprocessor. This preprocessor,like all the others, is just a function that's executed every time a new packet is available. My problem is that i want, when a new file gets exported from my previous code(so every 5 minutes), to read that file inside the preprocessor's function.
So, is there any way of getting the time and executing only if it's the desired time?
if (time is 15:36){
func(output.txt);}
i'm writing in c.
Thanks
You can do something like the following:
#include <time.h>
...
time_t t = time(NULL); //obtain current time in seconds
struct tm broken_time;
localtime_r(&t, &broken_time); // split time into fields
if(broken_time.tm_hour == 15 && broken_time.tm_min == 36) { //perform the check
func(output.txt);
}
Since you're using inotify, I'm assuming your environment supports POSIX signals.
You can use alarm() to raise a signal after a predetermined amount of time has passed and have the appropriate signal handler do whatever work you need to do. It would avoid what I think is going to end up being a very ugly infinite loop in your code.
So, in your case, the function handling SIGALRM would not need to worry what time it was, it would know that a predetermined amount of time has passed by the fact that it was entered. However, you'll need to provide some context that function can access to know what to do, kind of hard to suggest how without seeing your code.
I'm not entirely sure you're going down the right path with this, but using alarm() would probably be the sanest approach given what you described.
I have a program running two threads - they communicate using message queues.
In one thread, I call ioctl() to access the hardware decryptor. The code goes like:
void Decrypt
{
...
..
...
if(<condition 1>)
{.
...
...
retVal = ioctl(...);
comesInHere1++;
}
if(<condition 2>)
{
...
...
retVal = ioctl(...);
comesInHere2++;
}
comesInHere1 and comesInHere2 are used to count the number of times it goes in that particular if loop.
The entire program takes 80 ms to execute. But if I comment out the test variables (comesInHere1, comesInHere2 within the if loops), the execution time increases by 8 ms to 88 ms!
How is that possible? I cant comment out the variables now since it increases the time taken, cant keep them either - will get killed in code review :)
Kindly let me know
Thanks
Cache? It's possible that by adding a bit more data you're moving code to different cache lines that would somehow be placed together, causing thrashing. You could experiment by running on different systems and by adding padding data between variables that are used exclusively in each thread.
What happens if you serialize the processing onto a single core?