I want a button on my web page to abort an ongoing training. I realise that a callback provided to FIT can throw an exception so that it will abort when the epoch or batch ends but is there a way to abort it immediately?
No, there is no way to immediately stop the training process through the Tensorflow.js API. That said, you can spawn your own process to run the training, which you can kill anytime.
Using the Tensorflow.js API
To abort a running training of a model, you can set model.stopTraining to true. Tensorflow.js only checks after each batch if this variable is set. Therefore, there is no way to interrupt the training process directly by using the Tensorflow.js API.
Code Sample
await model.fit(x, y, {
callbacks: {
onBatchEnd: (batch, logs) => { // stop after batch
if (/* ... */) {
model.stopTraining = true;
}
},
onEpochEnd: (epoch, logs) => { // stop after epoch
if (/* ... */) {
model.stopTraining = true;
}
},
},
});
To directly abort the training process without waiting for the batch to finish, you have to kill the process itself.
Killing the training by killing the process
Killing the process means you need to separate your script into two individual parts. One part is the "managing" process keeping track of the first one and the other one does the actual training.
In the browser environment
In the browser you can use a WebWorker to run your training and then call worker.terminate() to kill the running process. However, it is not that easy to start the training process inside a web worker as you need to transfer your data and use an OffscreenCanvas (when using WebGL). Check out this github issue for more information.
Using Node.js
In the Node.js environment, it's easier. You can spawn a process using child_process.spawn and kill it by calling process.kill. You still have to come up with a communication scheme for the processes, but as your training process can read the data itself, the overhead should not be that big. That said, it is sill a very sophisticated task for just stopping the training.
Related
For a C application accessed via CGI-BIN, documentation online for accessing the process and breaking in GDB relies on manipulating the source code (i.e. adding an infinite loop), in order for the process to be available long enough for a developer to attach, exit the loop, and debug.
Is it feasible that a tool could monitor the process list, and attach via GDB, immediately breaking in order for a developer to achieve this without requiring source code changes?
The rough structure of what I have in mind to develop is something along the lines of:
1. My process monitors the process list on the system.
2. A process matching the name of my application, and owner Apache appears in the list.
3. My process immediately performs a 'pgrep' and 'gdb -p' command, then sending a break-point command to pause the process.
4. The developer can then access the process and look at the flow of execution.
Is this feasible as an idea or not possible due to some constraints (i.e. a race condition which may not always be fufilled?)
Is this feasible
Sure: a trivial shell script will do:
while true; do
PID=$(pgrep my_app)
if [[ -n "$PID" ]]; then
gdb -p "$PID"
fi
done
a race condition
The problem is that between pgrep and gdb -p the application may make significant progress, or even run to completion.
The only way to avoid that is to intercept all execve system calls on the system, as Tom Tromey's preattach.stp does.
My AutoIt script automates a mobile game. Sometimes the game will disconnect (due to an error or bad WiFi signal), causing the script to get off track and stop working correctly. I'm trying to test when/if this occurs, to then restart from the beginning.
Func _TestConnection()
$Continue = True
$DisconnectTest = PixelSearch(3146, 760, 3262, 791, 0x125C12, 10) ;search for popup
If Not (#error) Then
_LC(3112, 779) ;click Yes to reconnect
Sleep(10000)
Return $Continue = False
EndIf
EndFunc
This function will return False if an error pop-up is found. However I need to have this checked continuously because it can disconnect at any time.
Even reducing this to one line would help. For example: If _TestConnection() = False Then ExitLoop. But this won't return to where it left off; just exits the loop and continues with the rest of the script.
Another useful command would be If _TestConnection() = False Then Break, but this stops the program from working altogether.
Since you say you want to restart the automation script from the beginning, the easiest way is to have two processes running. The first one launches the automation script, and then without waiting for that to finish, runs the connection testing looping. When the connection fails, terminate the second process, reconnect, and then restart your automation script.
So I have this old, nasty piece of C code that I inherited on this project from a software engineer that has moved on to greener pastures. The good news is... IT RUNS! Even better news is that it appears to be bug free.
The problem is that it was designed to run on a server with a set of start up parameters input on the command line. Now, there is a NEW requirement that this server is reconfigurable (didn't see that one coming...). Basically, if the server receives a command over UDP, it either starts this program, stops it, or restarts it with new start up parameters passed in via the UDP port.
Basically the code that I'm considering using to run the obfuscated program is something like this (sorry I don't have the actual source in front of me, it's 12:48AM and I can't sleep, so I hope the pseudo-code below will suffice):
//my "bad_process_manager"
int manage_process_of_doom() {
while(true) {
if (socket_has_received_data) {
int return_val = ParsePacket(packet_buffer);
// if statement ordering is just for demonstration, the real one isn't as ugly...
if (packet indicates shutdown) {
system("killall bad_process"); // process name is totally unique so I'm good?
} else if (packet indicates restart) {
system("killall bad_process"); // stop old configuration
// start with new parameters that were from UDP packet...
system("./my_bad_process -a new_param1 -b new_param2 &");
} else { // just start
system("./my_bad_process -a new_param1 -b new_param2 &");
}
}
}
So as a result of the system() calls that I have to make, I'm wondering if there's a neater way of doing so without all the system() calls. I want to make sure that I've exhausted all possible options without having to crack open the C file. I'm afraid that actually manipulating all these values on the fly would result in having to rewrite the whole file I've inherited since it was never designed to be configurable while the program is running.
Also, in terms of starting the process, am I correct to assume that throwing the "&" in the system() call will return immediately, just like I would get control of the terminal back if I ran that line from the command line? Finally, is there a way to ensure that stderr (and maybe even stdout) gets printed to the same terminal screen that the "manager" is running on?
Thanks in advance for your help.
What you need from the server:
Ideally your server process that you're controlling should be creating some sort of PID file. Also ideally, this server process should hold an exclusive lock on the PID file as long as it is still running. This allows us to know if the PID file is still valid or the server has died.
Receive shutdown message:
Try to get a lock on the PID file, if it succeeds, you have nothing to kill (the server has died, if you proceed to the kill regardless, you may kill the wrong process), just remove the old PID file.
If the lock fails, read the PID file and do a kill() on the PID, remove the old PID file.
Receive start message:
You'll need to fork() a new process, then choose your flavor of exec() to start the new server process. The server itself should of course recreate its PID file and take a lock on it.
Receive restart message:
Same as Shutdown followed by Start.
I'm using C to scan a directory which contains frames extracted by ffmpeg. Now in the event that the last file is reached during the scan, I need to check for two conditions:
It's the last file as the video duration is over.
It's the last file as ffmpeg terminated abruptly and is no longer able to populate the directory
My C program workflow is like:
while(<there exists files in the directory>)
{
// iterate over them and do something with each frame
}
// coming out of the loop means no more files available...so I need a if condition
if(<check if ffmpeg is stopped>) // <-- need to know what to put inside the condition
{
// start it again
}
else
{
// video is over, nothing more left to do
}
I'm thinking I can do this using Process ID of ffmpeg, but how would I get that info? Any other alternative way of checking if ffpmeg has stopped?
Some metadata
OS : Windows 7
IDE : Dev C++
Language Used : C
You can definitely wait for FFmpeg process to finish. You normally obtain ffmpeg.exe process handle and either wait for it using wait functions, check its GetExitCodeProcess or both.
If you don't have a handle, but you do have a process identifier, OpenProcess will get you the handle.
Of course, you would not have to go into that trouble if you used native Windows APIs where you would be dealing with frames directly, not through files and external processes.
Not sure if this is a way to go in your case, but... in unix world a program that runs in background normally creates a publicly readable file with its process id. And then removes it when finished.
You could create this file from your batch script and then in your C program you could check if a pid from that file is running or not.
I am working on an application where I need to detect a system shutdown.
However, I have not found any reliable way get a notification on this event.
I know that on shutdown, my app will receive a SIGTERM signal followed by a SIGKILL. I want to know if there is any way to query if a SIGTERM is part of a shutdown sequence?
Does any one know if there is a way to query that programmatically (C API)?
As far as I know, the system does not provide any other method to query for an impending shutdown. If it does, that would solve my problem as well. I have been trying out runlevels as well, but change in runlevels seem to be instantaneous and without any prior warnings.
Maybe a little bit late. Yes, you can determine if a SIGTERM is in a shutting down process by invoking the runlevel command. Example:
#!/bin/bash
trap "runlevel >$HOME/run-level; exit 1" term
read line
echo "Input: $line"
save it as, say, term.sh and run it. By executing killall term.sh, you should able to see and investigate the run-level file in your home directory. By executing any of the following:
sudo reboot
sudo halt -p
sudo shutdown -P
and compare the difference in the file. Then you should have the idea on how to do it.
There is no way to determine if a SIGTERM is a part of a shutdown sequence. To detect a shutdown sequence you can either use use rc.d scripts like ereOn and Eric Sepanson suggested or use mechanisms like DBus.
However, from a design point of view it makes no sense to ignore SIGTERM even if it is not part of a shutdown. SIGTERM's primary purpose is to politely ask apps to exit cleanly and it is not likely that someone with enough privileges will issue a SIGTERM if he/she does not want the app to exit.
From man shutdown:
If the time argument is used, 5 minutes before the system goes down
the /etc/nologin file is created to ensure that further logins shall
not be allowed.
So you can test existence of /etc/nologin. It is not optimal, but probably best you can get.
Its a little bit of a hack but if the server is running systemd if you can run
/bin/systemctl list-jobs shutdown.target
... it will report ...
JOB UNIT TYPE STATE
755 shutdown.target start waiting <---- existence means shutting down
1 jobs listed.
... if the server is shutting down or rebooting ( hint: there's a reboot.target if you want to look specifically for that )
You will get No jobs running. if its not being shutdown.
You have to parse the output which is a bit messy as the systemctl doesnt return a different exit code for the two results. But it does seem reasonably reliable. You will need to watch out for a format change in the messages if you update the system however.
Making your application responding differently to some SIGTERM signals than others seems opaque and potentially confusing. It's arguable that you should always respond the same way to a given signal. Adding unusual conditions makes it harder to understand and test application behavior.
Adding an rc script that handles shutdown (by sending a special signal) is a completely standard way to handle such a problem; if this script is installed as part of a standard package (make install or rpm/deb packaging) there should be no worries about control of user machines.
I think I got it.
Source =
https://github.com/mozilla-b2g/busybox/blob/master/miscutils/runlevel.c
I copy part of the code here, just in case the reference disappears.
#include "libbb.h"
...
struct utmp *ut;
char prev;
if (argv[1]) utmpname(argv[1]);
setutent();
while ((ut = getutent()) != NULL) {
if (ut->ut_type == RUN_LVL) {
prev = ut->ut_pid / 256;
if (prev == 0) prev = 'N';
printf("Runlevel: prev=%c current=%c\n", prev, ut->ut_pid % 256);
endutent();
return 0;
}
}
puts("unknown");
see man systemctl, you can determine if the system is shutting down like this:
if [ "`systemctl is-system-running`" = "stopping" ]; then
# Do what you need
fi
this is in bash, but you can do it with 'system' in C
The practical answer to do what you originally wanted is that you check for the shutdown process (e.g ps aux | grep "shutdown -h" ) and then, if you want to be sure you check it's command line arguments and time it was started (e.g. "shutdown -h +240" started at 14:51 will shutdown at 18:51).
In the general case there is from the point of view of the entire system there is no way to do this. There are many different ways a "shutdown" can happen. For example someone can decide to pull the plug in order to hard stop a program that they now has bad/dangerous behaviour at shutdown time or a UPS could first send a SIGHUP and then simply fail. Since such a shutdown can happen suddenly and with no warning anywhere in a system there is no way to be sure that it's okay to keep running after a SIGHUP.
If a process receives SIGHUP you should basically assume that something nastier will follow soon. If you want to do something special and partially ignore SIGHUP then a) you need to coordinate that with whatever program will do the shutdown and b) you need to be ready that if some other system does the shutdown and kills you dead soon after a SIGHUP your software and data will survive. Write out any data you have and only continue writing to append-only files with safe atomic updates.
For your case I'm almost sure your current solution (treat all SIGHUPs as a shutdown) is the correct way to go. If you want to improve things, you should probably add a feature to the shutdown program which does a notify via DBUS or something similar.
When the system shuts down, the rc.d scripts are called.
Maybe you can add a script there that sends some special signal to your program.
However, I doubt you can stop the system shutdown that way.