How to propagate SIGTERM to a child process in a Bash script
Supervisord requires that the programs it is configured to run don’t daemonize themselves. Instead, they should run in the foreground and respond to the stop signal (TERM by default) by properly shutting down. Docker has a similar requirement for the commands specified in CMD and ENTRYPOINT instructions in Dockerfiles. In both cases, only the processes created directly by Supervisord or Docker receive the TERM signal and it is their responsibility to properly stop running child processes.
That is a problem if the actual server process is spawned by a shell script, as is often the case for a Java service:
#!/bin/bash # Prepare the JVM command line ... $JAVA_EXECUTABLE $JAVA_ARGS # Clean up ...
In this case the TERM signal is received by the shell process, but Bash will not forward that signal to the child process. This means that the shell process will stop, but the JVM will continue to run.
If the command that creates the child process is the last one in the shell script, then the problem
can be easily solved using
#!/bin/bash ... exec $JAVA_EXECUTABLE $JAVA_ARGS
Instead of creating a new process, this will replace the shell process by the JVM. In this case the TERM signal is received directly by the JVM and the problem is solved.
Things are more complicated if the shell script needs to perform some cleanup after the JVM
terminates. In this case we have no choice other than to create a child process, but we then need
to find a way to propagate the TERM signal to that child process and to wait for its completion
before executing the cleanup code. Here the
comes to the rescue: it allows to configure a command to be executed when the shell receives
specific signals. However, there is an important restriction:
When Bash receives a signal for which a trap has been set while waiting for a command to complete, the trap will not be executed until the command completes.
This makes it necessary to execute the JVM as a background process (using
&) and to wait for its
completion, so that the shell process can execute the trap while the child process is still running.
A solution often presented on the Web is to write the script as follows, using the
builtin to wait for the completion of the child process:
#!/bin/bash ... trap 'kill -TERM $PID' TERM $JAVA_EXECUTABLE $JAVA_ARGS & PID=$! wait $PID ...
However, this is not correct. To understand why, let’s look at how
wait interact with
When Bash is waiting for an asynchronous command via the
waitbuilt-in, the reception of a signal for which a trap has been set will cause the
waitbuilt-in to return immediately with an exit status greater than 128, immediately after which the trap is executed.
This means that the shell will start executing the instructions following the
(and may even exit) before the child process has terminated. One solution is to wait for the
termination of the child process in the trap:
#!/bin/bash ... trap 'kill -TERM $PID; wait $PID' TERM $JAVA_EXECUTABLE $JAVA_ARGS & PID=$! wait $PID ...
This works because the trap is executed in the same thread as the normal control flow and therefore
suspends the execution of the script. However, the solution is still incomplete if the script needs
to retrieve the exit status of the child process. Normally, this information is
returned by the
Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero.
This works unless
wait is interrupted by a signal, in which case it returns immediately with an
exit status greater than 128 (actually 128 plus the numeric value of the signal, which is 15 for SIGTERM).
One may want to use this information to distinguish between the two cases. However, this approach would be flawed
because the exit status of the child process may itself be greater than 128 (In particular the default signal
handler for SIGTERM causes the process to terminate with exit code 143).
One solution for this problem is to call
#!/bin/bash ... trap 'kill -TERM $PID; wait $PID' TERM $JAVA_EXECUTABLE $JAVA_ARGS & PID=$! wait $PID wait $PID EXIT_STATUS=$? ...
This works because
wait returns immediately (with the exit status of the process) if the process
has already exited. Waiting in the trap is then no longer necessary and the script can be simplified
#!/bin/bash ... trap 'kill -TERM $PID' TERM $JAVA_EXECUTABLE $JAVA_ARGS & PID=$! wait $PID wait $PID EXIT_STATUS=$? ...
However, the solution is still not perfect because it doesn’t handle SIGINT correctly.
When CTRL+C is pressed, the terminal sends a SIGINT to the foreground process group of the
terminal (which in the case considered here comprises the shell process and its child process).
The problem is that Bash configures commands started in the background to ignore SIGINT.
This means that CTRL+C only stops the script, but not the JVM.
To solve this issue and to mimic the behavior of the original script (which simply
$JAVA_EXECUTABLE $JAVA_ARGS in the foreground), it is sufficient to configure the
same trap for the INT signal:
#!/bin/bash ... trap 'kill -TERM $PID' TERM INT $JAVA_EXECUTABLE $JAVA_ARGS & PID=$! wait $PID wait $PID EXIT_STATUS=$? ...
Finally, it may also be a good idea to remove the trap after the first signal has been received or the JVM has stopped for some other reason:
#!/bin/bash ... trap 'kill -TERM $PID' TERM INT $JAVA_EXECUTABLE $JAVA_ARGS & PID=$! wait $PID trap - TERM INT wait $PID EXIT_STATUS=$? ...