Angelo Borsotti, November 2009

This document presents how to use signals in userspace Linux applications, showing several kinds of program patterns in C and C++ with them. Alternative (and better) patterns to signals are presented, too.

Signals are positioned in the threaded world, i.e., programs are assumed to be multi-threaded. Even a single-threaded program could call library functions that are multi-threaded, without knowing it, and thus become multi-threaded itself.

Sections in the document present first functionalities, and the code pattern to use to implement them.

Copying

Copyright (C) 2009 Angelo Borsotti.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.

Copyright owner waives GNU Free Documentation License's obligation to include a copy of the licence text if redistributing the covered work or derivatives thereof.

Waiver: this document contain program examples. They are displayed in monospace font. Permission is granted to copy and modify them for inclusion in programs.

Disclaimer: the same disclaimers of warranty and liability stated in sections 15, 16, and 17 of the GNU General Public License apply to the contents of this document.

Overview

Signals were introduced when there was no threading, and had two purposes: to interrupt the flow of execution when it was impossible to continue it (synchronous signals), to control it (stop, continue, kill), and to carry on activities in parallel with the man ones (e.g., asynchronous I/O).

With the introduction of threading, the last one, and partly also the second one, can be done better using threads.

Signals, while in principle a simple mechanism, in practice have a number of drawbacks and dark corners that make their use rather difficult. One of the drawbacks is that they interrupt the execution of a number of system calls that must then be restarted by the program. Another is that they interrupt execution of normal code asynchronously (instead of doing it at some defined places), and a further one is that their handlers can do a rather restricted set of actions.

This document shows how to perform the activities that traditionally have been done with signals, using signals when necessary, and using threads otherwise.

Semantics

Signals notify a process of the occurrence of some event: the synchronous ones notify about violation of some condition regarding the instructions being executed; the asynchronous ones about some software or hardware event. Typically, applications handle signals in the following way:

• When a sequence of actions is being executed, and a signal is received that indicates an event that can occur in those actions, the sequence is aborted, and some cleanup done, and the sequence either terminated or restarted. E.g., synchronous signals, alarm timers. Typically, the event is an error condition.
• When a parallel action is started (a new process, a non-blocking I/O, etc.), and a signal notified regarding some event occurred in the parallel action, the event is handled. E.g., a child process changing status, handled retrieving the status of the child, a SIGURG notifying out-of-band data in a network connection handling such data, a SIGIO handled transferring data, a SIGWINCH handled setting internal variables holding the window size, etc.
• When an asynchronous external request is made to change the flow of actions of a process, some action is taken to handle it. E.g., a kill request that terminates the process cleaning up the interprocess state.
• When an asynchronous external event is notified, the process must synchronize with it (signals used for inter-process communication). Signals can be used as condition variables (the Hoare's ones), or as events, or sort of: when a process sends another a signal, and the other is waiting, it proceeds. When several are sent, and the other has blocked them, the effect is as if only one has been sent (but realtime signals are instead queued).
• When a sequence of actions is being executed, and it is stopped or continued, some actions may need to be done to proceed seamlessly.
• When a sequence of actions is being executed that it addresses unmapped memory, some actions can be done to proceed seamlessly. (Wnfortunately, this cannot be implemented in a really transparent way.)

Basically, there are three kinds of semantics for signals: the one to abort a sequence of actions, the one to continue seamlessly it, and the one to support a parallel thread of actions. The handling is then done either interrupting a course of actions, or using a parallel course of actions. The latter can be done either using signal handlers (when the operations to be done are very simple), or dedicated threads. Signals have been introduced in Unix long before threads, and allowed some form of threading inside a same process to handle asynchronous events.

Signals need be handled with signal handlers only in few cases:

• Killing (including time supervision) can done with cancellation or testing pending signals (using non-blocking versions of blocking system calls), or for I/O, using pselect(), ppoll() or epoll_pwait().
• Asynchronous I/O and similar events can be handled with dedicated threads (except in the few cases in which the operations to be done are very simple, like, e.g., to retrieve the child exit status, for which signal handlers can be used).
• Synchronous signals.
• Signals used as interprocess events must be handled with a dedicated thread. This is the best and safest way to handle them. (It reduces the complexity of the program.)

Using a thread dedicated to handle signals is in most cases the same as using signal handlers, but without the restriction to run only asynch-signal-safe functions, and the nuisance of having system calls interrupted. There are few cases, however, in which handlers are needed. One is when synchronous signals occur. E.g., when accessing a memory mapped region at an address that is not mapped, we get a SIGSEGV or a SIGBUS, and in the handler we can map it, and return, thus making the code that accesses that memory completely unaware of this. We cannot use another thread because the signal is sent to the offending thread, and, even if we could, we could not restart the offending instruction.

This table reports the suggested handling of signals:

Bear in mind that, while interrupts really interrupt the current process (unless it is running with interrupts off), a signal might not interrupt a process with the same promptness. Therefore, we cannot be sure that a process is scheduled so promptly as to get all signals sent to it. Realtime signals are queued, and then at least are not lost.

Ideally, the signals that need little time to be served could be handled immediately, and the ones needing a long time (i.e., a time greater than the time between two arrivals of a same signal) could be queued by a thread and handled by another. However, signals are not meant to notify processes so frequently as to require such front-end/back-end architecture.

Note that signals are somehow more asynchronous than are interrupts with respect to the program that expects them. The program that uses interrupts to perform I/O has enabled a device to assert them. They occur at a point in time that lies in the interval from the enabling instant onwards (and often within some defined time). A kill signal is more unexpected, even if a program that wants to treat it has defined a handler for it. An event that makes a program terminate abruptly (e.g., a power failure) is even worse because it cannot be handled and can lead to leave data (e.g., disk files) inconsistent. To handle the latter, a stronger form of atomicity must be used, e.g., the writing of a block, that is likely to occur or not to occur even in case of failure. Operations that are made atomic by using locks can be interrupted by a failure.

Signal are simple means to communicating events without creating persistent objects. They have been used for this before threads were introduced in Unix.

This is a list of program patterns in which signals come into play:

• kill the operations done in a block of statements
• time supervise a blocks of statements
• synchronize with other processes, like, e.g., parent and children, and synchronize processes in general
• handle the completion of some previously initiated asynchronous action (e.g., I/O)
• handle notifications of changes or events from some servers
• handle urgent inputs (e.g., on sockets)
• handle unexpected contingencies, like, e.g., power failure
• handle recoverable errors, such as unmapped memory
• request a process to kill, stop, continue, reload configuration, recycle logs, provide information on its functioning, etc.

Note that, in accordance with the table above, most of these cases are handled with threads.

Since the disposition of most signals is to terminate the receiving process, possibly leaving persistent objects in an inconsistent state, signals need be handled. Take into account that a process can send all signals to another, even the ones that are meant to be generated only internally.

The examples

All uses of signals in this documents are shown with working examples. In them, frequently occurring actions are represented as functions that are hyperlinked. In actual programs, they need not be kept as functions; they can instead be inlined.

Error paths are indicated, but no error action is provided. Simple programs could handle them issuing an error message and then terminating; production-quality programs would provide appropriate error reporting and recovery.

Signals basic

This chapter describes how signals behave, and the basic techniques to deal with them.

Overview

Signals are sent either from processes to other processes, or by the system as a result of the violation of some condition (e.g., illegal instructions), or the occurrence of some event (e.g., the expiry of a timer). Sending signals shares some behaviour with interrupts:

• They may trigger the execution of user-defined code.
• When a signal is sent, and the target process is not running, the signal is queued to it. (I.e., it is kept pending.)
• Only one standard signal of a same kind is kept pending. Other signals of the same kind sent while one is still pending are lost. (I.e., standard signals do not guarantee delivery.) A signal remains pending until it is received. (I.e., it is persistent until receipt.) Realtime signals instead are queued, and therefore are not lost.
• Some signals supersede other signals and remove them if pending (e.g., SIGKILL).
• It is up to the target process to be quick enough to receive and process signals so as no to lose them.

The kernel very frequently (sometimes immediately, otherwise each time it switches from kernel mode to user mode, and at least almost every timer interrupt) checks if there are signals to deliver to running processes. If there are pending signals, the kernel takes one and consumes it:

• If a signal can be ignored, it ignores it (i.e., it is not blocked, and its disposition is SIG_IGN or its default action is to ignore) and throws it away.
• If it is not blocked, the kernel resumes the receiving process (when its turn comes) starting the signal handler of the signal (if any, otherwise a default handler).
• If it is blocked, it remains pending.
• At any point in time, each handler has a set of blocked signals attached, that is made of a process set plus an additional set, specific for each handler, plus the signal itself (unless otherwise specified). If a handler for the signal has been registered, the signals in this set become blocked when the handler is invoked.
• When there are many signals pending, the regular (AKA standard) ones are delivered first, in unspecified order, and then the realtime ones, in arrival order, with low-numbered ones taking priority.
• When there are several signals that can be received, they are all served before returning to the normal process execution.
• When a handler returns, if the process was interrupted during a system call, the system call can be restarted (SA_RESTART) or otherwise aborted returning EINTR. Otherwise, the process was not executing system calls, and thus it is restarted.
• Signal handlers can use only a limited set of system calls, and can be interrupted by the handlers of non-blocked signals.

The execution of the normal flow of control can be interrupted by several signals at the same time. It is up to handlers to block or ignore them so as to make the behaviour of a process manageable. E.g., a kill task signal should make a process ignore further signals of the same kind until it has handled the current one (not only in the handler, but in the process normal code also).

Lifetime

Signals are created:

• by processes, that send them
• by the kernel, that generates them
• by a process raising them to itself

Once created, a signal is pending.

When pending signals are given the target process and an action taken, they are delivered.

If delivering makes a signal handler run, the signals is caught.

Signals that are handled without the use of a handler are accepted.

In this document, handled means caught or accepted.

The lifetime of a signal is the interval of time between its creation and its delivery (possibly consisting in ignoring or discarding the signal).

Initial state of signals

The disposition of signals when a process makes an exec() is the default one for all signals that are caught by the creator, and the same as the creator's for the others. In particular, signals that are ignored by the creator are initially ignored by the created process. This allows a shell that does not support job control to set the disposition of the interrupt signal to ignore when it creates background processes so as not to interrupt them when interrupting the foreground process. However, the standard shells of Linux do support job control, and do not send interrupt signals to background jobs.

When the initial disposition (provided by the creator) is to ignore a signal, and a process wants to honor the creator's disposition to ignore signals, this is the scheme to be used:

    struct sigaction sa;
    sa.sa_flags = 0;
    sigsetmost(&sa.sa_mask);           // block most signals
    sigaction(SIGxxx,NULL,&sa);
    if (sa.sa_handler == SIG_DFL){
        sa.sa_handler = sig_int_handler;
        sigaction(SIGxxx,&sa,NULL);
    }

Ignoring signals that are set to be ignored by the creator applies only to the job control ones, or to signals that are agreed upon between the creator and the child. Actually, it should be (and it is) up to a shell to decide what processes to kill when the user interrupts the foreground process, and not to the background processes to take care of it. The suggestion is then to use this only when there is a real need for it.

A process that is created with fork() inherits the disposition of the creator. A process created by clone() can optionally inherit the disposition of the creator. A process created by pthread_swap() can choose to inherit the ignoring of signals from the parent.

It is possible to create a process with all signals blocked. (The mask is inherited in execv() and fork().) A process that is created with all signals unblocked can be killed before it decides to handle them. Since the signals that are ignored in the father are also ignored in the child process, it is also possible to create a process with all signals ignored, such a process can then register handlers as it likes without being terminated by a signal before doing it. For processes created by the shell, there is no way. The shell creates them as it likes. For ones created by a user process, the solution is to fork, then block all signals, call exec, and unblock them in the child. The process then has the possibility of ignoring the ones it wants to (thus discarding any such signals possibly pending). It would be fine to ignore all of them at the beginning, so as to clear the pending ones, but it is unlikely that there are any. Note that the pending signals of the creator are not pending signals for the created one.

Delivering signals

When a signal is sent to a process, a thread that does not block it or a thread that is waiting for it is chosen randomly, and the signal delivered to it. Signals that are directed to specific threads are delivered to them.

When several signal are pending, one is chosen to be delivered with this priority ordering:

1. if there is a standard signal, it is delivered, otherwise
2. if there is a realtime signal, it is delivered; if there are several, the one with the lowest number is delivered
3. if there are several lowest realtime signals, the one that was sent earlier is delivered

Sending signals

Signals can be sent using one of the system calls enlisted below. The API is not uniform:

All other system calls that send signals (raise(), killpg() and tgkill()) are wrappers to these.

Note that there is no way for a thread to send a signal with data to another thread: The one system call that sends signals with data is sigqueue(), that sends signals to processes only. However, if there is only one thread in the process that has registered a signal handler for the signal, that thread will receive the signal and its data.

sigqueue() can send a pointer as value: The pointer is an address in the user space of the sender. It can have a meaning in that of the receiver only if it is the process itself, or it has been created with fork() not followed by an exec(), or it is a thread of the same process as the sender. It does not in all other cases, and likely it causes a SIGSEGV if de-referenced.

sigqueue() allows sending signals with a value attached, not only for realtime signals, but for the others, too. In Linux, all signals carry the sender's pid and the data.

sigqueue() does not interpret 0 as the pid of the current process.

Note that there are no built-in means for a process to check that a process to which it is sending a signal can actually handle it. It would be nice if sending a signal could atomically return an indication telling if it succeeded (e.g., if the receiver has not set the disposition of the signal to ignore). This can be achieved by having the receiver send back another signal to acknowledge the handling.

After sending a signal to a process, always check its return status, since the receiver process could have terminated. Both kill() and sigqueue() return success when sending a signal to a zombie process (that cannot obviously handle it). Do the same when sending signals to threads. pthread_kill() returns ESRCH when sending a signal to a thread that has terminated but not yet joined.

A process can send a signal to another only if its real or effective user ID is the same as the real or saved set-user-ID of the other. The calls above return EPERM when the process does not have the privilege to send the signal.

Some signals are meant to be generated only internally (e.g., SIGSEGV). However, they can also be sent by other processes. The means to fend off them is described here.

Received signals

A process receives signals when:

• another process sends signals to it:
  • any process with kill()
  • an interactive shell when ^Z is introduced: the process receives SIGTSTP
  • same, when ^\ is introduced: it receives SIGQUIT
  • same, when ^C introduced: SIGINT
  • it has been run from within a terminal, and the terminal is closed: SIGHUP is received
  • it has been run from within a terminal, and the user is logged off: SIGCONT is received
• the process itself raises explicitly a signal
• it starts a timer, telling it to send signals
• it executes I/O operations, specifying to receive signals when some event occurs
• it violates one of a number of dynamic conditions, like, e.g., illegal instructions, accesses to memory that is not mapped for the process, etc.

Note that a process does not receive any signal when the computer is suspended, hibernated, or the system shut down (shutdown).

Blocking signals

Signals can be blocked (i.e., left pending, not causing handlers to run until unblocked). Note that blocking signals does not mean preventing a process to handle them: a process can have threads that are waiting for them.

Signals can get blocked because explicitly set in the process signal mask, or because the process is executing a handler registered with sigaction(). When a signal is delivered, the kernel adds the signals specified in sigaction() to the signal mask of the thread to which the signal is delivered, and restores it when the handler terminates.

To block signals:

    sigset_t newmask, oldmask;
    sigsetmost(&newmask);
    sigaddset(&newmask,SIGxxx);      // a signal to block
    sigaddset(&newmask,SIGyyy);      // another signal to block
    pthread_sigmask(SIG_BLOCK,&newmask,&oldmask);
    ...
    // to restore to the previous state:
    pthread_sigmask(SIG_SETMASK,&oldmask,NULL);

Note that the thread signal mask is counter-intuitive: bits are on for the signals that are blocked.

Blocking all signals except for SIGBUS, SIGFPE, SIGILL and is achieved with:

    void sigsetmost(sigset_t* set){
        sigfillset(set);
        sigdelset(set,SIGBUS);
        sigdelset(set,SIGFPE);
        sigdelset(set,SIGILL);
        sigdelset(set,SIGSEGV);
    }

If a thread creates a child process or another thread, it makes it inherit its set of blocked signals.

Pending signals

Signals that are not handled because blocked are pending. Actually, all signals between generation and delivery are pending. There is one set of pending signals, that is made of the process directed signals, and there is a set of pending signals for each thread, which is made of the signals directed to the thread.

sigpending() delivers the set of signals pending for the process united with the ones pending for the current thread. It does not remove pending signals.

To clear a pending signal, sigtimedwait() can be used, and also setting its disposition to ignore (and then resetting it back again). They have a different effect.

A solution is to use sigtimedwait(), that returns immediately if the timeout argument is zero, clearing any pending signals, if any. It removes a thread-specific pending signal, if any, and, if none exists, a process-specific one. If there are both, it removes only the thread-specific one:

    sigset_t sigpend;
    sigemptyset(&sigpend);
    sigaddset(&sigpend,SIGxxx);
    struct timespec ts = {0,0};      // do not wait
    siginfo_t siginfo;
    while (sigtimedwait(&sigpend,&siginfo,&ts) == -1 && errno == EINTR);

Another solution is to set the disposition of the signal to ignore and then back to what it was before. This removes the thread-directed pending signals of all threads and the process-directed ones:

    struct sigaction sa, oldsa;
    sa.sa_flags = 0;
    sa.sa_handler = SIG_IGN;
    sigfillset(&sa.sa_mask);
    sigaction(SIGxxx,&sa,&oldsa);
    sigaction(SIGxxx,&oldsa,&sa);

Note that a thread has no way to remove just a signal that is directed to itself: the first solution removes a process-directed signal if no thread-directed one exists, and the second removes all.

Note that a signal whose disposition is SIG_IGN and that is blocked, is delivered and kept pending. If its disposition is set again to SIG_IGN, it is removed from the pending signals.

Newly created processes by fork() have no pending signals; but exec() makes the process inherit the signals that have been raised between fork() and exec().

Synchronizing sending with receipt

A thread needs to delimit the intervals of time in which it receives signals, and handles them. It depends on the paradigm:

• supervised block: the block itself. At most, the entire process lifetime.
• process-wide signals, like, e.g., SIGTSTP: the entire process lifetime.
• IPC signals: while the thread is waiting.

Delimiting can either be done by unblocking/blocking signals, registering/de-registering handlers, and enabling handlers using per-handler flags. The first two have a similar cost; the latter has almost no cost.

There is a need to make a distinction between internal and external signals. The internal ones are the ones that can occur as a result of some operation made by a thread. E.g., timers, memory accesses, completion of I/O operations. They are used in supervised blocks, and thus they must only be handled as long as a supervised block lasts. The external ones are requests made by other threads and processes. The remaining part of this section deals only with the external ones.

Sending a signal to a receiver thread is a two-step action, and each step has its own caveats:

1. hooking the target (pid, tid)
2. sending the signal.

Hooking the target means ensuring that a process to interact with is there and it is the right one (i.e., not an homonym). This means to wait, or be notified when it starts, and to be notified when it ends. This is not a problem for related processes since fathers are notified about the doings of their children. On the contrary, unrelated processes must communicate their creation. This can be done in several ways, as described below. Making a sender aware of the termination of the receiver is more difficult, because termination can occur unexpectedly. Some solutions are described below.

Communicating creation/existence:

• when senders and receivers are completely separated, the sender can:
  • poll for a receiver with a given name. (It works when there is only one such receiver.) It does not allow running several copies of processes, since their names would clash.
  • poll for a receiver with a given name and attribute (working directory same as sender's, group, invocation argument, controlling terminal, etc.). The attribute must be known to both partners. It allows multiple instantiations.
  • poll for a receiver with a configured name, e.g., contained in an environment variable.
  • inotify wait for a message queue. Message queues have fairly sized names (NAME_MAX: 255), and thus can support multiple installations.
  • inotify wait for a file created in some known directory. It supports multiple installations.
  • use a combination of the ones above, starting from the most specific (e.g., an environment variable) and using some notion of distance between a sender and a receiver: owner, group, system, others.
• when senders and receivers are part of a same application:
  • use a file with a fixed name in a dedicated subdirectory of the user running the application to communicate the receiver pid. The file must be deleted at application startup, created by the receiver, and waited for by senders.

Some of these means require a sender process to poll for an object to come up; they differ in the time spent in polling. Polling for a process with a given name (i.e., scanning repeatedly /proc) is done until the process is created. Polling for a queue is done until the queue is created, creation that can be done by any of the processes that participate to the moot, or even by some startup script; thereafter a sender can wait for a message to come up (telling a receiver has started) without polling. The choice depends on how the application is made, but some means are better than others. E.g., the ones that allow multiple installations are better than the ones that work on a single one; the ones that do not require manual configuration are better than the manual ones.

This is the scheme for waiting for a message queue or a file to be created:

    int fd = inotify_init();                         // create inotify file descriptor
    if (fd == -1){
        ... error
    }
    int wd = inotify_add_watch(fd,path,IN_CREATE);   // path is the name of the directory
    if (wd == -1){
        ... error
    }
    struct inotify_event* ev;
    int evlen = sizeof(struct inotify_event) +
        pathconf("/tmp",_PC_NAME_MAX) + 1;
    ev = malloc(evlen);
    if (ev == NULL){
        ... error
    }
    for (;;){
        ssize_t siz = read(fd,ev,evlen);             // wait for notification
        if (siz == -1){
            if (errno == EINTR) continue;
            ... error
        }
        if (siz == 0) break;                         // end of file
        if ((IN_CREATE & ev->mask) != 0){
            if (strcmp(ev->name,filename) == 0){     // filename is the name of the file to watch
                break;
            }
        }
    }
    free(ev);
    int res = close(fd);
    if (res == -1){
        ... error
    }

Message queues appear as files in the /dev/mqueue directory (to be mounted with mount -t mqueue none /dev/mqueue if it is not already mounted).

This is the scheme for finding a process with a given name:

    int ret = 0;
    DIR* dir = opendir("/proc");                  // open directory of processes
    if (dir == NULL){
        ... error
    }
    int len = offsetof(struct dirent,d_name) +
        pathconf("/proc",_PC_NAME_MAX) + 1;
    struct dirent* entryp;
    entryp = malloc(len);
    struct dirent* res;
    for (;;){                                     // scan it
        if (readdir_r(dir,entryp,&res) != 0){     // get an entry
            ... error
        }
        if (res == NULL) break;
        if (entryp->d_name[0] == '.') continue;   // skip the ones that are not processes
        if (!isdigit(entryp->d_name[0])) continue;
        char buf[256];
        sprintf(buf,"/proc/%s/stat",entryp->d_name);
        FILE* fil = fopen(buf,"r");               // open stat file of this process
        if (fil == NULL){
            ... error
        }
        int pid;
        char* comm = buf;                         // get the command that created it
        fscanf(fil,"%d %s ",&pid,comm);
        int len = strlen(comm);
        comm[len-1] = 0;            // remove )
        comm++;                     // remove (
        fclose(fil);
        if (strcmp(comm,name) == 0){              // name is the name of the desired process
            ret = pid;
            break;
        }
    }
    free(entryp);
    if (closedir(dir) < 0){
        ... error
    }

This scheme can be adapted to look for processes that match some given attributes.

Communicating termination:

• related processes: wait(), waitpid().
• unrelated processes:
  • process connector (see /usr/include/linux/cn_proc)
  • polling with kill()
  • implement the receiver as a process that creates a child and informs the sender when it terminates.
  • watch /proc with inotify() to know when a process dies. Currently, it works only when the watcher and the watched are related processes. This is a Linux bug.

Note that using inotify() to be informed about process termination is not perfect: there is a race because the time from process termination and the waking up of read() is not null, and then the sender could have sent a signal while the receiver is dead. However, this should be not a big problem because this time is much lower than the pid recycle time.

Sending a signal can only be done when the sender is sure that the receiver (pid) is the right one, and the receiver is there to accept the signal (and not to lose it). Since the sender has no cheap means to know that the receiver is there, signals can only be used when receivers are always ready to accept them.

Interrupted system calls

Slow system calls are interrupted by signals. When a system call returns interruption, this means that it has not (entirely) fulfilled its task, because something happened which is not an error in the system call. Then either it is restarted, or it is abandoned because the overall operation has to be killed, or it is ignored. Ignoring is appropriate in cleanup only (e.g., as cleanup of file operations, a file can be closed without checking the result because nothing can further be done if the close fails). Restarting should be the default.

A number of I/O, file-locking, semaphore, etc. system calls (after having been silently interrupted) are restarted if the handler that interrupts them has been registered with the SA_RESTART option. Some 28 slow system calls are never restarted: the ones with timeouts, the ones for signals, and a number of others (see man 7 signal).

If while a process is blocked in a system call, a signal handler runs and executes a longjmp() that makes the process restart its execution from another point, the system call is never restarted. When a signal arrives, and the process is waiting in a slow system call, the system call is aborted, and possibly restarted if and when the handler returns normally. Handlers that are registered with SA_RESTART make a system call restarted when the handler returns normally. Upon jumping, errno is not set to EINTR. However, do take into account that a longjmp() is not asynch-signal-safe, and therefore it is plausible that the Linux documentation does not specify what happens when one is executed in a signal handler.

Some 15 slow system calls are interrupted when the process is stopped with a stop signal and then continued with a SIGCONT (see man 7 signal) making them return with EINTR when the process is continued (i.e., it receives SIGCONT), except for sleep() that returns a nonzero value. If there are several threads that are suspended on such a call, all of them are interrupted. N.B. the entries in man 7 signal that are tagged: "Linux 2.xxx" enlist calls that are interrupted only in that version of the kernel. Since SIGSTOP cannot be blocked, ignored, or handled, and SIGCONT continues anyway the process, EINTR (or interruption) must be tested after system calls.

    flag1 = 0; flag2 = 0;  ...          (1)
    ret = syscall(...);                 (2)
    if (ret == -1 && errno == EINTR){   (3)
       if (flag1) ...                   (4)
       if (flag2) ...
    }                                   (5)

Restarting system calls

The normal way to restart an interrupted system call is simply to execute it again. However, there can be special cases.

In some cases, an interrupted system call has partly fulfilled its task (e.g., sleep(), that returns the remaining time); when it has to be restarted, likely it should be requested to do only the remaining part. This is also the case of system calls with timeouts. Some system calls have a timeout argument that is the absolute time, and some others the relative (elapsed) one. They both denote a point in time beyond which the system call does not wait. If the process is suspended, time still goes on, and, when resumed, if time has passed that point, the system call must fail. Absolute timeouts are simpler to restart, but in theory they could expire after they have been computed and the system call to which they are passed is executed.

All system calls that have a timeout argument, except for select(), clock_nanosleep(), nanosleep() and sleep() do not return the remaining time when interrupted. System calls that require an absolute timeout can simply be restarted. For the ones that requires a relative one, the absolute time must be saved at the time the system call is executed, then the absolute time checked again when it is interrupted, and the difference made to check if the defined time has expired. If it has not, then the remaining timeout is passed, otherwise the system call is aborted.

Restarting with the remaining time is done as follows:

    // deliver in tend the current time plus ts
    void timeend(struct timespec* ts, struct timespec* tend){
        clock_gettime(CLOCK_REALTIME,tend);
        tend->tv_sec += ts->tv_sec;
        tend->tv_nsec += ts->tv_nsec;
        if (tend->tv_nsec > 1000000000){        // normalize
            tend->tv_sec++;
            tend->tv_nsec -= 1000000000;
        }
    }

    // deliver in ts the difference between tend and the current time,
    // return -1 is negative, 0 if null, and > 0 if positive
    int timediff(struct timespec* ts, struct timespec* tend){
        ts->tv_sec = tend->tv_sec;
        ts->tv_nsec = tend->tv_nsec;
        struct timespec tnow;
        clock_gettime(CLOCK_REALTIME,&tnow);
        ts->tv_sec -= tnow.tv_sec;
        ts->tv_nsec -= tnow.tv_nsec;
        if (ts->tv_nsec < 0){                   // normalize
            ts->tv_sec--;
            ts->tv_nsec = 1000000000 + ts->tv_nsec;
        }
        if (ts->tv_sec < 0 || ts->tv_nsec < 0){
            return -1;                          // time expired
        }
        return ts->tv_sec > 0 || ts->tv_nsec > 0;
    }

    ... before a system call
    struct timespec ts = max waiting time
    struct timespec tend;
    timeend(&ts,&tend);
    ...
    syscall(...);
    if (errno == EINTR){
        if (timediff(&ts,&tend) > 0) restart the syscall
    }

It is possible to implement a library of functions that wrap the 28 system calls that restart them when interrupted. However, it is not advisable to use the same function names as the interrupted ones (and link the library before the standard libraries), because it would impair the implementation of task kill with signals, because kill requests must be tested before restarting.

Non-event-driven signal handling

Blocking signals and testing if they are pending allows to use signals in a non-event-driven way.

This is one of the solutions of the kill problem.

Waiting for signals

Most signals can be treated having a thread that processes them. This allows performing I/O and all other operations that are not allowed in a signal handler. Realtime signals are queued by the kernel. (There is a limit to the signals queued, that can be changed with setrtlimit().) Concerning the others, queuing is needed only when they have low inter-arrival times, that sometimes are lower than processing times, which is seldom the case. Otherwise, there is a need to use a paradigm that is similar to that of drivers: having a top-half and a bottom half. By far, the simplest way to implement it is to have a thread that catches signals and queues them, and another that processes them. If the speed of the former is not sufficient, queuing must be done in a signal handler, but synchronization with the consumer thread is quite difficult because there are no system calls that can be used to do it. Luckily, in practice there is no need for it.

The codes of realtime signals are between SIGRTMIN+3 and SIGRTMAX (the first three are used by Linux). Their default disposition is to terminate the process.

To wait for a signal:

To wait for a signal, sigwait() is the best; to wait and get the attached data, the best is to make a loop with sigwaitinfo(), cycling when it returns EINTR.

sigwait() does not run handlers, and does not need a handler registered for the signals it is waiting on. It is as if it changed the disposition of such signals to be accepted by it. It overrides also the "ignore" (SIG_IGN) disposition. However, this seems to be a borderline case, and thus it is better not to have the disposition set to "ignore" for signals to be waited on.

N.B. sigwait() is meant to be called with signals blocked, and returns also with signals blocked: it allows to use them like events. If a signal that is not in the set passed to a sigwait() call occurs while the thread is suspended in it, the signal is handled according to its disposition.

siguspend() is similar to sigwait(), but it is a bit twisted: its argument denotes the complement of the signals to wait for. Moreover, the signals to wait for must have an handler. Otherwise, the default disposition is used, which is mostly to terminate the process.

This snippet waits for a signal to come. While waiting for the signal, the other signals are served.

    sigset_t mask, oldmask;
    sigemptyset(&mask);
    sigaddset(&mask,SIGxxx);             // signal(s) to wait for
    sigprocmask(SIG_BLOCK,&mask,&oldmask);
    int sig;                             // return signal that makes sigwait return
    sigwait(&mask,&sig);
    sigprocmask(SIG_SETMASK,&oldmask,NULL);
    ... process resumed

To devote a thread to handle a signal, a handler can be registered that re-sends the signal to the designated thread if it is caught by the wrong one:

    void* thread(void* data){
        increasePriority();                        // increase thread priority
        sigset_t mask;
        sigsetmost(&mask);                         // block most signals for this thread
        pthread_sigmask(SIG_BLOCK,&mask,NULL);
        sigemptyset(&mask);
        sigaddset(&mask,SIGxxx);                   // signal(s) to wait for
        int sig;
        sigwait(&mask,&sig);                       // accept the signal
        ...
        return NULL;
    }

    static pthread_t th;
    static void handler(int sig){
        if (!pthread_equal(pthread_self(),th)){    // it has been caught by the wrong thread
            pthread_kill(th,SIGxxx);               // redirect it
        }
    }

    int main(int argc, char* argv[]){
        ...
        struct sigaction sa;                       // register the handler
        sa.sa_flags = SA_RESTART;
        sa.sa_handler = handler;
        sigemptyset(&sa.sa_mask);
        sigaction(SIGxxx,&sa,NULL);
        int res = pthread_create(&th,NULL,&thread,NULL);  // create the signal thread
        if (res != 0){
            ... error
        }
        ... here the signal is re-sent to the signal thread
    }

This solution has a drawback, though. The thread that catches the signal, which could be any, could be executing a slow system call. Some slow system calls are never restarted when interrupted by a handler. They must then be restarted by the thread code. N.B. some pthreads functions are called in the handler. They are not in the list of the asynch-signal-safe ones, but are actually so in the Linux implementation.

There is another solution, in which the signal is blocked by all threads, except the one devoted to handle it so that the signal is delivered to it. The easiest way to do it is to block signals in the main thread before creating the others. But in so doing, any thread that creates a child with a fork() creates a child with that signal blocked, unless it unblocks it in the child.

    void* thread(void* data){
        increasePriority();                        // increase thread priority
        sigset_t mask;
        sigemptyset(&mask);
        sigaddset(&mask,SIGxxx);
        int sig;
        sigwait(&mask,&sig);
        ...
        return NULL;
    }

    int main(int argc, char* argv[]){
        sigset_t mask;
        sigsetmost(&mask);                         // block most signals for all threads
        pthread_sigmask(SIG_BLOCK,&mask,NULL);
        pthread_t th;
        int res = pthread_create(&th,NULL,&thread,NULL);
        if (res != 0){
           ... error
        }
        ... a fork here or in a thread creates a child with the signal blocked
    }

When dedicating a thread to handle signals, increase its priority:

    void increasePriority(){    
        int error;
        pthread_attr_t highprio;
        struct sched_param param;
        int policy;
        if ((error = pthread_attr_init(&highprio)) ||        // get current priority
            (error = pthread_attr_getschedparam(&highprio,&param)) ||
            (error = pthread_attr_getschedpolicy(&highprio,&policy))){
            ... error
        }
        if (param.sched_priority < sched_get_priority_max(policy)){
            param.sched_priority++;
            if (error = pthread_attr_setschedparam(&highprio,&param)){
                ... error
            }
        } else {
            ... cannot increase priority of thread
        }
    }

signalfd() creates a file descriptor on which signals can be received; without a need for handlers. It can help when one wants to wait for an operation on a file descriptor (e.g., an I/O operation) and a signal with a select(). There are no races: the signal is not lost since there is an atomic operation that allows to wait and to detect the signal at the same time. This supersedes the practice to register a handler that sends a byte to a pipe so as to get it with a select() or read(). This works if the process waits for some input, and wants to interrupt the wait with a signal (but make sure that there are no races when cancelling the input request when a signal occurs and is got with a select()). The running of a signal handler, instead, works with all the slow primitives. This is a means to make a signal persist until it is handled: that signal, which must be blocked before calling select()or read() is not lost if it occurs when the thread is not suspended, and it is handled at the first select() or read() executed. pselect() provides the same functionality: to wait on file descriptors and also on signals. signalfd() provides a unique interface (i.e., a file descriptor) towards events, which is more flexible than sigwait() because it can be used in select(). The scheme is:

    sigset_t mask, oldmask;
    sigemptyset(&mask);
    sigaddset(&mask,SIGxxx);                     // signals to accept
    sigaddset(&mask, ... );
    pthread_sigmask(SIG_BLOCK,&mask,&oldmask);   // n.b. not needed if already blocked
    int sfd = signalfd(-1,&mask,0);
    if (sfd < 0){
        ... error
    }
 
    struct signalfd_siginfo si;                  // returned info on the signal
    ssize_t res;
    res = read(sfd,&si,sizeof(si));
    if (res < 0){
        ... error
    }
    if (res != sizeof(si)){
        ... error
    }
    if (si.ssi_signo == SIGxxx){
        ... handle signal
    } else if (si.ssi_signo == ...){
        ... handle signal
    }
    close(sfd);
    ... clear any pending signals
    pthread_sigmask(SIG_SETMASK,NULL,&oldmask);

Threads

The use of signals must be placed in the multi-threaded context: programs that are single-threaded must be implemented as if they were multi-threaded. A single-threaded process might call a library function that internally creates threads, and thus becomes multi-threaded itself (possibly without the programmer knowing it).

There are process-directed signals and thread-directed signals. SIGSEGV, SIGFPE, SIGBUS, SIGILL, SIGSYS, and the ones generated with pthread_kill() are directed to specific threads; the others to the process.

Each thread can block incoming signals on a per-signal basis: each thread (including the main one) has its own signal mask. Blocking signals on a per-thread basis is also the way to tell what threads get what signals: a process-directed signal is delivered to a thread chosen between the ones that do not block it or are waiting for it, if any. E.g., if two threads call sigwait() for the same signal, an unspecified one is chosen. Moreover, each thread has its own pending signals.

The functions pthread_sigmask() and sigprocmask() apply to processes, threads, and signal handlers and deliver the same results (even if the documentation states that sigprocmask() has an unspecified behaviour on multi-threaded processes). When a handler runs, the signal mask is the one set by sigaction(), or-ed with that of the thread to which the signal has been delivered.

All threads share the same signal dispositions. E.g., sending SIGKILL to a thread (i.e., from a thread to another) kills the process.

Thread-directed signals can be handled in a thread specific way by threads that wait for them, whereas signals that are handled by signal handlers are treated in the same way even when they are caught by different threads (unless handlers distinguish among interrupted threads).

Signal handlers are a sort of shared resource. This is a problem only for signals that need a handler, and not for the ones that are accepted because we can have several threads waiting for the same signal, and handling them differently. Then, lay out a process-wide plan for using signals, defining what threads handle what process directed signals, and what signals are served with handlers. A thread that calls a library that sets a handler can disrupt other threads that relied on the handlers that were in force before. Then, in libraries, never register handlers.

Realtime signals have no predefined meaning, and thus can be used freely. However, in general they cannot be used as resources that are assigned to threads (e.g., for timers): there are processes that create a variable number of tasks, and with them it would be easy to run out of signals. They must be assigned a process-wide office, or none.

A solution to provide per-thread signal handlers is to use a thread private variable to hold the per-thread function pointer to the handler:

    typedef void (*funct_t) (int sig);
    static volatile __thread funct_t function;   // per-thread function pointer

    void handler(int sig){                       // unique, per-process handler
        if (function == NULL) return;            // actual handler not present
        function(sig);                           // call actual handler
    }

    ... handler setting, e.g., in main()

    struct sigaction sa;
    sa.sa_flags = SA_RESTART;
    sa.sa_handler = handler;
    sigsetmost(&sa.sa_mask);
    sigaction(SIGxxx,&sa,NULL);

    ... in a thread
    function = myhandler;                        // set the handler
    ... some action
    function = NULL;                             // reset it

Note that the (process-wide) handler is set only once, and all threads share its registration flags, and also the set of signals that are blocked in it (and thus in all per-thread handlers). This is not much of a restriction since these settings are quite common. Threads can also have nested sections in which they save and restore and set new handlers. Note also that the per-thread handler runs in the context of the thread that caught it originally (e.g., pthread_self() denotes the original thread, as well as thread private variables). This solution then applies to thread directed signals that need a thread specific signal handler, and not to process directed signals.

Note also that this does not work with existing libraries that set handlers (that would not work anyway since they conflict with other uses of the same signals, unless we redefine sigaction() with a library that is linked before libc.)

While in theory it would be preferable to let each thread have its own handlers, in practice not having this is not a big restriction since handlers are not that much used.

Threads execute simultaneously with other threads, also when they have caught a signal, and are thus executing a signal handler. Therefore, several simultaneous executions of a same signal handler can exist even if the handler has been registered with no SA_NODEFER flag.

Signal Handlers

Signal handlers may run at any time (asynchronously) when signals are unblocked in a piece of program. Some system calls (e.g., pselect(), ppool(), sigwait()) are called with signals blocked; they unblock signals only when waiting. In this case, handlers are not run asynchronously since they can be invoked only in well defined places in the program. When signals are delivered, if more than one signal is pending, the system can (and usually does) run all the handlers of such signals before returning to the execution of the normal code.

Signal handlers must be registered with sigaction(). The sigaction struct argument can be reused: it conveys only data to a sigaction() call, which does not use the struct afterwords. There is a need to initialize all fields (e.g., the mask of blocked signals) before passing it. Example:

        void handler(int signo){ ...}
        struct sigaction sa;
        sa.sa_flags = ...;               // any combination of flags
        sa.sa_handler = handler;
        sigemptyset(&sa.sa_mask);
        sigaddset(&sa.sa_mask,SIGxxx);   // signals to block in handler
        sigaction(SIGxxx,&sa,NULL);

Or:

        void handler(int signo, siginfo_t * si, void * context){ ...}
        struct sigaction sa;
        sa.sa_flags = SA_SIGINFO | ...;  // or any combination of flags
        sa.sa_sigaction = handler;
        sigemptyset(&sa.sa_mask);
        sigaddset(&sa.sa_mask,SIGxxx);   // signals to block in handler
        sigaction(SIGxxx,&sa,NULL);

It is possible to set a handler for several signals and then use a switch in it to define the actions for all signals handled. It is also possible in it to change the actions for a signal by testing a flag that is set in the program (but care must be paid to avoid races: see this example).

In handlers:

• do not call any function that is not in the list of the asynch-signal-safe ones. However, functions that are system calls (i.e. standard library system calls wrappers) are practically asynch-signal-safe except for the use of errno.
• save and restore errno when some system functions are called (or any other function that changes it). Even better, always save and restore it.
• to change shared flags, declare them as static volatile sig_atomic_t. However, any integer or pointer type can also be used.
• to change any other shared data that are accessed by a handler and the thread it interrupts, block the signal in all pieces of thread code that change such data.
• remember that accesses to variables are guaranteed not to be reordered only when they are all declared volatile. E.g., when a variable is initialized and a flag set to tell it, both must be declared volatile (accesses to volatile and non-volatile variables can be reordered). This pattern works properly also when the variable's type is not sig_atomic_t if the handler accesses it only after testing that it has been initialized. This constraint holds in the thread code only since handlers do not interrupt themselves (unless they are explicitly told to do so).
• declare shared variables that have a meaning for the current thread only as __thread (e.g., jump buffers that are set by a thread and used by its handlers).
• remember that a handler interrupts a thread, but runs simultaneously with all other threads and with the signal handlers run by such threads, and thus it cannot access data that are accessed also by them. I.e., a variable that is shared by a thread and one of its signal handlers must not be shared with other threads, because it ought to be protected by mutexes, but handlers cannot use them. A means to overcome this restriction is to use the __sync* gcc built-ins, or libraries alike, e.g.: libatomic_ops (or atomics in the upcoming C++0x standard when gcc will support it, i.e., some version beyond 4.5).
• block all other signals except for SIGBUS, SIGFPE, SIGILL, and SIGSEGV. (They occur when there are bugs or when inaccessible memory is accessed. When they are blocked, the behaviour of the program is undefined). This allows a handler to access shared data and perform safely complex operations on them, like, e.g., incrementing without bothering about other handlers accessing the same data.
• do not execute long operations, least of all the blocking ones.
• do not call any function that suspends the handler.
• a handler can access per-thread data, declared static volatile __thread sig_atomic_t.

Since it is not stated whether the standard string functions are re-entrant, you should assume that they are not. If you plan to make handlers do something more than barely raising a flag, you may need to build a library at least to manipulate strings to debug the handlers.

Handlers can be told to use an alternate stack when running. This is not much useful in normal cases. (It allows handlers to run also when a stack overflow has occurred, and a SIGSECV or SIGBUS generated.)

Libraries

Library functions that internally need to handle some signal must restore the signal handling when they return. A library function:

• if it registers a signal handler, must remember the old one and restore it (but see below)
• if it blocks signals, must restore the previous mask
• if it handles a signal, must ensure that none are pending upon return

In libraries, refrain from setting a handler, because it could disrupt signal handling in other threads.

Libraries must:

• honor cancellation registering cleanup handlers
• restart the 15 system calls that are interrupted by stop signals, and provide cleanup handlers
• at most disable cancellability, and never enable it

Unblockable and stray signals

The default disposition of many signals is to terminate the process. A process should then register handlers or create threads for all these signals, or block them if it does not want to be terminated from the outside by them. There is a protection mechanism that makes processes receive signals only from other processes with the same real or effective UID. However, there can exist processes that need a stronger protection (e.g., processes that update important data, whose integrity must be guaranteed). This, unfortunately, cannot be fully achieved, because SIGKILL can never be blocked, but it can be to some extent. Let's call stray signals the ones that are sent by a process to another, but were instead meant to be generated only internally to it.

SIGSEGV, SIGBUS, SIGFPE, and SIGILL must not be blocked because the program behaviour is otherwise undefined, unless they are generated with kill(). Even when they are generated by kill(), there is a chance that they are generated also internally. Therefore, they must never be blocked, which means that either they have the default disposition, which is to terminate the process, or they are caught by a handler. In either cases, another process can send them.

Stop signals always interrupt 15 system calls. SIGSTOP cannot be blocked, and SIGTSTP should normally not be blocked.

The suggested solution to cater for all this is:

1. to let stray synchronous signals act as kill signals, and other stray signals be discarded
2. to block in normal code all signals except SIGSEGV, SIGBUS, SIGFPE, and SIGILL
3. to restart in normal code the 15 system calls that are interrupted by stop signals
4. in supervised blocks, restart the system calls that are interrupted when there is no kill request pending
5. in threads that wait for signals other than SIGSEGV, SIGBUS, SIGFPE, and SIGILL, discard the signals when stray

Normal code here means statements that are not in blocks supervised by signals, i.e., ordinary thread code, including cleanup handlers. In cleanup handlers, these system calls should not occur, but should they do so, restart them.

Use of libraries:

• processes that use libraries that do not restart said 15 system calls when interrupted by stop signals may behave badly when stopped. They might be able to overcome this by restarting the piece of code that contains such library calls (if said calls return a suitable indication, and if they can be restarted). It is suggested to block all signals whose handlers return (i.e., terminate normally) so as not to make these libraries behave so also with other signals (besides the stop ones).
• processes that use libraries that restart such system calls unconditionally must block all signals except the ones whose handlers do not return, or the ones for which restarting is acceptable. I.e., they can use them in code that is not in signal supervised blocks.
• general libraries cannot be used in supervised blocks that are implemented with signals (because such libraries do not honor kill requests, or do not follow the constraints of supervised blocks).

Implementing a new library:

• restart the 15 system calls that are interrupted by stop signals, and provide cleanup handlers
• optionally, also restart the 13 additional system calls that are never restarted automatically. (It would make the library a bit more general, since it could be used both in processes that treat stray signals as kill signals, and in the ones that discard them.)

One such library can be used in normal code and also in supervised blocks implemented with threads.

    if ((siginfo->si_code == SI_USER || siginfo->si_code == SI_QUEUE) &&
        siginfo->si_pid != getpid()){
        ... external signal
    } 

Cost of signal functions

This is the approximate cost of signal functions, measured on an Athlon 64 X2 4200, 2.2 Ghz, both in terms of absolute execution time, and number of units, where a unit is the time taken by a memory-to-memory copy:

signal(), sigset(), sigvec(), sigpause(), siginterrupt() (which has the same purpose of SA_RESTART), etc., are old functions, not to be used any more.

Debugging aids

To debug signals, psignal() can be used to display a signal. Standard I/O can be used, but at the risk to making the process abort.

The main program

A main program (the main() function of a process) that does not change the default disposition of signals can be terminated by many signals that are sent by other processes. When this is not the desired behaviour, signals must be blocked, ignored, or handled.

This is the suggested scheme to cope with it. Basically, most signals are blocked throughout the program, and unblocked only when necessary, or are handled by dedicated threads.

The initial state of signals when a process starts is described here.

Killing applications, processes, threads, and tasks

There are a number of cases in which a course of actions need be prematurely terminated (killed):

• killing an application, i.e., a set of processes. E.g., the results of the applications are not needed any more, or the application hangs, or spins wildly.
• killing a single process. E.g., the process hangs, or spins wildly.
• killing a thread. E.g., a program runs a number of parallel threads to find something quickly. The first thread that finds it cancels the others (which are no longer needed).
• killing a task, i.e., a sequence of operations. E.g., a server is processing a request, and its results are not needed any more.

It has often been said that killing processes is not a good practice, but when there is something wrong with a process, the only one thing to do is to kill it, hoping that it has damaged the other processes and data as little as possible. After killing, a verify/repair program can be run to mend damage. Not killing does not solve the problem: the only alternative to kill a process is to reboot the system, unless the process keeps quiet and does no harm.

Graceful killing and forced killing are two opposite concepts. However, both are needed. The former allows stopping something, keeping all data and other resources consistent, and therefore is the preferred one. The latter is the emergency one, which may require some repair to be done, but is anyway better than rebooting.

In Linux, forced killing is fully supported only on processes. I.e., a process can kill another unconditionally, or, even better, can request another to perform graceful termination, and, if it does not, forcibly kill it. Thread killing instead is either collaborative or imperative. There is no way to try the soft one first, and then the hard one. Moreover, a thread can disable cancellation. This means that, when there are parallel activities in a system that can execute a possibly unreliable program, they should be implemented as processes. E.g., some Web browsers open new tabs or windows using processes, so as to be able to kill them, should they get stuck in some plugin or applet. Threads can also run code that is provided dynamically by using dynamic libraries, and would then be fault compartments if they could be forcibly cancelled. Unfortunately, it is not so, and one reason could be that threads share data. Thus, forcibly killing one of them is likely to place the process in an inconsistent state.

Killing implies a temporal relationship between the killer and its victim. With long-standing victims, this is not a problem. E.g., a process that enters an endless loop and stops producing results can be detected by measuring the production rate (and seeing it zero), and then killing. With endless applications such as servers (daemons), there is no problem, either. In other cases, there could be a need to provide some means to make sure that killing at least does not kill something else.

Promptness of killing could be termed kill latency. This is the maximum time elapsed between a kill request and when when a request is honored, like, e.g., the time between you push the brake pedal and when the car stops. I have never seen strong requirements for very low kill latencies (e.g., lower than a second). Perhaps the lowest latency is needed when SIGPWR occurs, which could be due to switching over power supply to a UPS, or the battery of a laptop running low.

• application: the latency is the duration of the longest path in the shutdown graph
• process: similar to applications, but referred to its threads
• thread: 15 μs (roughly) + duration of longest stretch + duration of cleanup handlers
• task: latency depends on the solution, at most the same as that of threads

The killer of a process or an application is a process (possibly an interactive shell). The killer of a thread or a task is (possibly another thread of) its process. Process control is done by having a process that starts, monitors, and kills or restarts other processes.

When killing, some error conditions might be encountered during cleanup. It is much better to continue it rather than terminating the process, because, in so doing, there is a chance to restore properly the application, process, thread, or task state.

Applications, processes, threads, and tasks are objects ranging from the largest to the smallest, in increasing level of granularity. The highest is the task (shortest code), but the thread one is appropriate for most of the cases, except perhaps for time supervision. Very seldom does a short sequence of actions need to be killed.

Needless to say, applications, processes, etc., need be designed, taking killing into account, in order to be killed.

Inconsistent states

When a process terminates, Linux performs some cleanup on the system objects accessed by the process. Note that a process terminates because it aborts spontaneously or is forcibly killed (there being no way to perform graceful kill when processes contain endless loops), or calls exit().

Note that when a process that holds a semaphore lock aborts (or is forcibly killed), the lock is not released. Aside from the inconsistent states due to system objects, there are also ones due to application objects.

Killing an application

When the application is made of a father process that creates all the others, killing the application is the same as killing the father process. When it is made by a set of unrelated processes, there is a need to tally all of them and kill them. The kill request is sent to a process in the application that shuts down all the others in an orderly fashion. The order in which processes are killed depends on the application. In general, there are processes that can be killed independently from others (and thus simultaneously), and others that must be killed before others. This makes up the shutdown graph. The main constraint on it is to maintain consistency of what is external to the application, and not to cause any problem in the application itself. E.g., an application that is made of a pipeline of processes should shut down by killing first the input stage of the pipeline rather than the output stage.

Killing a process

Process killing depends on what the process does. In any case, a control thread has to be provided that waits for a kill signal:

• when there is a known way to reset the interprocess state: make the control thread reset it and execute an exit(), or make the main register an atexit() handler and the control thread execute only an exit(). This can be done when the main thread or any other do not have any races in changing the state concurrently with the control thread.
• otherwise, make the control thread kill the main thread gracefully with pthread_cancel() (which in turn kills all its threads and processes), wait for a given time and then forcibly exit the process. This is the general solution.

Resetting the interprocess state

This applies when there is a known and safe (from races) way to reset the interprocess state. The scheme is:

    void* controlthread(void* data){
        increasePriority();                         // increase thread priority
        sigset_t mask;
        sigemptyset(&mask);
        sigaddset(&mask,SIGINT);
        int sig;
        sigwait(&mask,&sig);
        ... cleanup
        exit(EXIT_FAILURE);
    }
    int main(int argc, char* argv[]){
        sigset_t mask;
        sigsetmost(&mask);                          // block most signals
        pthread_sigmask(SIG_BLOCK,&mask,NULL);
        pthread_t cth;
        pthread_create(&cth,NULL,&controlthread,NULL);
        pthread_detach(cth);
        ... do the job
    }

General process kill

The process to kill can be either a child or an unrelated process. Processes that were created (as children) can be re-parented, becoming then unrelated.

To kill a process, first, graceful kill must be attempted, and then forced kill.

The processes that provide graceful kill must be killed using the means they provide, which could be sending a signal, or a message, or any other means of interprocess communication. The scheme described here makes use of a SIGINT signal (but any other signal can be used, instead). If the means to kill a process is not documented, you can try sending it a signal such as SIGINT, SIGTERM, SIGQUIT, SIGABRT, or at worst SIGKILL, or even all of them in sequence.

Graceful process kill is done by cancelling its main thread, which in turn undoes what can be undone, and in particular cancels the threads that are alive at that point (and that have something to clean up), and the created processes, too. However, a process kill function that retrieves all threads and cancels them would be handy to spare to reckon them. It could be used in simple programs as a convenience function, when threads can be cancelled in any order. But this needs to retrieve all the threads of a process, which is not possible. Moreover, if threads can be killed in any order, then probably they have little to clean up, and in such a case there is no need to kill them: they disappear at process exit. A similar reasoning applies to child processes.

Scheme of killer

The snippet of code that kills a child process is:

    {
        if (kill(pid,sig) == -1){   // pid: the one of the child to kill
            ... error
        }
        int i;
        for (i = 0; i < 3; i++){    // wait for the child to terminate
            pid_t pid = waitpid(pid,NULL,WNOHANG);
            if (pid == -1){
               ... error
            }
            if (pid != 0) goto killed;
            struct timespec ts = {1,0};
            if (nanosleep(&ts,NULL)) break;
        }
        kill(pid,SIGKILL);          // forcibly kill
        pid_t pid = waitpid(pid,NULL,0);
    } // killed
    killed:

In this example, the victim allows itself to be killed with a signal sig. Moreover, it is a related process, and thus must be waited for. Waiting for its completion is done with waitpid(). This also allows detection of its termination. If it were an unrelated process, it could still be killed by sending it a signal, but its termination must be detected, either probing it with kill(pid,0) or waiting for some reply. When several children have been created, they can be killed simultaneously. See here.

Scheme of victim

The scheme is:

    static pthread_t mainth;                    // tid of main thread
    static sem_t contrReady;                    // to wait until control thread ready
    static pid_t killer;                        // pid of killer process
    static pthread_t cth;                       // tid of control thread
    volatile sig_atomic_t kill_requested = 0;   // tell cleanup handlers external request
    volatile __thread void* kill_thread = NULL; // tell synchhandler to kill thread only
    volatile __thread sig_atomic_t failing = 0; // tell cleanup handlers thread is in error
    static __thread failure_t* failure = NULL;  // pointer to exception object

    // kill the process and all others in its group
    void forceKill(){
        kill(killer,SIGCHLD);                   // send reply before suicide (kill process group)
        struct sigaction sa;                    // do not generate zombies
        sa.sa_flags = 0;
        sa.sa_handler = SIG_IGN;
        sigemptyset(&sa.sa_mask);
        sigaction(SIGCHLD,&sa,NULL);
        kill(0,SIGKILL);                        // kill all the process group
    }

    // control thread to handle kill signals
    static int kill_underway = 0;               // to discard a SIGINT during killing
    void* controlthread(void* data){
        increasePriority();                     // increase thread priority
        sigset_t mask;
        sigsetmost(&mask);                      // block most signals for this thread
        pthread_sigmask(SIG_SETMASK,&mask,NULL);
        sigemptyset(&mask);                     // wait for kill signals
        sigaddset(&mask,SIGINT);
        sigaddset(&mask,SIGTERM);
        sigaddset(&mask,SIGQUIT);
        sigaddset(&mask,SIGABRT);
        sigaddset(&mask,SIGPWR);
        sigaddset(&mask,SIGXCPU);
        // possibly others
        int sig = 0;
        sem_post(&contrReady);                  // reached the fork-safe point
        int beingkilled = 0;
        for (;;){
            siginfo_t siginfo;
            sig = sigwaitinfo(&mask,&siginfo);
            if (sig == -1 && errno == EINTR){   // stop signals interrupt it
                continue;
            }
            switch (sig){
            case SIGINT: case SIGTERM: case SIGQUIT: case SIGABRT:
            case SIGPWR: case SIGXCPU:
                if (beingkilled) continue;
                beingkilled = 1;
                killer = siginfo.si_pid;              // all signals carry the sender
                if ((siginfo.si_code == SI_USER ||
                    siginfo.si_code == SI_QUEUE) &&
                    siginfo.si_pid != getpid()){      // external
                    kill_requested = 1;               // let cleanup handlers know it
                }
                pthread_cancel(mainth) != 0);         // cancel gracefully main thread
                struct timespec ts = {(int)data,0};
                nanosleep(&ts,NULL);                  // wait for graceful kill to happen
                forceKill();                          // kill the whole process group
            }
        }
    }

    // kill the process because of an error
    void abortprocess(){
        failing = 1;
        pthread_kill(cth,SIGINT);             // kill process
        pause();
        // no return possible here, wait to be cancelled
    }

    // catch synchronous signals
    static int synchsiggot = 0;               // to rekon signal occurred
    static void synchhandler(int signo, siginfo_t* info, void* context){
        int errno_save = errno;               // save errno
        if ((info->si_code != SI_USER &&
            info->si_code != SI_QUEUE) ||
            info->si_pid == getpid()){        // internal signal
            if (kill_thread != 0){            // thread wants to handle exception
                pthread_exit(PTHREAD_CANCELED);
            } else {
                if (synchsiggot){             // already got one, signal in cleanup
                    forceKill();
                }
                synchsiggot = 1;
                sigset_t sigset;              // unblock signal for cleanup handlers
                sigemptyset(&sigset);
                sigaddset(&sigset,signo);
                pthread_sigmask(SIG_UNBLOCK,&sigset,NULL);
                abortprocess();               // kill the process in error
            }
        } else {                              // external signal
            kill_requested = 1;
            pthread_kill(cth,SIGINT);         // kill process or return
        }
    }

    // first cleanup handler
    void sigcleanup(void* arg){
        kill(killer,SIGCHLD);                 // send reply to killer
        exit(EXIT_FAILURE);
    }

    // kill processes in the specified array
    void cleanupchildren(void* arg){
        pid_t* children = (pid_t*)arg;
        int nc;
        for (nc = 0; children[nc] != -1; nc++);       // determine nr. of children
        int j;
        int k;
        {
             for (j = 0; j < 6; j++){
                int sig;
                switch (j){
                case 0: sig = SIGINT; break;
                case 1: sig = SIGQUIT; break;
                case 2: sig = SIGTERM; break;
                case 3: sig = SIGABRT; break;
                case 4: sig = SIGPWR; break;
                case 5: sig = SIGXCPU; break;
                }
                // kill all processes in array
                for (k = 0; children[k] != -1; k++){
                    if (children[k] == 0) continue;         // done
                    pid_t pid = children[k];
                    if (pid < 0) pid = -pid;
                    kill(pid,sig);
                }
                int i;
                for (i = 0; i < 3; i++){                    // try, say, 3 times
                    for (k = 0; children[k] != -1; k++){
                        if (children[k] == 0) continue;     // done
                        if (children[k] > 0){               // child
                            pid_t pid = waitpid(children[k],NULL,WNOHANG);
                            if (pid > 0){                   // killed
                                children[k] = 0;            // clear its pid in array
                                if (--nc == 0) goto killed; // last
                            }
                        } else {                            // unrelated process
                            if (kill(-children[k],0) == -1){
                                if (errno == ESRCH){        // killed
                                    children[k] = 0;
                                    if (--nc == 0) goto killed;
                                }
                            }
                        }
                    }
                    struct timespec ts = {0,1000000};       // sleep 100ms
                    nanosleep(&ts,NULL);
                }
            }
            struct sigaction sa;                            // do not generate zombies
            sa.sa_flags = 0;
            sa.sa_handler = SIG_IGN;
            sigemptyset(&sa.sa_mask);
            sigaction(SIGCHLD,&sa,NULL);
            for (k = 0; children[k] != -1; k++){            // kill children 
                if (children[k] == 0) continue;             // done
                pid_t pid = children[k];
                if (pid < 0) pid = -pid;
                kill(pid,SIGKILL);                          // forcibly kill
            }
        } // killed
        killed:
    }

    // initialize the handling of signals
    void siginit(){
        mainth = pthread_self();
        sigset_t mask, oldmask;          // block most signals
        sigsetmost(&mask);
        sigdelset(&mask,SIGTSTP);        // unblock stop
        sigdelset(&mask,SIGTTIN);
        sigdelset(&mask,SIGTTOU);
        pthread_sigmask(SIG_BLOCK,&mask,&oldmask);

        struct sigaction sa;             // register handlers for synchronous abort signals
        sa.sa_flags = SA_SIGINFO | SA_RESTART;
        sa.sa_sigaction = synchhandler;
        sigsetmost(&sa.sa_mask);
        int synchsigs[] = {SIGBUS,SIGILL,SIGFPE};
        int i;
        for (i = 0; i < sizeof(synchsigs)/sizeof(int); i++){
            sigaction(synchsigs[i],&sa,NULL);
        }
        sa.sa_flags |= SA_ONSTACK;        // support threads that want to catch stackoverflow
        sigaction(SIGSEGV,&sa,NULL);
    }

    // create the control thread to handle signals
    void sigsetup(){
        sem_init(&contrReady,0,0);
        pthread_create(&cth,NULL,&controlthread,(void*)10);
        pthread_detach(cth);
        sem_wait(&contrReady);                      // wait it to reach a fork-safe point
    }

    int main(int argc, char* argv[]){
        siginit();
        pthread_cleanup_push(sigcleanup,NULL);      // register first cleanup handler
        sigsetup();
        ... place here the specific actions of the program
        pthread_cleanup_pop(0);
    }

Notes:

• the victim blocks all signals, except the ones that must never be blocked, and the stop ones.
• the victim creates a control thread that waits for SIGINT (and other kill signals), and, when one comes, performs a pthread_cancel() of the main thread, sleeps a bit, and then sends a reply signal (e.g., SIGCHLD) and sends a SIGKILL to the process group (which kills the current process and all its children). It is possible for a process to use SIGCHLD to notify termination also for non-related processes (providing that the processes that receive them do not attempt to wait for the senders). It could be possible, after having seen that graceful kill does not kill, to try sending a kill signal to all children and wait again before forcibly killing all of them immediately. It could provide some more chances to properly reset the interprocess state, but needs to retrieve the pids of all children, which is not easy, and likely does not pay off the effort. The sleep time could be passed as an argument, when the control thread is created. The victim, after having created the control thread, waits for it to reach the point at which it is ready to accept signals.
• the victim registers a first cleanup handler that sends a reply signal and makes an exit(EXIT_FAILURE). If this is quicker than the control thread, it terminates the process, otherwise the control thread terminates it.
• the victim reacts to cancellation by cancelling all the active threads and processes created by it (and not yet terminated) and undoing all other changes to the local and global state done. In particular, when the victim has created a child, it registers a cleanup handler that kills it (e.g., sends a signal to it), and polls its termination (because wait() does not support time supervision), and, if that does not occur, sends SIGKILL to it. I.e., the killer scheme above. If the victim has created several children, it can kill all of them at once, and wait for each of them. This step also kills children that have been re-parented on purpose. Note that it is up to the process to keep track of threads and processes, and then to kill the ones that are alive.
• the victim must not create new processes in its cleanup code. (Otherwise, killing might never end.)
• the victim can suicide itself with pthread_kill(controlthread,SIGINT).
• forced termination is reported as exit status 137, graceful termination as 1.
• SIGSEGV, SIGBUS, SIGFPE, and SIGILL must be handled, otherwise the program behaviour is undefined. When the process is not executing cleanup handlers, they produce here the same effect as does killing; when the process is executing cleanup, they act as forced kill.
• when a synchronous signal is received and the process killed, it is noted (setting synchsiggot) so as to forcibly kill the process, should it occur again, because this would occur in a cleanup handler. Then the process is killed by sending a request to the control thread, and the current thread paused, waiting to be cancelled. This is better than cancelling the main thread on the spot because the control thread forcibly kills the process if it does not cancel quickly. The current thread cannot be exited or cancelled on the spot, also, because that could cause a race with process kill: when thread exiting is faster than process kill, the main thread could exit before being killed, and, when process kill is faster, thread exiting could not be executed. Since the thread is pausing inside a handler, the signal needs to be unblocked explicitly before pausing; otherwise its cleanup handler would run with the signal blocked, causing immediate process abort, should the signal occur again.
• the code above is amenable to use in a library that can be used in all programs.

If the victim has several children to kill, it can kill all of them and then wait. Suppose there are two children: a quick and a slow one. If it waits for the quick first, and then for the slow, it will lose no time, and the same if it waits for the slow first. However, if the waited one does not terminate, all will be blocked. The technique is then to make a non-blocking waitpid() call for each process and loop over all the processes to kill, noting the ones that have been killed, and allowing a few spare runs, after which forcibly kill.

When a process receives from another (i.e., a shell script) a request to kill, and that process has children, the forced killing of children could be left to the overall killing of the victim. However, forcibly killing a child in place (i.e., in the cleanup handler) allows a more accurate control over the time to wait for its graceful killing with respect to an overall timeout. Moreover, we have to poll its termination anyway so as not to block cleanup. This means that forcibly killing it locally is not an extra cost. Additionally, it allows graceful kill to proceed.

To honor a kill request with cancellation, a thread must be created at the beginning of process execution, that waits for the kill signal (or any other interprocess communication means). Cancellation could also be started from within a signal handler that the main thread registers for it, and that after executing a pthread_cancel() executes a pthread_testcancel(), but this does not allow to forcibly kill the process when the cancellation fails. Note that a process cannot issue a cancellation request to another: it can send a signal, or a message. It is possible to register an handler for the main thread that executes a pthread_kill(), but a handler has all too many restrictions.

When forking off a new process, unblock SIGINT in the child if a process is being created that does not adhere to this scheme (i.e., it does not have a thread that handles SIGINT). Note that when a process forks another, it cannot be sure about what signals are blocked at the time the fork is executed. Therefore, it should initialize the signal mask in the child (it may execute an executable that does not initialize the mask when started).

If a pthread_cleanup_push() is executed, and then a fork() (which is the case here, for example), the child inherits the cleanup handler stack, but has no means to pop the cleanup handler. This can only be solved by calling an exec() after the fork(). Do not mix cancellation and creation of clones. The separation of process creation between forking and exec-ing serves only to allow to set a few things in between, such as unblocking signals, redirecting files, etc. A fork() must almost always be followed by an exec(). The program stretch in between is also a minefield much the same as a signal handler.

When a process is killed, and it knows that it has sent messages to queues that support purging of pending messages (which is a nice feature), the process can purge them as cleanup action. But not all that is done can be undone. Even though, a victim can send messages to the processes it is interworking with to inform them that it is quitting. Such processes can then cleanup the pending transactions they have with the victim. Moreover, as a general rule, processes that are interworking with others must be prepared to detect the disappearance of their partners, and to perform the necessary cleanup of pending transactions, such as discarding messages that have been sent by a process that is quitting. In order to do that, heartbeat messages must be passed between interworking processes (or some process monitor used).

Scheme of creation of a child process:

    pid_t child_pid = fork();
    if (child_pid == (pid_t)-1){
        ... error
    }
    if (child_pid == (pid_t)0){                       // child process
        pthread_sigmask(SIG_SETMASK,&oldmask,NULL);   // initial mask of father
        ... file redirection, etc.
        ... only asynch-signal-safe functions
        if (execl(exe-path,exe-name,arg,NULL) == -1){ // one of the exec functions
            ... error
        }
    }
    pthread_cleanup_push(cleanup,(void*)child_pid);
    ...
    child_pid = waitpid(child_pid,NULL,0);
    pthread_cleanup_pop(0);

Scheme of creation of several children:

    pid_t children[n];                            // n: number of children + 1
    memset(children,-1,sizeof(pid_t)*10);
    pthread_cleanup_push(cleanup,(void*)children);
    pid_t child_pid = fork();                     // create child
    ... exec as above
    children[0] = child_pid;                      // rekon pid, if re-parented store -child_pid
    ... create other children
    // wait for created process to complete
    int k;
    for (k = 0; children[k] != -1; k++){
        if (children[k] > 0){                     // child
            pid_t pid = waitpid(children[k],NULL,0);
            if (pid == -1) ... error
        } else {                                  // unrelated process
            for (;;){
                if (kill(-children[k],0) == -1){
                    if (errno == ESRCH) break;    // terminated
                }
                struct timespec ts = {1,0};
                nanosleep(&ts,NULL);
            }
        }
    }
    pthread_cleanup_pop(0);

Killing a thread

Examples cases in which thread kill is used:

• parallel search: kill is intrinsic in the nature of the algorithm
• server that processes requests, and supports killing of requests
• application that executes some user request in a separate thread, and that allows the user to kill it

Killing a thread can occur during process killing, or as part of the normal operation of a process (e.g., a process that starts a number of threads to make faster an operation using parallelism, like, e.g., a search, can kill threads once the operation has completed, or, e.g., an application with an user interface can use a thread to execute some user requests, that the user could kill). In the latter case, the program should be prepared to cope with killed threads.

A thread can (try to) kill another by executing a pthread_cancel(). The victim:

• if it has cancellation disabled, it does not terminate,
• otherwise, if it has asynchronous cancellation enabled it terminates at any point in its execution,
• otherwise (deferred cancellation) it terminates only at cancellation points (all blocking system calls and pthread_testcancel()),
• if it has already terminated (and possibly not yet been joined), it makes pthread_cancel() return ERSCH,
• it is it executing its cleanup handlers it makes pthread_cancel() return successfully, but makes pthread_join() return the value specified in pthread_exit() if the cleanup handlers were initiated by pthread_exit(), otherwise PTHREAD_CANCELED,
• a second pthread_cancel() has no effect.

The victim can enable, disable and change its cancellation type dynamically. It could disable cancellation during initialization, enable it as deferred thereafter, and enable it as asynchronous in long stretches of computation. A thread that performs only computation and does not change shared data can terminate gracefully with asynchronous cancellation.

The difference between asynchronous and deferred cancellations is that the programmer is sure that with the deferred one, a thread is not interrupted between calls, and thus it can update data without protecting the operation.

Scheme of victims

When a thread changes the system or program state (e.g., it allocates a resource) just before a cancellation point, and should restore that state before the thread is cancelled, it must push a cleanup handler before the cancellation point. When a thread has restored the changed state, it must pop the cleanup handler. This is the paradigm:

    allocate resource
    push cleanup handler
    .. call any function
    release resource
    pop cleanup handler

Changes to the system or program that must be undone are the ones that take the system or program in an inconsistent state, like, e.g., allocation of resources such as files or memory (that cause leaks if not released), partial updates of data, open communications with other programs, etc. Remember that the whole purpose of handling properly killing is to keep the system consistent.

It is difficult for a programmer to remember what functions contain cancellation points. This means that once the state has changed (e.g., a resource has been acquired), and then the program calls system or user functions, the programmer (not knowing if the called functions contain cancellation points), has to set up a cleanup handler. He could also remember the state change (e.g., resource acquired) in some data structure so as to make the cleanup handler release it parametrically. Note also that the called functions can evolve during time, and in a later version contain a cancellation point. This means that functions are potential cancellation points and must be considered as such. Enclosing calls in cleanup handlers push/pop pairs ensures that proper cleanup will be done, should the called functions contain cancellation points or otherwise. Cancellation works if at least all system calls that contain blocking points are cancellation points. This is actually the case (except some few exceptions, see below).

E.g., suppose there is a process that locks a semaphore, does some operations, and then unlocks it:

    sem_wait(sem);
    ...
    ... actions
    ...
    sem_post(sem);

To kill it, nothing must be done if the cancellation request occurs before or during sem_wait(), but the semaphore must be released if it occurs after it and before sem_post(). Of course, in this simple example this does not happen, but if one of the actions contains a cancellation point this can occur. The solution is:

    sem_wait(sem);
    pthread_cleanup_push(fn,args);
    ...
    ... actions
    ...
    pthread_cleanup_pop(0);
    sem_post(sem);

The last two statements can be also swapped. As soon as the state is restored (e.g., resource released), the cleanup handler must be popped. Popping and releasing can be done in any order, unless releasing is blocking, in which case it must be done before popping.

Note that when a system call is interrupted by a signal that lets the program continue, the state does not change, and no cleanup handler must be pushed. This can be detected by testing the EINTR error.

System calls can also be interrupted because they blocked the thread and a cancellation request occurred. As far as I know, there are no means in a cleanup handler to detect if it has been called because of the interruption of a system call or because of a pthread_testcancel().

When cancellability is enabled, pay attention when inserting statements that may contain cancellation points, such as trace statements. A solution could be to disable cancellation in the program parts in which they are used. Another is disabling cancellation in them.

On the other hand, it is convenient to exploit the fact that deferred cancellation does not interrupt a function anywhere: there are parts that need not be protected (either setting a cleanup handler or disabling cancellation).

In order to make a thread responsive to cancellation requests, make sure that in long program stretches there are cancellation points. Some system functions are cancellation points, and some others might be. When in such stretches only the latter are called, insert calls to pthread_testcancel().

Pay attention to the scope of variables: do take into account that cleanup push and pop are a block. Variables that are (re)defined in them are not visible outside. E.g.,

    pthread_cleanup_push(fn,args);
        int res;
        ...
    pthread_cleanup_pop(0);
    if (res) ...     // error

Cancellation of interleaved state changes

When a program makes a sequence of interleaved state changes, e.g.,:

    acquire resource 1
    acquire resource 2
    release resource 1
    release resource 2

the simplest way is to enclose all this in a cleanup handler that releases what resources are actually allocated, reckoning them in some data structure.

Cancellation of condition variables

Calls to wait for condition variables, pthread_cond_wait() and pthread_cond_timedwait(), lock the mutex when they are cancelled. There is a need to register a handler that unlocks it if the application needs to use that mutex again. E.g.,:

    pthread_mutex_lock(mutex);
    pthread_cleanup_push(cleanup,(void*)&mutex); // see below
    ...
    pthread_cond_wait(&cv,&mutex);
    pthread_cleanup_pop(0);

    void cleanup(void* arg){
        pthread_mutex_unlock((pthread_mutex_t*)arg);
    }

Cancellation of mutexes

This is the suggested paradigm for cleanup handlers with mutexes:

    pthread_cleanup_push_defer_np(pthread_mutex_unlock,(void *)&mut);
    pthread_mutex_lock(&mut);
    ...
    pthread_cleanup_pop_restore_np(1);

It works also with asynchronous cancellation.

Cancellation of barriers

POSIX barriers cannot be cancelled. When there is a need to cancel a thread that could wait on a barrier, the barrier must be implemented with semaphores. This is an implementation:

    initialization:
    sem_t arrival;
    sem_t departure;
    int counter = 0;
    int n = number of threads that must come to the barrier

    void cleanup(void* arg){
        int par = (int)arg;
        if (par) counter--;
    }

    void barrier_wait(int n){
        sem_wait(&arrival);
        counter++;
        if (counter < n){
            sem_post(&arrival);
        } else {
            sem_post(&departure);
        }
        pthread_cleanup_push(cleanup,(void*)(counter<n?1:0));
        sem_wait(&departure);
        pthread_cleanup_pop(0);
        counter--;
        if (counter > 0){
           sem_post(&departure);
        } else {
           sem_post(&arrival);
        }
    }

Cancellation of thread join

A thread that has registered a cleanup handler and is cancelled when joining another, e.g.,:

    pthead_cleanup_push(...);
    pthread_join(th);

needs to join the victim in the cleanup handler in order to terminate it properly, and not to leave a zombie thread around.

It is true that detaching it would make its cancellation faster when also the creator is being cancelled, but normally cancellation is fast, and thus joining the cancelled thread is not so a big loss of time. Detaching has a problem: when the main thread exits, all its threads are terminated at once. But then, when one detaches a thread it must be quite sure that the thread terminates well before the main one.

Cancellation of thread creation

See the scheme of killers.

Cancellation of process creation

To cancel a child process, a signal must be sent to it, and then its termination waited for. If that does not happen within a defined amount of time, the process must be forcibly terminated. The amount of time to wait depends on what the process is appointed to do.

See process kill.

Suicide

A thread kills itself executing pthread_exit(), which runs all the cleanup handlers. If it has to kill other threads, it must do it before suiciding, and if it wants to hand its cancellation to another thread, it must then pause() instead of exiting.

Cancellation in C++

A thread that receives a cancellation request (from itself or from another thread) can handle it using exception handling:

    extern "C"
    void* thread(void* arg){
        pthread_cleanup_push(cleanup,NULL);
        try {
            MyObject m;
            ....
        } catch (std::exception&){
            ... handle the standard exception
        } catch (...){
            ... handle cancellation exception
            throw;              // mandatory
        }
        pthread_cleanup_pop(0);
        return NULL;
    }

When a cancellation request is sent to the thread, and a catch-all handler is present, the try block is exited, and the destructors of the objects in its scope called (e.g., MyObject m above); then the handler is executed. The handler must end throwing again the exception. Then the cleanup handlers are executed, if any. I.e. it is as if exception handlers were cleanup handlers. Note that cancellation exceptions terminate threads, i.e., there is no way to recover them and keep threads going on.

Calling pthread_exit() has the same effect as cancellation (except for the value returned by the thread).

Protecting from cancellation

When deferred cancellation is enabled, code that executes only computation, including calls to functions that do not contain cancellation points, is not interrupted by cancellation requests. Remembering what system calls and also system library calls are cancellation points is not simple. The OpenGroup Base Specification enlists a number of calls which are cancellation points (at least in some cases), and a number of others that can be. All of them must be taken as cancellation points. This standard states that implementations must not introduce cancellation points in any other function specified in the standard. To stay on the safe side, it is better to check each function used reading its man pages.

Some synchronization functions, like, e.g., pthread_mutex_lock(), are not cancellation points. They can be used to perform operations that are not interrupted by other threads and also by cancellation (i.e., critical sections). When cancellable calls should be used inside one such critical section, the whole section must be protected disabling cancellation. Note that deferred cancellation leaves to threads the responsibility to honor cancellation requests, and threads should do it as soon as they can, avoiding getting blocked indefinitely or even for long amounts of time. The possibility for threads to use critical sections that are not cancelled in the middle allows them to update safely data (or to perform entirely some sequence of actions) thus preserving consistency of the program state.

To protect a piece of code that contains calls that are cancellation points (or are suspected to be), cancellation can be disabled:

    int oldstate;
    pthread_setcancelstate(PTHREAD_CANCEL_DISABLE,oldstate);
    ... non-cancellable section
    pthread_setcancelstate(oldstate,NULL);

When a function is provided that wraps a library one (e.g., a malloc that performs some additional checks), pay attention to honor the specification of the wrapped one for what concerns cancellability.

This allows to block killing, then do whatever operation we want without bothering to be killed, and then restore the previous state. Note, however, that in stretches of code in which calls that are not cancellation points are used, there is no need to use protection. But sometimes one does not know if a library function contains cancellation points. Suppose you want to implement a sort function and use threads to parallelize it, and use some synchronization calls that are basically blocking, but that used in such a place it would block very little. To make it safe, you could either handle cancellation, or disable it.

Cleanup handlers

During the execution of cleanup handlers, cancellation is disabled, and cannot be enabled (thus pay attention not to enter endless waiting). However, consider that killer and victim threads are cooperating, and then a killer should not issue two cancellation requests to the same thread (the second has no effect, and denotes poor coordination). Once a thread has started executing cleanup handlers, it becomes unresponsive to cancellation requests.

Cleanup handlers:

• can access the automatic variables of the function in which they are pushed.
• are executed before destructors for thread-private data because they can need to access such data.
• are also called when asynchronous cancellation is enabled.
• cannot call pthread_exit().
• have no built-in means to know if they have been started by pthread_exit() or pthread_cancel().
• are unable to change the value returned by pthread_join().

Cleanup handlers are not executed when exit() and _exit() are executed (and neither are per-thread data destructors). This is so because they are devoted to terminate processes. Objects that a process uses to communicate with other processes can be cleaned up with atexit().

Notes

• cleanup handlers push/pop must be in the same lexical scope, and this is checked at compile time. Jumping out of a cleanup handler push/pop block must not be done, and jumping into it too. Do not mix cleanup handlers and longjmp.
• the man pages do not state that pthread_cleanup_push and pthread_cleanup_pop are asynch-cancel-safe. However, they can be used when cancellability is asynchronous. There are asynch-cancel-safe functions, that can be called by a thread (including its cleanup handlers) that has asynchronous cancellation enabled. There is no list of them, but pthread_cancel(), pthread_setcancelstate() and pthread_setcanceltype()are asynch-cancel-safe. Asynch-cancel-safe means that it can be called when asynch cancellation is enabled since it does not cause anything to be corrupted. Functions that are not guaranteed to be asynch-cancel-safe could leave the process in an inconsistent state when cancellation is executed. Therefore, enable asynchronous cancellation when a thread is doing only computation.
• in libraries that need to change cancellability, save it on entry and restore it upon exit. A thread can run normally with asynchronous cancellability enabled, and disable it when calling functions that are not asynch-cancel-safe, but they are so many that it is better the other way round: run with deferred cancellability and enable the asynchronous one only in program stretches that do not call functions (as an alternative to calling pthread_testcancel()). pthread_setcancelstate() is not a cancellation point: this means that a function that does not want to be a cancellation point can really be so.
• a library must at most disable cancellability, and never enable it (a caller may want to disable cancellability before calling a library function so as not to have the thread killed).
• if a thread realizes that it can or should not continue, it can call pthread_exit(), that executes all the cleanup handlers on the stack.
• pthread_cleanup_push_defer_np() and pthread_cleanup_pop_defer_np() push/pop cleanup handlers and also set/restore deferred cancellability.
• do not use mutexes to protect code in which there are blocking calls, use semaphores instead; mutexes are not cancellation points.
• avoid mixing cancellation and signals in a same handler: a thread that supports cancellation must not be interrupted by signals; when it needs to process signals it should wait for them using sigwait().
• pthread_cancel() returns EPERM when issued to a thread that has completed its cleanup handlers, it was joinable, but has not yet joined, and also that it was detached and has terminated. I.e. it returns it when it cannot cancel the thread.
• inside a cleanup handler, no threads should be created. They would need to be cancelled too. Avoid calling there library functions that create threads.
• if the thread to cancel is executing a signal handler, it has the signal that activated the handler blocked.
• changing cancellability type costs the same as 25 memory-to-memory copies.
• cleanup push/pop costs the same as 15.9 memory-to-memory copies.
• a cleanup handler for a function should not call the very same function, unless it has cleaned up the internal state properly before calling it.
• rationale of POSIX cancellation
• reference document on cancellation
• reference document for system interfaces

Scheme of killers

The choices for a creator thread to synchronize with a created one to get its results are:

• join: this is a simple way to synchronize with a thread to get its results; it is somehow inflexible (e.g., there is no time supervised or multiple join). It should be used when there is a need to return a value from a thread, or to be sure that it has completed all cleanup handlers, and to make sure that it does not run any more. However, it does not guarantee that the memory used by the thread has been released when join terminates. Nevertheless, if a program wants to create threads only when the ones that it created before had completed their execution and released memory, it has a better chance to have it by joining than detaching. This is the most general and safe solution.
• synchronize with semaphores, barriers, etc. to get the result. If after having provided the result, a thread exits or does not access shared data, it can detach (pthread_detach()), otherwise another synchronization can be done, letting the creator know when it can reuse shared data. Note that this is more flexible than joining because it allows to wait for several threads in parallel.
• when a thread is created to perform some operation that does not have a result (e.g., to supervise something and notify the program when some conditions occur), the thread should run forever, and terminate upon request.

Cancellation does not change the detachable or joinable status of threads (unless a thread changes that itself), and therefore the killer has to wait for the victims as it does normally when it does not cancel them. The best alternative when cancellation is done depends then on what the thread does.

A killer and a victim proceed in parallel, and a victim could terminate independently from a killer. However, there are no races when killing is done because a thread that creates another kills it before joining or synchronizing, or at any time if it runs forever. There must not be threads that end spontaneously without notice.

After thread cancellation, victims continue to run, possibly executing some code before they reach a cancellation point, or executing cleanup handlers. But the killer wanted to kill them for some purpose. How can it know when its purpose has been achieved? The purpose could be to make the victims stop changing shared data, or consuming CPU, or memory:

• if the killer needs the victims to stop changing global or shared variables, then it must reach a point in its code in which it no longer uses their thread-id's, and the threads no longer access shared variables. The latter can be achieved using some semaphores or barriers, or nothing if the thread does not access shared variables. From that point on, the killer can again access those shared data, without bothering about the victims, that can then die with their own times. Think, e.g., to a program that creates threads to perform some search in parallel, and once one has found the result it cancels all of them and then needs to create others for another search and the latter need to access some global data, that are accessed also by the former.
• if the killer does not want to run new threads or allocate new resources until the ones used by the cancelled threads are released, then it should join the cancelled threads.
• if the killer does not want victims to consume CPU time, it can simply detach them and cancel them.
• take into account that tid's of detached or joined threads are (immediately) recycled. Thus, make sure that the thread is alive when cancelling (otherwise you risk to cancel another one). This is quite a problem because when you discover that a thread must be cancelled, you can cancel it, but the thread could have gone in the meantime. The only guarantee that a thread has not gone and its tid reused is when it is joinable. This means that all threads should be joinable if you want to cancel them, except when thread cancellation is done to perform a process cancellation (in which case they can be set to detached since after the cancellation of a thread its tid should no longer be needed), or when no more threads are created, or when the killer has waited for the victims to reach the point described above. Note that a program cannot be sure that new threads will not be created: it could call some library functions that internally create threads.

Treads are closely related objects. Often, killing a created thread does not only require to send it a cancellation request to it, but also to adjust the rest of the program to do without it. E.g., if there are 4 threads that must execute a parallel search for some result, and then wait on a barrier, killing one requires to adjust the parameter of the barrier.

The scheme to cancel a single thread and wait for it is:

    void cleanup(void* arg){
        pthread_t thr = (pthread_t)arg;
        pthread_cancel(thr);               // n.b. no error checking
        pthread_join(thr,NULL);            // no errors can occur
    }

    ... piece of program that creates a thread
    pthread_t th;
    pthread_cleanup_push(cleanup,&th);     // N.B. no cancellation points before creation
    pthread_create(&th,NULL,&thread,NULL); 
    ... do something
    pthread_cleanup_pop(0);

Thread cancellation is done without error checking because the thread could have terminated in the meantime, and cancelling a terminated thread returns an error. The registration of the cleanup handler is done before the creation of the thread because the thread can execute and even cancel its creator. If the registration of the cleanup handler were done after creation, there would be no cleanup handler to execute.

When there are several threads that must be cancelled, the killer could cancel them all, and then wait for all of them. When a program piece creates a variable number of threads and wants to cancel them, it can have a cleanup handler that receives an array of tid's as argument and cancels the ones pointed to by it, and then waits for them all with pthread_join() (that would not sequentialize cancellation and waiting for each thread). The program piece can reckon the thread tid's in an array. The assignment of the return value of a pthread_create() call is done unless the function returns an error (it is not a cancellation point). This means that it is safe to reckon tid's. The scheme for killing threads in parallel and joining them is:

    typedef struct threads_t {
        int       len;                   // number of elements
        pthread_t tids[0];               // their tids
    } threads_t;

    void cleanup(void* arg){
        threads_t* thr = (threads_t*)arg;
        int err;
        int i;
        for (i = 0; i < thr->len; i++){
            pthread_cancel(thr->tids[i]);     // n.b. no error checking
        }
        for (i = 0; i < thr->len; i++){
            pthread_join(thr->tids[i],NULL);  // no errors can occur
        }
        free(thr);
    }

    ... piece of program that creates several threads
    threads_t* thr = malloc(sizeof(threads_t)+sizeof(pthread_t)*n);  // n = number of threads
    thr->len = 0;
    pthread_cleanup_push(cleanup,(void*)thr);
    ...
    pthread_create(&thr->tids[thr->len++],NULL,NULL);
    ... other threads created
    pthread_cleanup_pop(0);

Note that thread cancellation is done without checking errors. This is so because when the cleanup handler is called, some of the threads might have terminated. Cancelling a terminated threads returns an error. However, attempting a cancellation is simpler than reckoning what threads are alive.

Killing a task

A task is supervised block. Supervised blocks are blocks of code in which a signal can occur, that must be handled terminating the block and cleaning up the program state. In languages such as C++ and Java, supervised blocks are bracketed statements that are made of a body, and a number of sections (exception handlers) to which control is implicitly (and sometimes explicitly) transferred when an event occurs. After executing them, the block is terminated. In C there is no such construct, and therefore, events are handled by checking their occurrence after any statement on which they can occur, or jumping at the end of the block, or cancelling the current thread when it coincides with the task.

A supervised block is meaningful when the supervised event can be generated during the execution of some actions occurring in the block, not with a signal that can occur at any time, even before or after the block, except for the case in which we want to protect the block so as to ensure that it performs a transaction (in which case when it is killed and it cannot complete the transaction, it rolls back). A means to ensure that a block is executed entirely is to block signals or to disable cancellation. A process that wants to support graceful kill must enclose all the code in a supervised block (which is done by default, if cancellability is used). As an example, consider time supervision, in which the supervised block starts exactly when a timer is started, and ends when it is stopped.

This is the scheme of a supervised block with some calls. E.g., a sequence of resources is allocated and the same released in reverse ordering, releasing that acts also as cleanup code (when a kill request is detected, control is transferred to the proper place in the cleanup code). Cleanup is protected against interruption, be it executed as the last part of normal operation, or because of killing:

    begin block
        ...
        allocate resource 1
        if killed goto r1
        ...
        allocate resource 2
        if killed goto r2
        ...
        disable killing           // cleanup code
        r2: release resource r2
        r1: release resource r1
        enable killing
    end block

Supervised blocks can be implemented by:

1. blocking signals and calling system calls that unblock them and atomically suspend the process. There are four such calls: pselect(), ppoll(), epoll_pwait() and read() on a signalfd(). A signal handler is also needed.
2. polling: never use blocking system calls (or functions that contain them), and instead use time supervised system calls, or calls that do not hang. Repeatedly invoke them and each time test if there are pending kill requests. A signal handler is also needed.
3. having a handler that executes a siglongjmp(), when the block contains only asynch-signal-safe function calls that do not have any cleanup to be done (i.e., that do not allocate resources, and do not leave inconsistent data if aborted). Note that these conditions are similar to the ones of asynchronous cancellation. Note also that using this ad-hoc solution is risky because the program could be modified in the future, forgetting these conditions. Enclosing a big bunch of code in a supervised block in which a signal performs a transfer of control to the end of the block is not a good idea because, even if non-re-entrant functions could be safely interrupted (which is not), no cleanup could be done, leaving the program in a bad state. Cooperative killing is instead the solution.
4. for short, non-blocking tasks, do nothing, and for the long ones (long enough to make killing meaningful), create a thread and cancel it when the supervised event occurs. The creation and joining of a thread costs the time of 10600 memory-to-memory copies. For recurring tasks (e.g., a server that processes requests), keep a thread dedicated to process tasks in sequence. To kill the current task, kill the thread, and then create it anew.
5. when a task can run untrusty code (e.g., that can block because of bugs, or that does not support cancellation), a process must be used since it allows forced killing. However, this is a rather uncommon case.

Solution 1 can be used only when there are no slow system calls other than the four ones above. Solution 2 implies polling, solution 3 is not general and should be avoided, and solution 4 is expensive. When solutions 1, 2 and 3, cannot be used, solution 4 is the only applicable one.

Solutions 1 and 2 can be used only for time supervision, and other internal signals, and not for synchronous signals (e.g., stack overflow) since they cannot be handled testing them at the next slow syscall aborted, but only terminating the block (e.g., cancelling the thread). Solution 3 can be used for all signals, and solution 4 for any kind of events.

Cleaning up means releasing allocated resources, and undoing what else can be undone. There is then a need to reckon what resources have been allocated at any point in time during the execution of a supervised block. Knowing what resources a process has at any one point in time is in general a good idea: it allows it to tell how much it is using and what. This can be done e.g., keeping a list of open files, etc. Note that in /proc/xxx files there is a list of some resources, but not all. E.g., there is no list of locked semaphores, and of course there are no resources enlisted that are not Linux objects.

When supervised blocks that need a signal handler (albeit empty) are used, make sure that the handler is agreed among all threads that need to handle the very same signal, i.e., that there are no two threads needing a different handler.

When supervised blocks are implemented with signals, and the originator of the signal sends process directed signals (e.g., a timer), make sure that there is only one thread at a time in one such supervised block, or that different signals are used. E.g., if there are several threads that need to do something when a signal occurs, they all cannot catch it because only one will get it. Of course, it is possible to have a single thread that accepts that signal and re-sends it to all other threads (but getting all their tid's is not simple, so, there would be a need for a sort of registration for the threads that want to receive it). Note that this needs to use another signal because it is not possible to bounce back the same one.

In supervised blocks that are implemented with signals that are generated internally (e.g., by timers), those signals must be removed if any is pending when the supervision starts (so as not to kill the block should one such signal be pending while there should be none), and also when it terminates (so as not to cause problems to what follows, like, e.g., interrupt system calls should the signal be generated when exiting the block). For signals that are not generated internally, there is no problem because such signals can come at any time, and it makes no difference if they occur immediately before the supervision, or just after.

In signal blocks that are implemented with signals, the handler:

• if the signal can always come from anywhere (i.e., internally from the process and also externally from others): set the kill flag,
• if the signal can only be internal: set the kill flag when the signal is not stray,
• otherwise, have a per-thread flag that controls its behaviour: when the flag is off, set the kill flag always, when it is on set the kill flag only when signals are stray. Each supervised block sets then the flag if it handles internal signals, and clears it otherwise.

In the following, only the first two cases are presented.

To allow killers to identify uniquely the instances of tasks to kill, tasks can be numbered (and task numbers recycled with a long recycle time), and kill requests accompanied by task numbers. This allows to protect against killing tasks implemented with threads, that have reused the tid of terminated threads.

See here an in-depth discussion of the implementation of supervised blocks.

Pselect(), ppoll(), etc.

Here is the scheme:

    static volatile __thread sig_atomic_t killFlag = 0;
    static void handler(int sig){
        killFlag = 1;
    }

    struct sigaction act;            // register handler
    act.sa_handler = handler;
    act.sa_flags = 0;
    sigemptyset(&act.sa_mask);
    sigaction(SIGxxx,&act,NULL);

    // supervised block
    sigset_t mask, oldmask;
    sigemptyset(&mask);
    sigaddset(&mask,SIGxxx);         // block signal. N.B. not needed if already blocked
    pthread_sigmask(SIG_BLOCK,&mask,&oldmask);
    pthread_sigmask(0,NULL,&mask);   // get the previous mask with this signal ..
    sigdelset(&mask,SIGxxx);         // .. unblocked
    clearKill();                     // clear pending signals, only if generated in the block
    ...
    struct timespec ts = {timeout,0};
    for (;;){
        int res = ppoll(&fds,1,&ts,&mask);
        if (res == -1){
            if (errno == EINTR){
                if (killFlag) ... kill the block
                continue;
            } else {
                ... error
            }
        } else if (res == 0){
            ... timedout
        } else {
            ... an event occurred
        }
        break;
    }
    clearKill();                     // clear pending signals, only if generated in the block    
    pthread_sigmask(SIG_SETMASK,&oldmask,NULL);  // restore previous mask

    // clear pending signals
    void clearKill(){
        sigset_t sigpend;
        sigemptyset(&sigpend);
        sigaddset(&sigpend,SIGxxx);
        struct timespec ts = {0,0};           // make sigtimedwait return immediately
        siginfo_t siginfo;                    // remove pending signals, if any
        while (sigtimedwait(&sigpend,&siginfo,&ts) == -1 && errno == EINTR);
    }

Having a signal handler is essential because ppoll() unblocks signals while waiting, and if a signal has the default disposition, it can terminate the process. In the code above, pselect() can be used with the same scheme.

If the convention to keep signals blocked (almost all) in non-supervised (normal) code is followed, there is no need to block the signals and restore the mask upon entry and exit of blocks. However, doing is does no harm.

Polling

To supervise a block of statements with polling:

1. unblock the signals that are devoted to kill the block, and provide for them a handler that sets a kill flag. Clear the kill flag before unblocking the signals. If the signal can legally come only internally from the same process, set the kill flag in the handler only when it is internal.
2. do not use blocking system calls in the block, but their nohanging or timeouted variants. After a system call:
   • if it succeeded, then proceed,
   • otherwise it timed out (errno = ETIMEDOUT or other timeout returns), nohanged (errno = EAGAIN), or interrupted: if the kill flag is set, exit the block, otherwise restart the system call,
   • otherwise it returned an error, exit the block.
3. in a long sequence of actions, test often the kill flag, and kill the block when set.

N.B. the poll loop must sleep for a while at each iteration.

We must test the kill flag often enough to have a responsive kill. This means testing when polling, and if there are long computations, testing it in them too.

This is the scheme of polling:

    static volatile __thread sig_atomic_t killFlag = 0;
    static void handler(int signo, siginfo_t* info, void* context){
        // if signal is stray, do not set the flag
        killFlag = 1;
    }

    // clear a pending kill request
    void clearKill(){
        killFlag = 0;
    }

    // test if kill has been requested
    int killRequested(){
        if (killFlag){
            errno = EINTR;
            killFlag = 0;
            return -1;
        }
        return 0;
    }

    // example of wrapping a timeouted system call (with absolute time)
    int psem_twait(sem_t* sem){
        if (killRequested()) return -1;   // not to loose the time of one poll
        struct timespec tsn = {1,0};
        for (;;){
            struct timespec ts;
            struct timespec tick = {1,0};
            timeend(&tick,&ts);
            if (sem_timedwait(sem,&ts) == 0) return 0;
            if (errno == ETIMEDOUT || errno == EINTR){
                if (killRequested()) return -1;
            } else {                      // some error
                return -1;
            }
            if (nanosleep(&tsn,NULL) -1 && errno == EINTR && killFlag){
                errno = EINTR;
                killFlag = 0;             // clear kill request
                return -1;
            }
        }
    }

    // example of wrapping a non-blocking system call
    int psem_wait(sem_t* sem){
        struct timespec ts = {1,0};
        for (;;){
            if (sem_trywait(sem) == 0) return 0;
            if (errno == EAGAIN){
                if (killRequested()) return -1;
            } else {                      // some error
                return -1;
            }
            if (nanosleep(&ts,NULL) -1 && errno == EINTR && killFlag){
                errno = EINTR;
                killFlag = 0;             // clear kill request
                return -1;
            }
        }
    }

    // example of wrapping a system call that is interrupted by stop signals
    int pepoll_wait(int epfd, struct epoll_event* events,
        int maxevents, int timeout){
        if (killRequested()) return -1;           // not to loose the time of one poll
        struct timespec ts = {timeout/1000,(timeout%1000)*1000000};
        struct timespec tend;
        timeend(&ts,&tend);                       // compute deadline
        int nfds;
        int i;
        timeout /= 10;                            // poll 10 times the timeout
        for (i = 0; i < 10; i++){
            nfds = epoll_wait(epfd,events,maxevents,timeout);
            if (nfds > 0) break;
            if (nfds == 0 || errno == EINTR){     // timeout or interruption
                if (killRequested()) return -1;
            } else {
                return -1;
            }
            if (timediff(&ts,&tend) <= 0) return 0;
        }
        return nfds;
    }

    ... before a supervised block:
    struct sigaction act;                 // register handler
    act.sa_sigaction = handler;
    act.sa_flags = SA_SIGINFO;            // N.B. do not restart slow system calls
    sigemptyset(&act.sa_mask);
    sigaction(SIGxxx,&act,NULL);

    // supervised block
    ...
    {                                     // supervised block
        sigset_t mask, oldmask;
        sigemptyset(&mask);
        sigaddset(&mask,SIGxxx);          // unblock signal
        pthread_sigmask(SIG_UNBLOCK,&mask,&oldmask);

        clearKill();                      // kill pending request, if generated internally
        ...
        int res = psem_wait(&sem);        // slow system call
        if (res == 0){
            ... success
        } else if (errno == EINTR){
            ... killed
            ... block signal
            ... cleanup
            ... unblock signal
            goto endblock;
        } else {
            ... error
        }
        ... other actions
        pthread_sigmask(SIG_SETMASK,&oldmask,NULL);  // restore previous mask
    } endblock:;

N.B. blocking signals in cleanup code does not cause any race with a signal that occurs right before blocking it: cleanup is reached because one such signal has already been caught.

Unblocking is needed because the code outside supervised blocks has normally signals blocked.

Task as thread

Here is the scheme of supervised block in which a task is implemented as a thread, and the supervision is a time supervision:

    typedef struct task_t {
        void* (*function) (void* arg);  // actual task function
        void*      arg;                 // function argument
        void*      retvalue;            // function return value
        pthread_t  th;                  // thread that executes the supervised actions
        sem_t      sem;                 // synchronization semaphore with thread
        failure_t* fail;                // pointer to failure object
    } task_t;

    // cleanup handler for thread
    void cleanup(void* arg){
        task_t* task = (task_t*)arg;
        sem_post(&task->sem);           // tell task terminated
    }

    // thread for executing the supervised actions
    void* thread(void* data){
        pthread_cleanup_push(cleanup,data);
        task_t* task = (task_t*)data;
        failure = task->fail;
        task->retvalue = task->function(task->arg);  // call actual task function
        pthread_cleanup_pop(1);
        return NULL;
    }

    // cleanup handler for convenience function
    static void cleanupt(void* arg){
        task_t* task = (task_t*)arg;
        pthread_cancel(task->th);             // n.b. no error checking
        pthread_join(task->th,NULL);          // no errors can occur
        sem_destroy(&task->sem);
    }

    // convenience wrapper function, that executes user function with timeout
    int timedTask(void* (*function) (void* arg), void* arg, void** ret, int timeout){
        task_t task;
        task.function = function;             // store function pointer
        task.arg = arg;                       // store function argument
        task.retvalue = NULL;                 // clear return value
        sem_init(&task.sem,0,0);              // initialize task termination semaphore
        task.fail = failure;                  // store exception object pointer

        int res = 0;
        pthread_cleanup_push(cleanupt,&task);
        res = pthread_create(&task.th,NULL,&thread,&task);
        if (res != 0){
            return res;
        }
        struct timespec ts;
        clock_gettime(CLOCK_REALTIME,&ts);    // start supervision
        ts.tv_sec += timeout/1000;            // timeout in millis
        ts.tv_nsec += (timeout % 1000) * 1000000;
        if (ts.tv_nsec > 1000000000){         // normalize
            ts.tv_sec++;
            ts.tv_nsec -= 1000000000;
        }
        res = sem_timedwait(&task.sem,&ts);   // wait until task completed or timeout
        if (res == -1 && errno == ETIMEDOUT){
            res = 0;
            pthread_cancel(task.th);
        }
        void* status;
        pthread_join(task.th,&status);
        if (status == PTHREAD_CANCELED){
            res = ETIMEDOUT;
        }
        sem_destroy(&task.sem);
        pthread_cleanup_pop(0);
        *ret = task.retvalue;                 // return task return value
        return res;
    }

    // example function that contains the actions to supervise
    void* funct(void* data){
       ... actions
    }

    ... code that defers the task to another thread and supervises it

    void* ret;
    int res = timedTask(funct,argument,&ret,timeout);
    if (res){
        ... task timed out, or creation error
    }

The thread that defers the task uses a semaphore as timer. This has the advantage that it can easily be created and destroyed, and it can be stopped by the task when it has ended before the time has expired.

This scheme contains a convenience function that can be used all the times there is a need to time supervise a sequence of actions: they can simply be put in a function, that is called. A similar technique can be used when the event to supervise is a different one. This example supports also exception handling.

If the supervision is not a time supervision, after having created the task thread, the creator has to wait for whatever event it must, and when it occurs, cancel the task thread. Note that the creator must use a system call that waits for two events to occur: the supervision one and the cancellation of the supervision made by the task thread.

The time supervision is started here by a thread that is not the one that executes the actions. This could be less precise than starting it from within the latter. However, that is not simple, and it is not precise anyway.

Preventing killing

To implement properly killing, either we instrument all the places in which a resource is allocated so as to treat killing, or we block killing where we do not want to treat it.

In library functions (and in object methods), killability could at most be disabled, but never enabled: the caller could want not to be killable.

General cleanup function

The use of a general cleanup function is possible for process, thread and task kill. It can be used when killing consists only in simple operations like, e.g., releasing resources. A table of allocated resources can be kept. They are system resources, but could also be process-wide resources contained in some shared segment. The table would indicate for each entry the kind of resource, and for the user ones, it would also contain a function pointer to the release function. A thread would register into the table any new resource acquired, and remove from it any released. The cleanup code would then scan the table and release all the resources in it. This can be used globally (i.e., for an entire thread or process), and also locally (for a task).

However, there are cases in which simply releasing is not enough (e.g., to close some connection there would be a need to send messages). This can be solved by making the code that executes actions that need to be undone provide a function to do it, that needs to be registered too. This allows also to preserve information hiding (which would not be the case if the general cleanup function accessed the internals of all modules that need cleanup).

With all kill paradigms it is possible to reckon resources allocated, and treat kill at the end of a block, Note that when the program detects a kill request, it must not continue, but jump to the cleanup code. Continuing could lead to unpredictable results, like, e.g., blocking (which should not happen with persistent events, but it could with simulated persistent signals: to kill a sequence of blocking system calls a sequence of signals must be sent until an acknowledge comes). Note that continuing does not occur when cancellation is used because when a cancellation request is acted, the thread jumps to cleanup handlers as soon as it encounters the first cancellation point, and does not continue any more with the code that the thread was executing before cancellation.

    pthread_sigmask(SIG_UNBLOCK,...);    // unblock signal
    ...
    if (killFlag) exit block   (1)
    systemcall(...);           (2)

    void handler(int signo){
        killFlag = 1;
    }

    if kill flag ...           (1)
    for (;;)
        timed wait             (2)
        if not timeout break;
        if kill flag ...       (3)
    }

    // test if kill has been requested
    int killRequested(){
        sigset_t sigpend;
        sigemptyset(&sigpend);                // sigpending does not clear it
        sigpending(&sigpend);
        if (sigismember(&sigpend,SIGxxx)){
            int sig;
            sigwait(&sigpend,&sig);           // clear kill request
            errno = EINTR;                    // report abortion
            return 1;
        }
        return 0;
    }

    // example of wrapping a system call using a time supervised one (with absolute time)
    int psem_twait(sem_t* sem){
        if (killRequested()) return -1;       // not to loose the time for one poll
        for (;;){
            struct timespec ts;
            struct timespec tick = {1,0};     // poll period 1 s
            timeend(&tick,&ts);
            if (sem_timedwait(sem,&ts) == 0) return 0;
            if (errno == ETIMEDOUT){
                if (killRequested()) return -1;
            } else {                          // some error
                return -1;
            }
        }
    }

    // example of wrapping a system call using a non-blocking one
    int psem_wait(sem_t* sem){
        if (killRequested()) return -1;       // not to loose the time for one poll
        for (;;){
            if (sem_trywait(sem) == 0) return 0;
            if (errno == EAGAIN){
                if (killRequested()) return -1;
            } else {                          // some error
                return -1;
            }
            struct timespec ts = {1,0};
            nanosleep(&ts,NULL);
        }
    }

    // example of wrapping a system call that is interrupted by stop signals
    int pepoll_wait(int epfd, struct epoll_event* events,
        int maxevents, int timeout){
        ... see the one for polling with signals unblocked
    }

    // supervised block
    ...
    {                                         // supervised block
        clearKill();                          // kill pending signal, if generated internally
        ...
        int res = psem_wait(&sem);            // slow system call
        if (res == 0){
            ... success
        } else if (errno == EINTR){
            ... killed
            ... cleanup. N.B. signals blocked
            goto endblock;
        } else {
            ... error
        }
        ... other actions
        clearKill();                          // kill pending signal, if generated internally
    } endblock:;

    if (setjmp(cxt) != 0){
        handle exception
    } else {
        try block
    }

    handler
       no need to block permanently signals: siglongjmp does it
       siglongjmp(context,1);

    // leave the signal unblocked at the beginning: it must kill
    save signal mask
    block SIGxxx
    register handler for SIGxxx
    if (sigsetjmp(context,1) != 0){
        cleanup
    } else {
        unblock SIGxxx
        supervised block
    }
    deregister handler
    restore signal mask

    restore signal mask
    deregister

    if (setjmp(context) != 0){
       cleanup
       (1)
       deregister handler
       (2)
       restore signal mask
    } else {
       unblock SIGxxx
       (3)
       supervised block
       (4)
       restore signal mask
       (5)
       deregister handler
    }

    static volatile __thread sigjmp_buf* jmp;
    static volatile sigjmp_buf nokill;

    handler
       sigjmp_buf* tmp;           // save current
       tmp = (sigjmp_buf*)jmp;
       jmp = NULL;                // make it NULL: prevent further jumping after handler jump
       if (tmp == NULL){
           pthread_exit(NULL);
       } else if (tmp == (sigjmp_buf*)&nokill){
           return;
       }
       siglongjmp(*tmp,1);
    }

    sigaction with SA_RESTART handler, all signals blocked
    ...
    sigjmp_buf buf;
    sigjmp_buf* prev = (sigjmp_buf*)jmp;     // save current
    if (sigsetjmp(buf,1) != 0){
        ... cleanup
    } else {
        jmp = (volatile sigjmp_buf*)&buf;    // (1)
        ... block of statements
        sigjmp_buf* save = (sigjmp_buf*)jmp; // save: start non-killable block
        jmp = &nokill;
        ... actions not killed
        jmp = save;                          // restore: end non killable block
    }
    jmp = prev;                              // restore previous supervision

    block signals
    acquire lock
    register lock acquired
    unblock signals

    acquire lock
    block signals
    register lock acquired
    unblock signals

    BEGIN_KILL
        acquire lock
    ON_KILL
        release lock
    END_KILL

    { // doit
        { // cleanup
            operation
            if (killFlag) break cleanup;
            system call
            if (errno == EINTR) break cleanup;
            break doit;
        }
        cleanup:
        perform cleanup here
    }
    doit:

    open(file1);
    open(file2);
    open(file3);

    open(file1);
    pthread_cleanup_push(cl1);
    open(file2);
    pthread_cleanup_push(cl2);
    open(file3);
    pthread_cleanup_push(cl3);

    {
        open(file1);
        if (killed) goto doit:
        open(file2);
        if (killed) goto doit:
        open(file3);
        if (killed) goto doit:
    } //
    doit:
    if (killed){
        if (file1 != null) close(file1);
        if (file2 != null) close(file2);
        if (file3 != null) close(file2);
    }

    void openservice(){
        ... code to open or start this service
    }
    void cleanup(void* arg){
        ... code to closer or stop this service
    }
    #define OPENSERVICE() pthread_cleanup_push(cleanup,NULL); openservice()
    #define CLOSESERVICE() pthread_cleanup_pop(1)

    // example of library use
    OPENSERVICE();
    ... other actions
    CLOSESERVICE();

Signals as IPC

Signals can be used as inter-process communication means (IPC).

All other IPC means, like, e.g., messages, semaphores, etc. require to create a permanent object, and to destroy it when it is no longer needed, while signals can be sent to any process without a need for system objects. This can be seen as a disadvantage, but it can be an advantage too: it allows to communicate with processes that are not yet created.

Semaphores, mailboxes, pipes, etc, need to be created, and even worse, need to be removed when processes terminate (even when they terminate because killed, with the risk to litter the system with no longer useful objects). Signals are also served before the process to which they are sent does anything else (providing it has registered a handler). Moreover, the idea of interrupting a sequence of statement is something basic that cannot be achieved with messages sent to threads waiting for them. Of course, the request made by a process to another to kill itself or the actions that it is executing can be implemented sending a message to it (having in it a thread receiving the message), but that thread needs anyway a means to interrupt the actions done by its process.

Signals can be used as semaphores or queues: waiting for a semaphore to become green or a queue to have an element to get is done by using sigwait(), or sigwaitinfo(); posting to a semaphore or a queue by sigqueue(), pthread_kill() or kill(). Synchronization among threads can better be done using one of the IPC means available (locks of various kinds, message queues, etc.). Therefore, signals as IPCs are restricted to synchronizing processes. Note that related processes know each other pids, while unrelated processes do not know them, and thus they must communicate (or find) their pids in order to synchronize with signals. A drawback is then that this exchange of data must be done, and another is that pids can denote processes that do not exist any more, or worse that are recycled and denote homonyms. This kind of synchronization is mostly used to issue requests to servers, or demons, i.e., processes that run permanently.

Another difference is that signals can carry data, i.e., are similar to messages, while semaphores do not.

This is the scheme of the server process (the thread in it that is devoted to handle signals):

    #define SIGCTRL (SIGRTMIN+6)               // signal used as IPC

    sigemptyset(&mask);
    sigaddset(&mask,SIGCTRL);                  // use a realtime signal, that is queued
    int sig = 0;
    int val = 0;
    for (;;){
        siginfo_t siginfo;
        sig = sigwaitinfo(&mask,&siginfo);     // wait request
        if (sig == -1){
            if (errno == EINTR) continue;
            ... error
        }
        val = siginfo.si_value.sival_int;      // extract associated value
        sigval_t sval;
        sval.sival_int = ...;                  // send reply with this value
        if (sigqueue(siginfo.si_pid,SIGCTRL,sval) < 0){   // send reply
            ... error
        }
    }

This is the scheme of the process that makes a request:

    sigval_t sval;
    sval.sival_int = val;                      // send request with this value
    if (sigqueue(pid,SIGCTRL,sval) < 0){
        ... error
    }
    siginfo_t siginfo;
    while (sig = sigwaitinfo(&mask,&siginfo) == -1 && errno == EINTR);  // wait reply
    if (sig == -1){
        ... error
    }
    val = siginfo.si_value.sival_int;          // get value from reply

Time supervision

Time supervision, although it is in general incompatible with the fact that processes and threads progress with the time that the scheduler gives them, is a need that occurs in applications. E.g., there are cases in realtime systems in which a sequence of operations must be done in a given time, otherwise something else need be done. There are several means to ensure that applications that have time constraints meet them. In these cases it is meaningful to supervise progress.

Time supervision can be applied to a single operation, usually a slow system call, or to a sequence of operations. Many slow system calls support time supervision. This works fine because supervision is started at the same time a process suspends, and therefore it supervises exactly the waiting time. Some system calls, though, require absolute time, which means that the current time need be got, and the deadline computed, and the process can get de-scheduled in between, thus making the deadline unreliable to some extent. Usually, however, the timeout is much bigger than the error caused by this. The same applies to the system calls that have no time supervision.

To time supervise a system call, like, e.g., a read(), we could run a thread that makes the read(), and another that starts the timer, and wait for one to end, but this also would not be perfect (the one that reads might get de-scheduled, letting the other come first). Upon time expiry a signal is can be sent. A problem is that once the handler has been set, and a signal unblocked, it can be delivered, and this can occur before the system call or function to time supervise is executed. If this occurs, we must not execute the call or function (otherwise it would not be supervised any more). This in practice is not a big problem because the chance that it occurs are few. However, the code must not contain the race all the same. A bigger problem is the expiry of the timer due to de-scheduling. This can be mended by using a long timer, or by using timed operations, when available.

Time supervision on I/O is provided by poll(). E.g.:

    struct pollfd fds;
    fds.fd = fileno;
    fds.events = kind of event to poll
    ... other file descriptors to watch, be their number n
    int res = poll(&fds,n,timeout-in-millis);
    if (res == -1){
        ... error
    }
    if (res == 0){
        ... timeout
    } else {
        ... events, in fds.revents
    }

Time supervision is a supervised block in which the kill signal is generated internally, and thus can be implemented as such:

1. pselect(), ppoll(), etc.: block signals, start a timer with a signal and abort the block when these calls are interrupted.
2. polling: use only time supervised or non-blocking calls (which is not a general solution because the code might call some library function that contains a blocking call), block signals, and poll that a kill request is pending. Most blocking system calls support a timeout (I/O, queues, locks, etc.), or at least a non-blocking operation.
3. longjmp: in some very simple cases it is possible to use a supervised block with a signal handler that makes a longjmp(). These are the cases in which the block of supervised actions contain only asynch-signal-safe calls, and the signal has a handler that blocks the other signals. An example is to time supervise I/O operations, but this can also be implemented with pselect(), ppoll(), etc.
4. create a thread and cancel it when the time expires. To support nested time supervisions, a thread must be created for each one. The scheme for this solution is here.

    typedef struct task_t {
        int timeout;                    // max execution time of task, in msec
        void* (*function) (void* arg);  // actual task function
        void* arg;                      // function argument
        void* retvalue;                 // function return value
        timer_t timer_id;               // id of the timer
    } task_t;

    // timeout function that cancels the thread
    void threadcancel(sigval_t sigval){
        pthread_cancel((pthread_t)sigval.sival_ptr);
    }

    // cleanup handler, that cancels the timer
    void cleanup(void* arg){
        task_t* task = (task_t*)arg;
        if (task->timer_id == NULL) return;          // argument not yet set
        timer_delete(task->timer_id);                // disarm and delete timer
    }

    // thread that starts/stops time supervision and executes the task
    void* thread(void* data){
        task_t* task = (task_t*)data;
        task->timer_id = NULL;                       // initialize task data
        task->retvalue = NULL;
        pthread_cleanup_push(cleanup,task);          // register timer cancellation

        struct sigevent event;
        event.sigev_notify = SIGEV_THREAD;
        event.sigev_notify_function = threadcancel;
        event.sigev_notify_attributes = NULL;
        event.sigev_value.sival_ptr = (void*)pthread_self();
        timer_t timer_id;
        if (timer_create(CLOCK_REALTIME,&event,&timer_id) < 0){
            pthread_exit(PTHREAD_CANCELED);
        }
        task->timer_id = timer_id;                   // remember timer it and attributes

        struct itimerspec itime;                     // arm timer
        itime.it_value.tv_sec = task->timeout/1000;
        itime.it_value.tv_nsec = (task->timeout % 1000) * 1000000;
        itime.it_interval.tv_sec = 0;
        itime.it_interval.tv_nsec = 0; 
        if (timer_settime(timer_id,0,&itime,NULL) < 0){
            pthread_exit(PTHREAD_CANCELED);
        }

        task->retvalue = task->function(task->arg);  // call actual task function
        pthread_cleanup_pop(1);                      // cancel timer
        return NULL;
    }

    // cleanup for convenience wrapper function
    static void cleanupt(void* arg){
        pthread_t thr = *((pthread_t*)arg);
        pthread_cancel(thr);                         // n.b. no error checking
        pthread_join(thr,NULL);                      // no errors can occur
    }

    // convenience wrapper function, that executes user function with timeout
    int timedTask(void* (*function) (void* arg), void* arg, int timeout){
        task_t task;
        task.timeout = timeout;                      // fill in time and function
        task.function = function;
        task.arg = arg;
        pthread_t th;                                // create thread to execute function
        pthread_cleanup_push(cleanupt,&th);
        int res = pthread_create(&th,NULL,&thread,&task);
        if (res != 0){
            return res;
        }
        void* status;                                // wait for its termination
        res = pthread_join(th,&status);
        if (res != 0){
            return res;
        }
        if (task.retvalue != NULL){                  // return value of function
            status = task.retvalue;
        }
        pthread_cleanup_pop(0);
        return (int)status;
    }

    void* funct(void* data){
        ... actual task
    }

    // example of use
    int res = timedTask(funct,argument,timeout);
    if (res){
        ... error
    }

Waiting for a child process

Waiting for children to terminate can be done with:

    for (;;){
        int status;
        pid_t pid = waitpid(-1,&status,0);
        if (pid == (pid_t)-1){
            if (errno == ECHILD) break;
            ... an error occurred
            break;
        }
    }

This scrap of code can be placed in any thread, including one dedicated to wait for children. If the SIG_IGN handler has been set for SICGCHLD, there is no need to wait to recover the status of children, and children do not become zombies. Note that this is not the same as the default disposition (even if it looks similar).

A process must wait for its children even when they have been killed sending SIGKILL, unless it sets the disposition of SIGCHLD to SIG_IGN.

Waiting for children must be done when they are designed to terminate, and also when they terminate unexpectedly, while they where supposed not to do.

Waiting for children can also be done in a handler registered for SIGCHLD, but this is not convenient because it would interrupt a number of system calls even when registered with the SA_RESTART option. When there is no need to get the children exit status, the disposition of SIGCHLD can be set to SIG_IGN, and nothing else done; otherwise, children can be waited either by the thread that created them or by any other (e.g., a dedicated one). Moreover, waitpid() resumes only when a child terminates, and not when SIGCHLD is sent by any process, while a handler is run when SIGCHLD occurs, also when sent by non-children processes.

Setting SIGCHLD to SIG_IGN or to an handler with SA_NOCLDWAIT prevents children to become zombies. However, init() in Linux clears quickly orphaned processes (i.e., children whose parent died before them), and also zombies. There is thus no need to set SIGCHLD to SIG_IGN to avoid zombies.

waitpid() can wait for any child that has a specific process group. Note that there can exist processes that belong to the same group, but that are not children, and thus that are not waited for.

Handling SIGSEGV and SIGBUS

SIGSEGV is generated when a memory address that does not belong to the process address space is accessed, or when an attempt is made to write into a read-only memory address. E.g., null pointers, page faults or stack overflow.

SIGBUS is generated when accessing a memory mapped file at an address that does not correspond to a position in the file.

Both these signals, after executing a handler, restart the very same instruction that caused them. Therefore, either the handler is able to cure the cause of the signal (e.g., map properly memory), or the handler must terminate the program or the affected thread.

SIGSEGV and SIGBUS are normally programming errors or failures, that can be handled with cancellation, reporting the exception (and therefore, when they occur in system calls, they must not make them return with an error, but terminate the thread). When they denote a programming error, little can be done but killing the process. When they are failures, and there is a means to recover them, the thread can be cancelled, the problem cured, and the thread recreated. In the latter case, the solution is then to put the code that might generate one of these signals in a thread, and cancel it when the signal occurs.

Do simply register/deregister a cleanup handler around statements that can cause one such signal. To tell the signal handler not to kill the process, but only the thread, a flag is set. This is the scheme:

    static volatile __thread sig_atomic_t kill_thread = 0;
    void cleanup(void* arg){
       ... report an exception
    }

    static void handler(int sig, siginfo_t* siginfo, void* context){
        if ((siginfo->si_code == SI_USER || siginfo->si_code == SI_QUEUE) &&
            siginfo->si_pid != getpid()){   // external signal
            ... process kill
        } else {
            if (kill_thread != 0){          // thread handles exception, kill it
                pthread_exit(PTHREAD_CANCELED);
            } else {                        // programming error
                ... process kill
            }
        }
    }

    pthread_cleanup_push(cleanup,NULL);
       ... code that can generate a programming error
    pthread_cleanup_pop(0);

    pthread_cleanup_push(cleanup,NULL);
    kill_thread = 1;                        // enable handling
       ... code that can generate a failure
    kill_thread = 0;                        // disables it
    pthread_cleanup_pop(0);

The latter is meant to be used on some specific statement or system call that can produce a SIGSEGV or SIGBUS when executed, and not on a large section of code, that could generate them also because of some programming error.

Note that the handler cancels the interrupted thread using pthread_exit(), which is not asynch-signal-safe. However this handler catches a synchronous signal, that cannot occur while executing this function, and thus it does not interrupt an execution of it.

When they are not programming errors or failures, they are used to remap memory, and not to terminate processes or threads (or supervised blocks). However, when they are generated from within user code, their handlers are executed, and when returning they restart silently the code, but when they occur in system calls (e.g., because a return argument points to unmapped memory), the behaviour is dependent on the system call, and may range from error returns to partial execution of the system call. A program that wants to remap silently memory accesses can then provide a third case to the handler above, supplying it a user function to call instead of killing the process. That program, however, must either be sure not to execute system calls that can address unmapped memory, or to check any error that they return and act accordingly. This latter case is not treated further here.

When there is a need to perform some memory remapping under the hood, in the handler, system calls such as mprotect() and mmap() can be safely called provided that errno be saved and restored (they cannot cause a SIGSEGV to occur while they are executing, and therefore there are never two executions of them at the same time).

Stack overflow

Stack overflow can occur at any function call, and then likely in library functions that are not prepared to handle it, and that do not even register cleanup handlers.

The stack should be dimensioned to support the deepest nesting, without reckoning recursion. Detection of stack overflow should be done only in functions that use much stack, and this occurs typically in recursive functions. It is better to control the depth of recursion than to rely on the system detecting stack overflow. Recovering from stack overflow is difficult: it could be done only, e.g., when there are alternative, less expensive ways to perform some task. E.g., a function needing a large amount of temporary storage could allocate it on the stack declaring an automatic variable. If it fails because of stack overflow, it could be called again telling it to allocate on the heap (which is more costly).

When a function is called, and there is no room in the stack for it, its stack frame is not placed in the stack, SIGSEGV is generated, its handler executed (if any), and if the handler terminates the thread, its cleanup handlers are run on the normal thread stack. The first cleanup handler executed must then use less stack than the offending function call, otherwise the process is immediately terminated with a segmentation fault (i.e., the default SIGSEGV disposition). E.g., suppose a function opens a file, but being close to the stack limit, it causes a stack overflow, and that the close function requires more stack than the open one: the process is terminated immediately. This means that the functions devoted to undo some operation must be thrifty with the stack. Since cleanup handlers unwind the stack, the subsequent ones have more stack at their disposal. N.B. if a thread has no alternate stack, and the handler is registered with SA_ONSTACK, the normal stack is used. Note also that there is no way to terminate a thread bypassing its cleanup handlers, and neither there is a way to extend a bit the stack of a thread (so as to have room to run its cleanup handlers).

Threads that want to terminate gracefully when a stack overflow occurs, must create an alternate stack. Consequently, the SIGSEGV handler must be registered with the SA_ONSTACK flag:

    void* thread(void* data){
        stack_t altstack;              // set up an alternate stack
        if ((altstack.ss_sp = malloc(SIGSTKSZ)) == NULL){
            ... error, no memory
        }
        altstack.ss_size = SIGSTKSZ;
        altstack.ss_flags = SS_ONSTACK;
        if (sigaltstack(&altstack,NULL) < 0){
            ... error
        }

        pthread_cleanup_push(cleanup,NULL);
        ... actions that can cause SIGSEGV because of stack overflow
        pthread_cleanup_pop(0);
        return NULL;
    }

    ... handler registration
    struct sigaction sa;
    sa.sa_flags = SA_SIGINFO | SA_RESTART | SA_ONSTACK;
    sa.sa_sigaction = handler;
    sigsetmost(&sa.sa_mask);
    sigaction(SIGSEGV,&sa,NULL);

An alternate stack is set for the thread that executes the operations that can cause the signal to occur so as to have room for the handler to execute (which may not be the case when a stack overflow occurs). Note that sigaltstack() sets an alternate stack for the signal handlers of the calling thread (and not the calling process, as said in the man pages) that are registered with SA_ONSTACK.

    static volatile  __thread sigjmp_buf* jmp;
    void handler(int sig, siginfo_t* siginfo, void* context){
        if ((siginfo->si_code == SI_USER || siginfo->si_code == SI_QUEUE) &&
            siginfo->si_pid != getpid()){   // external signal
            ... process kill       // or return, if stray signals discarded
        } else {
        sigjmp_buf* tmp;           // save current
        tmp = (sigjmp_buf*)jmp;
        jmp = NULL;                // make it NULL: prevent further jumping after handler jump
        if (tmp == NULL){
            ... process kill
        }
        siglongjmp(*tmp,1);
    }

    ... handler registration
    struct sigaction sa;
    sa.sa_flags = SA_RESTART | SA_SIGINFO;
    sa.sa_sigaction = handler;
    sigsetmost(&sa.sa_mask);
    sigaction(SIGSEGV,&sa,NULL);

    ... supervised block
    sigjmp_buf buf;
    if (sigsetjmp(buf,1) != 0){
        ... cleanup
    } else {
        jmp = (volatile sigjmp_buf*)&buf;
        ... actions that may generate the signal
        jmp = NULL;
    }

    if (info->si_code == SEGV_ACCERR || info->si_code == SEGV_MAPERR){
        ... internal signal
    }

Exception Handling

In this chapter, error reporting for threads that use cleanup handlers and signal handlers is presented. Cleanup handlers perform exception handling, i.e. they execute any action that is deemed appropriate to cater for failures or other premature termination (e.g., killing). This applies to all signals that denote errors, and to the explicit checking of dynamic conditions whose violations denote errors.

Exception handling means basically detecting failures, cleaning up the process state up to a point at which something can be done to recover (which can be retrying, trying alternatives, or doing less) or terminating the process if nothing can be done.

Premature termination occurs when:

1. the process is killed as a response of an external kill request,
2. the process is killed because that is the way we want to handle a stray signal,
3. the process is killed because an internal signal that denotes an unrecoverable (e.g., a programming) error is caught,
4. a thread suicides because it has detected an error (failure).

In the first two cases we are not interested in what the process (and all its threads) was exactly doing, in the others we need a detailed error reporting to tell what went wrong:

• unrecoverable (programming) error: we want to know the statement that caused the error, and how the program got there (e.g., the call stack).
• failure: we want to know what was wrong in our request or in the resources (hardware and software) used to fulfill it.

SIGSEGV cannot be left to its default disposition (barely no error reporting), and stray SIGSEGVs cannot produce an error (but, e.g., a message such as "process killed on user request", as all other kill requests). SIGSEGV can occur at any place in the code (e.g., at memory accesses made with pointers), and normally denotes a programming error. SIGBUS, in some specific circumstances, can denote a failure (e.g., a system call that accesses a memory mapped file that is truncated), that the program wants to handle.

The only one other signal that denotes the violation of a dynamic condition is a time supervision one. However with it, reporting the circumstances of the error is not difficult. Almost all system calls that detect the violation of a dynamic condition return with an error and not with a signal.

All these cases can be handled performing graceful kill and reporting an exception to higher levels, being the last the one that circumstantiates the error with respect to the external process interface. This can be done recording the error data in cleanup handlers. When an internal signal denotes a failure, we should catch it close to the point (setting a cleanup handler that reports an error). For system calls that do not generate this signal, but return an error or interruption, pthread_exit() can be called, reporting an error. When an error occurs, most of the times we need to perform cleanup much the same as we do when killing. There could be cases in which at some level of nesting a specific recovery can be done, and then the thread not terminated. This can be handled telling the handler not to cancel the thread (but in many cases no recovery can be done). It is then appropriate that cancellation be used both to serve an external request and to handle errors.

When an internal signal denotes a programming error, it can be reported by telling at least the thread or topmost function being executed (with a cleanup handler). Printing the call stack would be quite useful, but unfortunately, there are no means to do it (unless a coredump is generated, and analyzed). It would be nice to tell the source line in error, or at least the program (instruction) counter. This seems quite a difficult thing in C: I have never found a linker listing map that tells where the modules are placed in virtual memory, and neither a C compiler listing telling the relative addresses of instructions in the module (compilation unit). The process could have other threads extant at that point, that must be cancelled without reporting any error.

This is the scheme:

    void cleanup(void* arg){        // cleanup for function
        ... cleanup actions
        exceptionrethrow(..);
    }

    void function(){                // any function that can encounter errors
        ...
        pthread_cleanup_push(cleanup,arg);
            ...
            abortprocess();         // if unrecoverable error
            - or -
            statement               // that may cause a signal
            ...
        pthread_cleanup_pop(0);

        - or -
        exceptionthrow(...);        // if recoverable error (thread kill)
    }

    void cleanup0(void* arg){
        if (failing){               // cleanup because of error
            exceptionrethrow(...);
        }
    }

    void function0(){
        pthread_cleanup_push(cleanup0,NULL);   // caller that wants to wrap exceptions
        ...
        function();
        ...
        pthread_cleanup_pop(0);
    }

    typedef struct failure_t {
        struct failure_t* next;                // chained object
        ... any other error data
    } failure_t;

    // throw an exception from a cleanup handler
    void exceptionrethrow(...){
        failure_t* el;
        if (failure->next == NULL){            // use object passed by creator
            el = failure;
        } else {
            el = malloc(sizeof(failure_t));    // allocate a new object
            if (el == NULL) return;
            memset(el,0,sizeof(failure_t));    // initialize
        }
        el->next = failure->next;              // add to chain
        failure->next = el;
        ... record error data in failure object
        failing = 1;                           // note thread is failing
    }

    // throw an exception from a thread
    void exceptionthrow(char* sub){
        failure->next = failure;               // initialize chain
        ... record error data in failure object
        failing = 1;
        pthread_exit(PTHREAD_CANCELED);
    }
 
    // deliver the list of exceptions (and makes it linear)
    struct failure_t* getexceptions(void* p){
        if (p == NULL) return NULL;
        failure_t* f = (failure_t*)p;
        if (f->next == NULL) return NULL;      // only one, and not filled
        f = f->next;
        ((failure_t*)p)->next = NULL;          // open the cycle
        return f;
    }

    ... creation and joining of thread
    failure_t* fail = malloc(sizeof(failure_t));
    if (fail != NULL) memset(fail,0,sizeof(failure_t));
    pthread_cleanup_push(cleanupm,data);
    pthread_t th;                                  // create thread to execute function
    res = pthread_create(&th,NULL,&thread,fail);
    ...
    res = pthread_join(th,&status);                // wait for it
    pthread_cleanup_pop(0);
    failure_t* f = getexceptions(fail);            // get list of exceptions
    if (f != NULL){
        ... cope with exceptions
    }
    free(fail);

Handling SIGFPE

SIGFPE is difficult to handle: it should not be ignored (otherwise the program behaviour is undefined), it should not be handled by the default disposition (which is to abort the process), and then it needs to be caught. However, in the handler the only alternative is to terminate the process. It is also conceivable to try to skip the offending instruction, but this entails a code that is dependent on the instruction set of the processor. Moreover, the signal could occur many instructions later, as Tydeman states. Although it is possible for a compiler to generate the appropriate instructions to prevent this, there is no guarantee that this is done. This makes this signal even more difficult to handle.

To handle the floating point exceptions, the functions in fenv.h can instead be used:

    #include <fpe.h>
    ...
    fenv_t envp;
    feholdexcept(&envp);                 // disable generation of SIGFPE
    ...
    a = 1.0 / 0.0;                       // any floating point operation that raises an exception
    if (fetestexcept(FE_DIVBYZERO)) ...  // true if exception occurred

Handling SIGXCPU

SIGXCPU is sent to a process to notify it that it is reaching its limit of CPU consumption, and that soon after a SIGKILL will terminate it. The process has thus the chance to save or cleanup its important data. The CPU time limits are set as follows:

    struct rlimit rlim;
    rlim.rlim_cur = 1;             // soft limit, nr. of seconds
    rlim.rlim_max = 2;             // hard limit
    if (setrlimit(RLIMIT_CPU,&rlim) == -1){
        ... error
    }

Handling timers

A thread can be used to catch the signals generated by timers. A timer can be told to create a thread each time it tics. This is quite costly, and thus can be done for timers that tic only once:

    void thread(sigval_t sigval){
       ...
    }

    ... scrap of code that uses the timer
    struct sigevent event;
    timer_t timer_id;
    event.sigev_notify = SIGEV_THREAD;
    event.sigev_notify_function = thread;
    event.sigev_notify_attributes = NULL;
    if (timer_create(CLOCK_REALTIME,&event,&timer_id) < 0){
        ... error
    }

    struct itimerspec itime;
    itime.it_value.tv_sec = 1;
    itime.it_value.tv_nsec = 0;
    itime.it_interval.tv_sec = 0;
    itime.it_interval.tv_nsec = 0; 
    timer_settime(timer_id,0,&itime,NULL);
    ... when done:
    timer_delete(timer_id);

The created thread behaves as if it was detachable: it ceases to exist when its function returns. However, timers when told to run a function, create a thread for it, but create also another thread that is not terminated when timers are destroyed. This causes a leak of threads.

For timers that tic periodically, a thread can be created in advance, to wait for signals sent by the interval timers (without drift):

    void* thread(void* data){
        increasePriority();                        // increase thread priority
        sigset_t mask;
        sigsetmost(&mask);                         // block most signals
        pthread_sigmask(SIG_BLOCK,&mask,NULL);     // block all signals for this thread
        sigemptyset(&mask);
        sigaddset(&mask,SIGUSR1);                  // all signals carry the value
        for (;;){
            int sig;
            sigwait(&mask,&sig);
        }
    }

    ... scrap of code that uses the timer. N.B. SIGUSR1 must be blocked here

    pthread_t th;
    pthread_create(&th,NULL,&thread,NULL);
    struct sigevent event;
    timer_t timer_id;
    event.sigev_notify = SIGEV_SIGNAL;
    event.sigev_signo = SIGUSR1;
    if (timer_create(CLOCK_REALTIME,&event,&timer_id) < 0){
        ... error
    }

    struct itimerspec itime;
    itime.it_value.tv_sec = 1;
    itime.it_value.tv_nsec = 0;
    itime.it_interval.tv_sec = 1;
    itime.it_interval.tv_nsec = 0; 
    timer_settime(timer_id,0,&itime,NULL);
    ... when done:
    timer_delete(timer_id);timer_delete(timer_id);
    pthread_cancel(th);
    pthread_join(th,NULL);

Handling SIGSTOP, SIGTSTP and SIGCONT

These signals are sent to processes to stop (i.e., suspend) them and to continue (resume) them. SIGTSTP is sent when typing ^Z, SIGCONT when typing fg at a shell prompt when a program has been run by that shell. All of them can also be sent with kill(). SIGSTOP cannot be caught. SIGCONT can be caught, but eventually resumes the process anyway. One use of catching these signals is to set, or reset, the terminal to the desired operating mode, or to redraw it, or redisplay a prompt. If something need be done when both a process is interactively stopped (SIGTSTP) and when it is continued, and the actions are very simple, this is the scheme:

    static void handle_sigtstp(int signo){
        ... do what actions are needed before stopping
        raise(SIGSTOP);
        ... do what actions are needed before continuing
    }

    ... to register the handler
    struct sigaction sa;
    sigsetmost(&sa.sa_mask);
    sa.sa_handler = handle_sigtstp;
    sa.sa_flags = SA_RESTART;
    sigaction(SIGTSTP,&sa,NULL);

If there is a need to do some (very simple) actions only when SIGCONT is received, then a handler for it can be set. Otherwise, a thread can be dedicated to them:

    void* thread(void* data){
        increasePriority();                        // increase thread priority
        sigset_t mask;
        sigsetmost(&mask);
        pthread_sigmask(SIG_BLOCK,&mask,NULL);     // block all signals for this thread
        sigemptyset(&mask);
        sigaddset(&mask,SIGTSTP);
        sigaddset(&mask,SIGCONT);
        for (;;){
            int sig;
            sigwait(&mask,&sig);
            if (sig == SIGTSTP){
                ... do what actions are needed before stopping
                raise(SIGSTOP);
            } else {
                ... do what actions are needed before continuing
            }
        }
    }

    ... to create the thread (and destroy it).
    ... N.B. these signals blocked in all other threads
    pthread_t th;
    pthread_create(&th,NULL,&thread,NULL);
    ...
    pthread_cancel(th);
    pthread_join(th,NULL);

Rotating logs

Signals can be used to ask processes (or threads) to change the course of some action that they are doing repeatedly. This can be done informing them about the event with a flag. This is the case of processes that repeatedly execute a loop, or that perform frequently some actions. An example is processes that record log messages in some file. When there is a need to get the messages logged so far, the process can be told to close the current log file, and open a new one. Another example is processes that read a configuration file, and that can be told to reload it. If they execute repeatedly a loop, they can check if such a request has come.

This could be implemented registering a handler that sets a flag. However, that would interrupt many system calls that are not restarted automatically. A better solution is to use a thread to handle that signal. It can be handled by the control thread. This is an example of log rotation:

    static sem_t rotation_requested;

    void* controlthread(void* data){
        ...
            case SIGxxx:                    // signal devoted to rotate logs
                sem_post(&rotation_requested);
                break;
    }

    ... code that initializes the log file
    FILE* logfile;                          // log file
    int fileno = 0;                         // number of log file
    char filename[80];                      // string to build the name of the log file
    int fileopened = 0;                     // tell if log file is open
    sem_init(&rotation_requested,0,1);

    ... code that writes a message in the log file
    if (!sem_trywait(&rotation_requested)){ // a request to rotate has been made
        if (fileopened){                    // the log file is currently open
            fclose(logfile);                // close it
            fileopened = 0;
            fileno++;                       // use a new number next time
        }
    }
    if (!fileopened){                       // log file not open, build a new name and open it
        sprintf(filename,"tmp.log%d",fileno);
        logfile = fopen(filename,"w");
        fileopened = 1;
    }
    fprintf(logfile,message);

    ... code that closes the log file 
    if (fileopened){
        fclose(logfile);
    }
    sem_destroy(&rotation_requested);

N.B. since the flag is shared between two threads, and since there is no guarantee that accesses are not reordered, the flag is implemented with a semaphore. The control thread posts to it to let the other know that there is a request pending, and the other tries to get it, rotating the log when it got a request, and proceeding normally otherwise.

Power failure, suspension and hibernation

In Linux there is no way for a userland application to be notified about the occurrence of power failure, computer suspension or hibernation (and also resume), with signals or otherwise.

Applications could need to perform some operations when these events occur, like, e.g., closing a dial-up Internet connection.

    static __thread int var;  // declares a variable var, private for each thread
    ... var ...               // from within a thread addresses its private one

    if (killFlag) ... kill
    (1)
    sem_wait(sem);
    (2)
    if (killFlag){
        (3)
        if (errno == EINTR){
            (4)
            kill
        }
        (5)
        sem_post(sem);
    }

    (1)
    resourceAllocated = false;
    (2)
    sem_wait(sem);
    (3)
    if (killFlag){
        (4)
        if (errno == EINTR){
            (5)
            kill, but do not release sem
        } else {
            (6)
            kill and release sem
        }
    }
    resourceAllocated = true;

Java

Java does not have a notion of asynchronously interrupting threads. It has a notion of interrupting them at well defined points during execution. When a thread is interrupted, an exception is thrown. This makes interruption blend seamlessly with the exception handling mechanism, allowing thus threads to set up exception handlers to catch them, deciding then to terminate or to repair and recover.

A thread can post an interruption request to another, placing it into an interrupted status that makes it resume with an exception at the next execution of one of a set of (blocking) methods (belonging to several classes). The same occurs if the thread is already blocked in one such methods.

Java does not have a forced thread kill. Thread.destroy() is not implemented. As a consequence, there is no way to cure an application that has a wild thread (as with POSIX threads).

Java programs can register shutdown hooks, which are threads that are executed when the program (actually, the virtual machine) receives a signal.

SIGINT, SIGTERM, SIGHUP make the shutdown hooks run. SIGQUIT produces a dump of the current threads and garbage collection statistics. The other signals abort the program, some with an error message.

References

• http://www.gnu.org/software/libtool/manual/libc/Signal-Handling.html#Signal-Handling good introduction, it does not treat signals in threads
• Linux Application Development, 2nd edition: good, but it does not cover signals with threads
• Advanced programming in the unix environment: more extensive than the preceding two, it describes also the issue of using long jumps and using signals for time supervision.
• Linux programming bible: basic, it does not treat threads and races
• Linux programming by example: basic, but discusses parental control
• Linux system programming: basic
• Programming for AIX: treats signals with threads, but for AIX only
• http://www.linuxprogrammingblog.com/all-about-linux-signals?page=2: basic, but good
• Prentice Hall - Pthreads Primer - A Guide To Multithreaded Programming, Bil Lewis Daniel J. Berg: treats cancellation extensively. It is a bit outdated, though
• Pthreads Programming Bradford Nichols, Dick Buttlar and Jacqueline Proulx Farrell: it contains several examples of cancellation
• Using POSIX Threads: Programming with Pthreads by Brad Nichols,
• Programming with POSIX Threads, David Butenhof
• http://www.win.tue.nl/~aeb/linux/lk/lk.html#toc10: a good handbook of Linux
• MKS man pages
• OpenGroup
• Critical sections
• Programming with POSIX(R) Threads (Addison-Wesley Professional Computing Series) (Paperback) by David R. Butenhof (Author)
• Advanced Linux Programming, Mark Mitchell, Jeffrey Oldham, and Alex Samuel,New Riders Publishing
• O'Reilly, Understanding the Linux Kernel, 3rd edition