Everything ran well, except occasionally a test would time-out. I figured this was a problem with how I was setting up the stdin/stdout pipes to the subtask (each thread would fork() to run the sub-task and use a timed poll() to wait for results). On Friday, I finally sat down to figure out what was causing the test hangs.

Turns out it had nothing to do with sub-task input and output. Each thread is responsible for one subtask. The thread creates the necessary pipes, performs some other book keeping, and then forks. The child process then transforms a vector of command line arguments to an argv[] style array, sets up stdin/stdout with dup2, and calls execvp.

To debug, I modified the parent to print the hung pid which I would attach to using gdb. Every time I attached, the child process was stuck in the same place: waiting on a mutex used by the std::string implementation.

The child process doesn’t use any threading functions, and all of its parameters are passed on the stack. However, I didn’t consider that most of libstdc++ is thread safe (I can’t find a reference that specifies exactly what is and isn’t thread safe in glibc/libstdc++). Each time the process forked, it risked forking while the underlying std::string memory mutex was being held by another thread. As documented in the pthread_atfork man page, those mutexes are not useable after the fork, and must be re-initialized with pthread_mutex_init in the child process. This caused the child process to hang indefinitely waiting on a mutex that would be released only in the parent’s process space.

The solution was to move the std::string processing before the fork. That left only dup2 calls and the execvp after the fork. Result: no more hangs!