I didn’t run the installer as outlined in the post, but the method for 32bit library extraction is by far the easiest and the least likely to break your install. Instead of installing, I untar it somewhere and add a `bcomp` shell script which sets LD_LIBRARY_PATH and then runs bcompare.

Everything ran well, except occasionally a test would time-out. I figured this was a problem with how I was setting up the stdin/stdout pipes to the subtask (each thread would fork() to run the sub-task and use a timed poll() to wait for results). On Friday, I finally sat down to figure out what was causing the test hangs.

Turns out it had nothing to do with sub-task input and output. Each thread is responsible for one subtask. The thread creates the necessary pipes, performs some other book keeping, and then forks. The child process then transforms a vector of command line arguments to an argv[] style array, sets up stdin/stdout with dup2, and calls execvp.

To debug, I modified the parent to print the hung pid which I would attach to using gdb. Every time I attached, the child process was stuck in the same place: waiting on a mutex used by the std::string implementation.

The child process doesn’t use any threading functions, and all of its parameters are passed on the stack. However, I didn’t consider that most of libstdc++ is thread safe (I can’t find a reference that specifies exactly what is and isn’t thread safe in glibc/libstdc++). Each time the process forked, it risked forking while the underlying std::string memory mutex was being held by another thread. As documented in the pthread_atfork man page, those mutexes are not useable after the fork, and must be re-initialized with pthread_mutex_init in the child process. This caused the child process to hang indefinitely waiting on a mutex that would be released only in the parent’s process space.

The solution was to move the std::string processing before the fork. That left only dup2 calls and the execvp after the fork. Result: no more hangs!