...or Another Day in the Trenches Debugging Someone Else's Code
Symptom:
Process occasionally dies with segfaults and other random crashes. It's seemingly non-deterministic... completely random.
Analysis:
The ctime() library function (which produces a formatted string from a UNIX time and date value) isn't thread safe -- it uses a static buffer every time it is called. Internally, the ctime() library function pre-initializes its buffer to empty, by writing a null byte at the beginning every time it is called. This byte is there for only a few nanoseconds, since it is then immediately overwritten by the formatted string. But this brief instant where the string is left empty turned out to be the gasoline that some clueless developer set a match to.
The code in question was calling ctime(), taking the strlen() of the result, and copying that many characters minus 1 to another buffer (allocated on the heap), using strncpy(). The "-1" part was to remove the trailing newline that ctime() appends to the string. But this application is multi-threaded! So once in a while, thread 1 would be taking the strlen() of the result of ctime(), just as thread 2 was starting another ctime() conversion... so thread 1 gets a string length of 0, which it then subtracts 1 from, and consequently passes -1 as the length argument to strncpy().
Well, the length argument to strncpy() is unsigned, so -1 wraps around to the largest positive integer. Also, the strncpy() library function null-pads the destination string out to the specified length. Net effect of this bit of stupidity is to zero out all memory from the destination buffer to the end of the heap arena it was allocated from.
Upon hitting the end of the heap arena, thread 1 gets a segfault, causing the process to die. Or maybe it steps on a data pointer which is just about to be used by another thread, causing that thread to segfault first instead. Or maybe some other thread is trying to allocate or deallocate heap memory while this is going on, causing malloc() or free() to throw an assert.
Conclusion:
I hate amateur night multi-threaded C code...
