From e356ea321c8098cf1a83a67e27d64c44de08a298 Mon Sep 17 00:00:00 2001 From: Eric Andersen Date: Wed, 20 Feb 2002 09:18:50 +0000 Subject: Merge in the pthread library. This is the linuxthreads library taken from glibc 2.1.3 and ported to work with uClibc by Stefan Soucek and Erik Andersen (me). Stefan has hacked things up such that linuxthreads runs on MMU-less systems (tested only on arm-nommu). Erik cleaned things up and made it work properly as a shared library. -Erik --- libpthread/linuxthreads/FAQ.html | 1039 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 1039 insertions(+) create mode 100644 libpthread/linuxthreads/FAQ.html (limited to 'libpthread/linuxthreads/FAQ.html') diff --git a/libpthread/linuxthreads/FAQ.html b/libpthread/linuxthreads/FAQ.html new file mode 100644 index 000000000..21be33ec4 --- /dev/null +++ b/libpthread/linuxthreads/FAQ.html @@ -0,0 +1,1039 @@ + + +LinuxThreads Frequently Asked Questions + + +

LinuxThreads Frequently Asked Questions
+ (with answers)

+

[For LinuxThreads version 0.8]

+ +

+ +A. The big picture
+B. Getting more information
+C. Issues related to the C library
+D. Problems, weird behaviors, potential bugs
+E. Missing functions, wrong types, etc
+F. C++ issues
+G. Debugging LinuxThreads programs
+H. Compiling multithreaded code; errno madness
+I. X-Windows and other libraries
+J. Signals and threads
+K. Internals of LinuxThreads

+ +

+

+ +

A. The big picture

+ +

A.1: What is LinuxThreads?

+ +LinuxThreads is a Linux library for multi-threaded programming. +It implements the Posix 1003.1c API (Application Programming +Interface) for threads. It runs on any Linux system with kernel 2.0.0 +or more recent, and a suitable C library (see section C). +

+ +

A.2: What are threads?

+ +A thread is a sequential flow of control through a program. +Multi-threaded programming is, thus, a form of parallel programming +where several threads of control are executing concurrently in the +program. All threads execute in the same memory space, and can +therefore work concurrently on shared data.

+ +Multi-threaded programming differs from Unix-style multi-processing in +that all threads share the same memory space (and a few other system +resources, such as file descriptors), instead of running in their own +memory space as is the case with Unix processes.

+ +Threads are useful for two reasons. First, they allow a program to +exploit multi-processor machines: the threads can run in parallel on +several processors, allowing a single program to divide its work +between several processors, thus running faster than a single-threaded +program, which runs on only one processor at a time. Second, some +programs are best expressed as several threads of control that +communicate together, rather than as one big monolithic sequential +program. Examples include server programs, overlapping asynchronous +I/O, and graphical user interfaces.

+ +

A.3: What is POSIX 1003.1c?

+ +It's an API for multi-threaded programming standardized by IEEE as +part of the POSIX standards. Most Unix vendors have endorsed the +POSIX 1003.1c standard. Implementations of the 1003.1c API are +already available under Sun Solaris 2.5, Digital Unix 4.0, +Silicon Graphics IRIX 6, and should soon be available from other +vendors such as IBM and HP. More generally, the 1003.1c API is +replacing relatively quickly the proprietary threads library that were +developed previously under Unix, such as Mach cthreads, Solaris +threads, and IRIX sprocs. Thus, multithreaded programs using the +1003.1c API are likely to run unchanged on a wide variety of Unix +platforms.

+ +

A.4: What is the status of LinuxThreads?

+ +LinuxThreads implements almost all of Posix 1003.1c, as well as a few +extensions. The only part of LinuxThreads that does not conform yet +to Posix is signal handling (see section J). Apart +from the signal stuff, all the Posix 1003.1c base functionality, +as well as a number of optional extensions, are provided and conform +to the standard (to the best of my knowledge). +The signal stuff is hard to get right, at least without special kernel +support, and while I'm definitely looking at ways to implement the +Posix behavior for signals, this might take a long time before it's +completed.

+ +

A.5: How stable is LinuxThreads?

+ +The basic functionality (thread creation and termination, mutexes, +conditions, semaphores) is very stable. Several industrial-strength +programs, such as the AOL multithreaded Web server, use LinuxThreads +and seem quite happy about it. There used to be some rough edges in +the LinuxThreads / C library interface with libc 5, but glibc 2 +fixes all of those problems and is now the standard C library on major +Linux distributions (see section C).

+ +

+

+ +

B. Getting more information

+ +

B.1: What are good books and other sources of +information on POSIX threads?

+ +The FAQ for comp.programming.threads lists several books: +http://www.serpentine.com/~bos/threads-faq/.

+ +There are also some online tutorials. Follow the links from the +LinuxThreads web page: +http://pauillac.inria.fr/~xleroy/linuxthreads.

+ +

B.2: I'd like to be informed of future developments on +LinuxThreads. Is there a mailing list for this purpose?

+ +I post LinuxThreads-related announcements on the newsgroup +comp.os.linux.announce, +and also on the mailing list +linux-threads@magenet.com. +You can subscribe to the latter by writing +majordomo@magenet.com.

+ +

B.3: What are good places for discussing +LinuxThreads?

+ +For questions about programming with POSIX threads in general, use +the newsgroup +comp.programming.threads. +Be sure you read the +FAQ +for this group before you post.

+ +For Linux-specific questions, use +comp.os.linux.development.apps +and comp.os.linux.development.kernel. +The latter is especially appropriate for questions relative to the +interface between the kernel and LinuxThreads.

+ +

B.4: How should I report a possible bug in +LinuxThreads?

+ +If you're using glibc 2, the best way by far is to use the +glibcbug script to mail a bug report to the glibc +maintainers.

+ +If you're using an older libc, or don't have the glibcbug +script on your machine, then e-mail me directly +(Xavier.Leroy@inria.fr).

+ +In both cases, before sending the bug report, make sure that it is not +addressed already in this FAQ. Also, try to send a short program that +reproduces the weird behavior you observed.

+ +

B.5: I'd like to read the POSIX 1003.1c standard. Is +it available online?

+ +Unfortunately, no. POSIX standards are copyrighted by IEEE, and +IEEE does not distribute them freely. You can buy paper copies from +IEEE, but the price is fairly high ($120 or so). If you disagree with +this policy and you're an IEEE member, be sure to let them know.

+ +On the other hand, you probably don't want to read the standard. It's +very hard to read, written in standard-ese, and targeted to +implementors who already know threads inside-out. A good book on +POSIX threads provides the same information in a much more readable form. +I can personally recommend Dave Butenhof's book, Programming +with POSIX threads (Addison-Wesley). Butenhof was part of the +POSIX committee and also designed the Digital Unix implementations of +POSIX threads, and it shows.

+ +Another good source of information is the X/Open Group Single Unix +specification which is available both +on-line +and as a +book and CD/ROM. +That specification includes pretty much all the POSIX standards, +including 1003.1c, with some extensions and clarifications.

+ +

+

+ +

C. Issues related to the C library

+ +

C.1: Which version of the C library should I use +with LinuxThreads?

+ +The best choice by far is glibc 2, a.k.a. libc 6. It offers very good +support for multi-threading, and LinuxThreads has been closely +integrated with glibc 2. The glibc 2 distribution contains the +sources of a specially adapted version of LinuxThreads.

+ +glibc 2 comes preinstalled as the default C library on several Linux +distributions, such as RedHat 5 and up, and Debian 2. +Those distributions include the version of LinuxThreads matching +glibc 2.

+ +

C.2: My system has libc 5 preinstalled, not glibc +2. Can I still use LinuxThreads?

+ +Yes, but you're likely to run into some problems, as libc 5 only +offers minimal support for threads and contains some bugs that affect +multithreaded programs.

+ +The versions of libc 5 that work best with LinuxThreads are +libc 5.2.18 on the one hand, and libc 5.4.12 or later on the other hand. +Avoid 5.3.12 and 5.4.7: these have problems with the per-thread errno +variable.

+ +

C.3: So, should I switch to glibc 2, or stay with a +recent libc 5?

+ +I'd recommend you switch to glibc 2. Even for single-threaded +programs, glibc 2 is more solid and more standard-conformant than libc +5. And the shortcomings of libc 5 almost preclude any serious +multi-threaded programming.

+ +Switching an already installed +system from libc 5 to glibc 2 is not completely straightforward. +See the Glibc2 +HOWTO for more information. Much easier is (re-)installing a +Linux distribution based on glibc 2, such as RedHat 6.

+ +

C.4: Where can I find glibc 2 and the version of +LinuxThreads that goes with it?

+ +On prep.ai.mit.edu and its many, many mirrors around the world. +See http://www.gnu.org/order/ftp.html +for a list of mirrors.

+ +

C.5: Where can I find libc 5 and the version of +LinuxThreads that goes with it?

+ +For libc 5, see ftp://sunsite.unc.edu/pub/Linux/devel/GCC/.

+ +For the libc 5 version of LinuxThreads, see +ftp://ftp.inria.fr/INRIA/Projects/cristal/Xavier.Leroy/linuxthreads/.

+ +

C.6: How can I recompile the glibc 2 version of the +LinuxThreads sources?

+ +You must transfer the whole glibc sources, then drop the LinuxThreads +sources in the linuxthreads/ subdirectory, then recompile +glibc as a whole. There are now too many inter-dependencies between +LinuxThreads and glibc 2 to allow separate re-compilation of LinuxThreads. +

+ +

C.7: What is the correspondence between LinuxThreads +version numbers, libc version numbers, and RedHat version +numbers?

+ +Here is a summary. (Information on Linux distributions other than +RedHat are welcome.)

+ + + + + + + +
LinuxThreads C library RedHat
0.7, 0.71 (for libc 5) libc 5.x RH 4.2
0.7, 0.71 (for glibc 2) glibc 2.0.x RH 5.x
0.8 glibc 2.1.1 RH 6.0
0.8 glibc 2.1.2 not yet released
+

+ +

+

+ +

D. Problems, weird behaviors, potential bugs

+ +

D.1: When I compile LinuxThreads, I run into problems in +file libc_r/dirent.c

+ +You probably mean: +
+        libc_r/dirent.c:94: structure has no member named `dd_lock'
+
+I haven't actually seen this problem, but several users reported it. +My understanding is that something is wrong in the include files of +your Linux installation (/usr/include/*). Make sure +you're using a supported version of the libc 5 library. (See question C.2).

+ +

D.2: When I compile LinuxThreads, I run into problems with +/usr/include/sched.h: there are several occurrences of +_p that the C compiler does not understand

+ +Yes, /usr/include/sched.h that comes with libc 5.3.12 is broken. +Replace it with the sched.h file contained in the +LinuxThreads distribution. But really you should not be using libc +5.3.12 with LinuxThreads! (See question C.1.)

+ +

D.3: My program does fdopen() on a file +descriptor opened on a pipe. When I link it with LinuxThreads, +fdopen() always returns NULL!

+ +You're using one of the buggy versions of libc (5.3.12, 5.4.7., etc). +See question C.1 above.

+ +

D.4: My program creates a lot of threads, and after +a while pthread_create() no longer returns!

+ +This is known bug in the version of LinuxThreads that comes with glibc +2.1.1. An upgrade to 2.1.2 is recommended.

+ +

D.5: When I'm running a program that creates N +threads, top or ps +display N+2 processes that are running my program. What do all these +processes correspond to?

+ +Due to the general "one process per thread" model, there's one process +for the initial thread and N processes for the threads it created +using pthread_create. That leaves one process +unaccounted for. That extra process corresponds to the "thread +manager" thread, a thread created internally by LinuxThreads to handle +thread creation and thread termination. This extra thread is asleep +most of the time. + +

D.6: Scheduling seems to be very unfair when there +is strong contention on a mutex: instead of giving the mutex to each +thread in turn, it seems that it's almost always the same thread that +gets the mutex. Isn't this completely broken behavior?

+ +That behavior has mostly disappeared in recent releases of +LinuxThreads (version 0.8 and up). It was fairly common in older +releases, though. + +What happens in LinuxThreads 0.7 and before is the following: when a +thread unlocks a mutex, all other threads that were waiting on the +mutex are sent a signal which makes them runnable. However, the +kernel scheduler may or may not restart them immediately. If the +thread that unlocked the mutex tries to lock it again immediately +afterwards, it is likely that it will succeed, because the threads +haven't yet restarted. This results in an apparently very unfair +behavior, when the same thread repeatedly locks and unlocks the mutex, +while other threads can't lock the mutex.

+ +In LinuxThreads 0.8 and up, pthread_unlock restarts only +one waiting thread, and pre-assign the mutex to that thread. Hence, +if the thread that unlocked the mutex tries to lock it again +immediately, it will block until other waiting threads have had a +chance to lock and unlock the mutex. This results in much fairer +scheduling.

+ +Notice however that even the old "unfair" behavior is perfectly +acceptable with respect to the POSIX standard: for the default +scheduling policy, POSIX makes no guarantees of fairness, such as "the +thread waiting for the mutex for the longest time always acquires it +first". Properly written multithreaded code avoids that kind of heavy +contention on mutexes, and does not run into fairness problems. If +you need scheduling guarantees, you should consider using the +real-time scheduling policies SCHED_RR and +SCHED_FIFO, which have precisely defined scheduling +behaviors.

+ +

D.7: I have a simple test program with two threads +that do nothing but printf() in tight loops, and from the +printout it seems that only one thread is running, the other doesn't +print anything!

+ +Again, this behavior is characteristic of old releases of LinuxThreads +(0.7 and before); more recent versions (0.8 and up) should not exhibit +this behavior.

+ +The reason for this behavior is explained in +question D.6 above: printf() performs +locking on stdout, and thus your two threads contend very +heavily for the mutex associated with stdout. But if you +do some real work between two calls to printf(), you'll +see that scheduling becomes much smoother.

+ +

D.8: I've looked at <pthread.h> +and there seems to be a gross error in the pthread_cleanup_push +macro: it opens a block with { but does not close it! +Surely you forgot a } at the end of the macro, right? +

+ +Nope. That's the way it should be. The closing brace is provided by +the pthread_cleanup_pop macro. The POSIX standard +requires pthread_cleanup_push and +pthread_cleanup_pop to be used in matching pairs, at the +same level of brace nesting. This allows +pthread_cleanup_push to open a block in order to +stack-allocate some data structure, and +pthread_cleanup_pop to close that block. It's ugly, but +it's the standard way of implementing cleanup handlers.

+ +

D.9: I tried to use real-time threads and my program +loops like crazy and freezes the whole machine!

+ +Versions of LinuxThreads prior to 0.8 are susceptible to ``livelocks'' +(one thread loops, consuming 100% of the CPU time) in conjunction with +real-time scheduling. Since real-time threads and processes have +higher priority than normal Linux processes, all other processes on +the machine, including the shell, the X server, etc, cannot run and +the machine appears frozen.

+ +The problem is fixed in LinuxThreads 0.8.

+ +

D.10: My application needs to create thousands of +threads, or maybe even more. Can I do this with +LinuxThreads?

+ +No. You're going to run into several hard limits: + +(Other POSIX threads libraries have similar limitations, by the way.) +For all those reasons, you'd better restructure your application so +that it doesn't need more than, say, 100 threads. For instance, +in the case of a multithreaded server, instead of creating a new +thread for each connection, maintain a fixed-size pool of worker +threads that pick incoming connection requests from a queue.

+ +

+

+ +

E. Missing functions, wrong types, etc

+ +

E.1: Where is pthread_yield() ? How +comes LinuxThreads does not implement it?

+ +Because it's not part of the (final) POSIX 1003.1c standard. +Several drafts of the standard contained pthread_yield(), +but then the POSIX guys discovered it was redundant with +sched_yield() and dropped it. So, just use +sched_yield() instead. + +

E.2: I've found some type errors in +<pthread.h>. +For instance, the second argument to pthread_create() +should be a pthread_attr_t, not a +pthread_attr_t *. Also, didn't you forget to declare +pthread_attr_default?

+ +No, I didn't. What you're describing is draft 4 of the POSIX +standard, which is used in OSF DCE threads. LinuxThreads conforms to the +final standard. Even though the functions have the same names as in +draft 4 and DCE, their calling conventions are slightly different. In +particular, attributes are passed by reference, not by value, and +default attributes are denoted by the NULL pointer. Since draft 4/DCE +will eventually disappear, you'd better port your program to use the +standard interface.

+ +

E.3: I'm porting an application from Solaris and I +have to rename all thread functions from thr_blah to +pthread_blah. This is very annoying. Why did you change +all the function names?

+ +POSIX did it. The thr_* functions correspond to Solaris +threads, an older thread interface that you'll find only under +Solaris. The pthread_* functions correspond to POSIX +threads, an international standard available for many, many platforms. +Even Solaris 2.5 and later support the POSIX threads interface. So, +do yourself a favor and rewrite your code to use POSIX threads: this +way, it will run unchanged under Linux, Solaris, and quite a lot of +other platforms.

+ +

E.4: How can I suspend and resume a thread from +another thread? Solaris has the thr_suspend() and +thr_resume() functions to do that; why don't you?

+ +The POSIX standard provides no mechanism by which a thread A can +suspend the execution of another thread B, without cooperation from B. +The only way to implement a suspend/restart mechanism is to have B +check periodically some global variable for a suspend request +and then suspend itself on a condition variable, which another thread +can signal later to restart B.

+ +Notice that thr_suspend() is inherently dangerous and +prone to race conditions. For one thing, there is no control on where +the target thread stops: it can very well be stopped in the middle of +a critical section, while holding mutexes. Also, there is no +guarantee on when the target thread will actually stop. For these +reasons, you'd be much better off using mutexes and conditions +instead. The only situations that really require the ability to +suspend a thread are debuggers and some kind of garbage collectors.

+ +If you really must suspend a thread in LinuxThreads, you can send it a +SIGSTOP signal with pthread_kill. Send +SIGCONT for restarting it. +Beware, this is specific to LinuxThreads and entirely non-portable. +Indeed, a truly conforming POSIX threads implementation will stop all +threads when one thread receives the SIGSTOP signal! +One day, LinuxThreads will implement that behavior, and the +non-portable hack with SIGSTOP won't work anymore.

+ +

E.5: Does LinuxThreads implement +pthread_attr_setstacksize() and +pthread_attr_setstackaddr()?

+ +These optional functions are provided in recent versions of +LinuxThreads (0.8 and up). Earlier releases did not provide these +optional components of the POSIX standard.

+ +Even if pthread_attr_setstacksize() and +pthread_attr_setstackaddr() are now provided, we still +recommend that you do not use them unless you really have strong +reasons for doing so. The default stack allocation strategy for +LinuxThreads is nearly optimal: stacks start small (4k) and +automatically grow on demand to a fairly large limit (2M). +Moreover, there is no portable way to estimate the stack requirements +of a thread, so setting the stack size yourself makes your program +less reliable and non-portable.

+ +

E.6: LinuxThreads does not support the +PTHREAD_SCOPE_PROCESS value of the "contentionscope" +attribute. Why?

+ +With a "one-to-one" model, as in LinuxThreads (one kernel execution +context per thread), there is only one scheduler for all processes and +all threads on the system. So, there is no way to obtain the behavior of +PTHREAD_SCOPE_PROCESS. + +

E.7: LinuxThreads does not implement process-shared +mutexes, conditions, and semaphores. Why?

+ +This is another optional component of the POSIX standard. Portable +applications should test _POSIX_THREAD_PROCESS_SHARED +before using this facility. +

+The goal of this extension is to allow different processes (with +different address spaces) to synchronize through mutexes, conditions +or semaphores allocated in shared memory (either SVR4 shared memory +segments or mmap()ed files). +

+The reason why this does not work in LinuxThreads is that mutexes, +conditions, and semaphores are not self-contained: their waiting +queues contain pointers to linked lists of thread descriptors, and +these pointers are meaningful only in one address space. +

+Matt Messier and I spent a significant amount of time trying to design a +suitable mechanism for sharing waiting queues between processes. We +came up with several solutions that combined two of the following +three desirable features, but none that combines all three: +

+We concluded that kernel support is required to share mutexes, +conditions and semaphores between processes. That's one place where +Linus Torvalds's intuition that "all we need in the kernel is +clone()" fails. +

+Until suitable kernel support is available, you'd better use +traditional interprocess communications to synchronize different +processes: System V semaphores and message queues, or pipes, or sockets. +

+ +

+

+ +

F. C++ issues

+ +

F.1: Are there C++ wrappers for LinuxThreads?

+ +Douglas Schmidt's ACE library contains, among a lot of other +things, C++ wrappers for LinuxThreads and quite a number of other +thread libraries. Check out +http://www.cs.wustl.edu/~schmidt/ACE.html

+ +

F.2: I'm trying to use LinuxThreads from a C++ +program, and the compiler complains about the third argument to +pthread_create() !

+ +You're probably trying to pass a class member function or some +other C++ thing as third argument to pthread_create(). +Recall that pthread_create() is a C function, and it must +be passed a C function as third argument.

+ +

F.3: I'm trying to use LinuxThreads in conjunction +with libg++, and I'm having all sorts of trouble.

+ +>From what I understand, thread support in libg++ is completely broken, +especially with respect to locking of iostreams. H.J.Lu wrote: +
+If you want to use thread, I can only suggest egcs and glibc. You +can find egcs at +http://www.cygnus.com/egcs. +egcs has libsdtc++, which is MT safe under glibc 2. If you really +want to use the libg++, I have a libg++ add-on for egcs. +
+
+

+ +

G. Debugging LinuxThreads programs

+ +

G.1: Can I debug LinuxThreads program using gdb?

+ +Yes, but not with the stock gdb 4.17. You need a specially patched +version of gdb 4.17 developed by Eric Paire and colleages at The Open +Group, Grenoble. The patches against gdb 4.17 are available at +http://www.gr.opengroup.org/java/jdk/linux/debug.htm. +Precompiled binaries of the patched gdb are available in RedHat's RPM +format at http://odin.appliedtheory.com/.

+ +Some Linux distributions provide an already-patched version of gdb; +others don't. For instance, the gdb in RedHat 5.2 is thread-aware, +but apparently not the one in RedHat 6.0. Just ask (politely) the +makers of your Linux distributions to please make sure that they apply +the correct patches to gdb.

+ +

G.2: Does it work with post-mortem debugging?

+ +Not very well. Generally, the core file does not correspond to the +thread that crashed. The reason is that the kernel will not dump core +for a process that shares its memory with other processes, such as the +other threads of your program. So, the thread that crashes silently +disappears without generating a core file. Then, all other threads of +your program die on the same signal that killed the crashing thread. +(This is required behavior according to the POSIX standard.) The last +one that dies is no longer sharing its memory with anyone else, so the +kernel generates a core file for that thread. Unfortunately, that's +not the thread you are interested in. + +

G.3: Any other ways to debug multithreaded programs, then?

+ +Assertions and printf() are your best friends. Try to debug +sequential parts in a single-threaded program first. Then, put +printf() statements all over the place to get execution traces. +Also, check invariants often with the assert() macro. In truth, +there is no other effective way (save for a full formal proof of your +program) to track down concurrency bugs. Debuggers are not really +effective for subtle concurrency problems, because they disrupt +program execution too much.

+ +

+

+ +

H. Compiling multithreaded code; errno madness

+ +

H.1: You say all multithreaded code must be compiled +with _REENTRANT defined. What difference does it make?

+ +It affects include files in three ways: + +

+ +

H.2: Why is it so important that each thread has its +own errno variable?

+ +If all threads were to store error codes in the same, global errno +variable, then the value of errno after a system call or library +function returns would be unpredictable: between the time a system +call stores its error code in the global errno and your code inspects +errno to see which error occurred, another thread might have stored +another error code in the same errno location.

+ +

H.3: What happens if I link LinuxThreads with code +not compiled with -D_REENTRANT?

+ +Lots of trouble. If the code uses getc() or +putc(), it will perform I/O without proper interlocking +of the stdio buffers; this can cause lost output, duplicate output, or +just crash other stdio functions. If the code consults errno, it will +get back the wrong error code. The following code fragment is a +typical example: +
+        do {
+          r = read(fd, buf, n);
+          if (r == -1) {
+            if (errno == EINTR)   /* an error we can handle */
+              continue;
+            else {                /* other errors are fatal */
+              perror("read failed");
+              exit(100);
+            }
+          }
+        } while (...);
+
+Assume this code is not compiled with -D_REENTRANT, and +linked with LinuxThreads. At run-time, read() is +interrupted. Since the C library was compiled with +-D_REENTRANT, read() stores its error code +in the location pointed to by __errno_location(), which +is the thread-local errno variable. Then, the code above sees that +read() returns -1 and looks up errno. Since +_REENTRANT is not defined, the reference to errno +accesses the global errno variable, which is most likely 0. Hence the +code concludes that it cannot handle the error and stops.

+ +

H.4: With LinuxThreads, I can no longer use the signals +SIGUSR1 and SIGUSR2 in my programs! Why?

+ +The short answer is: because the Linux kernel you're using does not +support realtime signals.

+ +LinuxThreads needs two signals for its internal operation. +One is used to suspend and restart threads blocked on mutex, condition +or semaphore operations. The other is used for thread +cancellation.

+ +On ``old'' kernels (2.0 and early 2.1 kernels), there are only 32 +signals available and the kernel reserves all of them but two: +SIGUSR1 and SIGUSR2. So, LinuxThreads has +no choice but use those two signals.

+ +On recent kernels (2.2 and up), more than 32 signals are provided in +the form of realtime signals. When run on one of those kernels, +LinuxThreads uses two reserved realtime signals for its internal +operation, thus leaving SIGUSR1 and SIGUSR2 +free for user code. (This works only with glibc, not with libc 5.)

+ +

H.5: Is the stack of one thread visible from the +other threads? Can I pass a pointer into my stack to other threads? +

+ +Yes, you can -- if you're very careful. The stacks are indeed visible +from all threads in the system. Some non-POSIX thread libraries seem +to map the stacks for all threads at the same virtual addresses and +change the memory mapping when they switch from one thread to +another. But this is not the case for LinuxThreads, as it would make +context switching between threads more expensive, and at any rate +might not conform to the POSIX standard.

+ +So, you can take the address of an "auto" variable and pass it to +other threads via shared data structures. However, you need to make +absolutely sure that the function doing this will not return as long +as other threads need to access this address. It's the usual mistake +of returning the address of an "auto" variable, only made much worse +because of concurrency. It's much, much safer to systematically +heap-allocate all shared data structures.

+ +

+

+ +

I. X-Windows and other libraries

+ +

I.1: My program uses both Xlib and LinuxThreads. +It stops very early with an "Xlib: unknown 0 error" message. What +does this mean?

+ +That's a prime example of the errno problem described in question H.2. The binaries for Xlib you're using have not been +compiled with -D_REENTRANT. It happens Xlib contains a +piece of code very much like the one in question H.2. So, your Xlib fetches the error code from the +wrong errno location and concludes that an error it cannot handle +occurred.

+ +

I.2: So, what can I do to build a multithreaded X +Windows client?

+ +The best solution is to use X libraries that have been compiled with +multithreading options set. Linux distributions that come with glibc +2 as the main C library generally provide thread-safe X libraries. +At least, that seems to be the case for RedHat 5 and later.

+ +You can try to recompile yourself the X libraries with multithreading +options set. They contain optional support for multithreading; it's +just that the binaries provided by your Linux distribution were built +without this support. See the file README.Xfree3.3 in +the LinuxThreads distribution for patches and info on how to compile +thread-safe X libraries from the Xfree3.3 distribution. The Xfree3.3 +sources are readily available in most Linux distributions, e.g. as a +source RPM for RedHat. Be warned, however, that X Windows is a huge +system, and recompiling even just the libraries takes a lot of time +and disk space.

+ +Another, less involving solution is to call X functions only from the +main thread of your program. Even if all threads have their own errno +location, the main thread uses the global errno variable for its errno +location. Thus, code not compiled with -D_REENTRANT +still "sees" the right error values if it executes in the main thread +only.

+ +

This is a lot of work. Don't you have precompiled +thread-safe X libraries that you could distribute?

+ +No, I don't. Sorry. But consider installing a Linux distribution +that comes with thread-safe X libraries, such as RedHat 6.

+ +

I.3: Can I use library FOO in a multithreaded +program?

+ +Most libraries cannot be used "as is" in a multithreaded program. +For one thing, they are not necessarily thread-safe: calling +simultaneously two functions of the library from two threads might not +work, due to internal use of global variables and the like. Second, +the libraries must have been compiled with -D_REENTRANT to avoid +the errno problems explained in question H.2. +

+ +

I.4: What if I make sure that only one thread calls +functions in these libraries?

+ +This avoids problems with the library not being thread-safe. But +you're still vulnerable to errno problems. At the very least, a +recompile of the library with -D_REENTRANT is needed. +

+ +

I.5: What if I make sure that only the main thread +calls functions in these libraries?

+ +That might actually work. As explained in question I.1, +the main thread uses the global errno variable, and can therefore +execute code not compiled with -D_REENTRANT.

+ +

I.6: SVGAlib doesn't work with LinuxThreads. Why? +

+ +Because both LinuxThreads and SVGAlib use the signals +SIGUSR1 and SIGUSR2. See question H.4. +

+ + +

+

+ +

J. Signals and threads

+ +

J.1: When it comes to signals, what is shared +between threads and what isn't?

+ +Signal handlers are shared between all threads: when a thread calls +sigaction(), it sets how the signal is handled not only +for itself, but for all other threads in the program as well.

+ +On the other hand, signal masks are per-thread: each thread chooses +which signals it blocks independently of others. At thread creation +time, the newly created thread inherits the signal mask of the thread +calling pthread_create(). But afterwards, the new thread +can modify its signal mask independently of its creator thread.

+ +

J.2: When I send a SIGKILL to a +particular thread using pthread_kill, all my threads are +killed!

+ +That's how it should be. The POSIX standard mandates that all threads +should terminate when the process (i.e. the collection of all threads +running the program) receives a signal whose effect is to +terminate the process (such as SIGKILL or SIGINT +when no handler is installed on that signal). This behavior makes a +lot of sense: when you type "ctrl-C" at the keyboard, or when a thread +crashes on a division by zero or a segmentation fault, you really want +all threads to stop immediately, not just the one that caused the +segmentation violation or that got the SIGINT signal. +(This assumes default behavior for those signals; see question +J.3 if you install handlers for those signals.)

+ +If you're trying to terminate a thread without bringing the whole +process down, use pthread_cancel().

+ +

J.3: I've installed a handler on a signal. Which +thread executes the handler when the signal is received?

+ +If the signal is generated by a thread during its execution (e.g. a +thread executes a division by zero and thus generates a +SIGFPE signal), then the handler is executed by that +thread. This also applies to signals generated by +raise().

+ +If the signal is sent to a particular thread using +pthread_kill(), then that thread executes the handler.

+ +If the signal is sent via kill() or the tty interface +(e.g. by pressing ctrl-C), then the POSIX specs say that the handler +is executed by any thread in the process that does not currently block +the signal. In other terms, POSIX considers that the signal is sent +to the process (the collection of all threads) as a whole, and any +thread that is not blocking this signal can then handle it.

+ +The latter case is where LinuxThreads departs from the POSIX specs. +In LinuxThreads, there is no real notion of ``the process as a whole'': +in the kernel, each thread is really a distinct process with a +distinct PID, and signals sent to the PID of a thread can only be +handled by that thread. As long as no thread is blocking the signal, +the behavior conforms to the standard: one (unspecified) thread of the +program handles the signal. But if the thread to which PID the signal +is sent blocks the signal, and some other thread does not block the +signal, then LinuxThreads will simply queue in +that thread and execute the handler only when that thread unblocks +the signal, instead of executing the handler immediately in the other +thread that does not block the signal.

+ +This is to be viewed as a LinuxThreads bug, but I currently don't see +any way to implement the POSIX behavior without kernel support.

+ +

J.3: How shall I go about mixing signals and threads +in my program?

+ +The less you mix them, the better. Notice that all +pthread_* functions are not async-signal safe, meaning +that you should not call them from signal handlers. This +recommendation is not to be taken lightly: your program can deadlock +if you call a pthread_* function from a signal handler! +

+ +The only sensible things you can do from a signal handler is set a +global flag, or call sem_post on a semaphore, to record +the delivery of the signal. The remainder of the program can then +either poll the global flag, or use sem_wait() and +sem_trywait() on the semaphore.

+ +Another option is to do nothing in the signal handler, and dedicate +one thread (preferably the initial thread) to wait synchronously for +signals, using sigwait(), and send messages to the other +threads accordingly. + +

J.4: When one thread is blocked in +sigwait(), other threads no longer receive the signals +sigwait() is waiting for! What happens?

+ +It's an unfortunate consequence of how LinuxThreads implements +sigwait(). Basically, it installs signal handlers on all +signals waited for, in order to record which signal was received. +Since signal handlers are shared with the other threads, this +temporarily deactivates any signal handlers you might have previously +installed on these signals.

+ +Though surprising, this behavior actually seems to conform to the +POSIX standard. According to POSIX, sigwait() is +guaranteed to work as expected only if all other threads in the +program block the signals waited for (otherwise, the signals could be +delivered to other threads than the one doing sigwait(), +which would make sigwait() useless). In this particular +case, the problem described in this question does not appear.

+ +One day, sigwait() will be implemented in the kernel, +along with others POSIX 1003.1b extensions, and sigwait() +will have a more natural behavior (as well as better performances).

+ +

+

+ +

K. Internals of LinuxThreads

+ +

K.1: What is the implementation model for +LinuxThreads?

+ +LinuxThreads follows the so-called "one-to-one" model: each thread is +actually a separate process in the kernel. The kernel scheduler takes +care of scheduling the threads, just like it schedules regular +processes. The threads are created with the Linux +clone() system call, which is a generalization of +fork() allowing the new process to share the memory +space, file descriptors, and signal handlers of the parent.

+ +Advantages of the "one-to-one" model include: +

+The main disadvantage is more expensive context switches on mutex and +condition operations, which must go through the kernel. This is +mitigated by the fact that context switches in the Linux kernel are +pretty efficient.

+ +

K.2: Have you considered other implementation +models?

+ +There are basically two other models. The "many-to-one" model +relies on a user-level scheduler that context-switches between the +threads entirely in user code; viewed from the kernel, there is only +one process running. This model is completely out of the question for +me, since it does not take advantage of multiprocessors, and require +unholy magic to handle blocking I/O operations properly. There are +several user-level thread libraries available for Linux, but I found +all of them deficient in functionality, performance, and/or robustness. +

+ +The "many-to-many" model combines both kernel-level and user-level +scheduling: several kernel-level threads run concurrently, each +executing a user-level scheduler that selects between user threads. +Most commercial Unix systems (Solaris, Digital Unix, IRIX) implement +POSIX threads this way. This model combines the advantages of both +the "many-to-one" and the "one-to-one" model, and is attractive +because it avoids the worst-case behaviors of both models -- +especially on kernels where context switches are expensive, such as +Digital Unix. Unfortunately, it is pretty complex to implement, and +requires kernel support which Linux does not provide. Linus Torvalds +and other Linux kernel developers have always been pushing the +"one-to-one" model in the name of overall simplicity, and are doing a +pretty good job of making kernel-level context switches between +threads efficient. LinuxThreads is just following the general +direction they set.

+ +

+
Xavier.Leroy@inria.fr
+ + -- cgit v1.2.3