diff options
46 files changed, 1476 insertions, 851 deletions
@@ -209,6 +209,10 @@ Paolo 'Blaisorblade' Giarrusso <blaisorblade@yahoo.it> Patrick Mochel <mochel@digitalimplant.org> Paul Burton <paulburton@kernel.org> <paul.burton@imgtec.com> Paul Burton <paulburton@kernel.org> <paul.burton@mips.com> +Paul E. McKenney <paulmck@kernel.org> <paulmck@linux.ibm.com> +Paul E. McKenney <paulmck@kernel.org> <paulmck@linux.vnet.ibm.com> +Paul E. McKenney <paulmck@kernel.org> <paul.mckenney@linaro.org> +Paul E. McKenney <paulmck@kernel.org> <paulmck@us.ibm.com> Peter A Jonsson <pj@ludd.ltu.se> Peter Oruba <peter@oruba.de> Peter Oruba <peter.oruba@amd.com> diff --git a/Documentation/RCU/NMI-RCU.txt b/Documentation/RCU/NMI-RCU.rst index 881353fd5bff..180958388ff9 100644 --- a/Documentation/RCU/NMI-RCU.txt +++ b/Documentation/RCU/NMI-RCU.rst @@ -1,4 +1,7 @@ +.. _NMI_rcu_doc: + Using RCU to Protect Dynamic NMI Handlers +========================================= Although RCU is usually used to protect read-mostly data structures, @@ -9,7 +12,7 @@ work in "arch/x86/oprofile/nmi_timer_int.c" and in "arch/x86/kernel/traps.c". The relevant pieces of code are listed below, each followed by a -brief explanation. +brief explanation:: static int dummy_nmi_callback(struct pt_regs *regs, int cpu) { @@ -18,12 +21,12 @@ brief explanation. The dummy_nmi_callback() function is a "dummy" NMI handler that does nothing, but returns zero, thus saying that it did nothing, allowing -the NMI handler to take the default machine-specific action. +the NMI handler to take the default machine-specific action:: static nmi_callback_t nmi_callback = dummy_nmi_callback; This nmi_callback variable is a global function pointer to the current -NMI handler. +NMI handler:: void do_nmi(struct pt_regs * regs, long error_code) { @@ -53,11 +56,12 @@ anyway. However, in practice it is a good documentation aid, particularly for anyone attempting to do something similar on Alpha or on systems with aggressive optimizing compilers. -Quick Quiz: Why might the rcu_dereference_sched() be necessary on Alpha, - given that the code referenced by the pointer is read-only? +Quick Quiz: + Why might the rcu_dereference_sched() be necessary on Alpha, given that the code referenced by the pointer is read-only? +:ref:`Answer to Quick Quiz <answer_quick_quiz_NMI>` -Back to the discussion of NMI and RCU... +Back to the discussion of NMI and RCU:: void set_nmi_callback(nmi_callback_t callback) { @@ -68,7 +72,7 @@ The set_nmi_callback() function registers an NMI handler. Note that any data that is to be used by the callback must be initialized up -before- the call to set_nmi_callback(). On architectures that do not order writes, the rcu_assign_pointer() ensures that the NMI handler sees the -initialized values. +initialized values:: void unset_nmi_callback(void) { @@ -82,7 +86,7 @@ up any data structures used by the old NMI handler until execution of it completes on all other CPUs. One way to accomplish this is via synchronize_rcu(), perhaps as -follows: +follows:: unset_nmi_callback(); synchronize_rcu(); @@ -98,24 +102,23 @@ to free up the handler's data as soon as synchronize_rcu() returns. Important note: for this to work, the architecture in question must invoke nmi_enter() and nmi_exit() on NMI entry and exit, respectively. +.. _answer_quick_quiz_NMI: -Answer to Quick Quiz - - Why might the rcu_dereference_sched() be necessary on Alpha, given - that the code referenced by the pointer is read-only? +Answer to Quick Quiz: + Why might the rcu_dereference_sched() be necessary on Alpha, given that the code referenced by the pointer is read-only? - Answer: The caller to set_nmi_callback() might well have - initialized some data that is to be used by the new NMI - handler. In this case, the rcu_dereference_sched() would - be needed, because otherwise a CPU that received an NMI - just after the new handler was set might see the pointer - to the new NMI handler, but the old pre-initialized - version of the handler's data. + The caller to set_nmi_callback() might well have + initialized some data that is to be used by the new NMI + handler. In this case, the rcu_dereference_sched() would + be needed, because otherwise a CPU that received an NMI + just after the new handler was set might see the pointer + to the new NMI handler, but the old pre-initialized + version of the handler's data. - This same sad story can happen on other CPUs when using - a compiler with aggressive pointer-value speculation - optimizations. + This same sad story can happen on other CPUs when using + a compiler with aggressive pointer-value speculation + optimizations. - More important, the rcu_dereference_sched() makes it - clear to someone reading the code that the pointer is - being protected by RCU-sched. + More important, the rcu_dereference_sched() makes it + clear to someone reading the code that the pointer is + being protected by RCU-sched. diff --git a/Documentation/RCU/arrayRCU.txt b/Documentation/RCU/arrayRCU.rst index f05a9afb2c39..4051ea3871ef 100644 --- a/Documentation/RCU/arrayRCU.txt +++ b/Documentation/RCU/arrayRCU.rst @@ -1,19 +1,21 @@ -Using RCU to Protect Read-Mostly Arrays +.. _array_rcu_doc: +Using RCU to Protect Read-Mostly Arrays +======================================= Although RCU is more commonly used to protect linked lists, it can also be used to protect arrays. Three situations are as follows: -1. Hash Tables +1. :ref:`Hash Tables <hash_tables>` -2. Static Arrays +2. :ref:`Static Arrays <static_arrays>` -3. Resizeable Arrays +3. :ref:`Resizable Arrays <resizable_arrays>` Each of these three situations involves an RCU-protected pointer to an array that is separately indexed. It might be tempting to consider use of RCU to instead protect the index into an array, however, this use -case is -not- supported. The problem with RCU-protected indexes into +case is **not** supported. The problem with RCU-protected indexes into arrays is that compilers can play way too many optimization games with integers, which means that the rules governing handling of these indexes are far more trouble than they are worth. If RCU-protected indexes into @@ -24,16 +26,20 @@ to be safely used. That aside, each of the three RCU-protected pointer situations are described in the following sections. +.. _hash_tables: Situation 1: Hash Tables +------------------------ Hash tables are often implemented as an array, where each array entry has a linked-list hash chain. Each hash chain can be protected by RCU as described in the listRCU.txt document. This approach also applies to other array-of-list situations, such as radix trees. +.. _static_arrays: Situation 2: Static Arrays +-------------------------- Static arrays, where the data (rather than a pointer to the data) is located in each array element, and where the array is never resized, @@ -41,13 +47,17 @@ have not been used with RCU. Rik van Riel recommends using seqlock in this situation, which would also have minimal read-side overhead as long as updates are rare. -Quick Quiz: Why is it so important that updates be rare when - using seqlock? +Quick Quiz: + Why is it so important that updates be rare when using seqlock? + +:ref:`Answer to Quick Quiz <answer_quick_quiz_seqlock>` +.. _resizable_arrays: -Situation 3: Resizeable Arrays +Situation 3: Resizable Arrays +------------------------------ -Use of RCU for resizeable arrays is demonstrated by the grow_ary() +Use of RCU for resizable arrays is demonstrated by the grow_ary() function formerly used by the System V IPC code. The array is used to map from semaphore, message-queue, and shared-memory IDs to the data structure that represents the corresponding IPC construct. The grow_ary() @@ -60,7 +70,7 @@ the remainder of the new, updates the ids->entries pointer to point to the new array, and invokes ipc_rcu_putref() to free up the old array. Note that rcu_assign_pointer() is used to update the ids->entries pointer, which includes any memory barriers required on whatever architecture -you are running on. +you are running on:: static int grow_ary(struct ipc_ids* ids, int newsize) { @@ -112,7 +122,7 @@ a simple check suffices. The pointer to the structure corresponding to the desired IPC object is placed in "out", with NULL indicating a non-existent entry. After acquiring "out->lock", the "out->deleted" flag indicates whether the IPC object is in the process of being -deleted, and, if not, the pointer is returned. +deleted, and, if not, the pointer is returned:: struct kern_ipc_perm* ipc_lock(struct ipc_ids* ids, int id) { @@ -144,8 +154,10 @@ deleted, and, if not, the pointer is returned. return out; } +.. _answer_quick_quiz_seqlock: Answer to Quick Quiz: + Why is it so important that updates be rare when using seqlock? The reason that it is important that updates be rare when using seqlock is that frequent updates can livelock readers. diff --git a/Documentation/RCU/index.rst b/Documentation/RCU/index.rst index 5c99185710fa..81a0a1e5f767 100644 --- a/Documentation/RCU/index.rst +++ b/Documentation/RCU/index.rst @@ -7,8 +7,13 @@ RCU concepts .. toctree:: :maxdepth: 3 + arrayRCU + rcubarrier + rcu_dereference + whatisRCU rcu listRCU + NMI-RCU UP Design/Memory-Ordering/Tree-RCU-Memory-Ordering diff --git a/Documentation/RCU/lockdep-splat.txt b/Documentation/RCU/lockdep-splat.txt index 9c015976b174..b8096316fd11 100644 --- a/Documentation/RCU/lockdep-splat.txt +++ b/Documentation/RCU/lockdep-splat.txt @@ -99,7 +99,7 @@ With this change, the rcu_dereference() is always within an RCU read-side critical section, which again would have suppressed the above lockdep-RCU splat. -But in this particular case, we don't actually deference the pointer +But in this particular case, we don't actually dereference the pointer returned from rcu_dereference(). Instead, that pointer is just compared to the cic pointer, which means that the rcu_dereference() can be replaced by rcu_access_pointer() as follows: diff --git a/Documentation/RCU/rcu_dereference.txt b/Documentation/RCU/rcu_dereference.rst index bf699e8cfc75..c9667eb0d444 100644 --- a/Documentation/RCU/rcu_dereference.txt +++ b/Documentation/RCU/rcu_dereference.rst @@ -1,4 +1,7 @@ +.. _rcu_dereference_doc: + PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference() +=============================================================== Most of the time, you can use values from rcu_dereference() or one of the similar primitives without worries. Dereferencing (prefix "*"), @@ -8,7 +11,7 @@ subtraction of constants, and casts all work quite naturally and safely. It is nevertheless possible to get into trouble with other operations. Follow these rules to keep your RCU code working properly: -o You must use one of the rcu_dereference() family of primitives +- You must use one of the rcu_dereference() family of primitives to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU will complain. Worse yet, your code can see random memory-corruption bugs due to games that compilers and DEC Alpha can play. @@ -25,24 +28,24 @@ o You must use one of the rcu_dereference() family of primitives for an example where the compiler can in fact deduce the exact value of the pointer, and thus cause misordering. -o You are only permitted to use rcu_dereference on pointer values. +- You are only permitted to use rcu_dereference on pointer values. The compiler simply knows too much about integral values to trust it to carry dependencies through integer operations. There are a very few exceptions, namely that you can temporarily cast the pointer to uintptr_t in order to: - o Set bits and clear bits down in the must-be-zero low-order + - Set bits and clear bits down in the must-be-zero low-order bits of that pointer. This clearly means that the pointer must have alignment constraints, for example, this does -not- work in general for char* pointers. - o XOR bits to translate pointers, as is done in some + - XOR bits to translate pointers, as is done in some classic buddy-allocator algorithms. It is important to cast the value back to pointer before doing much of anything else with it. -o Avoid cancellation when using the "+" and "-" infix arithmetic +- Avoid cancellation when using the "+" and "-" infix arithmetic operators. For example, for a given variable "x", avoid "(x-(uintptr_t)x)" for char* pointers. The compiler is within its rights to substitute zero for this sort of expression, so that @@ -54,16 +57,16 @@ o Avoid cancellation when using the "+" and "-" infix arithmetic "p+a-b" is safe because its value still necessarily depends on the rcu_dereference(), thus maintaining proper ordering. -o If you are using RCU to protect JITed functions, so that the +- If you are using RCU to protect JITed functions, so that the "()" function-invocation operator is applied to a value obtained (directly or indirectly) from rcu_dereference(), you may need to interact directly with the hardware to flush instruction caches. This issue arises on some systems when a newly JITed function is using the same memory that was used by an earlier JITed function. -o Do not use the results from relational operators ("==", "!=", +- Do not use the results from relational operators ("==", "!=", ">", ">=", "<", or "<=") when dereferencing. For example, - the following (quite strange) code is buggy: + the following (quite strange) code is buggy:: int *p; int *q; @@ -81,11 +84,11 @@ o Do not use the results from relational operators ("==", "!=", after such branches, but can speculate loads, which can again result in misordering bugs. -o Be very careful about comparing pointers obtained from +- Be very careful about comparing pointers obtained from rcu_dereference() against non-NULL values. As Linus Torvalds explained, if the two pointers are equal, the compiler could substitute the pointer you are comparing against for the pointer - obtained from rcu_dereference(). For example: + obtained from rcu_dereference(). For example:: p = rcu_dereference(gp); if (p == &default_struct) @@ -93,7 +96,7 @@ o Be very careful about comparing pointers obtained from Because the compiler now knows that the value of "p" is exactly the address of the variable "default_struct", it is free to - transform this code into the following: + transform this code into the following:: p = rcu_dereference(gp); if (p == &default_struct) @@ -105,14 +108,14 @@ o Be very careful about comparing pointers obtained from However, comparisons are OK in the following cases: - o The comparison was against the NULL pointer. If the + - The comparison was against the NULL pointer. If the compiler knows that the pointer is NULL, you had better not be dereferencing it anyway. If the comparison is non-equal, the compiler is none the wiser. Therefore, it is safe to compare pointers from rcu_dereference() against NULL pointers. - o The pointer is never dereferenced after being compared. + - The pointer is never dereferenced after being compared. Since there are no subsequent dereferences, the compiler cannot use anything it learned from the comparison to reorder the non-existent subsequent dereferences. @@ -124,31 +127,31 @@ o Be very careful about comparing pointers obtained from dereferenced, rcu_access_pointer() should be used in place of rcu_dereference(). - o The comparison is against a pointer that references memory + - The comparison is against a pointer that references memory that was initialized "a long time ago." The reason this is safe is that even if misordering occurs, the misordering will not affect the accesses that follow the comparison. So exactly how long ago is "a long time ago"? Here are some possibilities: - o Compile time. + - Compile time. - o Boot time. + - Boot time. - o Module-init time for module code. + - Module-init time for module code. - o Prior to kthread creation for kthread code. + - Prior to kthread creation for kthread code. - o During some prior acquisition of the lock that + - During some prior acquisition of the lock that we now hold. - o Before mod_timer() time for a timer handler. + - Before mod_timer() time for a timer handler. There are many other possibilities involving the Linux kernel's wide array of primitives that cause code to be invoked at a later time. - o The pointer being compared against also came from + - The pointer being compared against also came from rcu_dereference(). In this case, both pointers depend on one rcu_dereference() or another, so you get proper ordering either way. @@ -159,13 +162,13 @@ o Be very careful about comparing pointers obtained from of such an RCU usage bug is shown in the section titled "EXAMPLE OF AMPLIFIED RCU-USAGE BUG". - o All of the accesses following the comparison are stores, + - All of the accesses following the comparison are stores, so that a control dependency preserves the needed ordering. That said, it is easy to get control dependencies wrong. Please see the "CONTROL DEPENDENCIES" section of Documentation/memory-barriers.txt for more details. - o The pointers are not equal -and- the compiler does + - The pointers are not equal -and- the compiler does not have enough information to deduce the value of the pointer. Note that the volatile cast in rcu_dereference() will normally prevent the compiler from knowing too much. @@ -175,7 +178,7 @@ o Be very careful about comparing pointers obtained from comparison will provide exactly the information that the compiler needs to deduce the value of the pointer. -o Disable any value-speculation optimizations that your compiler +- Disable any value-speculation optimizations that your compiler might provide, especially if you are making use of feedback-based optimizations that take data collected from prior runs. Such value-speculation optimizations reorder operations by design. @@ -188,11 +191,12 @@ o Disable any value-speculation optimizations that your compiler EXAMPLE OF AMPLIFIED RCU-USAGE BUG +---------------------------------- Because updaters can run concurrently with RCU readers, RCU readers can see stale and/or inconsistent values. If RCU readers need fresh or consistent values, which they sometimes do, they need to take proper -precautions. To see this, consider the following code fragment: +precautions. To see this, consider the following code fragment:: struct foo { int a; @@ -244,7 +248,7 @@ to some reordering from the compiler and CPUs is beside the point. But suppose that the reader needs a consistent view? -Then one approach is to use locking, for example, as follows: +Then one approach is to use locking, for example, as follows:: struct foo { int a; @@ -299,6 +303,7 @@ As always, use the right tool for the job! EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH +----------------------------------------- If a pointer obtained from rcu_dereference() compares not-equal to some other pointer, the compiler normally has no clue what the value of the @@ -308,7 +313,7 @@ guarantees that RCU depends on. And the volatile cast in rcu_dereference() should prevent the compiler from guessing the value. But without rcu_dereference(), the compiler knows more than you might -expect. Consider the following code fragment: +expect. Consider the following code fragment:: struct foo { int a; @@ -354,6 +359,7 @@ dereference the resulting pointer. WHICH MEMBER OF THE rcu_dereference() FAMILY SHOULD YOU USE? +------------------------------------------------------------ First, please avoid using rcu_dereference_raw() and also please avoid using rcu_dereference_check() and rcu_dereference_protected() with a @@ -370,7 +376,7 @@ member of the rcu_dereference() to use in various situations: 2. If the access might be within an RCU read-side critical section on the one hand, or protected by (say) my_lock on the other, - use rcu_dereference_check(), for example: + use rcu_dereference_check(), for example:: p1 = rcu_dereference_check(p->rcu_protected_pointer, lockdep_is_held(&my_lock)); @@ -378,14 +384,14 @@ member of the rcu_dereference() to use in various situations: 3. If the access might be within an RCU read-side critical section on the one hand, or protected by either my_lock or your_lock on - the other, again use rcu_dereference_check(), for example: + the other, again use rcu_dereference_check(), for example:: p1 = rcu_dereference_check(p->rcu_protected_pointer, lockdep_is_held(&my_lock) || lockdep_is_held(&your_lock)); 4. If the access is on the update side, so that it is always protected - by my_lock, use rcu_dereference_protected(): + by my_lock, use rcu_dereference_protected():: p1 = rcu_dereference_protected(p->rcu_protected_pointer, lockdep_is_held(&my_lock)); @@ -410,18 +416,19 @@ member of the rcu_dereference() to use in various situations: SPARSE CHECKING OF RCU-PROTECTED POINTERS +----------------------------------------- The sparse static-analysis tool checks for direct access to RCU-protected pointers, which can result in "interesting" bugs due to compiler optimizations involving invented loads and perhaps also load tearing. -For example, suppose someone mistakenly does something like this: +For example, suppose someone mistakenly does something like this:: p = q->rcu_protected_pointer; do_something_with(p->a); do_something_else_with(p->b); If register pressure is high, the compiler might optimize "p" out -of existence, transforming the code to something like this: +of existence, transforming the code to something like this:: do_something_with(q->rcu_protected_pointer->a); do_something_else_with(q->rcu_protected_pointer->b); @@ -435,7 +442,7 @@ Load tearing could of course result in dereferencing a mashup of a pair of pointers, which also might fatally disappoint your code. These problems could have been avoided simply by making the code instead -read as follows: +read as follows:: p = rcu_dereference(q->rcu_protected_pointer); do_something_with(p->a); @@ -448,7 +455,7 @@ or as a formal parameter, with "__rcu", which tells sparse to complain if this pointer is accessed directly. It will also cause sparse to complain if a pointer not marked with "__rcu" is accessed using rcu_dereference() and friends. For example, ->rcu_protected_pointer might be declared as -follows: +follows:: struct foo __rcu *rcu_protected_pointer; diff --git a/Documentation/RCU/rcubarrier.txt b/Documentation/RCU/rcubarrier.rst index a2782df69732..f64f4413a47c 100644 --- a/Documentation/RCU/rcubarrier.txt +++ b/Documentation/RCU/rcubarrier.rst @@ -1,4 +1,7 @@ +.. _rcu_barrier: + RCU and Unloadable Modules +========================== [Originally published in LWN Jan. 14, 2007: http://lwn.net/Articles/217484/] @@ -21,7 +24,7 @@ given that readers might well leave absolutely no trace of their presence? There is a synchronize_rcu() primitive that blocks until all pre-existing readers have completed. An updater wishing to delete an element p from a linked list might do the following, while holding an -appropriate lock, of course: +appropriate lock, of course:: list_del_rcu(p); synchronize_rcu(); @@ -32,13 +35,13 @@ primitive must be used instead. This primitive takes a pointer to an rcu_head struct placed within the RCU-protected data structure and another pointer to a function that may be invoked later to free that structure. Code to delete an element p from the linked list from IRQ -context might then be as follows: +context might then be as follows:: list_del_rcu(p); call_rcu(&p->rcu, p_callback); Since call_rcu() never blocks, this code can safely be used from within -IRQ context. The function p_callback() might be defined as follows: +IRQ context. The function p_callback() might be defined as follows:: static void p_callback(struct rcu_head *rp) { @@ -49,6 +52,7 @@ IRQ context. The function p_callback() might be defined as follows: Unloading Modules That Use call_rcu() +------------------------------------- But what if p_callback is defined in an unloadable module? @@ -69,10 +73,11 @@ in realtime kernels in order to avoid excessive scheduling latencies. rcu_barrier() +------------- We instead need the rcu_barrier() primitive. Rather than waiting for a grace period to elapse, rcu_barrier() waits for all outstanding RCU -callbacks to complete. Please note that rcu_barrier() does -not- imply +callbacks to complete. Please note that rcu_barrier() does **not** imply synchronize_rcu(), in particular, if there are no RCU callbacks queued anywhere, rcu_barrier() is within its rights to return immediately, without waiting for a grace period to elapse. @@ -88,79 +93,79 @@ must match the flavor of rcu_barrier() with that of call_rcu(). If your module uses multiple flavors of call_rcu(), then it must also use multiple flavors of rcu_barrier() when unloading that module. For example, if it uses call_rcu(), call_srcu() on srcu_struct_1, and call_srcu() on -srcu_struct_2(), then the following three lines of code will be required -when unloading: +srcu_struct_2, then the following three lines of code will be required +when unloading:: 1 rcu_barrier(); 2 srcu_barrier(&srcu_struct_1); 3 srcu_barrier(&srcu_struct_2); The rcutorture module makes use of rcu_barrier() in its exit function -as follows: +as follows:: - 1 static void - 2 rcu_torture_cleanup(void) - 3 { - 4 int i; + 1 static void + 2 rcu_torture_cleanup(void) + 3 { + 4 int i; 5 - 6 fullstop = 1; - 7 if (shuffler_task != NULL) { + 6 fullstop = 1; + 7 if (shuffler_task != NULL) { 8 VERBOSE_PRINTK_STRING("Stopping rcu_torture_shuffle task"); 9 kthread_stop(shuffler_task); -10 } -11 shuffler_task = NULL; -12 -13 if (writer_task != NULL) { -14 VERBOSE_PRINTK_STRING("Stopping rcu_torture_writer task"); -15 kthread_stop(writer_task); -16 |