path: root/Documentation/RCU/whatisRCU.rst

.. _whatisrcu_doc:

What is RCU?  --  "Read, Copy, Update"
======================================

Please note that the "What is RCU?" LWN series is an excellent place
to start learning about RCU:

| 1.	What is RCU, Fundamentally?  http://lwn.net/Articles/262464/
| 2.	What is RCU? Part 2: Usage   http://lwn.net/Articles/263130/
| 3.	RCU part 3: the RCU API      http://lwn.net/Articles/264090/
| 4.	The RCU API, 2010 Edition    http://lwn.net/Articles/418853/
| 	2010 Big API Table           http://lwn.net/Articles/419086/
| 5.	The RCU API, 2014 Edition    http://lwn.net/Articles/609904/
|	2014 Big API Table           http://lwn.net/Articles/609973/


What is RCU?

RCU is a synchronization mechanism that was added to the Linux kernel
during the 2.5 development effort that is optimized for read-mostly
situations.  Although RCU is actually quite simple once you understand it,
getting there can sometimes be a challenge.  Part of the problem is that
most of the past descriptions of RCU have been written with the mistaken
assumption that there is "one true way" to describe RCU.  Instead,
the experience has been that different people must take different paths
to arrive at an understanding of RCU.  This document provides several
different paths, as follows:

:ref:`1.	RCU OVERVIEW <1_whatisRCU>`

:ref:`2.	WHAT IS RCU'S CORE API? <2_whatisRCU>`

:ref:`3.	WHAT ARE SOME EXAMPLE USES OF CORE RCU API? <3_whatisRCU>`

:ref:`4.	WHAT IF MY UPDATING THREAD CANNOT BLOCK? <4_whatisRCU>`

:ref:`5.	WHAT ARE SOME SIMPLE IMPLEMENTATIONS OF RCU? <5_whatisRCU>`

:ref:`6.	ANALOGY WITH READER-WRITER LOCKING <6_whatisRCU>`

:ref:`7.	FULL LIST OF RCU APIs <7_whatisRCU>`

:ref:`8.	ANSWERS TO QUICK QUIZZES <8_whatisRCU>`

People who prefer starting with a conceptual overview should focus on
Section 1, though most readers will profit by reading this section at
some point.  People who prefer to start with an API that they can then
experiment with should focus on Section 2.  People who prefer to start
with example uses should focus on Sections 3 and 4.  People who need to
understand the RCU implementation should focus on Section 5, then dive
into the kernel source code.  People who reason best by analogy should
focus on Section 6.  Section 7 serves as an index to the docbook API
documentation, and Section 8 is the traditional answer key.

So, start with the section that makes the most sense to you and your
preferred method of learning.  If you need to know everything about
everything, feel free to read the whole thing -- but if you are really
that type of person, you have perused the source code and will therefore
never need this document anyway.  ;-)

.. _1_whatisRCU:

1.  RCU OVERVIEW
----------------

The basic idea behind RCU is to split updates into "removal" and
"reclamation" phases.  The removal phase removes references to data items
within a data structure (possibly by replacing them with references to
new versions of these data items), and can run concurrently with readers.
The reason that it is safe to run the removal phase concurrently with
readers is the semantics of modern CPUs guarantee that readers will see
either the old or the new version of the data structure rather than a
partially updated reference.  The reclamation phase does the work of reclaiming
(e.g., freeing) the data items removed from the data structure during the
removal phase.  Because reclaiming data items can disrupt any readers
concurrently referencing those data items, the reclamation phase must
not start until readers no longer hold references to those data items.

Splitting the update into removal and reclamation phases permits the
updater to perform the removal phase immediately, and to defer the
reclamation phase until all readers active during the removal phase have
completed, either by blocking until they finish or by registering a
callback that is invoked after they finish.  Only readers that are active
during the removal phase need be considered, because any reader starting
after the removal phase will be unable to gain a reference to the removed
data items, and therefore cannot be disrupted by the reclamation phase.

So the typical RCU update sequence goes something like the following:

a.	Remove pointers to a data structure, so that subsequent
	readers cannot gain a reference to it.

b.	Wait for all previous readers to complete their RCU read-side
	critical sections.

c.	At this point, there cannot be any readers who hold references
	to the data structure, so it now may safely be reclaimed
	(e.g., kfree()d).

Step (b) above is the key idea underlying RCU's deferred destruction.
The ability to wait until all readers are done allows RCU readers to
use much lighter-weight synchronization, in some cases, absolutely no
synchronization at all.  In contrast, in more conventional lock-based
schemes, readers must use heavy-weight synchronization in order to
prevent an updater from deleting the data structure out from under them.
This is because lock-based updaters typically update data items in place,
and must therefore exclude readers.  In contrast, RCU-based updaters
typically take advantage of the fact that writes to single aligned
pointers are atomic on modern CPUs, allowing atomic insertion, removal,
and replacement of data items in a linked structure without disrupting
readers.  Concurrent RCU readers can then continue accessing the old
versions, and can dispense with the atomic operations, memory barriers,
and communications cache misses that are so expensive on present-day
SMP computer systems, even in absence of lock contention.

In the three-step procedure shown above, the updater is performing both
the removal and the reclamation step, but it is often helpful for an
entirely different thread to do the reclamation, as is in fact the case
in the Linux kernel's directory-entry cache (dcache).  Even if the same
thread performs both the update step (step (a) above) and the reclamation
step (step (c) above), it is often helpful to think of them separately.
For example, RCU readers and updaters need not communicate at all,
but RCU provides implicit low-overhead communication between readers
and reclaimers, namely, in step (b) above.

So how the heck can a reclaimer tell when a reader is done, given
that readers are not doing any sort of synchronization operations???
Read on to learn about how RCU's API makes this easy.

.. _2_whatisRCU:

2.  WHAT IS RCU'S CORE API?
---------------------------

The core RCU API is quite small:

a.	rcu_read_lock()
b.	rcu_read_unlock()
c.	synchronize_rcu() / call_rcu()
d.	rcu_assign_pointer()
e.	rcu_dereference()

There are many other members of the RCU API, but the rest can be
expressed in terms of these five, though most implementations instead
express synchronize_rcu() in terms of the call_rcu() callback API.

The five core RCU APIs are described below, the other 18 will be enumerated
later.  See the kernel docbook documentation for more info, or look directly
at the function header comments.

rcu_read_lock()
^^^^^^^^^^^^^^^
	void rcu_read_lock(void);

	Used by a reader to inform the reclaimer that the reader is
	entering an RCU read-side critical section.  It is illegal
	to block while in an RCU read-side critical section, though
	kernels built with CONFIG_PREEMPT_RCU can preempt RCU
	read-side critical sections.  Any RCU-protected data structure
	accessed during an RCU read-side critical section is guaranteed to
	remain unreclaimed for the full duration of that critical section.
	Reference counts may be used in conjunction with RCU to maintain
	longer-term references to data structures.

rcu_read_unlock()
^^^^^^^^^^^^^^^^^
	void rcu_read_unlock(void);

	Used by a reader to inform the reclaimer that the reader is
	exiting an RCU read-side critical section.  Note that RCU
	read-side critical sections may be nested and/or overlapping.

synchronize_rcu()
^^^^^^^^^^^^^^^^^
	void synchronize_rcu(void);

	Marks the end of updater code and the beginning of reclaimer
	code.  It does this by blocking until all pre-existing RCU
	read-side critical sections on all CPUs have completed.
	Note that synchronize_rcu() will **not** necessarily wait for
	any subsequent RCU read-side critical sections to complete.
	For example, consider the following sequence of events::

	         CPU 0                  CPU 1                 CPU 2
	     ----------------- ------------------------- ---------------
	 1.  rcu_read_lock()
	 2.                    enters synchronize_rcu()
	 3.                                               rcu_read_lock()
	 4.  rcu_read_unlock()
	 5.                     exits synchronize_rcu()
	 6.                                              rcu_read_unlock()

	To reiterate, synchronize_rcu() waits only for ongoing RCU
	read-side critical sections to complete, not necessarily for
	any that begin after synchronize_rcu() is invoked.

	Of course, synchronize_rcu() does not necessarily return
	**immediately** after the last pre-existing RCU read-side critical
	section completes.  For one thing, there might well be scheduling
	delays.  For another thing, many RCU implementations process
	requests in batches in order to improve efficiencies, which can
	further delay synchronize_rcu().

	Since synchronize_rcu() is the API that must figure out when
	readers are done, its implementation is key to RCU.  For RCU
	to be useful in all but the most read-intensive situations,
	synchronize_rcu()'s overhead must also be quite small.

	The call_rcu() API is a callback form of synchronize_rcu(),
	and is described in more detail in a later section.  Instead of
	blocking, it registers a function and argument which are invoked
	after all ongoing RCU read-side critical sections have completed.
	This callback variant is particularly useful in situations where
	it is illegal to block or where update-side performance is
	critically important.

	However, the call_rcu() API should not be used lightly, as use
	of the synchronize_rcu() API generally results in simpler code.
	In addition, the synchronize_rcu() API has the nice property
	of automatically limiting update rate should grace periods
	be delayed.  This property results in system resilience in face
	of denial-of-service attacks.  Code using call_rcu() should limit
	update rate in order to gain this same sort of resilience.  See
	checklist.txt for some approaches to limiting the update rate.

rcu_assign_pointer()
^^^^^^^^^^^^^^^^^^^^
	void rcu_assign_pointer(p, typeof(p) v);

	Yes, rcu_assign_pointer() **is** implemented as a macro, though it
	would be cool to be able to declare a function in this manner.
	(Compiler experts will no doubt disagree.)

	The updater uses this function to assign a new value to an
	RCU-protected pointer, in order to safely communicate the change
	in value from the updater to the reader.  This macro does not
	evaluate to an rvalue, but it does execute any memory-barrier
	instructions required for a given CPU architecture.

	Perhaps just as important, it serves to document (1) which
	pointers are protected by RCU and (2) the point at which a
	given structure becomes accessible to other CPUs.  That said,
	rcu_assign_pointer() is most frequently used indirectly, via
	the _rcu list-manipulation primitives such as list_add_rcu().

rcu_dereference()
^^^^^^^^^^^^^^^^^
	typeof(p) rcu_dereference(p);

	Like rcu_assign_pointer(), rcu_dereference() must be implemented
	as a macro.

	The reader uses rcu_dereference() to fetch an RCU-protected
	pointer, which returns a value that may then be safely
	dereferenced.  Note that rcu_dereference() does not actually
	dereference the pointer, instead, it protects the pointer for
	later dereferencing.  It also executes any needed memory-barrier
	instructions for a given CPU architecture.  Currently, only Alpha
	needs memory barriers within rcu_dereference() -- on other CPUs,
	it compiles to nothing, not even a compiler directive.

	Common coding practice uses rcu_dereference() to copy an
	RCU-protected pointer to a local variable, then dereferences
	this local variable, for example as follows::

		p = rcu_dereference(head.next);
		return p->data;

	However, in this case, one could just as easily combine these
	into one statement::

		return rcu_dereference(head.next)->data;

	If you are going to be fetching multiple fields from the
	RCU-protected structure, using the local variable is of
	course preferred.  Repeated rcu_dereference() calls look
	ugly, do not guarantee that the same pointer will be returned
	if an update happened while in the critical section, and incur
	unnecessary overhead on Alpha CPUs.

	Note that the value returned by rcu_dereference() is valid
	only within the enclosing RCU read-side critical section [1]_.
	For example, the following is **not** legal::

		rcu_read_lock();
		p = rcu_dereference(head.next);
		rcu_read_unlock();
		x = p->address;	/* BUG!!! */
		rcu_read_lock();
		y = p->data;	/* BUG!!! */
		rcu_read_unlock();

	Holding a reference from one RCU read-side critical section
	to another is just as illegal as holding a reference from
	one lock-based critical section to another!  Similarly,
	using a reference outside of the critical section in which
	it was acquired is just as illegal as doing so with normal
	locking.

	As with rcu_assign_pointer(), an important function of
	rcu_dereference() is to document which pointers are protected by
	RCU, in particular, flagging a pointer that is subject to changing
	at any time, including immediately after the rcu_dereference().
	And, again like rcu_assign_pointer(), rcu_dereference() is
	typically used indirectly, via the _rcu list-manipulation
	primitives, such as list_for_each_entry_rcu() [2]_.

.. 	[1] The variant rcu_dereference_protected() can be used outside
	of an RCU read-side critical section as long as the usage is
	protected by locks acquired by the update-side code.  This variant
	avoids the lockdep warning that would happen when using (for
	example) rcu_dereference() without rcu_read_lock() protection.
	Using rcu_dereference_protected() also has the advantage
	of permitting compiler optimizations that rcu_dereference()
	must prohibit.	The rcu_dereference_protected() variant takes
	a lockdep expression to indicate which locks must be acquired
	by the caller. If the indicated protection is not provided,
	a lockdep splat is emitted.  See Documentation/RCU/Design/Requirements/Requirements.rst
	and the API's code comments for more details and example usage.

.. 	[2] If the list_for_each_entry_rcu() instance might be used by
	update-side code as well as by RCU readers, then an additional
	lockdep expression can be added to its list of arguments.
	For example, given an additional "lock_is_held(&mylock)" argument,
	the RCU lockdep code would complain only if this instance was
	invoked outside of an RCU read-side critical section and without
	the protection of mylock.

The following diagram shows how each API communicates among the
reader, updater, and reclaimer.
::


	    rcu_assign_pointer()
	                            +--------+
	    +---------------------->| reader |---------+
	    |                       +--------+         |
	    |                           |              |
	    |                           |              | Protect:
	    |                           |