path: root/Documentation/x86/resctrl.rst

.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

===========================================
User Interface for Resource Control feature
===========================================

:Copyright: |copy| 2016 Intel Corporation
:Authors: - Fenghua Yu <fenghua.yu@intel.com>
          - Tony Luck <tony.luck@intel.com>
          - Vikas Shivappa <vikas.shivappa@intel.com>


Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).

This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
flag bits:

=============================================	================================
RDT (Resource Director Technology) Allocation	"rdt_a"
CAT (Cache Allocation Technology)		"cat_l3", "cat_l2"
CDP (Code and Data Prioritization)		"cdp_l3", "cdp_l2"
CQM (Cache QoS Monitoring)			"cqm_llc", "cqm_occup_llc"
MBM (Memory Bandwidth Monitoring)		"cqm_mbm_total", "cqm_mbm_local"
MBA (Memory Bandwidth Allocation)		"mba"
=============================================	================================

To use the feature mount the file system::

 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps]] /sys/fs/resctrl

mount options are:

"cdp":
	Enable code/data prioritization in L3 cache allocations.
"cdpl2":
	Enable code/data prioritization in L2 cache allocations.
"mba_MBps":
	Enable the MBA Software Controller(mba_sc) to specify MBA
	bandwidth in MBps

L2 and L3 CDP are controlled separately.

RDT features are orthogonal. A particular system may support only
monitoring, only control, or both monitoring and control.  Cache
pseudo-locking is a unique way of using cache control to "pin" or
"lock" data in the cache. Details can be found in
"Cache Pseudo-Locking".


The mount succeeds if either of allocation or monitoring is present, but
only those files and directories supported by the system will be created.
For more details on the behavior of the interface during monitoring
and allocation, see the "Resource alloc and monitor groups" section.

Info directory
==============

The 'info' directory contains information about the enabled
resources. Each resource has its own subdirectory. The subdirectory
names reflect the resource names.

Each subdirectory contains the following files with respect to
allocation:

Cache resource(L3/L2)  subdirectory contains the following files
related to allocation:

"num_closids":
		The number of CLOSIDs which are valid for this
		resource. The kernel uses the smallest number of
		CLOSIDs of all enabled resources as limit.
"cbm_mask":
		The bitmask which is valid for this resource.
		This mask is equivalent to 100%.
"min_cbm_bits":
		The minimum number of consecutive bits which
		must be set when writing a mask.

"shareable_bits":
		Bitmask of shareable resource with other executing
		entities (e.g. I/O). User can use this when
		setting up exclusive cache partitions. Note that
		some platforms support devices that have their
		own settings for cache use which can over-ride
		these bits.
"bit_usage":
		Annotated capacity bitmasks showing how all
		instances of the resource are used. The legend is:

			"0":
			      Corresponding region is unused. When the system's
			      resources have been allocated and a "0" is found
			      in "bit_usage" it is a sign that resources are
			      wasted.

			"H":
			      Corresponding region is used by hardware only
			      but available for software use. If a resource
			      has bits set in "shareable_bits" but not all
			      of these bits appear in the resource groups'
			      schematas then the bits appearing in
			      "shareable_bits" but no resource group will
			      be marked as "H".
			"X":
			      Corresponding region is available for sharing and
			      used by hardware and software. These are the
			      bits that appear in "shareable_bits" as
			      well as a resource group's allocation.
			"S":
			      Corresponding region is used by software
			      and available for sharing.
			"E":
			      Corresponding region is used exclusively by
			      one resource group. No sharing allowed.
			"P":
			      Corresponding region is pseudo-locked. No
			      sharing allowed.

Memory bandwidth(MB) subdirectory contains the following files
with respect to allocation:

"min_bandwidth":
		The minimum memory bandwidth percentage which
		user can request.

"bandwidth_gran":
		The granularity in which the memory bandwidth
		percentage is allocated. The allocated
		b/w percentage is rounded off to the next
		control step available on the hardware. The
		available bandwidth control steps are:
		min_bandwidth + N * bandwidth_gran.

"delay_linear":
		Indicates if the delay scale is linear or
		non-linear. This field is purely informational
		only.

"thread_throttle_mode":
		Indicator on Intel systems of how tasks running on threads
		of a physical core are throttled in cases where they
		request different memory bandwidth percentages:

		"max":
			the smallest percentage is applied
			to all threads
		"per-thread":
			bandwidth percentages are directly applied to
			the threads running on the core

If RDT monitoring is available there will be an "L3_MON" directory
with the following files:

"num_rmids":
		The number of RMIDs available. This is the
		upper bound for how many "CTRL_MON" + "MON"
		groups can be created.

"mon_features":
		Lists the monitoring events if
		monitoring is enabled for the resource.

"max_threshold_occupancy":
		Read/write file provides the largest value (in
		bytes) at which a previously used LLC_occupancy
		counter can be considered for re-use.

Finally, in the top level of the "info" directory there is a file
named "last_cmd_status". This is reset with every "command" issued
via the file system (making new directories or writing to any of the
control files). If the command was successful, it will read as "ok".
If the command failed, it will provide more information that can be
conveyed in the error returns from file operations. E.g.
::

	# echo L3:0=f7 > schemata
	bash: echo: write error: Invalid argument
	# cat info/last_cmd_status
	mask f7 has non-consecutive 1-bits

Resource alloc and monitor groups
=================================

Resource groups are represented as directories in the resctrl file
system.  The default group is the root directory which, immediately
after mounting, owns all the tasks and cpus in the system and can make
full use of all resources.

On a system with RDT control features additional directories can be
created in the root directory that specify different amounts of each
resource (see "schemata" below). The root and these additional top level
directories are referred to as "CTRL_MON" groups below.

On a system with RDT monitoring the root directory and other top level
directories contain a directory named "mon_groups" in which additional
directories can be created to monitor subsets of tasks in the CTRL_MON
group that is their ancestor. These are called "MON" groups in the rest
of this document.

Removing a directory will move all tasks and cpus owned by the group it
represents to the parent. Removing one of the created CTRL_MON groups
will automatically remove all MON groups below it.

All groups contain the following files:

"tasks":
	Reading this file shows the list of all tasks that belong to
	this group. Writing a task id to the file will add a task to the
	group. If the group is a CTRL_MON group the task is removed from
	whichever previous CTRL_MON group owned the task and also from
	any MON group that owned the task. If the group is a MON group,
	then the task must already belong to the CTRL_MON parent of this
	group. The task is removed from any previous MON group.


"cpus":
	Reading this file shows a bitmask of the logical CPUs owned by
	this group. Writing a mask to this file will add and remove
	CPUs to/from this group. As with the tasks file a hierarchy is
	maintained where MON groups may only include CPUs owned by the
	parent CTRL_MON group.
	When the resource group is in pseudo-locked mode this file will
	only be readable, reflecting the CPUs associated with the
	pseudo-locked region.


"cpus_list":
	Just like "cpus", only using ranges of CPUs instead of bitmasks.


When control is enabled all CTRL_MON groups will also contain:

"schemata":
	A list of all the resources available to this group.
	Each resource has its own line and format - see below for details.

"size":
	Mirrors the display of the "schemata" file to display the size in
	bytes of each allocation instead of the bits representing the
	allocation.

"mode":
	The "mode" of the resource group dictates the sharing of its
	allocations. A "shareable" resource group allows sharing of its
	allocations while an "exclusive" resource group does not. A
	cache pseudo-locked region is created by first writing
	"pseudo-locksetup" to the "mode" file before writing the cache
	pseudo-locked region's schemata to the resource group's "schemata"
	file. On successful pseudo-locked region creation the mode will
	automatically change to "pseudo-locked".

When monitoring is enabled all MON groups will also contain:

"mon_data":
	This contains a set of files organized by L3 domain and by
	RDT event. E.g. on a system with two L3 domains there will
	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
	directories have one file per event (e.g. "llc_occupancy",
	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
	files provide a read out of the current value of the event for
	all tasks in the group. In CTRL_MON groups these files provide
	the sum for all tasks in the CTRL_MON group and all tasks in
	MON groups. Please see example section for more details on usage.

Resource allocation rules
-------------------------

When a task is running the following rules define which resources are
available to it:

1) If the task is a member of a non-default group, then the schemata
   for that group is used.

2) Else if the task belongs to the default group, but is running on a
   CPU that is assigned to some specific group, then the schemata for the
   CPU's group is used.

3) Otherwise the schemata for the default group is used.

Resource monitoring rules
<