diff options
76 files changed, 3771 insertions, 1034 deletions
diff --git a/Documentation/PCI/boot-interrupts.rst b/Documentation/PCI/boot-interrupts.rst new file mode 100644 index 000000000000..d078ef3eb192 --- /dev/null +++ b/Documentation/PCI/boot-interrupts.rst @@ -0,0 +1,155 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Boot Interrupts +=============== + +:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com> + +Overview +======== + +On PCI Express, interrupts are represented with either MSI or inbound +interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a +given Core IO converts the legacy interrupt messages from PCI Express to +MSI interrupts. If the IO-APIC is disabled (via the mask bits in the +IO-APIC table entries), the messages are routed to the legacy PCH. This +in-band interrupt mechanism was traditionally necessary for systems that +did not support the IO-APIC and for boot. Intel in the past has used the +term "boot interrupts" to describe this mechanism. Further, the PCI Express +protocol describes this in-band legacy wire-interrupt INTx mechanism for +I/O devices to signal PCI-style level interrupts. The subsequent paragraphs +describe problems with the Core IO handling of INTx message routing to the +PCH and mitigation within BIOS and the OS. + + +Issue +===== + +When in-band legacy INTx messages are forwarded to the PCH, they in turn +trigger a new interrupt for which the OS likely lacks a handler. When an +interrupt goes unhandled over time, they are tracked by the Linux kernel as +Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it +reaches a specific count with the error "nobody cared". This disabled IRQ +now prevents valid usage by an existing interrupt which may happen to share +the IRQ line. + + irq 19: nobody cared (try booting with the "irqpoll" option) + CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1 + Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020 + Call Trace: + <IRQ> + ? dump_stack+0x46/0x5e + ? __report_bad_irq+0x2e/0xb0 + ? note_interrupt+0x242/0x290 + ? nNIKAL100_memoryRead16+0x8/0x10 [nikal] + ? handle_irq_event_percpu+0x55/0x70 + ? handle_irq_event+0x4f/0x80 + ? handle_fasteoi_irq+0x81/0x180 + ? handle_irq+0x1c/0x30 + ? do_IRQ+0x41/0xd0 + ? common_interrupt+0x84/0x84 + </IRQ> + + handlers: + irq_default_primary_handler threaded usb_hcd_irq + Disabling IRQ #19 + + +Conditions +========== + +The use of threaded interrupts is the most likely condition to trigger +this problem today. Threaded interrupts may not be reenabled after the IRQ +handler wakes. These "one shot" conditions mean that the threaded interrupt +needs to keep the interrupt line masked until the threaded handler has run. +Especially when dealing with high data rate interrupts, the thread needs to +run to completion; otherwise some handlers will end up in stack overflows +since the interrupt of the issuing device is still active. + +Affected Chipsets +================= + +The legacy interrupt forwarding mechanism exists today in a number of +devices including but not limited to chipsets from AMD/ATI, Broadcom, and +Intel. Changes made through the mitigations below have been applied to +drivers/pci/quirks.c + +Starting with ICX there are no longer any IO-APICs in the Core IO's +devices. IO-APIC is only in the PCH. Devices connected to the Core IO's +PCIe Root Ports will use native MSI/MSI-X mechanisms. + +Mitigations +=========== + +The mitigations take the form of PCI quirks. The preference has been to +first identify and make use of a means to disable the routing to the PCH. +In such a case a quirk to disable boot interrupt generation can be +added.[1] + + Intel® 6300ESB I/O Controller Hub + Alternate Base Address Register: + BIE: Boot Interrupt Enable + 0 = Boot interrupt is enabled. + 1 = Boot interrupt is disabled. + + Intel® Sandy Bridge through Sky Lake based Xeon servers: + Coherent Interface Protocol Interrupt Control + dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2: + When this bit is set. Local INTx messages received from the + Intel® Quick Data DMA/PCI Express ports are not routed to legacy + PCH - they are either converted into MSI via the integrated IO-APIC + (if the IO-APIC mask bit is clear in the appropriate entries) + or cause no further action (when mask bit is set) + +In the absence of a way to directly disable the routing, another approach +has been to make use of PCI Interrupt pin to INTx routing tables for +purposes of redirecting the interrupt handler to the rerouted interrupt +line by default. Therefore, on chipsets where this INTx routing cannot be +disabled, the Linux kernel will reroute the valid interrupt to its legacy +interrupt. This redirection of the handler will prevent the occurrence of +the spurious interrupt detection which would ordinarily disable the IRQ +line due to excessive unhandled counts.[2] + +The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or +disable) the redirection of the interrupt handler to the PCH interrupt +line. The option can be overridden by either pci=ioapicreroute or +pci=noioapicreroute.[3] + + +More Documentation +================== + +There is an overview of the legacy interrupt handling in several datasheets +(6300ESB and 6700PXH below). While largely the same, it provides insight +into the evolution of its handling with chipsets. + +Example of disabling of the boot interrupt +------------------------------------------ + +Intel® 6300ESB I/O Controller Hub (Document # 300641-004US) + 5.7.3 Boot Interrupt + https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf + +Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families +Datasheet - Volume 2: Registers (Document # 330784-003) + 6.6.41 cipintrc Coherent Interface Protocol Interrupt Control + https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf + +Example of handler rerouting +---------------------------- + +Intel® 6700PXH 64-bit PCI Hub (Document # 302628) + 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt + https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf + + +If you have any legacy PCI interrupt questions that aren't answered, email me. + +Cheers, + Sean V Kelley + sean.v.kelley@linux.intel.com + +[1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/ +[2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/ +[3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/ diff --git a/Documentation/PCI/index.rst b/Documentation/PCI/index.rst index 6768305e4c26..8f66feaafd4f 100644 --- a/Documentation/PCI/index.rst +++ b/Documentation/PCI/index.rst @@ -16,3 +16,4 @@ Linux PCI Bus Subsystem pci-error-recovery pcieaer-howto endpoint/index + boot-interrupts diff --git a/Documentation/PCI/pcieaer-howto.rst b/Documentation/PCI/pcieaer-howto.rst index 18bdefaafd1a..0b36b9ebfa4b 100644 --- a/Documentation/PCI/pcieaer-howto.rst +++ b/Documentation/PCI/pcieaer-howto.rst @@ -156,12 +156,6 @@ default reset_link function, but different upstream ports might have different specifications to reset pci express link, so all upstream ports should provide their own reset_link functions. -In struct pcie_port_service_driver, a new pointer, reset_link, is -added. -:: - - pci_ers_result_t (*reset_link) (struct pci_dev *dev); - Section 3.2.2.2 provides more detailed info on when to call reset_link. @@ -212,15 +206,10 @@ error_detected(dev, pci_channel_io_frozen) to all drivers within a hierarchy in question. Then, performing link reset at upstream is necessary. As different kinds of devices might use different approaches to reset link, AER port service driver is required to provide the -function to reset link. Firstly, kernel looks for if the upstream -component has an aer driver. If it has, kernel uses the reset_link -callback of the aer driver. If the upstream component has no aer driver -and the port is downstream port, we will perform a hot reset as the -default by setting the Secondary Bus Reset bit of the Bridge Control -register associated with the downstream port. As for upstream ports, -they should provide their own aer service drivers with reset_link -function. If error_detected returns PCI_ERS_RESULT_CAN_RECOVER and -reset_link returns PCI_ERS_RESULT_RECOVERED, the error handling goes +function to reset link via callback parameter of pcie_do_recovery() +function. If reset_link is not NULL, recovery function will use it +to reset the link. If error_detected returns PCI_ERS_RESULT_CAN_RECOVER +and reset_link returns PCI_ERS_RESULT_RECOVERED, the error handling goes to mmio_enabled. helper functions @@ -243,9 +232,9 @@ messages to root port when an error is detected. :: - int pci_cleanup_aer_uncorrect_error_status(struct pci_dev *dev);` + int pci_aer_clear_nonfatal_status(struct pci_dev *dev);` -pci_cleanup_aer_uncorrect_error_status cleanups the uncorrectable +pci_aer_clear_nonfatal_status clears non-fatal errors in the uncorrectable error status register. Frequent Asked Questions diff --git a/Documentation/devicetree/bindings/pci/amlogic,meson-pcie.txt b/Documentation/devicetree/bindings/pci/amlogic,meson-pcie.txt index 84fdc422792e..b6acbe694ffb 100644 --- a/Documentation/devicetree/bindings/pci/amlogic,meson-pcie.txt +++ b/Documentation/devicetree/bindings/pci/amlogic,meson-pcie.txt @@ -18,7 +18,6 @@ Required properties: - reg-names: Must be - "elbi" External local bus interface registers - "cfg" Meson specific registers - - "phy" Meson PCIE PHY registers for AXG SoC Family - "config" PCIe configuration space - reset-gpios: The GPIO to generate PCIe PERST# assert and deassert signal. - clocks: Must contain an entry for each entry in clock-names. @@ -26,13 +25,13 @@ Required properties: - "pclk" PCIe GEN 100M PLL clock - "port" PCIe_x(A or B) RC clock gate - "general" PCIe Phy clock - - "mipi" PCIe_x(A or B) 100M ref clock gate for AXG SoC Family - resets: phandle to the reset lines. -- reset-names: must contain "phy" "port" and "apb" - - "phy" Share PHY reset for AXG SoC Family +- reset-names: must contain "port" and "apb" - "port" Port A or B reset - "apb" Share APB reset -- phys: should contain a phandle to the shared phy for G12A SoC Family +- phys: should contain a phandle to the PCIE phy +- phy-names: must contain "pcie" + - device_type: should be "pci". As specified in designware-pcie.txt @@ -43,9 +42,8 @@ Example configuration: compatible = "amlogic,axg-pcie", "snps,dw-pcie"; reg = <0x0 0xf9800000 0x0 0x400000 0x0 0xff646000 0x0 0x2000 - 0x0 0xff644000 0x0 0x2000 0x0 0xf9f00000 0x0 0x100000>; - reg-names = "elbi", "cfg", "phy", "config"; + reg-names = "elbi", "cfg", "config"; reset-gpios = <&gpio GPIOX_19 GPIO_ACTIVE_HIGH>; interrupts = <GIC_SPI 177 IRQ_TYPE_EDGE_RISING>; #interrupt-cells = <1>; @@ -58,17 +56,15 @@ Example configuration: ranges = <0x82000000 0 0 0x0 0xf9c00000 0 0x00300000>; clocks = <&clkc CLKID_USB - &clkc CLKID_MIPI_ENABLE &clkc CLKID_PCIE_A &clkc CLKID_PCIE_CML_EN0>; clock-names = "general", - "mipi", "pclk", "port"; - resets = <&reset RESET_PCIE_PHY>, - <&reset RESET_PCIE_A>, + resets = <&reset RESET_PCIE_A>, <&reset RESET_PCIE_APB>; - reset-names = "phy", - "port", + reset-names = "port", "apb"; + phys = <&pcie_phy>; + phy-names = "pcie"; }; diff --git a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt b/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt deleted file mode 100644 index 4a0475e2ba7e..000000000000 --- a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt +++ /dev/null @@ -1,27 +0,0 @@ -* Cadence PCIe endpoint controller - -Required properties: -- compatible: Should contain "cdns,cdns-pcie-ep" to identify the IP used. -- reg: Should contain the controller register base address and AXI interface - region base address respectively. -- reg-names: Must be "reg" and "mem" respectively. -- cdns,max-outbound-regions: Set to maximum number of outbound regions - -Optional properties: -- max-functions: Maximum number of functions that can be configured (default 1). -- phys: From PHY bindings: List of Generic PHY phandles. One per lane if more - than one in the list. If only one PHY listed it must manage all lanes. -- phy-names: List of names to identify the PHY. - -Example: - -pcie@fc000000 { - compatible = "cdns,cdns-pcie-ep"; - reg = <0x0 0xfc000000 0x0 0x01000000>, - <0x0 0x80000000 0x0 0x40000000>; - reg-names = "reg", "mem"; - cdns,max-outbound-regions = <16>; - max-functions = /bits/ 8 <8>; - phys = <&ep_phy0 &ep_phy1>; - phy-names = "pcie-lane0","pcie-lane1"; -}; diff --git a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.yaml b/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.yaml new file mode 100644 index 000000000000..2996f8d4777c --- /dev/null +++ b/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.yaml @@ -0,0 +1,49 @@ +# SPDX-License-Identifier: GPL-2.0-only +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/pci/cdns,cdns-pcie-ep.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: Cadence PCIe EP Controller + +maintainers: + - Tom Joseph <tjoseph@cadence.com> + +allOf: + - $ref: "cdns-pcie.yaml#" + - $ref: "pci-ep.yaml#" + +properties: + compatible: + const: cdns,cdns-pcie-ep + + reg: + maxItems: 2 + + reg-names: + items: + - const: reg + - const: mem + +required: + - reg + - reg-names + +examples: + - | + bus { + #address-cells = <2>; + #size-cells = <2>; + + pcie-ep@fc000000 { + compatible = "cdns,cdns-pcie-ep"; + reg = <0x0 0xfc000000 0x0 0x01000000>, + <0x0 0x80000000 0x0 0x40000000>; + reg-names = "reg", "mem"; + cdns,max-outbound-regions = <16>; + max-functions = /bits/ 8 <8>; + phys = <&pcie_phy0>; + phy-names = "pcie-phy"; + }; + }; +... diff --git a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-host.txt b/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-host.txt deleted file mode 100644 index 91de69c713a9..000000000000 --- a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-host.txt +++ /dev/null @@ -1,66 +0,0 @@ -* Cadence PCIe host controller - -This PCIe controller inherits the base properties defined in -host-generic-pci.txt. - -Required properties: -- compatible: Should contain "cdns,cdns-pcie-host" to identify the IP used. -- reg: Should contain the controller register base address, PCIe configuration - window base address, and AXI interface region base address respectively. -- reg-names: Must be "reg", "cfg" and "mem" respectively. -- #address-cells: Set to <3> -- #size-cells: Set to <2> -- device_type: Set to "pci" -- ranges: Ranges for the PCI memory and I/O regions -- #interrupt-cells: Set to <1> |