From cfdfc14e7fb8ae77290e9d5afaeecc0a234a3846 Mon Sep 17 00:00:00 2001 From: Doug Meyer Date: Wed, 23 May 2018 13:18:05 -0700 Subject: switchtec: Use generic PCI Vendor ID and Class Code Move the Microsemi Switchtec PCI Vendor ID (same as PCI_VENDOR_ID_PMC_Sierra) to pci_ids.h. Also, replace Microsemi class constants with the standard PCI definitions. Signed-off-by: Doug Meyer [bhelgaas: restore SPDX (I assume it was removed by mistake), remove device ID definitions] Signed-off-by: Bjorn Helgaas Reviewed-by: Logan Gunthorpe --- drivers/pci/switch/switchtec.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) (limited to 'drivers/pci') diff --git a/drivers/pci/switch/switchtec.c b/drivers/pci/switch/switchtec.c index 47cd0c037433..9940cc70f38b 100644 --- a/drivers/pci/switch/switchtec.c +++ b/drivers/pci/switch/switchtec.c @@ -641,7 +641,7 @@ static int ioctl_event_summary(struct switchtec_dev *stdev, for (i = 0; i < SWITCHTEC_MAX_PFF_CSR; i++) { reg = ioread16(&stdev->mmio_pff_csr[i].vendor_id); - if (reg != MICROSEMI_VENDOR_ID) + if (reg != PCI_VENDOR_ID_MICROSEMI) break; reg = ioread32(&stdev->mmio_pff_csr[i].pff_event_summary); @@ -1203,7 +1203,7 @@ static void init_pff(struct switchtec_dev *stdev) for (i = 0; i < SWITCHTEC_MAX_PFF_CSR; i++) { reg = ioread16(&stdev->mmio_pff_csr[i].vendor_id); - if (reg != MICROSEMI_VENDOR_ID) + if (reg != PCI_VENDOR_ID_MICROSEMI) break; } @@ -1267,7 +1267,7 @@ static int switchtec_pci_probe(struct pci_dev *pdev, struct switchtec_dev *stdev; int rc; - if (pdev->class == MICROSEMI_NTB_CLASSCODE) + if (pdev->class == (PCI_CLASS_BRIDGE_OTHER << 8)) request_module_nowait("ntb_hw_switchtec"); stdev = stdev_create(pdev); @@ -1321,19 +1321,19 @@ static void switchtec_pci_remove(struct pci_dev *pdev) #define SWITCHTEC_PCI_DEVICE(device_id) \ { \ - .vendor = MICROSEMI_VENDOR_ID, \ + .vendor = PCI_VENDOR_ID_MICROSEMI, \ .device = device_id, \ .subvendor = PCI_ANY_ID, \ .subdevice = PCI_ANY_ID, \ - .class = MICROSEMI_MGMT_CLASSCODE, \ + .class = (PCI_CLASS_MEMORY_OTHER << 8), \ .class_mask = 0xFFFFFFFF, \ }, \ { \ - .vendor = MICROSEMI_VENDOR_ID, \ + .vendor = PCI_VENDOR_ID_MICROSEMI, \ .device = device_id, \ .subvendor = PCI_ANY_ID, \ .subdevice = PCI_ANY_ID, \ - .class = MICROSEMI_NTB_CLASSCODE, \ + .class = (PCI_CLASS_BRIDGE_OTHER << 8), \ .class_mask = 0xFFFFFFFF, \ } -- cgit v1.2.3 From ad281ecf1c7d4c905607e049b967bb6412587ca1 Mon Sep 17 00:00:00 2001 From: Doug Meyer Date: Wed, 23 May 2018 13:18:06 -0700 Subject: PCI: Add DMA alias quirk for Microsemi Switchtec NTB Add a quirk for the Microsemi Switchtec parts to allow DMA access via non-transparent bridging to work when the IOMMU is turned on. This exclusively addresses the ability of a remote NT endpoint to perform DMA accesses through the locally enumerated NT endpoint. Other aspects of the Switchtec NTB functionality, such as interrupts for doorbells and messages are independent of this quirk, and will work whether the IOMMU is on or off. When a requestor on one NT endpoint accesses memory on another NT endpoint, it does this via a devfn proxy ID. Proxy IDs are statically assigned to each NT endpoint by the NTB hardware as part of the release-from-reset sequence prior to PCI enumeration. These proxy IDs cannot be modified dynamically, and are not visible to the host during enumeration. When the Switchtec NTB driver loads it will map local requestor IDs, such as the root complex and transparent bridge DMA engines, to proxy IDs by populating those requestor IDs in hardware mapping table table entries. This establishes a fixed relationship between a requestor ID and a proxy ID. When a peer on a remote NT endpoint performs an access within a particular translation window in it's NT endpoint BAR address space, that access is translated to a DMA request on the local endpoint's bus. As part of the translation process, the original requestor ID has its devfn replaced with the proxy ID, and the bus portion of the BDF is replaced with the bus of the local NT endpoint. Thus, the DMA access from a remote NT endpoint will appear on the local bus to have come from the unknown devfn which the IOMMU will reject. Interrogate NTB hardware registers for each remote NT endpoint to obtain the proxy IDs that have been assigned to it and alias them to the local (enumerated) NT endpoint's device. The IOMMU then accepts the remote proxy IDs as if they were requests coming directly from the enumerated endpoint, giving remote requestors access to memory resources which the local host has made available. Note that the aliasing of the proxy IDs cannot be performed at the driver level given the current IOMMU architecture. Superficially this is because pci_add_dma_alias() symbol is not exported. Functionally, the current IOMMU design requires the aliasing to be performed prior to the creation of IOMMU groups. If a driver were to attempt to use pci_add_dma_alias() in its probe routine it would fail since the IOMMU groups have been set up by that time. If the Switchtec hardware supported dynamic proxy ID (re-)assignment this would be an issue, but it does not. To further clarify static proxy ID assignment: While the requester ID to proxy ID mapping can be dynamically changed, the number and value of proxy IDs given to an NT EP cannot, even for dynamic reconfiguration such as hot-add. Therefore, the chip configuration must account a priori for the proxy IDs needs, considering both static and dynamic system configurations. For example, a port on the chip may not having anything plugged into it at start of day; but it must have a sufficient number of proxy IDs assigned to accommodate the supported devices which may be hot-added. Switchtec NTB functionality with the IOMMU off is unchanged by this quirk. Signed-off-by: Doug Meyer [bhelgaas: use hard-coded Device IDs instead of adding #defines for each] Signed-off-by: Bjorn Helgaas Reviewed-by: Logan Gunthorpe --- drivers/pci/quirks.c | 140 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) (limited to 'drivers/pci') diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index f439de848658..d54a182a09cf 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -27,6 +27,7 @@ #include #include #include +#include #include /* isa_dma_bridge_buggy */ #include "pci.h" @@ -4753,3 +4754,142 @@ DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID, PCI_CLASS_MULTIMEDIA_HD_AUDIO, 8, quirk_gpu_hda); DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, PCI_CLASS_MULTIMEDIA_HD_AUDIO, 8, quirk_gpu_hda); + +/* + * Microsemi Switchtec NTB uses devfn proxy IDs to move TLPs between + * NT endpoints via the internal switch fabric. These IDs replace the + * originating requestor ID TLPs which access host memory on peer NTB + * ports. Therefore, all proxy IDs must be aliased to the NTB device + * to permit access when the IOMMU is turned on. + */ +static void quirk_switchtec_ntb_dma_alias(struct pci_dev *pdev) +{ + void __iomem *mmio; + struct ntb_info_regs __iomem *mmio_ntb; + struct ntb_ctrl_regs __iomem *mmio_ctrl; + struct sys_info_regs __iomem *mmio_sys_info; + u64 partition_map; + u8 partition; + int pp; + + if (pci_enable_device(pdev)) { + pci_err(pdev, "Cannot enable Switchtec device\n"); + return; + } + + mmio = pci_iomap(pdev, 0, 0); + if (mmio == NULL) { + pci_disable_device(pdev); + pci_err(pdev, "Cannot iomap Switchtec device\n"); + return; + } + + pci_info(pdev, "Setting Switchtec proxy ID aliases\n"); + + mmio_ntb = mmio + SWITCHTEC_GAS_NTB_OFFSET; + mmio_ctrl = (void __iomem *) mmio_ntb + SWITCHTEC_NTB_REG_CTRL_OFFSET; + mmio_sys_info = mmio + SWITCHTEC_GAS_SYS_INFO_OFFSET; + + partition = ioread8(&mmio_ntb->partition_id); + + partition_map = ioread32(&mmio_ntb->ep_map); + partition_map |= ((u64) ioread32(&mmio_ntb->ep_map + 4)) << 32; + partition_map &= ~(1ULL << partition); + + for (pp = 0; pp < (sizeof(partition_map) * 8); pp++) { + struct ntb_ctrl_regs __iomem *mmio_peer_ctrl; + u32 table_sz = 0; + int te; + + if (!(partition_map & (1ULL << pp))) + continue; + + pci_dbg(pdev, "Processing partition %d\n", pp); + + mmio_peer_ctrl = &mmio_ctrl[pp]; + + table_sz = ioread16(&mmio_peer_ctrl->req_id_table_size); + if (!table_sz) { + pci_warn(pdev, "Partition %d table_sz 0\n", pp); + continue; + } + + if (table_sz > 512) { + pci_warn(pdev, + "Invalid Switchtec partition %d table_sz %d\n", + pp, table_sz); + continue; + } + + for (te = 0; te < table_sz; te++) { + u32 rid_entry; + u8 devfn; + + rid_entry = ioread32(&mmio_peer_ctrl->req_id_table[te]); + devfn = (rid_entry >> 1) & 0xFF; + pci_dbg(pdev, + "Aliasing Partition %d Proxy ID %02x.%d\n", + pp, PCI_SLOT(devfn), PCI_FUNC(devfn)); + pci_add_dma_alias(pdev, devfn); + } + } + + pci_iounmap(pdev, mmio); + pci_disable_device(pdev); +} +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8531, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8532, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8533, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8534, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8535, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8536, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8543, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8544, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8545, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8546, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8551, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8552, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8553, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8554, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8555, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8556, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8561, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8562, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8563, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8564, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8565, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8566, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8571, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8572, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8573, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8574, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8575, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_MICROSEMI, 0x8576, + quirk_switchtec_ntb_dma_alias); -- cgit v1.2.3 From f778a0d26fe3912424401e7b997155094de36487 Mon Sep 17 00:00:00 2001 From: Logan Gunthorpe Date: Wed, 30 May 2018 14:13:11 -0600 Subject: PCI: Expand documentation for pci_add_dma_alias() Seeing there's been some confusion about the use of pci_add_dma_alias(), expand the comment to describe why it must be called early and how early it must be called. Also, expand on the purpose of this function and common reasons it would be used. [The comment was reworded to some extent by Alex Williamson] Signed-off-by: Logan Gunthorpe Signed-off-by: Bjorn Helgaas Cc: Alex Williamson Cc: Doug Meyer --- drivers/pci/pci.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) (limited to 'drivers/pci') diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 97acba712e4e..ea4a9d3be9f9 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -5387,8 +5387,19 @@ int pci_set_vga_state(struct pci_dev *dev, bool decode, * @dev: the PCI device for which alias is added * @devfn: alias slot and function * - * This helper encodes 8-bit devfn as bit number in dma_alias_mask. - * It should be called early, preferably as PCI fixup header quirk. + * This helper encodes an 8-bit devfn as a bit number in dma_alias_mask + * which is used to program permissible bus-devfn source addresses for DMA + * requests in an IOMMU. These aliases factor into IOMMU group creation + * and are useful for devices generating DMA requests beyond or different + * from their logical bus-devfn. Examples include device quirks where the + * device simply uses the wrong devfn, as well as non-transparent bridges + * where the alias may be a proxy for devices in another domain. + * + * IOMMU group creation is performed during device discovery or addition, + * prior to any potential DMA mapping and therefore prior to driver probing + * (especially for userspace assigned devices where IOMMU group definition + * cannot be left as a userspace activity). DMA aliases should therefore + * be configured via quirks, such as the PCI fixup header quirk. */ void pci_add_dma_alias(struct pci_dev *dev, u8 devfn) { -- cgit v1.2.3