From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACF84C2BB85 for ; Tue, 14 Apr 2020 17:05:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 53D9C206A2 for ; Tue, 14 Apr 2020 17:05:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="tSYqciql" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 53D9C206A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7591A8E0029; Tue, 14 Apr 2020 13:04:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E32F8E0001; Tue, 14 Apr 2020 13:04:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 584F38E0029; Tue, 14 Apr 2020 13:04:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 37FD68E0001 for ; Tue, 14 Apr 2020 13:04:44 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 053218245571 for ; Tue, 14 Apr 2020 17:04:44 +0000 (UTC) X-FDA: 76707084888.21.soda50_8098d30498135 X-HE-Tag: soda50_8098d30498135 X-Filterd-Recvd-Size: 13780 Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Tue, 14 Apr 2020 17:04:43 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id h9so15238162wrc.8 for ; Tue, 14 Apr 2020 10:04:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o3CxBoBUQYvmGdKZIOcmkRHgIzTXyZyZ4t2wW/1SA7g=; b=tSYqciqlXtpjJ7vaQSTxjL5MZonaq8ohjbCSix8+DjN0AmayPZAIpS7LY/zTiW9lAV POiAPMbwtCGrqliYceZC5WVj5SXlQqBR90xihQd2WhNlVOzwxkZumgzVIHC8uJ1+X9Ld NY+/1Ctz+wLSZedF+dev8JUV1ocwF1jxxUy+lZv/DbEkXH3kKqGVr6BD+Eki/pfZ/emV B9dC1Qd8PVYzfTTA5YYtlwx2mJZSYHYzkmcH1Br3A9STc52frl2YwBhfk2usMwVWSQHq CdylWdrs5XXvI8yseewVv4rOEzHYZhFARjPFhBzziAmN5msb5YOwMdQkgDktUijog+71 /KaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o3CxBoBUQYvmGdKZIOcmkRHgIzTXyZyZ4t2wW/1SA7g=; b=a7+fCGB7NTqkCrIo0kl71GLbjT1SOvftwNE0Ci2dQQHTW5rsboJ3aAKKskwGhb+/qU q3zPf5vmpR5JWzeexqn2R6OVCgMnQgtnfHEQmRURP5IMrRP5vAwa9UMGlc0ZPkiEgzSB RdsjNm+Qs1oPiqAmcGIlwBj7qIVWQ0GEsDsgYxYIhMzq18rDHNMII9Kgn1Se99IRoxZ6 JXHx6D0S2S1yUqkRwq5pADlr5cFkiQp0pLpAfYPKt3jAQFR2RJ33YiMgw+Zj5iSEg+fy IsdoJ++Ge/3c0rP5du3jsn1dGs3KzYxPUzpYmIQND2OQ04XeBQBDNs3WCgFrcPnGU7XR x2wg== X-Gm-Message-State: AGi0Puau/+4AQcr+TSRxZt8FGO4MdbGh5vLo4nxHG0QF9LPbiOOGsNqp s0FvY6Tp6LlfcBZN+faT+H3azw== X-Google-Smtp-Source: APiQypIIDp3Aj7gJSpVJ/LpltmID+bFQ6t9Z5J6KgWOyiu1YsJcaTx88+obo6kMWqL+PJTWdhscFwQ== X-Received: by 2002:a5d:610e:: with SMTP id v14mr15760928wrt.159.1586883882233; Tue, 14 Apr 2020 10:04:42 -0700 (PDT) Received: from localhost.localdomain ([2001:171b:226b:54a0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id x18sm19549147wrs.11.2020.04.14.10.04.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Apr 2020 10:04:41 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-mm@kvack.org Cc: joro@8bytes.org, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, kevin.tian@intel.com, baolu.lu@linux.intel.com, Jonathan.Cameron@huawei.com, jacob.jun.pan@linux.intel.com, christian.koenig@amd.com, zhangfei.gao@linaro.org, jgg@ziepe.ca, xuzaibo@huawei.com, Jean-Philippe Brucker Subject: [PATCH v5 17/25] iommu/arm-smmu-v3: Implement mm operations Date: Tue, 14 Apr 2020 19:02:45 +0200 Message-Id: <20200414170252.714402-18-jean-philippe@linaro.org> X-Mailer: git-send-email 2.26.0 In-Reply-To: <20200414170252.714402-1-jean-philippe@linaro.org> References: <20200414170252.714402-1-jean-philippe@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hook SVA operations to support sharing page tables with the SMMUv3: * dev_enable/disable/has_feature for device drivers to modify the SVA state. * sva_bind/unbind and sva_get_pasid to bind device and address spaces. * The mm_attach/detach/clear/invalidate/free callbacks from iommu-sva The clear() operation has to detach the page tables while DMA may still be running (because the process died). To avoid any event 0x0a print (C_BAD_CD) we disable translation and error reporting, without clearing CD.V. PCIe Translation Requests and Page Requests are silently denied. The detach() operation always happens whether or not clear() was invoked, and properly disables the CD. Signed-off-by: Jean-Philippe Brucker --- v4->v5: Add clear() operation --- drivers/iommu/Kconfig | 1 + drivers/iommu/arm-smmu-v3.c | 214 +++++++++++++++++++++++++++++++++++- 2 files changed, 211 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 411a7ee2ab12d..8118f090a51b3 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -435,6 +435,7 @@ config ARM_SMMU_V3 tristate "ARM Ltd. System MMU Version 3 (SMMUv3) Support" depends on ARM64 select IOMMU_API + select IOMMU_SVA select IOMMU_IO_PGTABLE_LPAE select GENERIC_MSI_IRQ_DOMAIN help diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index d209d85402a83..6640c2ac2a7c5 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -36,6 +36,7 @@ #include =20 #include "io-pgtable-arm.h" +#include "iommu-sva.h" =20 /* MMIO registers */ #define ARM_SMMU_IDR0 0x0 @@ -739,6 +740,13 @@ struct arm_smmu_option_prop { static DEFINE_XARRAY_ALLOC1(asid_xa); static DEFINE_SPINLOCK(contexts_lock); =20 +/* + * When a process dies, DMA is still running but we need to clear the pg= d. If we + * simply cleared the valid bit from the context descriptor, we'd get ev= ent 0x0a + * which are not recoverable. + */ +static struct arm_smmu_ctx_desc invalid_cd =3D { 0 }; + static struct arm_smmu_option_prop arm_smmu_options[] =3D { { ARM_SMMU_OPT_SKIP_PREFETCH, "hisilicon,broken-prefetch-cmd" }, { ARM_SMMU_OPT_PAGE0_REGS_ONLY, "cavium,cn9900-broken-page1-regspace"}, @@ -1649,7 +1657,9 @@ static int __arm_smmu_write_ctx_desc(struct arm_smm= u_domain *smmu_domain, * (2) Install a secondary CD, for SID+SSID traffic. * (3) Update ASID of a CD. Atomically write the first 64 bits of the * CD, then invalidate the old entry and mappings. - * (4) Remove a secondary CD. + * (4) Quiesce the context without clearing the valid bit. Disable + * translation, and ignore any translation fault. + * (5) Remove a secondary CD. */ u64 val; bool cd_live; @@ -1666,8 +1676,11 @@ static int __arm_smmu_write_ctx_desc(struct arm_sm= mu_domain *smmu_domain, val =3D le64_to_cpu(cdptr[0]); cd_live =3D !!(val & CTXDESC_CD_0_V); =20 - if (!cd) { /* (4) */ + if (!cd) { /* (5) */ val =3D 0; + } else if (cd =3D=3D &invalid_cd) { /* (4) */ + val &=3D ~(CTXDESC_CD_0_S | CTXDESC_CD_0_R); + val |=3D CTXDESC_CD_0_TCR_EPD0; } else if (cd_live) { /* (3) */ val &=3D ~CTXDESC_CD_0_ASID; val |=3D FIELD_PREP(CTXDESC_CD_0_ASID, cd->asid); @@ -1884,7 +1897,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_share_asi= d(u16 asid) return NULL; } =20 -__maybe_unused static struct arm_smmu_ctx_desc *arm_smmu_alloc_shared_cd(struct mm_stru= ct *mm) { u16 asid; @@ -1978,7 +1990,6 @@ static struct arm_smmu_ctx_desc *arm_smmu_alloc_sha= red_cd(struct mm_struct *mm) return ERR_PTR(ret); } =20 -__maybe_unused static void arm_smmu_free_shared_cd(struct arm_smmu_ctx_desc *cd) { if (arm_smmu_free_asid(cd)) { @@ -3008,6 +3019,16 @@ static int arm_smmu_attach_dev(struct iommu_domain= *domain, struct device *dev) master =3D dev_iommu_priv_get(dev); smmu =3D master->smmu; =20 + /* + * Checking that SVA is disabled ensures that this device isn't bound t= o + * any mm, and can be safely detached from its old domain. Bonds cannot + * be removed concurrently since we're holding the group mutex. + */ + if (iommu_sva_enabled(dev)) { + dev_err(dev, "cannot attach - SVA enabled\n"); + return -EBUSY; + } + arm_smmu_detach_dev(master); =20 mutex_lock(&smmu_domain->init_mutex); @@ -3107,6 +3128,99 @@ arm_smmu_iova_to_phys(struct iommu_domain *domain,= dma_addr_t iova) return ops->iova_to_phys(ops, iova); } =20 +static void arm_smmu_mm_invalidate(struct device *dev, int pasid, void *= entry, + unsigned long iova, size_t size) +{ + /* TODO: Invalidate ATC */ +} + +static int arm_smmu_mm_attach(struct device *dev, int pasid, void *entry= , + bool attach_domain) +{ + struct arm_smmu_ctx_desc *cd =3D entry; + struct iommu_domain *domain =3D iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain =3D to_smmu_domain(domain); + + /* + * If another device in the domain has already been attached, the + * context descriptor is already valid. + */ + if (!attach_domain) + return 0; + + return arm_smmu_write_ctx_desc(smmu_domain, pasid, cd); +} + +static void arm_smmu_mm_clear(struct device *dev, int pasid, void *entry= ) +{ + struct arm_smmu_ctx_desc *cd =3D entry; + struct iommu_domain *domain =3D iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain =3D to_smmu_domain(domain); + + /* + * DMA may still be running. Keep the cd valid but disable + * translation, so that new events will still result in stall. + */ + arm_smmu_write_ctx_desc(smmu_domain, pasid, &invalid_cd); + + /* + * The ASID allocator won't broadcast the final TLB invalidations + * for this ASID, so we need to do it manually. + */ + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + + /* TODO: invalidate ATC */ +} + +static void arm_smmu_mm_detach(struct device *dev, int pasid, void *entr= y, + bool detach_domain, bool cleared) +{ + struct arm_smmu_ctx_desc *cd =3D entry; + struct iommu_domain *domain =3D iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain =3D to_smmu_domain(domain); + + if (detach_domain) { + arm_smmu_write_ctx_desc(smmu_domain, pasid, NULL); + + if (!cleared) + /* See comment in arm_smmu_mm_clear() */ + arm_smmu_tlb_inv_asid(smmu_domain->smmu, cd->asid); + } + + /* TODO: invalidate ATC */ +} + +static void *arm_smmu_mm_alloc(struct mm_struct *mm) +{ + return arm_smmu_alloc_shared_cd(mm); +} + +static void arm_smmu_mm_free(void *entry) +{ + arm_smmu_free_shared_cd(entry); +} + +static struct io_mm_ops arm_smmu_mm_ops =3D { + .alloc =3D arm_smmu_mm_alloc, + .invalidate =3D arm_smmu_mm_invalidate, + .attach =3D arm_smmu_mm_attach, + .clear =3D arm_smmu_mm_clear, + .detach =3D arm_smmu_mm_detach, + .free =3D arm_smmu_mm_free, +}; + +static struct iommu_sva * +arm_smmu_sva_bind(struct device *dev, struct mm_struct *mm, void *drvdat= a) +{ + struct iommu_domain *domain =3D iommu_get_domain_for_dev(dev); + struct arm_smmu_domain *smmu_domain =3D to_smmu_domain(domain); + + if (smmu_domain->stage !=3D ARM_SMMU_DOMAIN_S1) + return ERR_PTR(-EINVAL); + + return iommu_sva_bind_generic(dev, mm, &arm_smmu_mm_ops, drvdata); +} + static struct platform_driver arm_smmu_driver; =20 static @@ -3225,6 +3339,7 @@ static void arm_smmu_remove_device(struct device *d= ev) =20 master =3D dev_iommu_priv_get(dev); smmu =3D master->smmu; + WARN_ON(iommu_sva_disable(dev)); arm_smmu_detach_dev(master); iommu_group_remove_device(dev); iommu_device_unlink(&smmu->iommu, dev); @@ -3344,6 +3459,90 @@ static void arm_smmu_get_resv_regions(struct devic= e *dev, iommu_dma_get_resv_regions(dev, head); } =20 +static bool arm_smmu_iopf_supported(struct arm_smmu_master *master) +{ + return false; +} + +static bool arm_smmu_dev_has_feature(struct device *dev, + enum iommu_dev_features feat) +{ + struct arm_smmu_master *master =3D dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + if (!(master->smmu->features & ARM_SMMU_FEAT_SVA)) + return false; + + /* SSID and IOPF support are mandatory for the moment */ + return master->ssid_bits && arm_smmu_iopf_supported(master); + default: + return false; + } +} + +static bool arm_smmu_dev_feature_enabled(struct device *dev, + enum iommu_dev_features feat) +{ + struct arm_smmu_master *master =3D dev_iommu_priv_get(dev); + + if (!master) + return false; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return iommu_sva_enabled(dev); + default: + return false; + } +} + +static int arm_smmu_dev_enable_sva(struct device *dev) +{ + struct arm_smmu_master *master =3D dev_iommu_priv_get(dev); + struct iommu_sva_param param =3D { + .min_pasid =3D 1, + .max_pasid =3D 0xfffffU, + }; + + param.max_pasid =3D min(param.max_pasid, (1U << master->ssid_bits) - 1)= ; + return iommu_sva_enable(dev, ¶m); +} + +static int arm_smmu_dev_enable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_has_feature(dev, feat)) + return -ENODEV; + + if (arm_smmu_dev_feature_enabled(dev, feat)) + return -EBUSY; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return arm_smmu_dev_enable_sva(dev); + default: + return -EINVAL; + } +} + +static int arm_smmu_dev_disable_feature(struct device *dev, + enum iommu_dev_features feat) +{ + if (!arm_smmu_dev_feature_enabled(dev, feat)) + return -EINVAL; + + switch (feat) { + case IOMMU_DEV_FEAT_SVA: + return iommu_sva_disable(dev); + default: + return -EINVAL; + } +} + static struct iommu_ops arm_smmu_ops =3D { .capable =3D arm_smmu_capable, .domain_alloc =3D arm_smmu_domain_alloc, @@ -3362,6 +3561,13 @@ static struct iommu_ops arm_smmu_ops =3D { .of_xlate =3D arm_smmu_of_xlate, .get_resv_regions =3D arm_smmu_get_resv_regions, .put_resv_regions =3D generic_iommu_put_resv_regions, + .dev_has_feat =3D arm_smmu_dev_has_feature, + .dev_feat_enabled =3D arm_smmu_dev_feature_enabled, + .dev_enable_feat =3D arm_smmu_dev_enable_feature, + .dev_disable_feat =3D arm_smmu_dev_disable_feature, + .sva_bind =3D arm_smmu_sva_bind, + .sva_unbind =3D iommu_sva_unbind_generic, + .sva_get_pasid =3D iommu_sva_get_pasid_generic, .pgsize_bitmap =3D -1UL, /* Restricted during device attach */ }; =20 --=20 2.26.0