From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D96CC25B76 for ; Tue, 11 Jun 2024 07:49:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97FC96B0093; Tue, 11 Jun 2024 03:49:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92DE86B0095; Tue, 11 Jun 2024 03:49:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CF386B0096; Tue, 11 Jun 2024 03:49:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5F6666B0093 for ; Tue, 11 Jun 2024 03:49:10 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 105EE16124B for ; Tue, 11 Jun 2024 07:49:10 +0000 (UTC) X-FDA: 82217832060.13.E6FD04C Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by imf12.hostedemail.com (Postfix) with ESMTP id 0EB204001C for ; Tue, 11 Jun 2024 07:49:07 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PYxcdTYV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of zyjzyj2000@gmail.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=zyjzyj2000@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718092148; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kecLidF3g75Y0M4dgmXIi2U7qv7nfX0gCWhXKA3lhlo=; b=X9ip1HThwCMRdhFFBV73Lxr37FWm/lvyb48VMRtcCnauIP8I8EWCnHPuxtA/FWfR9twpZn m0xGpicHa6Fx9V05e9IKXCdRRTSNc/DfprSBLlgRLFo/JgS9Q2e8LjJi7EmSFzqftekOf8 3jkXrwHc4c2vol2HNNXf8Ty58nP9BPA= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PYxcdTYV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of zyjzyj2000@gmail.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=zyjzyj2000@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718092148; a=rsa-sha256; cv=none; b=z1/ES5hpcA/WO3MQZ4Tt5CUif/b1C2OJk7Q3r4PJ3LNshXxHzlDVf3p4m3ErKABbrG+MeQ Ug9xc79rBtGl1CJ4yilp0docsZZpm4wjr2VWblwnBu5j3VbeBcyQ5cmDCSjSAc/Xpzy+Vg siVVuyBEmPP5YofFR4TtPwXiM3IXYuY= Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a6f11a2d18aso257561266b.2 for ; Tue, 11 Jun 2024 00:49:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718092146; x=1718696946; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=kecLidF3g75Y0M4dgmXIi2U7qv7nfX0gCWhXKA3lhlo=; b=PYxcdTYV8m2WrWA+RogNihN8JiveETd9YPKHf5N+rN6afCZKDJ+A0OJ4Xax798qnK8 /Ilvfx0YJviC5YE2IGrGFCgmPMVv+qKyDXnzLj4nTiUJj4O5OYGYJ5XcOpEdV2uQoxt8 mvnjMfurucfkvEbSoHAMXkqEgLeXO99XlYm/uXU4IGs5qcZWEhNU/fzjESp5BizcRR6y AvhlPx6/txfd0putRO0eElqqUXr20agXFVBSCCzrpgg7YUVuXbSFDlfPlxKAVn452w5O TxAIgGrokbnxynU/ErCppPN89rzErengC4m44mSD/3syfSqEMk1ihLz/sNy7XvLbbI1+ X44A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718092146; x=1718696946; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kecLidF3g75Y0M4dgmXIi2U7qv7nfX0gCWhXKA3lhlo=; b=KKRwjaGsl3Fv6DX3IMi/Ee51ByL97kFmWm/+5EmReytvbTRJ+al2UprqEhpX5ivhGt AdvdiWhdG9gIS1AHbf4Yr8qEHFn4vLEdoh2T8WuN7YwKLiPqo+3oymmG2nlSHuBeRgTi nnI/hRtj0GBySVpkAKexxmpHf3cSOC91gRfE6lnmdFNYySCEnqauaTdGEyl7LK4lkeRN 88NFXkcHteHgC+1gTpEaFPjKO+bKX2IFSppYvLQRqymJnNw5EBB2D3YRnGhKt9Ym5uCZ cPWuj2fbx/8KGq4nN7ghEonVVUlIs8IhLEqw8q+bYGe9oNoif6HlnZaL4EThJXu0huIz ndNQ== X-Forwarded-Encrypted: i=1; AJvYcCV/dl+URldkI0IwPiS9bUgKPQaOrBe+pVwzmjq+haieMFeDmZl0UbTdeXEX9XCqg4qoyWcPU0NRE6u9amHtmT2EIAg= X-Gm-Message-State: AOJu0YyFeUYKvhCVcqzpZFCCX7i/xDXM1buNK5p//csQ1N8N5QYwk8zL vTaRKXuGVLyJbQQt30xeP3BDHxqRJMHW/xqecYrk7+Xc4L3ZNl1Q X-Google-Smtp-Source: AGHT+IFzcNCL4DB7c7G7kLyb+7UFm4o2GGJXJ/lKj59bEb2GT+3Qy+R5dFmfo0Jxs1fqaAcSpOVYtA== X-Received: by 2002:a17:906:6a0a:b0:a6f:1d19:c0b1 with SMTP id a640c23a62f3a-a6f1d19c496mr372629566b.18.1718092146075; Tue, 11 Jun 2024 00:49:06 -0700 (PDT) Received: from [10.16.124.60] ([212.227.34.98]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a6f2942b02fsm145877166b.167.2024.06.11.00.49.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Jun 2024 00:49:05 -0700 (PDT) Message-ID: <900d1d56-28ea-4c6d-b8c7-749a952e5f4b@gmail.com> Date: Tue, 11 Jun 2024 09:49:03 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to two steps To: "Zeng, Oak" , Jason Gunthorpe Cc: Leon Romanovsky , Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , "Brost, Matthew" , "Hellstrom, Thomas" , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , "Tian, Kevin" , Alex Williamson , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , "linux-doc@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-block@vger.kernel.org" , "linux-rdma@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-nvme@lists.infradead.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , "Williams, Dan J" , "jack@suse.com" , "Bommu, Krishnaiah" , "Ghimiray, Himal Prasad" References: <20240503164239.GB901876@ziepe.ca> <20240610161826.GA4966@unreal> <20240610172501.GJ791043@ziepe.ca> Content-Language: en-US From: Zhu Yanjun In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0EB204001C X-Stat-Signature: qpj6sr8os6h7w6bgh39use448uu9skqb X-Rspam-User: X-HE-Tag: 1718092147-265276 X-HE-Meta: U2FsdGVkX1/JfTa2lejxegxiVnAXY+VyHxp08Tqlv2rkwz92jXDmX0I4WMNDuV6zlt4sdLsoL3QgFru2kzwPol71JUVbVxJ6TpgnruN7lyiXaUv75ntNDnb0IinhYg0JUpm5S1k2Cb2ZXt0wsCuTfbcCvkX1GqOoE9pJzjVXXa7CfRQlnOJw1FvNDenn1gaiy44LEaIrpdZ2R/CNSOmRO70o7rIX5skvj2Xw0Y9eZf96ixipi2JXjUMT6aIdYqDrDEYYmfOG2FFHbwSE5Bl8SqRvcgrWMp80LUG5UcOt3Ipq5rpyvk+kv85vQjbgrHnyeUxxdtlw7q/nBZt5HsoKt3G3oKDY5Uu1ax5QAhq+QFMGsJzqLzFD3oJVeW6ba6dVxONEXcGl9uPqYDAYrcHu905m0KMFEYdbKlOG0VeqvsB25z38PlyEjfdZpoTESIxauTHp+D+0HATNybnSBUGA9tOJF9Pv6oZRG2wwat4VJHDzatD+ybDMuFqPUZ4oj1Yj10oXsz9vqz2O4KN+bElzEiI8fwRDxDvpGhHNRGSpVHK9datw002qte4wEJgCcoG0Poz+9HSztda4gArdxtQcIxnDi5cIeDbGw/EE/ATZNgk4UClDDmFJMB8FKjUo6yAjB9KfjjFKOxANs0KpCfSrSxoevx9aJyB9E/oUNduLTrMk2o58/HhMUGoZSANDVJHnvb8I5Typn+vK8OPmg1k70MOWQYjJaLewvojTRcceUPSfoNo7jnVRC73Qn+QrRwNxUeTRJhx4HLSHhihURIzEG/snaJQXYKQWYgLJevlWexnhLXH9HfIEWOhq9q9tdCPnU/yVwSdYE0PRnWrE0xV7t8QedXR+Uw4FCuyo/X5g/fQnpavmBw0PwkahKH6XsWOIgHfDvAexUATN/zf1ZGmm4YZS+NFev8PLjwLo0O67dzOT/A9PaAuBn9NzM8iKEVVulx9m3kvCflTNN99Lhmx Ypri9hh+ 9HJBDTBrmeb9ox7FJQ8SUEdMpEdaFz+vcUYn+ygRoayuwDs3O8LsN4llhAdqLqvSKiT/cwUrLCb6a4jv/qM+F89NbWGD/YW+Bcc7CafhL4k8AELAW5LJRSyhCSe+3wyyoWSO+LbyEEugzu2oxhFkhBPbUTc3ufWPKlSjNhsYGDZvnlE9eDag2DFs1OCprdH5Ry4bVnQfsGsCTgyWvGT4j+31TDXMdj1WqAl23tor3Djf+zI7JDPM94x+EXYIJxQnHA8lxXgS2I/DKuVeJWmnSobcFhDnsipin/77G/v+4CAAR5MpQsq4wvxiVoxx9pKxuI4J9z6mhXfdcSP6RfxyofW6w3wtRluYgGTCYWodTM3TDawpG8OfhrWNg5wW4s1fQ9sCgQ4oL1Rtgz+ndcXPUESELJlmVoxpXOAhbb6l9xGxzusso0ZeFudkyOZuBP547WoZtP66GzQinGcTmzCjojjaJJpp4O/yqu91rF7yGh+mmln1lXDbZ+agzwvTqqsjKrmEfwrEkmwjQrGxtBOtBhBWV8skJ7rcysQSSLXNaYvKu3ATGc6WBTcH3uoGIUzxjeECj1tWv5ePcXRNPRJ1Xi5Crv/agWYdNzH7lQM1iJIpg5QxFSb9fefH31H/Z3SrGz8zBw7+L9YTtBq/jlBDVaV8xxjQ748FIaKIn6KBNRpXywP5jAb3LqqroTtJoZilnFJR5tUEts2DVVHcojwLsYL4xkmR0icDURbtfG1fubNybkKdEvkaLHhgcZTR/sw2j+xNc9ymzvCnjhYxCTNKnPzFuQoehWP5oROOHVfz93Aphks8YmxEJZoWRKOjdSL3oHB818ViOmfliGQr2iiE9Y3FlWLgF8J4NybhV/r4QojPrAiRHGOWgW8VsJpn3Cdc358++p7ScYLTaFaNWHAJltSNzyLNNFJKz5WkN6knpO7dGcCmSyu46Z4uVFDqPHRPDsGOtygI/WG5eUBfeeJYH+o+fMVI4 YWbBWO+b lvp5D7FwcTO0QlViX5b9Eye1xONHUrkQDE35wWuHCdwNXAKZqitx/7PaMQSOQBMqszM3AqyNv0nz+mY7lFoXAxqML+ymavVxWJkY+KO+fyWLp7alK0LHgxW5syyWpw3nKDg1nju3aaKOcAom0v9s+BWRb8D/7VHI7xXbci7HzLkNZZVVg0iNgs2LYMUolseehqogbHkX6vLYIPJX6FL7U6okuQbyW8por+Hh+tVQUHNHYReaQUQKnAbTBZG9LbfY8vIkD8scj3sceuGV2f99rjhB74+p4b6ZulyZkMVlHbWQVvMuP4+PAI6TvX2FxBPw3/478iHmYC/f3oTBEQ5JL3v001XVIYCUVK9uY6wFgtsZhQRFypzeWcyOOwQrt7Kv+XiLO7fvhT+B7ZOFKO3gUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 10.06.24 23:28, Zeng, Oak wrote: > Hi Jason, Leon, > > I was able to fix the issue from my side. Things work fine now. Can you enlarge the dma list, then make tests with fio? Not sure if the performance is better or not. Thanks, Zhu Yanjun > I got two questions though: > > 1) The value returned from dma_link_range function is not contiguous, see below print. The "linked pa" is the function return. > I think dma_map_sgtable API would return some contiguous dma address. Is the dma-map_sgtable api is more efficient regarding the iommu page table? i.e., try to use bigger page size, such as use 2M page size when it is possible. With your new API, does it also have such consideration? I vaguely remembered Jason mentioned such thing, but my print below doesn't look like so. Maybe I need to test bigger range (only 16 pages range in the test of below printing). Comment? > > [17584.665126] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 18ef3f000 > [17584.665146] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 190d00000 > [17584.665150] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 190024000 > [17584.665153] drm_svm_hmmptr_map_dma_pages iova.dma_addr = 0x0, linked pa = 178e89000 > > 2) in the comment of dma_link_range function, it is said: " @dma_offset needs to be advanced by the caller with the size of previous page that was linked + DMA address returned for the previous page". > Is this description correct? I don't understand the part "+ DMA address returned for the previous page ". > In my codes, let's say I call this function to link 10 pages, the first dma_offset is 0, second is 4k, third 8k. This worked for me. I didn't add the previously returned dma address. > Maybe I need more test. But any comment? > > Thanks, > Oak > >> -----Original Message----- >> From: Jason Gunthorpe >> Sent: Monday, June 10, 2024 1:25 PM >> To: Zeng, Oak >> Cc: Leon Romanovsky ; Christoph Hellwig ; >> Robin Murphy ; Marek Szyprowski >> ; Joerg Roedel ; Will >> Deacon ; Chaitanya Kulkarni ; >> Brost, Matthew ; Hellstrom, Thomas >> ; Jonathan Corbet ; Jens >> Axboe ; Keith Busch ; Sagi >> Grimberg ; Yishai Hadas ; >> Shameer Kolothum ; Tian, Kevin >> ; Alex Williamson ; >> Jérôme Glisse ; Andrew Morton > foundation.org>; linux-doc@vger.kernel.org; linux-kernel@vger.kernel.org; >> linux-block@vger.kernel.org; linux-rdma@vger.kernel.org; >> iommu@lists.linux.dev; linux-nvme@lists.infradead.org; >> kvm@vger.kernel.org; linux-mm@kvack.org; Bart Van Assche >> ; Damien Le Moal >> ; Amir Goldstein >> ; josef@toxicpanda.com; Martin K. Petersen >> ; daniel@iogearbox.net; Williams, Dan J >> ; jack@suse.com; Zhu Yanjun >> ; Bommu, Krishnaiah >> ; Ghimiray, Himal Prasad >> >> Subject: Re: [RFC RESEND 00/16] Split IOMMU DMA mapping operation to >> two steps >> >> On Mon, Jun 10, 2024 at 04:40:19PM +0000, Zeng, Oak wrote: >>> Thanks Leon and Yanjun for the reply! >>> >>> Based on the reply, we will continue use the current version for >>> test (as it is tested for vfio and rdma). We will switch to v1 once >>> it is fully tested/reviewed. >> I'm glad you are finding it useful, one of my interests with this work >> is to improve all the HMM users. >> >> Jason -- Best