From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DD7CC77B75 for ; Mon, 22 May 2023 11:26:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BA38900004; Mon, 22 May 2023 07:26:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26796900002; Mon, 22 May 2023 07:26:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1300F900004; Mon, 22 May 2023 07:26:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id F3952900002 for ; Mon, 22 May 2023 07:26:22 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B2F2C406CC for ; Mon, 22 May 2023 11:26:22 +0000 (UTC) X-FDA: 80817662604.15.6B262B3 Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf20.hostedemail.com (Postfix) with ESMTP id A88D31C0018 for ; Mon, 22 May 2023 11:26:20 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=UDra9+da; spf=pass (imf20.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=quarantine) header.from=collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684754781; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GY5pe0jBxu2ovvOYSiu+8+fq+4W7w980avoP5bsLBks=; b=hHFpur3lSmYDyCMTDxdgWS118FafydI1jpOrBu34SP8Z8Cdxh7H/GyL/D14K2GXIqMxhIX Ul3zsdWkWLbIvoAVLCJVqJit5G5Q+yAYM3MNPbYEkl0VSO1daGd7xDFldxZjHLJFhMiSXt LQPTPFRCpyHW2bbF/r86GUIl0sX7Hc4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684754781; a=rsa-sha256; cv=none; b=xx57vCmqgU6F8fnQS4yv+uik6/VHqBDiIwR3yPvBC/xJgM/JTRhyC0oMkNWxfN6+dCvND+ pkT6BfFw+SNPrsXXzoSCju9WxKGBIDSc4z/ctjpvM+Pa+0cYhS2PEGnD91dQ7FBDrqoUgA ex029HpLqPb7fS6hR+bCTZU2u3fWVNM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=UDra9+da; spf=pass (imf20.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=quarantine) header.from=collabora.com Received: from [192.168.10.48] (unknown [119.155.11.156]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id 2957166031C5; Mon, 22 May 2023 12:26:11 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1684754778; bh=LQG2IlUMjHSbgYMY07PFPfmnRDnifZ4ZmeuWipX+Des=; h=Date:Cc:Subject:To:References:From:In-Reply-To:From; b=UDra9+daVWN3ILIqe8Nvyz4bfNp2gNAq+NWv458WpSCRF3UqLR4qak0MSucQAed0G IP5gAf1sUCB6OeenfcuLa+uX3RZpgsCZRhpsJGg8j63MNAZ70bKIVVib/CF+0xdF9b DJk+wDhGVQ7BT1ht1hPNLeXN/vLTYotqPAttoDeHUs9HvIdNjapqRURe/coyuUyHil FKsdHGlj1gaD+fKcrxLAT1e5sIv6H/Js32UArp95cqBHxOhByMWq7Wg1EuGycodHdB D/DsELoIgNwdjqGMyumWPBoiDldhwQ30UHl+b8a3gWUBuYDfE6A1nD8dln8sfjgtVJ kCwTivpdmn1oQ== Message-ID: <0edfaf12-66f2-86d3-df1c-f5dff10fb743@collabora.com> Date: Mon, 22 May 2023 16:26:07 +0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Cc: Muhammad Usama Anjum , Paul Gofman , Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Cyrill Gorcunov , =?UTF-8?B?TWljaGHFgiBNaXJvc8WCYXc=?= , Andrew Morton , Suren Baghdasaryan , Andrei Vagin , Alex Sierra , Matthew Wilcox , Pasha Tatashin , Danylo Mocherniuk , Axel Rasmussen , "Gustavo A . R . Silva" , David Hildenbrand , Dan Williams , linux-kernel@vger.kernel.org, Mike Rapoport , linux-fsdevel@vger.kernel.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com, Nadav Amit Subject: Re: [PATCH RESEND v15 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs Content-Language: en-US To: Peter Xu , linux-mm@kvack.org References: <20230420060156.895881-1-usama.anjum@collabora.com> <20230420060156.895881-3-usama.anjum@collabora.com> From: Muhammad Usama Anjum In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: A88D31C0018 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: yba5om4uwrnpw4xt1tscqy3nki3cdibf X-HE-Tag: 1684754780-944667 X-HE-Meta: U2FsdGVkX1+wgaQXayImJqGo1VYVCm5EgiS44aahmhhdbewwrSE7giZfKY/UrjEsG16Sgdkft/pHGkBTYejk/q03/zSXz92cD8aZx29e7CR464oZsnj61POtJE6Qa0ynTEPWOGxRMCHGFC1zW/bywJodIVZOAsnVVHwPToW+Hu2DQazoXbbjpyLBWYkspO6+U2zViazInZaQlcw+RV8bcY2ZRgRGfxsf7IXoc/7cSXo+GpVqzNfIJ3CPYC5Bk7jlozXMEoUDQladFeWPRvMoX7nEKKTPD/d1bzCbBp+LcVPGdFtu0xmejQyCy86lKYYCsfpDqvvAgxaNFNg8aUqMwkEy94wbao3jyG60ZgUaKf+8TQ+aLhIOaWfAMtl6EZo4tKt4qLTYeWCPvoc5qbqFfh+/g+q9LVy7dcbNl5TL465785LUJOBepgWidm/b3tnXzOY6B+BWpmccfZmVLzB/pkBy5m0poq0e+u+k8qutzeZSs/KMX8puTytUOJt2yNSuAnjRnqfDHadNaMvek1splvdMBwPgySM5nuYRxW0aDf5eC1uGnKFFpqn/hE7wpyZ6r1TmwnOu1nCEFvmKbaiJFjGHJ3mWb1UvpgydYgwgbrkmS9zDKE1rVnvUs1AraMVQr4s/9YwhC1G/YGeXwP3eMJ8vJ9MtUNPmidNxTB64rLroq3SpC8agQSWr3Ka6g1Bx7cTlBCE1K//DInuQa783TU34cnyMKuzKSlXYKViBWvL9EycT5gFEQwwpXB8j6feqM+cSaPXQ/jrhQJMIYG9FSrKS7//LFg/DN5zhFOgRd56s6f35MEClXlEm7atEMpJ3u4EBN4F5XV3wqbo/jNDPWIM6aXxmvDvmcdIhle2ukCDbIJ6+d2bY5J/8NApQu8mNFuuOZIo1iujTbK9GKn6h0mrgKsmNoBQnyoGZ3jNbcblVmxxad23ausFRulLsvuljO+MKMgqAbNkfeZ/H17v OFTfpjMy fFkSPv+Gf++u555iNQI2QCcBD+jKdLgkSYbi99NbIqtltjtcZ7jVrzrTGrO6K1g8KAaNNBbc8rGt+Lp3JYY9dqTsl1PR4Q0wkPfeHOkk+qLKmakQ+q1Et5XQTgH45l22t7sur3ktPjSfp5bd9ZqTsgYz/b/elYPwLuGcPVUEH6MLmtjnjLbp2iKgUnB8NdhdnLqVRj/owPGI2bfs0hqhIZIp99PmfD5AbIsasYq9W48vIdddrcpLP/DeoDub9L+7GUMzx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/22/23 3:24 PM, Muhammad Usama Anjum wrote: > On 4/26/23 7:13 PM, Peter Xu wrote: >> Hi, Muhammad, >> >> On Wed, Apr 26, 2023 at 12:06:23PM +0500, Muhammad Usama Anjum wrote: >>> On 4/20/23 11:01 AM, Muhammad Usama Anjum wrote: >>>> +/* Supported flags */ >>>> +#define PM_SCAN_OP_GET (1 << 0) >>>> +#define PM_SCAN_OP_WP (1 << 1) >>> We have only these flag options available in PAGEMAP_SCAN IOCTL. >>> PM_SCAN_OP_GET must always be specified for this IOCTL. PM_SCAN_OP_WP can >>> be specified as need. But PM_SCAN_OP_WP cannot be specified without >>> PM_SCAN_OP_GET. (This was removed after you had asked me to not duplicate >>> functionality which can be achieved by UFFDIO_WRITEPROTECT.) >>> >>> 1) PM_SCAN_OP_GET | PM_SCAN_OP_WP >>> vs >>> 2) UFFDIO_WRITEPROTECT >>> >>> After removing the usage of uffd_wp_range() from PAGEMAP_SCAN IOCTL, we are >>> getting really good performance which is comparable just like we are >>> depending on SOFT_DIRTY flags in the PTE. But when we want to perform wp, >>> PM_SCAN_OP_GET | PM_SCAN_OP_WP is more desirable than UFFDIO_WRITEPROTECT >>> performance and behavior wise. >>> >>> I've got the results from someone else that UFFDIO_WRITEPROTECT block >>> pagefaults somehow which PAGEMAP_IOCTL doesn't. I still need to verify this >>> as I don't have tests comparing them one-to-one. >>> >>> What are your thoughts about it? Have you thought about making >>> UFFDIO_WRITEPROTECT perform better? >>> >>> I'm sorry to mention the word "performance" here. Actually we want better >>> performance to emulate Windows syscall. That is why we are adding this >>> functionality. So either we need to see what can be improved in >>> UFFDIO_WRITEPROTECT or can I please add only PM_SCAN_OP_WP back in >>> pagemap_ioctl? >> >> I'm fine if you want to add it back if it works for you. Though before >> that, could you remind me why there can be a difference on performance? > I've looked at the code again and I think I've found something. Lets look > at exact performance numbers: > > I've run 2 different tests. In first test UFFDIO_WRITEPROTECT is being used > for engaging WP. In second test PM_SCAN_OP_WP is being used. I've measured > the average write time to the same memory which is being WP-ed and total > time of execution of these APIs: > > **avg write time:** > | No of pages | 2000 | 8192 | 100000 | 500000 | > |------------------------|------|------|--------|--------| > | UFFDIO_WRITEPROTECT | 2200 | 2300 | 4100 | 4200 | > | PM_SCAN_OP_WP | 2000 | 2300 | 2500 | 2800 | > > **Execution time measured in rdtsc:** > | No of pages | 2000 | 8192 | 100000 | 500000 | > |------------------------|------|-------|--------|--------| > | UFFDIO_WRITEPROTECT | 3200 | 14000 | 59000 | 58000 | > | PM_SCAN_OP_WP | 1900 | 7000 | 38000 | 40000 | > > Avg write time for UFFDIO_WRITEPROTECT is 1.3 times slow. The execution > time is 1.5 times slower in the case of UFFDIO_WRITEPROTECT. So > UFFDIO_WRITEPROTECT is making writes slower to the pages and execution time > is also slower. > > This proves that PM_SCAN_OP_WP is better than UFFDIO_WRITEPROTECT. Although > PM_SCAN_OP_WP and UFFDIO_WRITEPROTECT have been implemented differently. We > should have seen no difference in performance. But we have quite a lot of > difference in performance here. PM_SCAN_OP_WP takes read mm lock, uses > walk_page_range() to walk over pages which finds VMAs from address ranges > to walk over them and pagemap_scan_pmd_entry() is handling most of the work > including tlb flushing. UFFDIO_WRITEPROTECT is also taking the mm lock and > iterating from all the different page directories until a pte is found and > then flags are updated there and tlb is flushed for every pte. > > My next deduction would be that we are getting worse performance as we are > flushing tlb for one page at a time in case of UFFDIO_WRITEPROTECT. While > we flush tlb for 512 pages (moslty) at a time in case of PM_SCAN_OP_WP. > I've just verified this by adding some logs to the change_pte_range() and > pagemap_scan_pmd_entry(). Logs are attached. I've allocated memory of 1000 > pages and write-protected it with UFFDIO_WRITEPROTECT and PM_SCAN_OP_WP. > The logs show that UFFDIO_WRITEPROTECT has flushed tlb 1000 times of size 1 > page each time. While PM_SCAN_OP_WP has flushed only 3 times of bigger > sizes. I've learned over my last experience that tlb flush is very > expensive. Probably this is what we need to improve if we don't want to add > PM_SCAN_OP_WP? > > The UFFDIO_WRITEPROTECT uses change_pte_range() which is very generic > function and I'm not sure if can try to not do tlb flushes if uffd_wp is > true. We can try to do flush somewhere else and hopefully we should do only > one flush if possible. It will not be so straight forward to move away from > generic fundtion. Thoughts? I've just tested this theory of not doing per pte flushes and only did one flush on entire range in uffd_wp_range(). But it didn't improve the situation either. I was wrong that tlb flushes may be the cause. > > >> Thanks, >> > -- BR, Muhammad Usama Anjum