From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6C83EB64DD for ; Mon, 7 Aug 2023 03:33:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E469B6B0072; Sun, 6 Aug 2023 23:33:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DF60E8D0002; Sun, 6 Aug 2023 23:33:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C979F8D0001; Sun, 6 Aug 2023 23:33:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B9BE76B0072 for ; Sun, 6 Aug 2023 23:33:07 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6D4E914047A for ; Mon, 7 Aug 2023 03:33:07 +0000 (UTC) X-FDA: 81095887614.25.688F742 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by imf18.hostedemail.com (Postfix) with ESMTP id A73461C0009 for ; Mon, 7 Aug 2023 03:33:05 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=jt1NcVky; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of avagin@gmail.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=avagin@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691379185; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PS6IeiZC33tTmiycd5oF9VziudnWFwA4fXSnPFJgzm0=; b=xjeWrMdX2XkexBlrdqIWKsNcv0pPpuEmVvAW0/UwCzvCH5Xg7Aou+Xmg8n43qCrLGbaXsu HH79VaVlczMlj40Pa3CZLzuj0R/j1E1Dh2ygSoq3yfMwgtGqf6OzTd7/SIbP+K5XBJHKP9 Qidf8YbcNLV9HSwiw7nQBNzFoQp/OD8= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=jt1NcVky; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of avagin@gmail.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=avagin@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691379185; a=rsa-sha256; cv=none; b=nhHamg6s5rCJW5gsJ/TtjjYhkqv8H1wBJrrLRMcVDhqeZgwiOBKvsXJhCxb3b/3r0jde1q KP+nGU4Y/RpMyPv10rYumAbhu8h+oC7Hla7AMfR16JyhvJyS1htUUEg4HCP4pif2NDrtGQ Yat/QOSCKlNq1G+91b6kIud3ZNR7Ibc= Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-d0b597e7ac1so4701968276.1 for ; Sun, 06 Aug 2023 20:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691379185; x=1691983985; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=PS6IeiZC33tTmiycd5oF9VziudnWFwA4fXSnPFJgzm0=; b=jt1NcVkycFoQG/yHGtB+0jJlF/PZNamsiqBz0RYMIxTM2FlSh/KhjEUFsd4aHnhmP6 MIhB9FV/feBXmmOswUlFFjoABsS06vHrlIE7o9OtVnaXAJ66xaoRySmjFz/hP+hlylat nmRdzA9UoLApEKAH12FPgawBGQ4j8n52ukQCgNh3Oxt1MTXRI+QvXD4qU2HrCGhDA/eq S81N6xgsx7YR+qmVSykEcXlQlbvpyNttJ8kmT0uQgRkAX46FLXaHjXRCAGyusZrFz+sB /qeAWIvGmiYogbK/PblG/PRiZ4NINhBEMonwBXNXWdaRYuFmAKV54FBzEQRXUk3aL7un /xAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691379185; x=1691983985; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PS6IeiZC33tTmiycd5oF9VziudnWFwA4fXSnPFJgzm0=; b=NRAdLMWd2mvbGcsT9Zx1n67t4ix0LwT5BsMyrpHNEmcIYOdetpeBkm5irYR9wSdvnu VMQztdDJPF1o6xMDcNPV4XjWBCrgc/tSi9G6qU2TFeB0uQ7kUY4y24Wrvvr1/qu2A0xd sr0zX+mTU5ju+YyciIZw0CPbC3XvG7woJdx/eJsgf0uoK1GBw1/eNPsLJfN5VnaiHkkl vFUG7U2821mT0Lqq9/DKjZjLzyOkTzim+hlF4Qt6Zs0+A9Oqm8chdYgBSIcTdM7DiIAp sDIRtc4vPKrPqadvK49q/a5tqxpYW9WPzBS7PJHvLbnUS7/XS4koI6LuQdG4byWp+Iep Nmug== X-Gm-Message-State: AOJu0Yx+rERpyXr1eqcQa/ndGlu5We+/XTX7dt9iihM1wCWHl2bjDJau Yy3tzJ31QUl5JoibfFQ+ZuItWk163IP7oZR7r5Q= X-Google-Smtp-Source: AGHT+IGPWU9smSGwGTr83XrOkHe/s5ajPSR2PQbL5KXhpXzqbR4Ms/gkpEunaKMSN++k3I+OK1Jz5YNuHLhT1H1UCHQ= X-Received: by 2002:a25:26c9:0:b0:d0d:786:2999 with SMTP id m192-20020a2526c9000000b00d0d07862999mr7742055ybm.29.1691379184552; Sun, 06 Aug 2023 20:33:04 -0700 (PDT) MIME-Version: 1.0 References: <20230727093637.1262110-1-usama.anjum@collabora.com> <20230727093637.1262110-3-usama.anjum@collabora.com> <6b6a4e1c-a9e9-9592-d5b4-3c9210c8b650@collabora.com> In-Reply-To: <6b6a4e1c-a9e9-9592-d5b4-3c9210c8b650@collabora.com> From: Andrei Vagin Date: Sun, 6 Aug 2023 20:32:53 -0700 Message-ID: Subject: Re: WIP: Performance improvements To: Muhammad Usama Anjum Cc: =?UTF-8?B?TWljaGHFgiBNaXJvc8WCYXc=?= , Paul Gofman , Peter Xu , David Hildenbrand , Andrew Morton , Danylo Mocherniuk , Cyrill Gorcunov , Mike Rapoport , Nadav Amit , Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Matthew Wilcox , Pasha Tatashin , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com, =?UTF-8?B?TWljaGHFgiBNaXJvc8WCYXc=?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: A73461C0009 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 67xnm1rexhizmmcau7zzwtrq3zjws6sa X-HE-Tag: 1691379185-979006 X-HE-Meta: U2FsdGVkX1+VPBVT9a6uzlEbnayStNLCs1izRw/RL2QsiR4tz7a9piAfnEW4bfOuhHfnoJ82krpd79lPqSOy/PP2hWlt9/VqJU6+AeX97DT7L+X4uDUW8xh2FNwsyZbcn6ALH8kyfFYREVwDMPbUj0mYK4NMNBmNia4kYVpcyPVJZgtye9j17JguksmwUgL0dh8cZHBwan8WRsgMB4tbBET2VMp0UqSWPld2yPr7QxKgTiGprvEDTpZxpELg/f4+rtdOq1GbAiInsLDFHQd2Nh2fGsF4IduoRyyih9cN0w93TJMQ4fNg8jPQLQJucTzb3WOcxfCMQKGZHMWZCxt6u2DmG9RwgiI8eIb+6K6kv2ZjO7mNZVFYYHBhVfwHWXvDO93g1OTViAIXRp7Op8NTSg9OGqWnu+3zTS/ZC12BWJzgDv81220UhY4c1hZbj2NM6saetstBU3fpe42w8mLkv4QIY9VycgFTXZbJCJ7BAvt8ynVEAJds2yoAL9FQxTTLuDWzufABO/4Nqs1SPaqmNolpj1sbJ0MdYl1kUEuMkB83pCjTqeRgAFW0F53e19OHo32StRm75uiOoGjFVff87c0iS5Wc1CdSs8y70Co+qZpJLxs390G3EDJ5bS7wAGcj8ld1YfKLYiKuWLkRGnuq3GnnLPSNTzJ9Tz8BmtyNO9L2LOnaGkqEj+R2h8itmuVoVyqT5Igv2GP/plRwni5Avv223u69IAsUYO/LANSERgZ9Dstb7oPaesjTJD7leSl2EdjP9FdTIuXyOlJmW+tLP1nFS76BlkLDYkwb99hp6nviqA2kMPqonKhX5NEw8JmIKBA8EDuUzSpZNrFp81QaXaszmOriMO0CwO51DdunSt7L5CZyu0wSfhgjribpKLIzUKU0dGAryhdQKd8nilmhH6dsBFSOJQMyAU5xGmNAh3h8C9TYFq/t4bcpz4V6s9EBR1o/dc2lZmYNfxW48MH tNWdWZn0 1iVTEMDCmhWUiOcj+XrsKIT1swYZXOHjhcqv/Pk4vi8ahqf8bzEhJwwjj3gRT91d6ygyKF9Eac3489OKeR/a8gF9rRONQfuFvKNKuqpURBQ3h0jrQLgRuElMYzJ6Nd06LNaPS7RvRTmMTdFc0LugUqN7+NoOci+rjPIpGyQFZrngjwEZQ4PHsZO9aI3VI0KBaAqwzTsVsGKfz2xKZRqsvgFNVnpnwem6XbELnrWBas5jcy8W2rxdVorQF6GrzqA2e6l1hsl8RKVGxu/b5kC2/sCj88SrufcqYaMCuErBrlJpyQ22Jwv3RvsUbUmjUy97HNoiyotT621+rH2LO/VnIzoSG+O2rUlLyn2PPyUV1p/zjGKczTXBg0L9K5iGpNAc9VekxrQ9mZuVOG46+4zpe4s70E/EymbbKt3rQfo1Kp24unCJ2ewaFOiihaw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jul 28, 2023 at 4:02=E2=80=AFAM Muhammad Usama Anjum wrote: > > We are optimizing for more performance. Please find the change-set below > for easy review before next revision and post your comments: > > - Replace memcpy() with direct copy as it proving to be very expensive > - Don't check if PAGE_IS_FILE if no mask needs it as it is very > expensive to check per pte > - Add question in comment for discussion purpose > - Add fast path for exclusive WP for ptes > --- > fs/proc/task_mmu.c | 54 ++++++++++++++++++++++++++++++++++++---------- > 1 file changed, 43 insertions(+), 11 deletions(-) > > diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c > index 7e92c33635cab..879baf896ed0b 100644 > --- a/fs/proc/task_mmu.c > +++ b/fs/proc/task_mmu.c > @@ -1757,37 +1757,51 @@ static int pagemap_release(struct inode *inode, > struct file *file) > PAGE_IS_HUGE) > #define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPAS= YNC) > > +#define MASKS_OF_INTEREST(a) (a.category_inverted | a.category_mask | = \ > + a.category_anyof_mask | a.return_mask) > + > struct pagemap_scan_private { > struct pm_scan_arg arg; > + unsigned long masks_of_interest; > unsigned long cur_vma_category; > struct page_region *vec_buf, cur_buf; > unsigned long vec_buf_len, vec_buf_index, found_pages, end_addr; > struct page_region __user *vec_out; > }; > > -static unsigned long pagemap_page_category(struct vm_area_struct *vma, > +static unsigned long pagemap_page_category(struct pagemap_scan_private *= p, > + struct vm_area_struct *vma, > unsigned long addr, pte_t pte) > { > unsigned long categories =3D 0; > > if (pte_present(pte)) { > - struct page *page =3D vm_normal_page(vma, addr, pte); > + struct page *page; > > categories |=3D PAGE_IS_PRESENT; > if (!pte_uffd_wp(pte)) > categories |=3D PAGE_IS_WRITTEN; > - if (page && !PageAnon(page)) > - categories |=3D PAGE_IS_FILE; > + > + if (p->masks_of_interest & PAGE_IS_FILE) { > + page =3D vm_normal_page(vma, addr, pte); > + if (page && !PageAnon(page)) > + categories |=3D PAGE_IS_FILE; > + } > + > if (is_zero_pfn(pte_pfn(pte))) > categories |=3D PAGE_IS_PFNZERO; > } else if (is_swap_pte(pte)) { > - swp_entry_t swp =3D pte_to_swp_entry(pte); > + swp_entry_t swp; > > categories |=3D PAGE_IS_SWAPPED; > if (!pte_swp_uffd_wp_any(pte)) > categories |=3D PAGE_IS_WRITTEN; > - if (is_pfn_swap_entry(swp) && !PageAnon(pfn_swap_entry_to= _page(swp))) > - categories |=3D PAGE_IS_FILE; > + > + if (p->masks_of_interest & PAGE_IS_FILE) { > + swp =3D pte_to_swp_entry(pte); > + if (is_pfn_swap_entry(swp) && !PageAnon(pfn_swap_= entry_to_page(swp))) > + categories |=3D PAGE_IS_FILE; > + } > } > > return categories; > @@ -1957,9 +1971,7 @@ static bool pagemap_scan_push_range(unsigned long > categories, > if (p->vec_buf_index >=3D p->vec_buf_len) > return false; > > - memcpy(&p->vec_buf[p->vec_buf_index], cur_buf, > - sizeof(*p->vec_buf)); > - ++p->vec_buf_index; > + p->vec_buf[p->vec_buf_index++] =3D *cur_buf; > } > > cur_buf->start =3D addr; > @@ -2095,9 +2107,24 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, > unsigned long start, > return 0; > } > > + if (!p->vec_buf) { > + /* Fast path for performing exclusive WP */ > + for (addr =3D start; addr !=3D end; pte++, addr +=3D PAGE= _SIZE) { > + if (pte_uffd_wp(ptep_get(pte))) > + continue; > + make_uffd_wp_pte(vma, addr, pte); > + if (!flush) { > + start =3D addr; > + flush =3D true; > + } > + } > + ret =3D 0; > + goto flush_and_return; > + } > + > for (addr =3D start; addr !=3D end; pte++, addr +=3D PAGE_SIZE) { > unsigned long categories =3D p->cur_vma_category | > - pagemap_page_category(vma, add= r, ptep_get(pte)); > + pagemap_page_category(p, vma, = addr, ptep_get(pte)); > unsigned long next =3D addr + PAGE_SIZE; > > ret =3D pagemap_scan_output(categories, p, addr, &next); > @@ -2119,6 +2146,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, > unsigned long start, > } > } > > +flush_and_return: > if (flush) > flush_tlb_range(vma, start, addr); > > @@ -2284,6 +2312,9 @@ static int pagemap_scan_init_bounce_buffer(struct > pagemap_scan_private *p) > * consecutive ranges that have the same categories returned acro= ss > * walk_page_range() calls. > */ > + // Question: Increasing the vec_buf_len increases the execution s= peed. > + // But it'll increase the memory needed to run the IOCTL. Can we = do > something here? > + // Right now only have space for 512 entries of page_region The main problem here is that walk_page_range is executed for 512 pages. I think we need to execute it for the whole range and interrupt it when we need to drain the bounce buffer. For a trivial program that scans its address space the time is reduced from= 20 seconds to 0.001 seconds. The test program and perf data are here: https://gist.github.com/avagin/c5a22f3c78f8cb34281602dfe9c43d10 > p->vec_buf_len =3D min_t(size_t, PAGEMAP_WALK_SIZE >> PAGE_SHIFT, > p->arg.vec_len - 1); > p->vec_buf =3D kmalloc_array(p->vec_buf_len, sizeof(*p->vec_buf), > @@ -2329,6 +2360,7 @@ static long do_pagemap_scan(struct mm_struct *mm, > unsigned long uarg) > if (ret) > return ret; > > + p.masks_of_interest =3D MASKS_OF_INTEREST(p.arg); > ret =3D pagemap_scan_init_bounce_buffer(&p); > if (ret) > return ret; > -- > 2.39.2 >