From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCEF0C433EF for ; Wed, 27 Oct 2021 17:50:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4C57E610A3 for ; Wed, 27 Oct 2021 17:50:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4C57E610A3 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7E55C6B0071; Wed, 27 Oct 2021 13:50:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 793F46B0072; Wed, 27 Oct 2021 13:50:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65C866B0073; Wed, 27 Oct 2021 13:50:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 400116B0071 for ; Wed, 27 Oct 2021 13:50:35 -0400 (EDT) Received: from forelay.prod.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by fograve02.hostedemail.com (Postfix) with ESMTP id 2375B3EEFA for ; Wed, 27 Oct 2021 17:42:51 +0000 (UTC) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D49A718549F34 for ; Wed, 27 Oct 2021 17:42:41 +0000 (UTC) X-FDA: 78742937322.03.77236C8 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) by imf03.hostedemail.com (Postfix) with ESMTP id 196DC3003987 for ; Wed, 27 Oct 2021 17:42:36 +0000 (UTC) Received: by mail-yb1-f179.google.com with SMTP id m63so8273498ybf.7 for ; Wed, 27 Oct 2021 10:42:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=kBCZKPDNQAHmIZq7XduSwhYmuOetWWPl476PrH0YNN0=; b=c1yWnQHxZHmMmNhoUfzo1TMFW/D1F1/UZAPStMR8kK2shT1bIlw4Vz87gK7e2W7C5d qZyapuNaU8LqKQ7yqG6aDFRV+/6fDIgicnSOt3MVoQv473UjZ4sNsN4GIxWCZwq8cvz/ aXmvuWfYbWB3uoKmZf1L4BOJHgIU1r0S7jpDyMini7wC2QsiwWXh8jlTuABGzgvarTKU /PrRXTDJJn3cYwR7jGGdjH6OTWA1VuF5N24mb12ryhSfQelkF94CPVK+/s519yEvWEjG Yhp5wZgFsroCt/Ds03/trmCuPE988KKx9V74jYCqtKI/RerFz/T6u7PNkcYkzyvbUlou AfYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=kBCZKPDNQAHmIZq7XduSwhYmuOetWWPl476PrH0YNN0=; b=heZzRdTiAQ9cPsw4rzwu5+URcJZN1mKan6Y7phuj6JM3JcAQL/o7WQaPWxRR459bQp sFIORVoX6Jv42l5IVPghyHuR3qKt8fiXB88aiXZNvTi8F04pkzYQttX5X/1BSm8rFJLL r/qC5H9sGch2Jjd6zls+CKxsH6tuz7zhkvGkdAotj4DZXDveGBgerXYMxy/dkPoI5O+u kOXU97bqrDW4sSH5hvFsU+duhYyxRrTnzu9WR5dHcCGn/CdgidrS8eCp1uE/pmzkzHKA etmrKoPwt3s7KZANh6QJHZrJGO/iOXNxMkzhNsVH9AHs/ykRG5FxKGzaYA1ipwVEkrf2 32eA== X-Gm-Message-State: AOAM532Cd1FeW2tOWlcqOLZu/kC6B1NXuQKiZOUYkff4gZrQWonnDfBT lzwbUHxIdbWbkTa7OmbzypaQ/iM0XDVj5JzgVN8fUg== X-Google-Smtp-Source: ABdhPJyzyQR0M+no29qupLGsoz3t3Qt4vFELy82tvV2wN2FrbWSg0TFPJJgUZMaYPBh3o4cyCdgrlqUnK/2uL8vDe4c= X-Received: by 2002:a25:30b:: with SMTP id 11mr33445740ybd.202.1635356560450; Wed, 27 Oct 2021 10:42:40 -0700 (PDT) MIME-Version: 1.0 References: <20211022014658.263508-1-surenb@google.com> In-Reply-To: From: Suren Baghdasaryan Date: Wed, 27 Oct 2021 10:42:29 -0700 Message-ID: Subject: Re: [PATCH 1/1] mm: prevent a race between process_mrelease and exit_mmap To: Matthew Wilcox Cc: "Liam R. Howlett" , Michal Hocko , Andrew Morton , David Rientjes , Johannes Weiner , Roman Gushchin , Rik van Riel , Minchan Kim , Christian Brauner , Christoph Hellwig , Oleg Nesterov , David Hildenbrand , Jann Horn , Shakeel Butt , Andy Lutomirski , Christian Brauner , Florian Weimer , Jan Engelhardt , Linux API , linux-mm , LKML , kernel-team , "Kirill A. Shutemov" , Andrea Arcangeli Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 196DC3003987 X-Stat-Signature: gsu6ad3ewnqup75b5q5quiutwym58q5m Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=c1yWnQHx; spf=pass (imf03.hostedemail.com: domain of surenb@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1635356556-57510 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 27, 2021 at 10:35 AM Matthew Wilcox wrote: > > On Wed, Oct 27, 2021 at 09:08:21AM -0700, Suren Baghdasaryan wrote: > > Unconditional mmap_write_lock around free_pgtables in exit_mmap seems > > to me the most semantically correct way forward and the pushback is on > > the basis of regressing performance of the exit path. I would like to > > measure that regression to confirm this. I don't have access to a big > > machine but will ask someone in another Google team to try the test > > Michal wrote here > > https://lore.kernel.org/all/20170725142626.GJ26723@dhcp22.suse.cz/ on > > a server with and without a custom patch. > > Sorry to hijack this, but could you ask that team to also test this > patch? I think there's probably a good-sized win here, but I have no > profiles to share at this point. I've only done light testing, and > it may have bugs. > > NB: I only did the exit() path here. fork() conversion is left as an > exercise for the reader^W^W Liam. To clarify, this patch does not change the mmap_write_lock portion of exit_mmap. Do you want to test it in isolation or with the locking changes in exit_mmap I mentioned? > > From 5f9daa14a5e58c86a73eccf59abe23d131004926 Mon Sep 17 00:00:00 2001 > From: "Matthew Wilcox (Oracle)" > Date: Wed, 27 Oct 2021 13:28:35 -0400 > Subject: [PATCH] mm: Add vmavec > > The vmavec lets us allocate and free batches of VMAs instead of > one at a time. Should improve fork() and exit() performance. > > Signed-off-by: Matthew Wilcox (Oracle) > --- > include/linux/vmavec.h | 38 ++++++++++++++++++++++++++++++++++++++ > kernel/fork.c | 17 +++++++++++++++++ > mm/mmap.c | 30 ++++++++++++++++++++++-------- > 3 files changed, 77 insertions(+), 8 deletions(-) > create mode 100644 include/linux/vmavec.h > > diff --git a/include/linux/vmavec.h b/include/linux/vmavec.h > new file mode 100644 > index 000000000000..8a324e2e1258 > --- /dev/null > +++ b/include/linux/vmavec.h > @@ -0,0 +1,38 @@ > +/* > + * A vma vector is an array of vm_area_structs, with a counter. > + */ > + > +struct vm_area_struct; > + > +#define VMAVEC_SIZE 15 > + > +struct vmavec { > + unsigned char nr; > + void *vmas[VMAVEC_SIZE]; > +}; > + > +#define VMAVEC(name) struct vmavec name = { } > + > +static inline bool vmavec_full(struct vmavec *vmavec) > +{ > + return vmavec->nr == VMAVEC_SIZE; > +} > + > +static inline bool vmavec_empty(struct vmavec *vmavec) > +{ > + return vmavec->nr == 0; > +} > + > +static inline > +void vmavec_push(struct vmavec *vmavec, struct vm_area_struct *vma) > +{ > + vmavec->vmas[vmavec->nr++] = vma; > +} > + > +static inline struct vm_area_struct *vmavec_pop(struct vmavec *vmavec) > +{ > + return vmavec->vmas[--vmavec->nr]; > +} > + > +void vm_area_free_vec(struct vmavec *vmavec); > +void vm_area_alloc_vec(struct mm_struct *mm, struct vmavec *vmavec); > diff --git a/kernel/fork.c b/kernel/fork.c > index 38681ad44c76..ea7e8bd00be8 100644 > --- a/kernel/fork.c > +++ b/kernel/fork.c > @@ -97,6 +97,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -375,6 +376,22 @@ void vm_area_free(struct vm_area_struct *vma) > kmem_cache_free(vm_area_cachep, vma); > } > > +void vm_area_alloc_vec(struct mm_struct *mm, struct vmavec *vmavec) > +{ > + int i; > + > + vmavec->nr = kmem_cache_alloc_bulk(vm_area_cachep, GFP_KERNEL, > + VMAVEC_SIZE, vmavec->vmas); > + for (i = 0; i < vmavec->nr; i++) > + vma_init(vmavec->vmas[i], mm); > +} > + > +void vm_area_free_vec(struct vmavec *vmavec) > +{ > + kmem_cache_free_bulk(vm_area_cachep, vmavec->nr, vmavec->vmas); > + vmavec->nr = 0; > +} > + > static void account_kernel_stack(struct task_struct *tsk, int account) > { > void *stack = task_stack_page(tsk); > diff --git a/mm/mmap.c b/mm/mmap.c > index 88dcc5c25225..bff4e94eec8c 100644 > --- a/mm/mmap.c > +++ b/mm/mmap.c > @@ -47,6 +47,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -172,19 +173,24 @@ void unlink_file_vma(struct vm_area_struct *vma) > } > } > > -/* > - * Close a vm structure and free it, returning the next. > - */ > -static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) > +static void __remove_vma(struct vm_area_struct *vma) > { > - struct vm_area_struct *next = vma->vm_next; > - > might_sleep(); > if (vma->vm_ops && vma->vm_ops->close) > vma->vm_ops->close(vma); > if (vma->vm_file) > fput(vma->vm_file); > mpol_put(vma_policy(vma)); > +} > + > +/* > + * Close a vm structure and free it, returning the next. > + */ > +static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) > +{ > + struct vm_area_struct *next = vma->vm_next; > + > + __remove_vma(vma); > vm_area_free(vma); > return next; > } > @@ -3125,6 +3131,7 @@ void exit_mmap(struct mm_struct *mm) > { > struct mmu_gather tlb; > struct vm_area_struct *vma; > + VMAVEC(vmavec); > unsigned long nr_accounted = 0; > > /* mm's last user has gone, and its about to be pulled down */ > @@ -3179,9 +3186,16 @@ void exit_mmap(struct mm_struct *mm) > while (vma) { > if (vma->vm_flags & VM_ACCOUNT) > nr_accounted += vma_pages(vma); > - vma = remove_vma(vma); > - cond_resched(); > + __remove_vma(vma); > + vmavec_push(&vmavec, vma); > + vma = vma->vm_next; > + if (vmavec_full(&vmavec)) { > + vm_area_free_vec(&vmavec); > + cond_resched(); > + } > } > + if (!vmavec_empty(&vmavec)) > + vm_area_free_vec(&vmavec); > vm_unacct_memory(nr_accounted); > } > > -- > 2.33.0 >