From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 394D4EB64DA for ; Wed, 28 Jun 2023 13:43:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A14CD8D0002; Wed, 28 Jun 2023 09:43:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 99E198D0001; Wed, 28 Jun 2023 09:43:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 818738D0002; Wed, 28 Jun 2023 09:43:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 687E98D0001 for ; Wed, 28 Jun 2023 09:43:40 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2B56616082A for ; Wed, 28 Jun 2023 13:43:40 +0000 (UTC) X-FDA: 80952274200.15.6745111 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id D73A11C0027 for ; Wed, 28 Jun 2023 13:43:37 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Rdau3CLC; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687959817; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BR1e31KWtCzKshHk3i9sGJsYk/tQaHs2cFTTZ86spN4=; b=oA9Sdp0Tmeb3WNXwgjFJUAn9qzdLdrijj8moluw3aadZUIip/9OSGlRkxjmxD61UBo9Iv0 +SkzJ1YXTEbVBExVDpgLRzyS1NsHYVc1RMHiE9QICncZmzyzdYWMReYcZ6/+n02ZO4zO2G gCjQvouCOLoSqioGLrQRmJ+hgmp4XtM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Rdau3CLC; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687959818; a=rsa-sha256; cv=none; b=qlzRw6V+IfSd9ClzM6OQIb328/ZKoAfF8KRPCWHIlmDU+5m8vcpbWVFjYyuUfWk77LhOTc h9vhwruOxyTjsb0w1JfyOpwGIw/vai+jcUJPLOc9cNcnw5FN+B9YXO0VqwmZOdRH+iHykj cqAOj96M9UKerylqtazsSeIfUi7brbk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687959817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BR1e31KWtCzKshHk3i9sGJsYk/tQaHs2cFTTZ86spN4=; b=Rdau3CLCo129YFjIfmROC8eBB4x5O4P5c6oSuCMvUfOTpbr6TSDjIiHAmnD6tlvQvS8i/e O/2GPLmKQZw6msY6p0yqXYcm+XRyZAInFFFAvEaVjdFpGnhHpI/IED5lsNNjyAW+UTnaeX Fyd1CmNCFN+qLr3bW9xX1rzlNdggvBc= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-158-rMhhk-unMEWNT5lPRqPXnw-1; Wed, 28 Jun 2023 09:43:32 -0400 X-MC-Unique: rMhhk-unMEWNT5lPRqPXnw-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-402fa256023so2117811cf.0 for ; Wed, 28 Jun 2023 06:43:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687959812; x=1690551812; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BR1e31KWtCzKshHk3i9sGJsYk/tQaHs2cFTTZ86spN4=; b=jFhLYpEZJOiLyiK7iqaz05VZzzOqNGI6Gs6HD6igMd0WIF0h5ePCifq6/qODP8l/16 aZm9pDmP0hX4LIvcAvi9S/ZD1R8O2rPhmkkyLwI+OQMsZF/KfD9A8biNBQRPr3d5bxvV 4Q3Jw3NOIWM5VGr5EwL5eYdDZlLsenNA/fWxnPIMXlgZjQOja5oRsK40sGhiXdemJPaW AgVFdBPGdLAeny/N3ecv5T5rEDcvqnCjFKjplFaUscQ2cdAzC3L5HDYIDOKLxk6uKoy4 0XwjPSGPse47BMZjeNERf/WZMro5qTUSaigfdGEYxDGYEWmCG5DNtjhX+ou8IRyLJryF 6klw== X-Gm-Message-State: AC+VfDzWMQ+ozxRK4O9PeNhRpfibxfOdKYyZ3MbD94cDhUC6hfU8RVgs ZMWT08d1bL1JjS35Lff4Gti1HFCapUQrbLIs5jJgCYqSKlcNE/ttmQgnrkuuXA5rJoB4Jxmy52D Anj/MKIZx4mc= X-Received: by 2002:ac8:5713:0:b0:400:9bd5:3387 with SMTP id 19-20020ac85713000000b004009bd53387mr12057656qtw.0.1687959812241; Wed, 28 Jun 2023 06:43:32 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4GFumLvOCPj0F9b49es3pXW0L/ClPJqVb/4kHTZ+ZSwVk61r/fK/ZvKcbBwMMSNkp8sCzcFg== X-Received: by 2002:ac8:5713:0:b0:400:9bd5:3387 with SMTP id 19-20020ac85713000000b004009bd53387mr12057630qtw.0.1687959811891; Wed, 28 Jun 2023 06:43:31 -0700 (PDT) Received: from x1n (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fw15-20020a05622a4a8f00b00400a5ca26fesm4279836qtb.2.2023.06.28.06.43.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 06:43:31 -0700 (PDT) Date: Wed, 28 Jun 2023 09:43:29 -0400 From: Peter Xu To: Suren Baghdasaryan Cc: akpm@linux-foundation.org, willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, ying.huang@intel.com, david@redhat.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com Subject: Re: [PATCH v4 5/6] mm: handle swap page faults under per-VMA lock Message-ID: References: <20230628071800.544800-1-surenb@google.com> <20230628071800.544800-6-surenb@google.com> MIME-Version: 1.0 In-Reply-To: <20230628071800.544800-6-surenb@google.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Queue-Id: D73A11C0027 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: n1safizq9z31mpc1purimnohpozxkros X-HE-Tag: 1687959817-322248 X-HE-Meta: U2FsdGVkX19PAiIsU5DDfqmn+xARYesxoJWYcpP9f1Uf/nMdxKUSl55jH51wgFW3Jmb3UdIn2gna4ilECSNHEG0DgEDmwNpUIBDWQzCKyowyg+nMPrbwjEzJ2QIQaue2kyGKbd1LW94bvYAD2z5d/cGZQdIvR1WQVMlvrfAAhkF0liyvF2PFe/dlfnUXX9bQznnXqUlWseQs9TNlexGgar0P9YeDUnHHOW08z3FYGQ667q+Ph4037Igxwi1UQcozthpP5S738RyklX9nhuohWvpz+C4EOz1iHK6wDMlJZDxhJDlKZ09hRW/d042yVm4bAQ8Cdqnmf3aeGdVCIK/TNAKGtVcROS0kcgCqTbYEckB+XrY1egSHSMCm2HwrZLG+T9DLeGbQANq8WM1C5P6wYlrpWegsQTm868u9LU/sP+AG6Okp3s02hv89qun76sop9SwzZ0Qg1cH4vU/SNdUQQBmpaotLJy9xUqnu2PriUsNcJXrxe4MDvQJvSv8dgS1QSsL1vF+zOwZoXqf9T6UPqU1vOqNi4GPCJjFhCsXJE+ur3czGfI1X6Y/9K4NVupkxb6RSPAUC99aOzc0GwDdKPlDx9UdBxpwBTvj5cum+btc1J6+a6hhOvOtsqti4+cRTWNlSYIbd1ZQWywFsh9XlJT87aghRfQKOa5/c8Ha3HxNjA3p3NBGSK4f3UauZ8qeBSnZoWI8pPIA6RaCLz7cTXVSlNqfY1QK21aQgdTD6hF9vIodGJ1Rm2UVkDVRquPIpmIACy8F0XWGe9IEwZVapwXfHs75Eb2mNCLM0ItmDj2CBQrHbjYDj3FfSEsX77F+byI/IIcHobE9A1G4MjEBAuHp0yfY/pZ1dsVEa3A5Qa6W3/duV8F7vDoAGr/KXdnJeP8yCRx38MQDmyBtq3VU1hNUIFaiJRzR+qizG9fjzpEfbvkUyXouHLn5tdXRnRR0sCT5Ttq8j4Y14CqkM1OM JzZlzAlt nkDgJQhd6LD9Mtpzr6zX8KDeFKwrkcfd91vzbnUWy8/JSSz5z7Z+RSSULyz68DEZDRP6zcdtCeivaVyhWgfnKvUuuxyBpZNJnFcnXuvyyC00+34++3czSJufpn0p9xgev/Jm+O4+HQQMOM7HSvt35srL9WetoFMHMNcDHg/d3t7w/D0riSObpJplqQiHMME3KusaWFIuKTsTkSk6fq6BCHItoHRgcWms916wpv9xQ59RuhNdnsKK/KWDPFxVSbeaZkR1mQ4SQ8OnKawTXac0onj8F68qakBDihSDrF8evbopBcSRy5hNgAd8NSfXQ8xS8KjSSDWqmQWTieIf5QyFvwxGnkJU1gr0drz9oc73vYp+FBOzSEeBepPfQwjs84z30fVf4NFJFIuUMoXunmbLjQZQon+wigBJndYlB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 28, 2023 at 12:17:59AM -0700, Suren Baghdasaryan wrote: > When page fault is handled under per-VMA lock protection, all swap page > faults are retried with mmap_lock because folio_lock_or_retry has to drop > and reacquire mmap_lock if folio could not be immediately locked. > Follow the same pattern as mmap_lock to drop per-VMA lock when waiting > for folio and retrying once folio is available. > With this obstacle removed, enable do_swap_page to operate under > per-VMA lock protection. Drivers implementing ops->migrate_to_ram might > still rely on mmap_lock, therefore we have to fall back to mmap_lock in > that particular case. > Note that the only time do_swap_page calls synchronous swap_readpage > is when SWP_SYNCHRONOUS_IO is set, which is only set for > QUEUE_FLAG_SYNCHRONOUS devices: brd, zram and nvdimms (both btt and > pmem). Therefore we don't sleep in this path, and there's no need to > drop the mmap or per-VMA lock. > > Signed-off-by: Suren Baghdasaryan Acked-by: Peter Xu One nit below: > --- > mm/filemap.c | 25 ++++++++++++++++--------- > mm/memory.c | 16 ++++++++++------ > 2 files changed, 26 insertions(+), 15 deletions(-) > > diff --git a/mm/filemap.c b/mm/filemap.c > index 52bcf12dcdbf..7ee078e1a0d2 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -1699,31 +1699,38 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) > return ret; > } > > +static void release_fault_lock(struct vm_fault *vmf) > +{ > + if (vmf->flags & FAULT_FLAG_VMA_LOCK) > + vma_end_read(vmf->vma); > + else > + mmap_read_unlock(vmf->vma->vm_mm); > +} > + > /* > * Return values: > * 0 - folio is locked. > * VM_FAULT_RETRY - folio is not locked. > - * mmap_lock has been released (mmap_read_unlock(), unless flags had both > - * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in > - * which case mmap_lock is still held. > + * mmap_lock or per-VMA lock has been released (mmap_read_unlock() or > + * vma_end_read()), unless flags had both FAULT_FLAG_ALLOW_RETRY and > + * FAULT_FLAG_RETRY_NOWAIT set, in which case the lock is still held. > * > * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0 > - * with the folio locked and the mmap_lock unperturbed. > + * with the folio locked and the mmap_lock/per-VMA lock is left unperturbed. > */ > vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf) > { > - struct mm_struct *mm = vmf->vma->vm_mm; > unsigned int flags = vmf->flags; > > if (fault_flag_allow_retry_first(flags)) { > /* > - * CAUTION! In this case, mmap_lock is not released > - * even though return VM_FAULT_RETRY. > + * CAUTION! In this case, mmap_lock/per-VMA lock is not > + * released even though returning VM_FAULT_RETRY. > */ > if (flags & FAULT_FLAG_RETRY_NOWAIT) > return VM_FAULT_RETRY; > > - mmap_read_unlock(mm); > + release_fault_lock(vmf); > if (flags & FAULT_FLAG_KILLABLE) > folio_wait_locked_killable(folio); > else > @@ -1735,7 +1742,7 @@ vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf) > > ret = __folio_lock_killable(folio); > if (ret) { > - mmap_read_unlock(mm); > + release_fault_lock(vmf); > return VM_FAULT_RETRY; > } > } else { > diff --git a/mm/memory.c b/mm/memory.c > index 345080052003..76c7907e7286 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -3712,12 +3712,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > if (!pte_unmap_same(vmf)) > goto out; > > - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { > - ret = VM_FAULT_RETRY; > - vma_end_read(vma); > - goto out; > - } > - > entry = pte_to_swp_entry(vmf->orig_pte); > if (unlikely(non_swap_entry(entry))) { > if (is_migration_entry(entry)) { > @@ -3727,6 +3721,16 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > vmf->page = pfn_swap_entry_to_page(entry); > ret = remove_device_exclusive_entry(vmf); > } else if (is_device_private_entry(entry)) { > + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { > + /* > + * migrate_to_ram is not yet ready to operate > + * under VMA lock. > + */ > + vma_end_read(vma); > + ret |= VM_FAULT_RETRY; Here IIUC ret==0 is guaranteed, so maybe "ret = VM_FAULT_RETRY" is slightly clearer. > + goto out; > + } > + > vmf->page = pfn_swap_entry_to_page(entry); > vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, > vmf->address, &vmf->ptl); > -- > 2.41.0.162.gfafddb0af9-goog > -- Peter Xu