From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DD88C433F5 for ; Thu, 6 Jan 2022 21:05:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0682B6B0072; Thu, 6 Jan 2022 16:05:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F32536B0073; Thu, 6 Jan 2022 16:05:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAB836B0074; Thu, 6 Jan 2022 16:05:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id C578E6B0072 for ; Thu, 6 Jan 2022 16:05:32 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 874E5181FE083 for ; Thu, 6 Jan 2022 21:05:32 +0000 (UTC) X-FDA: 79001093304.04.566338B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 03CF010000E for ; Thu, 6 Jan 2022 21:05:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1641503131; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bfNEPHx1xDfNOOR4ZfuMpjosG1jEJyn3a9yDJ8UdTKY=; b=ZLKlFr4C/HWFUXDCwBLknBS18TzBJ+XskL5d1Hnof86EIpIEekJe3iETWUWYExBBJa/ij8 +GjsmuBJiEPKvOmpR3rBCWf9Ldd9bpFWSRP5b0T/uvz4M0itp66FBberIwK7o1ZBDQvAn5 qaaG5lSjzI8I4ns2NNmRjzuMa7XXSb4= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-653-pwYAO2JxMoyDPEVx387Jmw-1; Thu, 06 Jan 2022 16:05:30 -0500 X-MC-Unique: pwYAO2JxMoyDPEVx387Jmw-1 Received: by mail-oo1-f70.google.com with SMTP id b26-20020a4ac29a000000b002dac1c5b232so2235199ooq.2 for ; Thu, 06 Jan 2022 13:05:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bfNEPHx1xDfNOOR4ZfuMpjosG1jEJyn3a9yDJ8UdTKY=; b=kvz9NM/1d0/XdTFM19muSVHDamus/gQ/X15+XGSiJtxHVxIuGNEWgpckUeCs0rqvBi Vu3zFlMOZrHm24MH4RIsYTw0ZkCITYs4L8jl0dfWRsHqoKeRqatZcGxVm5rGmwn9Yhw+ G6DpMG3zkTdWKd8PLcJRYWem/c8v4xnJIHx+3oFoe0cKHy36x7lDINnrSxmaW6UrytmJ /SyRkl6X5Cm5ZNmLK6PWQKoK/XGbgqglrhD4VE4fHEzScE9aM4j94PHdCkokhHC17I9W VuXNaAH/77+U5ypvqvFFeWjQ35lAiYLHf27B1tvGneIXVPXwyKHS7N5Mwgq1Hu+3rVjz 9HjQ== X-Gm-Message-State: AOAM532U7NFaUM2CQ+SxHaxUaBMlMHxfQ77kLG+zprXYfIXwBh8mHABD xwGWlTTJc1t+fL9g+fHfIRyyixlJSz8D8hV6WAed53xRDCuf+/rT9N6ZL5UHGlcP9xcy/XVM7gy WY1A5nvZaHjU= X-Received: by 2002:a4a:dd08:: with SMTP id m8mr37690431oou.25.1641503129795; Thu, 06 Jan 2022 13:05:29 -0800 (PST) X-Google-Smtp-Source: ABdhPJxJJHuHyvhX6fNO7GsfowW/AaCDltzBsOD9eqaeIHK+3O5wnNGjVkNDhjuWIf6dAb0cTtDPNA== X-Received: by 2002:a4a:dd08:: with SMTP id m8mr37690401oou.25.1641503129553; Thu, 06 Jan 2022 13:05:29 -0800 (PST) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id r23sm615480oiw.20.2022.01.06.13.05.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jan 2022 13:05:29 -0800 (PST) Date: Thu, 6 Jan 2022 14:05:27 -0700 From: Alex Williamson To: Jason Gunthorpe Cc: Daniel Jordan , Alexander Duyck , Andrew Morton , Ben Segall , Cornelia Huck , Dan Williams , Dave Hansen , Dietmar Eggemann , Herbert Xu , Ingo Molnar , Johannes Weiner , Josh Triplett , Michal Hocko , Nico Pache , Pasha Tatashin , Peter Zijlstra , Steffen Klassert , Steve Sistare , Tejun Heo , Tim Chen , Vincent Guittot , linux-mm@kvack.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org Subject: Re: [RFC 08/16] vfio/type1: Cache locked_vm to ease mmap_lock contention Message-ID: <20220106140527.5c292d34.alex.williamson@redhat.com> In-Reply-To: <20220106123456.GZ2328285@nvidia.com> References: <20220106004656.126790-1-daniel.m.jordan@oracle.com> <20220106004656.126790-9-daniel.m.jordan@oracle.com> <20220106005339.GX2328285@nvidia.com> <20220106011708.6ajbhzgreevu62gl@oracle.com> <20220106123456.GZ2328285@nvidia.com> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZLKlFr4C; spf=none (imf05.hostedemail.com: domain of alex.williamson@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=alex.williamson@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 03CF010000E X-Stat-Signature: cmqfjshgypw6yegpzw6fbk8dcoee37ey X-HE-Tag: 1641503131-916154 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 6 Jan 2022 08:34:56 -0400 Jason Gunthorpe wrote: > On Wed, Jan 05, 2022 at 08:17:08PM -0500, Daniel Jordan wrote: > > On Wed, Jan 05, 2022 at 08:53:39PM -0400, Jason Gunthorpe wrote: > > > On Wed, Jan 05, 2022 at 07:46:48PM -0500, Daniel Jordan wrote: > > > > padata threads hold mmap_lock as reader for the majority of their > > > > runtime in order to call pin_user_pages_remote(), but they also > > > > periodically take mmap_lock as writer for short periods to adjust > > > > mm->locked_vm, hurting parallelism. > > > > > > > > Alleviate the write-side contention with a per-thread cache of locked_vm > > > > which allows taking mmap_lock as writer far less frequently. > > > > > > > > Failure to refill the cache due to insufficient locked_vm will not cause > > > > the entire pinning operation to error out. This avoids spurious failure > > > > in case some pinned pages aren't accounted to locked_vm. > > > > > > > > Cache size is limited to provide some protection in the unlikely event > > > > of a concurrent locked_vm accounting operation in the same address space > > > > needlessly failing in case the cache takes more locked_vm than it needs. > > > > > > Why not just do the pinned page accounting once at the start? Why does > > > it have to be done incrementally? > > > > Yeah, good question. I tried doing it that way recently and it did > > improve performance a bit, but I thought it wasn't enough of a gain to > > justify how it overaccounted by the size of the entire pin. > > Why would it over account? We'd be guessing that the entire virtual address mapping counts against locked memory limits, but it might include PFNMAP pages or pages that are already account via the page pinning interface that mdev devices use. At that point we're risking that the user isn't concurrently doing something else that could fail as a result of pre-accounting and fixup later schemes like this. Thanks, Alex