From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B5D7C433EF for ; Mon, 20 Jun 2022 12:24:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E34BF6B0073; Mon, 20 Jun 2022 08:24:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBDA08E0001; Mon, 20 Jun 2022 08:24:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C37546B0075; Mon, 20 Jun 2022 08:24:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B1D406B0073 for ; Mon, 20 Jun 2022 08:24:27 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 8A14D60D6A for ; Mon, 20 Jun 2022 12:24:27 +0000 (UTC) X-FDA: 79598532174.06.D9855A6 Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com [209.85.219.173]) by imf20.hostedemail.com (Postfix) with ESMTP id 1B2591C00A7 for ; Mon, 20 Jun 2022 12:24:26 +0000 (UTC) Received: by mail-yb1-f173.google.com with SMTP id v81so18708604ybe.0 for ; Mon, 20 Jun 2022 05:24:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/5JxyhKk/uMc2fQhNdOt2zyaowNO+vOl5nRe37m7DAA=; b=PE6NpoyZLTwUcLsqzDbtJnrnxAIyaVdtaXAicCknBnBwh8ZdqPc+LdCbBV9G0aYWTp OH08dpop/cUwHHDAwRoOZn8NuJbm7em0mCvXsUn5KrkE2uLBzx4qEpmZlCmJ1JT+Dfo2 L2g+1N68ykc71e7QB9eNTR6L/SfFYKD49pmyUm+IL+kgwtlzsAFOFxYmtVjO901MBCa0 FXrtKRIwLIRyoFGhr+u6OUEnyG6aXt9C5BbkXb3gIq/00R90GPJyncy94IdAi1QYzzKR vgiDWLdFFfvrAAZFOdl0BFPX5q6CTFy21BUmyoGXj99JPg6vLckZ2H/SPEJs5BrMF/te qm2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/5JxyhKk/uMc2fQhNdOt2zyaowNO+vOl5nRe37m7DAA=; b=uvWlEXIN2X9iMkUt7b/c+QDoFtgjXWldaU3a7NHS1PgpOTX1RcB4ffVtXL4dZ0weIq CcxYDJqacqEFj8cTn+XFvVCUrABWffNKT1W9GIoqo9kSYfR+QAsIdhDGJkXgc9x7wIn2 sv7Io9JO/hGz3bxCNFCZabyfNYDXwm9FgE14WS2pbiVcuyZtHAqhbpltsnDF/zf0qE9z 63Mtka3NUeGboY5Uilor1wIgIwNBiXAOrsrLH0FrD8EV/brLIj9I+cq7BLCIo+Hq20T4 SJmQRUVscDYCFNXwpoYHUTJjPaT7/WzQ6ZtaelRwRi+/yLx70ZhG4S4LZH/0rRURIbkZ eD2A== X-Gm-Message-State: AJIora8FQoXGk4y8ZSb/uKCkd68acxH7pabgnYAr5L7Ji4mb8I7OF8kn nNlE6l5D76M30EYjpaqUW0Dqqkw0IZRXR3M71iQ= X-Google-Smtp-Source: AGRyM1vMxS63SWLYtulgVh/kvFZnhALPPWNXYa9hXtgHK5dVSCko6xQfAiztdTyivXeoAx7i5Qb7kZtG3NdmkeicRKE= X-Received: by 2002:a25:c08f:0:b0:668:fcfc:1b2a with SMTP id c137-20020a25c08f000000b00668fcfc1b2amr6724569ybf.242.1655727866015; Mon, 20 Jun 2022 05:24:26 -0700 (PDT) MIME-Version: 1.0 References: <20220531200041.24904-1-alex.sierra@amd.com> <20220531200041.24904-2-alex.sierra@amd.com> <3ac89358-2ce0-7d0d-8b9c-8b0e5cc48945@redhat.com> <02ed2cb7-3ad3-8ffc-6032-04ae1853e234@amd.com> <87bkuo898d.fsf@nvdebian.thelocal> <877d5b90sp.fsf@nvdebian.thelocal> In-Reply-To: <877d5b90sp.fsf@nvdebian.thelocal> From: Oded Gabbay Date: Mon, 20 Jun 2022 15:23:59 +0300 Message-ID: Subject: Re: [PATCH v5 01/13] mm: add zone device coherent type memory support To: Alistair Popple Cc: "Sierra Guiza, Alejandro (Alex)" , David Hildenbrand , Jason Gunthorpe , rcampbell@nvidia.com, Matthew Wilcox , "Kuehling, Felix" , amd-gfx list , linux-xfs@vger.kernel.org, linux-mm , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Maling list - DRI developers , Andrew Morton , linux-ext4@vger.kernel.org, Christoph Hellwig Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655727867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/5JxyhKk/uMc2fQhNdOt2zyaowNO+vOl5nRe37m7DAA=; b=4O4GlR0x3uJebhTXjB3QGeFroYiAWP7AgfZjE0y3+Ed6XAqiWD7++LBmnPStB1+j5kTxzB hHSTJRjjB1R86DMbpOfIWHILdobYp4q+D2MUu2MeSdK5D95NX7+rzjU9rVu9YJNqp3c3iv NJysk6BY25/sKcGdWKrBF1CacPtg8KM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655727867; a=rsa-sha256; cv=none; b=FF8fuwhyqTcIWdZwxxn5alJiNy04wmXeMpx+/lf8sErtEUwKxJc87/cRVUn6fEscOW2QbB IjuxlZz0AXLYkDCqueBXgVF2XIq/FtLhMK235XHEu8+oRLFV6igfVdtraoYEVun2W0G0Br T3d2fpemE/owCiUYpYfgB0nRr+kPSeo= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PE6NpoyZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of oded.gabbay@gmail.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=oded.gabbay@gmail.com Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PE6NpoyZ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of oded.gabbay@gmail.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=oded.gabbay@gmail.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: dx73wkyf3x3xx3u5zxb366gg4dwocku5 X-Rspamd-Queue-Id: 1B2591C00A7 X-HE-Tag: 1655727866-766957 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 20, 2022 at 11:50 AM Alistair Popple wrote: > > > Oded Gabbay writes: > > > On Mon, Jun 20, 2022 at 3:33 AM Alistair Popple wrote: > >> > >> > >> Oded Gabbay writes: > >> > >> > On Fri, Jun 17, 2022 at 8:20 PM Sierra Guiza, Alejandro (Alex) > >> > wrote: > >> >> > >> >> > >> >> On 6/17/2022 4:40 AM, David Hildenbrand wrote: > >> >> > On 31.05.22 22:00, Alex Sierra wrote: > >> >> >> Device memory that is cache coherent from device and CPU point of view. > >> >> >> This is used on platforms that have an advanced system bus (like CAPI > >> >> >> or CXL). Any page of a process can be migrated to such memory. However, > >> >> >> no one should be allowed to pin such memory so that it can always be > >> >> >> evicted. > >> >> >> > >> >> >> Signed-off-by: Alex Sierra > >> >> >> Acked-by: Felix Kuehling > >> >> >> Reviewed-by: Alistair Popple > >> >> >> [hch: rebased ontop of the refcount changes, > >> >> >> removed is_dev_private_or_coherent_page] > >> >> >> Signed-off-by: Christoph Hellwig > >> >> >> --- > >> >> >> include/linux/memremap.h | 19 +++++++++++++++++++ > >> >> >> mm/memcontrol.c | 7 ++++--- > >> >> >> mm/memory-failure.c | 8 ++++++-- > >> >> >> mm/memremap.c | 10 ++++++++++ > >> >> >> mm/migrate_device.c | 16 +++++++--------- > >> >> >> mm/rmap.c | 5 +++-- > >> >> >> 6 files changed, 49 insertions(+), 16 deletions(-) > >> >> >> > >> >> >> diff --git a/include/linux/memremap.h b/include/linux/memremap.h > >> >> >> index 8af304f6b504..9f752ebed613 100644 > >> >> >> --- a/include/linux/memremap.h > >> >> >> +++ b/include/linux/memremap.h > >> >> >> @@ -41,6 +41,13 @@ struct vmem_altmap { > >> >> >> * A more complete discussion of unaddressable memory may be found in > >> >> >> * include/linux/hmm.h and Documentation/vm/hmm.rst. > >> >> >> * > >> >> >> + * MEMORY_DEVICE_COHERENT: > >> >> >> + * Device memory that is cache coherent from device and CPU point of view. This > >> >> >> + * is used on platforms that have an advanced system bus (like CAPI or CXL). A > >> >> >> + * driver can hotplug the device memory using ZONE_DEVICE and with that memory > >> >> >> + * type. Any page of a process can be migrated to such memory. However no one > >> >> > Any page might not be right, I'm pretty sure. ... just thinking about special pages > >> >> > like vdso, shared zeropage, ... pinned pages ... > >> >> > >> >> Hi David, > >> >> > >> >> Yes, I think you're right. This type does not cover all special pages. > >> >> I need to correct that on the cover letter. > >> >> Pinned pages are allowed as long as they're not long term pinned. > >> >> > >> >> Regards, > >> >> Alex Sierra > >> > > >> > What if I want to hotplug this device's coherent memory, but I do > >> > *not* want the OS > >> > to migrate any page to it ? > >> > I want to fully-control what resides on this memory, as I can consider > >> > this memory > >> > "expensive". i.e. I don't have a lot of it, I want to use it for > >> > specific purposes and > >> > I don't want the OS to start using it when there is some memory pressure in > >> > the system. > >> > >> This is exactly what MEMORY_DEVICE_COHERENT is for. Device coherent > >> pages are only allocated by a device driver and exposed to user-space by > >> a driver migrating pages to them with migrate_vma. The OS can't just > >> start using them due to memory pressure for example. > >> > >> - Alistair > > Thanks for the explanation. > > > > I guess the commit message confused me a bit, especially these two sentences: > > > > "Any page of a process can be migrated to such memory. However no one should be > > allowed to pin such memory so that it can always be evicted." > > > > I read them as if the OS is free to choose which pages are migrated to > > this memory, > > and anything is eligible for migration to that memory (and that's why > > we also don't > > allow it to pin memory there). > > > > If we are not allowed to pin anything there, can the device driver > > decide to disable > > any option for oversubscription of this memory area ? > > I'm not sure I follow your thinking on how oversubscription would work > here, however all allocations are controlled by the driver. So if a > device's coherent memory is full a driver would be unable to migrate > pages to that device until pages are freed by the OS due to being > unmapped or the driver evicts pages by migrating them back to normal CPU > memory. > > Pinning of pages is allowed, and could prevent such migrations. However > this patch series prevents device coherent pages from being pinned > longterm (ie. with FOLL_LONGTERM), so it should always be able to evict > pages eventually. > > > Let's assume the user uses this memory area for doing p2p with other > > CXL devices. > > In that case, I wouldn't want the driver/OS to migrate pages in and > > out of that memory... > > The OS will not migrate pages in or out (although it may free them if no > longer required), but a driver might choose to. So at the moment it's > really up to the driver to implement what you want in this regards. I see. In other words, we don't want to allow long-term pinning but the driver can decide it doesn't want to evict pages out of that memory, until they are freed. Thanks, Oded > > > So either I should let the user pin those pages, or prevent him from > > doing (accidently or not) > > oversubscription in this memory area. > > As noted above pages can be pinned, but not long-term. > > - Alistair > > > wdyt ? > > > >> > >> > Oded > >> > > >> >> > >> >> > > >> >> >> + * should be allowed to pin such memory so that it can always be evicted. > >> >> >> + * > >> >> >> * MEMORY_DEVICE_FS_DAX: > >> >> >> * Host memory that has similar access semantics as System RAM i.e. DMA > >> >> >> * coherent and supports page pinning. In support of coordinating page > >> >> >> @@ -61,6 +68,7 @@ struct vmem_altmap { > >> >> >> enum memory_type { > >> >> >> /* 0 is reserved to catch uninitialized type fields */ > >> >> >> MEMORY_DEVICE_PRIVATE = 1, > >> >> >> + MEMORY_DEVICE_COHERENT, > >> >> >> MEMORY_DEVICE_FS_DAX, > >> >> >> MEMORY_DEVICE_GENERIC, > >> >> >> MEMORY_DEVICE_PCI_P2PDMA, > >> >> >> @@ -143,6 +151,17 @@ static inline bool folio_is_device_private(const struct folio *folio) > >> >> > In general, this LGTM, and it should be correct with PageAnonExclusive I think. > >> >> > > >> >> > > >> >> > However, where exactly is pinning forbidden? > >> >> > >> >> Long-term pinning is forbidden since it would interfere with the device > >> >> memory manager owning the > >> >> device-coherent pages (e.g. evictions in TTM). However, normal pinning > >> >> is allowed on this device type. > >> >> > >> >> Regards, > >> >> Alex Sierra > >> >> > >> >> >