From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC5DDCDB474 for ; Tue, 17 Oct 2023 19:00:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 670558003F; Tue, 17 Oct 2023 15:00:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61F4680009; Tue, 17 Oct 2023 15:00:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E6B08003F; Tue, 17 Oct 2023 15:00:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3F51A80009 for ; Tue, 17 Oct 2023 15:00:14 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DE8B980D46 for ; Tue, 17 Oct 2023 19:00:13 +0000 (UTC) X-FDA: 81355868706.22.0FE4658 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id ACC9B40014 for ; Tue, 17 Oct 2023 19:00:11 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IEH9xjJQ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697569211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ufWNltmtm9MEQJk/HA+zcNGOIx26ETEdVffppRZYG34=; b=P2gt0fe6sKK5DJC0w8sqNTT+XulPYaShzSioSR4UrQlez/niLxmEzLVyL3R+jgIgLX2AKM 34m+OmdoWGDgLyEFT1003CsbOfhsYYwZiOb7eBdYP9QIFnHliOxVbtboPVfWaDsjaqxC5q T5Mp0iBlvchSmsQw9sWVXiwoFeKYoA0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IEH9xjJQ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697569211; a=rsa-sha256; cv=none; b=QhSCYLidOZ26WU8pQOfTsIyYlX4+ZIFK+6aLGKgqi/FJqQiqPlJWkwKXsnQUVqHJ3Xt6bE pWJdwxRKcpFBuvCJ0Y/toI+uwKDoD+GBHb2qdz/3auI9M6KU+SgoI041yCEmjX/GCP6BAl IY2aUPWMq5lEs9PJVWs/bUd1/IMtSOg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1697569210; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ufWNltmtm9MEQJk/HA+zcNGOIx26ETEdVffppRZYG34=; b=IEH9xjJQ/RnsWjTybSQFQRXOPDBbDcYn8jFTWu/e6Bv8tYzcnZZhXSSWqmqhC2FJB180tV mN4Z/rRRIJGFSNFJzD1HDYlbLJx9NMUQgT3LUO63hEYKDwFsF1cggsa+yGEbVduedKZ3nG meSjraKs7/Mfbfa9cLjSNhcw0qrdNmc= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-339-fgNweuslM6GVbAP5z-AgKg-1; Tue, 17 Oct 2023 14:59:59 -0400 X-MC-Unique: fgNweuslM6GVbAP5z-AgKg-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-65d0ea9d271so11894076d6.1 for ; Tue, 17 Oct 2023 11:59:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697569199; x=1698173999; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ufWNltmtm9MEQJk/HA+zcNGOIx26ETEdVffppRZYG34=; b=hAwwMA18iKjM4koSCtKURAo07RnnSm6JCg+KvkqC/ypVBvho8p8L88ZXQVvz9+19H2 aufA9bYjjXtRQSxw/cvkICdInJrJAnzvRJFJMI8Ir8GKBNSVz5X3aIwmaTeLbGGShEQc qogG+wXrDkEGOsId621e3Dp3+oBcatIi+jJJl06F7lTJ2185jIq3Gnetu3LuYPoS3ZJz spJktgaOVwORohrNXJPI517Dz3S+wEvnXIpfkVxOqB7y0P3ONqB55O1xkrrmV09d2nUm yS6ZkKlV7B+Fsn7D5dGT0w+awHhRa8P+yi510BDlAXa5xyWTw8uHG7A6E9EGvLrY8VG1 zjXw== X-Gm-Message-State: AOJu0YxHFznsI+BLgKDSBxdYr5RMOi88DgUX5Z5aAaJfGJsHBBsb5Ic2 FG9bJkSaqIrwWnWUJDCHER/7efOI4IbPDlvdrGwhSKonZaW4x+/X3+ll0aY5kH6yxofVOZSCj8N iTkJEP9mSuQA= X-Received: by 2002:a05:6214:5b04:b0:668:e12a:1d1c with SMTP id ma4-20020a0562145b0400b00668e12a1d1cmr3335078qvb.3.1697569198756; Tue, 17 Oct 2023 11:59:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEAk0RrwzkoW0kvuWM0ng2wrDDvK/L2bekMKE0E2CdX9MEwa+UJ0DieBoe9T3ZhdsG80o5WLA== X-Received: by 2002:a05:6214:5b04:b0:668:e12a:1d1c with SMTP id ma4-20020a0562145b0400b00668e12a1d1cmr3335060qvb.3.1697569198391; Tue, 17 Oct 2023 11:59:58 -0700 (PDT) Received: from x1n (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id pz2-20020ad45502000000b00655e428604esm768163qvb.137.2023.10.17.11.59.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 11:59:58 -0700 (PDT) Date: Tue, 17 Oct 2023 14:59:55 -0400 From: Peter Xu To: David Hildenbrand Cc: Lokesh Gidra , Suren Baghdasaryan , akpm@linux-foundation.org, viro@zeniv.linux.org.uk, brauner@kernel.org, shuah@kernel.org, aarcange@redhat.com, hughd@google.com, mhocko@suse.com, axelrasmussen@google.com, rppt@kernel.org, willy@infradead.org, Liam.Howlett@oracle.com, jannh@google.com, zhangpeng362@huawei.com, bgeffon@google.com, kaleshsingh@google.com, ngeoffray@google.com, jdduke@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@android.com Subject: Re: [PATCH v3 2/3] userfaultfd: UFFDIO_MOVE uABI Message-ID: References: <214b78ed-3842-5ba1-fa9c-9fa719fca129@redhat.com> <478697aa-f55c-375a-6888-3abb343c6d9d@redhat.com> <205abf01-9699-ff1c-3e4e-621913ada64e@redhat.com> <12588295-2616-eb11-43d2-96a3c62bd181@redhat.com> <8d187891-f131-4912-82d8-13112125b210@redhat.com> MIME-Version: 1.0 In-Reply-To: <8d187891-f131-4912-82d8-13112125b210@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Queue-Id: ACC9B40014 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4czot498ihsjfdxenfma166mpikcf76g X-HE-Tag: 1697569211-877912 X-HE-Meta: U2FsdGVkX19bRNT0yvfDEu1EG8+2lUFUuIW39SGsrMKkqZ0XH6077yOix8fhdF0vFB8929ttRxOpXJO7onXRnRBHGx+azCEgZAx9QejvXJejT+/OmeJMCN+xB+3+DjNcLKmMJEPom+ZCjU+6luVoylBkFpESW/w4EhFMjhWqD54jH+Vopu9e7AVHW4wl/m06CcfSMHESFinj6fusxMaa8yVQTAx5upFVrjbCYSQprz96lAmoAftTXQ327aIUqRJxj4LuVQif4PLS3EBBsqwf4csnd7FzQBWn8vqhrqf/1nIy6HH1O3Kp9qoBMJwkRApfVprlmF9y8E31kyzfQgTqGNiMid7K4p9bM9yEdMshptcqu74wm9QPfKpeheEXzvAit5NNSxQDyFN5tuELGy0iC/4U49msKkmm7N3T+fJ/d7Au5fBY7ChgT49pCmBP9w5ajKQyCLM47kgyTTbd5PWO2ry5DG9Ijymc2jRcaiSZLS+ftZ+7lFZOwHQ5cFh18t/7Oj/V83FJYgDZnuesBEG2nkgM8mBxPaEeQTCf8FXIs5Km6AR+3il+2HS4YKwFwsWC1et+CWiJku/CbcczItpWWNFNWOoovi7FpCX3KnbxdYJTKGz2keJO6u6BcTp2WTELhgrOaLgolu6Ntvf/wgW6XmUvWFo5vs3AeDb/zBJQmimwigZpaLSu1q1qDSaHXPmfk8LIvu0T/AWQkPFcTwC/yGv/uFaQp2mmMj5nUQkrzASY8ojT65gz90Q/5kXa4DYRp1ls2Az5+3mtPQ1T9q6hRCpbfLgQ96LzYVXN98M2XDiOrxwfDTbHJg1nu9YQcXV4fDgiD7geUDjOFVO1Fd1eApSkcfsKxh/YYjd62lfuW0zckDx+YKno6Nqe52rSnDSEe3sj76NfnHyXUp8ntZQII2hg+ySk/xmCnDTA2aL2Agq1cWeSqtJT09fX/cuH6lx0BBvX7kxi9PQRZMOKQlA SpIGcoFm Ck4890MoObOUv9FzszNl3DqMREYOW854Vh+CUUdmmqAi7MaRIVXQ7sXxhiCP+3HhdRk9A9hTkveSH5FfMADySZih8Q/CvLHMJk8NLSaiHpF/PpZ2dSzRvXZJBxpRadpPHg8oftmrt1yegJeui7y01C6ppKVdbTxwFdcN3MQxy9YBfWbJslb5Qw8CYClV5+G6+Ya7ErIaAwKNshnrOzswdWstC6aTDnWOOODdKRKjhjp3zPzC7zunPpCAtOc8gmFACkcxpxdmsq54pAS6J9eQ4qTDAp034SIeyxzTMw/ywz310OnUY7i5aZgAd69ktG3wt1ZSH9yZz/LYMnov5UMadVvkqvZ52/pGEI6O+Jj+IJ0hx7bPT088Ivdw34xx9IVrZicaDuBX0awhCU9sxIwio6sWNgO4ZUyuHlUcxVcETjDobp804faoD5gFJ06NRLao5O/H/kbqZZFZjguRhY22dRrDY7EglV9QcgH0ZbguwzBPcGuw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: David, On Tue, Oct 17, 2023 at 05:55:10PM +0200, David Hildenbrand wrote: > Don't get me wrong, but this feature is already complicated enough that we > should really think twice if we want to make this even more complicated and > harder to maintain -- because once it's in we all know it's hard to remove > and we can easily end up with a maintenance nightmare without sufficiently > good use cases. Yes I agree it's non-trivial. My point is adding cross-mm doesn't make it even more complicated.. afaics. For example, could you provide a list of things that will be different to support single mm or cross mm? I see two things that can be different, but I'd rather have all of them even if single-mm.. - cgroup: I assume single-mm may avoid uncharge and charge again, but I prefer it be there even if we only allow single-mm. For example, I'm not 100% sure whether memcg won't start to behave differently according to vma attribute in the future. - page pinning: I assume for single-mm we can avoid checking page pinning based on the fact that MMF_HAS_PINNED is per-mm, but I also prefer we fail explicitly on pinned pages over UFFDIO_MOVE because it doesn't sound correct, and avoid future changes on top of pinning solution that can change the assumption that "move a pin page within mm" is ok. Is there anything else that will be different? Did I miss something important? [...] > BTW, wasn't there a way to do VM live-upgrade using fork() and replacing the > binary? I recall that there was at some time either an implementation in > QEMU or a proposal for an implementation; but I don't know how VM memory was > provided. It's certainly harder to move VM memory using fork(). Maybe you meant the cpr project. I didn't actually follow that much previously (and will need to follow more after I took the migration duties.. when there's a new post), but IIUC at least the latest version needs to go with file memory only, not anonymous: https://lore.kernel.org/all/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/ Guest RAM must be non-volatile across reboot, which can be achieved by backing it with a dax device, or /dev/shm PKRAM as proposed in... Guest RAM must be backed by a memory backend with share=on, but cannot be memory-backend-ram. The memory is re-mmap'd in the updated process, so guest ram is efficiently preserved in place My understanding is there used to have solution for anonymous but that needs extra kernel changes (MADV_DOEXEC). https://lore.kernel.org/linux-mm/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/ I saw that you were part of the discussion, so maybe you will remember some more clue of that part. IIUC one core requirement of the whole approach is also that it will cover VFIO and maintenance of device DMA mappings, in which case it'll be different with any approach to leverage UFFDIO_MOVE because VFIO will not be allowed here; again I hope we start with forbid pinning. But it should be much cleaner on the design when with UFFDIO_MOVE, just not working with VFIO. One thing I'd need to measure is latency of UFFDIO_MOVE on page fault resolutions. I expect no more than tens of microseconds or even less. Should be drastically smaller than remote postcopy anyway. I'm probably off topic.. To go back: let's try to figure out what is special with cross-mm support. It'll be very weird in the future for anyone to propose a patch just add a feature flag and declaring cross-mm support, if the code is mostly all there. Nothing stops us from discussing what a cross-mm design will need. [...] > Is that and will that remain the case? I know people have been working on > transparent user-space swapping using monitor processes using uffd. I > thought there would have been ways to achieve that without any corporation > of the dst. Any example? For what I am aware, all corporation requires uffd desc forwarding. I think the trick here is any userfaultfd desc must be created by its own process, so far nobody else. That's more or less saying "I want to do this" from its own opinion. The next is forwarding that to someone else. Parent process is fine taking uffd of child with EVENT_FORK, as I mentioned, but besides that nothing else I can think of that can violate this guard to manipulate a random process. Thanks, -- Peter Xu