From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01A03C433FE for ; Tue, 18 Oct 2022 09:27:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B3886B0078; Tue, 18 Oct 2022 05:27:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 663176B007B; Tue, 18 Oct 2022 05:27:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 551B96B007D; Tue, 18 Oct 2022 05:27:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 45F386B0078 for ; Tue, 18 Oct 2022 05:27:56 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1B97C160F93 for ; Tue, 18 Oct 2022 09:27:56 +0000 (UTC) X-FDA: 80033543352.17.CFB1177 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf01.hostedemail.com (Postfix) with ESMTP id 895C240042 for ; Tue, 18 Oct 2022 09:27:55 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id f140so13553564pfa.1 for ; Tue, 18 Oct 2022 02:27:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Rqb+aVpBsUbDEx7v0ZoQPE3JbRRMNEAN9tkBEWFtU+k=; b=sjpRVNDsx7InUD1sZ0hNW11SsPAOVf/U9F6PWMdb/rcR0J5P/+9BPG/i+jJQ8o5uzJ MB5l6mch73YCxJQkHnQ3kbyr7eQRZEUL+rh3P78SeGqEY2eRFOEm389Xlqc1Islcr6th WnHFGTbqPfud6d478+G/Xn1RCJgEulc9xVCPl6BdYBTPWlTLGnkFmlY0sYstClwBaU1Z Nc/lDjZNfsDOpY+hfoGJ2hs81xWvmAziDvsOpQZoaV4YhqA6+TDX9rytiCWe2Zfryok8 YPkto9bJ86Hk9WVCa71s5KwOX4kTG3ckbSsqlnCXvOrSuUlfH4DcaM2vlKW9520BR1HW 320g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rqb+aVpBsUbDEx7v0ZoQPE3JbRRMNEAN9tkBEWFtU+k=; b=qizw5PAqr4aZl5oIGS74gwvmUP6m/VlSJEqXIRp2GVuIjVUp7o8KVI/2Rl37vD7xa+ oVQnmo8mjbfMh/NPMUR58ogNsQCVKoPCzlpkPD+Ms6fmmQF+7gMpAijGv2tpZ2/JLNU4 ubHVUKKQ8J+EB8CPUBYyP+SoFb6dx9dPXkdXg7iGSZVWLlPQh8997qFDbLCx3hAFls+X 6py6sW66vjUobqnN9cCooqhjCbeajWORlyFznO6qfrC7gWMEuX53UWiwxJN15kRUTXan oX8pr1VTJHOldyUe8qEzRxDYY7Eljxp9aZuT9El4PZQwCRjWCzcFYxXrM3pQ0ULnYR/9 3BBA== X-Gm-Message-State: ACrzQf1GpRx6pZqLmVx4NXSjFwfjJWTo4QqaSXpThecLeSVBbfDS4wBB AdqN50cjYvIJzpZycMh1sHncKcv3mCPaFIsvTLM2UQ== X-Google-Smtp-Source: AMsMyM5eFSsStLxpRdShe9iJjt6fkVyLXgfU9vn4kgqDaDqaSQwiONHiI1Kka/58zD9swX46xe92agyCycA7WN5ZKXo= X-Received: by 2002:a05:6a00:1946:b0:565:c337:c530 with SMTP id s6-20020a056a00194600b00565c337c530mr1992756pfk.47.1666085274296; Tue, 18 Oct 2022 02:27:54 -0700 (PDT) MIME-Version: 1.0 References: <20221012081526.73067-1-huangjie.albert@bytedance.com> <2aaf2c3a-6e49-abb9-b9c8-19ce87404982@redhat.com> <2f41fc4c-68eb-ab7d-970b-fcb10f474fd4@redhat.com> <03b90a2f-2854-6e19-6ccd-41f9933d8813@redhat.com> In-Reply-To: <03b90a2f-2854-6e19-6ccd-41f9933d8813@redhat.com> From: =?UTF-8?B?6buE5p2w?= Date: Tue, 18 Oct 2022 17:27:42 +0800 Message-ID: Subject: Re: [PATCH] mm: hugetlb: support get/set_policy for hugetlb_vm_ops To: David Hildenbrand Cc: Muchun Song , Mike Kravetz , Andrew Morton , linux-mm@kvack.org, linux-kernel Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666085275; a=rsa-sha256; cv=none; b=FsACMJf9rXtc7YPX6jGjirFHgfzcXQ6YAYwIFFw8YR6bKyRaJw85SH+g0dlwA/lnIu2vh6 kdAB+I6n/RGvedMKDPPF1Dd0wYSv+GkDiRT1SJ1O7HdaypiIgK40HrORa6VDY4EXWAEdZO fM3sSPKD6nxx4tomZWvdXl4HtdqxzzQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=sjpRVNDs; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of huangjie.albert@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=huangjie.albert@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666085275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rqb+aVpBsUbDEx7v0ZoQPE3JbRRMNEAN9tkBEWFtU+k=; b=yL1oSvR1nJXE1SF/DQblfpD7TkVsqcer+rExeOmdfhlI5GrhcJMB5kQ4s6z9zFuESaY4E0 SPWHrv9x/M544pfPSgb+ABmKNI23piHAyG0hn+dUngO76jYO61Tgpp2ANRvVZtyHuN12zE d4az0Do3Hp4qDT0FROae2r07Jjn2Eus= X-Rspamd-Queue-Id: 895C240042 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=sjpRVNDs; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of huangjie.albert@bytedance.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=huangjie.albert@bytedance.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Stat-Signature: hdonph33yiqqpcoou697bz5p9zj39cak X-HE-Tag: 1666085275-588647 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: David Hildenbrand =E4=BA=8E2022=E5=B9=B410=E6=9C=8817=E6= =97=A5=E5=91=A8=E4=B8=80 20:00=E5=86=99=E9=81=93=EF=BC=9A > > On 17.10.22 13:46, =E9=BB=84=E6=9D=B0 wrote: > > David Hildenbrand =E4=BA=8E2022=E5=B9=B410=E6=9C=881= 7=E6=97=A5=E5=91=A8=E4=B8=80 19:33=E5=86=99=E9=81=93=EF=BC=9A > >> > >> On 17.10.22 11:48, =E9=BB=84=E6=9D=B0 wrote: > >>> David Hildenbrand =E4=BA=8E2022=E5=B9=B410=E6=9C= =8817=E6=97=A5=E5=91=A8=E4=B8=80 16:44=E5=86=99=E9=81=93=EF=BC=9A > >>>> > >>>> On 12.10.22 10:15, Albert Huang wrote: > >>>>> From: "huangjie.albert" > >>>>> > >>>>> implement these two functions so that we can set the mempolicy to > >>>>> the inode of the hugetlb file. This ensures that the mempolicy of > >>>>> all processes sharing this huge page file is consistent. > >>>>> > >>>>> In some scenarios where huge pages are shared: > >>>>> if we need to limit the memory usage of vm within node0, so I set q= emu's > >>>>> mempilciy bind to node0, but if there is a process (such as virtiof= sd) > >>>>> shared memory with the vm, in this case. If the page fault is trigg= ered > >>>>> by virtiofsd, the allocated memory may go to node1 which depends o= n > >>>>> virtiofsd. > >>>>> > >>>> > >>>> Any VM that uses hugetlb should be preallocating memory. For example= , > >>>> this is the expected default under QEMU when using huge pages. > >>>> > >>>> Once preallocation does the right thing regarding NUMA policy, there= is > >>>> no need to worry about it in other sub-processes. > >>>> > >>> > >>> Hi, David > >>> thanks for your reminder > >>> > >>> Yes, you are absolutely right, However, the pre-allocation mechanism > >>> does solve this problem. > >>> However, some scenarios do not like to use the pre-allocation mechani= sm, such as > >>> scenarios that are sensitive to virtual machine startup time, or > >>> scenarios that require > >>> high memory utilization. The on-demand allocation mechanism may be be= tter, > >>> so the key point is to find a way support for shared policy=E3=80=82 > >> > >> Using hugetlb -- with a fixed pool size -- without preallocation is li= ke > >> playing with fire. Hugetlb reservation makes one believe that on-deman= d > >> allocation is going to work, but there are various scenarios where tha= t > >> can go seriously wrong, and you can run out of huge pages. > >> > >> If you're using hugetlb as memory backend for a VM without > >> preallocation, you really have to be very careful. I can only advise > >> against doing that. > >> > >> > >> Also: why does another process read/write *first* to a guest physical > >> memory location before the OS running inside the VM even initialized > >> that memory? That sounds very wrong. What am I missing? > >> > > > > for example : virtio ring buffer. > > For the avial descriptor, the guest kernel only gives an address to > > the backend, > > and does not actually access the memory. > > Okay, thanks. So we're essentially providing uninitialized memory to a > device? Hm, that implies that the device might have access to memory > that was previously used by someone else ... not sure how to feel about > that, but maybe this is just the way of doing things. > > The "easy" user-space fix would be to simply similarly mbind() in the > other processes where we mmap(). Has that option been explored? This can also solve the problem temporarily, but we need to change all processes that share memory with it, so it can't be done once and for all > > -- > Thanks, > > David / dhildenb >