From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8050FC001B0 for ; Wed, 16 Aug 2023 07:43:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE8F828000C; Wed, 16 Aug 2023 03:43:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D994D8D0021; Wed, 16 Aug 2023 03:43:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C39AD28000C; Wed, 16 Aug 2023 03:43:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B0B4C8D0021 for ; Wed, 16 Aug 2023 03:43:47 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 79FF91403B0 for ; Wed, 16 Aug 2023 07:43:47 +0000 (UTC) X-FDA: 81129178494.04.ADF6B05 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 34B8918000B for ; Wed, 16 Aug 2023 07:43:45 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dsJJ19GF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692171825; a=rsa-sha256; cv=none; b=JCfIBIc3vSuR5orWGlfGeW4upQpH6qAnhhkG9sa9d9s9oLfnpwccKbe10sT+g3WwVXyB7u dkPGcQlOlkiLlWQjm3B/0PhJcuUeGeXmcv0+mt9HWlXIOhCnOEqWNauHmRSq97q2T3H9jO ReSA3mByLHUk8S+meVP9AOXPG+2LHg4= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dsJJ19GF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692171825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UpE1Wk6RtuDhxL6BBbhMSsceHjp56cBYmdiTeUYHIMA=; b=d0P7NqAw8ukBN0v2sF5MjDlMiZ9TZ6I+QpKTK9Uv16pyapHjyvfF7/9cdDasGrP2e22fho fcaqbbpZ3e16IhLKcl8zwxF9C4IgRy1Euz2xG+Sih5ppOhh7RWAQiEcbf1BHcWVmIzzZgI 0YGfZhxnEviKQSV2Cq+R1f2u9ZR+G/c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1692171824; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UpE1Wk6RtuDhxL6BBbhMSsceHjp56cBYmdiTeUYHIMA=; b=dsJJ19GFtJyYrPANtmjUeTL2rChQA3Qf6SeqAuGiPeVTw2PTtLi8194uRYSZ44gLvOJCqF wn9QtsRVbQRsO9e7hL+E5A8r2CI93ySqp7I+Ky0D7EDV/5VnVWkdysSgEu0YO1qiUv+8Wm S/N0cJFEH5E1V+qxwEfpZx+yfT0cyiI= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-618-y_c27USrMg-5K6sU-14epQ-1; Wed, 16 Aug 2023 03:43:43 -0400 X-MC-Unique: y_c27USrMg-5K6sU-14epQ-1 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-317d5b38194so2861291f8f.0 for ; Wed, 16 Aug 2023 00:43:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692171822; x=1692776622; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UpE1Wk6RtuDhxL6BBbhMSsceHjp56cBYmdiTeUYHIMA=; b=iVcqH58ltIeL1T5LRJ/+Co6Li8pu0NFRaWx4LcKPFHoBNTE3I3rK2eYlsU14uPIMTr mTRVe17+LsYk+VFX1AUNG6Lb4ASnoTo84gQTEgY9EeqW+I3885GeaQogmiJgX1ECrJAo 3nV01n30dJCaO2nS8Bw0awrq9jpVentFqQ2tUfmZ11R2W/qndKF9fkJeaNpLT76bErmi 0eh+hbnZJTRRgu6rHrkR7EmLqbv2G2nlGFWmtCEkNdS1dPRYmeRXU2+ShjfUPwURAmSG tO8ltK+E7TPkhnd2ldkaTQQayj+UNduo0PXfAQN4KfeJfizdV4xnAZxixeytiBQF1oHS Odjg== X-Gm-Message-State: AOJu0Yww4U/oDW2QFNEuOJs+GJeaYOWpyEi0rTfJ6z4TrdATa/e4Kd3m FLKdMjAs7LLVE4C3KEoeUnpcTmON/mx+fnPsf+KwR4ThjGoXyjhYTGfqHYtf7XCncoqe1p5zjnT u66XB7Tgo78M= X-Received: by 2002:adf:e749:0:b0:316:f4b9:a952 with SMTP id c9-20020adfe749000000b00316f4b9a952mr2948465wrn.31.1692171822264; Wed, 16 Aug 2023 00:43:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEoxs2uyCX8yiPq64wznfoV64WTc9z+A6YDeGesjPO6fW3H11bdDvj+MzomGCloxPcmqbtE4Q== X-Received: by 2002:adf:e749:0:b0:316:f4b9:a952 with SMTP id c9-20020adfe749000000b00316f4b9a952mr2948445wrn.31.1692171821805; Wed, 16 Aug 2023 00:43:41 -0700 (PDT) Received: from ?IPV6:2003:cb:c74b:8b00:5520:fa3c:c527:592f? (p200300cbc74b8b005520fa3cc527592f.dip0.t-ipconnect.de. [2003:cb:c74b:8b00:5520:fa3c:c527:592f]) by smtp.gmail.com with ESMTPSA id o6-20020adfeac6000000b0031aca6cc69csm444421wrn.2.2023.08.16.00.43.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 16 Aug 2023 00:43:41 -0700 (PDT) Message-ID: Date: Wed, 16 Aug 2023 09:43:40 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 To: John Hubbard , Yan Zhao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, pbonzini@redhat.com, seanjc@google.com, mike.kravetz@oracle.com, apopple@nvidia.com, jgg@nvidia.com, rppt@kernel.org, akpm@linux-foundation.org, kevin.tian@intel.com, Mel Gorman References: <20230810085636.25914-1-yan.y.zhao@intel.com> <41a893e1-f2e7-23f4-cad2-d5c353a336a3@redhat.com> <6b48a161-257b-a02b-c483-87c04b655635@redhat.com> <1ad2c33d-95e1-49ec-acd2-ac02b506974e@nvidia.com> <846e9117-1f79-a5e0-1b14-3dba91ab8033@redhat.com> <4271b91c-90b7-4b48-b761-b4535b2ae9b7@nvidia.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in a VM In-Reply-To: <4271b91c-90b7-4b48-b761-b4535b2ae9b7@nvidia.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 34B8918000B X-Stat-Signature: m8zebcdpqu8jsmxuf1qybinkx984hw99 X-HE-Tag: 1692171825-151466 X-HE-Meta: U2FsdGVkX183E2mBC3ipIeFQFp5SlshbjMKvuFybRkX+zPmKWzicRYyH6fZ0c7xmppVwbQOP1ztfVpeHumyp8ulYDSvEh2GVNBiqpxV04eXLbTy4PiQf3/iuZXUW3+nz+QoUF9a65vJ90rHqpUVdnMv51VcbDYELhTvzMSfcbWCLxaK1ri0XPzuYSw4PUycBWfH2ma+60u/2CgEn7UNKAQbGoXZLJlAfnTrzros+ji5xGjZRz3KLG+aY8M01sExu4TIGzXWOcxI0QJaQFX4s8o1UiXf/UCM49jdAjOJ76nByzt3s8MdA+nZ0rWrG/OeESuaNPQKeC8EbJHI21n4DQzTQ7CQgfXJBCBeFSDRElposK2u6+3OdWe5tMwkkXwnXWX/If3tCGSfvEweBVjqXneIDCkAC1tVaj1UvRrKvjFHfxxeF0/wuK8oO5lt82usyMfqJc6Gmw+uP7B4r8jxzaxDH4fzc/gMV6DbGPSDNvSePryBa7hpUaP78Rov7JLQZiNlX/Kw7hLz1/jf1ycuYVEsOOOTAjXkQ9Y+I4cqwvhQDAMfcW64V1WkRcudqbVkFy14sPGnDzLr8KcTwAcdi/xYR6VEt0ZScigpbgnTREJdgC5o/ukHjJ2E1hHs2Q1ZFSRmwW3q+s03wsNmpD8znK5R7t/6hLedEQSTtA7BU6FkY0vmh2emvIBsEPfnPQw/fklmrCDdpneGtWGL06KolKkIF9NCLy4oh97MpPJVI19s0BCPAzjLQXeZjeTV834sEpUeSD8o+aSzak6PNoRB7hxOs2LACu8L789xgdtJTok4lAZMEn4yvr3VDsf+JM4H/DZSEsWaI+E+SrMweD13sCI3oJhX1LLNnzBE/9yBwn+NC8KEFL1XCrYYwYMHFX9QuFU6S4kUlmomNXWx/oXbrW50jsuTpfTvxaDrN5KjeUaAZe6vde64bl1QLlmvBXD70tEfnqZkAIGyIDsxUyzi zFV11v35 boEosP0ij3ktayxolJ2TephQyRLYSUBVQKsAs1ojbA/buEa6lsVIBCAqeaZHOSR0vW88w4F+mSZXtj42p42dEDJ+W7cSGKXLYrPp0rmMLgXU0D7j6oU51DarSK5aHcSp2GgJJ98IJF2fzx1dQDwQchFVGLbhYL9ljP/+7bR/aTdCkyZBemP+Ec4nZbbSV0U18j0RdS2lJvS9HIS/1HwzpmHjzF7hbaMy/7uv/JnRjU8uy2oO5qhJ81RIe3perwJ/xq0a6SeZ40Ej+qtAT4yYY3Kse1UW7KhXDD3RzoS4pkkHIOd1ogKR6XQBZVrxb/hczhF/FvpLFD+QFi9QSHmnqroWzw8huNWxE26caEwcEVxOI8rYWmdgDWrA3fBMiWvVQVKXZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 15.08.23 04:34, John Hubbard wrote: > On 8/14/23 02:09, Yan Zhao wrote: > ... >>> hmm_range_fault()-based memory management in particular might benefit >>> from having NUMA balancing disabled entirely for the memremap_pages() >>> region, come to think of it. That seems relatively easy and clean at >>> first glance anyway. >>> >>> For other regions (allocated by the device driver), a per-VMA flag >>> seems about right: VM_NO_NUMA_BALANCING ? >>> >> Thanks a lot for those good suggestions! >> For VMs, when could a per-VMA flag be set? >> Might be hard in mmap() in QEMU because a VMA may not be used for DMA until >> after it's mapped into VFIO. >> Then, should VFIO set this flag on after it maps a range? >> Could this flag be unset after device hot-unplug? >> > > I'm hoping someone who thinks about VMs and VFIO often can chime in. At least QEMU could just set it on the applicable VMAs (as said by Yuan Yao, using madvise). BUT, I do wonder what value there would be for autonuma to still be active for the remainder of the hypervisor. If there is none, a prctl() would be better. We already do have a mechanism in QEMU to get notified when longterm-pinning in the kernel might happen (and, therefore, MADV_DONTNEED must not be used): * ram_block_discard_disable() * ram_block_uncoordinated_discard_disable() -- Cheers, David / dhildenb