From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B74DC004D4 for ; Thu, 19 Jan 2023 23:07:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE5F46B0080; Thu, 19 Jan 2023 18:07:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C95A06B0081; Thu, 19 Jan 2023 18:07:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B36756B0082; Thu, 19 Jan 2023 18:07:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A429C6B0080 for ; Thu, 19 Jan 2023 18:07:16 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 6CEF5A0D8B for ; Thu, 19 Jan 2023 23:07:16 +0000 (UTC) X-FDA: 80373086472.18.3C0360F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id F21C12000F for ; Thu, 19 Jan 2023 23:07:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HNveYhQK; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674169634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3bBBNfWfdOJojqP+jjp+TX9gqoRqSR4QzF0ynIoERPA=; b=WTZfeHFD1YQgBkVCrcfYVXL3FBsV2WVE3WLvdjCLBLMhjYOHtJNQ+yTkY3H7m0tV2Wrmpq cApQe1T5cKG8tH8QtCGe7bhqIhTVHbMwdMbfe9JIiR+zW4WGQqNkrq9L6AUegrieAaxn/C 8lJlxqrLoxvCJyA0tP2KTpJBA++NguQ= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HNveYhQK; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674169634; a=rsa-sha256; cv=none; b=ejUHXhlL1A1Ho/JU7t+pzWsEN3RhUQ3L3PF8btiWLVj6ENdoSqVExcWAnFeYoo3dDuap3X PoxBgueqEUVQiZONW4hKLjbTn9+7AB4P4aNMHJxUHueMAK2fi1hFb+ZFID8IlXsC+dCnuK uGzp5pA8rWIn5guGIADrsgKsKEZ4eCc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674169633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=3bBBNfWfdOJojqP+jjp+TX9gqoRqSR4QzF0ynIoERPA=; b=HNveYhQKTFOuzJKwszX0DzDgBQKWlUrc8bD6WJoXaROmUVvNgOoZiwHhPRdQycPEmppQAq ouI0Y34AQPp59comqVuQ6RAoC0LvwfpxmaC4Qf8cK/HMAnUecuW9/wj2wmtYcLZn3KywYi XKNqo2zVNvSc5P+HKrUpQhGo+kLYDAo= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-622-UZEYY-cCMz-b9NglnD5Xbg-1; Thu, 19 Jan 2023 18:07:12 -0500 X-MC-Unique: UZEYY-cCMz-b9NglnD5Xbg-1 Received: by mail-qk1-f197.google.com with SMTP id h13-20020a05620a244d00b006fb713618b8so2324779qkn.0 for ; Thu, 19 Jan 2023 15:07:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3bBBNfWfdOJojqP+jjp+TX9gqoRqSR4QzF0ynIoERPA=; b=8RO5ctxAomSQpdoJuuhDLZ72UVyuruCyut//zNTj4P4j2PoK2F8zYzewVJ+Ua73oLe AtCRmINqerfFNTlUm/Vq1gPbeNOMPQ4u1eeWZYVJpfFxmwartJqhp11K3szOl/tgQale mMFzHYypcLL/tQCVBBeN1xqUFVHtqAtMdUXJbFkvP1kKNFuIP96zc3yUOUKQRTs8Pmnc VhRpeP3mcQ9/KA6MYEGc7SMPCd679ORgKMwLx+LFkY2NJOMxR3d9jkucPVlmoBwfYFnc ytGP9FBAHYyOHUq4t7OX4KKWHvlDD3fyddGMSkbe4gVKAorZ4SytIj+a7Bnkwt6qjIlV Hy9A== X-Gm-Message-State: AFqh2kpXJN5vx7GIGm44z/jk6fUD+ctQDQYl8qy1t/qVOW3yEskTCHM4 hcTx3FU0Ar60P9F3ZeEauHrhonKhuY94RGjTsodT/DfcY3n0fRSWI73Cj/6LWaekfUfEG69HeGe lFmf49jbJdaM= X-Received: by 2002:ac8:7ed7:0:b0:3b6:3260:fa1d with SMTP id x23-20020ac87ed7000000b003b63260fa1dmr16943267qtj.45.1674169631434; Thu, 19 Jan 2023 15:07:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXuNz3WrIEGSC+WUmM66adbpLmN9mzuwPFh7nzVrtEe2Y5cC3Y0zC4cPMmT+xlLot5h/N6JKIA== X-Received: by 2002:ac8:7ed7:0:b0:3b6:3260:fa1d with SMTP id x23-20020ac87ed7000000b003b63260fa1dmr16943241qtj.45.1674169631097; Thu, 19 Jan 2023 15:07:11 -0800 (PST) Received: from x1n (bras-base-aurron9127w-grc-56-70-30-145-63.dsl.bell.ca. [70.30.145.63]) by smtp.gmail.com with ESMTPSA id f8-20020a05620a408800b006b5cc25535fsm25661835qko.99.2023.01.19.15.07.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Jan 2023 15:07:10 -0800 (PST) Date: Thu, 19 Jan 2023 18:07:08 -0500 From: Peter Xu To: James Houghton Cc: Mike Kravetz , David Hildenbrand , Muchun Song , David Rientjes , Axel Rasmussen , Mina Almasry , Zach O'Keefe , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range Message-ID: References: <6548b3b3-30c9-8f64-7d28-8a434e0a0b80@redhat.com> MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Queue-Id: F21C12000F X-Stat-Signature: wsyzxzkh4bji5ix6rccoqkyc3fi63i51 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1674169633-363709 X-HE-Meta: U2FsdGVkX195b37v3rECyH19kAKaYinTkz1kFH20vllaaBExnQZVE5nssCoTKgU4idt4wUAEWcxfF5ZYirT70dJ6BYqS6BQwAKdONyko/Fd8ixbXM4DHdu/4GR0s1NIgWSHaBStjzKd/W2FgNBk0DRqbxW/gbv2rAId9dQDT6Y3JJVA3XLt2QgIxFxYOcioQ56QTtYNo0QwVM7BYDRb19laA+8ePVODBpQHu1NgeIDkfxZ1f1W0b1V49XAXKfnE/04L6tdYNq2quD7zp05yun70sSI5FrUQlXve0CwONuvgWinxX1g3mtZfaMcvO5XD/3tPbprYiHCKdJdDT3lpMn0U9nlcH1Sbm2g8vV3+0LjZIsXHLK1ITQVadFnHSmU8JeSk/jjdEtM+9uKyAY47OVybnWgL9uwETV1jH5Xz3fKwkT68f/XZasvz71KDOMXgVJVnAmFwq0XCleN68EpycdKgPCmqfQDQ6kRKBOuuLeKNtNkPlKW7Z0dZ6/ChZAr6Hy+UZhHjth5qdYjR29r3kPhoY0AhpUVZ3kRjbaPjOfLfBeuCdeYo8rqTw5l8+M/Yn6FrS5DGPOR//JgEfTapyhCAJ4ddAMjoYD4VDrBQrFkzNVXv0rx1zf1Uhb5J7IRkfkwAykvZKmexBtVNrNZAeUP+dr/DqlPNyxpbKr4aCjIhWNATkehyhfeWaiBmok/9tyVdubrf7MkXOG4UdUCwGQkji4C2aX11Nj/Chr//njconk+PISxDYIKPXaYPbB5rRGTB3bwpuUzzocZNMLCrzDFCVXbdWv4LPRllR++lA6bllQCh9PdoDYmZF552xR9NJrF290dxbkzzO2hWwQwvLPW699NRhTU2eS1VJGtVxnLf7NYM6J3daJ/Gp+uCfNnKzRaBgNiOGATJBjl3aRiZUbe7saPCTiyuvLD42IoFkqUSc/7M1wxe1TX8NpySdcLzoxyCS5e3A73AdYGJHiTc DlagEdf2 Agjri98cpviBDyuhAnBNcja08Bz+S11Grx25b5o3wrE6T8n8iebSmaCiyy+RNXG0qoB0kQsq0ihmPchu+PRRyLZyHroEkczAemCp16bcBhmMeI18h4g9B1eTdKz2a+Gi/CORK7u3yRdw+fOSulHL9oM4xIdT05zmHGptgI+N0oZdAw34aIXRJvNaHELXcvq7rFBY4bZ222e1vRa/dzq9VhE0Zj99x7GC+ef2ONdsoyCVx4ynuRZWn9YhDdKOo6vjLfr2jBMOS6i/zbqJnWeWEB9KQsPaNnjvgE9SFVQpGA21GxtUTSYv5vl3fZWx2w6IQHOF6j4waKhb4W+XK89kP+/vYTltO6hrGRg8E+mSeaEEaPPaKL6+1LMbU9hyVGvLUpd96E+2hX710IlrB9Ffk/ng9ZCcnwND47QsYliMd92y4nXWy+etuQt5Ky24Mw0qpqH24aJo5bssHe8M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jan 19, 2023 at 02:35:12PM -0800, James Houghton wrote: > On Thu, Jan 19, 2023 at 2:23 PM Peter Xu wrote: > > > > On Thu, Jan 19, 2023 at 02:00:32PM -0800, Mike Kravetz wrote: > > > I do not know much about the (primary) live migration use case. My > > > guess is that page table lock contention may be an issue? In this use > > > case, HGM is only enabled for the duration the live migration operation, > > > then a MADV_COLLAPSE is performed. If contention is likely to be an > > > issue during this time, then yes we would need to pass around with > > > something like hugetlb_pte. > > > > I'm not aware of any such contention issue. IMHO the migration problem is > > majorly about being too slow transferring a page being so large. Shrinking > > the page size should resolve the major problem already here IIUC. > > This will be problematic if you scale up VMs to be quite large. Do you mean that for the postcopy use case one can leverage e.g. 2M mappings (over 1G) to avoid lock contentions when VM is large? I agree it should be more efficient than having 512 4K page installed, but I think it'll make the page fault resolution slower too if some thead is only looking for a 4k portion of it. > Google upstreamed the "TDP MMU" for KVM/x86 that removed the need to take > the MMU lock for writing in the EPT violation path. We found that this > change is required for VMs >200 or so vCPUs to consistently avoid CPU > soft lockups in the guest. After the kvm mmu rwlock convertion, it'll allow concurrent page faults even if only 4K pages are used, so it seems not directly relevant to what we're discussing here, no? > > Requiring each UFFDIO_CONTINUE (in the post-copy path) to serialize on > the same PTL would be problematic in the same way. Pte-level pgtable lock only covers 2M range, so I think it depends on which is the address that the vcpu is faulted on? IIUC the major case should be that the faulted threads are not falling upon the same 2M range. > > > > > AFAIU 4K-only solution should only reduce any lock contention because locks > > will always be pte-level if VM_HUGETLB_HGM set. When walking and creating > > the intermediate pgtable entries we can use atomic ops just like generic > > mm, so no lock needed at all. With uncertainty on the size of mappings, > > we'll need to take any of the multiple layers of locks. > > > > Other than taking the HugeTLB VMA lock for reading, walking/allocating > page tables won't need any additional locking. Actually when revisiting the locks I'm getting a bit confused on whether the vma lock is needed if pmd sharing is anyway forbidden for HGM. I raised a question in the other patch of MADV_COLLAPSE, maybe they're related questions so we can keep it there. > > We take the PTL to allocate the next level down, but so does generic > mm (look at __pud_alloc, __pmd_alloc for example). Maybe I am > misunderstanding. Sorry you're right, please ignore that. I don't know why I had that impression that spinlocks are not needed in that process. Actually I am also curious why atomics won't work (by holding mmap read lock, then do cmpxchg(old_entry=0, new_entry) upon the pgtable entries). I think it's possible I just missed something else. -- Peter Xu