From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24F24C433B4 for ; Thu, 1 Apr 2021 20:11:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A38B3610CF for ; Thu, 1 Apr 2021 20:11:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A38B3610CF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F2CDD6B00B2; Thu, 1 Apr 2021 16:11:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EDC4B6B00B3; Thu, 1 Apr 2021 16:11:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D06DC6B00B4; Thu, 1 Apr 2021 16:11:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id ABB3A6B00B2 for ; Thu, 1 Apr 2021 16:11:02 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 6CE6E1832B5AB for ; Thu, 1 Apr 2021 20:11:02 +0000 (UTC) X-FDA: 77984891964.25.25A31DE Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by imf16.hostedemail.com (Postfix) with ESMTP id AA21080192E4 for ; Thu, 1 Apr 2021 20:11:01 +0000 (UTC) Received: by mail-ed1-f54.google.com with SMTP id dm8so3288019edb.2 for ; Thu, 01 Apr 2021 13:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lhY6AEaLMMkgvg+FYNnHd9fBKmOOgfnIt8T40ozDuzs=; b=qhOpzzTG2HENwjAGLhQUT1qBNU1sZlK/ZyWlrgvFX3P9sUtMnlnWdLZp9Uo5cP2Fd6 Hs/sownsomQPprhLo5myKczvg7bUA/PlLD+9lgatC7q4oPf8x7QJhRFe3PvNDz5iwe9u tbCIkWGZUsUSo5SPfrQefDj3XxpFfhs/TaiH0voqbJjPHCDj6hDZBjvvV2Cmj79jXL4Z GbV2XIqfwxAEQlIhpGxEvQVWFBFSxFWrA1iq55yN55TZrwu3ZCV6RcOgW+qtlLZYOZEL KR1CXcGAdiAAx9oQMNKA3s35SiIJtVQJ2dagGDzrBlZo8kernucRm3cusAPSJHYmDSTZ uw0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lhY6AEaLMMkgvg+FYNnHd9fBKmOOgfnIt8T40ozDuzs=; b=dZCkhpHrLGU404RDXnw5GAQB0yiX35Y32DEDF03x2Q7T9rQc9iIC577JD9vz/YFRsY LK4f6TIeLhoojzSmn0eOIypoDJpSjoSgilz/SCQ/UcQK2i27pbozJBEav3K1G7Yp6lPw X3NybS3S9s5Z7hI0K8Oz2GMk1zihLT9oJk8EdykJoHdoKJEk8HfhX5/fhUPaACHImQku b5EQcfL3MvQBiifj8IDGKo0nSEpTZ3umw9ZelHdnT3aIlXiSfVpg1XKv5R1T5hIHT1gm SRLRlZTug8iZ/sgMIylkcPR64sZbf/1+phedqiNToVmffLzP4rhqwDInwOTOMcnOM/H9 nDdA== X-Gm-Message-State: AOAM532q7XzVZJqncCjAzUbew/MndHv1R0Ui0izHKlLmhNPvPSPBbVGx 8eJlX9ibuiOr4K2Z3f4v2fgKyuODg89KcKCtDeU= X-Google-Smtp-Source: ABdhPJwmU2K77LdLdfvZWAExOfcct4DvD4TtqfUc5f/ANVMPG2uE3fZfJ2InJ7LexNueZknTnhK3VI3AE41ok9sI92Y= X-Received: by 2002:aa7:cc03:: with SMTP id q3mr12039422edt.366.1617307861047; Thu, 01 Apr 2021 13:11:01 -0700 (PDT) MIME-Version: 1.0 References: <20210329183312.178266-1-shy828301@gmail.com> <20210330164200.01a4b78f@thinkpad> <20210331134727.47bc1e6d@thinkpad> In-Reply-To: <20210331134727.47bc1e6d@thinkpad> From: Yang Shi Date: Thu, 1 Apr 2021 13:10:49 -0700 Message-ID: Subject: Re: [RFC PATCH 0/6] mm: thp: use generic THP migration for NUMA hinting fault To: Gerald Schaefer Cc: Mel Gorman , "Kirill A. Shutemov" , Zi Yan , Michal Hocko , Huang Ying , Hugh Dickins , hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, Andrew Morton , Linux MM , linux-s390@vger.kernel.org, Linux Kernel Mailing List , Alexander Gordeev Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AA21080192E4 X-Stat-Signature: ufg19zqo335zoa59jtxjah8gsob4kshk Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-ed1-f54.google.com; client-ip=209.85.208.54 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617307861-290914 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 31, 2021 at 4:47 AM Gerald Schaefer wrote: > > On Tue, 30 Mar 2021 09:51:46 -0700 > Yang Shi wrote: > > > On Tue, Mar 30, 2021 at 7:42 AM Gerald Schaefer > > wrote: > > > > > > On Mon, 29 Mar 2021 11:33:06 -0700 > > > Yang Shi wrote: > > > > > > > > > > > When the THP NUMA fault support was added THP migration was not supported yet. > > > > So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 > > > > THP migration has been supported so it doesn't make too much sense to still keep > > > > another THP migration implementation rather than using the generic migration > > > > code. It is definitely a maintenance burden to keep two THP migration > > > > implementation for different code paths and it is more error prone. Using the > > > > generic THP migration implementation allows us remove the duplicate code and > > > > some hacks needed by the old ad hoc implementation. > > > > > > > > A quick grep shows x86_64, PowerPC (book3s), ARM64 ans S390 support both THP > > > > and NUMA balancing. The most of them support THP migration except for S390. > > > > Zi Yan tried to add THP migration support for S390 before but it was not > > > > accepted due to the design of S390 PMD. For the discussion, please see: > > > > https://lkml.org/lkml/2018/4/27/953. > > > > > > > > I'm not expert on S390 so not sure if it is feasible to support THP migration > > > > for S390 or not. If it is not feasible then the patchset may make THP NUMA > > > > balancing not be functional on S390. Not sure if this is a show stopper although > > > > the patchset does simplify the code a lot. Anyway it seems worth posting the > > > > series to the mailing list to get some feedback. > > > > > > The reason why THP migration cannot work on s390 is because the migration > > > code will establish swap ptes in a pmd. The pmd layout is very different from > > > the pte layout on s390, so you cannot simply write a swap pte into a pmd. > > > There are no separate swp primitives for swap/migration pmds, IIRC. And even > > > if there were, we'd still need to find some space for a present bit in the > > > s390 pmd, and/or possibly move around some other bits. > > > > > > A lot of things can go wrong here, even if it could be possible in theory, > > > by introducing separate swp primitives in common code for pmd entries, along > > > with separate offset, type, shift, etc. I don't see that happening in the > > > near future. > > > > Thanks a lot for elaboration. IIUC, implementing migration PMD entry > > is *not* prevented from by hardware, it may be very tricky to > > implement it, right? > > Well, it depends. The HW is preventing proper full-blown swap + migration > support for PMD, similar to what we have for PTE, because we simply don't > have enough OS-defined bits in the PMD. A 5-bit swap type for example, > similar to a PTE, plus the PFN would not be possible. > > The HW would not prevent a similar mechanism in principle, i.e. we could > mark it as invalid to trigger a fault, and have some magic bits that tell > the fault handler or migration code what it is about. > > For handling migration aspects only, w/o any swap device or other support, a > single type bit could already be enough, to indicate read/write migration, > plus a "present" bit similar to PTE. But even those 2 bits would be hard to > find, though I would not entirely rule that out. That would be the tricky > part. > > Then of course, common code would need some changes, to reflect the > different swap/migration (type) capabilities of PTE and PMD entries. > Not sure if such an approach would be acceptable for common code. > > But this is just some very abstract and optimistic view, I have not > really properly looked into the details. So it might be even more > tricky, or not possible at all. Thanks a lot for the elaboration. > > > > > > > > > Not sure if this is a show stopper, but I am not familiar enough with > > > NUMA and migration code to judge. E.g., I do not see any swp entry action > > > in your patches, but I assume this is implicitly triggered by the switch > > > to generic THP migration code. > > > > Yes, exactly. The migrate_pages() called by migrate_misplaced_page() > > takes care of everything. > > > > > > > > Could there be a work-around by splitting THP pages instead of marking them > > > as migrate pmds (via pte swap entries), at least when THP migration is not > > > supported? I guess it could also be acceptable if THP pages were simply not > > > migrated for NUMA balancing on s390, but then we might need some extra config > > > option to make that behavior explicit. > > > > Yes, it could be. The old behavior of migration was to return -ENOMEM > > if THP migration is not supported then split THP. That behavior was > > not very friendly to some usecases, for example, memory policy and > > migration lieu of reclaim (the upcoming). But I don't mean we restore > > the old behavior. We could split THP if it returns -ENOSYS and the > > page is THP. > > OK, as long as we don't get any broken PMD migration entries established > for s390, some extra THP splitting would be acceptable I guess. There will be no migration PMD installed. The current behavior is a no-op if THP migration is not supported.