From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2363AC433DB for ; Tue, 30 Mar 2021 14:42:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6E030619CB for ; Tue, 30 Mar 2021 14:42:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6E030619CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9473D6B0081; Tue, 30 Mar 2021 10:42:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 91EB56B0083; Tue, 30 Mar 2021 10:42:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BF116B0085; Tue, 30 Mar 2021 10:42:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 5FE546B0081 for ; Tue, 30 Mar 2021 10:42:20 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1A17F8249980 for ; Tue, 30 Mar 2021 14:42:20 +0000 (UTC) X-FDA: 77976806040.14.FFE8991 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf25.hostedemail.com (Postfix) with ESMTP id B57DF60006D3 for ; Tue, 30 Mar 2021 14:42:10 +0000 (UTC) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 12UEYsJu087917; Tue, 30 Mar 2021 10:42:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=pp1; bh=QA7H5MVVrTQhEjHy1qS4HEFk81PKJ9zEPRLaF60nnFY=; b=BlJyfdUEAaLdf2Dvbp3bHLYnZzWcWqi8QelZmH9ElwCOSG+dFw339JpqWGGAmxGrakjD UTFmAVKf8hQE0/bapB0wXLgSazIFL/QfUpnNmLgWBvBtwFpiBNIymYh8N5J6mEuHkU4F wK3PYSX82DJqQvMZC/IABlehvAX5zUFk4mLLRigtPrvkUqZqFHpiEtCfsTwZc/GZHrB7 VgcHguys86lQOPP4cD1RmUGsPyG3DNtBxHj3zMxRPUCXbHsa5GMoGrrHmF1KQLZcjBNC xb6+nJmp63ZWbHK/qaDSpPRsZXgUA9PRBirgC5oebIuLHe6/NJaooyYGg4HPo3KL7/Cx iQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 37jhssv32v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 30 Mar 2021 10:42:08 -0400 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 12UEZH0J090959; Tue, 30 Mar 2021 10:42:08 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 37jhssv31j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 30 Mar 2021 10:42:07 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.43/8.16.0.43) with SMTP id 12UEg5uL006510; Tue, 30 Mar 2021 14:42:05 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma04ams.nl.ibm.com with ESMTP id 37hvb8jsmj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 30 Mar 2021 14:42:05 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 12UEfh7o35258624 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 30 Mar 2021 14:41:43 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D8E174204B; Tue, 30 Mar 2021 14:42:02 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2AE3642049; Tue, 30 Mar 2021 14:42:02 +0000 (GMT) Received: from thinkpad (unknown [9.171.41.230]) by d06av24.portsmouth.uk.ibm.com (Postfix) with SMTP; Tue, 30 Mar 2021 14:42:02 +0000 (GMT) Date: Tue, 30 Mar 2021 16:42:00 +0200 From: Gerald Schaefer To: Yang Shi Cc: mgorman@suse.de, kirill.shutemov@linux.intel.com, ziy@nvidia.com, mhocko@suse.com, ying.huang@intel.com, hughd@google.com, hca@linux.ibm.com, gor@linux.ibm.com, borntraeger@de.ibm.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, Alexander Gordeev Subject: Re: [RFC PATCH 0/6] mm: thp: use generic THP migration for NUMA hinting fault Message-ID: <20210330164200.01a4b78f@thinkpad> In-Reply-To: <20210329183312.178266-1-shy828301@gmail.com> References: <20210329183312.178266-1-shy828301@gmail.com> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.32; x86_64-redhat-linux-gnu) Content-Type: text/plain; charset=US-ASCII X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Yi7JXdLMXSe_hitBLtZAibAOcjZOsg2N X-Proofpoint-ORIG-GUID: qeRHACNgQgo-kf2zlYsY1JMCUCYB1sQR Content-Transfer-Encoding: 7bit X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369,18.0.761 definitions=2021-03-30_04:2021-03-30,2021-03-30 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 phishscore=0 impostorscore=0 bulkscore=0 suspectscore=0 clxscore=1011 priorityscore=1501 mlxscore=0 mlxlogscore=648 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103300107 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B57DF60006D3 X-Stat-Signature: i1mcj9nna6ugytrz4gr6s5i6uw67wrhs Received-SPF: none (linux.ibm.com>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=mx0a-001b2d01.pphosted.com; client-ip=148.163.156.1 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617115330-856083 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 29 Mar 2021 11:33:06 -0700 Yang Shi wrote: > > When the THP NUMA fault support was added THP migration was not supported yet. > So the ad hoc THP migration was implemented in NUMA fault handling. Since v4.14 > THP migration has been supported so it doesn't make too much sense to still keep > another THP migration implementation rather than using the generic migration > code. It is definitely a maintenance burden to keep two THP migration > implementation for different code paths and it is more error prone. Using the > generic THP migration implementation allows us remove the duplicate code and > some hacks needed by the old ad hoc implementation. > > A quick grep shows x86_64, PowerPC (book3s), ARM64 ans S390 support both THP > and NUMA balancing. The most of them support THP migration except for S390. > Zi Yan tried to add THP migration support for S390 before but it was not > accepted due to the design of S390 PMD. For the discussion, please see: > https://lkml.org/lkml/2018/4/27/953. > > I'm not expert on S390 so not sure if it is feasible to support THP migration > for S390 or not. If it is not feasible then the patchset may make THP NUMA > balancing not be functional on S390. Not sure if this is a show stopper although > the patchset does simplify the code a lot. Anyway it seems worth posting the > series to the mailing list to get some feedback. The reason why THP migration cannot work on s390 is because the migration code will establish swap ptes in a pmd. The pmd layout is very different from the pte layout on s390, so you cannot simply write a swap pte into a pmd. There are no separate swp primitives for swap/migration pmds, IIRC. And even if there were, we'd still need to find some space for a present bit in the s390 pmd, and/or possibly move around some other bits. A lot of things can go wrong here, even if it could be possible in theory, by introducing separate swp primitives in common code for pmd entries, along with separate offset, type, shift, etc. I don't see that happening in the near future. Not sure if this is a show stopper, but I am not familiar enough with NUMA and migration code to judge. E.g., I do not see any swp entry action in your patches, but I assume this is implicitly triggered by the switch to generic THP migration code. Could there be a work-around by splitting THP pages instead of marking them as migrate pmds (via pte swap entries), at least when THP migration is not supported? I guess it could also be acceptable if THP pages were simply not migrated for NUMA balancing on s390, but then we might need some extra config option to make that behavior explicit. See also my comment on patch #5 of this series. Regards, Gerald