From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15B2BC54FB9 for ; Fri, 17 Nov 2023 19:46:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82BA38D000A; Fri, 17 Nov 2023 14:46:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DA8C8D0003; Fri, 17 Nov 2023 14:46:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A3688D000A; Fri, 17 Nov 2023 14:46:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5C3F88D0003 for ; Fri, 17 Nov 2023 14:46:53 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3C9E31A0646 for ; Fri, 17 Nov 2023 19:46:53 +0000 (UTC) X-FDA: 81468479106.09.68F1A0F Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf04.hostedemail.com (Postfix) with ESMTP id E2FC640009 for ; Fri, 17 Nov 2023 19:46:49 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=VSQGOg2w; spf=pass (imf04.hostedemail.com: domain of sumanthk@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=sumanthk@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700250410; a=rsa-sha256; cv=none; b=2fMLVpCR10z1rOtKtSQMAWvBLOWcv9k1kl8m5CBqE3O/iTitP6bhMNVw6WuABlwG81cdox ny0Efgv0D3MDW5jDSxwjWcHT/kMPnic+hz11H71z3Zrl9MkGG6f2IrT7xgTN679yxnoiYn n+r9h8TNUAyILs0WNApEZ3xgKFMP5rA= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=VSQGOg2w; spf=pass (imf04.hostedemail.com: domain of sumanthk@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=sumanthk@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700250410; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V2het07mxfWMUnHT9sJPdfoPQsc50ihRh3IyKWgcirs=; b=WlFJO8GvG8fOLqS/XnzGiwuI8xS/AHVJc6r5IWfhIH7sKnq4nvCuexsSs8GfkO3AUBZ7l6 4nNidK5U7mGQ6Y1keUA6ZTK8t1yRnCd6RzrxT3Bk13cf03vwmFAfy3I1RPLnH+NtbHGXhf OCO+5jCLsPXtq64EryFr0yG9t4J1xM8= Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3AHJgFVU009656; Fri, 17 Nov 2023 19:46:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : in-reply-to; s=pp1; bh=V2het07mxfWMUnHT9sJPdfoPQsc50ihRh3IyKWgcirs=; b=VSQGOg2wbO9wdvRfJYoS2RKuOB7FMbmkX9Q87tqMPY5R5dSsW/htEwBIprv5CNlVetIo +iaQZ0w39j0Q9tAdbDlrazDQ3BoEEhdzDJgelHuFdSzCT82SzPvb59Qdw+7eWBlYq45z qt/5kEjHP3isK+7iIZekQdDUncNqZJWMYMsFs3rScTDtHO71al9elrWPZ4NW/9mNTUtz 33vjRY3r/43yPc5Rix1CPzteHtzSS+k9ybct6Yei9I2WKct0miIFLL3NmmjwsIImEhAw 7JcTjm/La8IOpZHzPQS5BOctb6LmutAZnyHFOTCYEwA+/GEIf4RCGUbtgHel3r4twtBM Dw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3ueejer5fw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 17 Nov 2023 19:46:45 +0000 Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3AHJgCsj009592; Fri, 17 Nov 2023 19:46:45 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3ueejer5eg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 17 Nov 2023 19:46:45 +0000 Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3AHISDNL009056; Fri, 17 Nov 2023 19:46:43 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3uanem88sh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 17 Nov 2023 19:46:43 +0000 Received: from smtpav07.fra02v.mail.ibm.com (smtpav07.fra02v.mail.ibm.com [10.20.54.106]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3AHJkerM22020816 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 17 Nov 2023 19:46:40 GMT Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1A6D120043; Fri, 17 Nov 2023 19:46:40 +0000 (GMT) Received: from smtpav07.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6776620040; Fri, 17 Nov 2023 19:46:39 +0000 (GMT) Received: from li-2b55cdcc-350b-11b2-a85c-a78bff51fc11.ibm.com (unknown [9.171.27.169]) by smtpav07.fra02v.mail.ibm.com (Postfix) with ESMTPS; Fri, 17 Nov 2023 19:46:39 +0000 (GMT) Date: Fri, 17 Nov 2023 20:46:37 +0100 From: Sumanth Korikkar To: David Hildenbrand Cc: Gerald Schaefer , linux-mm , Andrew Morton , Oscar Salvador , Michal Hocko , "Aneesh Kumar K.V" , Anshuman Khandual , Alexander Gordeev , Heiko Carstens , Vasily Gorbik , linux-s390 , LKML Subject: Re: [PATCH 0/8] implement "memmap on memory" feature on s390 Message-ID: References: <20231114180238.1522782-1-sumanthk@linux.ibm.com> <20231117140009.5d8a509c@thinkpad-T15> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: i039OZloyrjehCuPpyvMOCtYL7DKo22X X-Proofpoint-GUID: fLtOoATvfYaANKAn4lXdlzZ9nheuGiX3 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.987,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-11-17_19,2023-11-17_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=1 phishscore=0 mlxscore=1 adultscore=0 spamscore=1 mlxlogscore=202 malwarescore=0 impostorscore=0 bulkscore=0 clxscore=1015 suspectscore=0 lowpriorityscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311060000 definitions=main-2311170150 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E2FC640009 X-Stat-Signature: nrg9zsmqtym8p67td85gpgkutwff6uz5 X-Rspam-User: X-HE-Tag: 1700250409-14804 X-HE-Meta: U2FsdGVkX1+jTpwd8KItT+znL/1u3QHt0sEg5wC0HJa28TQgn3jgsusJ3kFAYUM3Dg2l9V0G1/rkk0eh5CxC8pCOoHSGvsbjoU8stg63nBLS3bnnHuP896xJi8YzuMNvv5gVCOwF+7QfF4po4MVtdCAmqpV7TLGz1Cb/cypouSGc+It+LbRVpl2bYwCC8v5dTmqsJZJLyU/VN9IC/rqVx4PjKwzd3S8vtoR/OqI87vlJqAPtqpRTvOokmtuiSJfaQhMXgWbPh7W0LJc+ZO4/8sjmcQXqpr74k72gWCtBV6JaNN9BWmvpdA7VyrcCbj428NdsIJZl2XkLYKbWwKFlGiKyx2vzjzOpUYB+sR49q29z3WzzKHRaoYtdhCTJHom+oJCdR7Jg9tEx6SRXA6bpPBR3w9ilage48RYUJRetg2xsBujLbgmfFw2OHYDMZX/1oYeY/uX5jPgjmH6/N1aWB5AKAc8inIroQagoOykibEFQjjkX6UfMGF+fSGIYjUmyyKXfQ+pAXUAtMz31lI43wfinoQrr0jg1u3yZrp4FCYPvFrMrwM9LPDLApGzBAhCjIrRMiHdLazp3GKNnsx6xLrtEps48iYUX/6D/fBnJ0zCQltIHZenff13V7MaIRdJdpOwnOyZRbMVqbIKxoR9YBuX1M2uLU2fwXwVUmkv0FRpKFMC82NTc5dD590Z+j/Ufczj97b6mg8TnXmEXjznx4m9eOuofUxcYQosS4QgdR9domkngSa5HjGmHSEIXz1kwdwjP5titjn9bJGsvefA5jRnr9wJVr6HMCrUtEM/VsG/RsAZ/mwElPqiCnbc5MtoP2ff/Y/KL62cOflH2ulp70+lqPQvIy7+a80IAz22AWwTkDsM8vGLlIoZz2td/vqd0Qo2v5ht2Cx5LctNFFk5UlbEAR4J0GihdI39dQTy5jWfTPhLvLSO2UAI3K0hIaSRouLp7ImyvcwMsaIjrNJD M97A0Mpm RiSuLVK1fT2E/F6LbOPIGUfFuph/tvXi9C2VBaXwiYK6bsqddfXdRC7Grdw6ILHERRS75yII1l1FCtd/79ItoM1VC8o/d1ljU4tB6BuWPu7fimhYVP/KFeNxyselJwNBoaV1TYcpAuHEAA8g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Nov 17, 2023 at 04:37:29PM +0100, David Hildenbrand wrote: > > It might make sense to > > 1) Send the first 3 out separately Ok sure, I will first send 3 patches as bug fixes with your feedback applied. > 2) Look into a simple variant that leaves __add_pages() calls alone and > only adds the new MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers -- > well, and deals with an inaccessible altmap, like the > page_init_poison() when the altmap might be inaccessible. Thanks for the valuable feedback. I just tried out quickly with disabling page_init_poison() and removing the hack in arch_add_memory() and arch_remove_memory(). Also used new MEM_PREPARE_ONLINE/MEM_FINISH_OFFLINE notifiers. The current testing result looks promising and seems to work and no issues found so far. I will also double check if there are any other memmap accesses in add_pages() phase. we will try to go for this approach currently, i.e. with the notifiers you suggested, and __add_pages() change. Do you have any suggestions with how we could check for inaccessible altmap? > 3) Look into a proper interface to add/remove memory instead of relying > on online/offline. agree for long term. > > 2) is certainly an improvement and might be desired in some cases. 3) is > more powerful (e.g., where you don't want an altmap because of > fragmentation) and future proof. > > I suspect there will be installations where an altmap is undesired: it > fragments your address space with unmovable (memmap) allocations. Currently, > runtime allocations of gigantic pages are affected. Long-term other large > allocations (if we ever see very large THP) will be affected. > > For that reason, we want to either support variable-sized memory blocks > long-term, or simulate that by "grouping" memory blocks that share a same > altmap located on the first memory blocks in that group: but onlining one > block forces onlining of the whole group. > > On s390x that adds all memory ahead of time, it's hard to make a decision > what the right granularity will be, and seeing sudden online/offline changed > behavior might be quite "surprising" for users. The user can give better > hints when adding/removing memory explicitly. Thanks for providing insights and details.