From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 267EDEE4996 for ; Mon, 21 Aug 2023 02:53:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C74D900002; Sun, 20 Aug 2023 22:53:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 777328D0001; Sun, 20 Aug 2023 22:53:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63EFF900002; Sun, 20 Aug 2023 22:53:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 54EB68D0001 for ; Sun, 20 Aug 2023 22:53:45 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 22C561401DF for ; Mon, 21 Aug 2023 02:53:45 +0000 (UTC) X-FDA: 81146591610.20.E44F173 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.115]) by imf01.hostedemail.com (Postfix) with ESMTP id 281B84000B for ; Mon, 21 Aug 2023 02:53:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JisZ5StT; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf01.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692586422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I3tQg85SYoX384f6+pVVf/3kTDlwShWlyl3wirJ4EUs=; b=Uj0tyMuKPjf1z8qJDxuNndmB8oqkNER5AJQ4XIdj/SbcDdVLM5X1HIzQfr0TNMtlgQqiei /i5yVQTQ6MLfA29K1iw5lp4ByuLY/D7143evT7/p4uYUcEn4wAw/bIBfqrCxwWqalKWxmL vFJKvsldD9103bmnYfTOU22mLp0z3GE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JisZ5StT; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf01.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692586422; a=rsa-sha256; cv=none; b=ZCdVgvG2rXNXnyq9Vb1xV+V5Jyt0Ks9sEC2FD04vSg+bjjVcQpmtzay8OOeqcM8jE3ZV5+ FDUN14xvTz4rEvPGUOVgEQOb1aprSMVWe5R8oe5iAhB12uDfbojEoZCQAo4PCr/bjKFpjc FZ0Sm/Sos7ss/VMMz2SMY9CXeoYfri4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692586421; x=1724122421; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=IkjujvBsdlsoDFEpFOiG+vgVcyT2VHYJYSw2RNjYJXE=; b=JisZ5StT0yY+V08NSLAH6x5FAa0wgS/UbjkY8yQpBvs6m2XCPgQ20tef LiXfimNcsDSVuUwQQ3ybZ03kkzRCzLEdETZiAq27FQ2CoREarFXudoLmz Owf4uVmV6QX2SMi3PeNsBOkZxY0uvvoTQpwd96ho/qv7uEKrh5hxzl+k3 cdSZxwDFmbKRtiJH9P+1faEGHfAnlm5bU+SLJNg+DDgMHo5SKENs3dDUj dHh9N7fJD5hyUkgSxcmYTg45yx6jVyEH3v1MULD+uDSkQ7dtM0cy5JAXl pHIzIPxvnhLlkONtORv1J0KhcJRaiITRW4WHbDhZ5JxCz2jTQaSSWzTHW w==; X-IronPort-AV: E=McAfee;i="6600,9927,10808"; a="373450687" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="373450687" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Aug 2023 19:53:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10808"; a="825768632" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="825768632" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Aug 2023 19:53:35 -0700 From: "Huang, Ying" To: Byungchul Park Cc: , , , , , , , , , , , , Subject: Re: [RFC 2/2] mm: Defer TLB flush by keeping both src and dst folios at migration References: <20230804061850.21498-1-byungchul@sk.com> <20230804061850.21498-3-byungchul@sk.com> <877cpx9jsx.fsf@yhuang6-desk2.ccr.corp.intel.com> <20230816001307.GA44941@system.software.com> <87r0o37qcn.fsf@yhuang6-desk2.ccr.corp.intel.com> <20230821012804.GA43847@system.software.com> Date: Mon, 21 Aug 2023 10:51:31 +0800 In-Reply-To: <20230821012804.GA43847@system.software.com> (Byungchul Park's message of "Mon, 21 Aug 2023 10:28:05 +0900") Message-ID: <878ra5ds5o.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Stat-Signature: 8isyk581p46j4qq56cu77sjbb5w6njs6 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 281B84000B X-HE-Tag: 1692586420-55927 X-HE-Meta: U2FsdGVkX1/QDls/5qH3b82xuwCCPvSJmPeiDYiRIYB4n/m/96gqGj1v2/fOu7LQLLHLsqSkLxOfFdRCBWqjFH5PVgB50dsHNIFjfyXO/jn9jnnPZ30y0VB2UFk8kSecK63dwuYAtuLADfx49YI5uRTg9qcPYv+FnBWAidvyyrL3zcGf2fsUacdRB9oxBhuusDcxpu8L1YHxb5gUdDifsoz1E8IMoDI8idnrj0Ky3zAz+koRcK/TzdqEFuwmaLRxZAyvekVWyB/XZaP5KV8KbonCXsmjbJWrt7EOT+HxqrPyun3GCvzZ8wChA5hHqFhty6skZfIUnRhkC6q3T4gH2Cf4KOtM05X+oWesdaQOGYCgvLLKBfTe83W46r1NzpzpiNOfuaNgTpu9rDgy09+bPbRH6QxJAZw4SNvfR8U8pUA3fKjXX05Jwj5rULDhEPluuBZtzBHy5yOc0SWTTBSOwMJHnIdMJbWifqHM6wXzVbzosX+cFh3Lajt3hfqHWTQ3wqrv0YQCThSeI/2I9/giWQA9+/tl31hJa65Hsyctz80i4RBoKXavVIGk07gHehSUstzeAepjzDO/nCo1r0WUUTwrZw7cH+Oax9bGMuxHn5JcDbzHU9XmogRPFGJ5IOxCylPKryD4yf/FktXJGeKR/FkzTmI+2rLYRLRtwRQsKlaoKKbGCgqjsg7GV2Vr6nBWDeiUKK/CkX8aEdHuPeyqgVF5LlG5GF8wVR0PJTNk9dg/rtdhaNOmWhR7rThTkNdOhDvGPUfWANBKhXW1TTz1tjNY6b3FS6NkdTFc4GQPqu6s/iVkizX2PiBWW6ZRDZa9DOCBKRpV+5ObIkXHDCBFNPlHUb3hnVkrTa0t8MXP7Yfiep013olOeRH97du2DKo3xMMHTgj83A9NIdXLRL5vVdW8XkXZTZuGngWJLaA7mB1n9n0x/zxhYSAvtUqHZTAi6gzv1fac0WtBTwzle4m wQrWwKjk 5dXFxiBTJRlUaAfJ1e894NpaiERHlqKJIlASSEwnVbWhY9rnPpgy2ot20oMyqpe0jHgnB91/8jF/e3zlCBNDD5wv/yJ8LOk4FBcec/PoWwxiPKZiYB+2bRg8hPVFMuuj+sAd+ZwHzIvhXUvk5LgyMOKnycDCbX5+sEPe7QN5gLixNt5G+7GwFVUL4bPATHx5HQ0hz52vM40kM5ok= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Byungchul Park writes: > On Wed, Aug 16, 2023 at 09:01:12AM +0800, Huang, Ying wrote: >> Byungchul Park writes: >> >> > On Tue, Aug 15, 2023 at 09:27:26AM +0800, Huang, Ying wrote: >> >> Byungchul Park writes: >> >> >> >> > Implementation of CONFIG_MIGRC that stands for 'Migration Read Copy'. >> >> > >> >> > We always face the migration overhead at either promotion or demotion, >> >> > while working with tiered memory e.g. CXL memory and found out TLB >> >> > shootdown is a quite big one that is needed to get rid of if possible. >> >> > >> >> > Fortunately, TLB flush can be defered or even skipped if both source and >> >> > destination of folios during migration are kept until all TLB flushes >> >> > required will have been done, of course, only if the target PTE entries >> >> > have read only permission, more precisely speaking, don't have write >> >> > permission. Otherwise, no doubt the folio might get messed up. >> >> > >> >> > To achieve that: >> >> > >> >> > 1. For the folios that have only non-writable TLB entries, prevent >> >> > TLB flush by keeping both source and destination of folios during >> >> > migration, which will be handled later at a better time. >> >> > >> >> > 2. When any non-writable TLB entry changes to writable e.g. through >> >> > fault handler, give up CONFIG_MIGRC mechanism so as to perform >> >> > TLB flush required right away. >> >> > >> >> > 3. TLB flushes can be skipped if all TLB flushes required to free the >> >> > duplicated folios have been done by any reason, which doesn't have >> >> > to be done from migrations. >> >> > >> >> > 4. Adjust watermark check routine, __zone_watermark_ok(), with the >> >> > number of duplicated folios because those folios can be freed >> >> > and obtained right away through appropreate TLB flushes. >> >> > >> >> > 5. Perform TLB flushes and free the duplicated folios pending the >> >> > flushes if page allocation routine is in trouble due to memory >> >> > pressure, even more aggresively for high order allocation. >> >> >> >> Is the optimization restricted for page migration only? Can it be used >> >> for other places? Like page reclaiming? >> > >> > Just to make sure, are you talking about the (5) description? For now, >> > it's performed at the beginning of __alloc_pages_slowpath(), say, before >> > page recaiming. Do you think it'd be meaningful to perform it during page >> > reclaiming? Or do you mean something else? >> >> Not for (5). TLB needs to be flushed during page reclaiming too. Can >> similar method be used to reduce TLB flushing there too? > > If you were talking about unmapping for swap while reclaiming, then I > think yes. The case can also take benefit from CONFIG_MIGRC. Yes. Thanks for explanation. -- Best Regards, Huang, Ying