From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F373C433E1 for ; Tue, 23 Jun 2020 02:51:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D1BB42076A for ; Tue, 23 Jun 2020 02:51:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D1BB42076A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4497A6B0006; Mon, 22 Jun 2020 22:51:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F9ED6B0008; Mon, 22 Jun 2020 22:51:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30F996B000E; Mon, 22 Jun 2020 22:51:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 162956B0006 for ; Mon, 22 Jun 2020 22:51:32 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B029B8248047 for ; Tue, 23 Jun 2020 02:51:31 +0000 (UTC) X-FDA: 76958950782.28.music15_110d4a326e38 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 86DD16D63 for ; Tue, 23 Jun 2020 02:51:31 +0000 (UTC) X-HE-Tag: music15_110d4a326e38 X-Filterd-Recvd-Size: 6789 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Tue, 23 Jun 2020 02:51:29 +0000 (UTC) IronPort-SDR: S4I5EUViEug+73/gj8HRN6pHy1fC3f/9z2u4AQ46jqiSrfrVvCdM04DOVsPi/YABpucReNZDxn 7Q1SQDtH5few== X-IronPort-AV: E=McAfee;i="6000,8403,9660"; a="124207047" X-IronPort-AV: E=Sophos;i="5.75,269,1589266800"; d="scan'208";a="124207047" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jun 2020 19:51:27 -0700 IronPort-SDR: stMi4x4u0lGE9oF6INyqSIVVZASuHoTY6RLBNhZfKd1cGfjSj+eDTH/4My3zENuLBHRiWuxxoE C4MywkOnf8rQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,269,1589266800"; d="scan'208";a="275201085" Received: from yhuang-dev.sh.intel.com (HELO yhuang-dev) ([10.239.159.23]) by orsmga003.jf.intel.com with ESMTP; 22 Jun 2020 19:51:24 -0700 From: "Huang\, Ying" To: Ralph Campbell Cc: Yang Shi , John Hubbard , Zi Yan , , , Linux MM , , Linux Kernel Mailing List , Jerome Glisse , "Christoph Hellwig" , Jason Gunthorpe , Ben Skeggs , Andrew Morton , Shuah Khan Subject: Re: [PATCH 13/16] mm: support THP migration to device private memory References: <20200619215649.32297-1-rcampbell@nvidia.com> <20200619215649.32297-14-rcampbell@nvidia.com> <4C364E23-0716-4D59-85A1-0C293B86BC2C@nvidia.com> Date: Tue, 23 Jun 2020 10:51:23 +0800 In-Reply-To: (Ralph Campbell's message of "Mon, 22 Jun 2020 17:05:37 -0700") Message-ID: <87zh8uze8k.fsf@yhuang-dev.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 X-Rspamd-Queue-Id: 86DD16D63 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Ralph Campbell writes: > On 6/22/20 4:54 PM, Yang Shi wrote: >> On Mon, Jun 22, 2020 at 4:02 PM John Hubbard wro= te: >>> >>> On 2020-06-22 15:33, Yang Shi wrote: >>>> On Mon, Jun 22, 2020 at 3:30 PM Yang Shi wrote= : >>>>> On Mon, Jun 22, 2020 at 2:53 PM Zi Yan wrote: >>>>>> On 22 Jun 2020, at 17:31, Ralph Campbell wrote: >>>>>>> On 6/22/20 1:10 PM, Zi Yan wrote: >>>>>>>> On 22 Jun 2020, at 15:36, Ralph Campbell wrote: >>>>>>>>> On 6/21/20 4:20 PM, Zi Yan wrote: >>>>>>>>>> On 19 Jun 2020, at 17:56, Ralph Campbell wrote: >>> ... >>>>>> Ying(cc=E2=80=99d) developed the code to swapout and swapin THP in= one piece: https://lore.kernel.org/linux-mm/20181207054122.27822-1-ying.= huang@intel.com/. >>>>>> I am not sure whether the patchset makes into mainstream or not. I= t could be a good technical reference >>>>>> for swapping in device private pages, although swapping in pages f= rom disk and from device private >>>>>> memory are two different scenarios. >>>>>> >>>>>> Since the device private memory swapin impacts core mm performance= , we might want to discuss your patches >>>>>> with more people, like the ones from Ying=E2=80=99s patchset, in t= he next version. >>>>> >>>>> I believe Ying will give you more insights about how THP swap works= . >>>>> >>>>> But, IMHO device memory migration (migrate to system memory) seems >>>>> like THP CoW more than swap. >>> >>> >>> A fine point: overall, the desired behavior is "migrate", not CoW. >>> That's important. Migrate means that you don't leave a page behind, e= ven >>> a read-only one. And that's exactly how device private migration is >>> specified. >>> >>> We should try to avoid any erosion of clarity here. Even if somehow >>> (really?) the underlying implementation calls this THP CoW, the actua= l >>> goal is to migrate pages over to the device (and back). >>> >>> >>>>> >>>>> When migrating in: >>>> >>>> Sorry for my fat finger, hit sent button inadvertently, let me finis= h here. >>>> >>>> When migrating in: >>>> >>>> - if THP is enabled: allocate THP, but need handle allocat= ion >>>> failure by falling back to base page >>>> - if THP is disabled: fallback to base page >>>> >>> >>> OK, but *all* page entries (base and huge/large pages) need to be cle= ared, >>> when migrating to device memory, unless I'm really confused here. >>> So: not CoW. >> >> I realized the comment caused more confusion. I apologize for the >> confusion. Yes, the trigger condition for swap/migration and CoW are >> definitely different. Here I mean the fault handling part of migrating >> into system memory. >> >> Swap-in just needs to handle the base page case since THP swapin is >> not supported in upstream yet and the PMD is split in swap-out phase >> (see shrink_page_list). >> >> The patch adds THP migration support to device memory, but you need to >> handle migrate in (back to system memory) case correctly. The fault >> handling should look like THP CoW fault handling behavior (before >> 5.8): >> - if THP is enabled: allocate THP, fallback if allocation is fail= ed >> - if THP is disabled: fallback to base page >> >> Swap fault handling doesn't look like the above. So, I said it seems >> like more THP CoW (fault handling part only before 5.8). I hope I >> articulate my mind. >> >> However, I didn't see such fallback is handled. It looks if THP >> allocation is failed, it just returns SIGBUS; and no check about THP >> status if I read the patches correctly. The THP might be disabled for >> the specific vma or system wide before migrating from device memory >> back to system memory. > > You are correct, the patch wasn't handling the fallback case. > I'll add that in the next version. For fallback, you need to split the THP in device firstly. Because you will migrate a base page instead a whole THP now. Best Regards, Huang, Ying >>> >>> thanks, >>> -- >>> John Hubbard >>> NVIDIA