From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22915CA9EA9 for ; Fri, 18 Oct 2019 21:44:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C639821925 for ; Fri, 18 Oct 2019 21:44:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eYJAofCd" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C639821925 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 722198E0005; Fri, 18 Oct 2019 17:44:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D2C68E0003; Fri, 18 Oct 2019 17:44:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C09F8E0005; Fri, 18 Oct 2019 17:44:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id 3596F8E0003 for ; Fri, 18 Oct 2019 17:44:23 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id C59781842ACD8 for ; Fri, 18 Oct 2019 21:44:22 +0000 (UTC) X-FDA: 76058234364.27.pet77_914ee814dd2e X-HE-Tag: pet77_914ee814dd2e X-Filterd-Recvd-Size: 6893 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 18 Oct 2019 21:44:22 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id p4so6719252qkf.5 for ; Fri, 18 Oct 2019 14:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=f6k6Qcz0XIMruasWL54uOmkX+OaPWhnAxt/2ABdxQoo=; b=eYJAofCde05cTsTfaMyE6JeN+7sR2G7MIGdQnW2wOcWHORSY/qB7ldYTVk8whSwnmB L8gKSJrf8GAQZ6KwNRjlzol1gOoIMpCr+14U3yY5JayWUT8uz9Hubf9i6wdwchFvG8wW KffxTYu1geR2SSmJW5dHX3DZUWn4gxpSWhDluMq5Za5xEyz9+xZTqOM/n+DLBJTuyO6d bP1zJpBEQZw+bjeAgNOihhWKAHHWsRoDBxznbeDZpa7s667hVNVykqZu0tblKtAjkZAx 7B1XHG/eSaCwNarrohViGtTx+/yL6wo9WRmnWkmEnkSujGaqy0b/ZJd+fkT9pgxiq40U EpFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=f6k6Qcz0XIMruasWL54uOmkX+OaPWhnAxt/2ABdxQoo=; b=P11Xd6C0u6A+dEQD95/CpTJJNTi0bvM/t026JRdIRd/K7VfKKQp1mE5exnQuw6YE1/ qFeaDXMBPJWidVjQ5PfimBpJloNIiaRI1d+dW4vv/W4IE564XIqBEovoETIsBg3E5Avo 2Q6LnRtVooHhb5tFbR/qxkP9YxYLY5EWitwBypxQRqCECWEvoL6XOXvjqvqpiBXuX0SK CqPQbPDLT/+LhNYCPHdwGE8SNqGOTDLznqrkmix7sAgG+RhRFgVx/br9d0SUrsdmsj9H cy6fODMjiuILDQ39ayWfMeFWbGIwth6lsJOW1tO0r53dYNOLiheH+xCBSEmR6+Wq4520 CGQA== X-Gm-Message-State: APjAAAXdLdNNd98cBiCL5+GT7GnPpvJb8+sxGbOkWEfGefII4eY8XyEe nxVrVSvcPBmzbt5j5SnoTkBW2MLOi2n3CX/zgvU= X-Google-Smtp-Source: APXvYqzDfShYzG1HI5hqk+rLhFI5e5bmwQUYPJHfZa3YoAdsk9Q3Ae7MIx6eul9el2hIln/xMdNwE22M0c2aRg6VzKM= X-Received: by 2002:a37:f70f:: with SMTP id q15mr10224368qkj.428.1571435061780; Fri, 18 Oct 2019 14:44:21 -0700 (PDT) MIME-Version: 1.0 References: <20191016221148.F9CCD155@viggo.jf.intel.com> <496566a6-2581-17f4-a4f2-e5def7f97582@intel.com> In-Reply-To: From: Yang Shi Date: Fri, 18 Oct 2019 14:44:08 -0700 Message-ID: Subject: Re: [PATCH 0/4] [RFC] Migrate Pages in lieu of discard To: Shakeel Butt Cc: Dave Hansen , Dave Hansen , LKML , Linux MM , Dan Williams , Jonathan Adams , "Chen, Tim C" Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 17, 2019 at 3:58 PM Shakeel Butt wrote: > > On Thu, Oct 17, 2019 at 10:20 AM Yang Shi wrote: > > > > On Thu, Oct 17, 2019 at 7:26 AM Dave Hansen wrote: > > > > > > On 10/16/19 8:45 PM, Shakeel Butt wrote: > > > > On Wed, Oct 16, 2019 at 3:49 PM Dave Hansen wrote: > > > >> This set implements a solution to these problems. At the end of the > > > >> reclaim process in shrink_page_list() just before the last page > > > >> refcount is dropped, the page is migrated to persistent memory instead > > > >> of being dropped. > > > ..> The memory cgroup part of the story is missing here. Since PMEM is > > > > treated as slow DRAM, shouldn't its usage be accounted to the > > > > corresponding memcg's memory/memsw counters and the migration should > > > > not happen for memcg limit reclaim? Otherwise some jobs can hog the > > > > whole PMEM. > > > > > > My expectation (and I haven't confirmed this) is that the any memory use > > > is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg > > > limit reclaim and global reclaim both end up doing migrations and > > > neither should have a net effect on the counters. > > > > Yes, your expectation is correct. As long as PMEM is a NUMA node, it > > is treated as regular memory by memcg. But, I don't think memcg limit > > reclaim should do migration since limit reclaim is used to reduce > > memory usage, but migration doesn't reduce usage, it just moves memory > > from one node to the other. > > > > In my implementation, I just skip migration for memcg limit reclaim, > > please see: https://lore.kernel.org/linux-mm/1560468577-101178-7-git-send-email-yang.shi@linux.alibaba.com/ > > > > > > > > There is certainly a problem here because DRAM is a more valuable > > > resource vs. PMEM, and memcg accounts for them as if they were equally > > > valuable. I really want to see memcg account for this cost discrepancy > > > at some point, but I'm not quite sure what form it would take. Any > > > feedback from you heavy memcg users out there would be much appreciated. > > > > We did have some demands to control the ratio between DRAM and PMEM as > > I mentioned in LSF/MM. Mel Gorman did suggest make memcg account DRAM > > and PMEM respectively or something similar. > > > > Can you please describe how you plan to use this ratio? Are > applications supposed to use this ratio or the admins will be > adjusting this ratio? Also should it dynamically updated based on the > workload i.e. as the working set or hot pages grows we want more DRAM > and as cold pages grows we want more PMEM? Basically I am trying to > see if we have something like smart auto-numa balancing to fulfill > your use-case. We thought it should be controlled by admins and transparent to the end users. The ratio is fixed, but the memory could be moved between DRAM and PMEM dynamically as long as it doesn't exceed the ratio so that we could keep warmer data in DRAM and colder data in PMEM. I talked this about in LSF/MM, please check this out: https://lwn.net/Articles/787418/ > > > > > > > > Also what happens when PMEM is full? Can the memory migrated to PMEM > > > > be reclaimed (or discarded)? > > > > > > Yep. The "migration path" can be as long as you want, but once the data > > > hits a "terminal node" it will stop getting migrated and normal discard > > > at the end of reclaim happens. > > > > I recalled I had a hallway conversation with Keith about this in > > LSF/MM. We all agree there should be not a cycle. But, IMHO, I don't > > think exporting migration path to userspace (or letting user to define > > migration path) and having multiple migration stops are good ideas in > > general. > > > > >