From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7777EC6FD1D for ; Wed, 15 Mar 2023 21:19:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D10226B0075; Wed, 15 Mar 2023 17:19:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CBF6F6B0078; Wed, 15 Mar 2023 17:19:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B86D96B007B; Wed, 15 Mar 2023 17:19:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A8FC56B0075 for ; Wed, 15 Mar 2023 17:19:31 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 71509C1217 for ; Wed, 15 Mar 2023 21:19:31 +0000 (UTC) X-FDA: 80572398942.22.9BF7865 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf10.hostedemail.com (Postfix) with ESMTP id 6B063C000C for ; Wed, 15 Mar 2023 21:19:29 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=J4hENgr3; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678915169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NwsJ4+G5J9g17XC5Oega2fzdoX3R4HN5m00Gv1AGrio=; b=bAIzqsZUCto/0fAT7MaUlh36gZgyjoIO1kPdhw81RHKIf3DokHu6jRaZZzZbgXkS+M1xE/ nObe33qmer9SWCj8Oh6gjBuBGhBA80v+MrhO0yKFIhP9w+193cKgNaXoI7uw0zIN2OZGnv GJINfrMuV148Ul2n9tnyJUaRWh2LCQ4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=J4hENgr3; spf=pass (imf10.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678915169; a=rsa-sha256; cv=none; b=dtqity7IuAigOXLb04MbgrO5N6rZpdf4zFSZRCBsb15ya17h6tH+F3Xk8pMOdEwRusnh1E vK1Km0La0xh+i7q/uAtvvf0Ps4rqVJMo9ZfSaupAJu4ujuOChLJ3aPFCC6QOZTS000tQce nJR7WF/0bsJm0ziIZPBXIpdWBNIvHKY= Received: by mail-qt1-f176.google.com with SMTP id l13so17823018qtv.3 for ; Wed, 15 Mar 2023 14:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; t=1678915168; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=NwsJ4+G5J9g17XC5Oega2fzdoX3R4HN5m00Gv1AGrio=; b=J4hENgr34NSGF6vyf2nmvDcQD6/muv0hdsHy09NIXn8lT7gniI+nN3rrtoNFk2ZMYA 8mmGoYBT75II1U4TG76NhTKQsb3gLBT3EbgmxXUGlLgMEhbQ67fPeCid7JCLAx3Zz7+7 omxNSH9aY6vY8KsFROJMmgxX+C+9yhSUFhpqknE2oyhNlQbkxezDB7Z92jO2lMaASCN/ lDHVA2APnVRLBvUKkcqHvurw7dD7ZMiJfaPF+ubWv3hC26Seao9b8uMeNX3tBAUVfq0K tgr3+qb8BmYcHsrz1fw4ll4Fa4vur9uUTcdR0x58449+V5TSWSCUsMKioHvbW/SBQsYV pS2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678915168; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=NwsJ4+G5J9g17XC5Oega2fzdoX3R4HN5m00Gv1AGrio=; b=3rHKU+gROueR77RBpYhuN2AxAkZLu5MuxCan5WMLsbC/gJpqOW62fZoDLsRilFL46z GNpTRE90F7xE22mbAR+RwOQp3zi+Suj1wiX7IOKl4De9JLqWWhpQ8voD8GDCphhlxjg7 Mi8Xul4TOt4cIVbNU7IeqoAYWV8iHCtJVVAmCyUkLyENG/BOO7aF03oJGkj5P+DQqrSJ h/I4eWa6N4cRRL9yOl3OjxnPdzfmrzqLC6Z2eWJWK3ig/oe0/ERxcbLDvhQyq24rJORu dmVV+Ic9Bhgvq8ZeQx0UqTVA8JIpU6htT9Yvd2FWnz6tqx1IcZmunvUYVnVjqMqr8VWn mX/Q== X-Gm-Message-State: AO0yUKXytigL9+QrasbTs118RMEPftvkUnIsdBthEgfhWzV3ETbOK8vc 9pPNRxvNfrB5iPsgZJlnz5Bmjw== X-Google-Smtp-Source: AK7set++jBmiiVxJ+t5MUOdawKP8WhjMZoYTetmJpxQaUcMEPqlctXi+QbRMbsaCZFI2F1xv+M2Vtw== X-Received: by 2002:ac8:5a0a:0:b0:3bb:7603:d68e with SMTP id n10-20020ac85a0a000000b003bb7603d68emr2513487qta.10.1678915168453; Wed, 15 Mar 2023 14:19:28 -0700 (PDT) Received: from localhost ([2620:10d:c091:400::5:62db]) by smtp.gmail.com with ESMTPSA id 64-20020a370343000000b00729a26e836esm4488474qkd.84.2023.03.15.14.19.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 14:19:28 -0700 (PDT) Date: Wed, 15 Mar 2023 17:19:27 -0400 From: Johannes Weiner To: David Hildenbrand Cc: Stefan Roesch , kernel-team@fb.com, linux-mm@kvack.org, riel@surriel.com, mhocko@suse.com, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, akpm@linux-foundation.org, Mike Kravetz Subject: Re: [PATCH v4 0/3] mm: process/cgroup ksm support Message-ID: <20230315211927.GB116016@cmpxchg.org> References: <20230310182851.2579138-1-shr@devkernel.io> <273a2f82-928f-5ad1-0988-1a886d169e83@redhat.com> <20230315210545.GA116016@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230315210545.GA116016@cmpxchg.org> X-Rspamd-Queue-Id: 6B063C000C X-Stat-Signature: 11kqqkm8dp5nsod4keifzzcn4ipm4d1g X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678915169-516565 X-HE-Meta: U2FsdGVkX1+VaQwLw5iISkhZBmWVpqrVLXuXWAXmfNxhT3xXLe1wmmH9IQwNpUKphVSztFh7JUuadDLnccIirD+8GuTuPxIBZ2X1Twij20/UAnB5uTLZS1P/UNDsEATnjjMSU75+S5gs4KkME+Z7z7FHgr7gY3tiEStFgZqZTz4MNC9UqCvpk5DjkibwsjLiYn4h6erP0Ysk+v+UyMmBjwQ9jSFKAU5ESN8ftb12Au5UmJTVQtr+mVFI3g5cXUqYwRl7ri+R13Rsa4bidoovcS4cmgqi5wbE0iv2SZgcM+4Ut+kUsTarDtcR2N63Cz/MIFuAY+4wZyKeRweX6NceRi9Cy0J3qLUJ4A3kca3Y6X+iaPH2FpmoE4XYzG7IbYfLPQdQ5A0W7iYaXnJRJaC0PYf6GqdSs2r5iDX1/4nyWWxSrPKP4JDJYguueyvXqDj6rNo2CKZebA3/RUzrucEhh6T0telr4cD2J5vzYbGdDPja6aGjazgw/clXGvMFg+Sj2jfD+l+2y77W7hrYMt/HejrLPrjSFIkhZhUUCVcoOneYnAg0ahthBql1/YpzRG0LsQEC66I/bx+k92UWJYZ6xeiNci/fzU5U4eiAIVkJwwiAvoc2hERldjuFYtpM4yHEbn/pBzlPHzQz5Ofq6thwGmxDbhdDOyM/5VlGtF3Xwmu8NtL2SIciG1QSiUmRsyFLcu/BZcOGc6ohZ4GDXjrKJvfbEamQDNlI+pTvkGpdiPDOGQOZevt9/kgxyD7h+P2tGZH7dJahWEQakPFuTd3lYzOYkAvQpdbA94Vp57ykDJk+z5AZWB0kIF7/9C5RyyUYUCPDFeIU+G9TH1Iu3Li4hFK8qu8F/72jz5aBCR0ZOtYhT5uYuLmWw2F6SXS145UCo9i+iMx/QJmXPqRiFUiKE916HW0yJneUW5656HimuuNoNO5DkeLxT5nbkv1tGtQsb7vv0pEJDT7KOgef9XP aqIW2mlQ rny2wgg1IzUVMBnxDTdi9H75Veh7aycMpMva1N5HLpdWxEIq+OdBC+aGTQAsI7ibLslUAzeklzr+m9uTyMgjVp8VS1HfQBrIn7QfspT1vkFsGq8QPe6vWrzNb22H2l75cGOCx/3fpzr/DXuLdQpXLNarpcL7scjs6pD9PDyd3jdVoBO5vUtQZPzj72XZQ35cm+eHgQsYFcx58XAz08SukxBKL9nV6w0RKiO52GO6i7Qja1X0kLsIpZ0wUrCfxjAEskJaV3+UschrTPkWHtWgysopowl8BERfSeBXd8ejm29NGg+FBrJ71mipGPxvMEH+acg6g6xVR+5XbLHY5ps3I+otvBMWAmx5ojd5Pk1Ad3rSPa2zIdOM8agwAbOnZs2wQ0jwEnkPp8KIkxmctbN2zghW5T5ZnFTgUw6nCNxVIqTifkvX/dQW3mfQxcBs2WCnrK4Y1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 15, 2023 at 05:05:47PM -0400, Johannes Weiner wrote: > On Wed, Mar 15, 2023 at 09:03:57PM +0100, David Hildenbrand wrote: > > On 10.03.23 19:28, Stefan Roesch wrote: > > > So far KSM can only be enabled by calling madvise for memory regions. To > > > be able to use KSM for more workloads, KSM needs to have the ability to be > > > enabled / disabled at the process / cgroup level. > > > > > > Use case 1: > > > The madvise call is not available in the programming language. An example for > > > this are programs with forked workloads using a garbage collected language without > > > pointers. In such a language madvise cannot be made available. > > > > > > In addition the addresses of objects get moved around as they are garbage > > > collected. KSM sharing needs to be enabled "from the outside" for these type of > > > workloads. > > > > > > Use case 2: > > > The same interpreter can also be used for workloads where KSM brings no > > > benefit or even has overhead. We'd like to be able to enable KSM on a workload > > > by workload basis. > > > > > > Use case 3: > > > With the madvise call sharing opportunities are only enabled for the current > > > process: it is a workload-local decision. A considerable number of sharing > > > opportuniites may exist across multiple workloads or jobs. Only a higler level > > > entity like a job scheduler or container can know for certain if its running > > > one or more instances of a job. That job scheduler however doesn't have > > > the necessary internal worklaod knowledge to make targeted madvise calls. > > > > > > Security concerns: > > > In previous discussions security concerns have been brought up. The problem is > > > that an individual workload does not have the knowledge about what else is > > > running on a machine. Therefore it has to be very conservative in what memory > > > areas can be shared or not. However, if the system is dedicated to running > > > multiple jobs within the same security domain, its the job scheduler that has > > > the knowledge that sharing can be safely enabled and is even desirable. > > > > > > Performance: > > > Experiments with using UKSM have shown a capacity increase of around 20%. > > > > Stefan, can you do me a favor and investigate which pages we end up > > deduplicating -- especially if it's mostly only the zeropage and if it's > > still that significant when disabling THP? > > > > > > I'm currently investigating with some engineers on playing with enabling KSM > > on some selected processes (enabling it blindly on all VMAs of that process > > via madvise() ). > > > > One thing we noticed is that such (~50 times) 20MiB processes end up saving > > ~2MiB of memory per process. That made me suspicious, because it's the THP > > size. > > > > What I think happens is that we have a 2 MiB area (stack?) and only touch a > > single page. We get a whole 2 MiB THP populated. Most of that THP is zeroes. > > > > KSM somehow ends up splitting that THP and deduplicates all resulting > > zeropages. Thus, we "save" 2 MiB. Actually, it's more like we no longer > > "waste" 2 MiB. I think the processes with KSM have less (none) THP than the > > processes with THP enabled, but I only took a look at a sample of the > > process' smaps so far. > > THP and KSM is indeed an interesting problem. Better TLB hits with > THPs, but reduced chance of deduplicating memory - which may or may > not result in more IO that outweighs any THP benefits. > > That said, the service in the experiment referenced above has swap > turned on and is under significant memory pressure. Unused splitpages > would get swapped out. The difference from KSM was from deduplicating > pages that were in active use, not internal THP fragmentation. Brainfart, my apologies. It could have been the ksm-induced splits themselves that allowed the unused subpages to get swapped out in the first place. But no, I double checked that workload just now. On a weekly average, it has about 50 anon THPs and 12 million regular anon. THP is not a factor in the reduction results.