From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A85ACC77B60 for ; Mon, 3 Apr 2023 09:49:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B9056B0075; Mon, 3 Apr 2023 05:49:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 140076B0078; Mon, 3 Apr 2023 05:49:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFACB6B007B; Mon, 3 Apr 2023 05:49:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DAA176B0075 for ; Mon, 3 Apr 2023 05:49:06 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A454EC09C6 for ; Mon, 3 Apr 2023 09:49:06 +0000 (UTC) X-FDA: 80639606292.12.799C08D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 510AC40016 for ; Mon, 3 Apr 2023 09:49:04 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jO7zvEOg; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680515344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UCVfONq4QX1WJA6KEQBIqDdCC3Zi4ZQ6oAhIXBiY4Aw=; b=BeGwKbeSUgTBYVF0m+NUAQttzwuSmu3f8KrpL2wvQcot43XGdHBekm1YbVw1j+BYryK2v/ PxFZVRNrEafsjvACLBU9rTsKtsjS0op7u2s+tH3RKkYWR2eKpYmNrdVrQuypjJf2dyyQ8P 8+9oJYqzwNas1pbqiFNphEQb1KynOD0= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jO7zvEOg; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680515344; a=rsa-sha256; cv=none; b=20elA1iaqC/ieLWzHZcG4VsouBXzw9+XINJG9zS/4J8ubpxIIwAS1vIj5MUNY7dgFFCIHo Je0IwdWMujA91SeYbwlua74DyNteKWRZiWDxOtlQ6kxkDIVsZPiGDLmh37J2ZT879uROUR PeHi2L/Jjp26w31rMaYqqK6F+VMck+o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680515343; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UCVfONq4QX1WJA6KEQBIqDdCC3Zi4ZQ6oAhIXBiY4Aw=; b=jO7zvEOgkKgF6O1Tb43/eiqFv7vFtilOq6sJhZiMCqOF2lSCMw7Fcl6o/ms9S70gpV54Io 5Fo4UY9WtlrPP0IKfutAVoHkVPCI7N3xAi2v96Z4yt5ZQj1J9l3zEywjdw6CLp/4QOa4OC ipVR+/9yzTWSQ1X4JRzbCxRY4sPslPg= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-261-C4nvNTpYNf-57tcUDjuYDA-1; Mon, 03 Apr 2023 05:49:02 -0400 X-MC-Unique: C4nvNTpYNf-57tcUDjuYDA-1 Received: by mail-wr1-f72.google.com with SMTP id i19-20020adfa513000000b002dc1cdac53fso3049186wrb.5 for ; Mon, 03 Apr 2023 02:49:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680515341; h=content-transfer-encoding:in-reply-to:subject:organization:from :references:cc:to:content-language:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UCVfONq4QX1WJA6KEQBIqDdCC3Zi4ZQ6oAhIXBiY4Aw=; b=iNGxGLZFpkZC1WP9RDaBQKAYr+xQIs/mNGrlVkRf1hyB9jCbhltm3wMpT3RASFXalH sBQ5C7GgYQNFCIBHB/1XeptYX6MDud7u5dHUXLXeAvDtxsdcBQz9i31hvSJoulukR5Yj kFof65teRU7J6jPqgaMEOdYvHzA35WAMjb9t5Tbhj5CfAI1AOw3m9JaCH2PsQSXtBkVO mOYSlmgNd8cpfLE0w+lc+VCgFROowHvvvxxvOa5C6gLtFOgcjGE60yMIvbpUI8m23ZxU FiYu3BJtDZnBZg3DJRpfa3F5KrQ3M2R81y9I3cYJNgbAIJXh6QS3aQQP7AF4dyunDxEk V+DQ== X-Gm-Message-State: AAQBX9eX2Mza5F03bNyW2JJdU1VCRJCcvUA/MI9zYgfWfqzbFTYZaTUw 7cRCY8sucPP+RePc8fFL5erq5/lGFGPqXELiSJzzNjGtr4Ck4Z0qSE0ZfAV+Z22Uqd6t2iD/oEX gAWQbcFPF+zQ= X-Received: by 2002:a5d:460b:0:b0:2d2:59cf:468f with SMTP id t11-20020a5d460b000000b002d259cf468fmr13074454wrq.15.1680515341386; Mon, 03 Apr 2023 02:49:01 -0700 (PDT) X-Google-Smtp-Source: AKy350arsxFGGG4mFkfzu2mP7wXMWNJhAfCsHB0QkcMrhf+hYAHPVuXlVhP82orVZo6dQiD7R1JfuA== X-Received: by 2002:a5d:460b:0:b0:2d2:59cf:468f with SMTP id t11-20020a5d460b000000b002d259cf468fmr13074432wrq.15.1680515341062; Mon, 03 Apr 2023 02:49:01 -0700 (PDT) Received: from ?IPV6:2003:cb:c702:5e00:8e78:71f3:6243:77f0? (p200300cbc7025e008e7871f3624377f0.dip0.t-ipconnect.de. [2003:cb:c702:5e00:8e78:71f3:6243:77f0]) by smtp.gmail.com with ESMTPSA id i16-20020adffc10000000b002c55ec7f661sm9350194wrr.5.2023.04.03.02.49.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Apr 2023 02:49:00 -0700 (PDT) Message-ID: Date: Mon, 3 Apr 2023 11:48:59 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 To: Stefan Roesch Cc: Johannes Weiner , Andrew Morton , kernel-team@fb.com, linux-mm@kvack.org, riel@surriel.com, mhocko@suse.com, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, Hugh Dickins References: <20230310182851.2579138-1-shr@devkernel.io> <20230328160914.5b6b66e4a5ad39e41fd63710@linux-foundation.org> <37dcd52a-2e32-c01d-b805-45d862721fbc@redhat.com> From: David Hildenbrand Organization: Red Hat Subject: Re: [PATCH v4 0/3] mm: process/cgroup ksm support In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 510AC40016 X-Stat-Signature: dfkgnmk1g57dbw133enjijks1e64jd7s X-Rspam-User: X-HE-Tag: 1680515344-536500 X-HE-Meta: U2FsdGVkX1+yWEzOGPVEu1+KzOcOPTKCulnxGI9y2VABLpBtLScVgiYyXk1HaJVFHa+eQ3FyUskR3Ua6OCZfbUcZkOLKNkaU3X5Qs2fX143Rs0cLuNt8fpTXOk1ZZSWBTC6exe5LifRaW0dfiCV/oAwfr55bKFy1mNpgkpmUfm3lgTn48IRlQvTlJcQBdJP2yc94cMDd2OXSQ4I/hzzcpbhGVvLAtOZXOD0wGnOTa/tQefV8TVRyOryBbXEeDlSQhgIMKqTWRO7eEvgelO1eGWubtgn3zUUoymgYfmF7VMVEGyoLcMhAXABAs1srPhIJtBbPJ8ZeyGN8eCeZh/cqDH/QpUlq86HfdYWD5vaa6LhxoXEUDVCThlK2/Hy5qBBCj8K3XyQYlCQqTiJRQCEge2LVjzG5+msGEuVZjzVKLw1h5ENU1fIhmSi/zkpo9t4dInMETLURm80Dkr8+gqdMc/TdlNv8KdqrGPRSCtmya3pC5EKzk2gk7DVO2dIwV/pGnHOz58vTqFNDDNqpDpodrJWA0rQeHlyMT7GZHFtxAkwsyor+KH+7u0D6a95yEKDy0LQ5gTEJnUFCLNNAaCXoJTvozsPNl9QMKHyP/98H4T+S47q7fiILDpE903uZDbhVSyC/lk6j4Rul4uz/iwEOMBabYHJ2TbNBApxyi/GKdy+OMLHPxXRJU/xrXnYfmXYBDeHPSh+S2V2CGMMaFGiokrr/dyfaMu07/f7UiJUfSB+VFhhcPhF/2UhC7Z5op42GDJNO3yBgyQKYYhG33vIaTB03mGyWzTt+FQRUCkDI6fq7IJtmIzWnE3B3ejZewhOTou4D4sYILfsz1bZwbXAosqgK5I44TLpsefluCuc2ir1hHrFy0sZ7xIjNERmiP7tWnBOhbl465e56XsBHOjQy5GTk+ADGOMMCY6H/49kYsQurViCQLuIQfQgreEKkYJo7G6M1e6hzh/vasBngc8D YhWE3bU4 5hjjpF5OJn0Zb2PoZ7vngIF4weCYrTih9JW/kPOJpYQbVheh0GvkZ/f2cSTeADGe6UCIlDrw9V1bmIKEAGEaMPxf2RD8bNi3rIrhE+SZj61Yrgs+y8QujtsMTfM8HYz4yGn6P8dx279btAaucoIYnu7j45Fc+/sjMpP3nRcneP3BJQo4dmLWkyUF5lbIYC7yG/ssJucVhQb7p6R7R1sLpTcbfCdZbxjEA4hjEE/x3sOARK+eLAjXKp73mdLvMnjcmjfTQmDYn3shGp+WzyEn+0wu/lIpnkX7xwiyVs8J0MYcPq8XaZ0mVXNobgfGoSiWWURkylB1uTLzPWps+DrGJ+q4z0mt20lk9wbU4q6cx1XcF0g5ymoCISzFrxPMk6wmx2sMBzc7m7tL0w2WeyWdnY8+BJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >>> Obviously we could spend months analysing which exact allocations are >>> identical, and then more months or years reworking the architecture to >>> deduplicate them by hand and in userspace. But this isn't practical, >>> and KSM is specifically for cases where this isn't practical. >>> Based on your request in the previous thread, we investigated whether >>> the boost was coming from the unintended side effects of KSM splitting >>> THPs. This wasn't the case. >>> If you have other theories on how the results could be bogus, we'd be >>> happy to investigate those as well. But you have to let us know what >>> you're looking for. >>> >> >> Maybe I'm bad at making such requests but >> >> "Stefan, can you do me a favor and investigate which pages we end up >> deduplicating -- especially if it's mostly only the zeropage and if it's >> still that significant when disabling THP?" >> >> "In any case, it would be nice to get a feeling for how much variety in >> these 20% of deduplicated pages are. " >> >> is pretty clear to me. And shouldn't take months. >> Just to clarify: the details I requested are not meant to decide whether to reject the patch set (I understand that it can be beneficial to have); I primarily want to understand if we're really dealing with a workload where KSM is able to deduplicate pages that are non-trivial, to maybe figure out if there are other workloads that could similarly benefit -- or if we could optimize KSM for these specific cases or avoid the memory deduplication altogether. In contrast to e.g.: 1) THP resulted in many zeropages we end up deduplicating again. The THP placement was unfortunate. 2) Unoptimized memory allocators that leave many identical pages mapped after freeing up memory (e.g., zeroed pages, pages all filled with poison values) instead of e.g., using MADV_DONTNEED to free up that memory. > > /sys/kernel/mm/ksm/pages_shared is over 10000 when we run this on an > Instagram workload. The workload consists of 36 processes plus a few > sidecar processes. Thanks! To which value is /sys/kernel/mm/ksm/max_page_sharing set in that environment? What would be interesting is pages_shared after max_page_sharing was set to a very high number such that pages_shared does not include duplicates. Then pages_shared actually expresses how many different pages we deduplicate. No need to run without THP in that case. Similarly, enabling "use_zero_pages" could highlight if your workload ends up deduplciating a lot of zeropages. But maxing out max_page_sharing would be sufficient to understand what's happening. > > Each of these individual processes has around 500MB in KSM pages. > That's really a lot, thanks. > Also to give some idea for individual VMA's > > 7ef5d5600000-7ef5e5600000 rw-p 00000000 00:00 0 (Size: 262144 KB, KSM: > 73160 KB) > I'll have a look at the patches today. -- Thanks, David / dhildenb