From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28505C636CC for ; Tue, 7 Feb 2023 04:37:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79D966B0074; Mon, 6 Feb 2023 23:37:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 74DCE6B0075; Mon, 6 Feb 2023 23:37:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EE1E6B0078; Mon, 6 Feb 2023 23:37:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4C9BD6B0074 for ; Mon, 6 Feb 2023 23:37:55 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1C1F51C6395 for ; Tue, 7 Feb 2023 04:37:55 +0000 (UTC) X-FDA: 80439238110.30.461EA68 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf18.hostedemail.com (Postfix) with ESMTP id 492091C0009 for ; Tue, 7 Feb 2023 04:37:53 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nHsuftEo; spf=pass (imf18.hostedemail.com: domain of jstultz@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=jstultz@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675744673; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w+0HCTBEcnsuxpAkRihzgpFaUXcz4XpnRKbh++S1fF8=; b=8lQN/uQkv2LdJcjW3RG7OBtu8hnNH0xlcXHz5VDomsF+eYDmSZpzEIozXrk+xmOV+TlaFU pXHNn4wX3bAyz8jcwk29XSx4kImvHIsuWd7QE5m6cZSb9Uz/mz2BjRxrK+c9uUA7wB3EAE h5NysRGATtjHM3nmBDV+ElC1DPCAsxU= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nHsuftEo; spf=pass (imf18.hostedemail.com: domain of jstultz@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=jstultz@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675744673; a=rsa-sha256; cv=none; b=3Xrv7l6Ma97DVvbWG0s1eJDkn1Z17w6uEpE68QiqwoVXfWLBWkDeExQP1qA/v9M5eEan5P CKqt3wXwA4KqCxBakniYl6rJGQYSwzv5kHL0euETfQ5mheQ26Tcvc0Dx/tFPDK/Mg4UKNr meqomnarMpItYzC7k9TWUEMY7+t9EI0= Received: by mail-yb1-f177.google.com with SMTP id 23so10072283ybf.10 for ; Mon, 06 Feb 2023 20:37:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=w+0HCTBEcnsuxpAkRihzgpFaUXcz4XpnRKbh++S1fF8=; b=nHsuftEo1aQKAm2ZVJ/nFPa+zRmJiiAJ7Qf7sD4IoESSTSeTpJDLVKSFlNEcbWekKT KdvTghoizpNwfkkusRa79/zDsO6GB0sSgC7prCC31jW2jV98LvutYHPv8BSK8gXYdhXo A3CBp4zptXI5fQ5V/v2yqYYiLin1bLJnS42gUbhcMk4PgH8RMfDs0/+r7Fk2+7DnZA1w DmRk/qB1WCLo8tyA03l2t65T5a09cDkzb6HCVNZ1KXH+bYEDoAdetqzzXuRc8E4nMcZW vezJNgkjvTufTswBsUjI0GfhOjMj6kNKd6YuNCQ5UcZ/wmK59xLhm8yB1AW3GhzBRnDL rSVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=w+0HCTBEcnsuxpAkRihzgpFaUXcz4XpnRKbh++S1fF8=; b=zrYwz4ZptFkrB8zY/5Dt2XZhWEWj77fiZ7CNT6SxU8MU5a43F9Hv6zWvIetLzP1MMI mX7b1bZtd4A6VFzx4tpCsU+TNAmyVHl1sSuF8gthkjyUBagKcujLAGj1NFQcJTUTOJ7x 38W+0cHs3Lb1Z/zeikUsAkktCN8AOaU5hnBuWszD9Xb0CK/cZ2jRqxfMKz2i4o3h/k3X I6kTqDHRscEAnuymuahjPUZV9F6UM537DdwO3yn5wFGSJfzexIh1SPn46dmyAf7JQC4n NHft0Rqu0DBnE2PWGl7ZMofCOBjRBaKU2at6iOudZi48oWpCtWS0Y0muWcxKOK1CuQTM H5nQ== X-Gm-Message-State: AO0yUKUkfAgBqjmUHQgDq8mLUGfT+A9h/B1F/NzFEgF+xui63Iq1DMmx CTgqSIOQSVY6lyuPyGZHe2sOia4XCBJSAA4udu6q X-Google-Smtp-Source: AK7set86jCFE95r0neU15sUjMrj5DOmSBeC4ESEUxmd0FJ68H6d3XGaqiZEAL2yyyMYLeaLE54lrOfm5ubSX6AA6x0k= X-Received: by 2002:a25:9e08:0:b0:7ba:e354:5aaf with SMTP id m8-20020a259e08000000b007bae3545aafmr246105ybq.37.1675744672187; Mon, 06 Feb 2023 20:37:52 -0800 (PST) MIME-Version: 1.0 References: <20230117082508.8953-1-jaewon31.kim@samsung.com> <20230117083103epcms1p63382eee1cce1077248a4b634681b0aca@epcms1p6> <20230125095646epcms1p2a97e403a9589ee1b74a3e7ac7d573f9b@epcms1p2> <20230125101957epcms1p2d06d65a9147e16f3281b13c085e5a74c@epcms1p2> <20230126044218epcms1p35474178c2f2b18524f35c7d9799e3aed@epcms1p3> <20230204150215epcms1p8d466d002c1e4dc2005d38f847adea6fa@epcms1p8> In-Reply-To: <20230204150215epcms1p8d466d002c1e4dc2005d38f847adea6fa@epcms1p8> From: John Stultz Date: Mon, 6 Feb 2023 20:37:40 -0800 Message-ID: Subject: Re: (2) [PATCH] dma-buf: system_heap: avoid reclaim for order 4 To: jaewon31.kim@samsung.com Cc: "T.J. Mercier" , "sumit.semwal@linaro.org" , "daniel.vetter@ffwll.ch" , "akpm@linux-foundation.org" , "hannes@cmpxchg.org" , "mhocko@kernel.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "jaewon31.kim@gmail.com" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 492091C0009 X-Rspam-User: X-Stat-Signature: 5w5z43mh6nn54m7zynxop19h5tadrcsu X-HE-Tag: 1675744673-311699 X-HE-Meta: U2FsdGVkX1/WKL/p1kUX5uDq1432fMpy8IwyJw7ecWintOHRa3ywR6iV3pA4ZjefzcYwXZcALInnWLJDXiG6q67OcGusa6lniko19Eou7OaaH2MgaWc9IxmkQRyuT6ic5s8UMFKzjRCE8Gga+6aDq5sNuXhWmnjiAuctTYc6isa55J247RewHCd9l4oNTcSUBUSqMHjLqscOCm2RJJQVSNr+U3ERzKTnJPYeqCyK2qpy7SX6qCDE7kYOP7L8cHS0aS0jFvf62Brnl72f8yX/kAfH3S0s+LTW+xpMPu66ucQ30OGpGlnjdxJh/T4lNsk3pcrvpu3S2TXrrjj9k0bAhI3YCZKljeKtCFqUy2L3Pl1l3ut3O9t01Yb/Ro5eBUDxNzWNstyh9QRoaxXvr7rFd/SFqXVHb+f0fiBjPQELWwF/SFnAe/iixdOIESFHKzLY622aU4NsdUmSQEaEqiBjU1fY1FMbhX5d9QPT3vQeedvlpa0a1N11ydgVMENMxgh7rXyx1IsFq6HFE8sGsd0tZYANSBRbFJK/D88ZyTzGxkkeSSl66veVjqTG1+o2nbf9jnWm2aYzSfJnmI3ZiYv0R/uCVRGyWxzavIus1EZHVYZRlO8lXIiu7XJnJ2EyRx4w9mSJSJ7KV4b/H2Gas/C6QHcfGfzUlKCZD+gQggXcWXsFqZzvO+vdn9CoIoTIDUJe/Z5rbcEI7PkQKwkZOcYOP+ASi6O53M/pLvJJIcKG7BcbMD3bAN7QJkGb3H09enftgkBPcPST9D8OgVzej4ESpNhft5kFDlljGMwN+fRRtjlZea/xf5X8+Fynk1pSKNkL6wPN7majkbwzWXCMWOrTJlistr6KJ1qoVbHvGUKKgi/pRo+fBQofaeSnF0qE5MwXtzzsIkbEWuKvr6tKk+ve9JZ3vqEHv8aINSqPRNetBN2bUPhhepnYueXJsOxzDK7diAOsQPd5yk/LuiQibN4 brpsTeSl 5r2GRYCSbAeiFwNXMLvRybUrMaK/9mIdnCGsBtC1XUwkec8uY0zTjhDjiXOWngna0nsMLOiPpg1fTYRPZJ4bXD/i2M/WFBIBdNN8PG4NGRYfKxKoHrZLwnuktF5GEepg37JWyIyP91FYx5w2I/l93CQaFJi4485XvSzJFXp9A3E7VdyJ4ers8b289ZOrR7J0ILKfqUKdtRgC3E4R+WITYMfk4nDrxI7hKuuBLlK+MvDbUZFqIujZyG07UvxQ8T57X8orF5Nqs/h7tsqsN0GVEgAEsi7xWvmwu/6rV5w7CgzEVYMPOCEEY6i6wW6ZQfQdwzyWBW1TOwpYfcOzb9gd8CKiSfVPGuhcBOAATyV1mR7b/IWgBuqt/EdXPkUixX03IcqwzfBrjtQB7FXVYQR5t7mKnDFz/JXbFD4MtrFytNKyfNr8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000027, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, Feb 4, 2023 at 7:02 AM Jaewon Kim wrote: > Hello John Stultz, sorry for late reply. > I had to manage other urgent things and this test also took some time to finish. > Any I hope you to be happy with following my test results. > > > 1. system heap modification > > To avoid effect of allocation from the pool, all the freed dma > buffer were passed to buddy without keeping them in the pool. > Some trace_printk and order counting logic were added. > > 2. the test tool > > To test the dma-buf system heap allocation speed, I prepared > a userspace test program which requests a specified size to a heap. > With the program, I tried to request 16 times of 10 MB size and > added 1 sleep between each request. Each memory was not freed > until the total 16 times total memory was allocated. Oof. I really appreciate all your effort that I'm sure went in to generate these numbers, but this wasn't quite what I was asking for. I know you've been focused on allocation performance under memory pressure, but I was hoping to see the impact of __using__ order 0 pages over order 4 pages in real world conditions (for camera or video recording or other use cases that use large allocations). These results seem to be still just focused on the difference in allocation performance between order 0 and order 4 with and without contention. That said, re-reading my email, I probably could have been more clear on this aspect. > 3. the test device > > The test device has arm64 CPU cores and v5.15 based kernel. > To get stable results, the CPU clock was fixed not to be changed > in run time, and the test tool was set to some specific CPU cores > running in the same CPU clock. > > 4. test results > > As we expected if order 4 exist in the buddy, the order 8, 4, 0 > allocation was 1 to 4 times faster than the order 8, 0, 0. But > the order 8, 0, 0 also looks fast enough. > > Here's time diff, and number of each order. > > order 8, 4, 0 in the enough order 4 case > > diff 8 4 0 > 665 usec 0 160 0 > 1,148 usec 0 160 0 > 1,089 usec 0 160 0 > 1,154 usec 0 160 0 > 1,264 usec 0 160 0 > 1,414 usec 0 160 0 > 873 usec 0 160 0 > 1,148 usec 0 160 0 > 1,158 usec 0 160 0 > 1,139 usec 0 160 0 > 1,169 usec 0 160 0 > 1,174 usec 0 160 0 > 1,210 usec 0 160 0 > 995 usec 0 160 0 > 1,151 usec 0 160 0 > 977 usec 0 160 0 > > order 8, 0, 0 in the enough order 4 case > > diff 8 4 0 > 441 usec 10 0 0 > 747 usec 10 0 0 > 2,330 usec 2 0 2048 > 2,469 usec 0 0 2560 > 2,518 usec 0 0 2560 > 1,176 usec 0 0 2560 > 1,487 usec 0 0 2560 > 1,402 usec 0 0 2560 > 1,449 usec 0 0 2560 > 1,330 usec 0 0 2560 > 1,089 usec 0 0 2560 > 1,481 usec 0 0 2560 > 1,326 usec 0 0 2560 > 3,057 usec 0 0 2560 > 2,758 usec 0 0 2560 > 3,271 usec 0 0 2560 > > From the perspective of responsiveness, the deterministic > memory allocation speed, I think, is quite important. So I > tested other case where the free memory are not enough. > > On this test, I ran the 16 times allocation sets twice > consecutively. Then it showed the first set order 8, 4, 0 > became very slow and varied, but the second set became > faster because of the already created the high order. > > order 8, 4, 0 in low memory > > diff 8 4 0 > 584 usec 0 160 0 > 28,428 usec 0 160 0 > 100,701 usec 0 160 0 > 76,645 usec 0 160 0 > 25,522 usec 0 160 0 > 38,798 usec 0 160 0 > 89,012 usec 0 160 0 > 23,015 usec 0 160 0 > 73,360 usec 0 160 0 > 76,953 usec 0 160 0 > 31,492 usec 0 160 0 > 75,889 usec 0 160 0 > 84,551 usec 0 160 0 > 84,352 usec 0 160 0 > 57,103 usec 0 160 0 > 93,452 usec 0 160 0 > > diff 8 4 0 > 808 usec 10 0 0 > 778 usec 4 96 0 > 829 usec 0 160 0 > 700 usec 0 160 0 > 937 usec 0 160 0 > 651 usec 0 160 0 > 636 usec 0 160 0 > 811 usec 0 160 0 > 622 usec 0 160 0 > 674 usec 0 160 0 > 677 usec 0 160 0 > 738 usec 0 160 0 > 1,130 usec 0 160 0 > 677 usec 0 160 0 > 553 usec 0 160 0 > 1,048 usec 0 160 0 > > > order 8, 0, 0 in low memory > > diff 8 4 0 > 1,699 usec 2 0 2048 > 2,082 usec 0 0 2560 > 840 usec 0 0 2560 > 875 usec 0 0 2560 > 845 usec 0 0 2560 > 1,706 usec 0 0 2560 > 967 usec 0 0 2560 > 1,000 usec 0 0 2560 > 1,905 usec 0 0 2560 > 2,451 usec 0 0 2560 > 3,384 usec 0 0 2560 > 2,397 usec 0 0 2560 > 3,171 usec 0 0 2560 > 2,376 usec 0 0 2560 > 3,347 usec 0 0 2560 > 2,554 usec 0 0 2560 > > diff 8 4 0 > 1,409 usec 2 0 2048 > 1,438 usec 0 0 2560 > 1,035 usec 0 0 2560 > 1,108 usec 0 0 2560 > 825 usec 0 0 2560 > 927 usec 0 0 2560 > 1,931 usec 0 0 2560 > 2,024 usec 0 0 2560 > 1,884 usec 0 0 2560 > 1,769 usec 0 0 2560 > 2,136 usec 0 0 2560 > 1,738 usec 0 0 2560 > 1,328 usec 0 0 2560 > 1,438 usec 0 0 2560 > 1,972 usec 0 0 2560 > 2,963 usec 0 0 2560 So, thank you for generating all of this. I think this all looks as expected, showing the benefit of your change to allocation under contention and showing the potential downside in the non-contention case. I still worry about the performance impact outside of allocation time of using the smaller order pages (via map and unmap through iommu to devices, etc), so it would still be nice to have some confidence this won't introduce other regressions, but I do agree the worse case impact is very bad. > Finally if we change order 4 to use HIGH_ORDER_GFP, > I expect that we could avoid the very slow cases. > Yeah. Again, this all aligns with the upside of your changes, which I'm eager for. I just want to have a strong sense of any regressions it might also cause. I don't mean to discourage you, especially after all the effort here. Do you think evaluating the before and after impact to buffer usage (not just allocation) would be doable in the near term? If you don't think so, given the benefit to allocation under pressure is large (and I don't mean to give you hurdles to jump), I'm willing to ack your change to get it merged, but if we later see performance trouble, I'll be quick to advocate for reverting it. Is that ok? thanks -john