From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CC10C30653 for ; Thu, 4 Jul 2024 13:28:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BE996B0093; Thu, 4 Jul 2024 09:28:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 447346B009A; Thu, 4 Jul 2024 09:28:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 273086B009B; Thu, 4 Jul 2024 09:28:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 00A5D6B0093 for ; Thu, 4 Jul 2024 09:28:39 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 810F7A12E3 for ; Thu, 4 Jul 2024 13:28:39 +0000 (UTC) X-FDA: 82302149958.23.97A4ED8 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf17.hostedemail.com (Postfix) with ESMTP id 8FF9940028 for ; Thu, 4 Jul 2024 13:28:36 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=T72QxSoq; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.219.53 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720099703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HzHi7K6AnkM89f012KBgFgO8w9r1OBzVl2izjnVI0tI=; b=m76dz6WNse62xROLExiFTaEq98ApDBfKMRkx7lDwWb2E0Sw904iWVH1ITdzqSM4RPjKzIT YwUP4khzfc6QkJDZcAf29UlX0OWqc3nDEE3HUelJVPSetgMhPluzW+bdl4Zp+XZr+gTE7w C7K1Ga5aAK8gZoHuQl+dfWeT+BDC+x0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=T72QxSoq; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.219.53 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720099703; a=rsa-sha256; cv=none; b=hLTx8bjFFa7IyAqeRGWMjRleTCqXM2cRf163K+bPDSDzIlRRaadm0jtDgp8156dYrpqxQy GeZmTBGbJK1nI2bY4w4v3gQzrvfb7oVqpZdVabAJ5QJTt2B6Lkff90rU+NT0RUGvr6BKtE vGNuRanK7kRtgrny5vMN+j0yxmgjVj0= Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6b5e0d881a1so13510086d6.1 for ; Thu, 04 Jul 2024 06:28:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720099715; x=1720704515; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=HzHi7K6AnkM89f012KBgFgO8w9r1OBzVl2izjnVI0tI=; b=T72QxSoqr75Rx8ghFt3vxafzGwiv4YTc+5kfibGLbUTIOTfU6nFJaI3wFZjr7Rbv67 MTNLOcnM0rdzUDWzmk84NVaIgr2AUH5B9zSo0VoiGBpvHYuYIEB2SdE7+IO8VaWGPUnO 20cWYTclF43XDcCuHbeKBYNnnRJ8jLrgopv3z7MJtk191gykfPg8eKe3LStiTHnyvkkG xtID7r8qRQf6prPDG/SYiX6eI6VCRch4MOr4I+5+2MvFYJqMOPd/1h92gKX0uSDp3vHi bH39HoySIYTE/GJbjauTGACTb8AeQ6VyGv3frJ7JeyEMfChwRvzhgAB5LjP2lNo9SRXT M1JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720099715; x=1720704515; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HzHi7K6AnkM89f012KBgFgO8w9r1OBzVl2izjnVI0tI=; b=W0XPpeZ1DITDjFpNFeyE0MCiOpaSDDbLi8ehlvSMxxp2D0EGGcHK0SgOFUtqnv173h OtcquPXHF8lQ3scUAZbdEeKW7370lHq7bI3KqPAHmm2s3pv2cIFk38HMXemWxWiA0VFO ORo3yb0CWLvrs1keZQT8LgMAKYxGMuTBra9RF0Ol7tCkuP0I/KAdNdVkCvYwEHbQlzS3 xIrXMtHR8u+e23nKBi80maOlUQ20X4RQpDa4qafonWRLCe4kZVdvlh+iivX1+zdGDMvf QpdyXfRHMg0rINGcHCbC1kEZnBWsvYNS7w/icWpNe+QoyRtWgvpvmtd/T6AbxtKaJSkk i8xA== X-Forwarded-Encrypted: i=1; AJvYcCV+dnsQKCtRUYt7yiK59YkvPabQXQXJW8xq1lng4lyh2HEXI1BR/U+Jcrz1DsuGgv8IFX4UnET0LBev4OAIk1bZaxM= X-Gm-Message-State: AOJu0Yxlu41CXYkfWp87pNM+6B6GD2KrcduKGY/q4H3BRSiiMQWkC/6o DpYlRrgPsdMDF3jW6hG7QKPAfjEFkml+wVLnhiPyegP8MaH+blaMFIRNsD8eyfiS476a+JEqmTH IqMjLdWTP/dSy0q4W3y1632ChNBg= X-Google-Smtp-Source: AGHT+IGxX3yz7enChXpJWyPVOuR53qPgU3wuQ646Fb7pZlVFt/CD/CIvHm1C4Ei4fi2op3R6Zx+WIvfYOhG/24xI5l8= X-Received: by 2002:a05:6214:1947:b0:6b5:684:2d5 with SMTP id 6a1803df08f44-6b5ee53130cmr24612716d6.4.1720099715499; Thu, 04 Jul 2024 06:28:35 -0700 (PDT) MIME-Version: 1.0 References: <20240701142046.6050-1-laoar.shao@gmail.com> <20240701195143.7e8d597abc14b255f3bc4bcd@linux-foundation.org> <874j98noth.fsf@yhuang6-desk2.ccr.corp.intel.com> <87wmm3kzmy.fsf@yhuang6-desk2.ccr.corp.intel.com> <87plrvkvo8.fsf@yhuang6-desk2.ccr.corp.intel.com> <87jzi3kphw.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87jzi3kphw.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Yafang Shao Date: Thu, 4 Jul 2024 21:27:59 +0800 Message-ID: Subject: Re: [PATCH] mm: Enable setting -1 for vm.percpu_pagelist_high_fraction to set the minimum pagelist To: "Huang, Ying" Cc: Andrew Morton , linux-mm@kvack.org, Matthew Wilcox , David Rientjes , Mel Gorman Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8FF9940028 X-Stat-Signature: 84nhxe7aojabca7jcs8tjrb9ti6z6xnj X-Rspam-User: X-HE-Tag: 1720099716-809815 X-HE-Meta: U2FsdGVkX18ng00En4aqrOFTuc1gFYHU2GJOVnlTfBScsJuMpgtCzBHdtJQiJ/z+2jIKfkwdiX0eNbZYoRnzCzpKXrAU38JPt7ObhBj0QbLjgbcfnweDZTA0f/GH+sTEQaAt6GIo+9NH/aSaQIivzJsapKqUKWiem/MeuXJAJaLx6+IH3u2ZqSdERGszm+Vew4bnPm0OputqFZ7GDP+TYb+bu6yC4hyn8f5QWoI4EQJ0y1X8h6zRTNzaQcRffESAAzSHumUjYjKr32M9Kg8B9bLCdDxMXes/WtNOspAnyVMNKMZ/FVAQ5Kn7hn4itPoI5uO988hWNionFt87wO+Zy5egHzTJGbo5buoNaCJJ7qOLuoFQprdl+2naATOVLgaRTsiQXetUh9VO4sE+utjIiDCgBgu1hHLDhGR9yrk0J7XKGAdB7fYIsVJlYMLQoZDblumcuSOL86eDb6IL/IKBhTQMBxH0j5mddHP/3LYVwlOellLzvIqh4DKxr8J7NCYbSy602Z22zKU1NPPZsjzvc+9iPMFe7xpaSZM/7mBeZ3+Bn5wDIaTE0W4ReAqT19rM/7nh8dWxozfMesJrl/Xb69gO7dt+ueNDHCJ5oshccOAwSfa5vMy2hFB1y27tw/FldPThvFJGzmiUPKcei94j9ukwdFYY5JeeBXgK5vuk5PcXNzqudhf7luPBj1Gb0GHVy5H7dMFvBD2MWnG4ShIl0tc7u3u3/MR79v0jESG1rrVPYZYSi5AXlXOxlehY/J7PHitiixUNDVU5Y3qzUqjqQfk6CDaME4i7ibebmmHMgqn4sh2Dfq2zJSpBSN3Irh4V3pcS5p27f5CxzxQjh0gfN8e0lT1qDepeEdrcuDixrmTlEG+Hv1k2jmuPWN0Qvn2QilqvjrtNfzKSae5K9Tjbg/RWxq1aeLYjJG5H8z+tOwnfi40KhtOQczSjlc6x1mCopq0iTGIqkNqMoayMWWo hDjMQfCG ylEBZreaKQW2BvXnWB/GkIStmC06FmQBpxTv5DMc8OzI37EAGgYpEh6sQaN3bIiJ5aipKu5Indiuy7JY3O53gETBg6l1uQiSPELS8jSFWLfqMKxbLwX1YzbaUHz3Vttll6NZ8/R9Q/a0MIiByd2CpkB23sJLXK0WmPWLSkXPVvWGR8M1scnEcZQQ6zvLCOY6/gWiCvi+kRdj92Vy3UCCPhYwGSQvGEWlEmiAGMwHQVchNv3u3Sv3xtGNhsNWapId0daP0iYrCGd4c8Uc92/jy8x8IYmdiZ9hl6vzSCP9t2SDg0iK78UTf1uCkwAkk6p430spO/xDMNYOpTLTIdXJR4TCeFGBL7ACTbVzrs45o/vN4R64GYzm7fOs0hOuKRuSrvplAoLjimbgZ7kzZsK96AOHeyodFLiGTEKKwYyKvdvHf5ABgurmvlL0A0mPMmBOGJ3WzN/3lJSGf06ry8ts2KR90pVnQHn9e/jBN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 3, 2024 at 1:36=E2=80=AFPM Huang, Ying w= rote: > > Yafang Shao writes: > > > On Wed, Jul 3, 2024 at 11:23=E2=80=AFAM Huang, Ying wrote: > >> > >> Yafang Shao writes: > >> > >> > On Wed, Jul 3, 2024 at 9:57=E2=80=AFAM Huang, Ying wrote: > >> >> > >> >> Yafang Shao writes: > >> >> > >> >> > On Tue, Jul 2, 2024 at 5:10=E2=80=AFPM Huang, Ying wrote: > >> >> >> > >> >> >> Yafang Shao writes: > >> >> >> > >> >> >> > On Tue, Jul 2, 2024 at 10:51=E2=80=AFAM Andrew Morton wrote: > >> >> >> >> > >> >> >> >> On Mon, 1 Jul 2024 22:20:46 +0800 Yafang Shao wrote: > >> >> >> >> > >> >> >> >> > Currently, we're encountering latency spikes in our contain= er environment > >> >> >> >> > when a specific container with multiple Python-based tasks = exits. These > >> >> >> >> > tasks may hold the zone->lock for an extended period, signi= ficantly > >> >> >> >> > impacting latency for other containers attempting to alloca= te memory. > >> >> >> >> > >> >> >> >> Is this locking issue well understood? Is anyone working on = it? A > >> >> >> >> reasonably detailed description of the issue and a descriptio= n of any > >> >> >> >> ongoing work would be helpful here. > >> >> >> > > >> >> >> > In our containerized environment, we have a specific type of c= ontainer > >> >> >> > that runs 18 processes, each consuming approximately 6GB of RS= S. These > >> >> >> > processes are organized as separate processes rather than thre= ads due > >> >> >> > to the Python Global Interpreter Lock (GIL) being a bottleneck= in a > >> >> >> > multi-threaded setup. Upon the exit of these containers, other > >> >> >> > containers hosted on the same machine experience significant l= atency > >> >> >> > spikes. > >> >> >> > > >> >> >> > Our investigation using perf tracing revealed that the root ca= use of > >> >> >> > these spikes is the simultaneous execution of exit_mmap() by e= ach of > >> >> >> > the exiting processes. This concurrent access to the zone->loc= k > >> >> >> > results in contention, which becomes a hotspot and negatively = impacts > >> >> >> > performance. The perf results clearly indicate this contention= as a > >> >> >> > primary contributor to the observed latency issues. > >> >> >> > > >> >> >> > + 77.02% 0.00% uwsgi [kernel.kallsyms] > >> >> >> > [k] mmput =E2=96= =92 > >> >> >> > - 76.98% 0.01% uwsgi [kernel.kallsyms] > >> >> >> > [k] exit_mmap =E2=96= =92 > >> >> >> > - 76.97% exit_mmap > >> >> >> > =E2=96= =92 > >> >> >> > - 58.58% unmap_vmas > >> >> >> > =E2=96= =92 > >> >> >> > - 58.55% unmap_single_vma > >> >> >> > =E2=96= =92 > >> >> >> > - unmap_page_range > >> >> >> > =E2=96= =92 > >> >> >> > - 58.32% zap_pte_range > >> >> >> > =E2=96= =92 > >> >> >> > - 42.88% tlb_flush_mmu > >> >> >> > =E2=96= =92 > >> >> >> > - 42.76% free_pages_and_swap_cache > >> >> >> > =E2=96= =92 > >> >> >> > - 41.22% release_pages > >> >> >> > =E2=96= =92 > >> >> >> > - 33.29% free_unref_page_list > >> >> >> > =E2=96= =92 > >> >> >> > - 32.37% free_unref_page_commit > >> >> >> > =E2=96= =92 > >> >> >> > - 31.64% free_pcppages_bulk > >> >> >> > =E2=96= =92 > >> >> >> > + 28.65% _raw_spin_lock > >> >> >> > =E2=96= =92 > >> >> >> > 1.28% __list_del_entry_v= alid > >> >> >> > =E2=96= =92 > >> >> >> > + 3.25% folio_lruvec_lock_irqsave > >> >> >> > =E2=96= =92 > >> >> >> > + 0.75% __mem_cgroup_uncharge_list > >> >> >> > =E2=96= =92 > >> >> >> > 0.60% __mod_lruvec_state > >> >> >> > =E2=96= =92 > >> >> >> > 1.07% free_swap_cache > >> >> >> > =E2=96= =92 > >> >> >> > + 11.69% page_remove_rmap > >> >> >> > =E2=96= =92 > >> >> >> > 0.64% __mod_lruvec_page_state > >> >> >> > - 17.34% remove_vma > >> >> >> > =E2=96= =92 > >> >> >> > - 17.25% vm_area_free > >> >> >> > =E2=96= =92 > >> >> >> > - 17.23% kmem_cache_free > >> >> >> > =E2=96= =92 > >> >> >> > - 17.15% __slab_free > >> >> >> > =E2=96= =92 > >> >> >> > - 14.56% discard_slab > >> >> >> > =E2=96= =92 > >> >> >> > free_slab > >> >> >> > =E2=96= =92 > >> >> >> > __free_slab > >> >> >> > =E2=96= =92 > >> >> >> > __free_pages > >> >> >> > =E2=96= =92 > >> >> >> > - free_unref_page > >> >> >> > =E2=96= =92 > >> >> >> > - 13.50% free_unref_page_commit > >> >> >> > =E2=96= =92 > >> >> >> > - free_pcppages_bulk > >> >> >> > =E2=96= =92 > >> >> >> > + 13.44% _raw_spin_lock > >> >> >> > > >> >> >> > By enabling the mm_page_pcpu_drain() we can find the detailed = stack: > >> >> >> > > >> >> >> > <...>-1540432 [224] d..3. 618048.023883: mm_page_pcp= u_drain: > >> >> >> > page=3D0000000035a1b0b7 pfn=3D0x11c19c72 order=3D0 migratetyp > >> >> >> > e=3D1 > >> >> >> > <...>-1540432 [224] d..3. 618048.023887: > >> >> >> > =3D> free_pcppages_bulk > >> >> >> > =3D> free_unref_page_commit > >> >> >> > =3D> free_unref_page_list > >> >> >> > =3D> release_pages > >> >> >> > =3D> free_pages_and_swap_cache > >> >> >> > =3D> tlb_flush_mmu > >> >> >> > =3D> zap_pte_range > >> >> >> > =3D> unmap_page_range > >> >> >> > =3D> unmap_single_vma > >> >> >> > =3D> unmap_vmas > >> >> >> > =3D> exit_mmap > >> >> >> > =3D> mmput > >> >> >> > =3D> do_exit > >> >> >> > =3D> do_group_exit > >> >> >> > =3D> get_signal > >> >> >> > =3D> arch_do_signal_or_restart > >> >> >> > =3D> exit_to_user_mode_prepare > >> >> >> > =3D> syscall_exit_to_user_mode > >> >> >> > =3D> do_syscall_64 > >> >> >> > =3D> entry_SYSCALL_64_after_hwframe > >> >> >> > > >> >> >> > The servers experiencing these issues are equipped with impres= sive > >> >> >> > hardware specifications, including 256 CPUs and 1TB of memory,= all > >> >> >> > within a single NUMA node. The zoneinfo is as follows, > >> >> >> > > >> >> >> > Node 0, zone Normal > >> >> >> > pages free 144465775 > >> >> >> > boost 0 > >> >> >> > min 1309270 > >> >> >> > low 1636587 > >> >> >> > high 1963904 > >> >> >> > spanned 564133888 > >> >> >> > present 296747008 > >> >> >> > managed 291974346 > >> >> >> > cma 0 > >> >> >> > protection: (0, 0, 0, 0) > >> >> >> > ... > >> >> >> > ... > >> >> >> > pagesets > >> >> >> > cpu: 0 > >> >> >> > count: 2217 > >> >> >> > high: 6392 > >> >> >> > batch: 63 > >> >> >> > vm stats threshold: 125 > >> >> >> > cpu: 1 > >> >> >> > count: 4510 > >> >> >> > high: 6392 > >> >> >> > batch: 63 > >> >> >> > vm stats threshold: 125 > >> >> >> > cpu: 2 > >> >> >> > count: 3059 > >> >> >> > high: 6392 > >> >> >> > batch: 63 > >> >> >> > > >> >> >> > ... > >> >> >> > > >> >> >> > The high is around 100 times the batch size. > >> >> >> > > >> >> >> > We also traced the latency associated with the free_pcppages_b= ulk() > >> >> >> > function during the container exit process: > >> >> >> > > >> >> >> > 19:48:54 > >> >> >> > nsecs : count distribution > >> >> >> > 0 -> 1 : 0 | = | > >> >> >> > 2 -> 3 : 0 | = | > >> >> >> > 4 -> 7 : 0 | = | > >> >> >> > 8 -> 15 : 0 | = | > >> >> >> > 16 -> 31 : 0 | = | > >> >> >> > 32 -> 63 : 0 | = | > >> >> >> > 64 -> 127 : 0 | = | > >> >> >> > 128 -> 255 : 0 | = | > >> >> >> > 256 -> 511 : 148 |***************** = | > >> >> >> > 512 -> 1023 : 334 |*************************= ***************| > >> >> >> > 1024 -> 2047 : 33 |*** = | > >> >> >> > 2048 -> 4095 : 5 | = | > >> >> >> > 4096 -> 8191 : 7 | = | > >> >> >> > 8192 -> 16383 : 12 |* = | > >> >> >> > 16384 -> 32767 : 30 |*** = | > >> >> >> > 32768 -> 65535 : 21 |** = | > >> >> >> > 65536 -> 131071 : 15 |* = | > >> >> >> > 131072 -> 262143 : 27 |*** = | > >> >> >> > 262144 -> 524287 : 84 |********** = | > >> >> >> > 524288 -> 1048575 : 203 |************************ = | > >> >> >> > 1048576 -> 2097151 : 284 |*************************= ********* | > >> >> >> > 2097152 -> 4194303 : 327 |*************************= ************** | > >> >> >> > 4194304 -> 8388607 : 215 |*************************= | > >> >> >> > 8388608 -> 16777215 : 116 |************* = | > >> >> >> > 16777216 -> 33554431 : 47 |***** = | > >> >> >> > 33554432 -> 67108863 : 8 | = | > >> >> >> > 67108864 -> 134217727 : 3 | = | > >> >> >> > > >> >> >> > avg =3D 3066311 nsecs, total: 5887317501 nsecs, count: 1920 > >> >> >> > > >> >> >> > The latency can reach tens of milliseconds. > >> >> >> > > >> >> >> > By adjusting the vm.percpu_pagelist_high_fraction parameter to= set the > >> >> >> > minimum pagelist high at 4 times the batch size, we were able = to > >> >> >> > significantly reduce the latency associated with the > >> >> >> > free_pcppages_bulk() function during container exits.: > >> >> >> > > >> >> >> > nsecs : count distribution > >> >> >> > 0 -> 1 : 0 | = | > >> >> >> > 2 -> 3 : 0 | = | > >> >> >> > 4 -> 7 : 0 | = | > >> >> >> > 8 -> 15 : 0 | = | > >> >> >> > 16 -> 31 : 0 | = | > >> >> >> > 32 -> 63 : 0 | = | > >> >> >> > 64 -> 127 : 0 | = | > >> >> >> > 128 -> 255 : 120 | = | > >> >> >> > 256 -> 511 : 365 |* = | > >> >> >> > 512 -> 1023 : 201 | = | > >> >> >> > 1024 -> 2047 : 103 | = | > >> >> >> > 2048 -> 4095 : 84 | = | > >> >> >> > 4096 -> 8191 : 87 | = | > >> >> >> > 8192 -> 16383 : 4777 |************** = | > >> >> >> > 16384 -> 32767 : 10572 |*************************= ****** | > >> >> >> > 32768 -> 65535 : 13544 |*************************= ***************| > >> >> >> > 65536 -> 131071 : 12723 |*************************= ************ | > >> >> >> > 131072 -> 262143 : 8604 |*************************= | > >> >> >> > 262144 -> 524287 : 3659 |********** = | > >> >> >> > 524288 -> 1048575 : 921 |** = | > >> >> >> > 1048576 -> 2097151 : 122 | = | > >> >> >> > 2097152 -> 4194303 : 5 | = | > >> >> >> > > >> >> >> > avg =3D 103814 nsecs, total: 5805802787 nsecs, count: 55925 > >> >> >> > > >> >> >> > After successfully tuning the vm.percpu_pagelist_high_fraction= sysctl > >> >> >> > knob to set the minimum pagelist high at a level that effectiv= ely > >> >> >> > mitigated latency issues, we observed that other containers we= re no > >> >> >> > longer experiencing similar complaints. As a result, we decide= d to > >> >> >> > implement this tuning as a permanent workaround and have deplo= yed it > >> >> >> > across all clusters of servers where these containers may be d= eployed. > >> >> >> > >> >> >> Thanks for your detailed data. > >> >> >> > >> >> >> IIUC, the latency of free_pcppages_bulk() during process exiting > >> >> >> shouldn't be a problem? > >> >> > > >> >> > Right. The problem arises when the process holds the lock for too > >> >> > long, causing other processes that are attempting to allocate mem= ory > >> >> > to experience delays or wait times. > >> >> > > >> >> >> Because users care more about the total time of > >> >> >> process exiting, that is, throughput. And I suspect that the zo= ne->lock > >> >> >> contention and page allocating/freeing throughput will be worse = with > >> >> >> your configuration? > >> >> > > >> >> > While reducing throughput may not be a significant concern due to= the > >> >> > minimal difference, the potential for latency spikes, a crucial m= etric > >> >> > for assessing system stability, is of greater concern to users. H= igher > >> >> > latency can lead to request errors, impacting the user experience= . > >> >> > Therefore, maintaining stability, even at the cost of slightly lo= wer > >> >> > throughput, is preferable to experiencing higher throughput with > >> >> > unstable performance. > >> >> > > >> >> >> > >> >> >> But the latency of free_pcppages_bulk() and page allocation in o= ther > >> >> >> processes is a problem. And your configuration can help it. > >> >> >> > >> >> >> Another choice is to change CONFIG_PCP_BATCH_SCALE_MAX. In that= way, > >> >> >> you have a normal PCP size (high) but smaller PCP batch. I gues= s that > >> >> >> may help both latency and throughput in your system. Could you = give it > >> >> >> a try? > >> >> > > >> >> > Currently, our kernel does not include the CONFIG_PCP_BATCH_SCALE= _MAX > >> >> > configuration option. However, I've observed your recent improvem= ents > >> >> > to the zone->lock mechanism, particularly commit 52166607ecc9 ("m= m: > >> >> > restrict the pcp batch scale factor to avoid too long latency"), = which > >> >> > has prompted me to experiment with manually setting the > >> >> > pcp->free_factor to zero. While this adjustment provided some > >> >> > improvement, the results were not as significant as I had hoped. > >> >> > > >> >> > BTW, perhaps we should consider the implementation of a sysctl kn= ob as > >> >> > an alternative to CONFIG_PCP_BATCH_SCALE_MAX? This would allow us= ers > >> >> > to more easily adjust it. > >> >> > >> >> If you cannot test upstream behavior, it's hard to make changes to > >> >> upstream. Could you find a way to do that? > >> > > >> > I'm afraid I can't run an upstream kernel in our production environm= ent :( > >> > Lots of code changes have to be made. > >> > >> Understand. Can you find a way to test upstream behavior, not upstrea= m > >> kernel exactly? Or test the upstream kernel but in a similar but not > >> exactly production environment. > > > > I'm willing to give it a try, but it may take some time to achieve the > > desired results.. > > Thanks! After I backported the series "mm: PCP high auto-tuning," which consists of a total of 9 patches, to our 6.1.y stable kernel and deployed it to our production envrionment, I observed a significant reduction in latency. The results are as follows: nsecs : count distribution 0 -> 1 : 0 | = | 2 -> 3 : 0 | = | 4 -> 7 : 0 | = | 8 -> 15 : 0 | = | 16 -> 31 : 0 | = | 32 -> 63 : 0 | = | 64 -> 127 : 0 | = | 128 -> 255 : 0 | = | 256 -> 511 : 0 | = | 512 -> 1023 : 0 | = | 1024 -> 2047 : 2 | = | 2048 -> 4095 : 11 | = | 4096 -> 8191 : 3 | = | 8192 -> 16383 : 1 | = | 16384 -> 32767 : 2 | = | 32768 -> 65535 : 7 | = | 65536 -> 131071 : 198 |********* = | 131072 -> 262143 : 530 |************************ = | 262144 -> 524287 : 824 |**************************************= | 524288 -> 1048575 : 852 |**************************************= **| 1048576 -> 2097151 : 714 |********************************* = | 2097152 -> 4194303 : 389 |****************** = | 4194304 -> 8388607 : 143 |****** = | 8388608 -> 16777215 : 29 |* = | 16777216 -> 33554431 : 1 | = | avg =3D 1181478 nsecs, total: 4380921824 nsecs, count: 3708 Compared to the previous data, the maximum latency has been reduced to less than 30ms. Additionally, I introduced a new sysctl knob, vm.pcp_batch_scale_max, to replace CONFIG_PCP_BATCH_SCALE_MAX. By tuning vm.pcp_batch_scale_max from the default value of 5 to 0, the maximum latency was further reduced to less than 2ms. nsecs : count distribution 0 -> 1 : 0 | = | 2 -> 3 : 0 | = | 4 -> 7 : 0 | = | 8 -> 15 : 0 | = | 16 -> 31 : 0 | = | 32 -> 63 : 0 | = | 64 -> 127 : 0 | = | 128 -> 255 : 0 | = | 256 -> 511 : 0 | = | 512 -> 1023 : 0 | = | 1024 -> 2047 : 36 | = | 2048 -> 4095 : 5063 |***** = | 4096 -> 8191 : 31226 |******************************** = | 8192 -> 16383 : 37606 |**************************************= * | 16384 -> 32767 : 38359 |**************************************= **| 32768 -> 65535 : 30652 |******************************* = | 65536 -> 131071 : 18714 |******************* = | 131072 -> 262143 : 7968 |******** = | 262144 -> 524287 : 1996 |** = | 524288 -> 1048575 : 302 | = | 1048576 -> 2097151 : 19 | = | avg =3D 40702 nsecs, total: 7002105331 nsecs, count: 172031 After multiple trials, I observed no significant differences between each attempt. Therefore, we decided to backport your improvements to our local kernel. Additionally, I propose introducing a new sysctl knob, vm.pcp_batch_scale_max, to the upstream kernel. This will enable users to easily tune the setting based on their specific workloads. --=20 Regards Yafang