From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 613D8C61D88 for ; Tue, 21 Nov 2023 01:15:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E77066B03C0; Mon, 20 Nov 2023 20:15:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E00316B03C7; Mon, 20 Nov 2023 20:15:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7A536B03C8; Mon, 20 Nov 2023 20:15:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B7E4E6B03C0 for ; Mon, 20 Nov 2023 20:15:57 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8D1FF160AD6 for ; Tue, 21 Nov 2023 01:15:57 +0000 (UTC) X-FDA: 81480194754.24.36ECB62 Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) by imf30.hostedemail.com (Postfix) with ESMTP id A8A408000A for ; Tue, 21 Nov 2023 01:15:55 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Aa0JTvpQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700529355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AjY00jS05kwn5BSYUnYd0PcyI2gRMkkzxEGPgeY8Yfc=; b=w4D3yX/+4P2l/dllrT+dIMoJyfEDIqkGmcLXFkmgFmTma+Eacn5T7geCyED5Q8Mkxis9S5 JxydUEnZLLGFZ89ak5LY/Y5LB9B73GGeBCHb7ao786PIJtA0h6w7i2lGL8vwRZJwzOmfMe TH4DwtOdnxJcUyPNgWNxC5xAL6X8D0M= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Aa0JTvpQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.49 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700529355; a=rsa-sha256; cv=none; b=3hIqj0Dskqoph929a5iSdxQPIN8rOiOrNR86/8dX8do3P2Kfq6MylHGjQkVtuhqEKgmSB+ JkH+eY9qWzrNRt8wOrNHbs+2/MdX3T0KwBjd1evVaIaO3c4p+3o2KX3NGTKC4ZPVtgcTDS 843Ce8vRfsn7LBqsvw0FHO3tK/jIOC4= Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-548b54ed16eso2378314a12.0 for ; Mon, 20 Nov 2023 17:15:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700529354; x=1701134154; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=AjY00jS05kwn5BSYUnYd0PcyI2gRMkkzxEGPgeY8Yfc=; b=Aa0JTvpQz7xaIZCW4kZHV0mzgjJgU9t8dMM8C79XuXYbI0bQzd95tGYteqY/WeG1yN g67ilUIcp4iF3H9Qpecc4Nac3ciB0A6UWi53+rTFZ4Lxr+zJvpkIQEJQHYQ4l4kCkUqI xwh/7G6AExoqAJF6nD1P5K2C4FgNLXhOLhU9fJ9ah4gEmb7jEbQty7yEs7F/xT4q84k6 yqBcWTkoKNJ4YggNGcW/PM/rg3Pw4+H4LPEGFny6NZeB0bsme0iij1ZsAw1ckkFUzKhP CY32qwyW/PMnMj3pDkN2+jFLC4xZLBp2wx6V00liYsTRi4QECTdtw8b2hQ20K+Vc9Erz oqDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700529354; x=1701134154; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AjY00jS05kwn5BSYUnYd0PcyI2gRMkkzxEGPgeY8Yfc=; b=iOMKiubza1jelXNqcfDB39g3cXhBNklQdvOLNxSGFoqPAmmhBBrTzVb7Nu0gCEYMLO f4IdB9uRyc1V6lO/e6PWkq/VrM8opa7AsBcAvDaUVhG6aQKqj7JRzJuINKF1Zl3q4k9b 0C29934lzhgLVjEcl9fhvoqjP7pnHANYhpDum77mtmDtiWSmc8Eze4loNOJ4Zp5ok5hF 5yytL0IW3LRZa3UfwmlzWVxui/RjP8j6OWVHgqQuNwMhTHBWcj1zuCA5QHQKOjPmWEQS 0fF2yEpxU9wKSj96FvRkF+Hyge+i2GacdvXUCmiiWUGixvjlyTOxTbKTugG0/jViHc3V Gd5w== X-Gm-Message-State: AOJu0YyCtXD/DzXH18gWw1fgfWALTIWsiA3qhQATHlLWl+slVe7rw8zc Nq8IYWyefhCtsw7MS9jXYxa5yvgkf6hIjEhVt4rHcQ== X-Google-Smtp-Source: AGHT+IGE9JGytLkYwuXYv424QTiXwuMTGFYTcr3WNFwtS4AArg7TkdCIA5Vrl8cQyU8mdcb0zDHhO7HTUB9eu8pbIRo= X-Received: by 2002:a17:906:51d0:b0:9eb:af0e:39da with SMTP id v16-20020a17090651d000b009ebaf0e39damr6746294ejk.46.1700529353980; Mon, 20 Nov 2023 17:15:53 -0800 (PST) MIME-Version: 1.0 References: <20231113130601.3350915-1-hezhongkun.hzk@bytedance.com> <8734x1cdtr.fsf@yhuang6-desk2.ccr.corp.intel.com> <87edgkapsz.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: <87edgkapsz.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Yosry Ahmed Date: Mon, 20 Nov 2023 17:15:15 -0800 Message-ID: Subject: Re: [PATCH] mm:zswap: fix zswap entry reclamation failure in two scenarios To: "Huang, Ying" Cc: Chris Li , Zhongkun He , Andrew Morton , Johannes Weiner , Nhat Pham , Seth Jennings , Dan Streetman , Vitaly Wool , linux-mm , LKML Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: fq93kr4qxobxct1qykx49yyqsd6gu65j X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A8A408000A X-HE-Tag: 1700529355-634693 X-HE-Meta: U2FsdGVkX183Lo+Z6HKem8VmEBQi19S/ldw+TiCvb4AflbeaRQIJ5+JOz1s3F05n/zc8vgBUYlwRTPLFfg16tZg5gNHBL9Hv04pGt3YYr+N/lezGqlTWQLA5LuxJC21NcVwbiNEz7h9Dx1FYe02lh0fa3puj5QsL4JdTfDxZtotTCaSCO4yBX3bUk9WsLMpvHDUks9ycJv636x//7QZC6O2sLfnNnEi41VsW16kyH/sVdGmTeZdj4c+U25WrpUuXneUys/2hWfDdLAoz1IeWE4XgW1jya176YdgsHRrHElu4RTmNBamc5RGzYdV9AbZDZV839tnZDq1i7F+/x9HsSXtrKmYsshZtA1XjrPeiTgt2/+WMn45DAntoT8wuLIrTH7zUpbEmFy9pckza+jlK0XcoQCul/uSrWhO4IqWe3GlwYRUlXmjnV7OarDVZ8pVJnWgZJtVzCGyMGnSH+OD5jlPFTaVSpbwImnSmSnwjjDZ5+K3Em2CQIIX2cfzcZzNaCk5G7nXVBqgx6uqijjMqvja/n2dUaTZNIwlW1/d6fMNqay2MmBSC777czsRgARDxDIFnyOZzYRIKNESSUfpSWyIlymNzCBRJU6X2H6YVFlQtFtmrs3u5n9yv3/vB4Duaam1hDyYNwXUFB9OnhLKdQd/agzmtfAA9jUX1rrKvbIBw0f2eLU7gexbzYKNY7mUwgmagaer7tODQHiAY2e29UamAgZb3SUfii73OIz/dz2E5W+RVEdqIOMfzdp0Ahuy6aEv5FzbvayKlnq1ySUPTpxTV/sU914M+jo8wY+gzGg1px9Zpzhp+Z5f0EBKZDnzM7xhz+KVj+VdukY1rf3LUPjsNIX8hGstRidmsiFtyHPxEIiMokvI/WdiQqPqppkoTrZkIkJNyEZIJz1i9dCXdQIaf0FUD7PphwCyxEK3Rk/7UHfKTzgsaZRGpG4IsCXWOG4z/DC7Pw9oplS2/jU4 6H08eAUX BWW5HVBjvSDzhoXgpPwmtC0p585E2cFSOwPChaKasuxEVePYK1teXdBKCTmkO82PedRN+298aR5q/CxXtA3qzZMtT3fQG3aZ1FTHHCyFpeVMVVtqK//GBunDT7+v+D8yU+QuHihCSCnRhc99OxPe6JWuBMtC49LoGVULL+j4rLbzCq0zih0afEWNGNNgBjqFUI70pcdAGocTjBOA+7eUROgcva3b7pIqvxvVu06tJVQ2t4cuF3rLOJ/FvlUpunU/XiC8e3FviikI5kibieAHqQJ5tnw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Nov 20, 2023 at 4:57=E2=80=AFPM Huang, Ying = wrote: > > Yosry Ahmed writes: > > > On Sun, Nov 19, 2023 at 7:20=E2=80=AFPM Huang, Ying wrote: > >> > >> Chris Li writes: > >> > >> > On Thu, Nov 16, 2023 at 12:19=E2=80=AFPM Yosry Ahmed wrote: > >> >> > >> >> Not bypassing the swap slot cache, just make the callbacks to > >> >> invalidate the zswap entry, do memg uncharging, etc when the slot i= s > >> >> no longer used and is entering the swap slot cache (i.e. when > >> >> free_swap_slot() is called), instead of when draining the swap slot > >> >> cache (i.e. when swap_range_free() is called). For all parts of MM > >> >> outside of swap, the swap entry is freed when free_swap_slot() is > >> >> called. We don't free it immediately because of caching, but this > >> >> should be transparent to other parts of MM (e.g. zswap, memcg, etc)= . > >> > > >> > That will cancel the batching effect on the swap slot free, making t= he > >> > common case for swapping faults take longer to complete, righ? > >> > If I recall correctly, the uncharge is the expensive part of the swa= p > >> > slot free operation. > >> > I just want to figure out what we are trading off against. This is n= ot > >> > one side wins all situations. > >> > >> Per my understanding, we don't batch memcg uncharging in > >> swap_entry_free() now. Although it's possible and may improve > >> performance. > > > > Yes. It actually causes a long tail in swapin fault latency as Chris > > discovered in our prod. I am wondering if doing the memcg uncharging > > outside the slots cache will actually amortize the cost instead. > > > > Regardless of memcg charging, which is more complicated, I think we > > should at least move the call to zswap_invalidate() before the slots > > cache. I would prefer that we move everything non-swapfile specific > > outside the slots cache layer (zswap_invalidate(), > > arch_swap_invalidate_page(), clear_shadow_from_swap_cache(), > > mem_cgroup_uncharge_swap(), ..). However, if some of those are > > controversial, we can move some of them for now. > > That makes sense for me. > > > When draining free swap slots from the cache, swap_range_free() is > > called with nr_entries =3D=3D 1 anyway, so I can't see how any batching= is > > going on. If anything it should help amortize the cost. > > In swapcache_free_entries(), the sis->lock will be held to free multiple > swap slots via swap_info_get_cont() if possible. This can reduce > sis->lock contention. Ah yes that's a good point. Since most of these callbacks don't actually access sis, but use the swap entry value itself, I am guessing the reason we need to hold the lock for all these callbacks is to prevent swapoff and swapon reusing the same swap entry on a different swap device, right?