From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E1CECE7AA6 for ; Fri, 6 Sep 2024 03:00:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE69D6B0085; Thu, 5 Sep 2024 23:00:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C96DB6B0088; Thu, 5 Sep 2024 23:00:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5DD36B0089; Thu, 5 Sep 2024 23:00:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 97ECD6B0085 for ; Thu, 5 Sep 2024 23:00:50 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3594240D4E for ; Fri, 6 Sep 2024 03:00:50 +0000 (UTC) X-FDA: 82532811060.09.8BEC044 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf06.hostedemail.com (Postfix) with ESMTP id 7404A18001C for ; Fri, 6 Sep 2024 03:00:48 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=t5kd0Kws; spf=pass (imf06.hostedemail.com: domain of surenb@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725591518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JTsFBCFFaXLue3y9fxeSovsc+6lMLee2VRnbsZWDcbI=; b=eqRC8cam0/qEvAX33YOe2SOO0ddrgBgNZ0cd9vMqa9opw8oOOD4iGWWO7SLq+CRTeisBRm qMWbaGVIKolaYqKry0GE6XFQKuXgvDV6+3vyXqMbpOQgDj//TipQhvAkpZG83Edtix3n3R iIhxZOFqOo2Bbj8syHXux8G56jhHbio= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=t5kd0Kws; spf=pass (imf06.hostedemail.com: domain of surenb@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725591518; a=rsa-sha256; cv=none; b=T892/2Pm8U9xWrJcJQk3pA8j5hUimwM3Ytjm1+9+IRwjQX+ixUXXfRLSSbrbfDEv7iZh7q YxmNJ91TnlcO4rsQrYEZt4tHDl8EWAcxgY59bBlQDnBbvbCkGGOeW6GNFOEQpVt7hMtU/2 u0UkQ/6JCkSTqRZV/hryQsPeHNZpYrU= Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-4567fe32141so99571cf.0 for ; Thu, 05 Sep 2024 20:00:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725591647; x=1726196447; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=JTsFBCFFaXLue3y9fxeSovsc+6lMLee2VRnbsZWDcbI=; b=t5kd0Kwsa0xSkccz4psqg2BhWiXbSmq2QKVi3uZYWILKyIzdVjBxm2pWEBM0dQUS6t /tKKqjrlouGHRDVTBsAodImm6Rs+wI8dgGWA0gVpjOpKGBWR03zgX7yA6tuqj8N37zY5 sBzKh9PRO77HPzkQmpSJN8RJEsrz/PTivL+ByaGX5VcXXXrhtsPcTkbN+4RRiRYttSPN LwoRl+LULB1f5viehOVjaz4rawIM92Ajc2fswysm0u7sDx8UE+a+T99X0BUT62ZJYXhb O0Oj3Vr2shbC67Fpwm54BPKfgGPnYgE0s6gm7hk6O6W7heKyY9DxMKOJwsdtGYVvL1JG PXFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725591647; x=1726196447; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JTsFBCFFaXLue3y9fxeSovsc+6lMLee2VRnbsZWDcbI=; b=vvto9QInC2rfBwy3s3DeBWB2AM6csJ9KWy40CObBVE6DOoF/u/gKPhtGJQNfsfpoQ3 wczNFRGOu/ZAfTkSfW8OLvWMy5Dvlez3HSqFUOpI1Y2nEFqzJQEF7hDuJJOF3bJ+4WTe aTSBHie2oXQtpCQ++CwvUk7GpbRb6R/zM8p+khcTHKYwyfuyIQ9UNbCnYoedBLKXHtZ2 CAOjdneViVVbxD+QjfAdxe/jeVu5GfYRQcS8ana7IEOD9qf1USTTWteOCu5PzvPf+4xA R6Oe5bFpsD/5XcbULGLf3XfK1IzqMAadxxlRjxnd7Ftx/1XCK3T+ZuVFaLFGWl5MjdPD 2ZxA== X-Forwarded-Encrypted: i=1; AJvYcCWeqXHe70h8ZjJceykIMHR3lnU5Jv7SfyY4poxvUtVeIE4BfxvYWM06VC28PPi2RRR1aPzZSt4vIg==@kvack.org X-Gm-Message-State: AOJu0YzcCu+qmgEvs7jjGUZMuOP5X+deU7Hax2jpCG5RaXGFyDnRJCnG DneNyu8/oYH9gFFEOfDqSUoBWylJN1+1FD45vVE6VI5N1H/VU79642MF8Nz+WfavUrqY3XRj9AW CjS0sKk788SleOaLi/kHEOShLp5rEVca4dmlJ X-Google-Smtp-Source: AGHT+IGFhlRcKSnHb7qnr3x+BEWd0gahj8Hw8W2cGgAhJIvbRKdGt/cZk32y1QzFyFfzc119DwH5CPb0vdMQd13+ZBc= X-Received: by 2002:ac8:5852:0:b0:456:9498:6b53 with SMTP id d75a77b69052e-4580e501817mr766401cf.16.1725591647062; Thu, 05 Sep 2024 20:00:47 -0700 (PDT) MIME-Version: 1.0 References: <20240903213649.3566695-1-yuzhao@google.com> <20240903213649.3566695-2-yuzhao@google.com> In-Reply-To: <20240903213649.3566695-2-yuzhao@google.com> From: Suren Baghdasaryan Date: Thu, 5 Sep 2024 20:00:35 -0700 Message-ID: Subject: Re: [PATCH mm-unstable v1 2/3] mm/codetag: fix pgalloc_tag_split() To: Yu Zhao Cc: Andrew Morton , Kent Overstreet , Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7404A18001C X-Stat-Signature: u9j6s3udoog95xy3kimr6jy7oxai8sy8 X-Rspam-User: X-HE-Tag: 1725591648-208102 X-HE-Meta: U2FsdGVkX1/CegcpDfvoh6OpSgsnbDCcYx2uv8CXHk27Aw6ZkAN8UjliK3QjSS1N9y9M3Y9SRGGBzvhqrfDoZ24rlXWEQ2PInhBfxesSfy1unsRfTj/Nxw475yMsz7OYBJBxD20/QpYeMcIMwPzv6mKK3J4oNmYlpXOZX/SJ5R9IgGEhUWMplFIe1Q3IA8k8D2TgDX4X8s3WBA5Mkwyw5Sc+SgjNuCwIa8cHVMVNU0RIYscXQF2b1GMg4Ub8wZRxDCYh1yF+UedqD+ihzNzaWxjkNXibWay7VW+Bhe16kZW7VylOa3YyxpmvIajAl6233/idMcBQIPAkJzwBLMyb/TvxWbs3C72vb7mbBMVWQNgekyTZDByOWtoQr99dtmtWTOe8WzwPFTOuZUEbwQz0EEeYtP9IrrvDDKT3zoR/sXUx8Rh2Dk47O63gKstAqrkHOcje/8mHMHLtLPrLyp+p8E/u5+2N2wHy6DXpAWR196lZIe2Hm0Kb+c8Z9+0mpcslZzTBmyvxOPwg8CqBLtTVJYtGq81ZhY5lTRBMq8DrC/UPqGt+iZg6MLELmUWCxTw4nL6guK4umXBHJxarG4xB3RLUdlRKsx9//SgW8QPtQbPtDX9bEaOsjhBL+wrZjSas3t5Fh6Zf8yOjIVtng9klgAfYIc8nb0BPwmP9dKgSHG+Ec/ehzYwlRbNg/crD0PMMRMpSaSmn2ok6Htiq67RldPhSQ7Ap5BOOx6galyUGAbdJeHLICi2auvmHDquOergb1gR9oE2znkxisJUyg7mfxB3dseskbBDpGksKh3/5VhMojXDSPRpn9/DA19mzhDaT16t5MAfQsY9l/uYb7sarDENgL5RdwtUcOMgAk4yORcONXhX7eF34zlJ8ECpmN/EZO32tYmwvwH2tBQ/kYJnAZITW17gQme7nlepRBfWQ+uJu966wbJEsWk7vkdlMsYNXOwx+XclyM2K0nXcf+Cn 48B1R7/w VrSDfQYE98Crf0/ckpIeL2QrSBPdpRqrMgiuvpnq0+v4C7N4pqvToRn7l+hWu+By9dB0JSYA9BKpSDbWe8eM7DXfzksUW7oU71lINhyOvtKC9GlW2IOCwHT6uoHWG+ZI/RlPlqkM+zGU9ZpN5UYbhs2ciUaSAfdaiqbb9M+tvd9Wj/fCVzL8qwHkeJuixXHuXpQgAIH2jRntEh8II2Q1SymgOX5/iMddnGx2MOxZQhPKsM8bbb06mMNQQvY1F5Hqbno8YLj4AS3CZWwo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Sep 3, 2024 at 2:36=E2=80=AFPM Yu Zhao wrote: > > Only tag the new head pages when splitting one large folio to multiple > ones of a lower order. Tagging tail pages can cause imbalanced "calls" > counters, since only head pages are untagged by pgalloc_tag_sub() and > reference counts on tail pages are leaked, e.g., > # echo 2048kB >/sys/kernel/mm/hugepages/hugepages-1048576kB/demote_size > # echo 700 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages > # time echo 700 >/sys/kernel/mm/hugepages/hugepages-1048576kB/demote > # grep alloc_gigantic_folio /proc/allocinfo > > Before this patch: > 0 549427200 mm/hugetlb.c:1549 func:alloc_gigantic_folio > > real 0m2.057s > user 0m0.000s > sys 0m2.051s > > After this patch: > 0 0 mm/hugetlb.c:1549 func:alloc_gigantic_folio > > real 0m1.711s > user 0m0.000s > sys 0m1.704s > > Not tagging tail pages also improves the splitting time, e.g., by > about 15% when demoting 1GB hugeTLB folios to 2MB ones, as shown > above. > > Fixes: be25d1d4e822 ("mm: create new codetag references during page split= ting") > Signed-off-by: Yu Zhao > --- > include/linux/mm.h | 30 ++++++++++++++++++++++++++++++ > include/linux/pgalloc_tag.h | 31 ------------------------------- > mm/huge_memory.c | 2 +- > mm/hugetlb.c | 2 +- > mm/page_alloc.c | 4 ++-- > 5 files changed, 34 insertions(+), 35 deletions(-) > > diff --git a/include/linux/mm.h b/include/linux/mm.h > index b31d4bdd65ad..a07e93adb8ad 100644 > --- a/include/linux/mm.h > +++ b/include/linux/mm.h > @@ -4137,4 +4137,34 @@ void vma_pgtable_walk_end(struct vm_area_struct *v= ma); > > int reserve_mem_find_by_name(const char *name, phys_addr_t *start, phys_= addr_t *size); > > +#ifdef CONFIG_MEM_ALLOC_PROFILING > +static inline void pgalloc_tag_split(struct folio *folio, int old_order,= int new_order) > +{ > + int i; > + struct alloc_tag *tag; > + unsigned int nr_pages =3D 1 << new_order; > + > + if (!mem_alloc_profiling_enabled()) > + return; > + > + tag =3D pgalloc_tag_get(&folio->page); > + if (!tag) > + return; > + > + for (i =3D nr_pages; i < (1 << old_order); i +=3D nr_pages) { > + union codetag_ref *ref =3D get_page_tag_ref(folio_page(fo= lio, i)); > + > + if (ref) { > + /* Set new reference to point to the original tag= */ > + alloc_tag_ref_set(ref, tag); > + put_page_tag_ref(ref); > + } > + } > +} > +#else /* !CONFIG_MEM_ALLOC_PROFILING */ > +static inline void pgalloc_tag_split(struct folio *folio, int old_order,= int new_order) > +{ > +} > +#endif /* CONFIG_MEM_ALLOC_PROFILING */ > + > #endif /* _LINUX_MM_H */ > diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h > index 207f0c83c8e9..59a3deb792a8 100644 > --- a/include/linux/pgalloc_tag.h > +++ b/include/linux/pgalloc_tag.h > @@ -80,36 +80,6 @@ static inline void pgalloc_tag_sub(struct page *page, = unsigned int nr) > } > } > > -static inline void pgalloc_tag_split(struct page *page, unsigned int nr) > -{ > - int i; > - struct page_ext *first_page_ext; > - struct page_ext *page_ext; > - union codetag_ref *ref; > - struct alloc_tag *tag; > - > - if (!mem_alloc_profiling_enabled()) > - return; > - > - first_page_ext =3D page_ext =3D page_ext_get(page); > - if (unlikely(!page_ext)) > - return; > - > - ref =3D codetag_ref_from_page_ext(page_ext); > - if (!ref->ct) > - goto out; > - > - tag =3D ct_to_alloc_tag(ref->ct); > - page_ext =3D page_ext_next(page_ext); > - for (i =3D 1; i < nr; i++) { > - /* Set new reference to point to the original tag */ > - alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), ta= g); > - page_ext =3D page_ext_next(page_ext); > - } > -out: > - page_ext_put(first_page_ext); > -} > - > static inline struct alloc_tag *pgalloc_tag_get(struct page *page) > { > struct alloc_tag *tag =3D NULL; > @@ -142,7 +112,6 @@ static inline void clear_page_tag_ref(struct page *pa= ge) {} > static inline void pgalloc_tag_add(struct page *page, struct task_struct= *task, > unsigned int nr) {} > static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {= } > -static inline void pgalloc_tag_split(struct page *page, unsigned int nr)= {} > static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { ret= urn NULL; } > static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned= int nr) {} > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index 0993dfe9ae94..aa8a4c938ba9 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -3244,7 +3244,7 @@ static void __split_huge_page(struct page *page, st= ruct list_head *list, > /* Caller disabled irqs, so they are still disabled here */ > > split_page_owner(head, order, new_order); > - pgalloc_tag_split(head, 1 << order); > + pgalloc_tag_split(folio, order, new_order); Looks like here and in the next diff you are fixing an additional bug. I was assuming that we are always splitting into order-0 pages, which is not the case anymore. It's worth mentioning that in the changelog. With that added: Acked-by: Suren Baghdasaryan > > /* See comment in __split_huge_page_tail() */ > if (folio_test_anon(folio)) { > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index 3faf5aad142d..a8624c07d8bf 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -3778,7 +3778,7 @@ static long demote_free_hugetlb_folios(struct hstat= e *src, struct hstate *dst, > list_del(&folio->lru); > > split_page_owner(&folio->page, huge_page_order(src), huge= _page_order(dst)); > - pgalloc_tag_split(&folio->page, 1 << huge_page_order(src= )); > + pgalloc_tag_split(folio, huge_page_order(src), huge_page_= order(dst)); > > for (i =3D 0; i < pages_per_huge_page(src); i +=3D pages_= per_huge_page(dst)) { > struct page *page =3D folio_page(folio, i); > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index c242d61fc4fd..13ce8e8899ed 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -2822,7 +2822,7 @@ void split_page(struct page *page, unsigned int ord= er) > for (i =3D 1; i < (1 << order); i++) > set_page_refcounted(page + i); > split_page_owner(page, order, 0); > - pgalloc_tag_split(page, 1 << order); > + pgalloc_tag_split(page_folio(page), order, 0); > split_page_memcg(page, order, 0); > } > EXPORT_SYMBOL_GPL(split_page); > @@ -5020,7 +5020,7 @@ static void *make_alloc_exact(unsigned long addr, u= nsigned int order, > struct page *last =3D page + nr; > > split_page_owner(page, order, 0); > - pgalloc_tag_split(page, 1 << order); > + pgalloc_tag_split(page_folio(page), order, 0); > split_page_memcg(page, order, 0); > while (page < --last) > set_page_refcounted(last); > -- > 2.46.0.469.g59c65b2a67-goog >