From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 873FAC4345F for ; Fri, 12 Apr 2024 10:29:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03B536B008A; Fri, 12 Apr 2024 06:29:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F06A16B008C; Fri, 12 Apr 2024 06:29:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D80356B0092; Fri, 12 Apr 2024 06:29:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B23A16B008A for ; Fri, 12 Apr 2024 06:29:55 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 497DF140C58 for ; Fri, 12 Apr 2024 10:29:55 +0000 (UTC) X-FDA: 82000509150.22.5A59DFF Received: from mail-vk1-f169.google.com (mail-vk1-f169.google.com [209.85.221.169]) by imf05.hostedemail.com (Postfix) with ESMTP id 791D5100018 for ; Fri, 12 Apr 2024 10:29:53 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="O+Aw/z7g"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712917793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RDA9Fsvooy18fjo8cTSK3tEnpx1k9+s98KjFedtdKvs=; b=KEj/mLYw1PX2VmE46Stiqd8dvOg9pVgSJr67q5LI7O6LzSFouy8oAyG1/i7s2WEa0nCkTf dwoJtc/saVw0P8y8W2RVVFX1bsUNHr9bplOLFA4+ntTsV1Ct3Ty5jB83kHo3WnGrU0uBbY YKBQwQSPRjF37PUc9IBHPSdNPv2nfQM= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="O+Aw/z7g"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712917793; a=rsa-sha256; cv=none; b=E5Iyj6Xi0p3SKqZnvQ0vYFKkEA4qaCY8QM6xPGzCk7sKUbI29ON710pY9GcrHpWcBrwnYY +6l1x8fN3+0LVzbUJBnmMqTdQwBvR7PAVGWBfPXiXdE6yI1oBdt60ZQoNWviqitiN9ogNk Ff+w2xRq/gKXTAKQq4wuRFuiiAHq6q0= Received: by mail-vk1-f169.google.com with SMTP id 71dfb90a1353d-4daa68eef71so258192e0c.0 for ; Fri, 12 Apr 2024 03:29:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712917792; x=1713522592; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=RDA9Fsvooy18fjo8cTSK3tEnpx1k9+s98KjFedtdKvs=; b=O+Aw/z7gErU4hj9lNSNB/WhxO0rESphqS/1/njaXmO3j+9Kt71ZyBzillcr6wfIo98 vsYYVGhupvRB1H/4hFWAvtkDBe5mnRUa7RJAFuu2cyS+qL1G8qFI809jRdOB6fYlthaH xYrosolxR9SER8riqtVbpTO4e4RaLMcAZrDU/e6gLjikcl9/zFGPKnGpQXuFJrlXJy7M fznjHCSM0ehnxAXIWro/oNPMhUXSFUkJ6gXVNnUGAW+i1fECD0s4oJIRaSYYFdh50zoh M+tRI25YP6OcZtiVPEjNOg19yA2eRAYTQZwlNaYsolVbgOXPSYW7OT1J7prm6aqqnxTn 42cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712917792; x=1713522592; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RDA9Fsvooy18fjo8cTSK3tEnpx1k9+s98KjFedtdKvs=; b=mVRVRRgg58ondN825n0nj2x14YviknhbYxQgzM6noYRIa4EJ83AZIPkHxjcmrGYZnk Z5hvy9MiRbq6Oj2rowpWZSxGB8tN0W6vxdYlnXTHQfFWBpsm8p1DKqn9hbfB+mBWmzL9 UioAETI25hFwhzHBwdFIrTViD8nZN6r5lF1WsrsJxBQATJ3G/M8+tUu+zlqVxiSGNRTh Xlc/5xHfiDVE7zDZ+1bFSY0w9lxCotknHVLKZH+XnxLbNmFPBoKjygzU+rCyrXmr0y3K hCDEFJYTYnpJ8wOuVyfnv/XQ1fwNsRQst2CGy/Q9/uadhok/SllKyYamEvi1+XnaJr66 JaNA== X-Forwarded-Encrypted: i=1; AJvYcCWyUVg6Jr0fnzmPq+hxNfFyvM/qBmjVVR+2MrU3Y6a+Rp3mL7+noN0sdTjyS+L7+plrx56Ry6nB1IKFiuVo4B/pNLs= X-Gm-Message-State: AOJu0YxGPOWx3IjaB8b3L0WGNZf85eQPUQjVFYrEbmXirF6sfSrm/ux3 iFVSN+rDBaom0AXBAEUbt3AZtRJKz5HWfstzliqFHhvKdkkrnEYSQ2sWtIDR0maqk/Fiw727A/k 8sb9X2fR3+5VPs+3RH8Eu8Hf+3Js= X-Google-Smtp-Source: AGHT+IGe6v65rFbNtZ9Y82mossVRD62kRKetDMzyaq+ykRQZYS0zZlj2LzN+WsFpiMna22CdaNC/uRl3TxSh9oAKCno= X-Received: by 2002:a05:6122:3c54:b0:4d8:4a7f:c166 with SMTP id fv20-20020a0561223c5400b004d84a7fc166mr2487170vkb.12.1712917792571; Fri, 12 Apr 2024 03:29:52 -0700 (PDT) MIME-Version: 1.0 References: <20240412101756.296971-1-21cnbao@gmail.com> <744fc49e-d91b-4f5a-9a27-1a25c50c2154@arm.com> In-Reply-To: <744fc49e-d91b-4f5a-9a27-1a25c50c2154@arm.com> From: Barry Song <21cnbao@gmail.com> Date: Fri, 12 Apr 2024 22:29:41 +1200 Message-ID: Subject: Re: [PATCH v5 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters To: Ryan Roberts Cc: akpm@linux-foundation.org, cerasuolodomenico@gmail.com, chrisl@kernel.org, corbet@lwn.net, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, peterx@redhat.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 791D5100018 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 5xghetxedmhj44rniyfdk8ec46abth1j X-HE-Tag: 1712917793-43409 X-HE-Meta: U2FsdGVkX1+7Z6EmkNUp2UwKQQPmWT5sH7dFA3UYHQfqcb+83GgXBbOuoEnJGj+A7+WUGCWgfqSVc9y1DONNFExVVNNBES7HgT5lkJPx01Itqxd2txKGja7nQjLDGKhwRK9La0+b+33EZF3jGgdT1J1g6q+lo9HFRMJgHU52DigNswuufcMh1TspCp1nQeHf2h0MN15iKwSGhHysAhk50ACSnRRlKKeIjo56rnExvZrs3MeYvOVx1C4AOwrQgsE+Y8mRi/QgVCMs+GtDw+HbGIBj4j2ZlBwtBTLVIuT4Xr9qLJ4z0TFgJmLX6jFHkpzcSGiykF4OqB5C08W9rLIxCyDD4XLh78FYmAMbqrH1fphvsADH2kUu/88c2z0LLLIxjgHU+V81OF7cACvsW/EeQxb7DcBqxcElrgbFlhGWsSloO+mbVwQyI2UfEBS3EnyIbbt7q9pZy2VppobQcQpg43mQsm2d3RfyH03jw3oxkp+bGIvaTnt66yc/W4vD1loxVVUKp5FHGCdmz9ghoUrby0L0thKLYxHEFi2Mfz0os9saxiaMxCoHcHFo9IDtqIsjrr0HCjiFmE+bNI86R7wFvHvd5yh+/cDWxHxHbbWNasH+rVOp+Vvm6vs6re0BHZHV4kARxlJAOXAmNRitUgw+l+9HDZ1+GIljhLOaUEGAKTtxCDBMfTvso2/SOffuafzvuDAYzeAE9OTzS0X4ufnZde0nAdD39QQbjmYjq9W7tsqXrXxDLlxl1iIKUeczhZgio8wxa5lF7rzF/PzdmNrlswsR9rDPyY/PrfmmkKjHmr/A3FGVWn3dHkMxNNMhf9hSRL+Kq9+/lxcvT8n8ENvxJ4mXa7zDkqbzGU2SiFQHmalzIVbLfzPxdsjihHatMnO4hV/uzEuHy46NuX0kV4CR4qTrhiI8Jtk6rRlYC+YtiuAwIHS/2PWrimxueToqyXGzpZlYXcTQK5lBhgfVKXh iv6lGqDD UVo7rbvPKHJDT6V1W9rlR6acywOHYlAsnMSYgAgZjFKA2efi6HPnPIvUIz5jRfH6dP0hkCxnlmxJxxBy946RibndslZJJsQedk6ZCRhvgzF0No/Qlmtwk9HF9zAKShGytgmpD/vDP7zalBFhUVaE7aCx+pKeomN1y3gIvD+JTGwcF8qwn7w9M1Eee2BE+v2qHX9DJ8Cs/spoyAhbWKCHWEjYH1LHapq0dpS47aweQd6oYVYlcHSehjCmbSDy+4B209+yWO3lxXd3Fo9sKsWNJ9tfVvXw7azNPQ/2s38zO04MS0dIcBwax+OKCIxiFRdONcECyXykWnS3dHmXIUK59FXSo/r6wDswY/3j3/AbqBjwpz+rBG8Gaq7CaqfymjIrILftEpScX8vpLozERXZ3hXMA1stRJG6HPAezO0Y3f8VCHXEq4ZAng6Y+z/8UkVaT7yo7aZ5tJbFu5nkJV4GX29OiT9fpykdc7LgOu54vUsGbyeksLzDD83rXR94deRGaAvU+HVGEvX8ft7hJ/Wc+wUytqpfKchludAoa9JmK5Sb4k8v656UdqxVxPkHxJdtNXEVCVS6k1mV4P1q2Wh43GrVbsWSWGulTOLeB3Dvi59YTPUJEWAg2xNDFjQsM+Scgt8sxGGdZMxOj5kyPvSktxyfUjKrRYJjZiWpeP5hEjMCKBNFzT+L9GldFkZpA3U64gjIeV9BPFbeJWE5TDYfAn21Bkiriv0bomCQbWcHorPccKPUEw2u07ZEhImGTvHTfUHT2bJhKF5uWP+4M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 12, 2024 at 10:25=E2=80=AFPM Ryan Roberts wrote: > > On 12/04/2024 11:17, Barry Song wrote: > > On Fri, Apr 12, 2024 at 9:56=E2=80=AFPM Ryan Roberts wrote: > >> > >> On 12/04/2024 10:43, Barry Song wrote: > >>> On Fri, Apr 12, 2024 at 9:27=E2=80=AFPM Ryan Roberts wrote: > >>>> > >>>> Hi Barry, > >>>> > >>>> 2 remaining comments - otherwise looks good. (same comments I just m= ade in the > >>>> v4 conversation). > >>>> > >>>> On 12/04/2024 08:37, Barry Song wrote: > >>>>> From: Barry Song > >>>>> > >>>>> Profiling a system blindly with mTHP has become challenging due to = the > >>>>> lack of visibility into its operations. Presenting the success rat= e of > >>>>> mTHP allocations appears to be pressing need. > >>>>> > >>>>> Recently, I've been experiencing significant difficulty debugging > >>>>> performance improvements and regressions without these figures. It= 's > >>>>> crucial for us to understand the true effectiveness of mTHP in real= -world > >>>>> scenarios, especially in systems with fragmented memory. > >>>>> > >>>>> This patch establishes the framework for per-order mTHP > >>>>> counters. It begins by introducing the anon_fault_alloc and > >>>>> anon_fault_fallback counters. Additionally, to maintain consistency > >>>>> with thp_fault_fallback_charge in /proc/vmstat, this patch also tra= cks > >>>>> anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. > >>>>> Incorporating additional counters should now be straightforward as = well. > >>>>> > >>>>> Signed-off-by: Barry Song > >>>>> Cc: Chris Li > >>>>> Cc: David Hildenbrand > >>>>> Cc: Domenico Cerasuolo > >>>>> Cc: Kairui Song > >>>>> Cc: Matthew Wilcox (Oracle) > >>>>> Cc: Peter Xu > >>>>> Cc: Ryan Roberts > >>>>> Cc: Suren Baghdasaryan > >>>>> Cc: Yosry Ahmed > >>>>> Cc: Yu Zhao > >>>>> --- > >>>>> include/linux/huge_mm.h | 51 ++++++++++++++++++++++++++++++++++ > >>>>> mm/huge_memory.c | 61 +++++++++++++++++++++++++++++++++++++= ++++ > >>>>> mm/memory.c | 3 ++ > >>>>> mm/page_alloc.c | 4 +++ > >>>>> 4 files changed, 119 insertions(+) > >>>>> > >>>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > >>>>> index e896ca4760f6..c5beb54b97cb 100644 > >>>>> --- a/include/linux/huge_mm.h > >>>>> +++ b/include/linux/huge_mm.h > >>>>> @@ -264,6 +264,57 @@ unsigned long thp_vma_allowable_orders(struct = vm_area_struct *vma, > >>>>> enforce_sysfs, orders); > >>>>> } > >>>>> > >>>>> +enum mthp_stat_item { > >>>>> + MTHP_STAT_ANON_FAULT_ALLOC, > >>>>> + MTHP_STAT_ANON_FAULT_FALLBACK, > >>>>> + MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, > >>>>> + __MTHP_STAT_COUNT > >>>>> +}; > >>>>> + > >>>>> +struct mthp_stat { > >>>>> + unsigned long stats[0][__MTHP_STAT_COUNT]; > >>>>> +}; > >>>>> + > >>>>> +extern struct mthp_stat __percpu *mthp_stats; > >>>>> + > >>>>> +static inline void count_mthp_stat(int order, enum mthp_stat_item = item) > >>>>> +{ > >>>>> + if (order <=3D 0 || order > PMD_ORDER || !mthp_stats) > >>>>> + return; > >>>>> + > >>>>> + this_cpu_inc(mthp_stats->stats[order][item]); > >>>>> +} > >>>>> + > >>>>> +static inline void count_mthp_stats(int order, enum mthp_stat_item= item, long delta) > >>>>> +{ > >>>>> + if (order <=3D 0 || order > PMD_ORDER || !mthp_stats) > >>>>> + return; > >>>>> + > >>>>> + this_cpu_add(mthp_stats->stats[order][item], delta); > >>>>> +} > >>>>> + > >>>>> +/* > >>>>> + * Fold the foreign cpu mthp stats into our own. > >>>>> + * > >>>>> + * This is adding to the stats on one processor > >>>>> + * but keeps the global counts constant. > >>>>> + */ > >>>>> +static inline void mthp_stats_fold_cpu(int cpu) > >>>>> +{ > >>>>> + struct mthp_stat *fold_stat; > >>>>> + int i, j; > >>>>> + > >>>>> + if (!mthp_stats) > >>>>> + return; > >>>>> + fold_stat =3D per_cpu_ptr(mthp_stats, cpu); > >>>>> + for (i =3D 1; i <=3D PMD_ORDER; i++) { > >>>>> + for (j =3D 0; j < __MTHP_STAT_COUNT; j++) { > >>>>> + count_mthp_stats(i, j, fold_stat->stats[i][j]= ); > >>>>> + fold_stat->stats[i][j] =3D 0; > >>>>> + } > >>>>> + } > >>>>> +} > >>>> > >>>> This is a pretty horrible hack; I'm pretty sure just summing for all= *possible* > >>>> cpus should work. > >>>> > >>>>> + > >>>>> #define transparent_hugepage_use_zero_page() = \ > >>>>> (transparent_hugepage_flags & = \ > >>>>> (1< >>>>> diff --git a/mm/huge_memory.c b/mm/huge_memory.c > >>>>> index dc30139590e6..21c4ac74b484 100644 > >>>>> --- a/mm/huge_memory.c > >>>>> +++ b/mm/huge_memory.c > >>>>> @@ -526,6 +526,50 @@ static const struct kobj_type thpsize_ktype = =3D { > >>>>> .sysfs_ops =3D &kobj_sysfs_ops, > >>>>> }; > >>>>> > >>>>> +struct mthp_stat __percpu *mthp_stats; > >>>>> + > >>>>> +static unsigned long sum_mthp_stat(int order, enum mthp_stat_item = item) > >>>>> +{ > >>>>> + unsigned long sum =3D 0; > >>>>> + int cpu; > >>>>> + > >>>>> + cpus_read_lock(); > >>>>> + for_each_online_cpu(cpu) { > >>>>> + struct mthp_stat *this =3D per_cpu_ptr(mthp_stats, cp= u); > >>>>> + > >>>>> + sum +=3D this->stats[order][item]; > >>>>> + } > >>>>> + cpus_read_unlock(); > >>>>> + > >>>>> + return sum; > >>>>> +} > >>>>> + > >>>>> +#define DEFINE_MTHP_STAT_ATTR(_name, _index) = \ > >>>>> +static ssize_t _name##_show(struct kobject *kobj, = \ > >>>>> + struct kobj_attribute *attr, char *buf) = \ > >>>>> +{ = \ > >>>>> + int order =3D to_thpsize(kobj)->order; = \ > >>>>> + = \ > >>>>> + return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index))= ; \ > >>>>> +} = \ > >>>>> +static struct kobj_attribute _name##_attr =3D __ATTR_RO(_name) > >>>>> + > >>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC= ); > >>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FA= LLBACK); > >>>>> +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_F= AULT_FALLBACK_CHARGE); > >>>>> + > >>>>> +static struct attribute *stats_attrs[] =3D { > >>>>> + &anon_fault_alloc_attr.attr, > >>>>> + &anon_fault_fallback_attr.attr, > >>>>> + &anon_fault_fallback_charge_attr.attr, > >>>>> + NULL, > >>>>> +}; > >>>>> + > >>>>> +static struct attribute_group stats_attr_group =3D { > >>>>> + .name =3D "stats", > >>>>> + .attrs =3D stats_attrs, > >>>>> +}; > >>>>> + > >>>>> static struct thpsize *thpsize_create(int order, struct kobject *p= arent) > >>>>> { > >>>>> unsigned long size =3D (PAGE_SIZE << order) / SZ_1K; > >>>>> @@ -549,6 +593,12 @@ static struct thpsize *thpsize_create(int orde= r, struct kobject *parent) > >>>>> return ERR_PTR(ret); > >>>>> } > >>>>> > >>>>> + ret =3D sysfs_create_group(&thpsize->kobj, &stats_attr_group)= ; > >>>>> + if (ret) { > >>>>> + kobject_put(&thpsize->kobj); > >>>>> + return ERR_PTR(ret); > >>>>> + } > >>>>> + > >>>>> thpsize->order =3D order; > >>>>> return thpsize; > >>>>> } > >>>>> @@ -691,6 +741,11 @@ static int __init hugepage_init(void) > >>>>> */ > >>>>> MAYBE_BUILD_BUG_ON(HPAGE_PMD_ORDER < 2); > >>>>> > >>>>> + mthp_stats =3D __alloc_percpu((PMD_ORDER + 1) * sizeof(mthp_s= tats->stats[0]), > >>>>> + sizeof(unsigned long)); > >>>> > >>>> Personally I think it would be cleaner to allocate statically using > >>>> ilog2(MAX_PTRS_PER_PTE) instead of PMD_ORDER. > >>> > >>> Hi Ryan, > >>> > >>> I don't understand why MAX_PTRS_PER_PTE is the correct size. For ARM6= 4, > >>> > >>> #define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) > >>> > >>> #define MAX_PTRS_PER_PTE PTRS_PER_PTE > >>> > >>> #define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3)) > >>> > >>> while PAGE_SIZE is 16KiB or 64KiB, PTRS_PER_PTE can be a huge number? > >>> > >>> > >>> Am I missing something? > >> > >> PTRS_PER_PTE is the number of PTE entries in a PTE table. On arm64 its= as follows: > >> > >> PAGE_SIZE PAGE_SHIFT PTRS_PER_PTE > >> 4K 12 512 > >> 16K 14 2048 > >> 64K 16 8192 > >> > >> So (PTRS_PER_PTE * PAGE_SIZE) =3D PMD_SIZE > >> > >> PMD_ORDER is ilog2(PMD_SIZE / PAGE_SIZE) =3D ilog2(PTRS_PER_PTE) > >> > >> MAX_PTRS_PER_PTE is just the maximum value that PTRS_PER_PTE will ever= have, > >> (and its equal to PTRS_PER_PTE except for powerpc). > >> > >> Pretty sure the math is correct? > > > > I am not convinced the math is correct :-) > > > > while page size is 64KiB, the page table is as below, > > PMD_ORDER =3D L2 index bits =3D [41:29] =3D 13 !=3D ilog2(8192) > > 1 << 13 =3D 8192 > > Right? So: > > ilog2(8192) =3D 13 > > What's wrong with that? > > I even checked in Python to make sure I'm not going mad: > > >>> import math > >>> math.log2(8192) > 13.0 You're correct. My mind fixated on the '16' in the line '64K 16 8192'. I mistakenly thought ilog2(8192) equals 16. Apologies for the confusion. > > > > > > > +--------+--------+--------+--------+--------+--------+--------+-------= -+ > > |63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 = 0| > > +--------+--------+--------+--------+--------+--------+--------+-------= -+ > > | | | | | > > | | | | v > > | | | | [15:0] in-page of= fset > > | | | +----------> [28:16] L3 index > > | | +--------------------------> [41:29] L2 index > > | +-------------------------------> [47:42] L1 index (= 48-bit) > > | [51:42] L1 index (= 52-bit) > > +-------------------------------------------------> [63] TTBR0/1 > > > > while page size is 4KiB, the page table is as below, > > > > +--------+--------+--------+--------+--------+--------+--------+-------= -+ > > |63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 = 0| > > +--------+--------+--------+--------+--------+--------+--------+-------= -+ > > | | | | | | > > | | | | | v > > | | | | | [11:0] in-page of= fset > > | | | | +-> [20:12] L3 index > > | | | +-----------> [29:21] L2 index > > | | +---------------------> [38:30] L1 index > > | +-------------------------------> [47:39] L0 index > > +-------------------------------------------------> [63] TTBR0/1 > > > > PMD_ORDER =3D L2 index bits =3D [29:21] =3D 9 =3D ilog2(512). > > > > You are only correct while page size =3D 4KiB. > > > > > > > > >