From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03563C4167B for ; Tue, 28 Nov 2023 07:13:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 617A96B0286; Tue, 28 Nov 2023 02:13:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F8436B02FD; Tue, 28 Nov 2023 02:13:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E1116B02FF; Tue, 28 Nov 2023 02:13:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3CEDC6B0286 for ; Tue, 28 Nov 2023 02:13:57 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F1C1980113 for ; Tue, 28 Nov 2023 07:13:56 +0000 (UTC) X-FDA: 81506498472.01.41D9967 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 4B979100020 for ; Tue, 28 Nov 2023 07:13:55 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=E6WQ8FwO; spf=pass (imf05.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701155635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fhOegZ/wwGmSsAku1dCJzHGzzuUFANNUfNm5UN7G3gI=; b=k9eUtkypHui4/p6IkJ/V6dDeEPratWfqq/xLL/IxIML9zLFAZ1T962lCbrRNmBajcCPBgK tDA4pceB1ujkIyuN83mFdz8jFSs5fx76qFrhuGHOxDn5ZcP96vF1K6OYatwI+SYecUwIva V8xv3PI3jFZc3oACGdIpfSTlh9Z89c8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701155635; a=rsa-sha256; cv=none; b=HlFgYuvmPDQZUFrn+Q/yuvJO1DHYkdAwvV8Fej4xadiF/mSitodPdul0r0DYTrwXUWo8No DSIUifeszbefLJ7x7yfeoftGVsz7D5/hd/8yu8l9PfDH0ndMK5tw1N4luyvwbchAva3Pbc DlXx9ly2Ii8QVGrJdYbIOaxzMCVC9cg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=E6WQ8FwO; spf=pass (imf05.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 3367461298; Tue, 28 Nov 2023 07:13:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5DCC6C433C7; Tue, 28 Nov 2023 07:13:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701155633; bh=IAmUZcFaVB6MmvVExaG8kzHoLyfFSQRpUzr1m1lRIkQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=E6WQ8FwO8/butwJcmjzgFlyNd5SS6M/+gXRWbqGKHkwlghDbdydfU3In6q0t4H9hL wjVOKmZrEAzpcPmAzz3wG0+GtX0ChN54XQXGgD0N6/LNVl9QWqHLFBTe3V2AJB0JUv 9bMATdMqYqHYjrJCTeaqQVBbSa5v4kRlFPZeStR+r0DIdzkxGYPZay8tqBh/3u+12N QXImKdMnpKHWbrdeDyfgTxnTQdhMYkN+oQGm8bkMS7anlH/Lxcub0S/H+R8SfaTsxX VBOwQOMI245ETsHOtUEB6Tf/6mZdnuCPtzGzrcn5h6pBgLvDFw/WkvmdSMErbL84eU BNREH1X4wIYLQ== Date: Tue, 28 Nov 2023 09:13:39 +0200 From: Mike Rapoport To: Serge Semin Cc: Thomas Bogendoerfer , Andrew Morton , Matthew Wilcox , Alexey Malahov , Arnd Bergmann , Aleksandar Rikalo , Aleksandar Rikalo , Dragan Mladjenovic , Chao-ying Fu , Jiaxun Yang , Yinglu Yang , Tiezhu Yang , Marc Zyngier , linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 5/7] mm/mm_init.c: Extend init unavailable range doc info Message-ID: <20231128071339.GJ636165@kernel.org> References: <20231122182419.30633-1-fancer.lancer@gmail.com> <20231122182419.30633-6-fancer.lancer@gmail.com> <20231123101854.GF636165@kernel.org> <20231124081900.GG636165@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: qob4kcdoci66n1dbsjwazygbtfrmprs9 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4B979100020 X-Rspam-User: X-HE-Tag: 1701155635-959557 X-HE-Meta: U2FsdGVkX19y2L+Yg6Jt7Zzh7JnvH85dTltw8dgYQgXn+njKj2awOkPzjahheDgn1LQBLE3N/vKbDjXH9BB6KAVH0OrErkB8+dPa+fWYa8m/4nHxi+0iRErrjL5UMTZC76wGC6IDxvKkMgOpZbUyK71yj607SL5QM1pj1BatozMvMnndNowDCaQSSpehWEVIz48tUhgkBiWEV82sh4CVRv0LZVv9nfEudIXUIdit1Uo03kGCTbcnIm7QDHdW+jwm4i5mM6bHZeQLLX6eJ67xcUChqHZzCVb+9IdauZu9h9FSywDYtrWL3NvNCO+zQCXpEuaMGFhR/IC4m5sNd3dHfNBMwPV2twKzl/ZIjRYN3E/PiFvrdMNgdxwgYEbAl3VBaTtyXanFqhL5QzNKvtXXF1/wZqU1eBl40cOwCNJ6PAfWTo023xNYPB+2JJ7XLRsoPnPxVQ24rFoi9wKMdDTcnsOOYCC6+mGhfz723NggM6hp3mXxPeXtcTFaankSQ4+niT/yFAHTwu6409h8SberY0p1Xy37kmz3siop3llJqE3Jhnebxu+FSbVYHMauZt5hauQyt8nciXk+JfDEd2bX8jVM/VGEcMdtnJ9+Jx79dlBQK3iRFR3hv74i5fSSuXjTOaKLnnPT4D/Vy+je9KUHBCAw7riwReScOZlmRXPUr0c/ykTpkksBdH7I0LklhbkKiX59W68AMSVP2Bc1PpYm/m/0rhrkYkQaW5oJzEaMCywd87Q9WO172LmVBe/EC6ODFfQnT7Bri9M37F0VUGb19ONv1nNjKEGanMMFGmrlDtEdb0jtfPNbTdfNXLJPU0CUbIa05f2qcmy+yejqAQehiay0mi5goY7N3FjK3qu/zFYdyeJJPsTkpmCDSqrux/njtaFCh456ubaZK59RVuyNvRpb7vsZSh/HlLsDxvSocWccmJTDAUMQmCwQVaLi1XfecR+YxOqBtBum08tVsbC Cms0T/0c p942GcT/NS0WrR/lZkLNCf57SYih4XEepL3fMG7mWN0fG+yEfPLZpeN8BGhYKkjrXHzCgpYelUGoELaC3OdVbibxgbRPl82SLb5z4Efwa1hV04ayQ41AZCJEYRXQdw7dMalPkHLUFBhOilB1FajOGrwrr1tqiU3uyvtznIJWyP4xPcmCbdLh9VrvX60L9BjdgP0EkXt5Sj5WoaWYmzYrK3sirFeRXWSpLZ4G+ncjMn9EErfPbyhmtrnhTMNVJmkNBeI9ohbzUHUoiqv4bEInN66zXPx6nV6hWFjARLn/02HPiEORdbvLnYBtTOq1CLziYCApQP7wsRcKml+SGmnR1dP6f5vx9oClP2a3bpwuCUexKFlUmUStJwm/f36Bp5pUzTg28aUZyedSwpi4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Nov 24, 2023 at 02:18:44PM +0300, Serge Semin wrote: > On Fri, Nov 24, 2023 at 10:19:00AM +0200, Mike Rapoport wrote: > > On Thu, Nov 23, 2023 at 01:42:39PM +0300, Serge Semin wrote: > > > On Thu, Nov 23, 2023 at 12:18:54PM +0200, Mike Rapoport wrote: > > > > On Wed, Nov 22, 2023 at 09:24:03PM +0300, Serge Semin wrote: > > > > > Besides of the already described reasons the pages backended memory holes > > > > > might be persistent due to having memory mapped IO spaces behind those > > > > > ranges in the framework of flatmem kernel config. Add such note to the > > > > > init_unavailable_range() method kdoc in order to point out to one more > > > > > reason of having the function executed for such regions. > > > > > > > > > > Signed-off-by: Serge Semin > > > > > > > > > > --- > > > > > > > > > > Please let me know if the IO-space pages must be initialized somehow > > > > > differently rather relying on free_area_init() executing the > > > > > init_unavailable_range() method. > > > > > > > > > > > Maybe I'm missing something, but why do you need struct pages in the > > > > IO space? > > > > > > In my case at the very least that's due to having a SRAM device > > > available in the middle of the MMIO-space. The region is getting > > > mapped using the ioremap_wc() method (Uncached Write-Combine CA), > > > which eventually is converted to calling get_vm_area() and > > > ioremap_page_range() (see ioremap_prot() function on MIPS), which in > > > its turn use the page structs for mapping. Another similar case is > > > using ioremap_wc() in the PCIe outbound ATU space mapping of > > > the graphic/video cards framebuffers. > > > > ioremap_page_range() does not need struct pages, but rather physical > > addresses. > > Unless I miss something or MIPS32 is somehow special/wrong in that > matter, but from my just got experience it actually does at least in > the framework of the __update_cache() implementation which is called > in the set_ptes() method (former set_pte_at()), which in its turn > is eventually invoked by vmap_range_noflush() and finally by > ioremap_page_range(). See the patch > [PATCH 3/7] mips: Fix max_mapnr being uninitialized on early stages > Link: https://lore.kernel.org/linux-mips/20231122182419.30633-4-fancer.lancer@gmail.com/ > of this series and the stack-trace of the bug fixed by that patch. > > Is it wrong that on MIPS32 ioremap_page_range() eventually relies on > the page structs? It has been like that for, I don't know, long time. > If so then the sparse memory config might be broken on MIPS32..( Do you mind posting your physical memory layout? If I understand correctly, you have a hole in your RAM and there is MMIO region somewhere in that hole. With FLATMEM the memory map exists for that hole and hence pfn_valid() returns 1 for the MMIO range as well. That makes __update_cache() to check folio state and that check would fail if the memory map contained garbage. But since the hole in the memory map is initialized with init_unavailable_range() you get a valid struct page/struct folio and everything is fine. With that, the init_unavailable_range() docs need not mention IO space at all, they should mention holes within FLATMEM memory map. As for SPARSEMEM, if the hole does not belong to any section, pfn_valid() will be false for it and __update_cache() won't try to access memory map. > > > In general having the pages array defined for the IO-memory is > > > required for mapping the IO-space other than just uncached (my sram > > > case for example) or, for instance, with special access attribute for > > > the user-space (if I am not missing something in a way VM works in > > > that case). > > > > > No, struct pages are not required to map IO space. If you need to map MMIO > > to userspace there's remap_pfn_range() for that. > > Is this correct for both flat and sparse memory config? In anyway > please see my comment above about the problem I recently got. > > > > > My guess is that your system has a hole in the physical memory mappings and > > with FLATMEM that hole will have essentially unused struct pages, which are > > initialized by init_unavailable_range(). But from mm perspective this is > > still a hole even though there's some MMIO ranges in that hole. > > Absolutely right. Here is the physical memory layout in my system. > 0 - 128MB: RAM > 128MB - 512MB: Memory mapped IO > 512MB - 768MB..8.256GB: RAM > > > > > Now, if that hole is large you are wasting memory for unused memory map and > > it maybe worth considering using SPARSEMEM. > > Do you think it's worth to move to the sparse memory configuration in > order to save the 384MB of mapping with the 16K page model? AFAIU flat > memory config is more performant. Performance is critical on the most > of the SoC applications especially when using the 10G ethernet or > the high-speed PCIe devices. > > -Serge(y) > > > > > > -Serge(y) > > > > > > > > > > > > --- > > > > > mm/mm_init.c | 1 + > > > > > 1 file changed, 1 insertion(+) > > > > > > > > > > diff --git a/mm/mm_init.c b/mm/mm_init.c > > > > > index 077bfe393b5e..3fa33e2d32ba 100644 > > > > > --- a/mm/mm_init.c > > > > > +++ b/mm/mm_init.c > > > > > @@ -796,6 +796,7 @@ overlap_memmap_init(unsigned long zone, unsigned long *pfn) > > > > > * - physical memory bank size is not necessarily the exact multiple of the > > > > > * arbitrary section size > > > > > * - early reserved memory may not be listed in memblock.memory > > > > > + * - memory mapped IO space > > > > > * - memory layouts defined with memmap= kernel parameter may not align > > > > > * nicely with memmap sections > > > > > * > > > > > -- > > > > > 2.42.1 > > > > > > > > > > > > > -- > > > > Sincerely yours, > > > > Mike. > > > > > > > > -- > > Sincerely yours, > > Mike. -- Sincerely yours, Mike.