From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47E08C369C2 for ; Thu, 24 Apr 2025 12:01:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C95FB6B00B1; Thu, 24 Apr 2025 08:01:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C1FA56B00B2; Thu, 24 Apr 2025 08:01:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6EA66B00B3; Thu, 24 Apr 2025 08:01:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 83C1B6B00B1 for ; Thu, 24 Apr 2025 08:01:47 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EA709121F1D for ; Thu, 24 Apr 2025 12:01:47 +0000 (UTC) X-FDA: 83368798254.15.2032897 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) by imf02.hostedemail.com (Postfix) with ESMTP id F389080007 for ; Thu, 24 Apr 2025 12:01:45 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=DStacqIY; dmarc=none; spf=pass (imf02.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.41 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745496106; a=rsa-sha256; cv=none; b=mYm74aZkT7RxHKXOGS7O7p4Ivc8n9bL7/eneg7RnrRAq27WEZr+Mnxa5npBmsenEgwbBb7 OEkcgLNiRuCjbCaFtag0WdIhTZd94cl9h9Mj+4nB4t/llZ2wq+w7Pn8AZbZYpfY7fQ6nZv +deqldbpvT21zc4VOoB5MHmsWB5BldU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=DStacqIY; dmarc=none; spf=pass (imf02.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.219.41 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745496106; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oZL+E3RNHbpFWOEVEZ+vdGqOh3AavrEOxiuPbcP0VN0=; b=ICByciMoUyfoRTkk8NCAVtiPpqiE2oZedNwxwuooqOHA9C6e2mkOruHjy3gFtegkFs2ulU ATaMIgeerO01c5HK7lJv27Gbr+GeWNy9U+taO04FdH0wCi+ADfg/1UX1/2aBf6mMw1A+Si W1K/0aqkYy1JWQ4VVzSorDHm2nVMoYw= Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-6e8f4c50a8fso9648716d6.1 for ; Thu, 24 Apr 2025 05:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1745496105; x=1746100905; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=oZL+E3RNHbpFWOEVEZ+vdGqOh3AavrEOxiuPbcP0VN0=; b=DStacqIYv2KavyAzoe7VF98uzZM1oi2VMcncrTUkvIEEerkJKsitNJLAfafJMKOl5T PjQrP24Q+ZfE6YaPQAwEbu+5biETpXjGEBiRui2xm9Tu577sGlUOR1TG4kJ8vCML5LNl EJ7xxbSI5/Hsj1Jli8JR1bZRN1Fz72jvExxcsT0G5sbKmoIDPe/LQVavVgoS//YtYO61 R3JyF/5+sbLCT8WT5JjijpejPCtp5sfVEyWDaBpxweaFjC/YDcLMat8EAFe8XRYb2pIl 7B2p2G8yIwiDjoVN/54gEhm7Br0mXc0pkF7bUYacyVmX82o+EdjQUUdvj2Es37yl6rXj WDpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745496105; x=1746100905; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=oZL+E3RNHbpFWOEVEZ+vdGqOh3AavrEOxiuPbcP0VN0=; b=NJc6wPV1Shkx/65eNkRjb5KQpPSsS76DVnkWcNcKwGxOmnxYw7b8JWBWIKlpmutru5 4n3kLju+5sObbUHKZOus29VEIXfdnDbSo8q6weX5xILwuzas7WAnsSCPvd/UqyE2R4UE +NbAV1/kzhzWQ6iUumWI+U7p4GINfgVNlsmMD0xdFd77ZSoMW5rGTZnxdRZ7VoiJKBcJ wfz6yzoSLfTlSo0REs28jZWKaFL1rmmUcQESNaC9NuibFLEE9GRBusPfyaAAzG46wQHf xOyifk+7TUDPJy4NP+g8GTxa76gSDS1j/B6+CPU2R+J0qzKjMkOEjdlDjy4bPDBLBICY tJSQ== X-Forwarded-Encrypted: i=1; AJvYcCV1d8ynOO/DULRRGbwTUpPPKmd5eTDipJ3qSHl+yYVzWEN5aathcTa7Dr9VOxLBSmRRDLk42cwLnw==@kvack.org X-Gm-Message-State: AOJu0YxZkOgb9LVRP/K9MgQ/z77jGs2ymyF3C1h7rT9ZqH48gUY4Np11 Y8APmPdqUePvyfIgL0T5d6EpXUuIatDCbb51YdN1OF4OIGKC41yFCtpXte1uimg= X-Gm-Gg: ASbGncvd2wCNVYPFo/WCdBEpkohREyXDidY9CVAp6PwYfr/xZBbdyg/ETDGoktFXC8H NH1/eeOTmKIZPPUCCB9If2ia94rouaxu/vzDq2mCv7JijxGxV/kWVN643VgbJU9jTq2hfrlFA3b 9I2ujlSXoM8lQVyWraFep6Ve0cbYYmgFgMaGeOcI7X/AXJq/TTuGZnni2xx3bOqi5MopVAuAqH8 oo6Go1UIYiID/j9C4REBgKCmRZ3kBL0BmP0/UCS9/nxu4snSBsswRDEWWGCHaguVjdVSwnGQ2GJ dCt5jzto0gU9iyLC4+Dpg/i7/HneZeO7zj32Tn78lVDuNlA6Dg08lEuBWAyj1FpK8C4MR9A7g4H 3RkyBIh3piq9Vaf9LbIY= X-Google-Smtp-Source: AGHT+IExjEYw2YciNBoIdkd9/pqHrN50d+y5G+B8wJfJbAXK4p9NBvT5uybfEubjkC+mcuj7maAkHA== X-Received: by 2002:a05:6214:d02:b0:6f2:d260:b2f4 with SMTP id 6a1803df08f44-6f4bfc95a23mr43959916d6.37.1745496104857; Thu, 24 Apr 2025 05:01:44 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-167-219-86.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.167.219.86]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6f4c093870bsm8528646d6.45.2025.04.24.05.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Apr 2025 05:01:44 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1u7vGx-00000007T99-2aD0; Thu, 24 Apr 2025 09:01:43 -0300 Date: Thu, 24 Apr 2025 09:01:43 -0300 From: Jason Gunthorpe To: jane.chu@oracle.com Cc: logane@deltatee.com, hch@lst.de, gregkh@linuxfoundation.org, willy@infradead.org, kch@nvidia.com, axboe@kernel.dk, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org Subject: Re: Report: Performance regression from ib_umem_get on zone device pages Message-ID: <20250424120143.GX1213339@ziepe.ca> References: <20250423232828.GV1213339@ziepe.ca> <84867704-1b25-422a-8c56-6422a2ef50a9@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <84867704-1b25-422a-8c56-6422a2ef50a9@oracle.com> X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: F389080007 X-Stat-Signature: dtxojfhcp5csordcxxaqk8uaaqsodhcp X-Rspam-User: X-HE-Tag: 1745496105-674947 X-HE-Meta: U2FsdGVkX1/epOI4JjQ0INNDqg4GUCmxqkFXUyYgmJPa22aOzgUhISdg0zjGqKCJRm0oVn9x4xTOlsaK7ickLa34oigcafGrxGPzgA6/9ENJMBrkk189jwxlRJ92ss4x7VaXwwzZDkWirHV51NEWkcv4JRcIvGYnnO9JHHshHMkzuOhEcf3A0mkmwIdW2Yfi1hIiXNvJED+aH8EH6Sk6RcsqGr5sEx3GTR5Q2qrSIHsIHDADfLqkEU50h8ZgGA8i6owtjWgBmIvk8YlRoqTulXvKVcQ7TGNUZrlzOu/ka5D10NiAiX1bzH+q4u5prTcCL+wLhwaWHgof7FZO/IFzgXP2NNmZ+nqn2TTTC78QAw7Vp6+Q3QnY5t3SJlwuR88TJXdeP0ze8v67OLmF92ngy78Rig1v8dRLkerr2fiNe0+KXninfzsQbpmC+nVjfzMSCZtVKzGh9AZQfanrPq8SoRjmUTmmOHIxfE0lW5n+mp8d66Gtt3hpNMb/iAkShxT7KWE+SNylmIZwO1GxZKBorUr+sg9Q/0x2ZrWMTH2dXLYnbSlJd+O/pHbq8R8/RGtegs698rWndHQkgCKr6LKDKsPJ5egsJ1I/bW+yY8flhV3l3w7pmTpZEqZ0vaFiTJEAhRxkqSHUwRAFPALRo2abt6rD4xQgBKzAXPBwGjs/4tvX96k9p08QPFzUYBqfRxaOF5Q54ipf8gOP6M6iZy+q7QeLJMojC3Iy821lHWpAXqUwKajIMZunAhFrHRCu65s5eKCkngSRv1d5iY9oVbSe8dHnsT9d4Wruvbxhq4MNESUf2NgiYwR0v2kmjQ9Yq+OGQ8WWDB4aRsh0BPFY08WvCl9t+OHHavrAjdGOBDLeemjuuYm9ddiOWvpNMmX4IJhIhIiSznnvi+7ioqYJb2ya4okTUe7BmLD0HyVlVi3ay28LyehOdRIivXRVKX6Zgj2Vc7TmSkeSH4uUimXGQ+b ZCfprFEk NdToS0evv8WswONvAZN86waav6TfMuXYHba/zLZJYvMJCd6yQyZCDg5FD3bEBti4lnsNYbTe/HkU2brucwJK7jCSqXlxkqO8ssDMgGxxsrzn1OkYq5ZKZMBIGguJM3qvQuKTUlG17V5rpOXhX00Jcu4zD9XVGlbXfCffjQQlkLXh5xucXd7EiirvzQMAKjPA3Ecc3abjWA8BeFSEOA+tpiEz6cevPDS7Gk5wwPRdVQ+6dqOtRJW3DYrxKtNFme9SqQv4jIdlVUHOzkHnTBWzXfFEI5+D7tLQi095d10sE72YP7CMG+OGjHRPH3M6tAcb21jbLBVr34WjPFwzm98cJBhrDxI7NLy9U+trL8RJ8fKv/DAE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Apr 23, 2025 at 10:35:06PM -0700, jane.chu@oracle.com wrote: > > On 4/23/2025 4:28 PM, Jason Gunthorpe wrote: > > > The flow of a single test run: > > > 1. reserve virtual address space for (61440 * 2MB) via mmap with PROT_NONE > > > and MAP_ANONYMOUS | MAP_NORESERVE| MAP_PRIVATE > > > 2. mmap ((61440 * 2MB) / 12) from each of the 12 device-dax to the > > > reserved virtual address space sequentially to form a continual VA > > > space > > Like is there any chance that each of these 61440 VMA's is a single > > 2MB folio from device-dax, or could it be? > > > > IIRC device-dax does could not use folios until 6.15 so I'm assuming > > it is not folios even if it is a pmd mapping? > > I just ran the mr registration stress test in 6.15-rc3, much better! > > What's changed? is it folio for device-dax? none of the code in > ib_umem_get() has changed though, it still loops through 'npages' doing I don't know, it is kind of strange that it changed. If device-dax is now using folios then it does change the access pattern to the struct page array somewhat, especially it moves all the writes to the head page of the 2MB section which maybe impacts the the caching? Jason