From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB96C02188 for ; Mon, 27 Jan 2025 18:54:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82412280197; Mon, 27 Jan 2025 13:54:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AD49280191; Mon, 27 Jan 2025 13:54:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64E53280197; Mon, 27 Jan 2025 13:54:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 417D0280191 for ; Mon, 27 Jan 2025 13:54:15 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C4DCF8037E for ; Mon, 27 Jan 2025 18:54:14 +0000 (UTC) X-FDA: 83054132028.20.0A57D72 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) by imf20.hostedemail.com (Postfix) with ESMTP id A449C1C0009 for ; Mon, 27 Jan 2025 18:54:12 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=WHVspOA6; spf=pass (imf20.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.49 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738004052; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sIyBoJGroUyTZpx6n/1BSJQGh7wzm8syWELiQ274Xzo=; b=Eirc/ne9JruqDP84Dy6LjMAJiaD/d1aaCoC8VKQO0dVKR+2G03BlesnU1CntWXU1uz+xjb mqyxkTcH8INzUrhG+BC5/HA5/66UF4bi8lDAtA91DBb2RLMzhW91liajGJwp5UwLzzmtBZ A62A1Mjxc8PkwXRtkJ9BZIKeiWPeXXE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=WHVspOA6; spf=pass (imf20.hostedemail.com: domain of axboe@kernel.dk designates 209.85.166.49 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738004052; a=rsa-sha256; cv=none; b=07yz0izQja/WTQGOLMeN/SuaGtAQyGYkyDkBEnzDhGJxtvP/D9sEX+7JOBJQVJe+nEInrG UsOEN+vD+DVtTXn2kmjVmGeMVMOOOWxpS01VoQbdJlmQQ2YKxy8VU4WIrVQjnkV/2r67jK MzL17K/6dlQIEhOTXt/Foc2RpNvBiFc= Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-851c4ee2a37so324919239f.3 for ; Mon, 27 Jan 2025 10:54:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1738004051; x=1738608851; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=sIyBoJGroUyTZpx6n/1BSJQGh7wzm8syWELiQ274Xzo=; b=WHVspOA6Iz9IbgVQEFzgi2Q9eYMPUhRRSMcQEnAWoRLqBpb2EdII8ESH4VKQEd0IxK mXsiYH/JlNYfqQui0RZueSvy0Im0c6hwKhedimbE+fpMDFqaWLF6nmtslUD1jg9psW+f wlXziYg6WVmIcUNfc0GsCoq2E1PhpB3XSGFA/FM6NfZVtrtZQ7U+xjaWZY/cXEcKqtoe h2EFgXMvkYG5tlXS0u19jMEHij1eL6aBn09ZgI4n/Xs4HU2oeo73WZgPzVhx1x8Ph+Rp N0Fzbejulz+Lckpv9sGeD+gJiOozXu3CExcCZ6WVXws0s2mW/ClTmtZkfUbau+mPE0lD vuYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738004051; x=1738608851; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sIyBoJGroUyTZpx6n/1BSJQGh7wzm8syWELiQ274Xzo=; b=mSl/HRbMg1HGmcDojDlV5U7s0o8xy8ONxacGPpD+hH3+IRpPab8ag/nuCYzgcLfoOG 4FjefwqlnnILukUr03yWC/QYLv4YQhlR0Rw6Ui97o6btA5ZreJUOVJLyQueKDy4UyHXt +AFfuRvbE2kQHc0PJPVOXZ9nEk0xeyqYxR0D/3iFZcT/mobPEI+ftADwHaXkZ5pYqh/B GDJuwYF9XG9Bs/gFrxdilUFy/dI8mXEb3XYOXDBEpDrC/n095Is2zwn2EwlveNhj0znw YLS0/3puRMdTI3PXifmSo64QVP4pChZjic5OuuZfl6GuBRUyC9OUoAaXSjyhPq3nSIby ynTQ== X-Forwarded-Encrypted: i=1; AJvYcCWkJ8vXJNEfMLbPyvy2vH39sDzjbdJdAuW6EB8peFqsd+7dvTkNNj0jCN+dWPzmfIf4wKFNEZ6hgg==@kvack.org X-Gm-Message-State: AOJu0YyLasLFDIUZWmiQpst245SQCHhxZPfDTcr5LwFEWry9dFxZrJni nSo+SDi5aMNe8REH4VC3pM1Wbo//xORMVxCNcYkxJxbZpJ+Oj9qlBXkPGeGhVUo= X-Gm-Gg: ASbGnctcslEV5kDG4N6mkba+PyPDxT1kgW1vTOWmD9mm3YjTv3+JlA50SUMXlfiq1Xe 6JCfHfDJwrw1ykipVcLe/jzGwrtWj/2rS4t91LgtAMplxzNSd6M2Gd3aS1A6VbI1ystQcOLvnB8 J3jd8ewzmjHqNP6Y4b0abrUByneSUWUgmnpXWAzb+Lh8fxwTGYSrIZyKKKEIAGxbsuldPaVaiYr Udsg2jk55y2b1cAUCUQ02DWv7KJFwDrMK6GLCSzhYOTjZM/uaI6jdukSgxeCsL/9/WtLkDcO0WB Sg== X-Google-Smtp-Source: AGHT+IFv9j+W6RuVCqdh4BJBXNTF9oEOZFuXXEm2c24r/6X+ifQU1oBGtRPGZlVEgbHvG/0hDG84JA== X-Received: by 2002:a05:6602:6d1a:b0:84c:ea14:931f with SMTP id ca18e2360f4ac-851b627fbd6mr3512940939f.11.1738004051522; Mon, 27 Jan 2025 10:54:11 -0800 (PST) Received: from [192.168.1.150] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id ca18e2360f4ac-8521dbb3acdsm288925439f.0.2025.01.27.10.54.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 10:54:10 -0800 (PST) Message-ID: <108b6ae7-a272-482b-b3da-60f1fc6617ee@kernel.dk> Date: Mon, 27 Jan 2025 11:54:09 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Direct I/O performance problems with 1GB pages To: Andres Freund , David Hildenbrand Cc: Matthew Wilcox , linux-mm@kvack.org, linux-block@vger.kernel.org, Muchun Song , Jane Chu , Pavel Begunkov References: <6ulkhmnl4rot5vrywoxvoewko7vbgkhypcwxjccghdu26kwsx5@bnseuzrsedte> Content-Language: en-US From: Jens Axboe In-Reply-To: <6ulkhmnl4rot5vrywoxvoewko7vbgkhypcwxjccghdu26kwsx5@bnseuzrsedte> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A449C1C0009 X-Stat-Signature: zj4dunwmgwfpjz4mod95fgoje9q7tt99 X-Rspam-User: X-HE-Tag: 1738004052-867617 X-HE-Meta: U2FsdGVkX1+ase0PgLlgBZ7DIH/9l00T5ZMZLIY6fryiGf2qf1dwoPPbLAyoY5aAjQ37A6v4b9a+VmNDZHpTbxTsdl1GInWxWxb5h5EZLdW+NighmMytiT3BQqe6MsCr66CRw4lW8/Ozve0Ih34TVfiBbbpm7X73cq6ku5w8hEyGbD/hvhJBHWG5GV+NaOCk+XcZC7oAQPwEhWnl3JIt7tIs+xvIOJl6qnR/CoeVoJou9LSP2C812BODyu6q0dUjADl22X7SYS1ks4J44MemdK+N8PgCq/0tJ4//WS6sjmw4vv1EeTVbYW5Q9FD0RgupiAEsSS4gjhDCwtG+BU99R978xMfY1mIp9XCoWyafXOxZAr2YmVFebflmeVhMHo4kBSveWfFAQoxpyYMMRqTGdDPDopAnWmONljBikQ3k/AdvkfxSH510ffZDbQDA82BiAOf3fDHpNgQh9q6BP+4WxYevZRHHfjG97DomffA0fU9K6NcDry+jsachhSmGNwn0AabCEtg6XkIaq+FIqcys2ddjCnt7m6KPhAbWyJHEdhbIhiIPG3cpFTDP3FRjhWstW1HtMhMBL109NAQNXIYuMdMCtvfRlsC2iDb5Ue8nB7kIduB69m4r96xH8DhTdp09PN7iNNEYAkY+uBV0b/KnFtSfc5o5WVJwUL/PBwSpVc1DC0/y/e/B3Y8nlc4ocyOJiwbpsmd3OU0X1at8yoElO2+tDSXayynxU93wSFOsw5mbyLbqVgrqdPSsskgKw7gQjW0TDtLr/hnDS1+yUygfMt76vOHZqAkh9TYb7xTvhXsHimLLMzyxe/lItLn7T/qAgmIR8tzRPMZzknb8LWG4wO3yBVLn3j0Llpzy3VCQOgcWaG1Ey5DpLZ6PCGaxBS9pub/mKuyqx3h0Ys1iatEq06vorl2lkepdimrusfq/sali5FglZgr6PuNO4gzpt6aV0X6cwUJqdDzDUe3YD+R bWALFQgM JzGfP6Ag6f0MOdiTcsTNYMscmCf7UG5p6Nxe98wAznLEQ7btYszoCYhNeP2Gr1oSkP4pw1+dh6sAqaw6VIxw+bkaOy6d6m6HKUmdekuSZ7nUwrGiQpHijptmiY4GuBfKn83bN/ICFr1da0Evr0lg3fUYYVO+fXfTVO2d7JqUu/N5LdzAnoc0TfWZYhD7cUOvnMGrtwG78/1JbXHrh5wsBJMU9aU9NRx8QjsWIxy64Ytu63QGjLQOqqp8DT+LqjGASkPhY4gHZg+652+S+dKxRa7Nn7wcoPtpLJVaej1ZP++yQ+xh4L5Cyniw2grd3gmfhE/FC5sdoq458xOnYOdeNzkMObS+zI9gkIA9bwskHaA4nrauYVye79Oz4SCCCSWA+8TOVctYwpoUTZKKD6uLZLm+U0E/gd0V3/i2E/ShCyQTDY3o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000015, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 1/27/25 11:21 AM, Andres Freund wrote: >> That's precisely what io-uring fixed buffers do :) > > I looked at using them at some point - unfortunately it seems that there is > just {READ,WRITE}_FIXED not {READV,WRITEV}_FIXED. It's *exceedingly* common > for us to do reads/writes where source/target buffers aren't wholly > contiguous. Thus - unless I am misunderstanding something, entirely plausible > - using fixed buffers would unfortunately increase the number of IOs > noticeably. > > Should have sent an email about that... > > I guess we could add some heuristic to use _FIXED if it doesn't require > splitting an IO into too many sub-ios. But that seems pretty gnarly. Adding Pavel, he's been working on support registered buffers with readv/writev. > I dimly recall that I also ran into some around using fixed buffers as a > non-root user. It might just be the accounting of registered buffers as > mlocked memory and the difficulty of configuring that across > distributions. But I unfortunately don't remember any details anymore. Should just be accounting. -- Jens Axboe