From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F122C02188 for ; Mon, 27 Jan 2025 21:31:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1DA02801AF; Mon, 27 Jan 2025 16:31:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BA6122801A8; Mon, 27 Jan 2025 16:31:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A204A2801AF; Mon, 27 Jan 2025 16:31:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7C13E2801A8 for ; Mon, 27 Jan 2025 16:31:45 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2513FA075F for ; Mon, 27 Jan 2025 21:31:45 +0000 (UTC) X-FDA: 83054528970.02.C3D7EB4 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by imf12.hostedemail.com (Postfix) with ESMTP id 132814000A for ; Mon, 27 Jan 2025 21:31:42 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PnS50ldh; spf=pass (imf12.hostedemail.com: domain of asml.silence@gmail.com designates 209.85.218.52 as permitted sender) smtp.mailfrom=asml.silence@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738013503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H8Ix5bawk3vcAfD33vhDdhZSL9A70/0kQOyvn23giZE=; b=syfvSW78eI51yHbjq0hI7zXJbGVSPIi0/B8UUeGdnNYNVaqgfZHLBmnOIN/cqQ3SrFsvdc /EtGLwHlKB2rv3jlbgrtj8FcB9IHcRk0TRbDONBSYyWFu17pn7hlHctJlOWUxz7jywlT4u SCBcVb0n1nC7h48eV38YJkk2afL+JKw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PnS50ldh; spf=pass (imf12.hostedemail.com: domain of asml.silence@gmail.com designates 209.85.218.52 as permitted sender) smtp.mailfrom=asml.silence@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738013503; a=rsa-sha256; cv=none; b=3s7f8uwKzuLUnHIOD0QEVsRz22GknLiQqY8NM2WUtgww8NmQgihRcrdZt1x23ms8kA+/E5 a7b5arKHsPFP3j2WeMEdFyFe84RItFqlK9EU39fnz34YpyQq+jp0+5OKmqoceW8bOcpbPt 7Uz7zWudZ6H4w88onZbkpvMqpkZfN8c= Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-aa689a37dd4so991147366b.3 for ; Mon, 27 Jan 2025 13:31:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738013501; x=1738618301; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=H8Ix5bawk3vcAfD33vhDdhZSL9A70/0kQOyvn23giZE=; b=PnS50ldhUgbxaWgs6Rj+fSxquRpVXfdMpb8u1kh90WM/RGsmS29gVvyhmC/6xKtgDi oC1uJgkbC3fLz4pTcd6un+OlEgUFoM+Ojo8jIkbocTbaZeD5AoygyEYWVsFdT6H+SBE/ D6w5u9m5+495U96MRqfniR4inVWJoUHPBBHw/lJIJm3bsSUbMcIff2fIfL61JKZYzo4u V/MjwJsd7M9DYwxjPi4GBrXW2NNd0JS00U47mPR5lISCA2OjldZykhSdjPteY6ECtbaF BbMRk6/NZ7te+NvUjhy7iYWT5V4XVZ48rU25bONl+KHrpp5VL8/r4SG1zB46edpbOwep HoZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738013501; x=1738618301; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=H8Ix5bawk3vcAfD33vhDdhZSL9A70/0kQOyvn23giZE=; b=AUksyHyagm4EOwJc2NEx0LFftPQF49e21w7K4I+mdL+5DnatKnvjrgHpce/bpAp56W UbpiLwUT4aAopjerXudGR2Zh8+j+RCt6my+BAOESgF2Q7JznjGuh1UnJUsEQxZrJJ802 W6pyUUpeYmGwRTuW08v4zjctpy4OZK93+gVqX+n28P4LgX0i6Miibgm66UQkPhuDr/80 f0yg0JeMKQqnccBknqjI+O2Fr2p/N69+UU2mYPTZNxzycY8dbXUjvOQ04dG4/oqf2ZPb wbyExolbiUqj+5R6V4izu408DHHkOExvlvxGed48q2VGKMXf/DG0RzXIZigxK6LoVNXg rYXg== X-Forwarded-Encrypted: i=1; AJvYcCXG3in4JwXgNcVQ5qVA7QQPdGlP/jCffYjuQ9ADaw63smWQ1DUV1bUJIh4BTFS2D/9JI105PJtO3A==@kvack.org X-Gm-Message-State: AOJu0Yxum7iADPBXlrAFmii7BX++DHnKNQysLmOUEROiUvXDdlW+7+Er 9Qup6B3wuKnwlJ4TKwULlowSr6c/4Tse3Ld5ozp3D0U92dpTpky7 X-Gm-Gg: ASbGnctQG7A1WbPOpbnpLchBFQg3kv5No9++G1tOVH5ix5dsuk2ZbxOoSIXiGdSzaQa 6z/yZ2BuXFFQF5oDs7h99nBneS2HrOTL7bG3vgFI5i3nikPrqcaxmxEwQgkyP5yv3DR0UwhF/fW WFDde5cIFNKwQF6dnL9wQbkQaixgcQ5MqBFH0a74RdjdVZBTLN/W79y+r+/SojYVgtymWGY/yS4 0gBDCmoNP7g/KK/RBTKeKpYEQQiaTOP1qPzgjGeHz8Ims1qr1TRp4Bd2lh3hwhWbRETLaCI3An1 mCO8DVfvmIhlQA== X-Google-Smtp-Source: AGHT+IHVJPLDwM6pv2kJlP80pKtzffSY3kmJQHtOcyku7NWQen1NwlAloUpu1svNA6mHUq0uRNRsxA== X-Received: by 2002:a17:907:3da1:b0:aa6:8bb4:5030 with SMTP id a640c23a62f3a-ab38aedf0eamr4227794166b.0.1738013501094; Mon, 27 Jan 2025 13:31:41 -0800 (PST) Received: from [192.168.8.100] ([148.252.145.92]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ab6760fc485sm646346566b.160.2025.01.27.13.31.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 13:31:40 -0800 (PST) Message-ID: <31cfafe8-c6bb-43fc-883b-356a23d4f434@gmail.com> Date: Mon, 27 Jan 2025 21:32:03 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Direct I/O performance problems with 1GB pages To: Jens Axboe , Andres Freund , David Hildenbrand Cc: Matthew Wilcox , linux-mm@kvack.org, linux-block@vger.kernel.org, Muchun Song , Jane Chu References: <6ulkhmnl4rot5vrywoxvoewko7vbgkhypcwxjccghdu26kwsx5@bnseuzrsedte> <108b6ae7-a272-482b-b3da-60f1fc6617ee@kernel.dk> Content-Language: en-US From: Pavel Begunkov In-Reply-To: <108b6ae7-a272-482b-b3da-60f1fc6617ee@kernel.dk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 132814000A X-Stat-Signature: cgfjnfkjdirn5r5qpb78fcftsd73f433 X-HE-Tag: 1738013502-315168 X-HE-Meta: U2FsdGVkX191C00jxXohQEk3VpLTYq/YQbUFym9BAcbhpTvFyc4uVRP0GBV4P+T3ynChkmdQ5Hvc6/xlZOaLv8/MFjDbZ8coLzje8o6oE0D3zZoEj0iP6ktmIKUshywbqGVaLynYhXERgVb3hSTTE6tEeVwwmuiNGLY9bP/qeoyoJ5pK9bTxHOKX3lwg0M2aqo0vL8Pv1lZ8LIhoWtzLZ3sX4JxC+vePO5FyktYvB7SqVDf3Ko2U//0Aohbt2+ov6hOvltYervoYN/i1685KvjGv0AGzdH0vMEuTiwHGbx2T7N8ekfDYwjoO6gTlvfG380qbtSGzGgOHaFXc2IwTfvJAtO5RhAGU5ccwm8X4PRIvCJq5ttxFxprFuB5nc5h2PNECjaXoJqqnZvqbQzShc7VPn1ESK4wyqnMleJaINPgRgNy+G1GK/EFN8srkAi33y+kjNWcleJaqcrGYg5GneaCXuTzy2CBEW7sA/zH1W4d2WKKDUck/dN77NmsOR4+B9kIHsF9HleSGkBSWEV8EJEWYz48Q8r3NrQX5VDBMpJsIjtFOzoNSCFfYyldqivrpkYS2s21Yn00bBC+kYL/tldqV0NKKeNCFIKrtxUwyb6q9hay3rS6TNh6Ayd6foEaL348Rd656Ov9j41gcaCgIneOhUtw660TqFip4b4yh+9Zu2URrHWVx72CkSPcyfA9qw0NDzyLHzUNpn0PCJGSyjaFdgEE/ksO1cQo1HTfDupUjeDpHsXdToE25QSp9pERFJpM0Ia8YiUKeO8Z3BjbrPWhIndFOTkLsAL7+J1ZkDhHlNVjsCIlXNY3XYc16U6rKkER3IYUzm+QhNSTzrTTK9C/SpD+SynyDwtofkknHKC30yXS5oqqRol1sG0yysGMynS3UQdmNZD/Ugy3BQDfBK39jPQsOYKJuIIWGkr2ZrXBV6RRr3v4lKYyruxdR0/19kWMaE9G9VGFVYxB3etQ u6FB35wX VsqNdnvkwj/bpY9dEXDIY52/1j8mI6hOt9Unjr1VBedI5WWdGc3jHwEwc/mY4sLhNb9hYUiq4Fczy1gVuQYqU4u7MKFFXk5duNRPwZNbHxPdEFwEEyTGRV0aSGkcwdBEhJw5PEXZpiTCIf71MFfnFD7c6IYLMdXZKw5c/iSGE+/MF90zEifBvNOhUT/yGMIQxHUCnZlcRuR1Qrh0XFJ+uU2T1dmlrnNW2z+RLfyH5PnlTnS44IBMGtla3Rj08LnFmYn10GXaHQ4qkKfG0OVJXxivttGoAVoKXUlbuPb9nERHUz6Hef0zI3ybf7N7So2H2L40gZFOnySzIvGuFHpq9GTovt1eIKkX9YRpYxEAs/FThsuE4f7FkSR++tO8c0MhbFkQq/7qYzdCbjRwGKThDjRqCUBKO+UmRnDJ6oFZ5GikN5wUW1m/4AXjASA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000055, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 1/27/25 18:54, Jens Axboe wrote: > On 1/27/25 11:21 AM, Andres Freund wrote: >>> That's precisely what io-uring fixed buffers do :) >> >> I looked at using them at some point - unfortunately it seems that there is >> just {READ,WRITE}_FIXED not {READV,WRITEV}_FIXED. It's *exceedingly* common >> for us to do reads/writes where source/target buffers aren't wholly >> contiguous. Thus - unless I am misunderstanding something, entirely plausible >> - using fixed buffers would unfortunately increase the number of IOs >> noticeably. >> >> Should have sent an email about that... >> >> I guess we could add some heuristic to use _FIXED if it doesn't require >> splitting an IO into too many sub-ios. But that seems pretty gnarly. > > Adding Pavel, he's been working on support registered buffers with > readv/writev. Yes, there are patches from last year out there, I'm going to pick them up and, hopefully for 6.15. I'll keep you in the loop. >> I dimly recall that I also ran into some around using fixed buffers as a >> non-root user. It might just be the accounting of registered buffers as >> mlocked memory and the difficulty of configuring that across >> distributions. But I unfortunately don't remember any details anymore. > > Should just be accounting. > -- Pavel Begunkov