From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5385C25B76 for ; Tue, 11 Jun 2024 08:20:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 066B96B0082; Tue, 11 Jun 2024 04:20:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0162F6B0092; Tue, 11 Jun 2024 04:20:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1F626B0095; Tue, 11 Jun 2024 04:20:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C3FE16B0082 for ; Tue, 11 Jun 2024 04:20:58 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4372D811E1 for ; Tue, 11 Jun 2024 08:20:58 +0000 (UTC) X-FDA: 82217912196.30.32439EA Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by imf21.hostedemail.com (Postfix) with ESMTP id 232C01C0011 for ; Tue, 11 Jun 2024 08:20:55 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=szeredi.hu header.s=google header.b=hREdUWEw; spf=pass (imf21.hostedemail.com: domain of miklos@szeredi.hu designates 209.85.219.174 as permitted sender) smtp.mailfrom=miklos@szeredi.hu; dmarc=pass (policy=quarantine) header.from=szeredi.hu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718094056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TsjHdLyVH+/QRYOQgnmFiCH9v5EDjsTLPrAqffUv+uc=; b=eJlDAs26E0rfTUk7KHRWOdgRGw/oyMwL7wc5e/PM7C1/hy1HQjttm3KFlRw+2eCHIZfAFw sW5yrY1oEUqdD6mcOk98hBFqOb11fxKzbYjcnof1hO0b8XLoBNGTPASGswCORDWrrcwsj6 eHWONn/o/zNDou/z/ebpQ9ulduHewgc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718094056; a=rsa-sha256; cv=none; b=ptSTQ3WTABvE3A0iPmm2HVoqFycCCu2aU3yOZxXsaqrZ/rvWdYIQIIO+zLqYN86xbdDwT6 gTHWKoxbO7eSs26FOFmxg2iYTLj1Xn/3fqflkd6mCNKdPOKjrOJ+Kx4YgsCA3mU59G3ANt ItknlQgxN6LGMapeVI+BlP81WUcRZPU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=szeredi.hu header.s=google header.b=hREdUWEw; spf=pass (imf21.hostedemail.com: domain of miklos@szeredi.hu designates 209.85.219.174 as permitted sender) smtp.mailfrom=miklos@szeredi.hu; dmarc=pass (policy=quarantine) header.from=szeredi.hu Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-dfaff6bf125so3717004276.3 for ; Tue, 11 Jun 2024 01:20:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=szeredi.hu; s=google; t=1718094055; x=1718698855; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=TsjHdLyVH+/QRYOQgnmFiCH9v5EDjsTLPrAqffUv+uc=; b=hREdUWEwW2Ia8f/HX8XfnAAz4MXC+N4qh0PQYQlY3bzNGxVFTvnVEFK5zJv43cf/P5 e/+jfVjoyQOzrapUpK6dqyvBw/jsJBSM8DNxsnqNgTFsgCpV2wuFK85R0B3O8WaU0jui 4sx2p818fuQIha041SjIEtzWrmq/XQlakh3AM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718094055; x=1718698855; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TsjHdLyVH+/QRYOQgnmFiCH9v5EDjsTLPrAqffUv+uc=; b=mMpjeDj//kAKSuzwqZoCN75RJZU/2U8iXJ309AKmuRaqwHNEoMNrLPZn6ne95Bog+P SAr4yl3zCBkhzqrpe0dyX34/+GrCqiGnJN6B0dYleS7rAYz10ubEQpmlDwN24jbKKF93 PpTve7KFpzPRK7TK7Zdyxeu34isyJ3aCFZDe5t8/Fpb2Lr1M5j3O50IJdDQzBiwzJeAu im7D7iouUoPNf259PxyEUrgcIc2iTltPI8A7xT2ZjsqTMZaf/m3zdDTgfPvofWceFz/T v/pmB3NjKZkSD0HHeazMDUxy4m+AY16tabOPv9n6Tvi6csI9ZOHFEboTYI3aOetC+Txl MXgw== X-Forwarded-Encrypted: i=1; AJvYcCUlJGsIMjHTN9vUbB/8soh4AqQnC0mNvWLJ3Pl9gcLoMp5Ee1B2Xw61U0/UwndbGL6RCw7u2BMj/QuI8JTcpp5oiy8= X-Gm-Message-State: AOJu0YwFTe4oDVDl+HulPNan56ist2u1wBCFVPIwWTZyxuibNzVeDLdL fskKS02z+wOLqBtXg3HRwcprTIaHUHM/rgT6Qtgq0Yq9dn42vmHiGZWFfmEXRyHSQfU3nDyjNqq +kYTsnpTtqm0n235wMJiHIohWNahsz70VR/m8RA== X-Google-Smtp-Source: AGHT+IGV43mp6SccLEJHmFQqXVggcZgGWtTaNQ77TCVt0LJrCuabfnXvVEhVcOdoMIPnLgFfeGrshI120aFHrmTB4yk= X-Received: by 2002:a25:ce11:0:b0:dfa:fff4:8ef5 with SMTP id 3f1490d57ef6-dfafff492e6mr9926637276.48.1718094054913; Tue, 11 Jun 2024 01:20:54 -0700 (PDT) MIME-Version: 1.0 References: <20240529-fuse-uring-for-6-9-rfc2-out-v1-0-d149476b1d65@ddn.com> In-Reply-To: <20240529-fuse-uring-for-6-9-rfc2-out-v1-0-d149476b1d65@ddn.com> From: Miklos Szeredi Date: Tue, 11 Jun 2024 10:20:42 +0200 Message-ID: Subject: Re: [PATCH RFC v2 00/19] fuse: fuse-over-io-uring To: Bernd Schubert Cc: Amir Goldstein , linux-fsdevel@vger.kernel.org, bernd.schubert@fastmail.fm, Andrew Morton , linux-mm@kvack.org, Ingo Molnar , Peter Zijlstra , Andrei Vagin , io-uring@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam03 X-Stat-Signature: kdzawa8tncm9isp13sgsfspn51j8pf53 X-Rspamd-Queue-Id: 232C01C0011 X-Rspam-User: X-HE-Tag: 1718094055-887433 X-HE-Meta: U2FsdGVkX1/cmgNJZSn54NSA6Lp3OuxF08tv/37lKfQ2PwHefZW5Llzwvu2uPSwb6w4jZC/za3Bj/fczgqI0VhBBnRAJEe7HuqYiQUlHlBjgcBc0fcfgDCJ5FsrPvyvyC+fzP4H1C2ezESURzyD7E3ys/7LN7HwJuMt5YksZtheMyOHsxhHpH7Cecsiv0mFJ8ZFZlD5O3qLoBRW3T70DCm8c33nO/SxSeHnMs2hAxZLVTzIgQw680IvLD2btruEY/6vl0PvarNyWmrYaMt/I8p0nZrGCNKY6iU0CG4i9ZLOfP5ojg1LJo309IjJRKZAtcb7V6rB+N7gzQS6SKMh/NgouyhtIU8OPVeLp6pKIvkQJoaB/sTz7nwuOmJJ8/3EOW+lxA5sd3WyK3qTiwv8ObMljsGHxTkpLMkVD0xXHHhww8D9PfXa1iJFdk0Gt/2sJWu4+Kg9lNKmWd8YNwIS2qqgj+AlhCmPCsRMNwwDrapc/pXfpq9e1JOuGQcCDMKkZSLw/uf3PP8hP8WQRLDIDN0hYqQHCHoQQPvqUNm8ELY5qoJaHHzfzsxqYKzQbt4DW6Ng3DC32UgtfR27Fsn0Fua/XD1z2fsakLkHhMSFI6onRi4+hgRgrggs62gBn0NLaOS2uXlxSnIS6HlLTFigOTHBLknUghENfVujOxpWoRd01wOisX+a+xigS0IUHhdvjohdnl1+WtTRmjAqoa2YC4qTy58CtrOOjGq5F5b1qCdQPM/87AUVDMSdkUYYcqBRYvrI+XoxlAweaTYPLFSRlndJyo+75YqCneC8wNMr58J/Hx1zBcWUdSd2gpUNJ0X/vRNHyHHA0dDATus1jaY0Y0YMgK+kJufu+UZn/DgTnubt/aTfzD+OoCTHLz6WGhR/wtbzq3g9+eSmupOAua4HznOTr+8/w9e9dFGkakRRvVGNwgNftc53CgaaoG9PdvxUiTJJ73RKr7YzK9oY+GSw eM7VhIsO VVAZl3cJ51qy+fjBQxfRXOo7q1uAcFORd5FG23pYFf1G182ZBMTQz6si5XgJOVXOmNgp1JqJADQkyNQYmh8yGnLGdpvI09RjJWUs3Yp6RzQ/LfOmeCzjknJL8KDo741VLX7A/fjBnD+0/APksh01lvH4r7/ilF7NgZEFQQpIcovtQGoUhfW27kTsDyoL2W9rQr9/VfY2bQxgj3bccduuVBAk4rEugPYKL1kugYhTq3g2LS5KTuz7ltyuHBw/OB2dQfmZm9w92jQr7CPMen8XbMWaHTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, 29 May 2024 at 20:01, Bernd Schubert wrote: > > From: Bernd Schubert > > This adds support for uring communication between kernel and > userspace daemon using opcode the IORING_OP_URING_CMD. The basic > appraoch was taken from ublk. The patches are in RFC state, > some major changes are still to be expected. Thank you very much for tackling this. I think this is an important feature and one that could potentially have a significant effect on fuse performance, which is something many people would love to see. I'm thinking about the architecture and there are some questions: Have you tried just plain IORING_OP_READV / IORING_OP_WRITEV? That's would just be the async part, without the mapped buffer. I suspect most of the performance advantage comes from this and the per-CPU queue, not from the mapped buffer, yet most of the complexity seems to be related to the mapped buffer. Maybe there's an advantage in using an atomic op for WRITEV + READV, but I'm not quite seeing it yet, since there's no syscall overhead for separate ops. What's the reason for separate async and sync request queues? > Avoiding cache line bouncing / numa systems was discussed > between Amir and Miklos before and Miklos had posted > part of the private discussion here > https://lore.kernel.org/linux-fsdevel/CAJfpegtL3NXPNgK1kuJR8kLu3WkVC_ErBPRfToLEiA_0=w3=hA@mail.gmail.com/ > > This cache line bouncing should be addressed by these patches > as well. Why do you think this is solved? > I had also noticed waitq wake-up latencies in fuse before > https://lore.kernel.org/lkml/9326bb76-680f-05f6-6f78-df6170afaa2c@fastmail.fm/T/ > > This spinning approach helped with performance (>40% improvement > for file creates), but due to random server side thread/core utilization > spinning cannot be well controlled in /dev/fuse mode. > With fuse-over-io-uring requests are handled on the same core > (sync requests) or on core+1 (large async requests) and performance > improvements are achieved without spinning. I feel this should be a scheduler decision, but the selecting the queue needs to be based on that decision. Maybe the scheduler people can help out with this. Thanks, Miklos