From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 213FCC433EF for ; Thu, 16 Jun 2022 21:52:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BA326B0071; Thu, 16 Jun 2022 17:52:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 867E06B0072; Thu, 16 Jun 2022 17:52:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 730B86B0073; Thu, 16 Jun 2022 17:52:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 62DEA6B0071 for ; Thu, 16 Jun 2022 17:52:20 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 318E235279 for ; Thu, 16 Jun 2022 21:52:20 +0000 (UTC) X-FDA: 79585448040.24.E45A322 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf26.hostedemail.com (Postfix) with ESMTP id 8A87114009F for ; Thu, 16 Jun 2022 21:52:19 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25GKPTwL020276 for ; Thu, 16 Jun 2022 14:52:18 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=Ynwi9SDirpjeaykEJbYhjcV/3AMQY22jWj0npJc0zwc=; b=T+KrILDy+2qw2Spbt8zh88B6udlxLj5RiMNAizsQ3xpk6rcwlDkncLomG9GmnLHiFvho L0LEz4KXCVZhyXBAfCmoWBKC+o6BeiKF9na2cCPhbhuLqlUTFIK9+SNUdf2ChQ/ZceiR dK1TzRz8hG40uY6lhJoE6ECfzRKx8Nju2uo= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gqve42c29-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 16 Jun 2022 14:52:18 -0700 Received: from snc-exhub201.TheFacebook.com (2620:10d:c085:21d::7) by snc-exhub204.TheFacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 16 Jun 2022 14:52:17 -0700 Received: from twshared25107.07.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Thu, 16 Jun 2022 14:52:17 -0700 Received: by devvm225.atn0.facebook.com (Postfix, from userid 425415) id ABE25108B709F; Thu, 16 Jun 2022 14:22:23 -0700 (PDT) From: Stefan Roesch To: , , , , CC: , , , , , , Christoph Hellwig Subject: [PATCH v9 05/14] iomap: Add async buffered write support Date: Thu, 16 Jun 2022 14:22:12 -0700 Message-ID: <20220616212221.2024518-6-shr@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220616212221.2024518-1-shr@fb.com> References: <20220616212221.2024518-1-shr@fb.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe Content-Type: text/plain X-Proofpoint-ORIG-GUID: rTcL5b91QuDUuVC88NvaJ8bDtvn6cdVg X-Proofpoint-GUID: rTcL5b91QuDUuVC88NvaJ8bDtvn6cdVg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-16_18,2022-06-16_01,2022-02-23_01 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655416339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ynwi9SDirpjeaykEJbYhjcV/3AMQY22jWj0npJc0zwc=; b=g3a9f1CbsouNvpzax7h+2JD5xQMmhcLXjMSPY3VZ6wpo4/UTTbKzLGEt8nJU5VeaDYu5Gw lcVPmOYr32n4tQI6oyGPTjTzpN1KJxd9lYX3zFPkQPFII5cLnVIbqeSNLhlUA3lMKrd0bJ g2ocRGhq98l1KRcM6tzL6f2eiJt0CLY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655416339; a=rsa-sha256; cv=none; b=8cVhN7jYndg96K0ztioQ6OTOYTFT5yJjkV3TlVk6BmHmuagahCWWsQJ/M1bKYdkE/aPBea K5I4SqzwQU+c4Fgwj0WdntwwG65/ntB36dUPBOEg6Gqs0NelbnrRkkhuVBzSY/2OXO80Dn KpJjV0VkCDxYfsjVMobFwexrt9JkTe0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=fb.com header.s=facebook header.b=T+KrILDy; dmarc=pass (policy=reject) header.from=fb.com; spf=none (imf26.hostedemail.com: domain of "prvs=7166657324=shr@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=7166657324=shr@fb.com" X-Stat-Signature: umdmrmwonp6n3ga5chx9c6peco6nyfwz X-Rspamd-Queue-Id: 8A87114009F Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=fb.com header.s=facebook header.b=T+KrILDy; dmarc=pass (policy=reject) header.from=fb.com; spf=none (imf26.hostedemail.com: domain of "prvs=7166657324=shr@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=7166657324=shr@fb.com" X-Rspamd-Server: rspam01 X-Rspam-User: X-HE-Tag: 1655416339-757956 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This adds async buffered write support to iomap. This replaces the call to balance_dirty_pages_ratelimited() with the call to balance_dirty_pages_ratelimited_flags. This allows to specify if the write request is async or not. In addition this also moves the above function call to the beginning of the function. If the function call is at the end of the function and the decision is made to throttle writes, then there is no request that io-uring can wait on. By moving it to the beginning of the function, the write request is not issued, but returns -EAGAIN instead. io-uring will punt the request and process it in the io-worker. By moving the function call to the beginning of the function, the write throttling will happen one page later. Signed-off-by: Stefan Roesch Reviewed-by: Jan Kara Reviewed-by: Christoph Hellwig --- fs/iomap/buffered-io.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 3c97b713f831..83cf093fcb92 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -559,6 +559,7 @@ static int __iomap_write_begin(const struct iomap_ite= r *iter, loff_t pos, loff_t block_size =3D i_blocksize(iter->inode); loff_t block_start =3D round_down(pos, block_size); loff_t block_end =3D round_up(pos + len, block_size); + unsigned int nr_blocks =3D i_blocks_per_folio(iter->inode, folio); size_t from =3D offset_in_folio(folio, pos), to =3D from + len; size_t poff, plen; =20 @@ -567,6 +568,8 @@ static int __iomap_write_begin(const struct iomap_ite= r *iter, loff_t pos, folio_clear_error(folio); =20 iop =3D iomap_page_create(iter->inode, folio, iter->flags); + if ((iter->flags & IOMAP_NOWAIT) && !iop && nr_blocks > 1) + return -EAGAIN; =20 do { iomap_adjust_read_range(iter->inode, folio, &block_start, @@ -584,7 +587,12 @@ static int __iomap_write_begin(const struct iomap_it= er *iter, loff_t pos, return -EIO; folio_zero_segments(folio, poff, from, to, poff + plen); } else { - int status =3D iomap_read_folio_sync(block_start, folio, + int status; + + if (iter->flags & IOMAP_NOWAIT) + return -EAGAIN; + + status =3D iomap_read_folio_sync(block_start, folio, poff, plen, srcmap); if (status) return status; @@ -613,6 +621,9 @@ static int iomap_write_begin(const struct iomap_iter = *iter, loff_t pos, unsigned fgp =3D FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE | FGP_NO= FS; int status =3D 0; =20 + if (iter->flags & IOMAP_NOWAIT) + fgp |=3D FGP_NOWAIT; + BUG_ON(pos + len > iter->iomap.offset + iter->iomap.length); if (srcmap !=3D &iter->iomap) BUG_ON(pos + len > srcmap->offset + srcmap->length); @@ -632,7 +643,7 @@ static int iomap_write_begin(const struct iomap_iter = *iter, loff_t pos, folio =3D __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT= , fgp, mapping_gfp_mask(iter->inode->i_mapping)); if (!folio) { - status =3D -ENOMEM; + status =3D (iter->flags & IOMAP_NOWAIT) ? -EAGAIN : -ENOMEM; goto out_no_page; } if (pos + len > folio_pos(folio) + folio_size(folio)) @@ -750,6 +761,8 @@ static loff_t iomap_write_iter(struct iomap_iter *ite= r, struct iov_iter *i) loff_t pos =3D iter->pos; ssize_t written =3D 0; long status =3D 0; + struct address_space *mapping =3D iter->inode->i_mapping; + unsigned int bdp_flags =3D (iter->flags & IOMAP_NOWAIT) ? BDP_ASYNC : 0= ; =20 do { struct folio *folio; @@ -762,6 +775,11 @@ static loff_t iomap_write_iter(struct iomap_iter *it= er, struct iov_iter *i) bytes =3D min_t(unsigned long, PAGE_SIZE - offset, iov_iter_count(i)); again: + status =3D balance_dirty_pages_ratelimited_flags(mapping, + bdp_flags); + if (unlikely(status)) + break; + if (bytes > length) bytes =3D length; =20 @@ -770,6 +788,10 @@ static loff_t iomap_write_iter(struct iomap_iter *it= er, struct iov_iter *i) * Otherwise there's a nasty deadlock on copying from the * same page as we're writing to, without it being marked * up-to-date. + * + * For async buffered writes the assumption is that the user + * page has already been faulted in. This can be optimized by + * faulting the user page. */ if (unlikely(fault_in_iov_iter_readable(i, bytes) =3D=3D bytes)) { status =3D -EFAULT; @@ -781,7 +803,7 @@ static loff_t iomap_write_iter(struct iomap_iter *ite= r, struct iov_iter *i) break; =20 page =3D folio_file_page(folio, pos >> PAGE_SHIFT); - if (mapping_writably_mapped(iter->inode->i_mapping)) + if (mapping_writably_mapped(mapping)) flush_dcache_page(page); =20 copied =3D copy_page_from_iter_atomic(page, offset, bytes, i); @@ -806,8 +828,6 @@ static loff_t iomap_write_iter(struct iomap_iter *ite= r, struct iov_iter *i) pos +=3D status; written +=3D status; length -=3D status; - - balance_dirty_pages_ratelimited(iter->inode->i_mapping); } while (iov_iter_count(i) && length); =20 return written ? written : status; @@ -825,6 +845,9 @@ iomap_file_buffered_write(struct kiocb *iocb, struct = iov_iter *i, }; int ret; =20 + if (iocb->ki_flags & IOCB_NOWAIT) + iter.flags |=3D IOMAP_NOWAIT; + while ((ret =3D iomap_iter(&iter, ops)) > 0) iter.processed =3D iomap_write_iter(&iter, i); if (iter.pos =3D=3D iocb->ki_pos) --=20 2.30.2