From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6741C433DF for ; Sat, 27 Jun 2020 07:33:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9D86E2081A for ; Sat, 27 Jun 2020 07:33:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="u0A0PX8v" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9D86E2081A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lst.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3133F6B0093; Sat, 27 Jun 2020 03:33:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 29DA26B0096; Sat, 27 Jun 2020 03:33:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1419B6B0098; Sat, 27 Jun 2020 03:33:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0008.hostedemail.com [216.40.44.8]) by kanga.kvack.org (Postfix) with ESMTP id EA9C66B0093 for ; Sat, 27 Jun 2020 03:33:08 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E14B81EE6 for ; Sat, 27 Jun 2020 07:33:07 +0000 (UTC) X-FDA: 76974175614.09.tail83_5d0e16a26e5c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id C42BD180AD801 for ; Sat, 27 Jun 2020 07:33:07 +0000 (UTC) X-HE-Tag: tail83_5d0e16a26e5c X-Filterd-Recvd-Size: 9085 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Sat, 27 Jun 2020 07:33:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=neOgU92RsrWDuksyQ2Z1wFdOnnGrSBcBhrDlZznOqq0=; b=u0A0PX8v6CDjtRXkekDJ9GPZcA FENXXFhI2izz8lcBcKb5aWOhXCPP2z4UVHieA7JWGSgOpclTA8JLr2Pp1pX2NQ83uXTaPwMmz5vAU U2X35b2VMZAlui75iR9AdUpKd9NGKPiN9uhpWEHxitnCXeatyhWsBdhNbQfPMM+0yjf4gSx3Z3HaH GWNHb/bJK5RuN1uXcAatmVe3NAk4BLtr/tn3Z2XPqLxGW1D9WJjqmZ2uo9zwBhUeXVOc4oJhuCyfR pxw/1+5SLMZzgs5r+Y0Gbv0CQqwFjeqFeXcCMxPp7rSw3Wq8fhJlCzQsJb5+fN9oNwPgV6zq4T/cc YpgPDRLQ==; Received: from [2001:4bb8:184:76e3:595:ba65:ae56:65a6] (helo=localhost) by casper.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jp5KS-0006q3-6c; Sat, 27 Jun 2020 07:32:51 +0000 From: Christoph Hellwig To: Jens Axboe , Tejun Heo Cc: Dennis Zhou , Li Zefan , Johannes Weiner , dm-devel@redhat.com, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 13/14] blk-cgroup: remove blkcg_bio_issue_check Date: Sat, 27 Jun 2020 09:31:58 +0200 Message-Id: <20200627073159.2447325-14-hch@lst.de> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200627073159.2447325-1-hch@lst.de> References: <20200627073159.2447325-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html X-Rspamd-Queue-Id: C42BD180AD801 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: blkcg_bio_issue_check is a giant inline function that does three entirely different things. Factor out the blk-cgroup related bio initalization into a new helper, and the open code the sequence in the only caller, relying on the fact that all the actual functionality is stubbed out for non-cgroup builds. Signed-off-by: Christoph Hellwig --- block/blk-cgroup.c | 34 +++++++++++++++++++++++ block/blk-core.c | 7 ++++- block/blk-throttle.c | 5 ++-- block/blk.h | 2 ++ include/linux/blk-cgroup.h | 56 ++------------------------------------ 5 files changed, 47 insertions(+), 57 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index d21ec2acd716e7..1ce94afc03bcfd 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1813,6 +1813,40 @@ void bio_clone_blkg_association(struct bio *dst, s= truct bio *src) } EXPORT_SYMBOL_GPL(bio_clone_blkg_association); =20 +static int blk_cgroup_io_type(struct bio *bio) +{ + if (op_is_discard(bio->bi_opf)) + return BLKG_IOSTAT_DISCARD; + if (op_is_write(bio->bi_opf)) + return BLKG_IOSTAT_WRITE; + return BLKG_IOSTAT_READ; +} + +void blk_cgroup_bio_start(struct bio *bio) +{ + int rwd =3D blk_cgroup_io_type(bio), cpu; + struct blkg_iostat_set *bis; + + cpu =3D get_cpu(); + bis =3D per_cpu_ptr(bio->bi_blkg->iostat_cpu, cpu); + u64_stats_update_begin(&bis->sync); + + /* + * If the bio is flagged with BIO_CGROUP_ACCT it means this is a split + * bio and we would have already accounted for the size of the bio. + */ + if (!bio_flagged(bio, BIO_CGROUP_ACCT)) { + bio_set_flag(bio, BIO_CGROUP_ACCT); + bis->cur.bytes[rwd] +=3D bio->bi_iter.bi_size; + } + bis->cur.ios[rwd]++; + + u64_stats_update_end(&bis->sync); + if (cgroup_subsys_on_dfl(io_cgrp_subsys)) + cgroup_rstat_updated(bio->bi_blkg->blkcg->css.cgroup, cpu); + put_cpu(); +} + static int __init blkcg_init(void) { blkcg_punt_bio_wq =3D alloc_workqueue("blkcg_punt_bio", diff --git a/block/blk-core.c b/block/blk-core.c index a9769c1a287546..76cfd5709f66cd 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1073,8 +1073,13 @@ generic_make_request_checks(struct bio *bio) if (unlikely(!current->io_context)) create_task_io_context(current, GFP_ATOMIC, q->node); =20 - if (!blkcg_bio_issue_check(q, bio)) + if (blk_throtl_bio(bio)) { + blkcg_bio_issue_init(bio); return false; + } + + blk_cgroup_bio_start(bio); + blkcg_bio_issue_init(bio); =20 if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) { trace_block_bio_queue(q, bio); diff --git a/block/blk-throttle.c b/block/blk-throttle.c index ac008345050010..9d00f62c05ecdf 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -2158,9 +2158,10 @@ static inline void throtl_update_latency_buckets(s= truct throtl_data *td) } #endif =20 -bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blkg, - struct bio *bio) +bool blk_throtl_bio(struct bio *bio) { + struct request_queue *q =3D bio->bi_disk->queue; + struct blkcg_gq *blkg =3D bio->bi_blkg; struct throtl_qnode *qn =3D NULL; struct throtl_grp *tg =3D blkg_to_tg(blkg ?: q->root_blkg); struct throtl_service_queue *sq; diff --git a/block/blk.h b/block/blk.h index 3a120a070dac82..41a50880c94e98 100644 --- a/block/blk.h +++ b/block/blk.h @@ -288,10 +288,12 @@ int create_task_io_context(struct task_struct *task= , gfp_t gfp_mask, int node); extern int blk_throtl_init(struct request_queue *q); extern void blk_throtl_exit(struct request_queue *q); extern void blk_throtl_register_queue(struct request_queue *q); +bool blk_throtl_bio(struct bio *bio); #else /* CONFIG_BLK_DEV_THROTTLING */ static inline int blk_throtl_init(struct request_queue *q) { return 0; } static inline void blk_throtl_exit(struct request_queue *q) { } static inline void blk_throtl_register_queue(struct request_queue *q) { = } +static inline bool blk_throtl_bio(struct bio *bio) { return false; } #endif /* CONFIG_BLK_DEV_THROTTLING */ #ifdef CONFIG_BLK_DEV_THROTTLING_LOW extern ssize_t blk_throtl_sample_time_show(struct request_queue *q, char= *page); diff --git a/include/linux/blk-cgroup.h b/include/linux/blk-cgroup.h index 8ab043c911f233..431b2d18bf4074 100644 --- a/include/linux/blk-cgroup.h +++ b/include/linux/blk-cgroup.h @@ -517,14 +517,6 @@ static inline void blkg_put(struct blkcg_gq *blkg) if (((d_blkg) =3D __blkg_lookup(css_to_blkcg(pos_css), \ (p_blkg)->q, false))) =20 -#ifdef CONFIG_BLK_DEV_THROTTLING -extern bool blk_throtl_bio(struct request_queue *q, struct blkcg_gq *blk= g, - struct bio *bio); -#else -static inline bool blk_throtl_bio(struct request_queue *q, struct blkcg_= gq *blkg, - struct bio *bio) { return false; } -#endif - bool __blkcg_punt_bio_submit(struct bio *bio); =20 static inline bool blkcg_punt_bio_submit(struct bio *bio) @@ -540,50 +532,6 @@ static inline void blkcg_bio_issue_init(struct bio *= bio) bio_issue_init(&bio->bi_issue, bio_sectors(bio)); } =20 -static inline bool blkcg_bio_issue_check(struct request_queue *q, - struct bio *bio) -{ - struct blkcg_gq *blkg =3D bio->bi_blkg; - bool throtl =3D false; - - throtl =3D blk_throtl_bio(q, blkg, bio); - if (!throtl) { - struct blkg_iostat_set *bis; - int rwd, cpu; - - if (op_is_discard(bio->bi_opf)) - rwd =3D BLKG_IOSTAT_DISCARD; - else if (op_is_write(bio->bi_opf)) - rwd =3D BLKG_IOSTAT_WRITE; - else - rwd =3D BLKG_IOSTAT_READ; - - cpu =3D get_cpu(); - bis =3D per_cpu_ptr(blkg->iostat_cpu, cpu); - u64_stats_update_begin(&bis->sync); - - /* - * If the bio is flagged with BIO_CGROUP_ACCT it means this is a - * split bio and we would have already accounted for the size of - * the bio. - */ - if (!bio_flagged(bio, BIO_CGROUP_ACCT)) { - bio_set_flag(bio, BIO_CGROUP_ACCT); - bis->cur.bytes[rwd] +=3D bio->bi_iter.bi_size; - } - bis->cur.ios[rwd]++; - - u64_stats_update_end(&bis->sync); - if (cgroup_subsys_on_dfl(io_cgrp_subsys)) - cgroup_rstat_updated(blkg->blkcg->css.cgroup, cpu); - put_cpu(); - } - - blkcg_bio_issue_init(bio); - - return !throtl; -} - static inline void blkcg_use_delay(struct blkcg_gq *blkg) { if (WARN_ON_ONCE(atomic_read(&blkg->use_delay) < 0)) @@ -657,6 +605,7 @@ static inline void blkcg_clear_delay(struct blkcg_gq = *blkg) atomic_dec(&blkg->blkcg->css.cgroup->congestion_count); } =20 +void blk_cgroup_bio_start(struct bio *bio); void blkcg_add_delay(struct blkcg_gq *blkg, u64 now, u64 delta); void blkcg_schedule_throttle(struct request_queue *q, bool use_memdelay)= ; void blkcg_maybe_throttle_current(void); @@ -710,8 +659,7 @@ static inline void blkg_put(struct blkcg_gq *blkg) { = } =20 static inline bool blkcg_punt_bio_submit(struct bio *bio) { return false= ; } static inline void blkcg_bio_issue_init(struct bio *bio) { } -static inline bool blkcg_bio_issue_check(struct request_queue *q, - struct bio *bio) { return true; } +static inline void blk_cgroup_bio_start(struct bio *bio) { } =20 #define blk_queue_for_each_rl(rl, q) \ for ((rl) =3D &(q)->root_rl; (rl); (rl) =3D NULL) --=20 2.26.2