From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B4E3C433ED for ; Wed, 7 Apr 2021 14:43:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 401A961242 for ; Wed, 7 Apr 2021 14:43:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 401A961242 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 921EA6B007D; Wed, 7 Apr 2021 10:43:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D2216B007E; Wed, 7 Apr 2021 10:43:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 772896B0080; Wed, 7 Apr 2021 10:43:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 579826B007D for ; Wed, 7 Apr 2021 10:43:56 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0A23E181AF5C7 for ; Wed, 7 Apr 2021 14:43:56 +0000 (UTC) X-FDA: 78005840472.22.485DE18 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf05.hostedemail.com (Postfix) with ESMTP id 21BC2E00011B for ; Wed, 7 Apr 2021 14:43:55 +0000 (UTC) Received: by mail-qv1-f47.google.com with SMTP id u3so6955074qvj.8 for ; Wed, 07 Apr 2021 07:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=/btP8VyRZ+PtRHEtHpKhEBU92313kMIz6dauarfwMLw=; b=FGKaXVsd1J+4FesmYRgTHlNBQriTvi5/iDTaOcVP69WtLq6kN3edC0JqVIFnlLsYCA L72IWqTaluslQzvNuGEb7aMe3IME2oWNyoARDuYUeRNb3welcW+9okrJ7XAOC+swKB4v dMVZ1vwObj+LmQxNnjAWnDXIx49wdxr69g2CrT35DofA+OxaPgZqdQ2NjBafPNOZq8za IdG91Gjcaj0dJ3qIsp2CXt9SOgAWykDlCyJwJmXog8jWDKPbMM+3PA94uAw0Dg5E5jSY 6r5WeWHI1+TX5j0RBeDnWTHBfvVDq1edl1gDZywRllmKWg4xKZ28aCg+MrLm+65tRXbP nSqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=/btP8VyRZ+PtRHEtHpKhEBU92313kMIz6dauarfwMLw=; b=C3CTmg0VVA/Gha79WZIq9hMnqbnch2l+xJk4Q9N884VHZEx9CwO5Og2Hpw4+QMAbdT ITVaW7o96j8eNu5snPHjYWo0HL7LIKuSCXLd0TwW6uN2Gvc/jWAk1jzqeLIGF2F5mYdn bPVEIgpVBeNOHFMpD0udHyUhHd8CgDWvx1FZ7Dp/thO2hAyin09IcFST1gU0ahEaH5kM IfoBpTTqHXEO5jw2I0E3Pr7wdaXZZMN2TXY6+OjNuaKcj1VzKJyZVcPNTDgvN3ZfHacr 7YL0SQ+1sQi0cbaFKucT05m3aoJf9YXWsUValOz//teowggUppdt0ZYvph7qdi9SpCjE Zyig== X-Gm-Message-State: AOAM531yQj+6c8+JeUVdsQNfA15dJjkTUJuUpljSpgUbM8Cuj+gYUWgU NXddVDDCgBgcViiaPdytFDQ= X-Google-Smtp-Source: ABdhPJx72scUaq5TlcVed08ZTISLmlNbmEHtmG617kEvTB0RSvFRyEXWpI7TqiBxKteAikcYxNjhiQ== X-Received: by 2002:a0c:9e5e:: with SMTP id z30mr3809678qve.61.1617806634967; Wed, 07 Apr 2021 07:43:54 -0700 (PDT) Received: from dschatzberg-fedora-PC0Y6AEN.dhcp.thefacebook.com ([2620:10d:c091:480::1:781f]) by smtp.gmail.com with ESMTPSA id 75sm18097680qkj.134.2021.04.07.07.43.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Apr 2021 07:43:54 -0700 (PDT) Date: Wed, 7 Apr 2021 10:43:52 -0400 From: Dan Schatzberg To: Hillf Danton Cc: Jens Axboe , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH 1/3] loop: Use worker per cgroup instead of kworker Message-ID: References: <20210402191638.3249835-1-schatzberg.dan@gmail.com> <20210403020902.1384-1-hdanton@sina.com> <20210407065300.1478-1-hdanton@sina.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210407065300.1478-1-hdanton@sina.com> X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 21BC2E00011B X-Stat-Signature: tc4osshwc46k9gmqpsc79j4q95za3j3r Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=mail-qv1-f47.google.com; client-ip=209.85.219.47 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617806635-931452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Apr 07, 2021 at 02:53:00PM +0800, Hillf Danton wrote: > On Tue, 6 Apr 2021 Dan Schatzberg wrote: > >On Sat, Apr 03, 2021 at 10:09:02AM +0800, Hillf Danton wrote: > >> On Fri, 2 Apr 2021 12:16:32 Dan Schatzberg wrote: > >> > +queue_work: > >> > + if (worker) { > >> > + /* > >> > + * We need to remove from the idle list here while > >> > + * holding the lock so that the idle timer doesn't > >> > + * free the worker > >> > + */ > >> > + if (!list_empty(&worker->idle_list)) > >> > + list_del_init(&worker->idle_list); > >> > >> Nit, only queue work if the worker is inactive - otherwise it is taking > >> care of the cmd_list. > > > >By worker is inactive, you mean worker is on the idle_list? Yes, I > >think you're right that queue_work() is unnecessary in that case since > >each worker checks empty cmd_list then adds itself to idle_list under > >the lock. A couple other corner cases - When worker is just allocated, it needs a queue_work() and rootcg always needs a queue_work() since it never sits on the idle_list. It does add to the logic a bit rather than just unconditionally invoking queue_work() > > > >> > >> > + work = &worker->work; > >> > + cmd_list = &worker->cmd_list; > >> > + } else { > >> > + work = &lo->rootcg_work; > >> > + cmd_list = &lo->rootcg_cmd_list; > >> > + } > >> > + list_add_tail(&cmd->list_entry, cmd_list); > >> > + queue_work(lo->workqueue, work); > >> > + spin_unlock_irq(&lo->lo_work_lock); > >> > } > >> [...] > >> > + /* > >> > + * We only add to the idle list if there are no pending cmds > >> > + * *and* the worker will not run again which ensures that it > >> > + * is safe to free any worker on the idle list > >> > + */ > >> > + if (worker && !work_pending(&worker->work)) { > >> > >> The empty cmd_list is a good enough reason for worker to become idle. > > > >This is only true with the above change to avoid a gratuitous > >queue_work(), right? > > It is always true because of the empty cmd_list - the idle_list is the only > place for the worker to go at this point. > > >Otherwise we run the risk of freeing a worker > >concurrently with loop_process_work() being invoked. > > My suggestion is a minor optimization at most without any change to removing > worker off the idle_list on queuing work - that cuts the risk for you. If I just change this line from if (worker && !work_pending(&worker->work)) { to if (worker) { then the following sequence of events is possible: 1) loop_queue_work runs, adds a command to the worker list 2) loop_process_work runs, processes a single command and then drops the lock and reschedules 3) loop_queue_work runs again, acquires the lock, adds to the list and invokes queue_work() again 4) loop_process_work resumes, acquires lock, processes work, notices list is empty and adds itself to the idle_list 5) idle timer fires and frees the worker 6) loop_process_work runs again (because of the queue_work in 3) and accesses freed memory The !work_pending... check prevents 4) from adding itself to the idle_list so this is not possible. I believe we can only make this change if we also make the other change you suggested to avoid gratuitous queue_work()