From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E187C433F5 for ; Mon, 13 Sep 2021 19:42:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 405C8610CE for ; Mon, 13 Sep 2021 19:42:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 405C8610CE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 8BB5D900002; Mon, 13 Sep 2021 15:42:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 86AB36B0071; Mon, 13 Sep 2021 15:42:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70B97900002; Mon, 13 Sep 2021 15:42:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0249.hostedemail.com [216.40.44.249]) by kanga.kvack.org (Postfix) with ESMTP id 639956B006C for ; Mon, 13 Sep 2021 15:42:50 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1C5392BC1A for ; Mon, 13 Sep 2021 19:42:50 +0000 (UTC) X-FDA: 78583572900.18.F98409B Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf05.hostedemail.com (Postfix) with ESMTP id DBBD95058CF2 for ; Mon, 13 Sep 2021 19:42:49 +0000 (UTC) Received: by mail-pl1-f173.google.com with SMTP id w6so6557923pll.3 for ; Mon, 13 Sep 2021 12:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=vfK/1h9QsUFCkVSDS838FPeS8MNkiZ5ITQXUKp/fEoc=; b=gnZjn1peutkcFGHrbl+EvIvj3MVzt6NF2XejDwjSLdZsr1JwRZX37aYu8hpINGLIte 9s2XV12tlatZGnJ16IuAY4yiVZbSJtCzOvJhH9HmdgucHqacf2hIvUoB9E8CGXg0oV+l zXqOSfQWbjrN7fnRucE7bPiz/9pFiobuTyKuM+46E+vnINgrjdekPrkM1YGd8BNinsbB Sp7ZaVo2mB/55o7Hih2/qHCL2sDeXRW2BFPs3eqm/ZjbFWwhWB67noPGU9CLdOGHCLsu rXopYBWv666ssVHTySfRIBFWRAH2TLnjITC5U0emsfkF06g5DixShylS7dbB3WfhdKU4 hQ4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=vfK/1h9QsUFCkVSDS838FPeS8MNkiZ5ITQXUKp/fEoc=; b=m7VLmqbPk0elLbQDuC8SdWJPVXyNUOtOKddhUWWFAvtzTH6sI3ZIZw8ivK0u1aQymM FbOQGT0YZh6V47j6tuXYI3FarfHwPnDVWu3ngG2fCZcF9oHUtfWXfHbf3k+vclthZhyu Vxe4DTmDLg37hxxPegTXjeg8FCx6+ZNQmhmARhIkWg/Ke/9MQG0dbm7gl+LRwKMz53uE pXrrk+BhEVkfmZqXIHUJnFvNp7FwRwL9FSAjJxD8FjuVzBoP1afy8MEx4cP1hpv6Kn6H 29eXivlcooCiOzOizJQJzHPKwZtkawkuJL2gcCc5pJrq1ah5xkn/dq+5MuddI4Xx8m0J HWYQ== X-Gm-Message-State: AOAM530grDRy2h1sqeM9yzpZP9e5Qy+l6RhdwRCNlApfDFeZy8iR86hs Q+u6UgyBHgYk6BikHtSNsVA= X-Google-Smtp-Source: ABdhPJxdH+GSCsRwPAmdvdU0wBkmkttJzaWrnnoKbWQhOL8+ok29XYaQSUFzAp3Z3aYFr9gIFbu5+A== X-Received: by 2002:a17:90b:33c8:: with SMTP id lk8mr1255361pjb.241.1631562168700; Mon, 13 Sep 2021 12:42:48 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-e24f-43ff-fee6-449f.res6.spectrum.com. [2603:800c:1a02:1bae:e24f:43ff:fee6:449f]) by smtp.gmail.com with ESMTPSA id p5sm8003697pfp.218.2021.09.13.12.42.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Sep 2021 12:42:48 -0700 (PDT) Date: Mon, 13 Sep 2021 09:42:46 -1000 From: Tejun Heo To: Shakeel Butt Cc: Feng Tang , Hillf Danton , LKML , Xing Zhengjun , Linux MM Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression Message-ID: References: <20210902215504.dSSfDKJZu%akpm@linux-foundation.org> <20210905124439.GA15026@xsang-OptiPlex-9020> <20210907033000.GA88160@shbuild999.sh.intel.com> <20210912111756.4158-1-hdanton@sina.com> <20210912132914.GA56674@shbuild999.sh.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gnZjn1pe; spf=pass (imf05.hostedemail.com: domain of htejun@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=htejun@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) X-Stat-Signature: d7d1ttht6irfrx3gofwaimzfjk4k6usf X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: DBBD95058CF2 X-HE-Tag: 1631562169-217062 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, On Mon, Sep 13, 2021 at 12:40:06PM -0700, Shakeel Butt wrote: > I did one more experiment with same workload but with system_wq > instead system_unbound_wq and there is clear difference in profile: > > With system_unbound_wq: > - 4.63% 0.33% mmap [kernel.kallsyms] [k] queue_work_on > 4.29% queue_work_on > - __queue_work > - 3.45% wake_up_process > - try_to_wake_up > - 2.46% ttwu_queue > - 1.66% ttwu_do_activate > - 1.14% activate_task > - 0.97% enqueue_task_fair > enqueue_entity > > With system_wq: > - 1.36% 0.06% mmap [kernel.kallsyms] [k] queue_work_on > 1.30% queue_work_on > - __queue_work > - 1.03% wake_up_process > - try_to_wake_up > - 0.97% ttwu_queue > 0.66% ttwu_do_activate > > Tejun, is this expected? i.e. queuing work on system_wq has a > different performance impact than on system_unbound_wq? Yes, system_unbound_wq is putting the work item on the global shared workqueue while the system_wq is per-cpu, so on a loaded system, overhead difference showing up isn't too surprising. Thanks. -- tejun