From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E16BC54EE9 for ; Tue, 6 Sep 2022 08:35:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D1858026A; Tue, 6 Sep 2022 04:35:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 581CD80224; Tue, 6 Sep 2022 04:35:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 448E78026A; Tue, 6 Sep 2022 04:35:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3216E80224 for ; Tue, 6 Sep 2022 04:35:31 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E1C6D40B37 for ; Tue, 6 Sep 2022 08:35:30 +0000 (UTC) X-FDA: 79881001620.26.651B055 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf21.hostedemail.com (Postfix) with ESMTP id B13031C0080 for ; Tue, 6 Sep 2022 08:35:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662453329; x=1693989329; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nTZ2w94RQp2R7AkcYI0Ec8o3HuGSc4e7kZ/lmVxt/v8=; b=GysGBryM/2nTXYl4fEf/NmLlmlzw+jGkBAXpDxXF2+n4HKmZI8Cb8p0h vHM1Jf5HaM/qRmaBePX6Kg75J2FdJlw0i4wwp7kI5gPACMvtj7h9N0Qjw Wb3xjFlYQDaMYF6R9uihfhxSi7kEQwJDk1uMhw9NyrdbBZUxr6On/jBe8 8FQZ+R8Q8Z4mF98R5BNJqIy8DGvNhwM0rPxWsndY+CvS3mCVeWXZL0C2E BvpoFIYQRKXUPSau4jFkyIzfMf8J2dLcJSArANkhGFX9oSWhi+QdArpXu 3K8YnMJdeW0vvuWxPhaS2FJeFIH862CTKqVlhZdoTnCgrNd/TVXU7Wu6E A==; X-IronPort-AV: E=McAfee;i="6500,9779,10461"; a="382833912" X-IronPort-AV: E=Sophos;i="5.93,293,1654585200"; d="scan'208";a="382833912" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Sep 2022 01:35:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,293,1654585200"; d="scan'208";a="675597073" Received: from linux-pnp-server-13.sh.intel.com ([10.239.176.176]) by fmsmga008.fm.intel.com with ESMTP; 06 Sep 2022 01:35:24 -0700 From: Jiebin Sun To: akpm@linux-foundation.org, vasily.averin@linux.dev, shakeelb@google.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, ebiederm@xmission.com, legion@kernel.org, manfred@colorfullife.com, alexander.mikhalitsyn@virtuozzo.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: tim.c.chen@intel.com, feng.tang@intel.com, ying.huang@intel.com, tianyou.li@intel.com, wangyang.guo@intel.com, jiebin.sun@intel.com Subject: [PATCH v3 0/2] ipc/msg: mitigate the lock contention in ipc/msg Date: Wed, 7 Sep 2022 00:54:28 +0800 Message-Id: <20220906165430.851424-1-jiebin.sun@intel.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902152243.479592-1-jiebin.sun@intel.com> References: <20220902152243.479592-1-jiebin.sun@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=GysGBryM; spf=pass (imf21.hostedemail.com: domain of jiebin.sun@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=jiebin.sun@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662453330; a=rsa-sha256; cv=none; b=R39VSVkYTEHE4hMrnpSE6yMyzxN3QQ/uknirf68wkqFNG8yACm1W9/OOvcpoTQkmhZsPj9 J6+E7KL7kTh93hzhgr2Z7KWRmnBd/MC5jDpDm07eRPSOmuX+lviuoKFgVK9wg9dd+b0oyx TgI4V9mXtQQN066uQzMUUSxnEjTWeAA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662453330; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nTZ2w94RQp2R7AkcYI0Ec8o3HuGSc4e7kZ/lmVxt/v8=; b=FkA6X4KqODnK93JNt6mLfaiFvVMjayYBSrtwVO5DmiDJ/izrBtPQ/eWYfTVHYQCWLgpBmX m4/ISvbUm2qYprXaD84fBuqA4SGs1LNxLwZQrHZgOXJa7BlfZCn3X3ZVY2poZA9SFLDwzK 7T4BNLRWtDa6ZOY5BEdgM8qcK1wyhCs= Authentication-Results: imf21.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=GysGBryM; spf=pass (imf21.hostedemail.com: domain of jiebin.sun@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=jiebin.sun@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: gie9aqbsmut4rk57nz69tbey1qd51irw X-Rspamd-Queue-Id: B13031C0080 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1662453329-968749 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, Here are two patches to mitigate the lock contention in ipc/msg. The 1st patch is to add the new function percpu_counter_add_local if only update the local counter without aggregating to global counter. This function could be used with percpu_counter_sum together if you need high accurate counter. The combination could bring obvious performance improvement than percpu_counter_add_batch if percpu_counter_add is frequently called and percpu_counter_sum is not in the critical path. The 2nd patch is to use percpu_counter instead of atomic update in ipc/msg. The msg_bytes and msg_hdrs atomic counters are frequently updated when IPC msg queue is in heavy use, causing heavy cache bounce and overhead. Change them to percpu_counter greatly improve the performance. Since there is one percpu struct per namespace, additional memory cost is minimal. Reading of the count done in msgctl call, which is infrequent. So the need to sum up the counts in each CPU is infrequent. Changes in v3: 1. Add comment and change log for the new function percpu_counter_add_local. Who should use it and who shouldn't. Changes in v2: 1. Separate the original patch into two patches. 2. Add error handling for percpu_counter_init. The performance gain increases as the threads of workload become larger. Performance gain: 3.38x CPU: ICX 8380 x 2 sockets Core number: 40 x 2 physical cores Benchmark: pts/stress-ng-1.4.0 -- system v message passing (160 threads) Regards Jiebin