From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 732D2C0015E for ; Tue, 11 Jul 2023 16:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03A706B0072; Tue, 11 Jul 2023 12:53:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2CC26B0074; Tue, 11 Jul 2023 12:53:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF49A6B0075; Tue, 11 Jul 2023 12:53:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CEE666B0072 for ; Tue, 11 Jul 2023 12:53:31 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 97A37140324 for ; Tue, 11 Jul 2023 16:53:31 +0000 (UTC) X-FDA: 80999927022.20.17383C1 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf23.hostedemail.com (Postfix) with ESMTP id 3FC1A140016 for ; Tue, 11 Jul 2023 16:53:27 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gV9rvdCD; spf=pass (imf23.hostedemail.com: domain of yu.ma@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=yu.ma@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689094409; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bpZKN3VukyMZTGQwqbuxnMeKA14E4h8FT0XV8eGRPsA=; b=2rEcGI3Y6G3wXQ/N97WgQdMW2pPUkC87BOnAr8XvAypAX/ZTKJGpmSpo1sZVyYzpRyb8jg pZX1xCg74TeYI11gqt7Pk/08CWG1NzSnRtkEXj3FG9wQLyEoFF0x9T93tQBNgseFXgIf9Q 3hbNli1xouCtmm60chA+jTn4nqHmUcI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gV9rvdCD; spf=pass (imf23.hostedemail.com: domain of yu.ma@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=yu.ma@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689094409; a=rsa-sha256; cv=none; b=8JfR8T46O5GBaHbaikP4ibOtwDfdVHGJtZprQ0S75MfP2AabNesLyKhV3YFY2x744qix/z A/a6oZFcFazkZCgkxehSnmCTpw8wNB6frziRBo0HJuG9SbLoqz8/Q/4yv/gq/GoenG04X2 YIqsQE+AwHymuMgthz+nxOP3yF/86us= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689094408; x=1720630408; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TrplBJ3bzzs7m5iDqf6Oh6aYMUB0SNPyHYHI7Jj6/7o=; b=gV9rvdCDM5YJsQJBBCCssoUzUvuvJ6oX6mv9GDLEFMRwjNXhODD5YGXL 2AQ+UP3CE4ShX+v902KOzbiYQuIF9T9/0+krWTaLhC0hRfjMRHqryC7/5 ZhnHoG2jFWInK7v4PVgJeqjAjXjh8PgC/9IGqft9wpwvRe7G8OitqAcbW uWTZRL3jU47VMge0Jnx0JMA35M4FdWnHrfSydV06UqW97TNX0M+PioYRm Ju6Fe7yMpsprEXrDgW+jBdKGyy/lv1dbT8UrpLEkhLkWL/ZBxmJTfIoti ZwoA4uTEBLBdKPaTXCuLCiDfSwDwe45TLL24/W9mkBPUcesS5k63rUvWQ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10768"; a="354539102" X-IronPort-AV: E=Sophos;i="6.01,197,1684825200"; d="scan'208";a="354539102" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jul 2023 09:53:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10768"; a="967873360" X-IronPort-AV: E=Sophos;i="6.01,197,1684825200"; d="scan'208";a="967873360" Received: from linux-pnp-server-20.sh.intel.com ([10.239.146.185]) by fmsmga006.fm.intel.com with ESMTP; 11 Jul 2023 09:53:18 -0700 From: Yu Ma To: yu.ma@intel.com, Liam.Howlett@Oracle.com, kirill@shutemov.name, akpm@linux-foundation.org Cc: dan.j.williams@intel.com, dave.hansen@intel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lipeng.zhu@intel.com, pan.deng@intel.com, shakeelb@google.com, tianyou.li@intel.com, tim.c.chen@intel.com, tim.c.chen@linux.intel.com Subject: [PATCH v2] mm/mmap: move vma operations to mm_struct out of the critical section of file mapping lock Date: Tue, 11 Jul 2023 13:20:20 -0400 Message-Id: <20230711172020.562250-1-yu.ma@intel.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 3FC1A140016 X-Rspam-User: X-Stat-Signature: kbjiga7n8695e1nnocyzott53inbt1zr X-Rspamd-Server: rspam01 X-HE-Tag: 1689094407-681840 X-HE-Meta: U2FsdGVkX18iLRl2zjtM9mykqdMcErky6ETEvvW9pimWKuHyayMMGO7JGsx9Jd3pTUqerPjhum5r+hxD1x7D9i2d8o9ya06mJAgP/D2YlAtPITOLLc1y6+ESNa44tNond7XNDDGp8uZE6tRFxFlv6b9GUae/CXcPLt7Mh+X3K4ex+pwEIkkj/jXar/92hh95IVwJasNx86YFhBhLNgIonEK73+xL0XZcGfcrVsDi1S/evdgUHSoKxyvgC/uYb2Gt76XpB0i8zk4gb6PK7mHA9uQVu1DDi6mQj/i0G/cSTSpsxhtAzs2k4sHjErLxmobkszOSVwbRmpAGvCMzk4bxQsbtJHOyJnoZnqIGSPWy0qRb+qBcNvi4FMfD5GgiCYPX+tdk97aVdmCC1FECQnwmFm6RU6BbuLGxWnl/BrXRq2y9OdsZc3GAh34UGCLP7RAzQvEx+Z1TiWJkgkob0noc25LdW5D9dle7vkH3oIuix23JXGPghL23thH/Ydvc/ia71g+Vymh1CCmzsHfHUnhcJuoMuX1dCasRi9vML4V47cwnub5ND/na5vRSXKPAG99SS9IofPgjRMPo90lwAk9XC13zd2YHlFFemHWsiYMpMGC0qIqOvrhg+ZAEq2/KQyDoQHBH2otyEORUOl54kXnzSZZlI7bn/EeUaDV8tB3JbkPdhCBiTjemEt62keL1R5NW6ONNIaUEo2v52ztBXsV47uwAPBGtoHepkOdvAoMaEeAjX1Yl2Y+HRPX8kJD/T6zvHoFxxU7cHnaCOFUihxH5cdca4LTRBpwuW5mCLwhuQF2Tg8mx0NKD4g9k13HgvacoM7eBGXTEiutNYaRj5NDx2GI7ad0xqmdZbCRhkS9EKU9tEGsscKTvcPWjL0D1RAO2h81B7Q5cSnsxqPDRYURNVbhJTfBymMriqhYJ0/qVIyx7KInfIvkl9O2wtpj1dwx9X4hgLNN4fVylFeR9ol5 KUX6FcFp M6cJyPpQxavhgYNkJyyFMxtSxwRIqjparb1dU+XYyN1CEJy2ilqCHX9VybUxYSfgZ4NbcavNJUkInRehJBMgR4VwX0lFplJoEcYjNixCaFJBuR8esAfIVBW88HJyxXj+x9HL+3Qz2M70xU8bekgMMRJrU4o61WmaUWl2OugU9F+2U9pARDrExxBpuHQ0h6Hzx/iz98mle2u0a+52HVCMCvxUAsINQTkodSTR7tuMdc6XytgZD5IkI+IWWX7Ql2HhgKi5CB+V/qtAgDuk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: UnixBench/Execl represents a class of workload where bash scripts are spawned frequently to do some short jobs. When running multiple parallel tasks, hot osq_lock is observed from do_mmap and exit_mmap. Both of them come from load_elf_binary through the call chain "execl->do_execveat_common->bprm_execve->load_elf_binary". In do_mmap,it will call mmap_region to create vma node, initialize it and insert it to vma maintain structure in mm_struct and i_mmap tree of the mapping file, then increase map_count to record the number of vma nodes used. The hot osq_lock is to protect operations on file’s i_mmap tree. For the mm_struct member change like vma insertion and map_count update, they do not affect i_mmap tree. Move those operations out of the lock's critical section, to reduce hold time on the lock. With this change, on Intel Sapphire Rapids 112C/224T platform, based on v6.0-rc6, the 160 parallel score improves by 12%. The patch has no obvious performance gain on v6.4-rc4 due to regression of this benchmark from this commit f1a7941243c102a44e8847e3b94ff4ff3ec56f25 (mm: convert mm's rss stats into percpu_counter). Related discussion and conclusion can be referred at the mail thread initiated by 0day as below: Link: https://lore.kernel.org/linux-mm/a4aa2e13-7187-600b-c628-7e8fb108def0@intel.com/ Reviewed-by: Tim Chen Signed-off-by: Yu Ma --- v1 -> v2: - Update vma_link() to reduce the hold time on file mapping lock as well. Based on v6.4-rc7, vma_link() is only called by insert_vm_struct () and copy_vma(), which are both protected by mmap_lock. --- mm/mmap.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index d600404580b2..6f42ca2ab84a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -444,14 +444,11 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) if (vma_iter_prealloc(&vmi)) return -ENOMEM; + vma_iter_store(&vmi, vma); + if (vma->vm_file) { mapping = vma->vm_file->f_mapping; i_mmap_lock_write(mapping); - } - - vma_iter_store(&vmi, vma); - - if (mapping) { __vma_link_file(vma, mapping); i_mmap_unlock_write(mapping); } @@ -2708,12 +2705,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr, if (vma_iter_prealloc(&vmi)) goto close_and_free_vma; - if (vma->vm_file) - i_mmap_lock_write(vma->vm_file->f_mapping); - vma_iter_store(&vmi, vma); mm->map_count++; if (vma->vm_file) { + i_mmap_lock_write(vma->vm_file->f_mapping); if (vma->vm_flags & VM_SHARED) mapping_allow_writable(vma->vm_file->f_mapping); -- 2.39.3