From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09479C433EF for ; Fri, 3 Sep 2021 01:57:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9658960FD7 for ; Fri, 3 Sep 2021 01:57:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9658960FD7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 28ECC8D0002; Thu, 2 Sep 2021 21:57:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 23E208D0001; Thu, 2 Sep 2021 21:57:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B78A8D0002; Thu, 2 Sep 2021 21:57:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id EF6878D0001 for ; Thu, 2 Sep 2021 21:57:40 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9CFE92CBC0 for ; Fri, 3 Sep 2021 01:57:40 +0000 (UTC) X-FDA: 78544600680.18.EEE8CF8 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf20.hostedemail.com (Postfix) with ESMTP id 036C1D0000AB for ; Fri, 3 Sep 2021 01:57:39 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 182MsGKE011369; Fri, 3 Sep 2021 01:57:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=rkHAlX6IBqy8crBChffRorzZD2BVDk1phF0XKST6I+g=; b=RuwK7oyllfDNgoBBBmA+VfrZtrlkMA2UtSyFfV8gJKAqxuMIXCc1mmKI6C+2Pq4Cze7o v7cG4H54paeofTwNcrQX0cV1zeZxo3UO4B+WFpBAOZcLWm/QL4Ag09J7AfxNiCPYNP/O LjEOi+xVuuL2IlOzODe9Fsyu8ktm7ksUB6Nd+PtezQICWoYpSZfBu2VtrbSQX9Hv2Ifd sc1hKy9nfDv1VHzmow/ZjidP253PoCZBy63r5UYqjexByA7449+gMRmkG1n3/LmWUz/0 +IpupYwW9Z2gMNoDS6o7EeJ4vSeLKYJWh9CxCFvEcplN9brHFzZFhrqEN5+WBb0dwp76 Dw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=corp-2020-01-29; bh=rkHAlX6IBqy8crBChffRorzZD2BVDk1phF0XKST6I+g=; b=WkXSdDq6lid7+Gf+o0fOcDVoxDBLIRumOTFjwjsr+fXmFqhSbiPDwfPKa3ZonXWDOoMp FJ9HUaWhQUzH5MnD6VKfDN2iAYyTMRiopSywnZDpiEbkahTUmR30fOtmNj4R1TC27OdO gKxJAkEyE3fx9l+ZtvzaaAFDS8USGa4arGrHZyvFqFuKUEOWH5lv9CsCaz/nQvTZWEJ0 BymrjBTxwE+5wDbOmVo5aIjpFuAygQpozFumG52ogB2I9PU0zgC2bPNsqfrBxSXG4LAp txNrQ2o+n9mNw3vbFizFHo9LxEEHtHUqbQHCDP97iepqyGAKLJX6lhr3auQwlAW46j4K vg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3atdw0mpg1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 03 Sep 2021 01:57:37 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 1831ttLp003108; Fri, 3 Sep 2021 01:57:36 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2107.outbound.protection.outlook.com [104.47.58.107]) by aserp3030.oracle.com with ESMTP id 3atdyxus97-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 03 Sep 2021 01:57:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h6smUS7WQrfeBbUtB9RDu69vKSSxRj2WKXqOZwkAZsDqBRuVexLp6QGNz7NglA2dzFQ73f4I+qTid9r2YC4BVXlQswbL18NUN2Mu8mMpLZIsJwSCyNWZqmOc67Iu7zdr1ItZhTbxO6eJJ79Uft4Fe/ay2ywWQKe7qcCYEGK89+caXElEtP7he6H/n3WfxZD8ajfCuu9clFFf024cnELaXhSvW8Iil75FOJU4AoSGnHJCHQHsyPdZGNXkWmFcvdHTU+tGW/3zcT9bGKrKzQirtQNEFV2L7an0Bo6SJPZ4E9ujhpTBKjGDJrnHOvnKJRKBymGorBnZa2Rn5Do/t7jY2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rkHAlX6IBqy8crBChffRorzZD2BVDk1phF0XKST6I+g=; b=GOIuEfxVlqRp0LimJEqeQhs8fTpkGvGL2MirVWQSdiAA7DSxjp7RnzdiB1aEpZ3F1wzLBRxPlxY7ObNOPDkR9Y//7K5DxmpYEid4Fe3oVnFbBTs8QdZGRXX924rkR/PsC36SqSnaBbSh83M/vHdBYvnhpI00hHirtlk+DrodpfpzocEigrKuW3BLJqBfr86iTGv7UCl4n5gNFcbe5Y3kYrXdqa6DnJxvsYyhFSEt+QeTn8WeOQZfYzvK92Atw5k/ci1k0f0V0lFC37oK9c7lI8WZ+z9z+z+cZEmlDfHsBceXHk/Bsu87j6ttVkKQJgwXL2jnavmMFt4svQaSclFXTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rkHAlX6IBqy8crBChffRorzZD2BVDk1phF0XKST6I+g=; b=x+ABJGAj8AwKbrY9h2V6kaQWEgMABxGQls+aV1+SnHZUOtT2yQB5q53ruItj71KdSiILPu3OeJbN77VDwbJZw7B37s5bpzljG5hEicHPX0FyEb63BnIDTkUyrs23/TElkV+trgop/kGOlyDQKNlFy+5L3nrNoymU5pEiDCmEg/s= Received: from DM6PR10MB4380.namprd10.prod.outlook.com (2603:10b6:5:223::19) by DM5PR1001MB2218.namprd10.prod.outlook.com (2603:10b6:4:35::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.24; Fri, 3 Sep 2021 01:57:34 +0000 Received: from DM6PR10MB4380.namprd10.prod.outlook.com ([fe80::f4b1:d890:7745:30a3]) by DM6PR10MB4380.namprd10.prod.outlook.com ([fe80::f4b1:d890:7745:30a3%4]) with mapi id 15.20.4478.020; Fri, 3 Sep 2021 01:57:34 +0000 From: Liam Howlett To: Vlastimil Babka CC: "linux-mm@kvack.org" , Michel Lespinasse Subject: Re: Strange mmap_lock tracepoint Thread-Topic: Strange mmap_lock tracepoint Thread-Index: AQHXnqhh7VimgmVH/ESa/cE4sY7tPquPAXcAgABoJ4CAAicnAA== Date: Fri, 3 Sep 2021 01:57:34 +0000 Message-ID: <20210903015726.emcfobqgddagvytd@revolver> References: <20210831203959.lwofr24z63wjsgkc@revolver> <1585872a-1562-c74a-b686-e0051fa75cda@suse.cz> <20210901170447.fcqdnw43q7cmamzu@revolver> In-Reply-To: <20210901170447.fcqdnw43q7cmamzu@revolver> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f02df589-4ff9-4a7e-3960-08d96e7e32c9 x-ms-traffictypediagnostic: DM5PR1001MB2218: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: yRiFReZsG8GVZuGRryCOAC80MDG4EkQnOeV0UvkCaI7KAlwvQKc48DQB2eMaxwqRr69DmoxeXTXoHVeoH79IE83u24kMSVuDeZznRqUn5bpT/Y3DTRkc6WXJp9vrW4SZhAOXmeWDP8lQrLkpaB0HjiGklyY7bhx2mLaUAGO81oDhvqX415Y6Tu3FtFw38JeJe+Qa5RpSyS7EYEiolIEa46b//ixnVFXU86h+GWh5CyWI5WGP4a/jyg2SB1nsXgCq1Sd1ROoOiFz6cj86mU6KaRwRbjl1tIqliAjbdAJq7EUMniBRXnGXYJJufupojzBoIkmMZlfOFRG8ty8bog4AY40M9atlxBSe6yzgUPk/uHkNWgMMtIObHsDrsu5Y/1BB/6dHM89K+KfsUwRc+3lhlW6hX9cEHGE9pz6NlqepDLzrYaa9RhSqTuxRgf0uI8xqu20mDUcVBOfl/4sPzxu28xnPV7lHejSm31wZ97YsUmrRomkiwesaN3GuDXDMBrFzy61ZNhf8IDcbCjVq8xhhAQWM6VKlJxqrW4QOufhDypWK89dA7ivg+Ajn7LLKmIBACrv7RX/b8A2d/eHOzrNPfw8aggX4QBCnD0c0KfOxi1s+t/Bh5OCDDjjBk6e4ayZ4Q5MzviLHuz7CRYg+glAHbLsfOnozuf4Q7oFiNzoRjXnJOMZKlHWbLgy8OPWQsewb3A72WRYqXrO5oLzY7M6WEIPo5tm60E5NAbaavIWpv2/kTAM24J1sMbTLsJcXZ9gg8Iz+H0w0oi9MrPWMKEbHFLkUO8M1fg5bQJvjNbwGheM= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4380.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(7916004)(376002)(366004)(136003)(396003)(39860400002)(346002)(53546011)(3716004)(71200400001)(8936002)(66476007)(6506007)(2906002)(86362001)(26005)(54906003)(186003)(66946007)(91956017)(316002)(66556008)(38100700002)(478600001)(7116003)(966005)(83380400001)(9686003)(6512007)(6486002)(64756008)(38070700005)(122000001)(44832011)(5660300002)(66446008)(76116006)(33716001)(1076003)(8676002)(6916009)(4326008);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LthM57WQieq/rjoEwWCPW1chf3a5HzrfSTWcNHgPK2KgafqRqT+YQRJ7OgiB?= =?us-ascii?Q?WICtieI08ab79ztrxVsfCkwssH/RsmuaK6xwdr4cyMJGboeddMwUiF4LLg6X?= =?us-ascii?Q?MwPtrb/kWirZyqs5oJhBI9BZR+mzpt/vvF/eJmdxO/SHpiwprvpEmfnvBUiE?= =?us-ascii?Q?mxO0afYGE/feh78+AWcnIoW38VOpDhhgmd/jmuSZ9cZG8HePaU+kdYBpYMgj?= =?us-ascii?Q?Jw0qOAd4v2KEYeqj3T24DlJjGeAqH9hOWBFXNZppSirn8yhMjwVXX9flgWRl?= =?us-ascii?Q?PwFXQEPFU+ncqLwc55ITQlMF284E0aqaslbb0XuvaDKbHQo3w5NTa4tLqV1S?= =?us-ascii?Q?A3s2RZYu6X0ZucdrvRR5pCKDMp0mB+BjS6h0T6C70rBhZxazQcaUXj5BE5Ja?= =?us-ascii?Q?hLnCFt9BFvqdLNVX4HdaGd/BcpcCvOmjP/g9oHzXFecjjo1+KS+kgUIv203d?= =?us-ascii?Q?Eck/2EI0cKsYfbXMZ6yv0ddVri4yNmy6SYLdPlwQzMNng/POXQpwSFhuhrw2?= =?us-ascii?Q?g78ddciZ32oIAl8pId1JdooWhlhttYtZbX9W9fdYByw/ARCKQHkvC1GXHOxB?= =?us-ascii?Q?f1LKXD3XG1AzOHzz6dlvZf5rg5n/7MaHl5qMwiNi1kOryN6mZTJvYtu4IBtB?= =?us-ascii?Q?TV016rTiL9+o9WdIddO7tE4InSw+jXRgxldGYzXy+nBgfbZc85bEshLzUIYb?= =?us-ascii?Q?6L9jkV2OhhAkg1F8jsJtVI8z2JLSfGx9vS3h5Qp3ByPpHgzvInp/Zu0/TeHm?= =?us-ascii?Q?OKm2Vrm7tCWwny9rEjFyOJQSL72c1tsYXdepKVfDeWk6V7fDs+aZpPtXJvg+?= =?us-ascii?Q?qAqYQ+nCNffW/seE5KTXBJJV37PtovkyzGQ5IJYbVDAzYYeXjmNTp557VerB?= =?us-ascii?Q?oy/8Sdb+k+u6VJHru/lAW6zfv/nsrtPB3XSkcDpEA4GwIRNjk2fb/fukXotz?= =?us-ascii?Q?6guRLB7OiTvB9IIJqwqSdezzpGSVFEYlSIEU0LrOlby3nimsbReLbJK4ntlU?= =?us-ascii?Q?I93LSolr78kWKcMuEMXylB20J36iUAnxKXQ7PpkLIfuLMi4qnr5V4UHggjZo?= =?us-ascii?Q?8XFjkCj9d/HlXDeq1JQPngLe+lhhPnwPtFIqqsLzmR8D+xFhAiZsrZdcd6tB?= =?us-ascii?Q?wTf1r0+VpXFJboeSVx8ufzbtLgX3xz9324pEl3NS3psEq0ToGSBgBd1K+6Qy?= =?us-ascii?Q?6ddfz4A+fnK74T6irqTQdnGL9HiSoVmYJxGJS8NKnt4FQiUvHIeOyfRzyX1v?= =?us-ascii?Q?kUL4533UWSsZLE3PVG299cx7mFA2mf6r8s0xyj8tD8QAZTJ1n8xB1kQ47v2e?= =?us-ascii?Q?L2sHvS1IoK+cbYH5VniZaScb?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-ID: <214BD3947453104FAF9F1D798BC1BE97@namprd10.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4380.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f02df589-4ff9-4a7e-3960-08d96e7e32c9 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Sep 2021 01:57:34.5204 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: EAyTXXlwzorJJR27rkajMWj9k7221rWxQwrd/0wrMWS/il4aftrrcW9qsn3MXusrH/JPzo4uWFYoByni34Am3A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1001MB2218 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10095 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 phishscore=0 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2108310000 definitions=main-2109030010 X-Proofpoint-GUID: K7oV9rufke80fRNrUVEp_8wV_Q3ddwsL X-Proofpoint-ORIG-GUID: K7oV9rufke80fRNrUVEp_8wV_Q3ddwsL Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=RuwK7oyl; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=WkXSdDq6; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=x+ABJGAj; spf=none (imf20.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=liam.howlett@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 036C1D0000AB X-Stat-Signature: hgrtom3r3zm4sda76ior3di938qx6i53 X-HE-Tag: 1630634259-221583 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: * Liam Howlett [210901 13:05]: > * Vlastimil Babka [210901 06:52]: > > On 8/31/21 22:40, Liam Howlett wrote: > > >=20 > > > Hello, > > >=20 > > > I've been trying to trace the mmap_lock calls using tracepoints and > > > captured this behaviour which I cannot explain. This was with the ma= ple > > > tree v2 patches running ebizzy with multiple threads in an x86_64 KVM > > > using 8 virtual CPUs. > > >=20 > > > AFAICT, there are zero callers that use the mmap_lock directly beside= s a > > > prefetchw(&mm->mmap_lock); > > >=20 > > >=20 > > > ebizzy-803 [000] .... 5376.655366: lock_release: 00000= 000de8cf25e lock > > > ebizzy-803 [000] .... 5376.655366: lock_release: 00000= 000b5e38448 ptlock_ptr(page) > > > ebizzy-803 [000] .... 5376.655367: lock_release: 00000= 0006b581afd &mm->mmap_lock > > > ebizzy-803 [000] .... 5376.655367: mmap_lock_released:= mm=3D000000001de7b122 memcg_path=3D write=3Dfalse > > > =20 > > > ebizzy-803 [000] .... 5376.655369: mmap_lock_start_loc= king: mm=3D000000001de7b122 memcg_path=3D write=3Dfalse > > > =20 > > > ebizzy-803 [000] .... 5376.655369: mmap_lock_acquire_r= eturned: mm=3D000000001de7b122 memcg_path=3D write=3Dfalse success=3Dfalse > > > =20 > > > ebizzy-803 [000] .... 5376.655369: mmap_lock_start_loc= king: mm=3D000000001de7b122 memcg_path=3D write=3Dfalse > >=20 > > I was thinking khugepaged interference, but this seems to be a system-w= ide > > tracing (judging from -0) so you would see khugepaged too, right? >=20 > Correct. All calls will go through Michel's tracepoints from what I can > tell. >=20 > >=20 > > In the other hand it seems strange to have a long list of just cpu 0 he= re. > > Are other CPU's missing or just the interleaving is imperfect because > > timestamps are not perfectly in sync between cpus, and in fact there wa= s > > another CPU who took the lock? >=20 > Yes, it could be the clock. I had used the default clock which is local > to CPUs. I was looking for documentation in 'time stamp' but should > have looked for 'clock'. I've re-run the test with counter, which > should remove the potential of incorrect ordering. I've also imported > into a spreadsheet and sorted by the counter to ensure there isn't > interleaving of printing causing issues. This is why the output below > has a slightly different whitespace than the original. >=20 >=20 > Please note that this is now next-20210811 directly. No maple tree > patches. >=20 >=20 > As per the grouping of each thread, I don't really have an answer. > There are places in the trace which do have different CPUs and threads > running interleaved: >=20 > ebizzy-1445 [000] .... 83693711 : lock_release: 0000000055bc357b &mm->= mmap_lock > ebizzy-1445 [000] .... 83693715 : mmap_lock_released: mm=3D00000000ce5= bd903 memcg_path=3D write=3Dfalse > ebizzy-1437 [003] .... 83693727 : mmap_lock_start_locking: mm=3D000000= 00ce5bd903 memcg_path=3D write=3Dfalse > ebizzy-1437 [003] .... 83693730 : lock_acquire: 0000000055bc357b try r= ead &mm->mmap_lock > ebizzy-1437 [003] .... 83693733 : mmap_lock_acquire_returned: mm=3D000= 00000ce5bd903 memcg_path=3D write=3Dfalse success=3Dtrue > ebizzy-1437 [003] .... 83693738 : lock_acquire: 000000005ada5d35 fs_re= claim > ebizzy-1437 [003] .... 83693742 : lock_acquire: 000000009ff6ca04 mmu_n= otifier_invalidate_range_start > ebizzy-1445 [000] .... 83693743 : mmap_lock_start_locking: mm=3D000000= 00ce5bd903 memcg_path=3D write=3Dfalse > ebizzy-1437 [003] .... 83693747 : lock_release: 000000009ff6ca04 mmu_n= otifier_invalidate_range_start >=20 >=20 > > > > >=20 > > > The following tracepoints were enabled: > > > events/mmap_lock/enable > > > events/lock/enable > > >=20 > > > My reading of the above trace is that the ebizzy thread dropped the l= ock > > > and immediately attempted to reacquire and detected it was in content= ion > > > so the thread added itself to the list and went to sleep only to wake= up > > > and get the lock? > >=20 > > Which path does this mmap_read_trylock() followed by immediate > > mmap_read_lock() anyway? I mean down_read() is implemented like this > > internally, but that wouldn't generate these mmap_lock_ events. >=20 > I think it's do_user_addr_fault()? >=20 > >=20 > > > Doesn't that mean the mmap_lock is _not_ contended? The lack of > > > contention makes sense since there is no tracepoint of an attempt to > > > acquire the lock between the dropping of the lock and the same MM wak= ing > > > up to get the lock. > > >=20 > > > What is even stranger is if I change the value of _Q_PENDING_LOOPS fr= om > > > (1 << 9) to (1 << 10), the benchmark performs better. > > >=20 > > > From the above trace and the effects of the _Q_PENDING_LOOPS change, = it > > > looks like the lock is somehow remaining in a state that causes a > > > failure to acquire the lock even when it is not held or contended? >=20 > So the new log is with trace_clock set to counter with next-20210811 > running ./ebizzy -m (only use mmap) on a KVM with 8 CPUs. I was able to get to the bottom what what was happening with a lot of help from Steven Rostedt. The logs can be explained by the following: - Sequence counters may be missed if a given CPU overruns its ring buffer. These are marked by #### CPU N buffer started #### or something similar. - There may also be a jump in the sequence counter by interrupts. - The mmap_lock logging was racy with other threads logging of mmap_lock. I have sent out a patch [1] to fix the race. - When downgrading the writer, the log will show that the lock is acquired as a read lock (write=3Dfalse) without a 'start locking' log. 1. https://lore.kernel.org/linux-mm/20210903013521.1802774-1-Liam.Howlett@= oracle.com/ Thanks everyone for the help in tracking down the odd behaviour. It was nice to find out it wasn't entirely user error :) Cheers, Liam=