From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D813AC4345F for ; Fri, 12 Apr 2024 08:07:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 222706B0089; Fri, 12 Apr 2024 04:07:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D2CC6B0093; Fri, 12 Apr 2024 04:07:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 024DB6B0095; Fri, 12 Apr 2024 04:07:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D5D8E6B0089 for ; Fri, 12 Apr 2024 04:07:29 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5FC11A0DC8 for ; Fri, 12 Apr 2024 08:07:29 +0000 (UTC) X-FDA: 82000150218.29.0CA41E7 Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2057.outbound.protection.outlook.com [40.107.215.57]) by imf24.hostedemail.com (Postfix) with ESMTP id 02195180019 for ; Fri, 12 Apr 2024 08:07:25 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=oppo.com header.s=selector1 header.b=oiRW40pV; dmarc=pass (policy=quarantine) header.from=oppo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf24.hostedemail.com: domain of lipeifeng@oppo.com designates 40.107.215.57 as permitted sender) smtp.mailfrom=lipeifeng@oppo.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712909246; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=QOMLzTAf430Yv7GQG6MHNG0SQhYE5EfZjOPPcbet9/k=; b=j5OCMt+AZMInsDSsecNxj8ZqnNjzh1TOF4zHRJY3DhXBvgVW1YwzuNiAfZVcIoKJwmmebi uE8qDPr0frcWlWNcTauiDDLp2YnTLzQAaI1Vbb0F0HYv21Wv1ggOsXyXx8GTeU94HQbnEK u+HesrnH+Bk7xQ/ojyhGIavfV51iV/o= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=oppo.com header.s=selector1 header.b=oiRW40pV; dmarc=pass (policy=quarantine) header.from=oppo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf24.hostedemail.com: domain of lipeifeng@oppo.com designates 40.107.215.57 as permitted sender) smtp.mailfrom=lipeifeng@oppo.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1712909246; a=rsa-sha256; cv=pass; b=hm4DPq8Zi782qxl5hbcL3E1XLH+qNQFcCfm+fRzCvWme777LGw/gzPL/y+5+HOtUdToHSQ E5FcmyucnM9nxMAdSElwtJbGh32ISvxzFua3S3Z8vKLPt1hH39v215mvhDchI+brjU3s0N DWtkjTzwZzzPBegDUz4wHHXDBO35qBA= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YfcW8XMlyMB2rpnBstSocOjM9Y7Sdv2fg5jeHH2q9XVW+juOr4cXaKvSbgq39ZED/V/T62mO4o3bkmifpmOUo8+433S+InVRGnBCSlB7n1QqP8uy7CGWUi1ANTXVRulN9JZ/fa2fhaRHEKvZhiOY+aW7E1D3SG597loItxeCfHyqsWlVLpsY8/LvQ1xGZxo1ZsuvrNSBvtk5UQGJy1fsIUYdiypLD3X4iJmHI5kHI3fZBJoCWeAYzIMLbk8RoEQa1MII6FRnCjays2LPYcm4llWBJ3XZljGCdaXP3iSsFb4PfrbQuzTPs1SiLtwh5l4HRj6HP6//9baMGwbhFkfKLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QOMLzTAf430Yv7GQG6MHNG0SQhYE5EfZjOPPcbet9/k=; b=XCUlTmVxQTbXhk5AQ4BwDHTHWa+x/JhklHUwuWlfbt5uIMj5Y9zmfm4ZgZ3Xy9FuLfRtpNljmVvUmhdaHeLQ7dZh0lfeSDlylWXmtuUnJZNjW/H8HIckHpaTg9aCPi7hGbz1zc3K9WegOMUdvLWLxIHw1rYN7BMlTHd9L5yvYKzs8euzq43fYUY0YUAZHoeSXHm8SLfkcyPhwJjnMFFi1Pkg05bVrU/ZgXaSzpq/dG86cWW+QrmOUOCRdE8zx5hJgv17dYbsWRJen85OmkNvk/8EP8GJlGqBaW7ngMMfsPiT7l8QdIyaHh8TrMHCE9YtuBWwRhsosLzlQUCsUKB68g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oppo.com; dmarc=pass action=none header.from=oppo.com; dkim=pass header.d=oppo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oppo.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QOMLzTAf430Yv7GQG6MHNG0SQhYE5EfZjOPPcbet9/k=; b=oiRW40pV7qG2cemGtEcrY7keIOQAnnf6v/zPqISTeC3KIN/JQwxvVYIAUg64EfcdJ4tW2Jjc3UP3TSV6mije4sK9vHMdS5afwdmgiSRGNX0h0sYFw0IxYUvNdKNQKFmNp/jQYlLXK63zv941bb4LF3CpLKNkg5Vrh08xDUmhi/Q= Received: from TYZPR02MB5595.apcprd02.prod.outlook.com (2603:1096:400:1c5::12) by JH0PR02MB7549.apcprd02.prod.outlook.com (2603:1096:990:69::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Fri, 12 Apr 2024 08:07:20 +0000 Received: from TYZPR02MB5595.apcprd02.prod.outlook.com ([fe80::60d1:4349:510c:8133]) by TYZPR02MB5595.apcprd02.prod.outlook.com ([fe80::60d1:4349:510c:8133%5]) with mapi id 15.20.7409.042; Fri, 12 Apr 2024 08:07:19 +0000 From: lipeifeng@oppo.com To: lipeifeng@oppo.com, akpm@linux-foundation.org, david@fromorbit.com, zhengqi.arch@bytedance.com, roman.gushchin@linux.dev, muchun.song@linux.dev Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH] mm/shrinker: add SHRINKER_NO_DIRECT_RECLAIM Date: Fri, 12 Apr 2024 16:07:06 +0800 Message-Id: <20240412080706.744-1-lipeifeng@oppo.com> X-Mailer: git-send-email 2.34.1 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: SI2PR01CA0040.apcprd01.prod.exchangelabs.com (2603:1096:4:193::14) To TYZPR02MB5595.apcprd02.prod.outlook.com (2603:1096:400:1c5::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: TYZPR02MB5595:EE_|JH0PR02MB7549:EE_ X-MS-Office365-Filtering-Correlation-Id: 934f5831-5776-4959-8a64-08dc5ac792f7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qL07g8B+u9atSSj7SjevhMwVZSWumxYnwIfSYqM4NHAxadvV/wpHYwTIXUcqjIYhTEYwvY3PuWzgQA4gVN064ZJHRnYdAHJU7aPABBOe+tvybLvG25TpePIn/VCnlZx3EVFxA6TEe2JqwC+3mWkitavU/Nfjn95zEkbG8gTHE/BFrJmuZn6Iw2T3gTaXr1iVWrQciHMIS8s2Wt6vzgno8Me7LkkdtMKDD+Rv5CWDC1WpqPFBuA+qBjThqu+w4XDzPAFi8xlWxaC176t9hm+qiKKsq+Z0tY0m5juyCy98yImXYo0OY5WCyf9jlUjwoKkfV0FQVptJs/VJ1F5TPn+5kW4OPt38yJSRn5mdGynzegNoWhn23ohNJDjDk+3G4err6vUEdoy71rdD46weq9cD6Nm3h/seF0TPj48AcZ3oUbxeqvFjgJzWC3BM+JEiT4/6kEkzTkhm8BTcQ8faaTdVQc/P6cT2xAelKj4gCv7jI/ifDGub6jlFNBjlVmaF0lqf16Yc6d7dzNLkreUOc73UkOM57IBSP3bGAudMUzjAD1pyc5ncMZHzY7+6xlm2FPPwDxv3qROdSrG5nNLknT2uk+S3csAt/P3o8Ni671rPo+na7LEQf1cPhpmawu2vyaXPLDOSkp5Zwe3rhWypKvTGlJDC1HpxMPUj9L12enUmShGrH2LelDSd9uVV93OL+c1Qt7dG6HLL9aD3jTAAmdJabLj4PJdKIOO9ZbzM0CLSvPI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:TYZPR02MB5595.apcprd02.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(52116005)(376005)(1800799015)(366007)(38350700005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?oGLf+IeVgpj+XI8CSi1Ax4kJCfFOPm1zPxQaAQJ7nlETaTOzputRPh2fx+4K?= =?us-ascii?Q?RRqFGnSnrUWLhvn5xO6eCDaWETr+XcwI9fBi85hx98tWw17EgIMp+vbENBkb?= =?us-ascii?Q?4Nre6EPTNgwiO90W84hq80A1t4/sOdGK7PPMDqvHzFG7i60RwG/dXbSBc4pg?= =?us-ascii?Q?hxOGE6I2MpOqRuc8arwutdgkSXUCraNN7y8qdDV/EYnQdSQeh4rPEKZnOTEt?= =?us-ascii?Q?n+Z5rpxhf5/N8W2GtUAZRsD9meaTZ2s8hz9K09H1KN8m5WPSHC4XlKTkb7HV?= =?us-ascii?Q?6bt40fvwLKHBgThUYIw4bqkaXg+81nBRRRyRZeUl9Rn4KF9qR5jvtx3hFyB3?= =?us-ascii?Q?EhLxQ9c3S9x/dMUgcWPwxUjB3lYs/Omh02tKDpfr+zBXXLBv0j++CPg/Fa0W?= =?us-ascii?Q?a6shsHY1tbMIJsU1fgnB58JFgXTtKU+1CZdYaLFxAa9Ek3g796Ry+/la1x1g?= =?us-ascii?Q?MqAKqgeLBPOEWC9oIQ/foCzslxqwt/2usQX4Jxk2XBDY0jPjpJx6sTYfE4hK?= =?us-ascii?Q?di5NVAvRgNx3tSff7nBoNxjSwhxU+AmVbqdKBLqZLIeIufsAJUf7J5wLynGE?= =?us-ascii?Q?Cr7AvYlZd3fuTeQPubZ7g6313UllNa83mVL9OzLbbURmnTfcsMSpgrzhfBow?= =?us-ascii?Q?nIhUeYu3M9mVgsQ4fWXmASk7klEcmWtmf+U6y3LNkFdiBeuUkz4HNk14eOd8?= =?us-ascii?Q?+GkxhkX1BXSmwNGcdmKDYdOkJ422OrmD8EYyC8krnT9ro+dviKe+yi83ujL1?= =?us-ascii?Q?hN7v9jgkAq5HlHqx8AwGSeWzuelZyRqzI3ky1yUqD+kmBbklNxGT/FhVinj+?= =?us-ascii?Q?Lvp0xqwa05sIIluSOHXR5r3Y8Mty7p2/d7FEQ+cENYTZBxiAzZJ2OfSN5YK+?= =?us-ascii?Q?1suSQritVElJwrTPeukzJTy9QQXvVsp1KP7T/GVl/otyF7X3oZS/Fa3pxWEi?= =?us-ascii?Q?lxe7jWq0HFtrEWcxNbNthgpRRe0umPo2n6aDVtspOlPOC9udnYwW+hlM/6JE?= =?us-ascii?Q?Z26QdZ1vbg4S9sCV4Lu1cKB3eX8lSnVYV0k2B3SBed+wY0NhWy/kSj4bkzL/?= =?us-ascii?Q?ofl1RLRMiVclqZcWJLqS1xnj7AY89rK0sxA7syFU4gAiga+IS3RpzOkb/Ntf?= =?us-ascii?Q?a6lCotV5GlL11cmv2k/YlYniBZ0XH2J1XHoZ0wtt8/tvC2UzSkcSNOURccQQ?= =?us-ascii?Q?i3wcxd1DLvLQjwX0VesHpA891/kWZtndB/zAnFk0BspL5DT+mrNdmX0o78gm?= =?us-ascii?Q?KeZlh3hvwH72b4DUWbDa6po7S5mOS1CaT+ls0BQfmtqwWfK7O+fXoQncTFdm?= =?us-ascii?Q?l8L6wYJ0yLVokEYiyzAylGLNbpNvHF+PAUBD+22ICiOp13DA1CjZ4+VOPjMH?= =?us-ascii?Q?cXip1YN4xzVKbAtOs9bVqSY0fxLEAd2ndDIOxxMbmo27LtbP68cNYdqZrEDe?= =?us-ascii?Q?tzYlm2LYNtcc82hk9uAmHc2OVoMxwCMiji0erxJzywo2d9jmPPQSBoR3OK3P?= =?us-ascii?Q?hgPFwAeUfxZ7rdYSX443eC2Cq/XQ6vdRVupC4ZzYoxZFmS4AB1M8zt/HSKCw?= =?us-ascii?Q?AXA29ZrVlUrB01qkfXXWKaSK5AVH/3M0JbFVJ346?= X-OriginatorOrg: oppo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 934f5831-5776-4959-8a64-08dc5ac792f7 X-MS-Exchange-CrossTenant-AuthSource: TYZPR02MB5595.apcprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2024 08:07:19.1962 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f1905eb1-c353-41c5-9516-62b4a54b5ee6 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SJTNBmGBqienfADFx52klyxu5DfPj7SZqkc868B8gVx8XM5nqFxOSwb5d0czHGWcGxHK+yBLEFsprTfPVyBYsg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: JH0PR02MB7549 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 02195180019 X-Stat-Signature: 9dacco3i73a4ikd535mdun9nh96twzh4 X-HE-Tag: 1712909245-793229 X-HE-Meta: U2FsdGVkX18uwCadi9DNEOxttV9JyeL7TvuLnfCt+HWa6jJNIFLDz1M0KYT1u/x7KjsmH/z/km5bOjRC9HMuWcT+yA9d6WAcWGKvjmWWK/a/5YJHJShZSaKSgLkm0I5OfLzGyxP82E8b65dhp5Xl4EPFIqeykXU096Jgrit+fWRXfFqbn1sR8Aewe8+EScjrSR04/jBQsKUVUiaRbA6cnZNfoKHozpoYgRNFR2rh8ZLWxITWAoLT+i9VVeHBDiaYmSXEfTt2qhapZACOtBD3CKO0oKBsnsdjRJXFqjB3qr65p9p5Bcq1jaVve7ltxFFDztkVxRj7opFUYgC8nld2oj0ZEKmdkOJ0sMZjH+yLKSWxtoB+B5MqNLAvieyFfSpK0JcXD0/waVkL41ntbWBlaX7xnRh/aO9oTNvYbeXSHkjkdX/f9g/xCCl60Bz4rm7O+yC9jnra119pLNyVtkka5h1eNmAtWMKmhlIP/SA7vnUUbAbSYQOoapP7iD5liDjee38Pbg7CE8GyEPZXjRjlNCsvdQQIuL1T0769WGgWnpUtj1pVK97+v5ApuXGxoulOT2y6ddbgORSMH3NeMJMEj6BMWx0Xm/8UXTqQe+8veSjrZ1yrxpycYXQu2h0UP2NfVS1Ft1hqFVlhsvhZ5IThOVgx+mZziqVk0VTpQDp1fg6b9yl1+KjFLje3HQSu9yNfkYRkiOfkctN7c2DISVFZwzYiuLpERuHf8kHrvSsvLsQFP0wZDOqBGx7Nqyv9BaNw01yhjTAYhcnW3ltYU00CS6Ke48QYI2zUc1g8NudGsjQeI5Z9Q7mCnDn2SNcfGtvVIQ8UbC+O7nwsAFE4pjv2iwO3EStBsbc3VcHWPekMSGCPpnV/he1BLfbvDsPZ6mLdE5YUBc7z6Elzy/N1608SbINepy5lCL2IjU0IeEvCskQ+ctVIhFjkbh/LGdiX5iOrHlO9sOvAhY0kQs9DLqp NUNJy7Qo cTpBie5X9nL5pnubwY6/bVHmaUwPYTL8EKVSM9RkOMtum0XiqYl4XO6+8eCo6GVCKsZUpy6TP37QJaa+F+deJi+4CLkdu5acD3OA25PZGB1PovzlgnU/xq2OiILi98R/JHCi29iFM5srjpeUlSeChRERpgSrm+MFJhsX9vn5f8Z/5pFDjddad1UGazKM7VWqCvRmcQW9BRMaVPRKpc6JhNRK39P24VOW8euEwUssj6eOKZl0fHXJUV9NYQwQqlO/1WwIDBV06qEGpKSmlUNbyOmF4O8sqKGXY1lqdQfz2f7SXKe0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peifeng Li In the case of insufficient memory, threads will be in direct_reclaim to reclaim memory, direct_reclaim will call shrink_slab to run sequentially each shrinker callback. If there is a lock-contention in the shrinker callback,such as spinlock,mutex_lock and so on, threads may be likely to be stuck in direct_reclaim for a long time, even if the memfree has reached the high watermarks of the zone, resulting in poor performance of threads. Example 1: shrinker callback may wait for spinlock static unsigned long mb_cache_shrink(struct mb_cache *cache, unsigned long nr_to_scan) { struct mb_cache_entry *entry; unsigned long shrunk = 0; spin_lock(&cache->c_list_lock); while (nr_to_scan-- && !list_empty(&cache->c_list)) { entry = list_first_entry(&cache->c_list, struct mb_cache_entry, e_list); if (test_bit(MBE_REFERENCED_B, &entry->e_flags) || atomic_cmpxchg(&entry->e_refcnt, 1, 0) != 1) { clear_bit(MBE_REFERENCED_B, &entry->e_flags); list_move_tail(&entry->e_list, &cache->c_list); continue; } list_del_init(&entry->e_list); cache->c_entry_count--; spin_unlock(&cache->c_list_lock); __mb_cache_entry_free(cache, entry); shrunk++; cond_resched(); spin_lock(&cache->c_list_lock); } spin_unlock(&cache->c_list_lock); return shrunk; } Example 2: shrinker callback may wait for mutex lock static unsigned long kbase_mem_evictable_reclaim_scan_objects(struct shrinker *s, struct shrink_control *sc) { struct kbase_context *kctx; struct kbase_mem_phy_alloc *alloc; struct kbase_mem_phy_alloc *tmp; unsigned long freed = 0; kctx = container_of(s, struct kbase_context, reclaim); // MTK add to prevent false alarm lockdep_off(); mutex_lock(&kctx->jit_evict_lock); list_for_each_entry_safe(alloc, tmp, &kctx->evict_list, evict_node) { int err; err = kbase_mem_shrink_gpu_mapping(kctx, alloc->reg, 0, alloc->nents); if (err != 0) { freed = -1; goto out_unlock; } alloc->evicted = alloc->nents; kbase_free_phy_pages_helper(alloc, alloc->evicted); freed += alloc->evicted; list_del_init(&alloc->evict_node); kbase_jit_backing_lost(alloc->reg); if (freed > sc->nr_to_scan) break; } out_unlock: mutex_unlock(&kctx->jit_evict_lock); // MTK add to prevent false alarm lockdep_on(); return freed; } In mobile-phone,threads are likely to be stuck in shrinker callback during direct_reclaim, with example like the following: <...>-2806 [004] ..... 866458.339840: mm_shrink_slab_start: dynamic_mem_shrink_scan+0x0/0xb8 ... priority 2 <...>-2806 [004] ..... 866459.339933: mm_shrink_slab_end: dynamic_mem_shrink_scan+0x0/0xb8 ... For the above reason, the patch introduces SHRINKER_NO_DIRECT_RECLAIM that allows driver to set shrinker callback not to be called in direct_reclaim unless sc->priority is 0. The reason why sc->priority=0 allows shrinker callback to be called in direct_reclaim is for two reasons: 1.Always call all shrinker callback in drop_slab that priority is 0. 2.sc->priority is 0 during direct_reclaim, allow direct_reclaim to call shrinker callback, to reclaim memory timely. Note: 1.Register_shrinker_prepared() default not to set SHRINKER_NO_DIRECT_RECLAIM, to maintain the current behavior of the code. 2.Logic of kswapd and drop_slab to call shrinker callback isn't affected. Signed-off-by: Peifeng Li --- include/linux/shrinker.h | 5 +++++ mm/shrinker.c | 36 ++++++++++++++++++++++++++++++++++-- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 1a00be90d93a..2d5a8b3a720b 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -130,6 +130,11 @@ struct shrinker { * non-MEMCG_AWARE shrinker should not have this flag set. */ #define SHRINKER_NONSLAB BIT(4) +/* + * Can shrinker callback be called in direct_relcaim unless + * sc->priority is 0? + */ +#define SHRINKER_NO_DIRECT_RECLAIM BIT(5) __printf(2, 3) struct shrinker *shrinker_alloc(unsigned int flags, const char *fmt, ...); diff --git a/mm/shrinker.c b/mm/shrinker.c index dc5d2a6fcfc4..3ac50da72494 100644 --- a/mm/shrinker.c +++ b/mm/shrinker.c @@ -544,7 +544,23 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (!memcg_kmem_online() && !(shrinker->flags & SHRINKER_NONSLAB)) continue; - + /* + * SHRINKER_NO_DIRECT_RECLAIM, mean that shrinker callback + * should not be called in direct_reclaim unless priority + * is 0. + */ + if ((shrinker->flags & SHRINKER_NO_DIRECT_RECLAIM) && + !current_is_kswapd()) { + /* + * 1.Always call shrinker callback in drop_slab that + * priority is 0. + * 2.sc->priority is 0 during direct_reclaim, allow + * direct_reclaim to call shrinker callback, to reclaim + * memory timely. + */ + if (priority) + continue; + } ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { clear_bit(offset, unit->map); @@ -658,7 +674,23 @@ unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, continue; rcu_read_unlock(); - + /* + * SHRINKER_NO_DIRECT_RECLAIM, mean that shrinker callback + * should not be called in direct_reclaim unless priority + * is 0. + */ + if ((shrinker->flags & SHRINKER_NO_DIRECT_RECLAIM) && + !current_is_kswapd()) { + /* + * 1.Always call shrinker callback in drop_slab that + * priority is 0. + * 2.sc->priority is 0 during direct_reclaim, allow + * direct_reclaim to call shrinker callback, to reclaim + * memory timely. + */ + if (priority) + continue; + } ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) ret = 0; -- 2.34.1