From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF4AAC4332F for ; Mon, 17 Oct 2022 18:20:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B6AF6B0074; Mon, 17 Oct 2022 14:20:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 767F06B0075; Mon, 17 Oct 2022 14:20:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60A306B0078; Mon, 17 Oct 2022 14:20:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 531F06B0074 for ; Mon, 17 Oct 2022 14:20:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 207F4120613 for ; Mon, 17 Oct 2022 18:20:10 +0000 (UTC) X-FDA: 80031255780.05.90CAC14 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf23.hostedemail.com (Postfix) with ESMTP id 8CE6E140038 for ; Mon, 17 Oct 2022 18:20:08 +0000 (UTC) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29HH2Svr024127; Mon, 17 Oct 2022 11:19:56 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=s2048-2021-q4; bh=t1yh+nb7pc3Eg4jKVqOXdw6E045DwaAOlvS3mZ97QV4=; b=A3TG48gqK+sLpq/z5nD/+lfr7pGlvOb/kMpUCWCoG+wPmtsGlhHltCLkQxv3pVS4jhsY bWk/cB4qc8mTAEMR7tSbX8qhywACiwZs9J3SbavgaStUAHyiLS06yw7IqBVvgS8OqijX oy6/an1pnHZzrhe39xWQum9H6eZHYXwsaW9lT7PD1/aNMBeo2dlIjHlQ95qBgquXHJzF vGjMTdr7gGDbq2dN7wdV2q1V6fzv9I/GXg6QMiwC9kuC6iADjl/XgtV8Dh1M1pnUx/tg MDpi0wIdKF8W3f5ULMhTIgHj2FSKOTxX3pOmrAvQjx8tTcX8gJDtFsySWH9J+zgBodk+ gA== Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3k7rwmb3c3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 17 Oct 2022 11:19:56 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TsN+uZpPmfmmueAWowA4z6znq6uedQ/oQawm+06ZuMZOOoLqX6xea0S0fGx48CI2e4i2XNtAa356dDa230UWIYPpWqXny2vabm9Iy5+QWwv44gxWHSD0Zsz+yCmzTh9wyx2ZxaW21DyjUPohOzd+CXNcIPi5OZyJUzhUxTaAxP7tIfDJoD041wFh+8wUTL+SJwLufZyHYWgD61R43z6QdiD7wyHc8+puiMLCyc4m/4RcUWbUNkSshE/m/2rtOyDl8Hw/3kYVCX199Epv0uv23hNIyQ6e0oLQXm/SJ/81M9+GkcJxQy8LlMOsHmZPRgnF+10/iC44T93H+NKmYGMfgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=t1yh+nb7pc3Eg4jKVqOXdw6E045DwaAOlvS3mZ97QV4=; b=oXGY+EldMpuUPEueo5Ye6reRNh4FKZdQIMcovYG8qb/3ZESVdeKUKOomncRZF1r7hvBjfdB1sY6uQdcVW7s1zGtZSRVCT3A0u4eIkI+QSMCVTyWub4fzdvaxCB8KYbNPHuKmYT0zDD82B4FqEGb0k7qno5p9vdijVWmko9VUujNMONviFSFS6jbj6veYWxN2aLmgyQunObXDLRBPVWj/ApaG0JZ9hWkD8fnTaVTkfUWj5sTiFJ7DgeC275DJtA183nnMgtZVykMP1O7cr/YwxuXDYZJT8H5S3qXqmRVwyPApDYuLgU7iZ8Ka4Wk2/AfYtIW7ciyKMAJ34SpdsqdVwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=meta.com; dmarc=pass action=none header.from=meta.com; dkim=pass header.d=meta.com; arc=none Received: from CY4PR15MB1734.namprd15.prod.outlook.com (2603:10b6:910:19::14) by BN8PR15MB2705.namprd15.prod.outlook.com (2603:10b6:408:c4::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.32; Mon, 17 Oct 2022 18:19:43 +0000 Received: from CY4PR15MB1734.namprd15.prod.outlook.com ([fe80::fc7c:15c:9a61:e445]) by CY4PR15MB1734.namprd15.prod.outlook.com ([fe80::fc7c:15c:9a61:e445%12]) with mapi id 15.20.5723.033; Mon, 17 Oct 2022 18:19:42 +0000 From: "Alex Zhu (Kernel)" To: Johannes Weiner CC: "linux-mm@kvack.org" , Kernel Team , "willy@infradead.org" , "riel@surriel.com" Subject: Re: [PATCH v3 3/3] mm: THP low utilization shrinker Thread-Topic: [PATCH v3 3/3] mm: THP low utilization shrinker Thread-Index: AQHY3ov4xE/oymI8I0CpAa1tUN4wBK4Mh70AgAAEmYCABmCfAA== Date: Mon, 17 Oct 2022 18:19:42 +0000 Message-ID: <237C93B3-7E90-44CE-85F9-BC366D81D66F@fb.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CY4PR15MB1734:EE_|BN8PR15MB2705:EE_ x-ms-office365-filtering-correlation-id: 9f470d07-a6fa-45a9-2327-08dab06c29b2 x-fb-source: Internal x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Ro31Yb85wEmrXR4DcThEMhcOycg1evi+JlTtJCNZzy/iDABR19CVTNpR4JmkdIOny7bObMTUd7lTaInjny7IipKt/7H9ASVT7a6xg4H8t7tDP8Zhpz5lnQRCylKrPpAIbnI/9LUSHtScAHN3aL1gtfYfpxokY1FyZ40YidwIdWdDDZJAXvOhxPRI5eGlz/i4EAFaT1pITjEMj/vh+N/RRVPknlHL+EbfpnFZFr01sOuhZ5fvfkD+dtYZMt470jrsgkCwuN2zzewPB8Fvc/zFCScdUPWB78GR6BzbgXj65Hb/3gxGSpjLesVF4Zr0LZ3xilOVwW+cK/GHGkwmOQhMKP5x+NMsbZ/ODOj3DtgwqQ9ud29zCug7AuqWPj4Mhx6eDo4xI4XPgYAXhq7r180bbXQXYY370nespKkvG07FATRNKq2sasKOnB3Ue+6wtTvbYHjfxbrbNaD09hl0E5trKsCCu1W3heSSfpQAGgc4sI/XyfCLBClgfUp+9vBcnPzHTnmrLII3tjh1L93YfoAwITlOBOhA0C6J5XMxyaawbtSzYGhP5PdFAVyfePcoVk2KzBihXZOel70UzKEJTe8p77/R3Y/YSqFMYPO2LRIGxCRWCOsjZNPovrbVv8kpKpUO62WyG3UjVa5zRUNXevi/V/VHjxOGrTVNOYPMeFKNRxsYA4iqvXiATe8nNSt416y42nmdsm0C+QClFdC4fMgv9X71ZVvLv+pDCSqhyiVMtRB8vJTCWAwQBREl8e2+3xNEFATqY5WJGpnGUBJ+Qvu4ng== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY4PR15MB1734.namprd15.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(376002)(366004)(346002)(136003)(39860400002)(396003)(451199015)(36756003)(186003)(2906002)(6486002)(64756008)(66446008)(66476007)(66556008)(66946007)(76116006)(4326008)(71200400001)(478600001)(316002)(54906003)(6916009)(38070700005)(91956017)(41300700001)(8936002)(6506007)(6512007)(5660300002)(8676002)(9686003)(53546011)(38100700002)(86362001)(33656002)(122000001)(83380400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?16lSB5Zx3YO7BOKWy/esZcBOMjox+9aPNv5moqQuqLA6BHtd7qX0BCLw/Ezm?= =?us-ascii?Q?ky1XCdAI4LRYKV8P3N6ySqhYIGk/2TN4DTzmG1KWgH3y1GUDzffuvp7y/Pa+?= =?us-ascii?Q?1jcd7gSKwM/Yw2/fZjqot9uL03cC5bSKgUQV7Gb24IgyfqO5aUcP/HEzsYDQ?= =?us-ascii?Q?zn7jSkK0npAxvncRGBU+EUQqRsvPVD/KnKkCT3VzHh8XqzBVowfUn1InQKmv?= =?us-ascii?Q?YnorZPMLo9Kr3ohIIstNTc7WeFC3vmp1inN6Pemq1N8xAIm3ZXvYGO6Pd3u8?= =?us-ascii?Q?qkvvCsgKkUv2IgMwIKeJhGKx/3c8yWbYpeE0iAEgmK/1hynEvymZnsGx3p6O?= =?us-ascii?Q?DV61kIdxp/rC1Uh04sbBhcuXQs7SqRvnnuEF6l4Q6aJs6t2gsNMXWuwIdw9W?= =?us-ascii?Q?j0+2RzHaTGL1LvhdTAIdXS5Ltc1xnEyj174cwsWN8XufMh8dxAjbHgB8lbAy?= =?us-ascii?Q?Uqs2baV6VT6s5I5z4MRN6U9PRr4JOXEU6nbHtSpSc6nTdpukgSdnUe2i2XHK?= =?us-ascii?Q?N+6Kdn0bGJJEf2AJt691iQq+HP/CyBn9J1f/Kb/ard21oSlGhLFw9tPddawH?= =?us-ascii?Q?Ho1x+ddmuq6cBRaqAzoif7BlQxaPc4r6vO+8S0SpL24LvsOJ7fKMdidFi7WS?= =?us-ascii?Q?gFAE+FJJ7OwTJ37HOVE2eGKsu6ug/jSPbbEHr/LaR2gnWeqUuuLQJimLebAI?= =?us-ascii?Q?2CByWCXPha1RonpgL9wi14wR9ejMWZ4ry1RMUv4xMKZEiEObUZPVmVagtdCm?= =?us-ascii?Q?JizwxL26eH1M8Q28heAIZ32AZN0gl8NZubZCpkKKTFuCwcksYqy9AOl2EPa0?= =?us-ascii?Q?ydimtnvFvL4wB0lorwstzR/xsSN6trMbglz87xE6Wo8FoMT1d5fZu+E8GY4u?= =?us-ascii?Q?CqzCa6KMpp8u3XjGtR3j6mGGSqVn84CyAj6GgW0q7U0ZI6QfgYbgnxo0pVDZ?= =?us-ascii?Q?ixkU4cjBGTQcjLSvQhNxcIELbb7z/sVWHvuMp2HuB9qePORcOMfe96XeJimy?= =?us-ascii?Q?C62zlQ+Eo3aeT2Ym3chUiKbNvl5cjUg+OfmCUCNZzDzW527EFFn0QGynQo6K?= =?us-ascii?Q?njV9+VimjgXZICyRa/OMipF2YVog5rM87UgPyJqBvE5vhifwujmBvm8QelzO?= =?us-ascii?Q?HUl0qLwi4qUtLZvwMBDhHrcpjP3kHfnL1f86Bv8LROXfZVMnsGw6JCkJNohc?= =?us-ascii?Q?j7+ewohOrEFUUPdJ8B0EzmTCp/2hYVW1yNHuPxh0tn7V02Sd476KexjvPwN8?= =?us-ascii?Q?bFUqMnztY0AOpvuMvb0ARxIxD7naYkDVidHOIL7B2TmLocnxZZFKUWW1x29D?= =?us-ascii?Q?kNp2vm365a+d6xueGVLAvE6NfsxOCiNCCUrwWKSnn4kOcyootBy2d8WPE6VF?= =?us-ascii?Q?v3N8gb2fdcSe23Ve9GnHj/f62nROQ0exVSk1ogssi5EVc7AEVlcaQ2bwv0Rz?= =?us-ascii?Q?kBTbhkTsbGkxoYl4NJ33sofvfA1AYUKruba4YCGkf8Y43nw68oz55FIhuqhc?= =?us-ascii?Q?0bnOwAdoMOnfip/JcLrIP7dgaJ6ANwEZa8tsZ77E7jEInhiukheL54yhWEvT?= =?us-ascii?Q?8YzGOzxxu065ZSlMeO//AUxl3XQVAz8Cqgxvaoib3Gtzkb9W3D6qyj9pGB2l?= =?us-ascii?Q?xAVteSUmiJOuq4OqwXCkbhc=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: meta.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CY4PR15MB1734.namprd15.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9f470d07-a6fa-45a9-2327-08dab06c29b2 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Oct 2022 18:19:42.8374 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: iQu+4dq4Va8Pf3wdz777CnWtONvULn0bc8htXrJ6Trjaq3w39lpyCDozSu2CebhaI3w2HLwDbVjpFbvwdiJwew== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR15MB2705 X-Proofpoint-GUID: -UOMhb4b6XxGB04VvP9UCH5i8FPJAI-H X-Proofpoint-ORIG-GUID: -UOMhb4b6XxGB04VvP9UCH5i8FPJAI-H X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-17_13,2022-10-17_02,2022-06-22_01 ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666030808; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t1yh+nb7pc3Eg4jKVqOXdw6E045DwaAOlvS3mZ97QV4=; b=hbkzwVKvKIX4jVvALdzqqJa5LhyCr1NO+uu+hAm4MQxXbrUaw7ISYYGujZoBn6DvekivQ3 3YaNghaNhQW+HnWO3g/jcPB+yDAPFmLHQ5mgufKc0vM7UKjIgkDUpA1+cIqLn68/tJLL0v Hh2I3J0Sr+uOtYChbMHHBws7sJX0vN4= ARC-Authentication-Results: i=2; imf23.hostedemail.com; dkim=pass header.d=meta.com header.s=s2048-2021-q4 header.b=A3TG48gq; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf23.hostedemail.com: domain of "prvs=128949a3e2=alexlzhu@meta.com" designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=128949a3e2=alexlzhu@meta.com"; dmarc=pass (policy=reject) header.from=meta.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1666030808; a=rsa-sha256; cv=pass; b=FIDN3gMDlHuIFBjBQoquqCnw57UfTAWxYPkj7IT/4iwMQfBG97epOifZ/MzjpQ63YcDGPZ 1+GFGCrz64NbEfEGrL7REg00ave4bKYa0Voh2q+VrO7mRPp9IAB5eGyKl/7kqLglr8lRmG sU5ziS7KMKBnWb+OJ9ZlOyBv7oXlTAg= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=meta.com header.s=s2048-2021-q4 header.b=A3TG48gq; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf23.hostedemail.com: domain of "prvs=128949a3e2=alexlzhu@meta.com" designates 67.231.145.42 as permitted sender) smtp.mailfrom="prvs=128949a3e2=alexlzhu@meta.com"; dmarc=pass (policy=reject) header.from=meta.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8CE6E140038 X-Stat-Signature: 86gj3bc4abuw4qc145ztgnodaogeuwu3 X-Rspam-User: X-HE-Tag: 1666030808-20656 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > On Oct 13, 2022, at 9:56 AM, Johannes Weiner wrote: >=20 > Oof, fatfingered send vs postpone. Here is the rest ;) >=20 > On Thu, Oct 13, 2022 at 12:39:53PM -0400, Johannes Weiner wrote: >> On Wed, Oct 12, 2022 at 03:51:47PM -0700, alexlzhu@fb.com wrote: >>> + >>> + if (get_page_unless_zero(head)) { >>> + if (!trylock_page(head)) { >>> + put_page(head); >>> + return LRU_SKIP; >>> + } >>=20 >> The trylock could use a comment: >=20 > /* Inverse lock order from add_underutilized_thp() */ >=20 >>> + list_lru_isolate(lru, item); >>> + spin_unlock_irq(lock); >>> + num_utilized_pages =3D thp_number_utilized_pages(head); >>> + bucket =3D thp_utilization_bucket(num_utilized_pages); >>> + if (bucket < THP_UTIL_BUCKET_NR - 1) { >>> + split_huge_page(head); >>> + spin_lock_irq(lock); >=20 > The re-lock also needs to be unconditional, or you'll get a double > unlock in the not-split case. >=20 >>> @@ -2737,6 +2800,7 @@ int split_huge_page_to_list(struct page *page, st= ruct list_head *list) >>> struct folio *folio =3D page_folio(page); >>> struct deferred_split *ds_queue =3D get_deferred_split_queue(&folio->p= age); >>> XA_STATE(xas, &folio->mapping->i_pages, folio->index); >>> + struct list_head *underutilized_thp_list =3D page_underutilized_thp_l= ist(&folio->page); >>> struct anon_vma *anon_vma =3D NULL; >>> struct address_space *mapping =3D NULL; >>> int extra_pins, ret; >>> @@ -2844,6 +2908,9 @@ int split_huge_page_to_list(struct page *page, st= ruct list_head *list) >>> list_del(page_deferred_list(&folio->page)); >>> } >>> spin_unlock(&ds_queue->split_queue_lock); >=20 > A brief comment would be useful: >=20 > /* Frozen refs lock out additions, test can be lockless */ >=20 >>> + if (!list_empty(underutilized_thp_list)) >>> + list_lru_del_page(&huge_low_util_page_lru, &folio->page, >>> + underutilized_thp_list); >>> if (mapping) { >>> int nr =3D folio_nr_pages(folio); >>>=20 >>> @@ -2886,6 +2953,7 @@ int split_huge_page_to_list(struct page *page, st= ruct list_head *list) >>> void free_transhuge_page(struct page *page) >>> { >>> struct deferred_split *ds_queue =3D get_deferred_split_queue(page); >>> + struct list_head *underutilized_thp_list =3D page_underutilized_thp_l= ist(page); >>> unsigned long flags; >>>=20 >>> spin_lock_irqsave(&ds_queue->split_queue_lock, flags); >>> @@ -2894,6 +2962,12 @@ void free_transhuge_page(struct page *page) >>> list_del(page_deferred_list(page)); >>> } >>> spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); >=20 > Here as well: >=20 > /* A dead page cannot be re-added, test can be lockless */ >=20 >>> + if (!list_empty(underutilized_thp_list)) >>> + list_lru_del_page(&huge_low_util_page_lru, page, underutilized_thp_l= ist); >>> + >>> + if (PageLRU(page)) >>> + __folio_clear_lru_flags(page_folio(page)); >>> + >>> free_compound_page(page); >>> } >>>=20 >>> @@ -2934,6 +3008,39 @@ void deferred_split_huge_page(struct page *page) >>> spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); >>> } >>>=20 >>> +void add_underutilized_thp(struct page *page) >>> +{ >>> + VM_BUG_ON_PAGE(!PageTransHuge(page), page); >>> + >>> + if (PageSwapCache(page)) >>> + return; >=20 > Why is that? >=20 >>> + >>> + /* >>> + * Need to take a reference on the page to prevent the page from gett= ing free'd from >>> + * under us while we are adding the THP to the shrinker. >>> + */ >>> + if (!get_page_unless_zero(page)) >>> + return; >>> + >>> + if (!is_anon_transparent_hugepage(page)) >>> + goto out_put; >>> + >>> + if (is_huge_zero_page(page)) >>> + goto out_put; >>> + >=20 > And here: >=20 > /* Stabilize page->memcg to allocate and add to the same list */ >=20 >>> + lock_page(page); >>> + >>> + if (memcg_list_lru_alloc(page_memcg(page), &huge_low_util_page_lru, G= FP_KERNEL)) >>> + goto out_unlock; >=20 > The testbot pointed this out already, but this allocation call needs > an #ifdef CONFIG_MEMCG_KMEM guard. Sounds good. I made the changes on Friday. Testing them out today. Will sen= d out v4 soon.=20