From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BF81C636CC for ; Tue, 7 Feb 2023 12:40:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D9736B00C2; Tue, 7 Feb 2023 07:40:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 289326B00C3; Tue, 7 Feb 2023 07:40:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 103DA6B00C4; Tue, 7 Feb 2023 07:40:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F276B6B00C2 for ; Tue, 7 Feb 2023 07:40:47 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B2485A01A7 for ; Tue, 7 Feb 2023 12:40:47 +0000 (UTC) X-FDA: 80440454934.11.D386A4E Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2046.outbound.protection.outlook.com [40.107.92.46]) by imf14.hostedemail.com (Postfix) with ESMTP id D34DF10000E for ; Tue, 7 Feb 2023 12:40:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=jwqlXbDK; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf14.hostedemail.com: domain of jgg@nvidia.com designates 40.107.92.46 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675773644; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7OLqe7VMmhmFUDnyrT1EKZWE0OVvHQc3C3TrmWacRec=; b=Q4RkL+V64s8CuAVoUN0+xyWESSZ3GlDg8RSs5a2DsKXyKdHxGs9WjoNg3OW2o8buryOWtK qlXb7Dzc4ztWC0Lt6P8Th0XPi/canNkFdJMFl8L58LmQ8+k53FPHhaRtKSMltQFobbm6Zl yAeiww6RrUEa+RRd+xkL7z95rS80e1Q= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=jwqlXbDK; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf14.hostedemail.com: domain of jgg@nvidia.com designates 40.107.92.46 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1675773644; a=rsa-sha256; cv=pass; b=m+FAlPdkFSwa3CMDXzzIpwwUiic9RdTuPIKUkn5YrByZQw9MpGb1gj49i/EBv6oywKb06E y1+wqtEPck+SB2A1+dixwdwG73O9G1YmpjaqzqPTvOAcp1ReW5VFM8Kl5wYSImQxL0IKVx HIgSKPS/O7mNz7r3tIztzGDL5N7Ut5E= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JBsaXxCgucfQjVD2/GPDW1hT/XeJYkx6/QuWbgAuEYJ6AcmsFTOgOe3jL3HUX8nfMdb6qKTxpKten0s2SQA8Ba8fbkfUDbx7SOCg5xu2IeWLlvILNWIUfKVP3Hbusp3IaUCrEbJum678IqhrgWeYtmaAABzaSynHIjtSlz3+Rv9wmDx6ws6jKETBdc4viicoO9PbE73RJogVZJZmLoL1Eg65Y7BSn5yhUYrT9eYCdCBw36z9CbTrX/vAHxrlqIqfodEfktXEygZFll6z4i3WxAZ4r40fFkg9Gk9ma24HVKzUrb8dKfqcXZpT+orUoc3F6esOurv2FgtFK8DEgFzvBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7OLqe7VMmhmFUDnyrT1EKZWE0OVvHQc3C3TrmWacRec=; b=QiJQXEJtmCMj19htE7W+poGrS+IwVcGGYrkCrQEtm5qdpC18zXFGl7gj1LqmWyZW1bNmGCkC0bLObzs+2B3boY6Hgm0qF2r38K+xR9/p//0xxs5SrS9IdpSvYwHBbhNh4SF5wTWLt+4z1S1hUehoar9Id68ni+L14/kwTKHm006n/ndhmaXoQfomYAAHWvlIBmZi9/5uWZ/mLHplvWIWQTJ2ZK7UqVCLy+1OjniUszBGaXc53eEW4E0VcyDbOMVWwFxY7LVZr76PuEn64n0qG3XcY5Nf5RCtQOzYKTulgbduYTqrtvrjvFcV+TgPdw5JBqlpEiAHdgQcBuTdvSQlyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7OLqe7VMmhmFUDnyrT1EKZWE0OVvHQc3C3TrmWacRec=; b=jwqlXbDKF1sP9dyW5h+KIZbZ/AI09pk4yHGhTv8zv09bK9VEqOKXnKqhc7luWlJetzLl+JPQyjemVdrZH2gf88dfSwyl/ThMzk4RkV5iCmmDu5zI8UNESlsSP1hkJmMi/a6RbnFk1FAAZCCoFkuD5ebPcIN8Ao1AHYmFdbXyjoeFYnEY0hLV+hPcPie/ET9dC64oKthmpieXj8LLecpwkTaQjjrak6tyQQEDJcie+l2OUTEAnKJ5bmyo2q1Vq0LZ71mSTgqrUShdjkaymElm/26UeWPaLfaYQZseQWSl3QecFkaiulBvBCR1lKl2C4RodfeieJgqSwHJv/v0CZpA5Q== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SJ2PR12MB7799.namprd12.prod.outlook.com (2603:10b6:a03:4c2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.36; Tue, 7 Feb 2023 12:40:40 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee%4]) with mapi id 15.20.6064.031; Tue, 7 Feb 2023 12:40:40 +0000 Date: Tue, 7 Feb 2023 08:40:39 -0400 From: Jason Gunthorpe To: Claudio Imbrenda Cc: David Hildenbrand , Alistair Popple , David Howells , Christoph Hellwig , John Hubbard , linux-mm@kvack.org, "Mike Rapoport (IBM)" , Christian Borntraeger , Janosch Frank Subject: Re: [PATCH v2 13/13] mm/gup: move private gup FOLL_ flags to internal.h Message-ID: References: <20230126154148.2442e4cd@p-imbrenda> <7388b6a6-85f5-2e61-e3dc-54de531308d0@redhat.com> <20230126163902.16898798@p-imbrenda> <20230130192104.41c5b426@p-imbrenda> <20230207123112.6900e10b@p-imbrenda> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230207123112.6900e10b@p-imbrenda> X-ClientProxiedBy: BL1PR13CA0097.namprd13.prod.outlook.com (2603:10b6:208:2b9::12) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SJ2PR12MB7799:EE_ X-MS-Office365-Filtering-Correlation-Id: 679dddc8-f7d2-4109-c674-08db09088532 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2SaR+OXpQyoB6ukGBLYluESmR3l2TeH6Vlf4mSvwCURPDik2Ma3QAGLo5tlJdQeuZ91k+GJM40Obcu7x4jf+/lyxrfIJE4bOPOk05zOpBUKONP7RiAAKzHkhqpOwrmBfNYm52YNZwmiR5WS6jNJiibrcAsDwCMd24Pd9fij+WgdTmhKu5zGfZVIKxDREkJn76x9e7ARk7fSCUjpsT7bD6ZpVoPoGIQc/E5Td0oFRIZUangWHHzqnjE0uvJFBZmSV770DWHdS2dpCZHFCaP0whh0SgfD3dz2KY+543escMvDX9XyCQC6AUVOExBEk98a+NCdL+Z+KJVsQpTJpeV0M6i3MsN8bkriO3HhanJWjgQzMOBsd2y7cwvzZdNcCMfHCab8M6Z3LUmbKcLp35SFO6Mki+SY/ki+LgaYEY7sjshJQ2aMrIU6Ve16Rd++5olifvDB4peJAkBksBjQy9EFaug4Iz8/St3Cg7jHBM3d9lLI7S+cCid7QJpnYzKX+0WlUZKLYX5GD4jNI6G76QYvfNz+da5Z3XVH1hlT9O814PRzyHaUtxISfOMFLORaOxYhPPPAUOoLMy/3eS/ELtYg6F+sSEhdDm1w1RGY6CTOCrraZrW0Opy+YbQ72cuV+LFO7bcwpoptT66XdHkE+z63J4Q== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(136003)(39860400002)(376002)(396003)(346002)(451199018)(2616005)(186003)(478600001)(6486002)(36756003)(86362001)(83380400001)(38100700002)(8936002)(2906002)(316002)(54906003)(41300700001)(6506007)(6512007)(26005)(66556008)(6916009)(5660300002)(66476007)(4326008)(8676002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?awpYYqS0UNMc0HWbQGzNMXCatkQjm0W5ikNeGaeaJAsRnHo3pK9dvTRAvD6+?= =?us-ascii?Q?nbwaiwU8Kv/lJSko0O35AuhDLbep4OSNiog49Fiu51cekOv+JE/Fz5fwQ/i2?= =?us-ascii?Q?DKj4m0+VIDfj1wng054k3XZN3KdeK8SwYj+ybNQwo+yYjq5gRtXOB5r1EKaL?= =?us-ascii?Q?SI3Q/38SzPUj/FAK9lG9vFW9awoav/2H2TQmP/Xr0tXZd257jS7GKsUratJ7?= =?us-ascii?Q?d6fO92wH8QKvjrBvHYGmTdkQESR/r5fwHCtGCbqFbdV16Sm9Dzf4zoTdcChr?= =?us-ascii?Q?8ebg4WEp05MayTc5tMhwOYu+0IDNYFo0V9EscCaxugv5lb826HmLbSh7Yjjg?= =?us-ascii?Q?JsQDZLkBY18jECk+KeUHcYGYjhs+QIbIQlsehePh3fT785UGYYEPfwCVRdUp?= =?us-ascii?Q?6Ajh1wAX5n0ghyvNWth9Rd/h6Fs72XRFtPuzEIupaaEE7tGD9eSdtRY/Ro5q?= =?us-ascii?Q?RiXaQwtFReYeWW92L2WjNL/NRhJbBtfr7PJ6WQELpacYi1alJXPqKdyIXZEi?= =?us-ascii?Q?EAYiHpYC9o0UJVn20JBXuuDkIrDI0l+hDH5cb9Th39o0eOcx2K4dFAZo6xyR?= =?us-ascii?Q?xNdO/C7WFsLMiKy17Jyz+XtY9UWA1byYLlu+ZhtcWSLew7TLJUUonL+uh6eV?= =?us-ascii?Q?XObhix0mjjYSw+woyjPKOYSMF97p+BYhbuF6o2B817eRYl2hSR3f+1EQELlF?= =?us-ascii?Q?Cuq+s9DTE6v8khReBfNBlyfe0D8/dm3YprDhNdP/XEbz0oxPhXwVutOqV7iX?= =?us-ascii?Q?TUXWNBFubMV38KmfH/AgnBC0TXLzbDiWfMIMKrfqM8t3LCRRo9BiJxHoY/80?= =?us-ascii?Q?9VxZkf6UGQ+tOAruXrtdUqwAKqBI/54etHAqlIziq3dA1xVx7uWklhZedEVJ?= =?us-ascii?Q?5HJU1vGk1n28UQSFRDZlBkTZ1zQyFwDVAJHHENpVWej/N8X05u+m6a120cv+?= =?us-ascii?Q?bG9f1i7NucEFbb23hldOnda+x8Z+AoAtK4ZxcXXcUMRjOKwO8yBQ3etoCtrN?= =?us-ascii?Q?hOoDzMF3KxAN/UHFKQpngpzvdjQHX9jWxzqdNpfWzfIzA4ihH2nL7N2YDlsA?= =?us-ascii?Q?cIAyraOoEqAv2IGgev7eB28Wk95Pj7YCefluM60ouOImUdw91+0FTU/+Prne?= =?us-ascii?Q?8R5/jUqdKrkcJYkgrKZxvhtxVwKGvnOled3u8P61fyikqQ/OpYCWWwMcFSeL?= =?us-ascii?Q?aIQXOQvkLDWHw0O634VacLtkjHyd4MZ4ZhU+qLzTX+5TldK3a4AUkwazR9rE?= =?us-ascii?Q?nZxXQN631jpgwgiT8c2sBON6fQpaJLVkE3qIVgjklRtqvvdEV/P43XKj+jNa?= =?us-ascii?Q?B7FG7UVx79MfeRrM4hQyqs3DvqTaZYKjjKvCZf7XRmb8Aa2sICixUT4P3ItS?= =?us-ascii?Q?3C/JUpA7k+9PX2tq8qE+zSkx4uobIQNl86DojLamujH3tLCVhoVJWrAs67v/?= =?us-ascii?Q?+jc2zWsJf4qfPvF5VhIVwalCszfssupCppWhADBnNxO6vquqg2Xkjh91OmJJ?= =?us-ascii?Q?RyrY/wLNU5BOkL2Hw0pz7RZe8sBZykZGETZvymk9OBZ35o5XVRfKUt9yhfj+?= =?us-ascii?Q?3PBmENYb7cGxn9z12y143C221QZG+VheESy/VmdH?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 679dddc8-f7d2-4109-c674-08db09088532 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2023 12:40:40.3861 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BVJUSNIiW+QYjZC4Nik/oRrugJ+TpNNRR8j9uRMvzbESpH0YlLKklg2ogD4+Qh+w X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7799 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D34DF10000E X-Rspam-User: X-Stat-Signature: teqi4yw7zeedfztinuwcdc5bhge7d7yc X-HE-Tag: 1675773644-404955 X-HE-Meta: U2FsdGVkX19nIepOWeY4cCQnPEK9Rd7crJP7STwhJOtbHttl3GqA5Eh/QDpvy7nWl1b1k4CRi0yvfH97UudM4SwKPlma/VBVfEFHDwTc0YCq6kLBqm+rzmVeSGIqKWiR7g09H8kJujN3CoPdrS8B/l8MlLq7MHoXLN+J61FPZ1iufslVAPbB1AqOK6PQWZy6TMxJx371G0tsF+NeTxcJHJh7+TgahpJldl0LxQ7ly3xxniqyPYloYNGK7acFU0xFTArX3ThsjUMxHFnY601Z1/0TNyOxcRGwRjyhrzVSXRausyCmNiigMdtIBBMjHaDpqKsh9I3bNInacKJ7EqsR7Vv7J9Jx7Ouq7rLWRnPfmVSh/kcUxvbmkwBujyv9it20M5+6auHilyJ5g/USOKalb2AxRTp6v6ca9GRmzrpjSMtKox6aJcirobyKWYz9cKCNEVqopFEe3WbZRH4RpXhMZkpQlyoQ33u3HxncEGYJpEvYXU+YK3z9SjX15cZYk1W/QlMfTvnBiltn0YkTRnQXRCZacob9gwV/ZAmYzV2MgfYoaEsUPuMQOPazNJf6j5nX6OOlh4iqO42G0o42+9dhDQ3cvnPslxuIzlf2UaO3ecrTt/BzSAeay92mTyAnvY/aUgpTetWcSvSrexSNiLpw5B01NZ2ZGjk9M5503wLRaxfYsF0l1/JmRoouGVkEFXU5/Rt1YDs1Amozqt6+ZRA0sOcM/I2LwtaccLxbN6Wbh+GL+FXHXeXVaB1ZCnVLzTYgtmqEAdmmSXLoA/ROyTUe4sXdGRlBQTpoFFVbLR92bBebHhIdXxlHPWnxctBKgIbj/uqxLfO+GjW3FL10cyJoARq3UHg5jk8hgw8gzkCznthlp5fXDLyXgJj3SYw+ksSimdRnX6NLgSU3qGPm8mTGXQ+HAG+5OeYLu58GlX5Ht77kBRWJIn2M4xekVrunASEe+CU/9jUij/3exi3g7J8 8YizMLqI limXlIumY/II7Hhv/WxpNzRg4wiHI0twp/s/JDEWfm+w1EH7CY5qeDqrdOJ8CdtuJjglbgHBnS4/wraQPUPp3tXcZwxEuRZzD1McNcQq7GF4zHA9S6i3ANX4gGJo+i2RHjuEEGolHdtWZyQnzzsKCmgrFa0/ZGXqy2oeSFT7UFVLwINSi2G6GJcse25xTdqCSVaUjygExtXuU9I1pvKlbRozrgulL7BCRSP0KFl4scZOHXf2l3wXfiZzmL8qbgc7gu8a0kx7IcMbnHM72j4CkuUohmWj5gA0HIeOCeCa9AI5nc9M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 07, 2023 at 12:31:12PM +0100, Claudio Imbrenda wrote: > On Mon, 30 Jan 2023 14:24:40 -0400 > Jason Gunthorpe wrote: > > > On Mon, Jan 30, 2023 at 07:21:04PM +0100, Claudio Imbrenda wrote: > > > On Thu, 26 Jan 2023 12:35:37 -0400 > > > Jason Gunthorpe wrote: > > > > > > > On Thu, Jan 26, 2023 at 04:39:02PM +0100, Claudio Imbrenda wrote: > > > > > > > > > I can tell you that the original goal of that function is to make sure > > > > > that there are no extra references. in particular, we want to prevent > > > > > I/O of any kind to be ongoing while the page becomes secure. (the I/O > > > > > will fail and, depending on which device it was, the whole system might > > > > > end up in a rather unhappy state) > > > > > > > > Sure, but if there is concurrent IO you just try again right? It > > > > doesn't wait for refs to drop for instance. > > > > > > > > So make the lock_page work the same way: > > > > > > the more I look at this, the less I understand why I wrote the code > > > like that. I do have a comment, though > > > > > > > > > > > diff --git a/arch/s390/kernel/uv.c b/arch/s390/kernel/uv.c > > > > index 9f18a4af9c1319..847ee50b8672c6 100644 > > > > --- a/arch/s390/kernel/uv.c > > > > +++ b/arch/s390/kernel/uv.c > > > > @@ -193,20 +193,11 @@ static int expected_page_refs(struct page *page) > > > > } > > > > > > > > static int make_secure_pte(pte_t *ptep, unsigned long addr, > > > > - struct page *exp_page, struct uv_cb_header *uvcb) > > > > + struct page *page, struct uv_cb_header *uvcb) > > > > { > > > > pte_t entry = READ_ONCE(*ptep); > > > > - struct page *page; > > > > int expected, cc = 0; > > > > > > > > - if (!pte_present(entry)) > > > > - return -ENXIO; > > > > - if (pte_val(entry) & _PAGE_INVALID) > > > > - return -ENXIO; > > > > - > > > > - page = pte_page(entry); > > > > - if (page != exp_page) > > > > - return -ENXIO; > > > > if (PageWriteback(page)) > > > > return -EAGAIN; > > > > expected = expected_page_refs(page); > > > > @@ -304,17 +295,25 @@ int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb) > > > > goto out; > > > > > > > > rc = -ENXIO; > > > > - page = follow_page(vma, uaddr, FOLL_WRITE); > > > > - if (IS_ERR_OR_NULL(page)) > > > > - goto out; > > > > - > > > > - lock_page(page); > > > > ptep = get_locked_pte(gmap->mm, uaddr, &ptelock); > > > > + > > > > + if (!pte_present(entry)) > > > > + goto out_unlock_pte; > > > > + if (pte_val(entry) & _PAGE_INVALID) > > > > + goto out_unlock_pte; > > > > > > I guess we also need to make sure the page was writable? > > > FOLL_WRITE made sure of that > > > > > > so I guess something like: > > > > > > if (!pte_write(entry)) > > > goto out_unlock_pte; > > > > Probably, that looks like an existing race that it wasn't re-checked :\ > > > > Jason > > how do we want to proceed with this? > > I can write a patch based on the above and see if anything breaks, but > it will take some thinking and testing and reviewing before I'll be > comfortable with it. I would be happy if you did that I think the code is clearly racey as written now Jason