From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85A6AC3DA63 for ; Tue, 23 Jul 2024 18:24:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D74396B00A8; Tue, 23 Jul 2024 14:24:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D23A46B00AA; Tue, 23 Jul 2024 14:24:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B77B36B00AD; Tue, 23 Jul 2024 14:24:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9632B6B00A8 for ; Tue, 23 Jul 2024 14:24:58 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 335BD8035A for ; Tue, 23 Jul 2024 18:24:58 +0000 (UTC) X-FDA: 82371843876.21.D2AB20C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id C72B9C001C for ; Tue, 23 Jul 2024 18:24:55 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JOs52ICA; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721759059; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mlSldsjWGkt137rHxqPtPFE0Ch/bG3EUj3dygNmEleA=; b=NjYCQOv0amV+89M9eeMJJRO2QRfzQALrjFpsX/LT5zTmaFkGhGB49m987BUOpM7qjYpVQG KAMMkZkq1cLyyC5Ni864lgXCBQTKlIqIMEl0MPf2xQY/FybepXxpUFZpw0ENWEgyb0T1YK eoDhYrFAI8YQ1BLHCZgN8nRXlZ+OzCs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JOs52ICA; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721759059; a=rsa-sha256; cv=none; b=Idc6nG0OEfJTt8SkESZbQe+VRKolxB4pKKyYHGnIeqI2Np8QRC1Ul1xlySgNtTXSRt2XpW KZN+ojR/81emQPmxAKTgQk1iai7gTzu7b4FhWns3JzC9oh6syTwr5Ni9VsCNqpDOlhuMpN 57ytJX2CbjE1I3dtCKkp+b9DoBo6VHs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721759095; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=mlSldsjWGkt137rHxqPtPFE0Ch/bG3EUj3dygNmEleA=; b=JOs52ICAb/BRygtEFQdU3ROKz3NJgX5KIwywmn6V0XG35KHPGy5QxrGuUhTr+E65wl6Sfa YJvgfbRgcfm8r7tJYENsHnqThiKM39oqV6MHFkTxQkPo/BWVYfSm8exMFQoAK1R8kMq0Rp DC7O7fpIagte4snOvM1D8yg5OcTE3Qs= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-240-nql1scjRNWKVc6yYvpy7VQ-1; Tue, 23 Jul 2024 14:24:53 -0400 X-MC-Unique: nql1scjRNWKVc6yYvpy7VQ-1 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-42725ef39e2so41753715e9.0 for ; Tue, 23 Jul 2024 11:24:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721759092; x=1722363892; h=content-transfer-encoding:in-reply-to:organization:autocrypt :content-language:references:cc:to:from:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=mlSldsjWGkt137rHxqPtPFE0Ch/bG3EUj3dygNmEleA=; b=kkZTXrqtCbmcZ+wSiXTziG/A4qFL4ku4p3YGZiEDwNPj/aZo+qCr+Qyw/yGfeoeQdo WXRXdvbAGwdCYQTRth+AqnjNuUxHWVx6X5iDbCSO7WhtFnPKNA+rLbNEq5AH7TU3+4DP /gfJ1x6qcl4LZ/Ko7+uXDknEX/6uvGz+SobcQ6VAWWIZ5YRUldwf08dp52TWbWv79zPh nsQw8nShvosEnXes3nYLqtQdJO93v/vUkNECBz4Gp++h32Zx5+2L0jrzVI4oI99T+wTW Nb1vh31AKzpgGFqRGKV7qoP1Dx2qDk4NwnIvT9SY7hGWsZOumjwdEiGymdULEXt4NnIl FwPQ== X-Forwarded-Encrypted: i=1; AJvYcCU2YW4idZ1u1cfEslDLMjmSsBEUWsUDAImBJ0i60RTXlnxo/8rwTovIR4iEzI7WItgAXQE1xFz5OL64lIgl4n9CCJA= X-Gm-Message-State: AOJu0YxfT3h6cuQHHnOuqjmy/YxYCmxwWWA8v7XGeSALW7ZI0awpGghr /R0zgJGGE063tz16apiW8MvWh8BbpKPbGuSjI/EBtPpA9X4HvEg/AoakCdsHT8RyLAuqje2Zm5/ ho3Sd+r2TTsoXsAqijGVcW8IRoKYJI2pr5X3s0lJdjSU1OgIQuTNcQoLe X-Received: by 2002:a05:600c:1c13:b0:426:5ef5:bcb1 with SMTP id 5b1f17b1804b1-427f7a15278mr2493755e9.6.1721759091948; Tue, 23 Jul 2024 11:24:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHq7GXtYAKFTz+qGMMS1RUEa8XcH1p15cy6QfGw/2Plhyb1cls798ESEkmZL12VE1vSOh74Nw== X-Received: by 2002:a05:600c:1c13:b0:426:5ef5:bcb1 with SMTP id 5b1f17b1804b1-427f7a15278mr2493625e9.6.1721759091497; Tue, 23 Jul 2024 11:24:51 -0700 (PDT) Received: from ?IPV6:2003:cb:c72f:ed00:9dc2:1adb:d133:4434? (p200300cbc72fed009dc21adbd1334434.dip0.t-ipconnect.de. [2003:cb:c72f:ed00:9dc2:1adb:d133:4434]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427d6901781sm182893445e9.14.2024.07.23.11.24.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 23 Jul 2024 11:24:51 -0700 (PDT) Message-ID: Date: Tue, 23 Jul 2024 20:24:49 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] mm: fix maxnode for mbind(), set_mempolicy() and migrate_pages() From: David Hildenbrand To: Jerome Glisse Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Andi Kleen References: <20240720173543.897972-1-jglisse@google.com> <0c390494-e6ba-4cde-aace-cd726f2409a1@redhat.com> <6be6453a-15ce-4305-9a7c-a66e57564785@redhat.com> Autocrypt: addr=david@redhat.com; keydata= xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9 PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9 M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3 AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8 kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt WNyWQQ== Organization: Red Hat In-Reply-To: <6be6453a-15ce-4305-9a7c-a66e57564785@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Stat-Signature: yz4xnrzfhj1whephw4p8sxjwwpo74j6a X-Rspam-User: X-Rspamd-Queue-Id: C72B9C001C X-Rspamd-Server: rspam02 X-HE-Tag: 1721759095-430278 X-HE-Meta: U2FsdGVkX1958aYVGUZZGnyaZmv/0qS6m4UMnvDyvbQmM2E1A1XEJkNb4RQXQpVLrBg0YfZFT/LRcGpPDYvPcKKNsOIdVXVnUpyJraW6H113HzQKJMrTZVfoeLRJGFwn1qKd9ayppz9DynLPe9p7M1yxoMOS2+VAl6ShSni/QSiIOZPnQftQR08WNonYxQx1tIa8gcd86DuSQ367tdcQGJEe+sw/UI6B3ZbwUsUjegsE5o6lBXdZZk74bYXvT1Jh0LVT7mr0M0IjvTh5nVnOkuExWwA8/9zPuNFoZ73Ad3CFsjgzJYjsrqgUqGB6Aqtpi2I4iw/5aiYd0vwToOzTjfF1sX+NPRY5K+fuqgxkMbrMvkh5BOMOK0P8iLncsMFXWOF4IUqR151g2oZ7bIlpdBmlZImBpAIOUgQ+AOcKOB/+U7Biub7x/ppsqH4PYwQq1qg04zpQOcwD90KnEX1tSr6y86Gpm/ssD9hFzxXBS1Px3g3AeAy3RU5r2QK0H1WONuMg712RKMuHoY+Z+J3C9gbnH4rKRrrcDcZlqQJ0XVD9E6w1Ge9yzZkWbPh9/dQeFEa1LBhXrtxaLELsZaWVkrrAkzgiH2cyO5K/Mfe3Tl7ReNuysH1jlpZQ5fi+KgEbzY34uwO/FYL24jnkYBtiFZVIOvLxuaQjA3cc0CHz+3+oTzw+BVbuaqqWz8RGfYUrdf7dmbdNNrzmkShYffMLhW7V80vJzJRg7jSx7mCzSyDBHCM2veIiXBXD5fKw3xWJvjBDqUKsWKrnU//nWRTRTbEv8eqkcQb1+QvNysIqjiRvx6C4Xtro0aA+1xt1+9Ho+Lghm2vONEVnxPEAm8nfYEHQ+TV83nQCaSqv8rbrZoXrbtDex7nrVHTTuX3SsdyXRodoR1rKaVUXBxSMLqy9tLw0h06ZM1TMKZQ27JIG0dIbpkeepXFXs/ZXDkkO/WRidBUHwQMQy66+DZgiPII UZ81xDmm Xr+QRrlFWxybrjd4u3RDS93AjoNw7lu7fFRn6cvgYVv+jijG6XAC7Hri7/cSbjf6Zw91upBDRb+Ei6E386YSuxezMdfe+aiOvqTD1iYxvj2Pv4dy7OytI2qkOc72K4TTfUmEcvIX7XMWh2UsCHzTe6Kd10iM0fqqomSOkkhwi7DL6Zy5zeVfWp7L3Uj6jQMquvlWZeT6vwwDCQIx13Efi5piG556ejBFXImW/izMlYciuSnHRLKPSNWZoxqlP9X0vtTgq6FzVNrzwltjL6hwN45ZVhJHllme+KJ2FDUMLRfOhKbLlN5wpIJZU4lfSEHUqGOGfok0oBNkma+WKrjF5s+Xjq4gnUZRTvUn3tDdDM//myDhbRde8y1ukb5+K0+BuMIxMv4eiGdK4qV9gZKvKe2sL51GTIdjsHlDdKAigizEkZHzdN/qWxiu9bUz3jg7blcCD9vcGSh7OUklDF4F65nsDR3aBg5dimhOWO9LCGy6afj3je59j50OkguGOHMzq4t79ruEXJE5m+61mOIhAEjixkxIT2CqjPKGJGne61cZ/S4k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 23.07.24 19:37, David Hildenbrand wrote: > On 23.07.24 18:33, Jerome Glisse wrote: >> On Mon, 22 Jul 2024 at 06:09, David Hildenbrand wrote: >>> >>> On 20.07.24 19:35, Jerome Glisse wrote: >>>> Because maxnode bug there is no way to bind or migrate_pages to the >>>> last node in multi-node NUMA system unless you lie about maxnodes >>>> when making the mbind, set_mempolicy or migrate_pages syscall. >>>> >>>> Manpage for those syscall describe maxnodes as the number of bits in >>>> the node bitmap ("bit mask of nodes containing up to maxnode bits"). >>>> Thus if maxnode is n then we expect to have a n bit(s) bitmap which >>>> means that the mask of valid bits is ((1 << n) - 1). The get_nodes() >>>> decrement lead to the mask being ((1 << (n - 1)) - 1). >>>> >>>> The three syscalls use a common helper get_nodes() and first things >>>> this helper do is decrement maxnode by 1 which leads to using n-1 bits >>>> in the provided mask of nodes (see get_bitmap() an helper function to >>>> get_nodes()). >>>> >>>> The lead to two bugs, either the last node in the bitmap provided will >>>> not be use in either of the three syscalls, or the syscalls will error >>>> out and return EINVAL if the only bit set in the bitmap was the last >>>> bit in the mask of nodes (which is ignored because of the bug and an >>>> empty mask of nodes is an invalid argument). >>>> >>>> I am surprised this bug was never caught ... it has been in the kernel >>>> since forever. >>> >>> Let's look at QEMU: backends/hostmem.c >>> >>> /* >>> * We can have up to MAX_NODES nodes, but we need to pass maxnode+1 >>> * as argument to mbind() due to an old Linux bug (feature?) which >>> * cuts off the last specified node. This means backend->host_nodes >>> * must have MAX_NODES+1 bits available. >>> */ >>> >>> Which means that it's been known for a long time, and the workaround >>> seems to be pretty easy. >>> >>> So I wonder if we rather want to update the documentation to match reality. >> >> [Sorry resending as text ... gmail insanity] >> >> I think it is kind of weird if we ask to supply maxnodes+1 to work >> around the bug. If we apply this patch qemu would continue to work as >> is while fixing users that were not aware of that bug. So I would say >> applying this patch does more good. Long term qemu can drop its >> workaround or keep it for backward compatibility with old kernel. > > Not really, unfortunately. The thing is that it requires a lot more > effort to sense support than simply pass maxnodes+1. So unless you know > exactly on which minimum kernel version your software runs (barely > happens), you will simply apply the workaround. > > I would assume that each and every sane user out there does that > already, judging that even that QEMU code is 10 years old (!). > > In any case, we have to document that behavior that existed since the > very beginning. Because it would be even *worse* if someone would > develop against a new kernel and would get a bunch of bug reports when > running on literally every old kernel out there :) > > So my best guess is that long-term it will create more issues when we > change the behavior ... but in any case we have to update the man pages. I'll add a pointer to a discussion from 20 (!) year ago [1]. The conclusion [2] / request from Andi there was that N65 is the right thing to do, and we (re)added the --maxnode. This was when this code was introduced. Assuming MAX_NODES is 128, we have to pass in 129 -- one more than the number of bits. So the man page might be actively misleading (unless I misinterpret it): "nodemask points to a bit mask of nodes containing up to maxnode bits. The bit mask size is rounded to the next multiple of sizeof(unsigned long), but the kernel will use bits only up to maxnode." That documentation should be fixed. [1] https://lkml.indiana.edu/hypermail/linux/kernel/0409.1/1538.html [2] https://lkml.indiana.edu/hypermail/linux/kernel/0409.1/1569.html -- Cheers, David / dhildenb