From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1A5ACF259D for ; Mon, 14 Oct 2024 04:49:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21ED76B0082; Mon, 14 Oct 2024 00:49:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1CF1B6B0083; Mon, 14 Oct 2024 00:49:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06F7D6B0085; Mon, 14 Oct 2024 00:49:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D69916B0082 for ; Mon, 14 Oct 2024 00:49:45 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C1E3E40B5D for ; Mon, 14 Oct 2024 04:49:40 +0000 (UTC) X-FDA: 82670979678.09.CCA9A82 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by imf08.hostedemail.com (Postfix) with ESMTP id 66022160007 for ; Mon, 14 Oct 2024 04:49:39 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=n19hxiOl; spf=pass (imf08.hostedemail.com: domain of jeffxu@chromium.org designates 209.85.167.173 as permitted sender) smtp.mailfrom=jeffxu@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728881242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aXygzV2KTIZpy73JYymw8mRNeey1AWkRtD0JC+polOw=; b=7XiCrXzrKRqD8DPz0kbCC46Y2bHReei4/88cocAgAe4cqczmM7cVHmiB0HwrkL7BRxdEjL nX7tonOGaX8oZtvsdRgrgGQX8ThIzgwz6Ch/eaeBPCAgFj6XmflYXWwtyDkq4/nQZeVpPL dWz4gAwIMYpk9gCkfy7MqBb5bRl0t6c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728881242; a=rsa-sha256; cv=none; b=KrSIqResnarzLukvPZtGZ8eJEC5PQPFs70fW2XwqB3whh7JmoVja6zKx23nnfsiIKrppIV SVOb7ThtpQMopAanNlYRCD1fQ7xz3XLTAXDomrzMHGfjLU3iN830s9LlTmTMYjwYHDC9Fd JxMH+J5LhRPqa/IW9c8DIW/Y6e304dM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=n19hxiOl; spf=pass (imf08.hostedemail.com: domain of jeffxu@chromium.org designates 209.85.167.173 as permitted sender) smtp.mailfrom=jeffxu@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3db3763e924so241274b6e.2 for ; Sun, 13 Oct 2024 21:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1728881382; x=1729486182; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=aXygzV2KTIZpy73JYymw8mRNeey1AWkRtD0JC+polOw=; b=n19hxiOlE9tZWemQGzqoHADshqwYt5jZbTKmcSiO94XMmXS7T9/f/YjzyFyqpJ3Ds5 BdTrI42j0W2vyNNgOQ4kj7cl13Q3LdF1VN6twsC0GuYjPQ/lYQg/LeCpEDN0sTNODPgV u6LPUj12TS7l99ecwJnGvOlfJdLQH60giSqhE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728881382; x=1729486182; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aXygzV2KTIZpy73JYymw8mRNeey1AWkRtD0JC+polOw=; b=WuhsfA3/o/ffJYo1tfk12XCv+pQsEelLFMc/ZC5XycEyO2r3VeNexyyQ429hjcCRLu kGhJXbV8IS9N+EYTKxWVseWJHsu4rM/aIPFuRALLtIVmCPHDOWHj6Ee0o5ARlnP4d2tQ YG7wagDbgo+Og+pSDiXSHdN9KaumtWVW2hvCQaDh5HuDJnUH0pXR/hwV1Po+VbLhuG3Q G8gqGjV2wD4mPksTDIKuTAf9WH9NzN0ubIbd4WcoT2FoCBsnDn7koK74owh9wAJ98UVm FVUys6PMAK+iN8YpXZzRCj034QbyV6MoO7vSmEdKpsC5GX2DVEufLwM0cTnhBgyru3W9 EY1w== X-Forwarded-Encrypted: i=1; AJvYcCVjUf5QtBe7NnNRBqHpxcM+OftphbEw3mmYuZkLSNeDNhu5ChJLNqI4EpQ/QqsaaYjlJEP8SaeUNw==@kvack.org X-Gm-Message-State: AOJu0YyMBOpklMtP8QCsxTykgOwJ6Wokrgz+spfN+yFNP0dnDbVEZq/6 O1hoImc3zZ4xzXR2+22nSzC42zvMrEjMYHcnA4H4dIwfT9X8sLqDxTIpcduYI0xXHN8UTvXeMCC justhq5WpTRQCd9xsgS3THzu0P4naFiTXwYbc X-Google-Smtp-Source: AGHT+IEW/1AiekjBhEnn2YIPy9eT6g762PBkV5jIpTSn0H9S7U6T1VGDPofVhO6vlb2jV6yz5IwbYQWfUPh8byf/bFE= X-Received: by 2002:a05:6870:558d:b0:27b:b2e0:6af with SMTP id 586e51a60fabf-2886dd5a7d4mr1886468fac.2.1728881382240; Sun, 13 Oct 2024 21:49:42 -0700 (PDT) MIME-Version: 1.0 References: <20240927185211.729207-1-jeffxu@chromium.org> <20240927185211.729207-2-jeffxu@chromium.org> <2vkppisejac42wnawjkd7qzyybuycu667yxwmsd4pfk5rwhiqc@gszyo5lu24ge> <2q6hzkvep2g3z6m2jrwbw2j3sbydf6tgj2obwd6hgmm7xzgsg3@ddr5ghmsia5k> In-Reply-To: From: Jeff Xu Date: Sun, 13 Oct 2024 21:49:30 -0700 Message-ID: Subject: Re: [PATCH v1 1/1] mseal: update mseal.rst To: Pedro Falcato Cc: akpm@linux-foundation.org, keescook@chromium.org, corbet@lwn.net, jorgelo@chromium.org, groeck@chromium.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mm@kvack.org, jannh@google.com, sroettger@google.com, linux-hardening@vger.kernel.org, willy@infradead.org, gregkh@linuxfoundation.org, torvalds@linux-foundation.org, deraadt@openbsd.org, usama.anjum@collabora.com, surenb@google.com, merimus@google.com, rdunlap@infradead.org, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, enh@google.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 66022160007 X-Stat-Signature: mfqd4o4fywm5dei5p8uykmdz6kioo7of X-HE-Tag: 1728881379-771590 X-HE-Meta: U2FsdGVkX1+31Pd+sd3zVhk29OuFicTpamNXqxChcH7WyQHbxnSybpG/8fvV3Q0xcNpahZhBBmBxKlrgRKlEDhpU3rcQpSrLmu0eGcBbqfRdqhWenYeKhBSm/y6KkYmwsPxMthwSFtvKirtBdHx5dYvX8hY+x50KsrH7K7YryP47WLjLDaJ8ea4qiBzbMfH4j1Wl383uNrY8EpAYpmv2E3+JuRj8XIgTGoncAsqYEVAcnW9NksHeK0mQlzf8CwqlOiVyLnF5iYiqAPRteBY1Z9un5aGrq6ky7Riq80alunMROiRi46OuaL0eLerXMA+Af/2NvtXDrBa4bDNwRHONwOy6GZ+Y8I4cybkopCRim12UjS7gw13tp3NrUjLSRhbKIIv48DmRzLjjoNxWzSPCY+vDjt7bXLLpLP/tWzmtIISuynBqwk83YJX1MY3K6AwFOsBhDT5fZN85zmdQs7b92/VaVwqoLi2iFUobxQodu5rPiopt//+rH1Uw1LDqcxT8Qc1WQKWadVMhlkAzr0t3dXCP9DsK/LWOiGFCLMz7vHKUFUUW9kkfqAkG2Fdup1a7D3XlBg+UGWFQS8f2s5XGZwZSKil8ak2JLtyOb7yFXafJhNpQ5+Ex0etrnWXjGlLhkRiUuULu/UBilcp/Q9qQUGrH9I3QrzXbiFgjM9hwSb9JzWQqc8W6ekSx8tz64Fr8FQzPfKeXOv8raM7DS7VBBgutFYkFRVHPdeKTXT0bOPJ1gWrhhLmPp2256f/wiGscAyO+dHNPAQyOsC7pA/6rUmEvc2RcxAvhWOEOMT82JZxxmlxIH7iV4+PPA/eORh4OlwIx8gYSsPyaL5ed9vgYr8skQy68UMb51wO1semL/hpSW2j3lMRdhP8tSONfUfK7AVEj+85BbcBEwgk8ESqMpczwlIFXMYlnyZGRjI4KzlX8ZzkSgbuU4Fx5SwU1CElpgz4bdNmKNdN9Oz2AE8a rbMSXQQi x9gsJG+7zlhGzvT2Pp2s10DjrRDyIyItOAK4yXi+MZ4C0kq3ZJYSh66iflklRnOiAjkwkmDp2naSZcfdPytI3YEUi0g1LX5BcbNo9b8wcXhsSyMUlkLsPKu0ZixGZ4BfKy1QqQkQWOrjUyEbe7KwETlp0o4Qr/EVl0clE/cP9rbtfxyin9jTSOP7D08MHCTH1+0UGb9Z34G2DF59jVfsTiwpgjQ5nC9DPx/k7ULM/u0Kupr+d0HtuzR83Ria13k87WA365HIgZN1Z3n1GkEVLMCYjXnMHSo3hkd4dvCoGgsIdu0s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Pedro On Fri, Oct 4, 2024 at 10:02=E2=80=AFAM Pedro Falcato wrote: > > On Mon, Sep 30, 2024 at 05:24:39PM -0700, Jeff Xu wrote: > > Hi Pedro > > > > On Sat, Sep 28, 2024 at 6:43=E2=80=AFAM Pedro Falcato wrote: > > > > > > On Fri, Sep 27, 2024 at 06:29:30PM GMT, Jeff Xu wrote: > > > > Hi Pedro, > > > > > > > > On Fri, Sep 27, 2024 at 3:59=E2=80=AFPM Pedro Falcato wrote: > > > > > > > > > + > > > > > > + Blocked mm syscall: > > > > > > + - munmap > > > > > > + - mmap > > > > > > + - mremap > > > > > > + - mprotect and pkey_mprotect > > > > > > + - some destructive madvise behaviors: MADV_DONTNEED, MAD= V_FREE, > > > > > > + MADV_DONTNEED_LOCKED, MADV_FREE, MADV_DONTFORK, MADV_W= IPEONFORK > > > > > > + > > > > > > + The first set of syscall to block is munmap, mremap, mmap. = They can > > > > > > + either leave an empty space in the address space, therefore= allow > > > > > > + replacement with a new mapping with new set of attributes, = or can > > > > > > + overwrite the existing mapping with another mapping. > > > > > > + > > > > > > + mprotect and pkey_mprotect are blocked because they changes= the > > > > > change > > > > > > + protection bits (rwx) of the mapping. > > > > > > + > > > > > > + Some destructive madvice behaviors (MADV_DONTNEED, MADV_FRE= E, > > > > > > + MADV_DONTNEED_LOCKED, MADV_FREE, MADV_DONTFORK, MADV_WIPEON= FORK) > > > > > > + for anonymous memory, when users don't have write permissio= n to the > > > > > > + memory. Those behaviors can alter region contents by discar= ding pages, > > > > > > + effectively a memset(0) for anonymous memory. > > > > > > > > > > What's the difference between anonymous memory and MAP_PRIVATE | = MAP_FILE? > > > > > > > > > MAP_FILE seems not used ? > > > > anonymous mapping is the mapping that is not backed by a file. > > > > > > MAP_FILE is actually defined as 0 usually :) But I meant file-backed = private mappings. > > > > > OK, we are on the same page for this. > > > > > > > The feature now, as is (as far as I understand!) will allow you t= o do things like MADV_DONTNEED > > > > > on a read-only file mapping. e.g .text. This is obviously wrong? > > > > > > > > > When a MADV_DONTNEED is called, pages will be freed, on file-backed > > > > mapping, if the process reads from the mapping again, the content > > > > will be retrieved from the file. > > > > > > > > > > Sorry, it was late and I gave you a crap example. Consider this: > > > a file-backed MAP_PRIVATE vma is marked RW. I write to it, then RO-it= + mseal. > > > > > > The attacker later gets me to MADV_DONTNEED that VMA. You've just los= t data. > > > > > > The big problem here is with anon _pages_, not anon vmas. > > > > > That depends on the app's threat-model. What you described seems to be > > a case below > > 1. The file is rw > > 2. The process opens the file as rw > > 3. the process mmap the fd as rw > > 4 The process writes the memory, and the change isn't flushed to the > > file on disk. > > 5 The process changes the mapping to RO > > 6. The process seals the mapping > > 7. The process is called MADV_DONTNEED , and because the change isn't > > flush to file on disk, so it loses the change, (retrieve the old data > > from disk when read from the mapped address later) > > > > I'm not sure this is a valid use case, the problem here seems to be > > that the app needs to flush the change from memory to disk if the > > expectation is writing is permanent. > > > > MAP_PRIVATE never does writeback. That's not what this is about. > I can trivially discard anonymous pages for private "file VMAs", which ar= en't > refilled with the exact same contents. This is a problem. > That is fair, I appreciate you providing the use case. I think mseal should support this . In addition, after reviewing MADV_DONTNEED, I believe we should also allow madvise(DONTNEED) when PROT is PROT_NONE. I will work on fixes for those, but before that, I like to make sure some of the existing fixes are backported to 6.10, which makes it easy to backport future fixes. > > In any case, the mseal currently just blocks a subset of madvise, those > > we know with a security implication. If there is something mseal needs > > to block additionally, one can always extend it by using the "flags" fi= eld. > > I do think the bar is high though, e.g. a valid use case to support tha= t. > > No, this has nothing to do with a flag. It's about providing sane semanti= cs. > > > > > > > For anonymous mapping, since there is no file backup, if process > > > > reads from the mapping, 0 is filled, hence equivalent to memset(0) > > > > > > > > > > + > > > > > > + Kernel will return -EPERM for blocked syscalls. > > > > > > + > > > > > > + When blocked syscall return -EPERM due to sealing, the memo= ry regions may or may not be changed, depends on the syscall being blocked: > > > > > > + - munmap: munmap is atomic. If one of VMAs in the given = range is > > > > > > + sealed, none of VMAs are updated. > > > > > > + - mprotect, pkey_mprotect, madvise: partial update might= happen, e.g. > > > > > > + when mprotect over multiple VMAs, mprotect might updat= e the beginning > > > > > > + VMAs before reaching the sealed VMA and return -EPERM. > > > > > > + - mmap and mremap: undefined behavior. > > > > > > > > > > mmap and mremap are actually not undefined as they use munmap sem= antics for their unmapping. > > > > > Whether this is something we'd want to document, I don't know hon= estly (nor do I think is ever written down in POSIX?) > > > > > > > > > I'm not sure if I can declare mmap/mremap as atomic. > > > > > > > > Although, it might be possible to achieve this due to munmap being > > > > atomic. I'm not sure as I didn't test this. Would you like to find > > > > out ? > > > > > > I just told you they use munmap under the hood. It's just that the re= quirement isn't actually > > > written down anywhere. > > > > > I knew about mmap/mremap calling munmap. I don't know what exactly you > > are asking though. In your patch and its discussion, you did not mentio= n > > the mmap/mremap (for sealing) is or should be atomic. > > > > My point is: since there isn't a clear statement from your patch descri= ption > > or POSIX, that mremap/mmap is atomic, and I haven't tested it myself w= ith > > regards to sealing, let's leave them as "undefined" for now. (I could = get back > > to this later after the merging window) > > > > > > > > > > > > > > > > > > Use cases: > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > - glibc: > > > > > > The dynamic linker, during loading ELF executables, can appl= y sealing to > > > > > > - non-writable memory segments. > > > > > > + mapping segments. > > > > > > > > > > > > - Chrome browser: protect some security sensitive data-structu= res. > > > > > > > > > > > > -Notes on which memory to seal: > > > > > > -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > - > > > > > > -It might be important to note that sealing changes the lifetim= e of a mapping, > > > > > > -i.e. the sealed mapping won=E2=80=99t be unmapped till the pro= cess terminates or the > > > > > > -exec system call is invoked. Applications can apply sealing to= any virtual > > > > > > -memory region from userspace, but it is crucial to thoroughly = analyze the > > > > > > -mapping's lifetime prior to apply the sealing. > > > > > > +Don't use mseal on: > > > > > > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > +Applications can apply sealing to any virtual memory region fr= om userspace, > > > > > > +but it is *crucial to thoroughly analyze the mapping's lifetim= e* prior to > > > > > > +apply the sealing. This is because the sealed mapping *won=E2= =80=99t be unmapped* > > > > > > +till the process terminates or the exec system call is invoked= . > > > > > > > > > > There should probably be a nice disclaimer as to how most people = don't need this or shouldn't use this. > > > > > At least in its current form. > > > > > > > > > Ya, the mseal is not for most apps. I mention the malloc example to= stress that. > > > > > > > > > > > > > > > - > > > > > > - > > > > > > -Additional notes: > > > > > > -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > As Jann Horn pointed out in [3], there are still a few ways to= write > > > > > > -to RO memory, which is, in a way, by design. Those cases are n= ot covered > > > > > > -by mseal(). If applications want to block such cases, sandbox = tools (such as > > > > > > -seccomp, LSM, etc) might be considered. > > > > > > +to RO memory, which is, in a way, by design. And those could b= e blocked > > > > > > +by different security measures. > > > > > > > > > > > > Those cases are: > > > > > > - > > > > > > -- Write to read-only memory through /proc/self/mem interface. > > > > > > -- Write to read-only memory through ptrace (such as PTRACE_POK= ETEXT). > > > > > > -- userfaultfd. > > > > > > + - Write to read-only memory through /proc/self/mem interfac= e (FOLL_FORCE). > > > > > > + - Write to read-only memory through ptrace (such as PTRACE_= POKETEXT). > > > > > > + - userfaultfd. > > > > > > > > > > I don't understand how this is not a problem, but MADV_DONTNEED i= s. > > > > > To me it seems that what we have now is completely useless, becau= se you can trivially > > > > > bypass it using /proc/self/mem, which is enabled on most Linux sy= stems. > > > > > > > > > > Before you mention ChromeOS or Chrome, I don't care. Kernel featu= res aren't designed > > > > > for Chrome. They need to work with every other distro and applica= tion as well. > > > > > > > > > > It seems to me that the most sensible change is blocking/somehow = distinguishing between /proc/self/mem and > > > > > /proc//mem (some other process) and ptrace. As in blocking /= proc/self/mem but allowing the other FOLL_FORCE's > > > > > as the traditional UNIX permission model allows. > > > > > > > > > IMO, it is a matter of Divide and Conquer. In a nutshell, mseal o= nly > > > > prevents VMA's certain attributes (such as prot bits) from changing= . > > > > It doesn't mean to say that sealed RO memory is immutable. To achie= ve > > > > that, the system needs to apply multiple security measures. > > > > > > No, it's a matter of providing a sane API without tons of edgecases. = Making a VMA immutable should make a VMA > > > immutable, and not require you to provide a crap ton of other mechani= sms in order to truly make it immutable. > > > If I call mseal, I expect it to be sealed, not "sealed except when it= 's not, lol". > > > > > > You haven't been able to quite specify what semantics are desirable o= ut of this whole thing. Making > > > prot flags "immutable" is completely worthless if you can simply writ= e to a random pseudofile and > > > have it bypass the whole thing (where a write to /proc/self/mem is se= mantically equivalent to > > > mprotect RW + write + mprotect RO). Making the vma immutable is compl= etely worthless > > > if I can simply wipe anon pages. There has to be some end goal here (= make contents immutable? > > > make sure VMA protection can't be changed? both?) which seems to be u= nclear from the kernel mmap-side. > > > > > > If you insist on providing half-baked APIs (and waving off any concer= ns), I'm sure this would've been better > > > implemented as a random bpf program for chrome. Maybe we could revert= this whole thing and give eBPF one > > > or two bits of vma flags for their own uses :) > > > > > Please reply to the above. We're struggling to understand exactly what se= mantics you want from this. > *That* is what we want to document and get set in stone, and we'll move f= rom there. > If you meant to make mseal to support blocking /proc/self/mem or ptrace or other cases, I welcome that. Please go ahead to implement it with a flag, the mseal already has a field for such future extension. The current mseal's semantic doesn't come out from vacuum, for system such as ChromeOS and Android, they are already relying heavily on security mechanisms to block /proc/self/mem or ptrace, as I pointed out previously, SELINUX/YAMA/Landlock/seccomp, and most recently CONFIG_PROC_MEM_RESTRICT_WRITE_ALL all contributes to that, therefore I don't have a use case or needs to make mseal to additionally block those. But I understand others might have different choices on which security mechanism to use, and I don't want to argue about which requirements are correct. Security isn't a true/false binary state, each system is different and no single security feature can achieve the absolute "secure" state. Thanks -Jeff > > > > > > > > For writing to /proc/pid/mem, it can be disabled via [1]. SELINUX = and > > > > Landlock can achieve the same protection too. > > > > > > I'm not blocking /proc/pid/mem, and my distro doesn't run any of thos= e security modules :/ > > > > > It is a choice you can make :-) > > Your feature needs to work without "extra choices". > > -- > Pedro