From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FF48C3271E for ; Tue, 9 Jul 2024 00:12:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AFF636B0099; Mon, 8 Jul 2024 20:12:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AAF236B009C; Mon, 8 Jul 2024 20:12:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94F376B009D; Mon, 8 Jul 2024 20:12:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 719CA6B009C for ; Mon, 8 Jul 2024 20:12:12 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E4F32A4DD4 for ; Tue, 9 Jul 2024 00:12:11 +0000 (UTC) X-FDA: 82318286862.01.1D5157D Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by imf06.hostedemail.com (Postfix) with ESMTP id 1A513180007 for ; Tue, 9 Jul 2024 00:12:09 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="VtHD/aMZ"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of tjmercier@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=tjmercier@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720483896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oVbaaS6scR3r/YQCj7vaVWFVSsqXZHGDW0NIfp1BYWA=; b=kFlQknpyoOqCjX6IrNyZCnxdhxDw3WAejCcfaM8pXMevT67Q49gxVCeCqyqHl4Wca+MKXW C0SEElCbakMk4Kem63KGVBNm+VPi3m4Y2FR+jcFz+ZmGS2eHdqAAyPFGLXQwYw0+OtJ2TR VjTZ1cM+dI7Rh5gl7olZdNVdGU6pQjA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720483896; a=rsa-sha256; cv=none; b=2v30L2LpRX29GKp6rIq3wvvfiEmtXD5eCFwwcWI1RNwGP6Mr6YmFee4gpiGr3JH3hs5h2W jbBwPEE4WY16V0VmVvMwiXFzG1TNxONJwZlzkscyU0V6Ji5DO/k5p7Uk9VzmcKjV8H3QdZ Ey+nJ0UWNYn6jbAc5RR/OvgrX81QBhk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="VtHD/aMZ"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of tjmercier@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=tjmercier@google.com Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-e03a6196223so5011712276.3 for ; Mon, 08 Jul 2024 17:12:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720483929; x=1721088729; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=oVbaaS6scR3r/YQCj7vaVWFVSsqXZHGDW0NIfp1BYWA=; b=VtHD/aMZu2op/YyHiFePcJiYEOZM4i7w77Hz7Wj+qrjUS2NyBb1gDrxRNcMmnBVrZx 6MbDrqg5G/T1h8S/cKFC16Pb4iU7gBZCMBI8QPzgZxfFaJ73+640ZznpCYctHICBs8mm qiWu9qgjxIdkXTMWBGJS1kk0w+SeNG1RveeJHKxx6Mbp2pudw1wvKIijQyoJ4bGgc9FA m3jUOqhXyP0aR6bXHFXu+eRSBOkjBhgeqFFm3PVd76xiRJ1WpP18FzJznAA8O6dWIvwF /LJtXJiSfhLMjJROMiTugMsj87LcvoQpyWfotGik/xoIWREsB7lUIZkAxVVaEVm1+x8y JAew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720483929; x=1721088729; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oVbaaS6scR3r/YQCj7vaVWFVSsqXZHGDW0NIfp1BYWA=; b=uqmWxguoBnjzww0NjGGbFG6W3NNCcor730RpRmMYkgD7eoQ14s9Ummrf0PFJzhz5qV ZvAgFFMCAgeJ4JZSkAW2KrUOLgUaxL0ovK4eGArmPUC9ZcNZyXxiYmIYeqFUnKa081pC e1yYT2pejRBCPJJkee9ajOTvuLyBE1Lz7d9YNZHthR9sy+Mp6qjmjCnz76JVS3S0LfsP gYaBC06MOGSLeRhwwGvdF5/yh0qrjzMaF9DegeJ4BHbVVm8sqZayyJORPliieUG1sd+R bGSIf7vvvAH1cW/13cYXefBMgZssQlwp/zMLnc1nLLpqmcYvuqFhYe29VAR4lOmRlHIl da6g== X-Forwarded-Encrypted: i=1; AJvYcCUEKMI7bAAVQGNMJr2JkGLiH6Iczj1bkt5icOCyhJK4tFcaYRJ6Qf5aY6DmE9HIXlFOoEcu54PuZFlmsmotCpHG75s= X-Gm-Message-State: AOJu0YwqNSMvYXoe6m+grnuUlXcj2V0/rOULicaW3So6TbbFdmkVCDNs CQFS02v6B7vnNPI5cv93b7m9MW5hKCwwJY3SZXsE3CvqvpZgr4luu6S5sRCEuyPz8qv9pW8deU/ +Sdyk1NmMEykmn9rPYRldid7b49rBbiYwm5WK X-Google-Smtp-Source: AGHT+IEvOP5YgTU9NpaKmjdo45GdXyB6IULqBv9Iv4UMXSFc+i2Xm5Ackgx2bhCetWSIWrnTOedKP/EoCHDgLCSgLhY= X-Received: by 2002:a25:c441:0:b0:e03:3d38:fb26 with SMTP id 3f1490d57ef6-e041b078a64mr1551144276.28.1720483928711; Mon, 08 Jul 2024 17:12:08 -0700 (PDT) MIME-Version: 1.0 References: <20240702084423.1717904-1-link@vivo.com> <27a62e44-9d85-4ef2-b833-e977af039758@vivo.com> In-Reply-To: From: "T.J. Mercier" Date: Mon, 8 Jul 2024 17:11:56 -0700 Message-ID: Subject: Re: [RFC PATCH 0/4] Introduce PMC(PER-MEMCG-CACHE) To: Huan Yang Cc: Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , Ryan Roberts , Chris Li , Dan Schatzberg , Kairui Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christian Brauner , opensource.kernel@vivo.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1A513180007 X-Stat-Signature: ogzcnd4g18sa67uq3atuzmwws3hiwidn X-Rspam-User: X-HE-Tag: 1720483929-603212 X-HE-Meta: U2FsdGVkX18IBg7mNLMPofbP7Qk0YJ/tr2h8wREfaGIOJHwIi9bxjNahCfjC9I4jl89w7xh/axyXCgJ8kqlXNWYyh2DM01e+XSOX2W8iACouT7IGyGv2YlG2dHhLwnmgrTtT4jwOzq8Ju6copfMWnViaiyMVc6nUmMeh/256w2/feasSn2TPROCuJIVKKUcmg/DUVS+bHniYwTLNIEyX6saaadcie82ftT7K05c6bHapQ/oKZthPcVQ31IDvpNffr94Oa7T/Q4IaYSpG+gfobUPWBM5NmS/PzUJKEe0c4IkP0ZRBmJCqnvii9xGE8xy8I+5lkKkLzUMufn8nH5IBibJRJFgYRo4reb+uPhqHLvnlTGfMTih2DYqXNkdF/YOkMB74XqV8E3MSns5Or9EjtU/kGJhXGS9ykwjylkxXlAjAtOrl8LIW8orYoU51RHU8+lHFNalWMMNbLMd2BOUKdnJiVZL+zsnOau+Rc69Yf6c+Fyb1/LiGv9zrM7a0VV99FK8U0mxW1yCNQSQ17qYBi08Zs0mDyTRBv6zlGhQAlE68bNF9tl6kk8n+Inco50lS/EBnWBBsKDPNxeceYjj/EYjocz7nWBux5f+ggMgrBnJ9UyeYquFSDXs05vVvyNgLEhigBEjgHP4lHFJagRzYow71nxYDW3zMWmS0Vkwn+utTNyEvh78I9cEpQ+NopR4a+K48Yu+QtU1LnpI2rr99MHNR8H7rlVivXVjamX0YlgRIVUqljlqbEFYsglbHAjLp60MSALGyTAyQSEQERo53Zz0pGQa9hQlaBcnfPP9Ed/J7taa8GBxcb9F6DLbsezzZDAa05oeJyZKJ7Tw18FrSkpETUwQYBP9cf1m5F7KmX0Dkd3AdHgjEGWTfqf0ZmLeOrljpaiWrtbuDShS5XUMCFL2Ta6D6w2gLYf5nfMrfx6hG9GIhGmYBcUn2hZL8jGmQtsWS7NeiNtPUTEebql/ Djit7byz yvUn0ggVvG71ZF/yaka12eMvvQaCUiewepgdjiMlbdfRoWxAikTyIEA0L78t1auME9E7Lywf5n5aeaWNG1CGgIWbVhLGKd3bu935qDf7QKGYH72uxuob95aHPWV+EI94AybL7wfzo7gC0oNnB3WRhvUpgANphA+X0+Zu6qTkIflsxQo/kpP+w/3aYUgmurtt6sP8H2JYIZY0fOKPgDYWN8FG0IsSi12hW/6LP9MixO/PcPsSeNc+3Iy9gEOnym8TJILxrSEh7k4b1NGpePul/bADLKYtQQOBd3L9ioYN2FBpm+49k7r7WDy4kLyNeAvz25Bgp8vWR0PlTnJaN1VWxsW5KoSAtH4J6jCiOgmXwvO/lvrg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 3, 2024 at 7:29=E2=80=AFPM Huan Yang wrote: > > > =E5=9C=A8 2024/7/4 6:59, T.J. Mercier =E5=86=99=E9=81=93: > > On Tue, Jul 2, 2024 at 7:23=E2=80=AFPM Huan Yang wrote: > >> > >> =E5=9C=A8 2024/7/3 3:27, Roman Gushchin =E5=86=99=E9=81=93: > >>> On Tue, Jul 02, 2024 at 04:44:03PM +0800, Huan Yang wrote: > >>>> This patchset like to talk abount a idea about PMC(PER-MEMCG-CACHE). > >>>> > >>>> Background > >>>> =3D=3D=3D > >>>> > >>>> Modern computer systems always have performance gaps between hardwar= e, > >>>> such as the performance differences between CPU, memory, and disk. > >>>> Due to the principle of locality of reference in data access: > >>>> > >>>> Programs often access data that has been accessed before > >>>> Programs access the next set of data after accessing a particula= r data > >>>> As a result: > >>>> 1. CPU cache is used to speed up the access of already accessed = data > >>>> in memory > >>>> 2. Disk prefetching techniques are used to prepare the next set = of data > >>>> to be accessed in advance (to avoid direct disk access) > >>>> The basic utilization of locality greatly enhances computer performa= nce. > >>>> > >>>> PMC (per-MEMCG-cache) is similar, utilizing a principle of locality = to enhance > >>>> program performance. > >>>> > >>>> In modern computers, especially in smartphones, services are provide= d to > >>>> users on a per-application basis (such as Camera, Chat, etc.), > >>>> where an application is composed of multiple processes working toget= her to > >>>> provide services. > >>>> > >>>> The basic unit for managing resources in a computer is the process, > >>>> which in turn uses threads to share memory and accomplish tasks. > >>>> Memory is shared among threads within a process. > >>>> > >>>> However, modern computers have the following issues, with a locality= deficiency: > >>>> > >>>> 1. Different forms of memory exist and are not interconnected (a= nonymous > >>>> pages, file pages, special memory such as DMA-BUF, various me= mory alloc in > >>>> kernel mode, etc.) > >>>> 2. Memory isolation exists between processes, and apart from spe= cific > >>>> shared memory, they do not communicate with each other. > >>>> 3. During the transition of functionality within an application,= a process > >>>> usually releases memory, while another process requests memor= y, and in > >>>> this process, memory has to be obtained from the lowest level= through > >>>> competition. > >>>> > >>>> For example abount camera application: > >>>> > >>>> Camera applications typically provide photo capture services as well= as photo > >>>> preview services. > >>>> The photo capture process usually utilizes DMA-BUF to facilitate the= sharing > >>>> of image data between the CPU and DMA devices. > >>>> When it comes to image preview, multiple algorithm processes are typ= ically > >>>> involved in processing the image data, which may also involve heap m= emory > >>>> and other resources. > >>>> > >>>> During the switch between photo capture and preview, the application= typically > >>>> needs to release DMA-BUF memory and then the algorithms need to allo= cate > >>>> heap memory. The flow of system memory during this process is manage= d by > >>>> the PCP-BUDDY system. > >>>> > >>>> However, the PCP and BUDDY systems are shared, and subsequently requ= ested > >>>> memory may not be available due to previously allocated memory being= used > >>>> (such as for file reading), requiring a competitive (memory reclamat= ion) > >>>> process to obtain it. > >>>> > >>>> So, if it is possible to allow the released memory to be allocated w= ith > >>>> high priority within the application, then this can meet the localit= y > >>>> requirement, improve performance, and avoid unnecessary memory recla= im. > >>>> > >>>> PMC solutions are similar to PCP, as they both establish cache pools= according > >>>> to certain rules. > >>>> > >>>> Why base on MEMCG? > >>>> =3D=3D=3D > >>>> > >>>> The MEMCG container can allocate selected processes to a MEMCG based= on certain > >>>> grouping strategies (typical examples include grouping by app or UID= ). > >>>> Processes within the same MEMCG can then be used for statistics, upp= er limit > >>>> restrictions, and reclamation control. > >>>> > >>>> All processes within a MEMCG are considered as a single memory unit, > >>>> sharing memory among themselves. As a result, when one process relea= ses > >>>> memory, another process within the same group can obtain it with the > >>>> highest priority, fully utilizing the locality of memory allocation > >>>> characteristics within the MEMCG (such as APP grouping). > >>>> > >>>> In addition, MEMCG provides feature interfaces that can be dynamical= ly toggled > >>>> and are fully controllable by the policy.This provides greater flexi= bility > >>>> and does not impact performance when not enabled (controlled through= static key). > >>>> > >>>> > >>>> Abount PMC implement > >>>> =3D=3D=3D > >>>> Here, a cache switch is provided for each MEMCG(not on root). > >>>> When the user enables the cache, processes within the MEMCG will sha= re memory > >>>> through this cache. > >>>> > >>>> The cache pool is positioned before the PCP. All order0 page release= d by > >>>> processes in MEMCG will be released to the cache pool first, and whe= n memory > >>>> is requested, it will also be prioritized to be obtained from the ca= che pool. > >>>> > >>>> `memory.cache` is the sole entry point for controlling PMC, here are= some > >>>> nested keys to control PMC: > >>>> 1. "enable=3D[y|n]" to enable or disable targeted MEMCG's cache > >>>> 2. "keys=3Dnid=3D%d,watermark=3D%u,reaper_time=3D%u,limit=3D%u" = to control already > >>>> enabled PMC's behavior. > >>>> a) `nid` to targeted a node to change it's key. or else all no= de. > >>>> b) The `watermark` is used to control cache behavior, caching = only when > >>>> zone free pages above the zone's high water mark + this wat= ermark is > >>>> exceeded during memory release. (unit byte, default 50MB, > >>>> min 10MB per-node-all-zone) > >>>> c) `reaper_time` to control reaper gap, if meet, reaper all ca= che in this > >>>> MEMCG(unit us, default 5s, 0 is disable.) > >>>> d) `limit` is to limit the maximum memory used by the cache po= ol(unit bytes, > >>>> default 100MB, max 500MB per-node-all-zone) > >>>> > >>>> Performance > >>>> =3D=3D=3D > >>>> PMC is based on MEMCG and requires performance measurement through t= he > >>>> sharing of complex workloads between application processes. > >>>> Therefore, at the moment, we unable to provide a better testing solu= tion > >>>> for this patchset. > >>>> > >>>> Here is the internal testing situation we provide, using the camera > >>>> application as an example. (1-NODE-1-ZONE-8GRAM) > >>>> > >>>> Test Case: Capture in rear portrait HDR mode > >>>> 1. Test mode: rear portrait HDR mode. This scene needs more than 800= M ram > >>>> which memory types including dmabuf(470M), PSS(150M) and APU(20= 0M) > >>>> 2. Test steps: take a photo, then click thumbnail to view the full i= mage > >>>> > >>>> The overall performance benefit from click shutter button to showing= whole > >>>> image improves 500ms, and the total slowpath cost of all camera thre= ads reduced > >>>> from 958ms to 495ms. > >>>> Especially for the shot2shot in this mode, the preview dealy of each= frame have > >>>> a significant improve. > >>> Hello Huan, > >>> > >>> thank you for sharing your work. > >> thanks > >>> Some high-level thoughts: > >>> 1) Naming is hard, but it took me quite a while to realize that you'r= e talking > >> Haha, sorry for my pool english > >>> about free memory. Cache is obviously an overloaded term, but per-mem= cg-cache > >>> can mean absolutely anything (pagecache? cpu cache? ...), so maybe it= 's not > >> Currently, my idea is that all memory released by processes under memc= g > >> will go into the `cache`, > >> > >> and the original attributes will be ignored, and can be freely request= ed > >> by processes under memcg. > >> > >> (so, dma-buf\page cache\heap\driver, so on). Maybe named PMP more > >> friendly? :) > >> > >>> the best choice. > >>> 2) Overall an idea to have a per-memcg free memory pool makes sense t= o me, > >>> especially if we talk 2MB or 1GB pages (or order > 0 in general). > >> I like it too :) > >>> 3) You absolutely have to integrate the reclaim mechanism with a gene= ric > >>> memory reclaim mechanism, which is driven by the memory pressure. > >> Yes, I all think about it. > >>> 4) You claim a ~50% performance win in your workload, which is a lot.= It's not > >>> clear to me where it's coming from. It's hard to believe the page all= ocation/release > >>> paths are taking 50% of the cpu time. Please, clarify. > >> Let me describe it more specifically. In our test scenario, we have 8G= B > >> of RAM, and our camera application > >> > >> has a complex set of algorithms, with a peak memory requirement of up = to > >> 3GB. > >> > >> Therefore, in a multi-application background scenario, starting the > >> camera and taking photos will create a > >> > >> very high memory pressure. In this scenario, any released memory will = be > >> quickly used by other processes (such as file pages). > >> > >> So, during the process of switching from camera capture to preview, > >> DMA-BUF memory will be released, > >> > >> while the memory used for the preview algorithm will be simultaneously > >> requested. > >> > >> We need to take a lot of slow path routes to obtain enough memory for > >> the preview algorithm, and it seems that the > >> > >> just released DMA-BUF memory does not provide much help. > >> > > Hi Huan, > HI T.J. > > > > I find this part surprising. Assuming the dmabuf memory doesn't first > > go into a page pool (used for some buffers, not all) and actually does > Actually, when PMC enabled, we let page free avoid free into page pool. > > get freed synchronously with fput, this would mean it gets sucked up > > by other supposedly background processes before it can be allocated by > > the preview process. I thought the preview process was the one most > > desperate for memory? You mention file pages, but where is this > > newly-freed memory actually going if not to the preview process? My > This was discovered through the meminfo observation program. > When the dma-buf is released, there is a noticeable increase in cache. > > This may be triggered by pagecache when loading the algorithm model. > > Additionally, the algorithm heap memory cannot benefit from the release > of the dma-buf. > I believe this is related to the migratetype. The stack/heap cannot > obtain priority access to > the dma-buf memory released by the kernel.(HIGHUSER_MOVABLE) > > So, PMC break it, share each memory. Even if it's incorrect :)(If my > understanding of the > fragmentation issue is incorrect, please correct me.) > Oh that would make sense, but then the memory *is* going to your preview process just not in the form you were hoping for. So model loading and your heap allocations were fighting for memory, probably thrashing the file pages? I guess it's more important to get the heap allocations done first for performance for your app, and I think I can understand how PMC would give a sort of priority to those over the file pages during the preview transition. Ok. Sorry I don't have an opinion on this part yet if that's what's happening. > > initial reaction was the same as Roman's that the PMC should be hooked > > up to reclaim instead of depending on the reaper. But I think this > > might suggest that wouldn't work because the system is under such high > > memory pressure that it'd be likely reclaim would have emptied the > > PMCs before the preview process could use it. > The point you raised is indeed very likely to happen, as there is immense > memory pressure. > Currently, we only open the PMC when the application is in the foreground= , > and close it when it goes to the background. > It is indeed unnecessary to drain the PMC when the application is in the > foreground, > and a longer reaper timeout would be more useful.(Thanks for the > flexibility provided by memcg.) > > > > One more thing I find odd is that for this to work a significant > > portion of your dmabuf pages would have to be order 0, but we're > > talking about a ~500M buffer. Does whatever exports this buffer not > > try to use higher order pages like here? > Yes, actually our heap configured order 8 4 0, but In our practical > application and observation processes, > it is often difficult to meet the high-order memory allocation, so > falling back to order 0 is the most common. > Therefore, for our MID_ORDER allocation, we use LOW_ORDER_GFP. > Just like the testing scenario I mentioned earlier, with 8GB of RAM and > the camera peaking at around 3GB, > > the fragmentation at this point will cause most of the DMA-BUF > allocations to fall back to order 0. > The use of PMC is for real-world, high-load applications. I don't think > it's very practical for regular applications. Got it, thanks. > > Thanks > HY > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree= /drivers/dma-buf/heaps/system_heap.c?h=3Dv6.9#n54 > > > > Thanks! > > -T.J. > > > >> But using PMC (let's call it that for now), we are able to quickly mee= t > >> the memory needs of the subsequent preview process > >> > >> with the just released DMA-BUF memory, without having to go through th= e > >> slow path, resulting in a significant performance improvement. > >> > >> (of course, break migrate type may not good.) > >> > >>> There are a lot of other questions, and you highlighted some of them = below > >>> (and these are indeed right questions to ask), but let's start with s= omething. > >>> > >>> Thanks > >> Thanks > >>