From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53A67C30658 for ; Wed, 3 Jul 2024 03:23:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79A8B6B007B; Tue, 2 Jul 2024 23:23:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 722E66B0082; Tue, 2 Jul 2024 23:23:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 575086B0083; Tue, 2 Jul 2024 23:23:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 30B186B007B for ; Tue, 2 Jul 2024 23:23:09 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id ABDB7C072F for ; Wed, 3 Jul 2024 03:23:08 +0000 (UTC) X-FDA: 82296995256.03.DBD76D1 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.18]) by imf26.hostedemail.com (Postfix) with ESMTP id A547E140002 for ; Wed, 3 Jul 2024 03:23:05 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VSMkl8xT; spf=pass (imf26.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.18 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719976974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FiZioEddncn6fp4txnA/bltw266e9RZTC/wzJPJZf2Q=; b=j/6BnwasT+0X694WYxcfRm11SUn+EYcanMWCcmUM0lSiej3tyw+VFYZBOLZfkbT16/GxCz 2uSvGZc5PJjQQ+H3lwBzvJIMyC0oFOmAWBDr7CkrN0/46UG/ME0oY/mvqz3kGGZ3k/vw2O JS+26gluV1/zOr+70rxWzjcO0jd32HY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VSMkl8xT; spf=pass (imf26.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.18 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719976974; a=rsa-sha256; cv=none; b=p+uhQz7c/Yg7clnlmuDhduaTkfFhzf8dyG+nNRg1Xsgkxz88otcwazr0ngOSbnBU3rPo9i G6JsxfSRe4xgqGrmdGflqtttFEahrc3w3jJYu2ugyCxeF1tGmqxBERNs0TbIl7D26JuCJN wUa12Juiz+QSE9jgTAKgXKpxFPfGDiQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1719976985; x=1751512985; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version:content-transfer-encoding; bh=np6tzASXqDJC5abYmKjfp4vxDEY1qJbui861TfuTIBc=; b=VSMkl8xTvu0KJG+sMIe84lgshz5EkTyVXOZ79ISqgio9ZhUqTbZmNnHV te9AVpidFleQaixnG0/c9VZGKW14ML2uaSy6bWZ8IHFALzfs7Q9+MNQXx ysGHSgh2fyEA2Rus0HX9ibh1aYP57qz0V1YdiQMmfVKBbcDEDSTevP+ME Og8ZFlWksbFWJQb6x5oHIgIxFJL0Vh7aKoZuFZrnv0j9Gf0hDYrwryN7d 3T3SBkJr4aypbaUvrJEENJ/L7ex1MkjjQ0yWQXh6ob4D4UpKt9a4saWV8 +vo3ByRtPt+T/DZyizbkQD/Dv9iNR9jY27mwwRtxroTYzodkB+YguEypA Q==; X-CSE-ConnectionGUID: CUzm/sjZS0+oAtVZVpeVww== X-CSE-MsgGUID: tYc/EBqpSvW5ZJCIWPciNg== X-IronPort-AV: E=McAfee;i="6700,10204,11121"; a="17320380" X-IronPort-AV: E=Sophos;i="6.09,181,1716274800"; d="scan'208";a="17320380" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa110.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Jul 2024 20:23:04 -0700 X-CSE-ConnectionGUID: HoXcJMZgQAiJYajvn8eVOw== X-CSE-MsgGUID: 94HpaDjdS3itUkIQPGb2VQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,180,1716274800"; d="scan'208";a="46164890" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa010-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Jul 2024 20:23:02 -0700 From: "Huang, Ying" To: Yafang Shao Cc: Andrew Morton , linux-mm@kvack.org, Matthew Wilcox , David Rientjes , Mel Gorman Subject: Re: [PATCH] mm: Enable setting -1 for vm.percpu_pagelist_high_fraction to set the minimum pagelist In-Reply-To: (Yafang Shao's message of "Wed, 3 Jul 2024 10:13:43 +0800") References: <20240701142046.6050-1-laoar.shao@gmail.com> <20240701195143.7e8d597abc14b255f3bc4bcd@linux-foundation.org> <874j98noth.fsf@yhuang6-desk2.ccr.corp.intel.com> <87wmm3kzmy.fsf@yhuang6-desk2.ccr.corp.intel.com> Date: Wed, 03 Jul 2024 11:21:11 +0800 Message-ID: <87plrvkvo8.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A547E140002 X-Stat-Signature: rkt4g1cf355x19q6g3kqqo7azuex6umd X-Rspam-User: X-HE-Tag: 1719976985-170678 X-HE-Meta: U2FsdGVkX1+3SLkQI7xl4NuEpQ8BYLo1va0FOnYvGqGLqfm3rnOUznBiHA8r4wdBfg0TR+etjftkha2Nfk9nneoS6WawZLhFS4Ld36bAiqtTiwlOwqMrgGJDRd/9R7N7lUjLh48wlhpufG9q3mTZOeIDEVUNNhI9NeKA+v7+YrsLUfzODWfI4GeQnUMOOwP81V0AJnYUxf2mCp3cm8vrPd5tCGVYatRVTYxPwp/c2PsSsU6oWsGr3NvRo6OAFmtFJMJHL/ySDDghNKrLOhcX/ATzqK7tbW+632iH8Z6E5ayq1uVew1D+anvgRUeoCI2J8AloLIZ8vBG2o5vrJ/cOkkyn/jAkNBy2xxUjVy/GMy6LDc7eLUXCgIbBFkxO3GxsYztRMa+NkgKOLI8OX9M8iDU7icrD6RhTPbqhm2skQYho5ViwbQOz1dSQrXNKUe64bqDhL9kCkskxu9rzdSQCvFfap0+eMMSH7T0pTfFkQ3omLN/qUliG713dfGYbUTUXDlHDSDQ+C/q2nykJYvLc6fnZo5cKuh0/8VQllVyi1cNo0PRPbSulwqZfoDkbyTG49JiDfRLdJHLIm7ydE46fDQJod7dG3qKotKhK2Dimb99yw9x7xg+tF8bLB5FVJoZ5NYGbTQPS/tV+8nc5uvffg8qwHRzFAUMGtqP50FhFzCcGCDex4oWvr5UtpYaQM/CdZ/uxOQl5B8srWGKdU23UfTDTzvDxTNPTPq+OUlYNQ8PsQzEyCrhmq9kkZP0yzW++fNuv0z8NyOshY41hMNOV3fdioB0+etnGQOif5pnECptdZozXDAiT19z4Bam/XZ8Rpy50MIfp1ec4dU+IN2dPFxJyT4ag04npNWLUi/sRQ81Awba+zECRHm+dw/iefAwifOD0b53VLjNLtVYn/0Wb4mgmNnCLaB9pj6QOd7jTR6vkYhaphgGfvh1ALuLZrE8KlT65qFHjnF0GRwlFL5U JyGtciTm LSufnenInnRg0Q7AEsILiCV1WvJ67BSanYQ/xFgsJjl7ql9yqcBaIThxy7B7sSpzAIdl8ZGBJytbXs4aXGjLk8M+QYz1zvGnZI9OQWBCQ8Xtx+8lsJv/TslhJy41C04JIVQC3p+agydX6f26FkVxexkVRhnWkQAGjl/syse+W0ZMUZgZAEALYqR1F9kXLkifXXFAeJ9gzSVvhPattppCsIF0eSO4eHwejB229VNfXEuBzG15BVo8zzIBWVdOFuAiLo83gjiD4XyArnJPi4y91h5/q6lW2kTSVusqs+eqgXvqw7K2hhqvgLmS1Lc3P+93TN9eOA0pjcal8s5dsH67bqKSDE1UKl9YWNMFbxQmY1JLrvPS4gjmHZ5Q5EQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Yafang Shao writes: > On Wed, Jul 3, 2024 at 9:57=E2=80=AFAM Huang, Ying = wrote: >> >> Yafang Shao writes: >> >> > On Tue, Jul 2, 2024 at 5:10=E2=80=AFPM Huang, Ying wrote: >> >> >> >> Yafang Shao writes: >> >> >> >> > On Tue, Jul 2, 2024 at 10:51=E2=80=AFAM Andrew Morton wrote: >> >> >> >> >> >> On Mon, 1 Jul 2024 22:20:46 +0800 Yafang Shao wrote: >> >> >> >> >> >> > Currently, we're encountering latency spikes in our container en= vironment >> >> >> > when a specific container with multiple Python-based tasks exits= . These >> >> >> > tasks may hold the zone->lock for an extended period, significan= tly >> >> >> > impacting latency for other containers attempting to allocate me= mory. >> >> >> >> >> >> Is this locking issue well understood? Is anyone working on it? A >> >> >> reasonably detailed description of the issue and a description of = any >> >> >> ongoing work would be helpful here. >> >> > >> >> > In our containerized environment, we have a specific type of contai= ner >> >> > that runs 18 processes, each consuming approximately 6GB of RSS. Th= ese >> >> > processes are organized as separate processes rather than threads d= ue >> >> > to the Python Global Interpreter Lock (GIL) being a bottleneck in a >> >> > multi-threaded setup. Upon the exit of these containers, other >> >> > containers hosted on the same machine experience significant latency >> >> > spikes. >> >> > >> >> > Our investigation using perf tracing revealed that the root cause of >> >> > these spikes is the simultaneous execution of exit_mmap() by each of >> >> > the exiting processes. This concurrent access to the zone->lock >> >> > results in contention, which becomes a hotspot and negatively impac= ts >> >> > performance. The perf results clearly indicate this contention as a >> >> > primary contributor to the observed latency issues. >> >> > >> >> > + 77.02% 0.00% uwsgi [kernel.kallsyms] >> >> > [k] mmput =E2=96=92 >> >> > - 76.98% 0.01% uwsgi [kernel.kallsyms] >> >> > [k] exit_mmap =E2=96=92 >> >> > - 76.97% exit_mmap >> >> > =E2=96=92 >> >> > - 58.58% unmap_vmas >> >> > =E2=96=92 >> >> > - 58.55% unmap_single_vma >> >> > =E2=96=92 >> >> > - unmap_page_range >> >> > =E2=96=92 >> >> > - 58.32% zap_pte_range >> >> > =E2=96=92 >> >> > - 42.88% tlb_flush_mmu >> >> > =E2=96=92 >> >> > - 42.76% free_pages_and_swap_cache >> >> > =E2=96=92 >> >> > - 41.22% release_pages >> >> > =E2=96=92 >> >> > - 33.29% free_unref_page_list >> >> > =E2=96=92 >> >> > - 32.37% free_unref_page_commit >> >> > =E2=96=92 >> >> > - 31.64% free_pcppages_bulk >> >> > =E2=96=92 >> >> > + 28.65% _raw_spin_lock >> >> > =E2=96=92 >> >> > 1.28% __list_del_entry_valid >> >> > =E2=96=92 >> >> > + 3.25% folio_lruvec_lock_irqsave >> >> > =E2=96=92 >> >> > + 0.75% __mem_cgroup_uncharge_list >> >> > =E2=96=92 >> >> > 0.60% __mod_lruvec_state >> >> > =E2=96=92 >> >> > 1.07% free_swap_cache >> >> > =E2=96=92 >> >> > + 11.69% page_remove_rmap >> >> > =E2=96=92 >> >> > 0.64% __mod_lruvec_page_state >> >> > - 17.34% remove_vma >> >> > =E2=96=92 >> >> > - 17.25% vm_area_free >> >> > =E2=96=92 >> >> > - 17.23% kmem_cache_free >> >> > =E2=96=92 >> >> > - 17.15% __slab_free >> >> > =E2=96=92 >> >> > - 14.56% discard_slab >> >> > =E2=96=92 >> >> > free_slab >> >> > =E2=96=92 >> >> > __free_slab >> >> > =E2=96=92 >> >> > __free_pages >> >> > =E2=96=92 >> >> > - free_unref_page >> >> > =E2=96=92 >> >> > - 13.50% free_unref_page_commit >> >> > =E2=96=92 >> >> > - free_pcppages_bulk >> >> > =E2=96=92 >> >> > + 13.44% _raw_spin_lock >> >> > >> >> > By enabling the mm_page_pcpu_drain() we can find the detailed stack: >> >> > >> >> > <...>-1540432 [224] d..3. 618048.023883: mm_page_pcpu_dra= in: >> >> > page=3D0000000035a1b0b7 pfn=3D0x11c19c72 order=3D0 migratetyp >> >> > e=3D1 >> >> > <...>-1540432 [224] d..3. 618048.023887: >> >> > =3D> free_pcppages_bulk >> >> > =3D> free_unref_page_commit >> >> > =3D> free_unref_page_list >> >> > =3D> release_pages >> >> > =3D> free_pages_and_swap_cache >> >> > =3D> tlb_flush_mmu >> >> > =3D> zap_pte_range >> >> > =3D> unmap_page_range >> >> > =3D> unmap_single_vma >> >> > =3D> unmap_vmas >> >> > =3D> exit_mmap >> >> > =3D> mmput >> >> > =3D> do_exit >> >> > =3D> do_group_exit >> >> > =3D> get_signal >> >> > =3D> arch_do_signal_or_restart >> >> > =3D> exit_to_user_mode_prepare >> >> > =3D> syscall_exit_to_user_mode >> >> > =3D> do_syscall_64 >> >> > =3D> entry_SYSCALL_64_after_hwframe >> >> > >> >> > The servers experiencing these issues are equipped with impressive >> >> > hardware specifications, including 256 CPUs and 1TB of memory, all >> >> > within a single NUMA node. The zoneinfo is as follows, >> >> > >> >> > Node 0, zone Normal >> >> > pages free 144465775 >> >> > boost 0 >> >> > min 1309270 >> >> > low 1636587 >> >> > high 1963904 >> >> > spanned 564133888 >> >> > present 296747008 >> >> > managed 291974346 >> >> > cma 0 >> >> > protection: (0, 0, 0, 0) >> >> > ... >> >> > ... >> >> > pagesets >> >> > cpu: 0 >> >> > count: 2217 >> >> > high: 6392 >> >> > batch: 63 >> >> > vm stats threshold: 125 >> >> > cpu: 1 >> >> > count: 4510 >> >> > high: 6392 >> >> > batch: 63 >> >> > vm stats threshold: 125 >> >> > cpu: 2 >> >> > count: 3059 >> >> > high: 6392 >> >> > batch: 63 >> >> > >> >> > ... >> >> > >> >> > The high is around 100 times the batch size. >> >> > >> >> > We also traced the latency associated with the free_pcppages_bulk() >> >> > function during the container exit process: >> >> > >> >> > 19:48:54 >> >> > nsecs : count distribution >> >> > 0 -> 1 : 0 | = | >> >> > 2 -> 3 : 0 | = | >> >> > 4 -> 7 : 0 | = | >> >> > 8 -> 15 : 0 | = | >> >> > 16 -> 31 : 0 | = | >> >> > 32 -> 63 : 0 | = | >> >> > 64 -> 127 : 0 | = | >> >> > 128 -> 255 : 0 | = | >> >> > 256 -> 511 : 148 |***************** = | >> >> > 512 -> 1023 : 334 |******************************= **********| >> >> > 1024 -> 2047 : 33 |*** = | >> >> > 2048 -> 4095 : 5 | = | >> >> > 4096 -> 8191 : 7 | = | >> >> > 8192 -> 16383 : 12 |* = | >> >> > 16384 -> 32767 : 30 |*** = | >> >> > 32768 -> 65535 : 21 |** = | >> >> > 65536 -> 131071 : 15 |* = | >> >> > 131072 -> 262143 : 27 |*** = | >> >> > 262144 -> 524287 : 84 |********** = | >> >> > 524288 -> 1048575 : 203 |************************ = | >> >> > 1048576 -> 2097151 : 284 |******************************= **** | >> >> > 2097152 -> 4194303 : 327 |******************************= ********* | >> >> > 4194304 -> 8388607 : 215 |************************* = | >> >> > 8388608 -> 16777215 : 116 |************* = | >> >> > 16777216 -> 33554431 : 47 |***** = | >> >> > 33554432 -> 67108863 : 8 | = | >> >> > 67108864 -> 134217727 : 3 | = | >> >> > >> >> > avg =3D 3066311 nsecs, total: 5887317501 nsecs, count: 1920 >> >> > >> >> > The latency can reach tens of milliseconds. >> >> > >> >> > By adjusting the vm.percpu_pagelist_high_fraction parameter to set = the >> >> > minimum pagelist high at 4 times the batch size, we were able to >> >> > significantly reduce the latency associated with the >> >> > free_pcppages_bulk() function during container exits.: >> >> > >> >> > nsecs : count distribution >> >> > 0 -> 1 : 0 | = | >> >> > 2 -> 3 : 0 | = | >> >> > 4 -> 7 : 0 | = | >> >> > 8 -> 15 : 0 | = | >> >> > 16 -> 31 : 0 | = | >> >> > 32 -> 63 : 0 | = | >> >> > 64 -> 127 : 0 | = | >> >> > 128 -> 255 : 120 | = | >> >> > 256 -> 511 : 365 |* = | >> >> > 512 -> 1023 : 201 | = | >> >> > 1024 -> 2047 : 103 | = | >> >> > 2048 -> 4095 : 84 | = | >> >> > 4096 -> 8191 : 87 | = | >> >> > 8192 -> 16383 : 4777 |************** = | >> >> > 16384 -> 32767 : 10572 |******************************= * | >> >> > 32768 -> 65535 : 13544 |******************************= **********| >> >> > 65536 -> 131071 : 12723 |******************************= ******* | >> >> > 131072 -> 262143 : 8604 |************************* = | >> >> > 262144 -> 524287 : 3659 |********** = | >> >> > 524288 -> 1048575 : 921 |** = | >> >> > 1048576 -> 2097151 : 122 | = | >> >> > 2097152 -> 4194303 : 5 | = | >> >> > >> >> > avg =3D 103814 nsecs, total: 5805802787 nsecs, count: 55925 >> >> > >> >> > After successfully tuning the vm.percpu_pagelist_high_fraction sysc= tl >> >> > knob to set the minimum pagelist high at a level that effectively >> >> > mitigated latency issues, we observed that other containers were no >> >> > longer experiencing similar complaints. As a result, we decided to >> >> > implement this tuning as a permanent workaround and have deployed it >> >> > across all clusters of servers where these containers may be deploy= ed. >> >> >> >> Thanks for your detailed data. >> >> >> >> IIUC, the latency of free_pcppages_bulk() during process exiting >> >> shouldn't be a problem? >> > >> > Right. The problem arises when the process holds the lock for too >> > long, causing other processes that are attempting to allocate memory >> > to experience delays or wait times. >> > >> >> Because users care more about the total time of >> >> process exiting, that is, throughput. And I suspect that the zone->l= ock >> >> contention and page allocating/freeing throughput will be worse with >> >> your configuration? >> > >> > While reducing throughput may not be a significant concern due to the >> > minimal difference, the potential for latency spikes, a crucial metric >> > for assessing system stability, is of greater concern to users. Higher >> > latency can lead to request errors, impacting the user experience. >> > Therefore, maintaining stability, even at the cost of slightly lower >> > throughput, is preferable to experiencing higher throughput with >> > unstable performance. >> > >> >> >> >> But the latency of free_pcppages_bulk() and page allocation in other >> >> processes is a problem. And your configuration can help it. >> >> >> >> Another choice is to change CONFIG_PCP_BATCH_SCALE_MAX. In that way, >> >> you have a normal PCP size (high) but smaller PCP batch. I guess that >> >> may help both latency and throughput in your system. Could you give = it >> >> a try? >> > >> > Currently, our kernel does not include the CONFIG_PCP_BATCH_SCALE_MAX >> > configuration option. However, I've observed your recent improvements >> > to the zone->lock mechanism, particularly commit 52166607ecc9 ("mm: >> > restrict the pcp batch scale factor to avoid too long latency"), which >> > has prompted me to experiment with manually setting the >> > pcp->free_factor to zero. While this adjustment provided some >> > improvement, the results were not as significant as I had hoped. >> > >> > BTW, perhaps we should consider the implementation of a sysctl knob as >> > an alternative to CONFIG_PCP_BATCH_SCALE_MAX? This would allow users >> > to more easily adjust it. >> >> If you cannot test upstream behavior, it's hard to make changes to >> upstream. Could you find a way to do that? > > I'm afraid I can't run an upstream kernel in our production environment :( > Lots of code changes have to be made. Understand. Can you find a way to test upstream behavior, not upstream kernel exactly? Or test the upstream kernel but in a similar but not exactly production environment. >> IIUC, PCP high will not influence allocate/free latency, PCP batch will. > > It seems incorrect. > Looks at the code in free_unref_page_commit() : > > if (pcp->count >=3D high) { > free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), > pcp, pindex); > } > > And nr_pcp_free() : > min_nr_free =3D batch; > max_nr_free =3D high - batch; > > batch =3D clamp_t(int, pcp->free_count, min_nr_free, max_nr_free); > return batch; > > The 'batch' is not a fixed value but changed dynamically, isn't it ? Sorry, my words were confusing. For 'batch', I mean the value of the "count" parameter of free_pcppages_bulk() actually. For example, if we change CONFIG_PCP_BATCH_SCALE_MAX, we restrict that. >> Your configuration will influence PCP batch via configuration PCP high. >> So, it may be reasonable to find a way to adjust PCP batch directly. >> But, we need practical requirements and test methods first. >> -- Best Regards, Huang, Ying