From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09BC2C4332F for ; Wed, 8 Nov 2023 22:10:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6EBEF8D00C7; Wed, 8 Nov 2023 17:10:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69C5A8D0073; Wed, 8 Nov 2023 17:10:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53B7B8D00C7; Wed, 8 Nov 2023 17:10:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3E5C38D0073 for ; Wed, 8 Nov 2023 17:10:39 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 143A1120210 for ; Wed, 8 Nov 2023 22:10:39 +0000 (UTC) X-FDA: 81436182198.26.FB11618 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 53C4C4000C for ; Wed, 8 Nov 2023 22:10:37 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ygx13piD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.172 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699481437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q07dVhyskdcQzQ0zJmGgGIEowrRlgCK5hZIQ8KSGCtA=; b=ZnxwPIUXdS91KsLFus+Ur4QzJXqUGGQRYUlO67FS9wkGBtIsc9W85PdygCATMugNSjchdH FaiUjJpUCA6RcGM064MpNo5WwAl8JPdlS2zNOM1hAfeFmExI9bacfOQfKlg39ZqSTodfoc qxqGjvG6iL9/ZqOprvFOtk1gfjrIyd4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ygx13piD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.172 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699481437; a=rsa-sha256; cv=none; b=DPDpoXYa1wUQMKxP/T4eiD7nwPXMLTKEVYaXQhzgkTVHklTtYbjC4RoU2jWn5P5yFSiLOJ eBxgo7A/dLxM63HW90LA7d2NKjF8T+FCkMH4E0n+FpEcP4qcnHvyioRVCzU+/wF6sVKQ+f dphZcMbI6OCUvIMsjseA18nrJc4xk40= Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-41cdc669c5eso30541cf.1 for ; Wed, 08 Nov 2023 14:10:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1699481436; x=1700086236; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Q07dVhyskdcQzQ0zJmGgGIEowrRlgCK5hZIQ8KSGCtA=; b=ygx13piD14mc306AHo40Q618XctqEfRewAZbt2bAYwehXTSllX9+fq891xvRoMaNom +CmUEn4uEPPSGlZwCbMZBCglrGUgssZ1BSnvXqko94IM/E6n9yDtRgbkDv1Q5/VWaaM8 LyRSGjQvELGETLeJn5tk5tFXB8AZH0NpuyqR3qTtnfINg8ZDhs+2ggpykt1z89gE3duR 9CBrb1ZRv1JCSC6qjaiRoceiN2zdSfT7N704XnN6IpPVXjT6TPbrnACyYA+Wqm9npQq1 k/vfeWjKBrBuLiZQ8olGENp/alRLo7r2CTJ7G1eFhHnAkKL8QRvdX57o9tXCdErVEbMH xSpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699481436; x=1700086236; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Q07dVhyskdcQzQ0zJmGgGIEowrRlgCK5hZIQ8KSGCtA=; b=vG+URsgHIzPCzLwDbHRw+W07n9DEAOgPiPPgSKw3sSfoDXkxyHChfwrQ7ScFSmhjbN qHA77Z+cOKI61VbkV5+HjykG7X+JCG7ASBpGXZpJ4c0uEYlJkC6g+Il68x3P0+3lTToL 1bZWP+7HCQn3ZOtUn94aiC3breRyqFM6tHC8lItBpOTEkfriNIUCFaTHHrsVGAVxxjFF Y/5pj/LhC1RVKwD9WJkCXBRvshk9nnrT1rNJTBA9XF021ZvvGmLnMf5NOvs4fLwFxjXa 2I0PfQ6X/YOgTMRrwBuW0SfuXQN05j+nir71x0wIxzIyF6YXL3S28zKMtn6JFNxdCHQJ kuYQ== X-Gm-Message-State: AOJu0YxgQt3vcRrKoBxE27/1Zxkpyk/qj0hV9/4UvE9u8mS+9k/osmMb fgJLnOZLPhDgOwz8hklyjminOyXC6iBpMquUMlK0CA== X-Google-Smtp-Source: AGHT+IFwdS0M345UBfrF/ZcX3miuqvoTqzqLIQmYW8Knj2DW7MTLUapu/stnh9ZQ0l3pxLD3Vya9vC+OZsyh7ou3hPE= X-Received: by 2002:a05:622a:588d:b0:41c:bfd9:b990 with SMTP id fh13-20020a05622a588d00b0041cbfd9b990mr115917qtb.17.1699481436189; Wed, 08 Nov 2023 14:10:36 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Yu Zhao Date: Wed, 8 Nov 2023 14:09:58 -0800 Message-ID: Subject: Re: high kswapd CPU usage with symmetrical swap in/out pattern with multi-gen LRU To: Jaroslav Pulchart Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Igor Raits , Daniel Secik , Charan Teja Kalla Content-Type: multipart/mixed; boundary="0000000000004ef1d40609ab5b5e" X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 53C4C4000C X-Stat-Signature: kgmfqfen7juo9g1iwq9yjxzj93xoy6s1 X-HE-Tag: 1699481437-42419 X-HE-Meta: U2FsdGVkX18/FwcyD5Fpt9puG+Fi3QgP2YHtm7ifnnpt9d3QOKQ/ZuFQVZzf8ewTLErZUvOlr90Bj8Bw9HGBvK2H2+yDBBPAwSOgpJj5ItVLjMS/cBNTR/txSn38otth0JrZYqhyNIRsUW7xjaWL27JZUVxP2hsfUTqUXRhUVjgdspud6fc2NaRV+M4/ahMUWouuuzaX7iNwiLU+f0sLQzrmuhaI++TgXKFPrkgJPLlRCaNKFpAN3fB+kVWi4jc4nXvbbxqIxupKK9O5UvoBa0VTlEMkbikmIK6apxM7Z9c/Y5MwdJQ0cT81POsoXu1BCddo95TthewNzgE/cEzWGT/Idh83cyBUcaHAJ6DsjZ9pSPOcGnm/I4eSd207dn+YchrNEzAO36tR1XlP+77yQdD8LuhZ0DVQpGep6w7qP0tdXUfCKyrOkqPA/YAQbgvyBlNK5Stk1rL+zs25YHPgVNvyT/u/W6hJb+B5Y4EDTxXqPhjW/nj71WUszJ8RfXPIzVSmY62dRRd9EB08HRq4FRR8t1K67OkRyynb8T3Hs+VeMaN/A8CwOJp7f/6Khy1qhDifHN5irnUFY3Cl0Hp5PaiesznLlsARI5CAc3M+PyX3npvOIsHfDdPJuev0xJMTg7q8OrK3P1B1JyoWWUWKqEoMp89ykjzr6hSUWrx/J6mViY0DLONeJClbRCXx3lkRGn3DOns1S+Ec7NJkgmLzGQN7VszehRo/M1yFypaVi+5XYBd1w6aiigfAT0PmcYuZz1KC6hIbbw6gEdjCoTR9G+nbZI1sjlDFM3ShezVuA+25pMefy+DomHWxk8CwMi1bkCNbwHWINBx6YyDJv4w6VFDTZwxf2jlDGfSQS6OhF/Arz7Keyn00XHMBIZdjabMD4JIgI28RDwwppsm93aVq093JPN0wzZFz/CzgJCpep67pmhtCbxev2mruV8pgg6i7xD2qV2Sq26A/lUOD+bh ifDCcdTE 4LmvsZPeX9iDILt2Cad6hJBe4Q07efdw1RKhIHvJQnqigl2KHdkrHMQgPXnmirV+9Rqg17uUfn4LwLM5lLCuER79M7m9rj4Zgec/DeBBLCtE+eekbUSu4HCJ6Ki53eO++n2Zz0cOB/FF5XicELcaPWiJaWpSNLJUHput1M7rBdUWY5nfsZKn5GiKVrDhr0bSuho/NZWtT/vsuAEXnL/7kNA6fewqsZb0SPD8qCQ4sRgPaNikzXkf/6FTedVmFlPANpI1Dd1xFeOv2nLnRla2bZKnY8/Un+aOcg9UdJSG8kjhPk6Bo0/6qUV24KMWlE7cIIKUdipgEKqNtvdJzD5vPX6Tcf/ne7iK4tLnIQZJ1QQ7ObN7vJEYscHTqERYMFh0hYeoXudvTa0X1h8ln1fko5tVeKGRa5uEzqigCnKN48tVIXFI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --0000000000004ef1d40609ab5b5e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Nov 8, 2023 at 12:04=E2=80=AFPM Jaroslav Pulchart wrote: > > > > > Hi Jaroslav, > > Hi Yu Zhao > > thanks for response, see answers inline: > > > > > On Wed, Nov 8, 2023 at 6:35=E2=80=AFAM Jaroslav Pulchart > > wrote: > > > > > > Hello, > > > > > > I would like to report to you an unpleasant behavior of multi-gen LRU > > > with strange swap in/out usage on my Dell 7525 two socket AMD 74F3 > > > system (16numa domains). > > > > Kernel version please? > > 6.5.y, but we saw it sooner as it is in investigation from 23th May > (6.4.y and maybe even the 6.3.y). v6.6 has a few critical fixes for MGLRU, I can backport them to v6.5 for you if you run into other problems with v6.6. > > > Symptoms of my issue are > > > > > > /A/ if mult-gen LRU is enabled > > > 1/ [kswapd3] is consuming 100% CPU > > > > Just thinking out loud: kswapd3 means the fourth node was under memory = pressure. > > > > > top - 15:03:11 up 34 days, 1:51, 2 users, load average: 23.34, > > > 18.26, 15.01 > > > Tasks: 1226 total, 2 running, 1224 sleeping, 0 stopped, 0 z= ombie > > > %Cpu(s): 12.5 us, 4.7 sy, 0.0 ni, 82.1 id, 0.0 wa, 0.4 hi, > > > 0.4 si, 0.0 st > > > MiB Mem : 1047265.+total, 28382.7 free, 1021308.+used, 767.6 = buff/cache > > > MiB Swap: 8192.0 total, 8187.7 free, 4.2 used. 25956.7 = avail Mem > > > ... > > > 765 root 20 0 0 0 0 R 98.3 0.0 > > > 34969:04 kswapd3 > > > ... > > > 2/ swap space usage is low about ~4MB from 8GB as swap in zram (was > > > observed with swap disk as well and cause IO latency issues due to > > > some kind of locking) > > > 3/ swap In/Out is huge and symmetrical ~12MB/s in and ~12MB/s out > > > > > > > > > /B/ if mult-gen LRU is disabled > > > 1/ [kswapd3] is consuming 3%-10% CPU > > > top - 15:02:49 up 34 days, 1:51, 2 users, load average: 23.05, > > > 17.77, 14.77 > > > Tasks: 1226 total, 1 running, 1225 sleeping, 0 stopped, 0 z= ombie > > > %Cpu(s): 14.7 us, 2.8 sy, 0.0 ni, 81.8 id, 0.0 wa, 0.4 hi, > > > 0.4 si, 0.0 st > > > MiB Mem : 1047265.+total, 28378.5 free, 1021313.+used, 767.3 = buff/cache > > > MiB Swap: 8192.0 total, 8189.0 free, 3.0 used. 25952.4 = avail Mem > > > ... > > > 765 root 20 0 0 0 0 S 3.6 0.0 > > > 34966:46 [kswapd3] > > > ... > > > 2/ swap space usage is low (4MB) > > > 3/ swap In/Out is huge and symmetrical ~500kB/s in and ~500kB/s out > > > > > > Both situations are wrong as they are using swap in/out extensively, > > > however the multi-gen LRU situation is 10times worse. > > > > From the stats below, node 3 had the lowest free memory. So I think in > > both cases, the reclaim activities were as expected. > > I do not see a reason for the memory pressure and reclaims. This node > has the lowest free memory of all nodes (~302MB free) that is true, > however the swap space usage is just 4MB (still going in and out). So > what can be the reason for that behaviour? The best analogy is that refuel (reclaim) happens before the tank becomes empty, and it happens even sooner when there is a long road ahead (high order allocations). > The workers/application is running in pre-allocated HugePages and the > rest is used for a small set of system services and drivers of > devices. It is static and not growing. The issue persists when I stop > the system services and free the memory. Yes, this helps. Also could you attach /proc/buddyinfo from the moment you hit the problem? > > > Could I ask for any suggestions on how to avoid the kswapd utilizatio= n > > > pattern? > > > > The easiest way is to disable NUMA domain so that there would be only > > two nodes with 8x more memory. IOW, you have fewer pools but each pool > > has more memory and therefore they are less likely to become empty. > > > > > There is a free RAM in each numa node for the few MB used in > > > swap: > > > NUMA stats: > > > NUMA nodes: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 > > > MemTotal: 65048 65486 65486 65486 65486 65486 65486 65469 65486 > > > 65486 65486 65486 65486 65486 65486 65424 > > > MemFree: 468 601 1200 302 548 1879 2321 2478 1967 2239 1453 2417 > > > 2623 2833 2530 2269 > > > the in/out usage does not make sense for me nor the CPU utilization b= y > > > multi-gen LRU. > > > > My questions: > > 1. Were there any OOM kills with either case? > > There is no OOM. The memory usage is not growing nor the swap space > usage, it is still a few MB there. > > > 2. Was THP enabled? > > Both situations with enabled and with disabled THP. My suspicion is that you packed the node 3 too perfectly :) And that might have triggered a known but currently a low priority problem in MGLRU. I'm attaching a patch for v6.6 and hoping you could verify it for me in case v6.6 by itself still has the problem? > > MGLRU might have spent the extra CPU cycles just to void OOM kills or > > produce more THPs. > > > > If disabling the NUMA domain isn't an option, I'd recommend: > > Disabling numa is not an option. However we are now testing a setup > with -1GB in HugePages per each numa. > > > 1. Try the latest kernel (6.6.1) if you haven't. > > Not yet, the 6.6.1 was released today. > > > 2. Disable THP if it was enabled, to verify whether it has an impact. > > I try disabling THP without any effect. Gochat. Please try the patch with MGLRU and let me know. Thanks! (Also CC Charan @ Qualcomm who initially reported the problem that ended up with the attached patch.) --0000000000004ef1d40609ab5b5e Content-Type: application/octet-stream; name="0001-mm-mglru-curb-kswapd-overshooting-high-wmarks.patch" Content-Disposition: attachment; filename="0001-mm-mglru-curb-kswapd-overshooting-high-wmarks.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_loqbbvwz0 RnJvbSBhMTg4MTY5ZDI2YjJkNDBmZTBhOTEzOTM3NjFjZjIyOTI5ODQ1NDVjIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBZdSBaaGFvIDx5dXpoYW9AZ29vZ2xlLmNvbT4KRGF0ZTogV2Vk LCA4IE5vdiAyMDIzIDE0OjU2OjU4IC0wNzAwClN1YmplY3Q6IFtQQVRDSF0gbW0vbWdscnU6IGN1 cmIga3N3YXBkIG92ZXJzaG9vdGluZyBoaWdoIHdtYXJrcwoKU2lnbmVkLW9mZi1ieTogWXUgWmhh byA8eXV6aGFvQGdvb2dsZS5jb20+Ci0tLQogbW0vdm1zY2FuLmMgfCA0MCArKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKystLS0tLS0tCiAxIGZpbGUgY2hhbmdlZCwgMzMgaW5zZXJ0aW9u cygrKSwgNyBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9tbS92bXNjYW4uYyBiL21tL3Ztc2Nh bi5jCmluZGV4IDZmMTMzOTRiMTEyZS4uZGMwYmQyY2MyN2UwIDEwMDY0NAotLS0gYS9tbS92bXNj YW4uYworKysgYi9tbS92bXNjYW4uYwpAQCAtNTM0MSwyMCArNTM0MSw0NyBAQCBzdGF0aWMgbG9u ZyBnZXRfbnJfdG9fc2NhbihzdHJ1Y3QgbHJ1dmVjICpscnV2ZWMsIHN0cnVjdCBzY2FuX2NvbnRy b2wgKnNjLCBib29sCiAJcmV0dXJuIHRyeV90b19pbmNfbWF4X3NlcShscnV2ZWMsIG1heF9zZXEs IHNjLCBjYW5fc3dhcCwgZmFsc2UpID8gLTEgOiAwOwogfQogCi1zdGF0aWMgdW5zaWduZWQgbG9u ZyBnZXRfbnJfdG9fcmVjbGFpbShzdHJ1Y3Qgc2Nhbl9jb250cm9sICpzYykKK3N0YXRpYyB1bnNp Z25lZCBsb25nIGdldF9ucl90b19yZWNsYWltKHN0cnVjdCBscnV2ZWMgKmxydXZlYywgc3RydWN0 IHNjYW5fY29udHJvbCAqc2MpCiB7CisJaW50IGk7CisJdW5zaWduZWQgbG9uZyBucl90b19yZWNs YWltOworCiAJLyogZG9uJ3QgYWJvcnQgbWVtY2cgcmVjbGFpbSB0byBlbnN1cmUgZmFpcm5lc3Mg Ki8KIAlpZiAoIXJvb3RfcmVjbGFpbShzYykpCiAJCXJldHVybiAtMTsKIAotCXJldHVybiBtYXgo c2MtPm5yX3RvX3JlY2xhaW0sIGNvbXBhY3RfZ2FwKHNjLT5vcmRlcikpOworCW5yX3RvX3JlY2xh aW0gPSBtYXgoc2MtPm5yX3RvX3JlY2xhaW0sIGNvbXBhY3RfZ2FwKHNjLT5vcmRlcikpOworCWlm IChzYy0+bnJfcmVjbGFpbWVkID49IG5yX3RvX3JlY2xhaW0pCisJCXJldHVybiAwOworCisJLyog ZG9uJ3QgYWJvcnQgZGlyZWN0IHJlY2xhaW0gdG8gYXZvaWQgcHJlbWF0dXJlIE9PTSAqLworCWlm ICghY3VycmVudF9pc19rc3dhcGQoKSkKKwkJcmV0dXJuIG5yX3RvX3JlY2xhaW07CisKKwkvKiBh Ym9ydCBvbmx5IGlmIGFsbCBlbGlnaWJsZSB6b25lcyBhcmUgYmFsYW5jZWQgKi8KKwlmb3IgKGkg PSAwOyBpIDw9IHNjLT5yZWNsYWltX2lkeDsgaSsrKSB7CisJCXVuc2lnbmVkIGxvbmcgd21hcms7 CisJCXN0cnVjdCB6b25lICp6b25lID0gbHJ1dmVjX3BnZGF0KGxydXZlYyktPm5vZGVfem9uZXMg KyBpOworCisJCWlmICghbWFuYWdlZF96b25lKHpvbmUpKQorCQkJY29udGludWU7CisKKwkJaWYg KHN5c2N0bF9udW1hX2JhbGFuY2luZ19tb2RlICYgTlVNQV9CQUxBTkNJTkdfTUVNT1JZX1RJRVJJ TkcpCisJCQl3bWFyayA9IHdtYXJrX3BhZ2VzKHpvbmUsIFdNQVJLX1BST01PKTsKKwkJZWxzZQor CQkJd21hcmsgPSBoaWdoX3dtYXJrX3BhZ2VzKHpvbmUpOworCisJCWlmICghem9uZV93YXRlcm1h cmtfb2tfc2FmZSh6b25lLCBzYy0+b3JkZXIsIHdtYXJrLCBzYy0+cmVjbGFpbV9pZHgpKQorCQkJ cmV0dXJuIG5yX3RvX3JlY2xhaW07CisJfQorCisJcmV0dXJuIGkgPiBzYy0+cmVjbGFpbV9pZHgg PyAwIDogbnJfdG9fcmVjbGFpbTsKIH0KIAogc3RhdGljIGJvb2wgdHJ5X3RvX3Nocmlua19scnV2 ZWMoc3RydWN0IGxydXZlYyAqbHJ1dmVjLCBzdHJ1Y3Qgc2Nhbl9jb250cm9sICpzYykKIHsKIAls b25nIG5yX3RvX3NjYW47CiAJdW5zaWduZWQgbG9uZyBzY2FubmVkID0gMDsKLQl1bnNpZ25lZCBs b25nIG5yX3RvX3JlY2xhaW0gPSBnZXRfbnJfdG9fcmVjbGFpbShzYyk7CiAJaW50IHN3YXBwaW5l c3MgPSBnZXRfc3dhcHBpbmVzcyhscnV2ZWMsIHNjKTsKIAogCS8qIGNsZWFuIGZpbGUgZm9saW9z IGFyZSBtb3JlIGxpa2VseSB0byBleGlzdCAqLwpAQCAtNTM3Niw3ICs1NDAzLDcgQEAgc3RhdGlj IGJvb2wgdHJ5X3RvX3Nocmlua19scnV2ZWMoc3RydWN0IGxydXZlYyAqbHJ1dmVjLCBzdHJ1Y3Qg c2Nhbl9jb250cm9sICpzYykKIAkJaWYgKHNjYW5uZWQgPj0gbnJfdG9fc2NhbikKIAkJCWJyZWFr OwogCi0JCWlmIChzYy0+bnJfcmVjbGFpbWVkID49IG5yX3RvX3JlY2xhaW0pCisJCWlmIChzYy0+ bnJfcmVjbGFpbWVkID49IGdldF9ucl90b19yZWNsYWltKGxydXZlYywgc2MpKQogCQkJYnJlYWs7 CiAKIAkJY29uZF9yZXNjaGVkKCk7CkBAIC01NDM3LDcgKzU0NjQsNiBAQCBzdGF0aWMgdm9pZCBz aHJpbmtfbWFueShzdHJ1Y3QgcGdsaXN0X2RhdGEgKnBnZGF0LCBzdHJ1Y3Qgc2Nhbl9jb250cm9s ICpzYykKIAlzdHJ1Y3QgbHJ1X2dlbl9mb2xpbyAqbHJ1Z2VuOwogCXN0cnVjdCBtZW1fY2dyb3Vw ICptZW1jZzsKIAljb25zdCBzdHJ1Y3QgaGxpc3RfbnVsbHNfbm9kZSAqcG9zOwotCXVuc2lnbmVk IGxvbmcgbnJfdG9fcmVjbGFpbSA9IGdldF9ucl90b19yZWNsYWltKHNjKTsKIAogCWJpbiA9IGZp cnN0X2JpbiA9IGdldF9yYW5kb21fdTMyX2JlbG93KE1FTUNHX05SX0JJTlMpOwogcmVzdGFydDoK QEAgLTU0NzAsNyArNTQ5Niw3IEBAIHN0YXRpYyB2b2lkIHNocmlua19tYW55KHN0cnVjdCBwZ2xp c3RfZGF0YSAqcGdkYXQsIHN0cnVjdCBzY2FuX2NvbnRyb2wgKnNjKQogCiAJCXJjdV9yZWFkX2xv Y2soKTsKIAotCQlpZiAoc2MtPm5yX3JlY2xhaW1lZCA+PSBucl90b19yZWNsYWltKQorCQlpZiAo c2MtPm5yX3JlY2xhaW1lZCA+PSBnZXRfbnJfdG9fcmVjbGFpbShscnV2ZWMsIHNjKSkKIAkJCWJy ZWFrOwogCX0KIApAQCAtNTQ4MSw3ICs1NTA3LDcgQEAgc3RhdGljIHZvaWQgc2hyaW5rX21hbnko c3RydWN0IHBnbGlzdF9kYXRhICpwZ2RhdCwgc3RydWN0IHNjYW5fY29udHJvbCAqc2MpCiAKIAlt ZW1fY2dyb3VwX3B1dChtZW1jZyk7CiAKLQlpZiAoc2MtPm5yX3JlY2xhaW1lZCA+PSBucl90b19y ZWNsYWltKQorCWlmICghaXNfYV9udWxscyhwb3MpKQogCQlyZXR1cm47CiAKIAkvKiByZXN0YXJ0 IGlmIHJhY2VkIHdpdGggbHJ1X2dlbl9yb3RhdGVfbWVtY2coKSAqLwotLSAKMi40Mi4wLjg2OS5n ZWEwNWYyMDgzZC1nb29nCgo= --0000000000004ef1d40609ab5b5e--