From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 824C8EE49A3 for ; Tue, 22 Aug 2023 15:30:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB208280043; Tue, 22 Aug 2023 11:30:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D613794001B; Tue, 22 Aug 2023 11:30:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2A0E280043; Tue, 22 Aug 2023 11:30:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id ADC1E94001B for ; Tue, 22 Aug 2023 11:30:46 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 65B27C03AB for ; Tue, 22 Aug 2023 15:30:46 +0000 (UTC) X-FDA: 81152128092.13.18D9254 Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 71EB540030 for ; Tue, 22 Aug 2023 15:30:44 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1nNMUj+J; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692718244; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9jLe3aS6oHv7LXqfot3AlJx+jHhWMRNhwylusvC3Hno=; b=caEArG5jn9iHgl2/2STEFZOhzGyFfwRAVeU4mgwMO9DXA4tkm4/PICqwEfUcQa2AHSU3xo XgcoP8o+U0Q2DTSj0kkT9tEbwhv3vsGKh5W5HJU39gDrCKP3h7qDXjFSkxq3Hv4SE8AV/o Qe9oyrGSthtgqSf6p1/vsj59lmEQWBE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1nNMUj+J; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of yosryahmed@google.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692718244; a=rsa-sha256; cv=none; b=RzRc2Hq19xso3BfEgGj2L/MmO/d1skc7OO6F2hjFyYzHxZp41zQ6dgqVrRWxkm+Sx+sdXl l0mdZRr+fFYtL2bwkwkCZ/paT4LetnixkvsOQhAqDHDK0ZON0nHNXbUE7JiUsWA9VFO61L RFjN7g0MgK3KnVENjSHnmMnd32OgdME= Received: by mail-lj1-f172.google.com with SMTP id 38308e7fff4ca-2bcb89b4767so38089491fa.3 for ; Tue, 22 Aug 2023 08:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692718242; x=1693323042; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9jLe3aS6oHv7LXqfot3AlJx+jHhWMRNhwylusvC3Hno=; b=1nNMUj+JgJ/msq8cZiBCa8v/TbP0d9ORcQHmlu9M6RZjcKQX2uBVnok+in04BF0Vm5 7+i96DocFBoQbs6OYoLWk1N4vsYFad0F6MfUJIG6SBHbJn0sQGeFtVZZ4a2HyReq1YRf d1TeFz/A+7H3zimfpnsaJZuOoO2shESI+2b9Gme610irXIm45tk9VHkHUKTzgBhc6t1K pyXfs8KOOM4WsfihKZLR7DK60sB7CDhIg+OqQtx2aglkouLVsfFCETc9lv81Od8a3kCf 58bhwp9+lhlyWY/vmP6/SneKCtCSGaja1ZfZKeEXYeicRuuo9GGCGgUmY/1IWpRVBMaj LIrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692718242; x=1693323042; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9jLe3aS6oHv7LXqfot3AlJx+jHhWMRNhwylusvC3Hno=; b=EgC2wZSGSjm8wwCfTpcnicAU0aGMGkNc1spjz20now1ujQlikzakyyGBR56gxAbV9b KK6f1hg/2VWnuKQbEpgOImZT3S/nbWqaNhq2O76SJ5BQ49aykKbZJty3143Ib3PJGiYS Z6DrZI6tKkHEC7sxyuCRmIZaFHyTbCZ/QjAEWaIDn9h4GFdE/AvVr9//HQNHPu29o+Zr NeLB4xnzyj/OUVk2jO+iarG5YLOls/qAYU9WeGP6+kPjs35Xy4+WvEU/GA4pcy1vkzum qfqcU04CzvEmx3tS9blN/epoIiyyVaSm2AR2thDj8zdC8wqZiDfa5Fm4NrZmYS8dyUIn vxIw== X-Gm-Message-State: AOJu0Yx+TkKvyLs5hIw62hlzF+dNqA8DX1krFLA4BbrsvzpKuOzmVQhm RRJMLbMLLRLx6w0CTUNnIqMbEWNW5EFDO/Pf3PhzOA== X-Google-Smtp-Source: AGHT+IHj5NEzUxDIG8nZNuwvZFRAKokOiiGQkNLlSwSEoa9qnMaznGTV9FDbXoxI5OM5wbSxZjgLunlmQRlGB+LZ6lE= X-Received: by 2002:a2e:8891:0:b0:2b9:f3b4:6808 with SMTP id k17-20020a2e8891000000b002b9f3b46808mr7836633lji.29.1692718242268; Tue, 22 Aug 2023 08:30:42 -0700 (PDT) MIME-Version: 1.0 References: <20230821205458.1764662-1-yosryahmed@google.com> <20230821205458.1764662-4-yosryahmed@google.com> In-Reply-To: From: Yosry Ahmed Date: Tue, 22 Aug 2023 08:30:05 -0700 Message-ID: Subject: Re: [PATCH 3/3] mm: memcg: use non-unified stats flushing for userspace reads To: Michal Hocko Cc: Andrew Morton , Johannes Weiner , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: multipart/mixed; boundary="0000000000008ab2d0060384ad3e" X-Rspamd-Queue-Id: 71EB540030 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: of5r4947kkbeq1yk3a5rrzq97zrzs7ie X-HE-Tag: 1692718244-784771 X-HE-Meta: U2FsdGVkX1+pzN9WYkcyBZPJjG1tRCnJ2ztE9eRR0MiTrojvjLor3NT3YoYy/eUTcZBZrsTPVLfeZU6Bw6NCXWUdk3yWh648NBSw6MNQMd1s8MLLO6+/4V5MSdzGyKcHlfqE4csFJrqcpLg+ufgDHr43xyWhHoWPgUIGm5z7+zeVJwV0xW8yCHh+njPO/GT4bk5XKXKBNXtXrhB+iYUEu8dHRjS6bVE3CiOeB7yhR22yQsyUfp29CUzZZMq75xrLmUc1R3wtfthd+FJaq7tsmNnuMuKTWUbej29GcpqIVfgpVoMD6Iex0M7PyX5wa0aSm6flYyidIhFC6qDaW9R2yaSJSpkQ9IypBdzlTFLT3UHnTEjTyYmFOVgsHlTEVJObYVvLRX1Rpjsufyeigbusexx95X+QXAFXhY3J02z7+cbyXBVGApzdCx1Sk+3wJWQwyEQDhw2XWKYFmrSQG/bKSA1UrCep/zHDGnDtrHzCgaKdOzxKRwtPuFC2hUI8N7KHax3/WQZjGHlwzgwO93iQQulVd/Gujr0SJP3++BPwLRjmTSOwH1K0QNmYNI8DdVpJAn1y0NndKpKrtfOeHFcd/KJyGQuAysEhpyoQTQeillgwigICt86323xzXMAOqkknPqPHh7MevHJyB1NwTrMW8kWL3UhXDDDbk76Fu7DqWxwAwZoKcDyQe6m4bB1RoY5deWxdyMBx6j9HYr4R16o3j+cJK1ZKZb2NljK4YO0yYmFHFArsm2EHda3kRYDwvxMkw9KfF/jLahKbaqbyGQ/wlu/taXRuKQEm9GbpyUTT8XF0dADS46yE5/rd7EXa7cGv0svP1jwO1T6u0Nn7eYeGTLhYbwybWraxNAtVYDZRjisGpT/JSA8eWpWGwY8dQDK+4ubMJpzLT0RQjIpHVmZvy7IKaJ7jUhWwDiOLeUpSPhb5FDuTwc8luN2BvAO5/Hq0MSkaVy5kz8oozy0+7UO NSrSrCrS uPyqXawZ7njcSMWzjn/8pmM2EpzbXtAjU4lwsJpzQHqVKo0ciXFF1k7Euyul4PvxPOvJH2FvyLHRKE36DG0fUmwTWMqA7LWoFyno/ElHNU0M5eIeW8IR/cTU46WJbSS6lq8ytWvD2fzETltvn7UK2s3IgK1tQ3asg9SBglZDDWaGARC0xCBAalyalowfWgvA4VDO6hTQAKD7vhzxd5JIGLkEo59exth4bAILLOADrgrzc4ef5zM/itqyepzBWBBA9392A2HE7Wskroz7DghjbXdEfHwmh0EvvqsmVrlLX4fQ72015qokhD+nVeRjS6zWt40Q0iYUjQ/aeFvZODeN89O2QBTgO7rAaB/2gB/iOBS31qSzC2rOEXRAli9xO9/DNVPNOuqZeVodkB99DHCi1VZFS0PZR9pQFVKyhwQ+zgF/k8pVhTFyYK1xT3lMx1ynfsbE7RYkWGqiMMaVOHumvUWRSXcbaKJr+Jgm1KJSskOymgnu5DoVJhHlZNu4uQ/6MNa+4n/llPpn+smcbNuE3eNwUFH2V4aVXZ1tVO/raDdcRDwGiUF9SLhHDM2qgcTNRym3q X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --0000000000008ab2d0060384ad3e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Aug 22, 2023 at 2:06=E2=80=AFAM Michal Hocko wrot= e: > > On Mon 21-08-23 20:54:58, Yosry Ahmed wrote: > > Unified flushing allows for great concurrency for paths that attempt to > > flush the stats, at the expense of potential staleness and a single > > flusher paying the extra cost of flushing the full tree. > > > > This tradeoff makes sense for in-kernel flushers that may observe high > > concurrency (e.g. reclaim, refault). For userspace readers, stale stats > > may be unexpected and problematic, especially when such stats are used > > for critical paths such as userspace OOM handling. Additionally, a > > userspace reader will occasionally pay the cost of flushing the entire > > hierarchy, which also causes problems in some cases [1]. > > > > Opt userspace reads out of unified flushing. This makes the cost of > > reading the stats more predictable (proportional to the size of the > > subtree), as well as the freshness of the stats. Since userspace reader= s > > are not expected to have similar concurrency to in-kernel flushers, > > serializing them among themselves and among in-kernel flushers should b= e > > okay. > > > > This was tested on a machine with 256 cpus by running a synthetic test > > The script that creates 50 top-level cgroups, each with 5 children (250 > > leaf cgroups). Each leaf cgroup has 10 processes running that allocate > > memory beyond the cgroup limit, invoking reclaim (which is an in-kernel > > unified flusher). Concurrently, one thread is spawned per-cgroup to rea= d > > the stats every second (including root, top-level, and leaf cgroups -- > > so total 251 threads). No regressions were observed in the total runnin= g > > time; which means that non-unified userspace readers are not slowing > > down in-kernel unified flushers: > > I have to admit I am rather confused by cgroup_rstat_flush (and > cgroup_rstat_flush_locked). The former says it can block but the later > doesn't ever block and even if it drops the cgroup_rstat_lock it merely > cond_rescheds or busy loops. How much of a contention and yielding can > you see with this patch? What is the worst case? How bad a random user > can make the situation by going crazy and trying to flush from many > different contexts? Userspace readers (or more generically non-unified flushers) can all collectively only block a single unified flusher at most. Specifically, one userspace reader goes to flush and holds cgroup_rstat_lock, meanwhile an in-kernel flusher (e.g. reclaim) goes and tries to flush, and spins on cgroup_rstat_lock. Other in-kernel (unified) flushers will just see another unified flusher in progress and skip. So userspace can only block a single in-kernel reclaimer. Not that it's not really that bad because: (a) As you note, cgroup_rstat_flush() does not really "block", it's cpu-bound. Even when it cond_resched()'s, it yields the lock first. So it can't really hold anyone hostage for long. (b) I assume a random user can only read their own stats, which should be a relatively small subtree, quick to flush. I am assuming a random user cannot read root's memory.stat (which is most expensive). (c) Excessive flushing doesn't really build up because there will be nothing to flush and the lock will be released very shortly after it's held. So to answer your question, I don't think a random user can really affect the system in a significant way by constantly flushing. In fact, in the test script (which I am now attaching, in case you're interested), there are hundreds of threads that are reading stats of different cgroups every 1s, and I don't see any negative effects on in-kernel flushers in this case (reclaimers). > -- > Michal Hocko > SUSE Labs --0000000000008ab2d0060384ad3e Content-Type: text/x-sh; charset="US-ASCII"; name="stress.sh" Content-Disposition: attachment; filename="stress.sh" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_llmgovtf0 IyEvYmluL2Jhc2gKCk5SX0wxX0NHUk9VUFM9NTAKTlJfTDJfQ0dST1VQU19QRVJfTDE9NQpOUl9M Ml9DR1JPVVBTPSQoKCBOUl9MMV9DR1JPVVBTICogTlJfTDJfQ0dST1VQU19QRVJfTDEgKSkKTlJf V09SS0VSU19QRVJfQ0dST1VQPTEwCldPUktFUl9NQj0xMApOUl9UT1RBTF9XT1JLRVJTPSQoKCBO Ul9XT1JLRVJTX1BFUl9DR1JPVVAgKiAoMSArIE5SX0wxX0NHUk9VUFMgKyBOUl9MMl9DR1JPVVBT KSApKQpMMV9DR1JPVVBfTElNSVRfTUI9JCgoIE5SX0wyX0NHUk9VUFNfUEVSX0wxICogTlJfV09S S0VSU19QRVJfQ0dST1VQICogV09SS0VSX01CIC8gNCApKQpUT1RBTF9NQj0kKCggTlJfVE9UQUxf V09SS0VSUyAqIFdPUktFUl9NQiApKQpUTVBGUz0kKG1rdGVtcCAtZCkKUk9PVD0iL3N5cy9mcy9j Z3JvdXAvIgpBRE1JTj0iL3N5cy9mcy9jZ3JvdXAvYWRtaW4iClpSQU1fREVWPSIvbW50L2RldnRt cGZzL3pyYW0wIgoKY2xlYW51cF9jZ3JvdXAoKSB7CiAgbG9jYWwgLXIgY2c9JDEKICBsb2NhbCAt ciBwcm9jcz0kKGNhdCAkY2cvY2dyb3VwLnByb2NzKQogIGlmIFtbIC1uICRwcm9jcyBdXTsgdGhl bgogICAga2lsbCAtS0lMTCAkcHJvY3MKICAgIHdhaXQgJHByb2NzIDI+L2Rldi9udWxsCiAgZmkK ICB3aGlsZSBbWyAtbiAkKGNhdCAkY2cvY2dyb3VwLnByb2NzKSBdXTsgZG8KICAgIHNsZWVwIDAu MQogIGRvbmUKICBybWRpciAkY2cKfQoKY2xlYW51cCgpIHsKICBmb3IgaSBpbiAkKHNlcSAkTlJf TDFfQ0dST1VQUyk7IGRvCiAgICBsMT0iJFJPT1QvcGFyZW50JGkiCiAgICBmb3IgaiBpbiAkKHNl cSAkTlJfTDJfQ0dST1VQU19QRVJfTDEpOyBkbwogICAgICBsMj0iJGwxL2NoaWxkJGoiCiAgICAg IGNsZWFudXBfY2dyb3VwICRsMgogICAgZG9uZQogICAgY2xlYW51cF9jZ3JvdXAgJGwxCiAgZG9u ZQoKICBjbGVhbnVwX2Nncm91cCAkQURNSU4KCiAgdW1vdW50ICRUTVBGUwogIHJtIC1yZiAkVE1Q RlMKCiAgc3dhcG9mZiAkWlJBTV9ERVYKICBlY2hvIDEgPiAiL3N5cy9ibG9jay96cmFtMC9yZXNl dCIKfQp0cmFwIGNsZWFudXAgSU5UIFFVSVQgRVhJVAoKcnVuX3N0YXRzX3JlYWRlcigpIHsKICBs b2NhbCAtciBjZ19ydW49JDEKICBsb2NhbCAtciBjZ19yZWFkPSQyCgogICMgcmVhZCB0aGUgc3Rh dHMgZXZlcnkgc2Vjb25kIHVudGlsIHdvcmtlcnMgYXJlIGRvbmUKICBlY2hvIDAgPiAiJGNnX3J1 bi9jZ3JvdXAucHJvY3MiCiAgd2hpbGUgW1sgJChscyAkVE1QRlMpIF1dOyBkbwogICAgY2F0ICIk Y2dfcmVhZC9tZW1vcnkuc3RhdCIgPiAvZGV2L251bGwKICAgIHNsZWVwIDEKICBkb25lCn0KCnJ1 bl93b3JrZXIoKSB7CiAgbG9jYWwgLXIgY2c9JDEKICBsb2NhbCAtciBmPSQyCgogIGVjaG8gMCA+ ICIkY2cvY2dyb3VwLnByb2NzIgogIHJtIC1yZiAiJGYiCiAgZGQgaWY9L2Rldi96ZXJvIG9mPSIk ZiIgYnM9MU0gY291bnQ9JFdPUktFUl9NQiBzdGF0dXM9bm9uZQogIGNhdCAiJGYiID4gL2Rldi9u dWxsCiAgcm0gIiRmIgp9CgojIFNldHVwIHpyYW0KZWNobyAkKChUT1RBTF9NQiA8PCAyMCkpID4g Ii9zeXMvYmxvY2svenJhbTAvZGlza3NpemUiCm1rc3dhcCAkWlJBTV9ERVYKc3dhcG9uICRaUkFN X0RFVgplY2hvICJTZXR1cCB6cmFtIGRvbmUiCgojIE1vdW50IHRtcGZzCm1vdW50IC10IHRtcGZz IG5vbmUgJFRNUEZTCgojIENyZWF0ZSBhZG1pbiBjZ3JvdXAKbWtkaXIgJEFETUlOCgojIENyZWF0 ZSB3b3JrZXIgY2dyb3Vwcywgc2V0IGxpbWl0cwplY2hvICIrbWVtb3J5IiA+ICIkUk9PVC9jZ3Jv dXAuc3VidHJlZV9jb250cm9sIgpmb3IgaSBpbiAkKHNlcSAkTlJfTDFfQ0dST1VQUyk7IGRvCiAg bDE9IiRST09UL3BhcmVudCRpIgogIG1rZGlyICRsMQogIGVjaG8gIittZW1vcnkiID4gIiRsMS9j Z3JvdXAuc3VidHJlZV9jb250cm9sIgogIGZvciBqIGluICQoc2VxICROUl9MMl9DR1JPVVBTX1BF Ul9MMSk7IGRvCiAgICBsMj0iJGwxL2NoaWxkJGoiCiAgICBta2RpciAkbDIKICBkb25lCiAgZWNo byAkKCggTDFfQ0dST1VQX0xJTUlUX01CIDw8IDIwKSkgPiAiJGwxL21lbW9yeS5tYXgiCmRvbmUK ZWNobyAiU2V0dXAgJE5SX0wxX0NHUk9VUFMgTDEgY2dyb3VwcyB3aXRoIGxpbWl0ICR7TDFfQ0dS T1VQX0xJTUlUX01CfU0gZG9uZSIKZWNobyAiRWFjaCBMMSBjZ3JvdXAgaGFzICROUl9MMl9DR1JP VVBTX1BFUl9MMSBMMiBjaGlsZHJlbiIKCiMgU3RhcnQgd29ya2VycyB0byBhbGxvY2F0ZSB0bXBm cyBtZW1vcnkKZm9yIGkgaW4gJChzZXEgJE5SX0wxX0NHUk9VUFMpOyBkbwogIGwxPSIkUk9PVC9w YXJlbnQkaSIKICBmb3IgaiBpbiAkKHNlcSAkTlJfTDJfQ0dST1VQU19QRVJfTDEpOyBkbwogICAg bDI9IiRsMS9jaGlsZCRqIgogICAgZm9yIGsgaW4gJChzZXEgJE5SX1dPUktFUlNfUEVSX0NHUk9V UCk7IGRvCiAgICAgIChydW5fd29ya2VyICIkbDIiICIkVE1QRlMvJGktJGotJGsiKSYKICAgIGRv bmUKICBkb25lCmRvbmUKCiMgU3RhcnQgc3RhdCByZWFkZXJzCihydW5fc3RhdHNfcmVhZGVyICIk QURNSU4iICIkUk9PVCIpJgpmb3IgaSBpbiAkKHNlcSAkTlJfTDFfQ0dST1VQUyk7IGRvCiAgbDE9 IiRST09UL3BhcmVudCRpIgogIChydW5fc3RhdHNfcmVhZGVyICIkQURNSU4iICIkbDEiKSYKICBm b3IgaiBpbiAkKHNlcSAkTlJfTDJfQ0dST1VQU19QRVJfTDEpOyBkbwogICAgbDI9IiRsMS9jaGls ZCRqIgogICAgIyBSYW4gc3RhdCByZWFkZXJzIGluIHRoZSBhZG1pbiBjZ3JvdXAgYXMgd2VsbCBh cyB0aGUgY2dyb3VwIGl0c2VsZgogICAgKHJ1bl9zdGF0c19yZWFkZXIgIiRBRE1JTiIgIiRsMiIp JgogICAgKHJ1bl9zdGF0c19yZWFkZXIgIiRsMiIgIiRsMiIpJgogIGRvbmUKZG9uZQoKIyBXYWl0 IGZvciB3b3JrZXJzCmVjaG8gIlJhbiAkTlJfV09SS0VSU19QRVJfQ0dST1VQIHdvcmtlcnMgcGVy IEwyIGNncm91cCwgZWFjaCBhbGxvY2F0aW5nICR7V09SS0VSX01CfU0sIHdhaXRpbmcuLiIKd2Fp dAoK --0000000000008ab2d0060384ad3e--