From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC361C433EF for ; Fri, 27 May 2022 02:26:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C1DF8D0003; Thu, 26 May 2022 22:26:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26D558D0001; Thu, 26 May 2022 22:26:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 132358D0003; Thu, 26 May 2022 22:26:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 055CA8D0001 for ; Thu, 26 May 2022 22:26:27 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CD694346D9 for ; Fri, 27 May 2022 02:26:26 +0000 (UTC) X-FDA: 79509933972.09.85A7B95 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf23.hostedemail.com (Postfix) with ESMTP id 70639140031 for ; Fri, 27 May 2022 02:26:03 +0000 (UTC) Received: by mail-pl1-f179.google.com with SMTP id n8so2997359plh.1 for ; Thu, 26 May 2022 19:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=GTmACc9N/3MLTj6poPRyGy9qcVFfRLI5Sz32EZV0z38=; b=COgXgMZDG8WkP+lglV2Qm8KiYg3OYMJ6q8zRmVVu4YCQhwnr7YHTclag8GvN6sPpqj snyshILu0mVF35uUzY3IlNV9lWALJRWgtcJCwaJ1up/VKMnneFH4I2c4Q/SuqpcT+xJS daVWNio46q8qRqPTPGJPc/nGy+VQMvmJMExBR9jsfPXJBRsunsvuvv5c2vRI0uGs/H42 MTteQDv/PoLH7V0fx1W/rcVnFrgqRC/gawvBOz1wSEkGDK7M7tYwSvQDJ1Mj4yC2oeGq NBkIf5iBnZvgvG6zmmMCX9Uo0b9ZtH+3JN2JND1LiwxJgkKye5t/48JP60vScOgML+au nF4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=GTmACc9N/3MLTj6poPRyGy9qcVFfRLI5Sz32EZV0z38=; b=ETBWw6M+HTWIXClzkgg0FYNKeUSmBjkUJzyLxIibdN7V189xD40Y8DWW9BdksHFtuG Zn7zvdL5wHwLBieIgEWO7qoCbEHpXJrVQsIDZgOprcJQQzZgT6axOmNdkkV7wcwTO5XM n5UQ1wOCCWEe+C2mPiyM2joep4Sd8ZEDNnJyNHP7AMyeqkDtxXjc5SBl8mQRN2q/DEip /7ePVhG8D9xf51wEC+hpBEia0sHWWFII7leMj8ZgpM9JrU9EhOSORVjv4cbvX0X+HA2t Hq1aeQ+hmQd2tDmozH+57qXKXbr71vQirZNXekG0igXRJ5kBF0Jk0dKZmcdCZK3i6/Tc ilEw== X-Gm-Message-State: AOAM531b+3RhfsXOCoSgZKIMuPyTTczxUpiwpxykjwFCEEU2E8oQECw3 5K5wikr+/2WR4oFTuCNFQBvwnQ== X-Google-Smtp-Source: ABdhPJwG+lmKSHcrxP6YtEa7GiL6Nr6zjRjr3nP+ihLp9L087u1RV9dVoYNbsckRgYc6uXjSvJ5U9w== X-Received: by 2002:a17:902:bf06:b0:14d:8c72:96c6 with SMTP id bi6-20020a170902bf0600b0014d8c7296c6mr40633154plb.156.1653618384636; Thu, 26 May 2022 19:26:24 -0700 (PDT) Received: from [10.255.70.20] ([139.177.225.235]) by smtp.gmail.com with ESMTPSA id j2-20020a170902690200b0015f4b7a012bsm2247113plk.251.2022.05.26.19.26.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 26 May 2022 19:26:23 -0700 (PDT) Message-ID: <27062739-74af-7deb-2486-45bb84888433@bytedance.com> Date: Fri, 27 May 2022 10:22:23 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.1 Subject: Re: Re: [PATCH 3/3] virtio_balloon: Introduce memory recover Content-Language: en-US To: "Michael S. Tsirkin" , akpm@linux-foundation.org, naoya.horiguchi@nec.com Cc: david@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, jasowang@redhat.com, virtualization@lists.linux-foundation.org, pbonzini@redhat.com, peterx@redhat.com, qemu-devel@nongnu.org References: <20220520070648.1794132-1-pizhenwei@bytedance.com> <20220520070648.1794132-4-pizhenwei@bytedance.com> <20220526151507-mutt-send-email-mst@kernel.org> From: zhenwei pi In-Reply-To: <20220526151507-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 70639140031 X-Stat-Signature: aziycmbczb1p8m7fu1yscm9ota9btpcr Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=COgXgMZD; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf23.hostedemail.com: domain of pizhenwei@bytedance.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=pizhenwei@bytedance.com X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1653618363-50512 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/27/22 03:18, Michael S. Tsirkin wrote: > On Fri, May 20, 2022 at 03:06:48PM +0800, zhenwei pi wrote: >> Introduce a new queue 'recover VQ', this allows guest to recover >> hardware corrupted page: >> >> Guest 5.MF -> 6.RVQ FE 10.Unpoison page >> / \ / >> -------------------+-------------+----------+----------- >> | | | >> 4.MCE 7.RVQ BE 9.RVQ Event >> QEMU / \ / >> 3.SIGBUS 8.Remap >> / >> ----------------+------------------------------------ >> | >> +--2.MF >> Host / >> 1.HW error >> >> The workflow: >> 1, HardWare page error occurs randomly. >> 2, host side handles corrupted page by Memory Failure mechanism, sends >> SIGBUS to the user process if early-kill is enabled. >> 3, QEMU handles SIGBUS, if the address belongs to guest RAM, then: >> 4, QEMU tries to inject MCE into guest. >> 5, guest handles memory failure again. >> >> 1-5 is already supported for a long time, the next steps are supported >> in this patch(also related driver patch): >> 6, guest balloon driver gets noticed of the corrupted PFN, and sends >> request to host side by Recover VQ FrontEnd. >> 7, QEMU handles request from Recover VQ BackEnd, then: >> 8, QEMU remaps the corrupted HVA fo fix the memory failure, then: >> 9, QEMU acks the guest side the result by Recover VQ. >> 10, guest unpoisons the page if the corrupted page gets recoverd >> successfully. >> >> Then the guest fixes the HW page error dynamiclly without rebooting. >> >> Emulate MCE by QEMU, the guest works fine: >> mce: [Hardware Error]: Machine check events logged >> Memory failure: 0x61646: recovery action for dirty LRU page: Recovered >> virtio_balloon virtio5: recovered pfn 0x61646 >> Unpoison: Unpoisoned page 0x61646 by virtio-balloon >> MCE: Killing stress:24502 due to hardware memory corruption fault at 7f5be2e5a010 >> >> The 'HardwareCorrupted' in /proc/meminfo also shows 0 kB. >> >> Signed-off-by: zhenwei pi >> --- >> drivers/virtio/virtio_balloon.c | 243 ++++++++++++++++++++++++++++ >> include/uapi/linux/virtio_balloon.h | 16 ++ >> 2 files changed, 259 insertions(+) >> >> diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c >> index f4c34a2a6b8e..f9d95d1d8a4d 100644 >> --- a/drivers/virtio/virtio_balloon.c >> +++ b/drivers/virtio/virtio_balloon.c >> @@ -52,6 +52,7 @@ enum virtio_balloon_vq { >> VIRTIO_BALLOON_VQ_STATS, >> VIRTIO_BALLOON_VQ_FREE_PAGE, >> VIRTIO_BALLOON_VQ_REPORTING, >> + VIRTIO_BALLOON_VQ_RECOVER, >> VIRTIO_BALLOON_VQ_MAX >> }; >> >> @@ -59,6 +60,12 @@ enum virtio_balloon_config_read { >> VIRTIO_BALLOON_CONFIG_READ_CMD_ID = 0, >> }; >> >> +/* the request body to commucate with host side */ >> +struct __virtio_balloon_recover { >> + struct virtio_balloon_recover vbr; >> + __virtio32 pfns[VIRTIO_BALLOON_PAGES_PER_PAGE]; >> +}; >> + > > > I don't think this idea of passing 32 bit pfns is going to fly. > What is wrong with just passing the pages normally as a s/g list? > this is what is done for the hints at the moment. > > neither should you use __virtio types for new functionality > (should all be __le), nor use __virtio for the struct name. > > Guest side sends GPA/PFN to host side by passing the pages normally as a s/g list, this is OK. But in this scenario, guest also needs to get status(recovered?corrupted?failed to recover?) of page from the host side. For a normal page(Ex, 4K), the host could return the status quite immediately. But for the 2M hugetlb of guest RAM, the host should be pending until the guest requests 512*4K to recover. Once the 2M hugepage gets recovered(or failed to recover), the host returns 512 PFNs with status to guest. There are at most 512 recover requests of a single 2M huge page. For example, the guest tries to recover a corrupted page: struct scatterlist status_sg, page_sg, *sgs[2]; sg_init_one(&status_sg, status, sizeof(*status)); sgs[0] = &status_sg; p = page_address(page); sg_init_one(&page_sg, p, PAGE_SIZE); sgs[1] = &page_sg; virtqueue_add_sgs(recover_vq, sgs, 1, 1, NULL, GFP_ATOMIC); The host handles 4K recover request on 2M hugepage, this request is pending until the full 2M huge page gets recovered(or failed). To avoid too many pending request in virt queue, I designed as this patch(should use __le), passing PFN in request body, using a single IN request only. ... >> --- a/include/uapi/linux/virtio_balloon.h >> +++ b/include/uapi/linux/virtio_balloon.h >> @@ -37,6 +37,7 @@ >> #define VIRTIO_BALLOON_F_FREE_PAGE_HINT 3 /* VQ to report free pages */ >> #define VIRTIO_BALLOON_F_PAGE_POISON 4 /* Guest is using page poisoning */ >> #define VIRTIO_BALLOON_F_REPORTING 5 /* Page reporting virtqueue */ >> +#define VIRTIO_BALLOON_F_RECOVER 6 /* Memory recover virtqueue */ >> >> /* Size of a PFN in the balloon interface. */ >> #define VIRTIO_BALLOON_PFN_SHIFT 12 > > Please get this feature recorded in the spec with the virtio TC. > They will also ask you to supply minimal documentation. > Sure! By the way, this feature depends on the memory&&memory-failure mechanism, what about sending the change of spec to virtio TC after Andrew and Naoya ack? -- zhenwei pi