From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BF5CC433FE for ; Fri, 18 Nov 2022 01:32:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C4A76B0078; Thu, 17 Nov 2022 20:32:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 94D876B007B; Thu, 17 Nov 2022 20:32:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C8558E0001; Thu, 17 Nov 2022 20:32:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6EA6D6B0078 for ; Thu, 17 Nov 2022 20:32:25 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 491F3C0367 for ; Fri, 18 Nov 2022 01:32:25 +0000 (UTC) X-FDA: 80144837850.25.977AE6A Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf29.hostedemail.com (Postfix) with ESMTP id EE2E2120012 for ; Fri, 18 Nov 2022 01:32:24 +0000 (UTC) Received: by mail-pg1-f202.google.com with SMTP id y9-20020a63e249000000b00476aad2eb4aso2215038pgj.17 for ; Thu, 17 Nov 2022 17:32:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=X3e5jC7h9awIX99sxD8I+UBHtQfnz5RCLC9SbQikEwk=; b=cYD4amwn9y7cVKu1KFLKIR+R7gkkTAzxoiXhMIjSQhqTUAY97l9LzXUjrnmOugun12 5b1mGMaL/q4dOAhilD9MYRtgAehXqJbJpN9eiebCdzG40lKX4YEHdmrDmxugYhQvQCQG XeWec0SEhu+sh5D4UJ8EVl5/0z30fVW/LdTA1Cqg123iXMVjNaNwXvjjpS8UhOc+pard lnV63W7Sq/nfo7MO759O51PxMY2Opxo4Erg3Y43QDTqQnl7PUJ7x2G1M/SThuT2bQLBM hwBFiFfPEsGpj5YoKTdUw093QJhnX7IkgoPA6lhwgaKQ/Zm3a3OTZaaOGIHl0WS68YxG S/NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=X3e5jC7h9awIX99sxD8I+UBHtQfnz5RCLC9SbQikEwk=; b=dxPIZE0m4v60U1i9UMFaAEyE1RaT7yuoLabsLfjHIKaFrdF6bV3aJpHTEnS7DLlpWp 6fM/WLu5rHH0Z0A56V/s4DEYYrNL91rTIQBUMz8nWlk5kH+aXstJ/Vkh+yFXK4K76r6q HfjZEV8g7pMaJbUxjs46ksR+2gFmudU0XZpSGWbJxz+Z3Ugv1sqHXErf0ANXZB34gp6H 5ZQuvOQSmYWdC4U3tA57ui/HKz9yaW74R4ZVtYn77pil9ssFwQ+oXmQT++VKOq+EVZnQ 5uwgulEwHWoARZ4ELKuMuNJIcbQ744eoyYdmmWkkreLpeDc0e2m3dx3eR+sB2U04YVZa bPog== X-Gm-Message-State: ANoB5plyQROdRmshM+MIB1iwk83KY50zPuJPw00rWQRrdBWuZORGZUlP gDc/pgd9mvFxuMvNWbjR/C0n0f3niNudtw== X-Google-Smtp-Source: AA0mqf4tj6saXkqYA0YZhtn/oY5bNj2bQ9d04xxijeQEOaTiv5fMTv1MFNyOduhF4GuR0ckTqtnnmJKFewTsBA== X-Received: from yjqkernel.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1837]) (user=jiaqiyan job=sendgmr) by 2002:a62:148f:0:b0:560:ee47:81e1 with SMTP id 137-20020a62148f000000b00560ee4781e1mr5597741pfu.63.1668735143995; Thu, 17 Nov 2022 17:32:23 -0800 (PST) Date: Thu, 17 Nov 2022 17:31:57 -0800 In-Reply-To: <20221118013157.1333622-1-jiaqiyan@google.com> Mime-Version: 1.0 References: <20221118013157.1333622-1-jiaqiyan@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221118013157.1333622-3-jiaqiyan@google.com> Subject: [PATCH v7 2/2] mm/khugepaged: recover from poisoned file-backed memory From: Jiaqi Yan To: kirill.shutemov@linux.intel.com, kirill@shutemov.name, shy828301@gmail.com, tongtiangen@huawei.com, akpm@linux-foundation.org Cc: tony.luck@intel.com, naoya.horiguchi@nec.com, linmiaohe@huawei.com, juew@google.com, jiaqiyan@google.com, linux-mm@kvack.org, osalvador@suse.de Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668735145; a=rsa-sha256; cv=none; b=TSG8Fwrt2Yhi2u7KOFKa7VbGYMsTKBcKXzjkhJZZ+55SfP7REp20U04BHr0QE0lUL7rbce UKCnX9HFy9iN1R94ifA5otHAM6HdVgnu73oYdLX/gqEnjxCDU1mmYP0tUkHYK7RVsAh9+1 F8S6qlegHNr6ClWtAvmxRklyLEQEnCU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cYD4amwn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3p-B2YwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3p-B2YwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668735145; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X3e5jC7h9awIX99sxD8I+UBHtQfnz5RCLC9SbQikEwk=; b=RKMyh1TWNGfBj/tRov6v8tNpVHSUHdhJHwDfPeFBCCJitWCzBMPTsfacl7bYEmXJf6GK6p xeyEV9+e2OUzzSp1mtv0LU9tbKQcvydQ7xuU/TeDn4Iln1/IcSzH5tAjVyNIxbFdRoLCYM 75UtxCaISRcFmhZD3cErkgjYSAOUr7I= Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cYD4amwn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3p-B2YwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3p-B2YwgKCGYNMEUMcERKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jiaqiyan.bounces.google.com X-Rspam-User: X-Stat-Signature: fx4cwus4agr8e67mp1hud9hctqt1qryb X-Rspamd-Queue-Id: EE2E2120012 X-Rspamd-Server: rspam11 X-HE-Tag: 1668735144-257369 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make collapse_file roll back when copying pages failed. More concretely: - extract copying operations into a separate loop - postpone the updates for nr_none until both scanning and copying succeeded - postpone joining small xarray entries until both scanning and copying succeeded - postpone the update operations to NR_XXX_THPS until both scanning and copying succeeded - for non-SHMEM file, roll back filemap_nr_thps_inc if scan succeeded but copying failed Tested manually: 0. Enable khugepaged on system under test. Mount tmpfs at /mnt/ramdisk. 1. Start a two-thread application. Each thread allocates a chunk of non-huge memory buffer from /mnt/ramdisk. 2. Pick 4 random buffer address (2 in each thread) and inject uncorrectable memory errors at physical addresses. 3. Signal both threads to make their memory buffer collapsible, i.e. calling madvise(MADV_HUGEPAGE). 4. Wait and then check kernel log: khugepaged is able to recover from poisoned pages by skipping them. 5. Signal both thread to inspect their buffer contents and make sure no data corruption. Signed-off-by: Jiaqi Yan --- mm/khugepaged.c | 74 ++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 29 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c1f225327bc05..434423e5e35da 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1776,7 +1776,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, struct collapse_control *cc) { struct address_space *mapping = file->f_mapping; - struct page *hpage; + struct page *hpage, *page, *tmp; pgoff_t index = 0, end = start + HPAGE_PMD_NR; LIST_HEAD(pagelist); XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); @@ -1821,7 +1821,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, xas_set(&xas, start); for (index = start; index < end; index++) { - struct page *page = xas_next(&xas); + page = xas_next(&xas); VM_BUG_ON(index != xas.xa_index); if (is_shmem) { @@ -2003,10 +2003,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } nr = thp_nr_pages(hpage); - if (is_shmem) - __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); - else { - __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); + if (!is_shmem) { filemap_nr_thps_inc(mapping); /* * Paired with smp_mb() in do_dentry_open() to ensure @@ -2017,21 +2014,10 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, smp_mb(); if (inode_is_open_for_write(mapping->host)) { result = SCAN_FAIL; - __mod_lruvec_page_state(hpage, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); goto xa_locked; } } - - if (nr_none) { - __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); - /* nr_none is always 0 for non-shmem. */ - __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); - } - - /* Join all the small entries into a single multi-index entry */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, hpage); xa_locked: xas_unlock_irq(&xas); xa_unlocked: @@ -2044,20 +2030,34 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, try_to_unmap_flush(); if (result == SCAN_SUCCEED) { - struct page *page, *tmp; - /* * Replacing old pages with new one has succeeded, now we - * need to copy the content and free the old pages. + * attempt to copy the contents. */ index = start; - list_for_each_entry_safe(page, tmp, &pagelist, lru) { + list_for_each_entry(page, &pagelist, lru) { while (index < page->index) { clear_highpage(hpage + (index % HPAGE_PMD_NR)); index++; } - copy_highpage(hpage + (page->index % HPAGE_PMD_NR), - page); + if (copy_highpage_mc(hpage + (page->index % HPAGE_PMD_NR), page)) { + result = SCAN_COPY_MC; + break; + } + index++; + } + while (result == SCAN_SUCCEED && index < end) { + clear_highpage(hpage + (index % HPAGE_PMD_NR)); + index++; + } + } + + if (result == SCAN_SUCCEED) { + /* + * Copying old pages to huge one has succeeded, now we + * need to free the old pages. + */ + list_for_each_entry_safe(page, tmp, &pagelist, lru) { list_del(&page->lru); page->mapping = NULL; page_ref_unfreeze(page, 1); @@ -2065,12 +2065,23 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, ClearPageUnevictable(page); unlock_page(page); put_page(page); - index++; } - while (index < end) { - clear_highpage(hpage + (index % HPAGE_PMD_NR)); - index++; + + xas_lock_irq(&xas); + if (is_shmem) + __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); + else + __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); + + if (nr_none) { + __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); + /* nr_none is always 0 for non-shmem. */ + __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); } + /* Join all the small entries into a single multi-index entry. */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, hpage); + xas_unlock_irq(&xas); SetPageUptodate(hpage); page_ref_add(hpage, HPAGE_PMD_NR - 1); @@ -2086,8 +2097,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, unlock_page(hpage); hpage = NULL; } else { - struct page *page; - /* Something went wrong: roll back page cache changes */ xas_lock_irq(&xas); if (nr_none) { @@ -2121,6 +2130,13 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, xas_lock_irq(&xas); } VM_BUG_ON(nr_none); + /* + * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. + * This undo is not needed unless failure is due to SCAN_COPY_MC. + */ + if (!is_shmem && result == SCAN_COPY_MC) + filemap_nr_thps_dec(mapping); + xas_unlock_irq(&xas); hpage->mapping = NULL; -- 2.38.1.584.g0f3c55d4c2-goog