From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15E67C433E7 for ; Wed, 2 Sep 2020 14:56:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7F002078E for ; Wed, 2 Sep 2020 14:56:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=soleen.com header.i=@soleen.com header.b="MnyrJeZW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7F002078E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 864F86B0085; Wed, 2 Sep 2020 10:27:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8166E6B0087; Wed, 2 Sep 2020 10:27:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7041D6B0088; Wed, 2 Sep 2020 10:27:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0115.hostedemail.com [216.40.44.115]) by kanga.kvack.org (Postfix) with ESMTP id 5805F6B0085 for ; Wed, 2 Sep 2020 10:27:23 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1976A181AEF0B for ; Wed, 2 Sep 2020 14:27:23 +0000 (UTC) X-FDA: 77218349166.30.level31_200e9c8270a1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id A4A6B180B3C8B for ; Wed, 2 Sep 2020 14:27:22 +0000 (UTC) X-HE-Tag: level31_200e9c8270a1 X-Filterd-Recvd-Size: 6166 Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Sep 2020 14:27:22 +0000 (UTC) Received: by mail-ed1-f68.google.com with SMTP id n13so5122264edo.10 for ; Wed, 02 Sep 2020 07:27:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6OOrr3ankKVwxGGNXToxy8jZJ4HxhbpceY6NaN50tLo=; b=MnyrJeZWxNanxx+EeVE/LB7iSafmhLfk8zHRAK7voOONuaQFQEp8FWuNryGn4WlrvK YiaPAiGMbL43ABgId2yZdBMutBWeYG80LAvLktNAUeReCRCAZKhDQWdZkH1TmA9HM64N 97zfvWaoPMpVrPbgWPW1wqg0HWi1CpGicWQ3wAt15K5BXPdJSkrWz8XKHqA6CzYunlJJ Ku9rD8Q0Iw5rjGvtQN99M3OpzxqqRu/OB1JoIRpo8DeZHDJ+oSIz3AjII/VHvkAxOYtx 6GwaI7NjueIhnz9V2rhOZmvj6CETY7E8dvX0Gzl7zMZbpyVd2hEnFU55UESMBey8sU0O dHtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6OOrr3ankKVwxGGNXToxy8jZJ4HxhbpceY6NaN50tLo=; b=cttYY77T3cund4lPwhFe8LxJMq9wJfAkYQE/tjFBrgdvOLqHtSt6lQro67YF7TXmDC p1r11kA7l03wGhZCKU28ifu1EdFUgGtFkKvtccNcncEefWB6xDhAL0WTyNHOnNp1Sgrm hHyUd0ilg9+RNKYW9IUgm5OCPDw7ZWhVfwQb7/c6kOO9RWCGXu3kPDiHv4lJiAm70rHr /yZdot5kjQY53R9uPULzzdOUkV6j4rZ/BREu1GwoodDiTY8lis1VKUgIrbIlEFKRAzT+ CpWUPQZUCyeKAKr6m+A5dWHA7looj40wvmwpleubK7jTzESakKgvarMKruQXsS1ds9fY fSvw== X-Gm-Message-State: AOAM530hhuOkbwkzFu/3L6OPnDxQqF2JQDQeFVUfUOq2rnK6nQNvSc4l N2+/mxf1ZrfBJefWV/paRJuH/3csutqMRiU4MBOSLfoO+m/ErA== X-Google-Smtp-Source: ABdhPJzNVkeIY5bugg1hnssT2HP/1orWqIA4ATC5qddKjKxgYam+9CmN4LlIuehn0Ap7B63fkyLj5KnhXd2jZIkpPrM= X-Received: by 2002:a05:6402:1d0f:: with SMTP id dg15mr352913edb.342.1599056840905; Wed, 02 Sep 2020 07:27:20 -0700 (PDT) MIME-Version: 1.0 References: <20200901124615.137200-1-pasha.tatashin@soleen.com> <20200902140851.GJ4617@dhcp22.suse.cz> In-Reply-To: <20200902140851.GJ4617@dhcp22.suse.cz> From: Pavel Tatashin Date: Wed, 2 Sep 2020 10:26:44 -0400 Message-ID: Subject: Re: [PATCH] mm/memory_hotplug: drain per-cpu pages again during memory offline To: Michal Hocko Cc: LKML , Andrew Morton , linux-mm Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: A4A6B180B3C8B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Sep 2, 2020 at 10:08 AM Michal Hocko wrote: > > On Tue 01-09-20 08:46:15, Pavel Tatashin wrote: > > There is a race during page offline that can lead to infinite loop: > > a page never ends up on a buddy list and __offline_pages() keeps > > retrying infinitely or until a termination signal is received. > > > > Thread#1 - a new process: > > > > load_elf_binary > > begin_new_exec > > exec_mmap > > mmput > > exit_mmap > > tlb_finish_mmu > > tlb_flush_mmu > > release_pages > > free_unref_page_list > > free_unref_page_prepare > > set_pcppage_migratetype(page, migratetype); > > // Set page->index migration type below MIGRATE_PCPTYPES > > > > Thread#2 - hot-removes memory > > __offline_pages > > start_isolate_page_range > > set_migratetype_isolate > > set_pageblock_migratetype(page, MIGRATE_ISOLATE); > > Set migration type to MIGRATE_ISOLATE-> set > > drain_all_pages(zone); > > // drain per-cpu page lists to buddy allocator. > > It is not really clear to me how we could have passed > has_unmovable_pages at this stage when the page is not PageBuddy. Is > this because you are using Movable Zones? Yes, we hot-remove memory from the movable zone. > > > > > Thread#1 - continue > > free_unref_page_commit > > migratetype = get_pcppage_migratetype(page); > > // get old migration type > > list_add(&page->lru, &pcp->lists[migratetype]); > > // add new page to already drained pcp list > > > > Thread#2 > > Never drains pcp again, and therefore gets stuck in the loop. > > > > The fix is to try to drain per-cpu lists again after > > check_pages_isolated_cb() fails. > > But this means that the page is not isolated and so it could be reused > for something else. No? The page is in a movable zone, has zero references, and the section is isolated (i.e. set_pageblock_migratetype(page, MIGRATE_ISOLATE);) is set. The page should be offlinable, but it is lost in a pcp list as that list is never drained again after the first failure to migrate all pages in the range. > > > Signed-off-by: Pavel Tatashin > > Cc: stable@vger.kernel.org > > --- > > mm/memory_hotplug.c | 9 +++++++++ > > 1 file changed, 9 insertions(+) > > > > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c > > index e9d5ab5d3ca0..d6d54922bfce 100644 > > --- a/mm/memory_hotplug.c > > +++ b/mm/memory_hotplug.c > > @@ -1575,6 +1575,15 @@ static int __ref __offline_pages(unsigned long start_pfn, > > /* check again */ > > ret = walk_system_ram_range(start_pfn, end_pfn - start_pfn, > > NULL, check_pages_isolated_cb); > > + /* > > + * per-cpu pages are drained in start_isolate_page_range, but if > > + * there are still pages that are not free, make sure that we > > + * drain again, because when we isolated range we might > > + * have raced with another thread that was adding pages to > > + * pcp list. > > + */ > > + if (ret) > > + drain_all_pages(zone); > > } while (ret); > > > > /* Ok, all of our target is isolated. > > -- > > 2.25.1 > > > > -- > Michal Hocko > SUSE Labs