From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3C85C63777 for ; Fri, 20 Nov 2020 21:17:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 37DC222253 for ; Fri, 20 Nov 2020 21:17:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="kf1qDma2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 37DC222253 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5A5FF6B0036; Fri, 20 Nov 2020 16:17:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 52E9E6B005C; Fri, 20 Nov 2020 16:17:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A7FC6B005D; Fri, 20 Nov 2020 16:17:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id 0B4076B0036 for ; Fri, 20 Nov 2020 16:17:19 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A8412364C for ; Fri, 20 Nov 2020 21:17:19 +0000 (UTC) X-FDA: 77506057398.27.chair54_3b130de2734e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 8E0253D663 for ; Fri, 20 Nov 2020 21:17:19 +0000 (UTC) X-HE-Tag: chair54_3b130de2734e X-Filterd-Recvd-Size: 3800 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Fri, 20 Nov 2020 21:17:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Sender:Reply-To:Content-ID:Content-Description; bh=PSvZ/V3hJPQm5eQAOv/bPZGbNpjwCHr1HibjUj9KFYU=; b=kf1qDma2luPsYIqJ/FUFFRoFIn m1/WSwGdytFgqwD+TrR++qy82KIODyPULbRDB7bys5Z/AjxMYurrjtxzsuWiaEJ3TbjSPItms1h2M +Cxhi80yPG/35YLHTrKWq5onJe+TUTjI8uTdKJmC8b7NY+BRRD4pIRF1P/x8t5nqRLLBaV14/vWQq +CCRKQi5I6uYosNQa9lBZu4F41yfyvtCJLHZZ5sQnMbg8uVQNLQHCVAAasto98KULnaGh/CirnpqL w0LuYBGi4a68M3m/u50kScjU25A0pcl6IBR/vpG3Sc/FwlsFJdEmo+Rr/xQnwt5BJeDzMEUl+rWtJ YXskBatQ==; Received: from willy by casper.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1kgDmF-0000k7-PU; Fri, 20 Nov 2020 21:17:07 +0000 Date: Fri, 20 Nov 2020 21:17:07 +0000 From: Matthew Wilcox To: David Hildenbrand Cc: Pavel Tatashin , linux-mm , Andrew Morton , Vlastimil Babka , LKML , Michal Hocko , Oscar Salvador , Dan Williams , Sasha Levin , Tyler Hicks , Joonsoo Kim , sthemmin@microsoft.com Subject: Re: Pinning ZONE_MOVABLE pages Message-ID: <20201120211707.GC4327@casper.infradead.org> References: <9452B231-23F3-48F5-A0E2-D6C5603F87F1@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <9452B231-23F3-48F5-A0E2-D6C5603F87F1@redhat.com> Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Nov 20, 2020 at 09:59:24PM +0100, David Hildenbrand wrote: >=20 > > Am 20.11.2020 um 21:28 schrieb Pavel Tatashin : > >=20 > > =EF=BB=BFRecently, I encountered a hang that is happening during memo= ry hot > > remove operation. It turns out that the hang is caused by pinned user > > pages in ZONE_MOVABLE. > >=20 > > Kernel expects that all pages in ZONE_MOVABLE can be migrated, but > > this is not the case if a user applications such as through dpdk > > libraries pinned them via vfio dma map. Kernel keeps trying to > > hot-remove them, but refcnt never gets to zero, so we are looping > > until the hardware watchdog kicks in. > >=20 > > We cannot do dma unmaps before hot-remove, because hot-remove is a > > slow operation, and we have thousands for network flows handled by > > dpdk that we just cannot suspend for the duration of hot-remove > > operation. > >=20 >=20 > Hi! >=20 > It=E2=80=98s a known problem also for VMs using vfio. I thought about t= his some while ago an came to the same conclusion: before performing long= -term pinnings, we have to migrate pages off the movable zone. After that= , it=E2=80=98s too late. We can't, though. VMs using vfio pin their entire address space (right?) so we end up with basically all of the !MOVABLE memory used for VMs and the MOVABLE memory goes unused (I'm thinking about the case of a machine which only hosts VMs and has nothing else to do with its memory). In that case, the sysadmin is going to reconfigure ZONE_MOVABLE away, and now we just don't have any ZONE_MOVABLE. So what's the point? ZONE_MOVABLE can also be pinned by mlock() and other such system calls. The kernel needs to understand that ZONE_MOVABLE memory may not actually be movable, and skip the unmovable stuff.