From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88937C2BA1A for ; Tue, 7 Apr 2020 18:15:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 487E82063A for ; Tue, 7 Apr 2020 18:15:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 487E82063A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C8F848E0006; Tue, 7 Apr 2020 14:15:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C194F8E0001; Tue, 7 Apr 2020 14:15:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE1C28E0006; Tue, 7 Apr 2020 14:15:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 954558E0001 for ; Tue, 7 Apr 2020 14:15:51 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5614E824999B for ; Tue, 7 Apr 2020 18:15:51 +0000 (UTC) X-FDA: 76681862502.04.mask29_330984741f35a X-HE-Tag: mask29_330984741f35a X-Filterd-Recvd-Size: 5201 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 7 Apr 2020 18:15:50 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id h9so4970585wrc.8 for ; Tue, 07 Apr 2020 11:15:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=30f4EXGKl1YJhrB5sA5rsC6Ua7MRsvS6emxG/UIc+z4=; b=fDA8ZGD6ZrS6CxlgCF5tykFqddec/iKEeNzXyBCPpiMa+qGasLtXIdl+EPijG7WGQH X63mbBE7BPGvdsKoISYganN5yg2bz3sbrk9cXHolGoHp2glY/a3WmuC+/c8E8xT5Bv7H v8TQOg0Qfnu8qdIXAbEjGDUBPVJSzXsQsaMuuqpeYl8s35tQrLooRvugRhr/r5qCEMNz h6QlA6P5kApupuTPg+uM5E5hqsF/inU6hQnJ7xPd5jcdgZG2dKMSvGd/LxE3Uid61p43 Wj1PPzggzXdZA5LxvuNz5Uny8kcxLr8sE6ROnIbdF1Zg4QcYzCbCLJA7h8Y4I86bGJrv cqIg== X-Gm-Message-State: AGi0PubS01Zx2I8qhwwdBGY3TvREqt9xJzie5jQmOgFV5UDdPUyaIzhP gM984yea79FtKCWsfedFPtE= X-Google-Smtp-Source: APiQypITHu4vN/S/wf/3welZh/HKE2B4GgI/qWph0fV/wXNSGjyHrTkeAOlxepCrV64wBXvSikvSFQ== X-Received: by 2002:adf:90ea:: with SMTP id i97mr3924822wri.123.1586283349499; Tue, 07 Apr 2020 11:15:49 -0700 (PDT) Received: from localhost (ip-37-188-180-223.eurotel.cz. [37.188.180.223]) by smtp.gmail.com with ESMTPSA id g2sm31444354wrs.42.2020.04.07.11.15.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Apr 2020 11:15:48 -0700 (PDT) Date: Tue, 7 Apr 2020 20:15:46 +0200 From: Michal Hocko To: Roman Gushchin Cc: Andrew Morton , Aslan Bakirov , linux-mm@kvack.org, kernel-team@fb.com, linux-kernel@vger.kernel.org, Rik van Riel , Mike Kravetz , Andreas Schaufler , Randy Dunlap , Joonsoo Kim Subject: Re: [PATCH v5 2/2] mm: hugetlb: optionally allocate gigantic hugepages using cma Message-ID: <20200407181546.GV18914@dhcp22.suse.cz> References: <20200407163840.92263-1-guro@fb.com> <20200407163840.92263-3-guro@fb.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200407163840.92263-3-guro@fb.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue 07-04-20 09:38:40, Roman Gushchin wrote: > Commit 944d9fec8d7a ("hugetlb: add support for gigantic page allocation at > runtime") has added the run-time allocation of gigantic pages. However it > actually works only at early stages of the system loading, when the > majority of memory is free. After some time the memory gets fragmented by > non-movable pages, so the chances to find a contiguous 1 GB block are > getting close to zero. Even dropping caches manually doesn't help a lot. > > At large scale rebooting servers in order to allocate gigantic hugepages > is quite expensive and complex. At the same time keeping some constant > percentage of memory in reserved hugepages even if the workload isn't > using it is a big waste: not all workloads can benefit from using 1 GB > pages. > > The following solution can solve the problem: > 1) On boot time a dedicated cma area* is reserved. The size is passed > as a kernel argument. > 2) Run-time allocations of gigantic hugepages are performed using the > cma allocator and the dedicated cma area > > In this case gigantic hugepages can be allocated successfully with a high > probability, however the memory isn't completely wasted if nobody is using > 1GB hugepages: it can be used for pagecache, anon memory, THPs, etc. > > * On a multi-node machine a per-node cma area is allocated on each node. > Following gigantic hugetlb allocation are using the first available > numa node if the mask isn't specified by a user. > > Usage: > 1) configure the kernel to allocate a cma area for hugetlb allocations: > pass hugetlb_cma=10G as a kernel argument > > 2) allocate hugetlb pages as usual, e.g. > echo 10 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages > > If the option isn't enabled or the allocation of the cma area failed, > the current behavior of the system is preserved. > > x86 and arm-64 are covered by this patch, other architectures can be > trivially added later. > > The patch contains clean-ups and fixes proposed and implemented by > Aslan Bakirov and Randy Dunlap. It also contains ideas and suggestions > proposed by Rik van Riel, Michal Hocko and Mike Kravetz. Thanks! > > Signed-off-by: Roman Gushchin > Tested-by: Andreas Schaufler > Acked-by: Mike Kravetz > Acked-by: Michal Hocko > Cc: Aslan Bakirov > Cc: Randy Dunlap > Cc: Rik van Riel > Cc: Joonsoo Kim Thanks a lot for addressing my review feedback! -- Michal Hocko SUSE Labs