From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FCE9C433E2 for ; Thu, 3 Sep 2020 17:02:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DE1802072A for ; Thu, 3 Sep 2020 17:02:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="dTBR0NUV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE1802072A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 40D916B005C; Thu, 3 Sep 2020 13:02:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BE4B6B005D; Thu, 3 Sep 2020 13:02:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D3D16B0062; Thu, 3 Sep 2020 13:02:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id 168726B005C for ; Thu, 3 Sep 2020 13:02:18 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BB26E3634 for ; Thu, 3 Sep 2020 17:02:17 +0000 (UTC) X-FDA: 77222368314.13.smell15_3909e08270ab Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id C18291814062D for ; Thu, 3 Sep 2020 17:02:13 +0000 (UTC) X-HE-Tag: smell15_3909e08270ab X-Filterd-Recvd-Size: 3445 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Sep 2020 17:02:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=zMmFLPhqWER0OTsSWmmyqG3pphhSoq3FXbT4jT7kkiE=; b=dTBR0NUVGVuwmwqmdzhccoXx6r 5r2nFdbvndAHYrvjwpj2ZMEpF3jDwgE4dt3agLUWBfAzFmEKev7/8ocuqebdks0QUr5i410WCUxxH N4TnhSRYDsiRxJm0vMbeLWjGQxVP6z1pY6TsmgxSdgCanRMHE++pHrj8kdp6zhGK7kV1uHoxpTXaD PzLxDqodxaV6gu4I4u0VKLS2t7EgqV/2fpfClv2ZzQ0wSVfD4CvF/YnTDQ3x6E/6WX8tOSnlepUJO cHNznnLQ9KP5Z7aOb57vWpiCUNlxHaSk/1syoM288tSpw5nfFBcy1aQKHc+Z6fdtz99yaBxdzbs9G OyfQpQBQ==; Received: from willy by casper.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDscX-00074S-AB; Thu, 03 Sep 2020 17:01:57 +0000 Date: Thu, 3 Sep 2020 18:01:57 +0100 From: Matthew Wilcox To: Jason Gunthorpe Cc: Roman Gushchin , Michal Hocko , Zi Yan , linux-mm@kvack.org, Rik van Riel , "Kirill A . Shutemov" , Shakeel Butt , Yang Shi , David Nellans , linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 00/16] 1GB THP support on x86_64 Message-ID: <20200903170157.GE14765@casper.infradead.org> References: <20200902180628.4052244-1-zi.yan@sent.com> <20200903073254.GP4617@dhcp22.suse.cz> <20200903162527.GF60440@carbon.dhcp.thefacebook.com> <20200903165051.GN24045@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200903165051.GN24045@ziepe.ca> X-Rspamd-Queue-Id: C18291814062D X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Sep 03, 2020 at 01:50:51PM -0300, Jason Gunthorpe wrote: > At least from a RDMA NIC perspective I've heard from a lot of users > that higher order pages at the DMA level is giving big speed ups too. > > It is basically the same dynamic as CPU TLB, except missing a 'TLB' > cache in a PCI-E device is dramatically more expensive to refill. With > 200G and soon 400G networking these misses are a growing problem. > > With HPC nodes now pushing 1TB of actual physical RAM and single > applications basically using all of it, there is definately some > meaningful return - if pages can be reliably available. > > At least for HPC where the node returns to an idle state after each > job and most of the 1TB memory becomes freed up again, it seems more > believable to me that a large cache of 1G pages could be available? You may be interested in trying out my current THP patchset: http://git.infradead.org/users/willy/pagecache.git It doesn't allocate pages larger than PMD size, but it does allocate pages *up to* PMD size for the page cache which means that larger pages are easier to create as larger pages aren't fragmented all over the system. If someone wants to opportunistically allocate pages larger than PMD size, I've put some preliminary support in for that, but I've never tested any of it. That's not my goal at the moment. I'm not clear whether these HPC users primarily use page cache or anonymous memory (with O_DIRECT). Probably a mixture.