From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74223C433DB for ; Tue, 22 Dec 2020 19:14:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E932723130 for ; Tue, 22 Dec 2020 19:13:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E932723130 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1A40C8D0009; Tue, 22 Dec 2020 14:13:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 12EEE8D0001; Tue, 22 Dec 2020 14:13:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F38BF8D0009; Tue, 22 Dec 2020 14:13:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id D62CB8D0001 for ; Tue, 22 Dec 2020 14:13:58 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A12481EF3 for ; Tue, 22 Dec 2020 19:13:58 +0000 (UTC) X-FDA: 77621868156.06.base22_240b9ea27462 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 6263C10046C6B for ; Tue, 22 Dec 2020 19:13:58 +0000 (UTC) X-HE-Tag: base22_240b9ea27462 X-Filterd-Recvd-Size: 5638 Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Tue, 22 Dec 2020 19:13:57 +0000 (UTC) Received: by mail-io1-f46.google.com with SMTP id m23so12991362ioy.2 for ; Tue, 22 Dec 2020 11:13:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=QJAkIaRAmbTjT49QW8oYv7QzM88P6GE6t4wSLLc/GcU=; b=aPF/YAZ9erWveRdWO/2hLJheEsEUVZm7epizh5j/VC7kd5b1Or/+tJGPitWc8dSQJP GYAtfRS/iPRCqfuJh9isOkZeiQaHR/W9ZPkmeyYncwIgToIvh4y8xmVs/5oK/tqpHpwj DICtixudSLvfYeRUBOCe0XmxF9mvT+NZwpUa5M36qLPwiW0Jlpze8AF/V0Ql5w9Q8O3W iOShXZkUa915bP42rj1Dmmjh3R4RqVNe7xyqCdswjsY1+7z8cdkhDh0KBMYV/9fVQFm8 +CutoBZSQk7bBC50n1TqdRbxekHfa/PPIO3Fh2PHqlr9OAzu5NuuAc/fmBau3hVgVHb7 SWcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=QJAkIaRAmbTjT49QW8oYv7QzM88P6GE6t4wSLLc/GcU=; b=LfwLj2mdY9xKaktccmS7OGjB7VfoKOXwSMqlpqtAgtDJRmvSGbyjl3/nlcjIst+bi5 Zqhda5DwtBatVgnA68wwqpwJ6YXqf9o0yGJwo20HCQe6x5H7/H8P4DkIK/o4Gtvd7OnU KmN6odZ3NybiCcpw0Qww3lluHeFt2eDTxTim5Npj6/Q9QZm0XG4RVsXQJcGPGN4k/78f mFN1svxXTpP1GfZHogkA7jG4d/rRKasygTAL1NlxnSXDe0lohhBiXZec3mdjkN/vMf2U DhY28HIzIvYTAzRkQylY7xGcTJqvxReX3BA4qZFBaSXlF9KOMEdM4UPespvss/cOlAY3 6zCg== X-Gm-Message-State: AOAM531ds1ApdGu1zwD0NunegdGFwdYE97XVCK4B0KujG0sYOARbBnv/ BkkCLX1vo4ElZCt/vo7P/UojWQ4F//8WNC4wumo= X-Google-Smtp-Source: ABdhPJwiH4X66f06EOxSYK8GgRP7F1wU/QPbZByPqPm8NohKnqOwio5inBXgbitqxNy/JmW3UyhM48dXA4yo6r9vqWA= X-Received: by 2002:a05:6602:150b:: with SMTP id g11mr18941096iow.88.1608664437267; Tue, 22 Dec 2020 11:13:57 -0800 (PST) MIME-Version: 1.0 References: <20201221162519.GA22504@open-light-1.localdomain> In-Reply-To: <20201221162519.GA22504@open-light-1.localdomain> From: Alexander Duyck Date: Tue, 22 Dec 2020 11:13:46 -0800 Message-ID: Subject: Re: [RFC v2 PATCH 0/4] speed up page allocation for __GFP_ZERO To: Alexander Duyck , Mel Gorman , Andrew Morton , Andrea Arcangeli , Dan Williams , "Michael S. Tsirkin" , David Hildenbrand , Jason Wang , Dave Hansen , Michal Hocko , Liang Li , linux-mm , LKML , virtualization@lists.linux-foundation.org Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Dec 21, 2020 at 8:25 AM Liang Li wrote: > > The first version can be found at: https://lkml.org/lkml/2020/4/12/42 > > Zero out the page content usually happens when allocating pages with > the flag of __GFP_ZERO, this is a time consuming operation, it makes > the population of a large vma area very slowly. This patch introduce > a new feature for zero out pages before page allocation, it can help > to speed up page allocation with __GFP_ZERO. > > My original intention for adding this feature is to shorten VM > creation time when SR-IOV devicde is attached, it works good and the > VM creation time is reduced by about 90%. > > Creating a VM [64G RAM, 32 CPUs] with GPU passthrough > ===================================================== > QEMU use 4K pages, THP is off > round1 round2 round3 > w/o this patch: 23.5s 24.7s 24.6s > w/ this patch: 10.2s 10.3s 11.2s > > QEMU use 4K pages, THP is on > round1 round2 round3 > w/o this patch: 17.9s 14.8s 14.9s > w/ this patch: 1.9s 1.8s 1.9s > ===================================================== > > Obviously, it can do more than this. We can benefit from this feature > in the flowing case: So I am not sure page reporting is the best thing to base this page zeroing setup on. The idea with page reporting is to essentially act as a leaky bucket and allow the guest to drop memory it isn't using slowly so if it needs to reinflate it won't clash with the applications that need memory. What you are doing here seems far more aggressive in that you are going down to low order pages and sleeping instead of rescheduling for the next time interval. Also I am not sure your SR-IOV creation time test is a good justification for this extra overhead. With your patches applied all you are doing is making use of the free time before the test to do the page zeroing instead of doing it during your test. As such your CPU overhead prior to running the test would be higher and you haven't captured that information. One thing I would be interested in seeing is what is the load this is adding when you are running simple memory allocation/free type tests on the system. For example it might be useful to see what the will-it-scale page_fault1 tests look like with this patch applied versus not applied. I suspect it would be adding some amount of overhead as you have to spend a ton of time scanning all the pages and that will be considerable overhead.