From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D203DC433E6 for ; Tue, 5 Jan 2021 02:14:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 809C522515 for ; Tue, 5 Jan 2021 02:14:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 809C522515 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CEFFA8D004C; Mon, 4 Jan 2021 21:14:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C52728D0036; Mon, 4 Jan 2021 21:14:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B41978D004C; Mon, 4 Jan 2021 21:14:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0081.hostedemail.com [216.40.44.81]) by kanga.kvack.org (Postfix) with ESMTP id 959418D0036 for ; Mon, 4 Jan 2021 21:14:25 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5DF458248047 for ; Tue, 5 Jan 2021 02:14:25 +0000 (UTC) X-FDA: 77670102090.27.alley29_38099c1274d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 41FC43D663 for ; Tue, 5 Jan 2021 02:14:25 +0000 (UTC) X-HE-Tag: alley29_38099c1274d5 X-Filterd-Recvd-Size: 6991 Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Tue, 5 Jan 2021 02:14:24 +0000 (UTC) Received: by mail-lf1-f44.google.com with SMTP id h22so69216270lfu.2 for ; Mon, 04 Jan 2021 18:14:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+xkQA9ZfR+79pp7Ulpm2NqZg2cDfkuU5Tk9TMW07Q/c=; b=L02eXfE4Jc7oT66d67QzqNj/RmpYwVhrjr6MB+Bxl5Ykw4KP9mwhqSGYxpwlcc6LPs /Md0OgTPowORc0tIgihkmp0cXgGl9FvYHClrCFudQ8cyUrZSEJxkcL8ELJvjSH1xbATH KvkVysRYh8uqCD5cS8vF6XjqittzzuOLQZrlBv4hY9DoEocTzzujuj+BjINCo8aEd0pT wasEfnStSTW9gkOyZY6cnDUD2vZTR/u58YzFvicQEN6mQ1xo8HKpWdk7H8+fZLTH+JL8 eUbQPVsYeCCvHaQv+Zulne+dKgbRIqyT/koWDTHdmPkb2+/i62ZNAgdwjE4Pgjrnd660 djGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+xkQA9ZfR+79pp7Ulpm2NqZg2cDfkuU5Tk9TMW07Q/c=; b=pecCrgYwtf1nJ8PweN34JHAEMWyUnVyY1oUoaoPTaaaMPuHrSjQG3jQwc9C8cs2Axz oB8ih4ddw4TGQI0kA26PJhhqlu3+wjkBb2tkUjfY2va7Gw0Kfem0blEv4pZxBvmvTk25 uMLmzu2hJq2Jyh6Al21Sesnp7TpVJRNxRg7KC6+rwd0EdqRVxsZmH+VkCXQgSCfZ7DaR /4kt7SBPFXIPIGe30leFsv/t5OuoFnldIda5xLSHjOlS3WXCE+g4VmYziVKSIfMXMgi+ TTxOkkRaysfh/pYs5E7ejBT3kNJUqJzXU40vP6jhNoIaXzLsEYt/+vudkB+wM7kv+74a NXnQ== X-Gm-Message-State: AOAM530mI1pv0Yr/vVua2bdyCXBk6WgCAdhDEchDjkCz4dejWVc3edpE 96TTrAGHoEyxj6lwOshXBcIw0IcXAa7M+6fL1Wk= X-Google-Smtp-Source: ABdhPJwS3i/LxCg32Rg+Qmv2lZZlfQUOdQkbfxLjoknBZAOR5Pc+gPjgDoCy+vUunI60PAIj6qPcLcr8EEpmDgxaH7w= X-Received: by 2002:a19:804a:: with SMTP id b71mr31198406lfd.504.1609812863497; Mon, 04 Jan 2021 18:14:23 -0800 (PST) MIME-Version: 1.0 References: <96BB0656-F234-4634-853E-E2A747B6ECDB@redhat.com> In-Reply-To: <96BB0656-F234-4634-853E-E2A747B6ECDB@redhat.com> From: Liang Li Date: Tue, 5 Jan 2021 10:14:11 +0800 Message-ID: Subject: Re: [RFC v2 PATCH 0/4] speed up page allocation for __GFP_ZERO To: David Hildenbrand Cc: Alexander Duyck , Mel Gorman , Andrew Morton , Andrea Arcangeli , Dan Williams , "Michael S. Tsirkin" , Jason Wang , Dave Hansen , Michal Hocko , Liang Li , linux-mm , LKML , virtualization@lists.linux-foundation.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > >>> In our production environment, there are three main applications have= such > >>> requirement, one is QEMU [creating a VM with SR-IOV passthrough devic= e], > >>> anther other two are DPDK related applications, DPDK OVS and SPDK vho= st, > >>> for best performance, they populate memory when starting up. For SPDK= vhost, > >>> we make use of the VHOST_USER_GET/SET_INFLIGHT_FD feature for > >>> vhost 'live' upgrade, which is done by killing the old process and > >>> starting a new > >>> one with the new binary. In this case, we want the new process starte= d as quick > >>> as possible to shorten the service downtime. We really enable this fe= ature > >>> to speed up startup time for them :) > > Am I wrong or does using hugeltbfs/tmpfs ... i.e., a file not-deleted bet= ween shutting down the old instances and firing up the new instance just so= lve this issue? You are right, it works for the SPDK vhost upgrade case. > > >> > >> Thanks for info on the use case! > >> > >> All of these use cases either already use, or could use, huge pages > >> IMHO. It's not your ordinary proprietary gaming app :) This is where > >> pre-zeroing of huge pages could already help. > > > > You are welcome. For some historical reason, some of our services are > > not using hugetlbfs, that is why I didn't start with hugetlbfs. > > > >> Just wondering, wouldn't it be possible to use tmpfs/hugetlbfs ... > >> creating a file and pre-zeroing it from another process, or am I missi= ng > >> something important? At least for QEMU this should work AFAIK, where y= ou > >> can just pass the file to be use using memory-backend-file. > >> > > If using another process to create a file, we can offload the overhead = to > > another process, and there is no need to pre-zeroing it's content, just > > populating the memory is enough. > > Right, if non-zero memory can be tolerated (e.g., for vms usually has to)= . I mean there is no need to pre-zeroing the file content obviously in user s= pace, the kernel will do it when populating the memory. > > If we do it that way, then how to determine the size of the file? it de= pends > > on the RAM size of the VM the customer buys. > > Maybe we can create a file > > large enough in advance and truncate it to the right size just before t= he > > VM is created. Then, how many large files should be created on a host? > > That=E2=80=98s mostly already existing scheduling logic, no? (How many vm= s can I put onto a specific machine eventually) It depends on how the scheduling component is designed. Yes, you can put 10 VMs with 4C8G(4CPU, 8G RAM) on a host and 20 VMs with 2C4G on another one. But if one type of them, e.g. 4C8G are sold out, customers can't by more 4C8G VM while there are some free 2C4G VMs, the resource reserved for them can be provided as 4C8G VMs > > You will find there are a lot of things that have to be handled properl= y. > > I think it's possible to make it work well, but we will transfer the > > management complexity to up layer components. It's a bad practice to le= t > > upper layer components process such low level details which should be > > handled in the OS layer. > > It=E2=80=98s bad practice to squeeze things into the kernel that can just= be handled on upper layers ;) > You must know there are a lot of functions in the kernel which can be done in userspace. e.g. Some of the device emulations like APIC, vhost-net backend which has userspace implementation. :) Bad or not depends on the benefits the solution brings. >From the viewpoint of a user space application, the kernel should provide high performance memory management service. That's why I think it should be done in the kernel. Thanks Liang