From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7FFDC433E2 for ; Mon, 22 Jun 2020 18:45:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6EA6B2076A for ; Mon, 22 Jun 2020 18:45:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="Rw1MHPXF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6EA6B2076A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 047216B0022; Mon, 22 Jun 2020 14:45:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F39DA6B0023; Mon, 22 Jun 2020 14:45:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E28736B0024; Mon, 22 Jun 2020 14:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id C7B046B0022 for ; Mon, 22 Jun 2020 14:45:10 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 7996E181AC9CB for ; Mon, 22 Jun 2020 18:45:10 +0000 (UTC) X-FDA: 76957725180.06.shoes71_0c12c5426e35 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 366EA10056F20 for ; Mon, 22 Jun 2020 18:45:10 +0000 (UTC) X-HE-Tag: shoes71_0c12c5426e35 X-Filterd-Recvd-Size: 4662 Received: from hqnvemgate26.nvidia.com (hqnvemgate26.nvidia.com [216.228.121.65]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Mon, 22 Jun 2020 18:45:09 +0000 (UTC) Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Mon, 22 Jun 2020 11:44:55 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Mon, 22 Jun 2020 11:45:08 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Mon, 22 Jun 2020 11:45:08 -0700 Received: from rcampbell-dev.nvidia.com (172.20.13.39) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 22 Jun 2020 18:44:59 +0000 Subject: Re: [PATCH 08/16] nouveau/hmm: fault one page at a time To: Jason Gunthorpe CC: , , , , , Jerome Glisse , "John Hubbard" , Christoph Hellwig , Ben Skeggs , Andrew Morton , Shuah Khan References: <20200619215649.32297-1-rcampbell@nvidia.com> <20200619215649.32297-9-rcampbell@nvidia.com> <20200622172233.GA2874652@mellanox.com> From: Ralph Campbell X-Nvconfidentiality: public Message-ID: Date: Mon, 22 Jun 2020 11:44:59 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: <20200622172233.GA2874652@mellanox.com> X-Originating-IP: [172.20.13.39] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1592851495; bh=edTvH4CLD4vItWTGQfSjP32xxPYuTSK47wQwLKtpC2o=; h=X-PGP-Universal:Subject:To:CC:References:From:X-Nvconfidentiality: Message-ID:Date:User-Agent:MIME-Version:In-Reply-To: X-Originating-IP:X-ClientProxiedBy:Content-Type:Content-Language: Content-Transfer-Encoding; b=Rw1MHPXFCHxs683/WECRpVcEISDyJHi11DYaKjk6Je+Hzsfmso1p97Ec8fYZM2+nM A+Whgg2ThDXh90YDXJAKBJwzuaP2WzJlG8bwz/1/VnakrGPCT7sb/aLJ+IleyTzN4h 9jH1TmNccMQ6YU399/RKKkKq2dPSZhr92L71xYLncMKZOVOR48E1BIFUCpW11mwjOE t+M8mWr0WSNGayrNmi4RHcERwZ+hLyLIlZZD5cjsTt043/R1L5B5ihX/JKG3DP2Aem qU3pd3KpJt+qQyE2dumEr5djt4hCCHzY/BBBtQZeeGZ0+hG3skWe1a6K8PfsLkY2WT 4nfL0AaTfpmSw== X-Rspamd-Queue-Id: 366EA10056F20 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 6/22/20 10:22 AM, Jason Gunthorpe wrote: > On Fri, Jun 19, 2020 at 02:56:41PM -0700, Ralph Campbell wrote: >> The SVM page fault handler groups faults into a range of contiguous >> virtual addresses and requests hmm_range_fault() to populate and >> return the page frame number of system memory mapped by the CPU. >> In preparation for supporting large pages to be mapped by the GPU, >> process faults one page at a time. In addition, use the hmm_range >> default_flags to fix a corner case where the input hmm_pfns array >> is not reinitialized after hmm_range_fault() returns -EBUSY and must >> be called again. > > Are you sure? hmm_range_fault is pretty expensive per call.. > > Jason > Short answer is no, I'm not 100% sure. The GPU might generate a list of dozens or hundreds of fault entries in the same 4K page, or sequential 4K pages, or some other stride. A single 2MB mapping might satisfy all of those after calling hmm_range_fault() for the first fault entry and then skipping all the other fault entries that fall into that range. So mostly, I'm proposing this change because it makes handling the compound page case and -EBUSY case simpler. As for performance, that is hard to say because nouveau is missing policies for whether to migrate data to GPU memory on a fault or to map system memory. Since GPU memory is much higher bandwidth, overall performance can be much higher if the data is migrated to the GPU's local memory but currently, migration is only performed explicitly under application request (via OpenCL clEnqueueSVMMigrateMem() call). If the GPU is only accessing system memory a few times, then it can be faster to map system memory and not migrate the data so it depends on the application. Then there is thrashing to consider if the GPU and CPU are both trying to access the same pages...