From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B10B1C433E1 for ; Wed, 24 Mar 2021 19:21:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4AFF661A07 for ; Wed, 24 Mar 2021 19:21:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4AFF661A07 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0F736B02FE; Wed, 24 Mar 2021 15:21:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC00A6B02FF; Wed, 24 Mar 2021 15:21:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B130D6B0300; Wed, 24 Mar 2021 15:21:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 92DA36B02FE for ; Wed, 24 Mar 2021 15:21:01 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 542F08248D52 for ; Wed, 24 Mar 2021 19:21:01 +0000 (UTC) X-FDA: 77955735522.23.5481FD2 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by imf08.hostedemail.com (Postfix) with ESMTP id A593280192E2 for ; Wed, 24 Mar 2021 19:20:59 +0000 (UTC) Received: by mail-wr1-f42.google.com with SMTP id e18so25598480wrt.6 for ; Wed, 24 Mar 2021 12:21:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=tyRazDGir5x3TAuTtidSefO0V++5Y3T6e12HWHHxi8o=; b=ixY6eN/EJw9qDTzwkGIO/z4YdG6nIFOcJDyZPNJhy7h2OqrGebjpnGa71AjAUaw9Ti xQcFJfz2I5MkpZNdVDBc8/U/vkkvbwlvdoVz5I8Dkg1NsiDHG3Ba0efClwh8fjerERNL 7Lg5lNbYmje5jKDhRmubW6Z/iISr8C+a6nZ6w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=tyRazDGir5x3TAuTtidSefO0V++5Y3T6e12HWHHxi8o=; b=bMQdTGG1ZXy/IgmY+pm39KDuiOdjpWD9IGfL+c1yP3asYe3/flg+yIChE2peUNveUJ 1zsamw/HGlHvNLjBUlGXMj2siuaVVo1zgtlpWhelx2TgLFX9eAqydwd/OaJQDOR/YB6z y8+MW1VxZyH1sUD8ecvXumMo5yN5VEurzlAZq9cc6mOIJTWiT9MOi6l2zlsD2/GF1QnR +qCCwi9ZVTH+h4qzdzIChcZ6q4J5R+EZx4sYAZx3avF5v/pVyK5mBLgdoqBRGUc/E2ep 8pUt/4uuOcWtvHXdPxvmIpRSr2CLGpCzAIld5qYnZ8KUq29MMIa7LQZAcUMV6myKAmG2 maiA== X-Gm-Message-State: AOAM530qps91tT56Kwz9+vvFmm8An0hN3vD5Xbp8l+zfj2Gjt3/cD7qO +izfUCasZ7LtjJ07rSpj6TnrKQ== X-Google-Smtp-Source: ABdhPJxjXE/HNfEm4Qs9it8hgxmYAi7iB2AHMj8ATHq4jb2vtkgCIH7680kqcrvTjxyhE1o0GL1+IQ== X-Received: by 2002:adf:a59a:: with SMTP id g26mr5080022wrc.271.1616613659702; Wed, 24 Mar 2021 12:20:59 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id y205sm4094896wmc.18.2021.03.24.12.20.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Mar 2021 12:20:58 -0700 (PDT) Date: Wed, 24 Mar 2021 20:20:57 +0100 From: Daniel Vetter To: Christian =?iso-8859-1?Q?K=F6nig?= Cc: Thomas =?iso-8859-1?Q?Hellstr=F6m_=28Intel=29?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu Subject: Re: [PATCH] drm/ttm: stop warning on TT shrinker failure Message-ID: Mail-Followup-To: Christian =?iso-8859-1?Q?K=F6nig?= , Thomas =?iso-8859-1?Q?Hellstr=F6m_=28Intel=29?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu References: <20808d08-b66c-13c3-f672-ebce216b2fa2@gmail.com> <03889c00-bb5d-ef20-12c6-7e77df073dd9@amd.com> <762c4597-e9bd-6d8d-51b5-16b04f913eb8@shipmail.org> <488c8996-1dd2-4928-a98a-4e72f3e0af64@amd.com> <31a52f86-e4af-f1d3-90b2-6eff8ec5f300@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <31a52f86-e4af-f1d3-90b2-6eff8ec5f300@amd.com> X-Operating-System: Linux phenom 5.7.0-1-amd64 X-Stat-Signature: n9abnn937zdro7trygq715soes36nfxk X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A593280192E2 Received-SPF: none (ffwll.ch>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=mail-wr1-f42.google.com; client-ip=209.85.221.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616613659-913828 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 24, 2021 at 01:07:44PM +0100, Christian K=F6nig wrote: >=20 >=20 > Am 24.03.21 um 13:01 schrieb Daniel Vetter: > > On Wed, Mar 24, 2021 at 01:00:28PM +0100, Christian K=F6nig wrote: > > > Am 24.03.21 um 12:55 schrieb Daniel Vetter: > > > > On Wed, Mar 24, 2021 at 11:19:13AM +0100, Thomas Hellstr=F6m (Int= el) wrote: > > > > > On 3/23/21 4:45 PM, Christian K=F6nig wrote: > > > > > > Am 23.03.21 um 16:13 schrieb Michal Hocko: > > > > > > > On Tue 23-03-21 14:56:54, Christian K=F6nig wrote: > > > > > > > > Am 23.03.21 um 14:41 schrieb Michal Hocko: > > > > > > > [...] > > > > > > > > > Anyway, I am wondering whether the overall approach is > > > > > > > > > sound. Why don't > > > > > > > > > you simply use shmem as your backing storage from the > > > > > > > > > beginning and pin > > > > > > > > > those pages if they are used by the device? > > > > > > > > Yeah, that is exactly what the Intel guys are doing for t= heir > > > > > > > > integrated > > > > > > > > GPUs :) > > > > > > > >=20 > > > > > > > > Problem is for TTM I need to be able to handle dGPUs and = those have all > > > > > > > > kinds of funny allocation restrictions. In other words I = need to > > > > > > > > guarantee > > > > > > > > that the allocated memory is coherent accessible to the G= PU > > > > > > > > without using > > > > > > > > SWIOTLB. > > > > > > > >=20 > > > > > > > > The simple case is that the device can only do DMA32, but= you also got > > > > > > > > device which can only do 40bits or 48bits. > > > > > > > >=20 > > > > > > > > On top of that you also got AGP, CMA and stuff like CPU c= ache behavior > > > > > > > > changes (write back vs. write through, vs. uncached). > > > > > > > OK, so the underlying problem seems to be that gfp mask (th= us > > > > > > > mapping_gfp_mask) cannot really reflect your requirements, = right?=A0 Would > > > > > > > it help if shmem would allow to provide an allocation callb= ack to > > > > > > > override alloc_page_vma which is used currently? I am prett= y sure there > > > > > > > will be more to handle but going through shmem for the whol= e life time > > > > > > > is just so much easier to reason about than some tricks to = abuse shmem > > > > > > > just for the swapout path. > > > > > > Well it's a start, but the pages can have special CPU cache s= ettings. So > > > > > > direct IO from/to them usually doesn't work as expected. > > > > > >=20 > > > > > > Additional to that for AGP and CMA I need to make sure that I= give those > > > > > > pages back to the relevant subsystems instead of just droppin= g the page > > > > > > reference. > > > > > >=20 > > > > > > So I would need to block for the swapio to be completed. > > > > > >=20 > > > > > > Anyway I probably need to revert those patches for now since = this isn't > > > > > > working as we hoped it would. > > > > > >=20 > > > > > > Thanks for the explanation how stuff works here. > > > > > Another alternative here that I've tried before without being s= uccessful > > > > > would perhaps be to drop shmem completely and, if it's a normal= page (no dma > > > > > or funny caching attributes) just use add_to_swap_cache()? If i= t's something > > > > > else, try alloc a page with relevant gfp attributes, copy and > > > > > add_to_swap_cache()? Or perhaps that doesn't work well from a s= hrinker > > > > > either? > > > > So before we toss everything and go an a great rewrite-the-world = tour, > > > > what if we just try to split up big objects. So for objects which= are > > > > bigger than e.g. 10mb > > > >=20 > > > > - move them to a special "under eviction" list > > > > - keep a note how far we evicted thus far > > > > - interleave allocating shmem pages, copying data and releasing t= he ttm > > > > backing store on a chunk basis (maybe 10mb or whatever, tunin= g tbh) > > > >=20 > > > > If that's not enough, occasionally break out of the shrinker enti= rely so > > > > other parts of reclaim can reclaim the shmem stuff. But just rele= asing our > > > > own pages as we go should help a lot I think. > > > Yeah, the later is exactly what I was currently prototyping. > > >=20 > > > I just didn't used a limit but rather a only partially evicted BOs = list > > > which is used when we fail to allocate a page. > > >=20 > > > For the 5.12 cycle I think we should just go back to a hard 50% lim= it for > > > now and then resurrect this when we have solved the issues. > > Can we do the 50% limit without tossing out all the code we've done t= hus > > far? Just so this doesn't get too disruptive. >=20 > Yeah, I just need to get back to v1 of this patch. Before you convinced= me > that the shrinker is the better approach .) I don't think there's anything else than the shrinker if you want dynamically sized memory usage. Or pinning it all. Implementing our own kswapd and not tying into direct reclaim does not sound like a good idea to me. -Daniel --=20 Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch