From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7C24C433E1 for ; Wed, 24 Mar 2021 11:55:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 334F461A0A for ; Wed, 24 Mar 2021 11:55:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 334F461A0A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 58C986B02A2; Wed, 24 Mar 2021 07:55:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53D086B02A5; Wed, 24 Mar 2021 07:55:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B6016B02A6; Wed, 24 Mar 2021 07:55:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id 1D4B76B02A2 for ; Wed, 24 Mar 2021 07:55:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id CFA8A8249980 for ; Wed, 24 Mar 2021 11:55:38 +0000 (UTC) X-FDA: 77954613156.19.A6C8169 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf05.hostedemail.com (Postfix) with ESMTP id AB85EE0011F8 for ; Wed, 24 Mar 2021 11:55:37 +0000 (UTC) Received: by mail-wm1-f53.google.com with SMTP id t5-20020a1c77050000b029010e62cea9deso1043877wmi.0 for ; Wed, 24 Mar 2021 04:55:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=gO58uuS+0G4MQjeMAkSvwrYzYSPAbUNJctRACLjk0bg=; b=OUEbMbeV2XMtinhZQiQFhGWWPrSed8DGkee/j1abaY7+gjIBFDFq7gm3fIROxw/tv7 Dl58YUKUlMglRAjIt3sVLQgrLCQcs/hHJrmpFuJ7HEnVuIQPDA+NJf1RmX88aFAJKAcK 99SwnPEbjAWAiGXs+X73ja2KaaeiSNX+XXnag= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=gO58uuS+0G4MQjeMAkSvwrYzYSPAbUNJctRACLjk0bg=; b=tSBkAxhNFCrhE/YYb6XNVV8usNt6/Fv9A8PxCVZDWDi3hUQI5gEri/HfweX3/uTlez dKihrXCmtDzRemV7mC8fadLzbHvEF4GJwpVaoUizeLOxWpKr7B7yf5Svl2mAOcUvlIB4 wz9hOt5XIVXhM2LctwtiqYb+8/Zghq2bTzXwAtSgk7t655MHP9fFt3XktOYVPEUH7za9 +Y6KaObGk+3uASBkduYUexWvLz9TJxGQT3e2KiU+oK4m3TFWm9QDP72PJQAjDIq1iBOR Vr5rwbwE1Ai3sp7NdlZxxBJiQehek1FOTLZ0gpOBYwx2AXWzHQfkqUuqiUbozb5i4GMY LFcg== X-Gm-Message-State: AOAM533qkOOg51fJIQEjUJ/B0nc1Ay41J5ZVU22F07bBdqCwIyxtCKvb Rj6DZUDg0aZGq/NJYUEo68Z8ug== X-Google-Smtp-Source: ABdhPJwhiD7SOUBfRgkNdfO4JdgqIMVd2CqvBvssLjFmgEs3XWs421vPNE2H+yLnhmkR35AAzLxCiw== X-Received: by 2002:a05:600c:19d1:: with SMTP id u17mr2527627wmq.141.1616586936320; Wed, 24 Mar 2021 04:55:36 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id q207sm2106956wme.36.2021.03.24.04.55.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Mar 2021 04:55:35 -0700 (PDT) Date: Wed, 24 Mar 2021 12:55:33 +0100 From: Daniel Vetter To: Thomas =?iso-8859-1?Q?Hellstr=F6m_=28Intel=29?= Cc: Christian =?iso-8859-1?Q?K=F6nig?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu Subject: Re: [PATCH] drm/ttm: stop warning on TT shrinker failure Message-ID: Mail-Followup-To: Thomas =?iso-8859-1?Q?Hellstr=F6m_=28Intel=29?= , Christian =?iso-8859-1?Q?K=F6nig?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu References: <75ff80c5-a054-d13d-85c1-0040addb45d2@gmail.com> <20808d08-b66c-13c3-f672-ebce216b2fa2@gmail.com> <03889c00-bb5d-ef20-12c6-7e77df073dd9@amd.com> <762c4597-e9bd-6d8d-51b5-16b04f913eb8@shipmail.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <762c4597-e9bd-6d8d-51b5-16b04f913eb8@shipmail.org> X-Operating-System: Linux phenom 5.7.0-1-amd64 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AB85EE0011F8 X-Stat-Signature: q5144tatoxguwbjg9xmo54bwonjrfbx3 Received-SPF: none (ffwll.ch>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=mail-wm1-f53.google.com; client-ip=209.85.128.53 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616586937-83940 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 24, 2021 at 11:19:13AM +0100, Thomas Hellstr=F6m (Intel) wrot= e: >=20 > On 3/23/21 4:45 PM, Christian K=F6nig wrote: > > Am 23.03.21 um 16:13 schrieb Michal Hocko: > > > On Tue 23-03-21 14:56:54, Christian K=F6nig wrote: > > > > Am 23.03.21 um 14:41 schrieb Michal Hocko: > > > [...] > > > > > Anyway, I am wondering whether the overall approach is > > > > > sound. Why don't > > > > > you simply use shmem as your backing storage from the > > > > > beginning and pin > > > > > those pages if they are used by the device? > > > > Yeah, that is exactly what the Intel guys are doing for their > > > > integrated > > > > GPUs :) > > > >=20 > > > > Problem is for TTM I need to be able to handle dGPUs and those ha= ve all > > > > kinds of funny allocation restrictions. In other words I need to > > > > guarantee > > > > that the allocated memory is coherent accessible to the GPU > > > > without using > > > > SWIOTLB. > > > >=20 > > > > The simple case is that the device can only do DMA32, but you als= o got > > > > device which can only do 40bits or 48bits. > > > >=20 > > > > On top of that you also got AGP, CMA and stuff like CPU cache beh= avior > > > > changes (write back vs. write through, vs. uncached). > > > OK, so the underlying problem seems to be that gfp mask (thus > > > mapping_gfp_mask) cannot really reflect your requirements, right?=A0= Would > > > it help if shmem would allow to provide an allocation callback to > > > override alloc_page_vma which is used currently? I am pretty sure t= here > > > will be more to handle but going through shmem for the whole life t= ime > > > is just so much easier to reason about than some tricks to abuse sh= mem > > > just for the swapout path. > >=20 > > Well it's a start, but the pages can have special CPU cache settings.= So > > direct IO from/to them usually doesn't work as expected. > >=20 > > Additional to that for AGP and CMA I need to make sure that I give th= ose > > pages back to the relevant subsystems instead of just dropping the pa= ge > > reference. > >=20 > > So I would need to block for the swapio to be completed. > >=20 > > Anyway I probably need to revert those patches for now since this isn= 't > > working as we hoped it would. > >=20 > > Thanks for the explanation how stuff works here. >=20 > Another alternative here that I've tried before without being successfu= l > would perhaps be to drop shmem completely and, if it's a normal page (n= o dma > or funny caching attributes) just use add_to_swap_cache()? If it's some= thing > else, try alloc a page with relevant gfp attributes, copy and > add_to_swap_cache()? Or perhaps that doesn't work well from a shrinker > either? So before we toss everything and go an a great rewrite-the-world tour, what if we just try to split up big objects. So for objects which are bigger than e.g. 10mb - move them to a special "under eviction" list - keep a note how far we evicted thus far - interleave allocating shmem pages, copying data and releasing the ttm backing store on a chunk basis (maybe 10mb or whatever, tuning tbh) If that's not enough, occasionally break out of the shrinker entirely so other parts of reclaim can reclaim the shmem stuff. But just releasing ou= r own pages as we go should help a lot I think. -Daniel --=20 Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch