From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5797AC433C1 for ; Wed, 24 Mar 2021 12:08:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E6D5D619FE for ; Wed, 24 Mar 2021 12:08:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E6D5D619FE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ffwll.ch Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7C8856B02B5; Wed, 24 Mar 2021 08:08:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79FB16B02B9; Wed, 24 Mar 2021 08:08:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F26A6B02BA; Wed, 24 Mar 2021 08:08:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id 3F6196B02B5 for ; Wed, 24 Mar 2021 08:08:23 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E0307181AEF3F for ; Wed, 24 Mar 2021 12:08:22 +0000 (UTC) X-FDA: 77954645244.10.027A0FE Received: from mail-ej1-f49.google.com (mail-ej1-f49.google.com [209.85.218.49]) by imf22.hostedemail.com (Postfix) with ESMTP id CFBE0C0007C5 for ; Wed, 24 Mar 2021 12:08:20 +0000 (UTC) Received: by mail-ej1-f49.google.com with SMTP id ce10so32442189ejb.6 for ; Wed, 24 Mar 2021 05:08:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=y/CPamrFRpFRFAnI28nNs6ii9WSN4afHD5HKOCB7J8c=; b=IT94N8Kudb1s7hMBK+meT8zGUaH5jVPBknlROYnHXpEJMka+WOm+/tD3pATq/adnsl nmsFwSLuAjPOBnobohRgG8l52MJLi4VT2B4zcrrP29bTsNwSv39XjBSasuAaG3LSy2iB LcIwsECqo/pmBIpQ/V+1Pp74MWfr4CFq13Sks= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=y/CPamrFRpFRFAnI28nNs6ii9WSN4afHD5HKOCB7J8c=; b=Jh/RAd4jGgwQ+ayUStBO0v8M2iEdnhwGNzuwua8E5omQVB1ptluRzipBvtij6wtwCh AAbDBACRWGSHFy99bt60feoVIqqFrmtUIvODJhyhHxCE959k2PTXwRm+MtajGWKC7PhD j+ibPsXzubVsqySn/ailuTgikGnuQMmCVrL6GajRKZE5Gp/y4EoJA6B8/Nv/9DJWnQt3 zT0Sa+OPmXNNeMuocwvofu3c1gqNruRsyR0ntjT8RisvCP/3CphYXZUQNlxqVy3wHXLl 1L1mYNGvOAas1VRkZ8p4dkMyT6Gwp59R5yKY393uKNVu9WKoO1fqk2mMGBD6eaHniBGn ACuw== X-Gm-Message-State: AOAM533W75Hg1DBUpSZczWH1y6Rf76HoKIWLAAfhXk6MvrlsgkM0QFBs C4J0U9pI983Tcw4sLfra0MeFbsiYcIykerkl X-Google-Smtp-Source: ABdhPJwUNXx1B+FbHKziqsaPTXmwCuAvrvj40VRzGIVe/yFkcrZJ+8u496Je7TqqruPqCdIVyk9sHQ== X-Received: by 2002:a05:6000:137b:: with SMTP id q27mr3176838wrz.168.1616587321808; Wed, 24 Mar 2021 05:02:01 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id c8sm2180230wmb.34.2021.03.24.05.02.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Mar 2021 05:02:01 -0700 (PDT) Date: Wed, 24 Mar 2021 13:01:59 +0100 From: Daniel Vetter To: Christian =?iso-8859-1?Q?K=F6nig?= Cc: Thomas =?iso-8859-1?Q?Hellstr=F6m_=28Intel=29?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu Subject: Re: [PATCH] drm/ttm: stop warning on TT shrinker failure Message-ID: Mail-Followup-To: Christian =?iso-8859-1?Q?K=F6nig?= , Thomas =?iso-8859-1?Q?Hellstr=F6m_=28Intel=29?= , Michal Hocko , Matthew Wilcox , dri-devel , Linux MM , amd-gfx list , Dave Chinner , Leo Liu References: <75ff80c5-a054-d13d-85c1-0040addb45d2@gmail.com> <20808d08-b66c-13c3-f672-ebce216b2fa2@gmail.com> <03889c00-bb5d-ef20-12c6-7e77df073dd9@amd.com> <762c4597-e9bd-6d8d-51b5-16b04f913eb8@shipmail.org> <488c8996-1dd2-4928-a98a-4e72f3e0af64@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <488c8996-1dd2-4928-a98a-4e72f3e0af64@amd.com> X-Operating-System: Linux phenom 5.7.0-1-amd64 X-Stat-Signature: 94ihsaqqo7boxooa7khegp4rrpkdk4ni X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: CFBE0C0007C5 Received-SPF: none (ffwll.ch>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mail-ej1-f49.google.com; client-ip=209.85.218.49 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616587700-663079 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 24, 2021 at 01:00:28PM +0100, Christian K=F6nig wrote: > Am 24.03.21 um 12:55 schrieb Daniel Vetter: > > On Wed, Mar 24, 2021 at 11:19:13AM +0100, Thomas Hellstr=F6m (Intel) = wrote: > > > On 3/23/21 4:45 PM, Christian K=F6nig wrote: > > > > Am 23.03.21 um 16:13 schrieb Michal Hocko: > > > > > On Tue 23-03-21 14:56:54, Christian K=F6nig wrote: > > > > > > Am 23.03.21 um 14:41 schrieb Michal Hocko: > > > > > [...] > > > > > > > Anyway, I am wondering whether the overall approach is > > > > > > > sound. Why don't > > > > > > > you simply use shmem as your backing storage from the > > > > > > > beginning and pin > > > > > > > those pages if they are used by the device? > > > > > > Yeah, that is exactly what the Intel guys are doing for their > > > > > > integrated > > > > > > GPUs :) > > > > > >=20 > > > > > > Problem is for TTM I need to be able to handle dGPUs and thos= e have all > > > > > > kinds of funny allocation restrictions. In other words I need= to > > > > > > guarantee > > > > > > that the allocated memory is coherent accessible to the GPU > > > > > > without using > > > > > > SWIOTLB. > > > > > >=20 > > > > > > The simple case is that the device can only do DMA32, but you= also got > > > > > > device which can only do 40bits or 48bits. > > > > > >=20 > > > > > > On top of that you also got AGP, CMA and stuff like CPU cache= behavior > > > > > > changes (write back vs. write through, vs. uncached). > > > > > OK, so the underlying problem seems to be that gfp mask (thus > > > > > mapping_gfp_mask) cannot really reflect your requirements, righ= t?=A0 Would > > > > > it help if shmem would allow to provide an allocation callback = to > > > > > override alloc_page_vma which is used currently? I am pretty su= re there > > > > > will be more to handle but going through shmem for the whole li= fe time > > > > > is just so much easier to reason about than some tricks to abus= e shmem > > > > > just for the swapout path. > > > > Well it's a start, but the pages can have special CPU cache setti= ngs. So > > > > direct IO from/to them usually doesn't work as expected. > > > >=20 > > > > Additional to that for AGP and CMA I need to make sure that I giv= e those > > > > pages back to the relevant subsystems instead of just dropping th= e page > > > > reference. > > > >=20 > > > > So I would need to block for the swapio to be completed. > > > >=20 > > > > Anyway I probably need to revert those patches for now since this= isn't > > > > working as we hoped it would. > > > >=20 > > > > Thanks for the explanation how stuff works here. > > > Another alternative here that I've tried before without being succe= ssful > > > would perhaps be to drop shmem completely and, if it's a normal pag= e (no dma > > > or funny caching attributes) just use add_to_swap_cache()? If it's = something > > > else, try alloc a page with relevant gfp attributes, copy and > > > add_to_swap_cache()? Or perhaps that doesn't work well from a shrin= ker > > > either? > > So before we toss everything and go an a great rewrite-the-world tour= , > > what if we just try to split up big objects. So for objects which are > > bigger than e.g. 10mb > >=20 > > - move them to a special "under eviction" list > > - keep a note how far we evicted thus far > > - interleave allocating shmem pages, copying data and releasing the t= tm > > backing store on a chunk basis (maybe 10mb or whatever, tuning tbh= ) > >=20 > > If that's not enough, occasionally break out of the shrinker entirely= so > > other parts of reclaim can reclaim the shmem stuff. But just releasin= g our > > own pages as we go should help a lot I think. >=20 > Yeah, the later is exactly what I was currently prototyping. >=20 > I just didn't used a limit but rather a only partially evicted BOs list > which is used when we fail to allocate a page. >=20 > For the 5.12 cycle I think we should just go back to a hard 50% limit f= or > now and then resurrect this when we have solved the issues. Can we do the 50% limit without tossing out all the code we've done thus far? Just so this doesn't get too disruptive. -Daniel --=20 Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch