From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f198.google.com (mail-pf0-f198.google.com [209.85.192.198]) by kanga.kvack.org (Postfix) with ESMTP id B996E680FEA for ; Thu, 16 Feb 2017 08:08:11 -0500 (EST) Received: by mail-pf0-f198.google.com with SMTP id 145so23160506pfv.6 for ; Thu, 16 Feb 2017 05:08:11 -0800 (PST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0084.outbound.protection.outlook.com. [104.47.1.84]) by mx.google.com with ESMTPS id s187si6891949pgb.401.2017.02.16.05.08.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Feb 2017 05:08:10 -0800 (PST) Subject: Re: [PATCH v3 net-next 08/14] mlx4: use order-0 pages for RX References: <20170213195858.5215-1-edumazet@google.com> <20170213195858.5215-9-edumazet@google.com> <20170214131206.44b644f6@redhat.com> <1487087488.8227.53.camel@edumazet-glaptop3.roam.corp.google.com> From: Tariq Toukan Message-ID: <37bc04eb-71c9-0433-304d-87fcf8b06be3@mellanox.com> Date: Thu, 16 Feb 2017 15:08:00 +0200 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Eric Dumazet , Tariq Toukan , Jesper Dangaard Brouer Cc: Tom Herbert , Eric Dumazet , Alexander Duyck , "David S . Miller" , netdev , Martin KaFai Lau , Saeed Mahameed , Willem de Bruijn , Brenden Blanco , Alexei Starovoitov , linux-mm On 15/02/2017 6:57 PM, Eric Dumazet wrote: > On Wed, Feb 15, 2017 at 8:42 AM, Tariq Toukan wrote: >> Isn't it the same principle in page_frag_alloc() ? >> It is called form __netdev_alloc_skb()/__napi_alloc_skb(). >> >> Why is it ok to have order-3 pages (PAGE_FRAG_CACHE_MAX_ORDER) there? > This is not ok. > > This is a very well known problem, we already mentioned that here in the past, > but at least core networking stack uses order-0 pages on PowerPC. You're right, we should have done this as well in mlx4 on PPC. > mlx4 driver suffers from this problem 100% more than other drivers ;) > > One problem at a time Tariq. Right now, only mlx4 has this big problem > compared to other NIC. We _do_ agree that the series improves the driver's quality, stability, and performance in a fragmented system. But due to the late rc we're in, and the fact that we know what benchmarks our customers are going to run, we cannot Ack the series and get it as is inside kernel 4.11. We are interested to get your series merged along another perf improvement we are preparing for next rc1. This way we will earn the desired stability without breaking existing benchmarks. I think this is the right thing to do at this point of time. The idea behind the perf improvement, suggested by Jesper, is to split the napi_poll call mlx4_en_process_rx_cq() loop into two. The first loop extracts completed CQEs and starts prefetching on data and RX descriptors. The second loop process the real packets. > > Then, if we _still_ hit major issues, we might also need to force > napi_get_frags() > to allocate skb->head using kmalloc() instead of a page frag. > > That is a very simple fix. > > Remember that we have skb->truesize that is an approximation, it will > never be completely accurate, > but we need to make it better. > > mlx4 driver pretends to have a frag truesize of 1536 bytes, but this > is obviously wrong when host is under memory pressure > (2 frags per page -> truesize should be 2048) > > >> By using netdev/napi_alloc_skb, you'll get that the SKB's linear data is a >> frag of a huge page, >> and it is not going to be freed before the other non-linear frags. >> Cannot this cause the same threats (memory pinning and so...)? >> >> Currently, mlx4 doesn't use this generic API, while most other drivers do. >> >> Similar claims are true for TX: >> https://github.com/torvalds/linux/commit/5640f7685831e088fe6c2e1f863a6805962f8e81 > We do not have such problem on TX. GFP_KERNEL allocations do not have > the same issues. > > Tasks are usually not malicious in our DC, and most serious > applications use memcg or such memory control. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org