From: Glauber Costa <glommer@parallels.com>
To: linux-mm@kvack.org
Cc: linux-fsdevel@vger.kernel.org,
containers@lists.linux-foundation.org,
Michal Hocko <mhocko@suse.cz>,
Johannes Weiner <hannes@cmpxchg.org>,
kamezawa.hiroyu@jp.fujitsu.com,
Andrew Morton <akpm@linux-foundation.org>,
Dave Shrinnker <david@fromorbit.com>,
Greg Thelen <gthelen@google.com>,
hughd@google.com, yinghan@google.com
Subject: [PATCH v2 00/28] memcg-aware slab shrinking
Date: Fri, 29 Mar 2013 13:13:42 +0400 [thread overview]
Message-ID: <1364548450-28254-1-git-send-email-glommer@parallels.com> (raw)
Hi,
Notes:
======
This is v2 of memcg-aware LRU shrinking. I've been testing it extensively
and it behaves well, at least from the isolation point of view. However,
I feel some more testing is needed before we commit to it. Still, this is
doing the job fairly well. Comments welcome.
Base work:
==========
Please note that this builds upon the recent work from Dave Chinner that
sanitizes the LRU shrinking API and make the shrinkers node aware. Node
awareness is not *strictly* needed for my work, but I still perceive it
as an advantage. The API unification is a major need, and I build upon it
heavily. That allows us to manipulate the LRUs without knowledge of the
underlying objects with ease. This time, I am including that work here as
a baseline.
Description:
============
This patchset implements targeted shrinking for memcg when kmem limits are
present. So far, we've been accounting kernel objects but failing allocations
when short of memory. This is because our only option would be to call the
global shrinker, depleting objects from all caches and breaking isolation.
The main idea is to associate per-memcg lists with each of the LRUs. The main
LRU still provides a single entry point and when adding or removing an element
from the LRU, we use the page information to figure out which memcg it belongs
to and relay it to the right list.
Patches:
========
1 and 2: improve handling of small number of shrinkable objects.
This is a scenario that is way more likely to appear under memcg,
although it is not memcg-specific. Already sent separately, but not
yet merged.
3 to 20: Dave's work to unify the LRU API and make it per-node. I had to make
minor changes to the patches to reflect new code in the tree and due to
build problems. I tried to keep them as unchanged as possible.
21 to 28: memcg targeted shrinking.
Main changes from *v1:
* merged comments from the mailing list
* reworked lru-memcg API
* effective proportional shrinking
* sanitized locking on the memcg side
* bill user memory first when kmem == umem
* various bugfixes
TODO:
* shrink dead memcgs when global pressure kicks in. (minor)
Dave Chinner (17):
dcache: convert dentry_stat.nr_unused to per-cpu counters
dentry: move to per-sb LRU locks
dcache: remove dentries from LRU before putting on dispose list
mm: new shrinker API
shrinker: convert superblock shrinkers to new API
list: add a new LRU list type
inode: convert inode lru list to generic lru list code.
dcache: convert to use new lru list infrastructure
list_lru: per-node list infrastructure
shrinker: add node awareness
fs: convert inode and dentry shrinking to be node aware
xfs: convert buftarg LRU to generic code
xfs: convert dquot cache lru to list_lru
fs: convert fs shrinkers to new scan/count API
drivers: convert shrinkers to new count/scan API
shrinker: convert remaining shrinkers to count/scan API
shrinker: Kill old ->shrink API.
Glauber Costa (11):
super: fix calculation of shrinkable objects for small numbers
vmscan: take at least one pass with shrinkers
hugepage: convert huge zero page shrinker to new shrinker API
vmscan: also shrink slab in memcg pressure
memcg,list_lru: duplicate LRUs upon kmemcg creation
lru: add an element to a memcg list
list_lru: also include memcg lists in counts and scans
list_lru: per-memcg walks
memcg: per-memcg kmem shrinking
list_lru: reclaim proportionaly between memcgs and nodes
super: targeted memcg reclaim
arch/x86/kvm/mmu.c | 35 ++-
drivers/gpu/drm/i915/i915_dma.c | 4 +-
drivers/gpu/drm/i915/i915_drv.h | 2 +-
drivers/gpu/drm/i915/i915_gem.c | 69 +++--
drivers/gpu/drm/i915/i915_gem_evict.c | 10 +-
drivers/gpu/drm/i915/i915_gem_execbuffer.c | 2 +-
drivers/gpu/drm/ttm/ttm_page_alloc.c | 48 ++-
drivers/gpu/drm/ttm/ttm_page_alloc_dma.c | 55 ++--
drivers/md/dm-bufio.c | 65 ++--
drivers/staging/android/ashmem.c | 44 ++-
drivers/staging/android/lowmemorykiller.c | 40 ++-
drivers/staging/zcache/zcache-main.c | 29 +-
fs/dcache.c | 215 +++++++------
fs/drop_caches.c | 1 +
fs/ext4/extents_status.c | 30 +-
fs/gfs2/glock.c | 30 +-
fs/gfs2/main.c | 3 +-
fs/gfs2/quota.c | 14 +-
fs/gfs2/quota.h | 4 +-
fs/inode.c | 174 +++++------
fs/internal.h | 5 +
fs/mbcache.c | 53 ++--
fs/nfs/dir.c | 20 +-
fs/nfs/internal.h | 4 +-
fs/nfs/super.c | 3 +-
fs/nfsd/nfscache.c | 31 +-
fs/quota/dquot.c | 39 ++-
fs/super.c | 107 ++++---
fs/ubifs/shrinker.c | 20 +-
fs/ubifs/super.c | 3 +-
fs/ubifs/ubifs.h | 3 +-
fs/xfs/xfs_buf.c | 167 +++++-----
fs/xfs/xfs_buf.h | 5 +-
fs/xfs/xfs_dquot.c | 7 +-
fs/xfs/xfs_icache.c | 4 +-
fs/xfs/xfs_icache.h | 2 +-
fs/xfs/xfs_qm.c | 274 +++++++++--------
fs/xfs/xfs_qm.h | 4 +-
fs/xfs/xfs_super.c | 12 +-
include/linux/dcache.h | 4 +
include/linux/fs.h | 25 +-
include/linux/list_lru.h | 112 +++++++
include/linux/memcontrol.h | 41 +++
include/linux/shrinker.h | 45 ++-
include/linux/swap.h | 2 +
include/trace/events/vmscan.h | 4 +-
lib/Makefile | 2 +-
lib/list_lru.c | 474 +++++++++++++++++++++++++++++
mm/huge_memory.c | 18 +-
mm/memcontrol.c | 373 ++++++++++++++++++++---
mm/memory-failure.c | 2 +
mm/slab_common.c | 1 -
mm/vmscan.c | 142 ++++++---
net/sunrpc/auth.c | 45 ++-
54 files changed, 2091 insertions(+), 836 deletions(-)
create mode 100644 include/linux/list_lru.h
create mode 100644 lib/list_lru.c
--
1.8.1.4
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next reply other threads:[~2013-03-29 9:13 UTC|newest]
Thread overview: 97+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-03-29 9:13 Glauber Costa [this message]
2013-03-29 9:13 ` [PATCH v2 01/28] super: fix calculation of shrinkable objects for small numbers Glauber Costa
2013-04-01 7:16 ` Kamezawa Hiroyuki
2013-03-29 9:13 ` [PATCH v2 02/28] vmscan: take at least one pass with shrinkers Glauber Costa
2013-04-01 7:26 ` Kamezawa Hiroyuki
2013-04-01 8:10 ` Glauber Costa
2013-04-10 5:09 ` Ric Mason
2013-04-10 7:32 ` Glauber Costa
2013-04-10 9:19 ` Dave Chinner
2013-04-08 8:42 ` Joonsoo Kim
2013-04-08 8:47 ` Glauber Costa
2013-04-08 9:01 ` Joonsoo Kim
2013-04-08 9:05 ` Glauber Costa
2013-04-09 0:55 ` Joonsoo Kim
2013-04-09 1:29 ` Dave Chinner
2013-04-09 2:05 ` Joonsoo Kim
2013-04-09 7:43 ` Glauber Costa
2013-04-09 9:08 ` Joonsoo Kim
2013-04-09 12:30 ` Dave Chinner
2013-04-10 2:51 ` Joonsoo Kim
2013-04-10 7:30 ` Glauber Costa
2013-04-10 8:19 ` Joonsoo Kim
2013-04-10 8:46 ` Wanpeng Li
2013-04-10 8:46 ` Wanpeng Li
2013-04-10 10:07 ` Dave Chinner
2013-04-10 14:03 ` JoonSoo Kim
2013-04-11 0:41 ` Dave Chinner
2013-04-11 7:27 ` Wanpeng Li
2013-04-11 9:25 ` Dave Chinner
2013-04-11 7:27 ` Wanpeng Li
2013-04-11 7:27 ` Wanpeng Li
[not found] ` <20130410025115.GA5872-Hm3cg6mZ9cc@public.gmane.org>
2013-04-10 8:46 ` Wanpeng Li
2013-03-29 9:13 ` [PATCH v2 03/28] dcache: convert dentry_stat.nr_unused to per-cpu counters Glauber Costa
2013-04-05 1:09 ` Greg Thelen
2013-04-05 1:15 ` Dave Chinner
2013-04-08 9:14 ` Glauber Costa
2013-04-08 13:18 ` Glauber Costa
2013-04-08 23:26 ` Dave Chinner
2013-04-09 8:02 ` Glauber Costa
2013-04-09 12:47 ` Dave Chinner
2013-03-29 9:13 ` [PATCH v2 04/28] dentry: move to per-sb LRU locks Glauber Costa
2013-03-29 9:13 ` [PATCH v2 05/28] dcache: remove dentries from LRU before putting on dispose list Glauber Costa
2013-04-03 6:51 ` Sha Zhengju
2013-04-03 8:55 ` Glauber Costa
2013-04-04 6:19 ` Dave Chinner
2013-04-04 6:56 ` Glauber Costa
2013-03-29 9:13 ` [PATCH v2 06/28] mm: new shrinker API Glauber Costa
2013-04-05 1:09 ` Greg Thelen
2013-03-29 9:13 ` [PATCH v2 07/28] shrinker: convert superblock shrinkers to new API Glauber Costa
2013-03-29 9:13 ` [PATCH v2 08/28] list: add a new LRU list type Glauber Costa
2013-04-04 21:53 ` Greg Thelen
2013-04-05 1:20 ` Dave Chinner
2013-04-05 8:01 ` Glauber Costa
2013-04-06 0:04 ` Dave Chinner
2013-03-29 9:13 ` [PATCH v2 09/28] inode: convert inode lru list to generic lru list code Glauber Costa
2013-03-29 9:13 ` [PATCH v2 10/28] dcache: convert to use new lru list infrastructure Glauber Costa
2013-04-08 13:14 ` Glauber Costa
2013-04-08 23:28 ` Dave Chinner
2013-03-29 9:13 ` [PATCH v2 11/28] list_lru: per-node " Glauber Costa
2013-03-29 9:13 ` [PATCH v2 12/28] shrinker: add node awareness Glauber Costa
2013-03-29 9:13 ` [PATCH v2 13/28] fs: convert inode and dentry shrinking to be node aware Glauber Costa
2013-03-29 9:13 ` [PATCH v2 14/28] xfs: convert buftarg LRU to generic code Glauber Costa
2013-03-29 9:13 ` [PATCH v2 15/28] xfs: convert dquot cache lru to list_lru Glauber Costa
2013-03-29 9:13 ` [PATCH v2 16/28] fs: convert fs shrinkers to new scan/count API Glauber Costa
2013-03-29 9:13 ` [PATCH v2 17/28] drivers: convert shrinkers to new count/scan API Glauber Costa
2013-03-29 9:14 ` [PATCH v2 18/28] shrinker: convert remaining shrinkers to " Glauber Costa
2013-03-29 9:14 ` [PATCH v2 19/28] hugepage: convert huge zero page shrinker to new shrinker API Glauber Costa
2013-03-29 9:14 ` [PATCH v2 20/28] shrinker: Kill old ->shrink API Glauber Costa
2013-03-29 9:14 ` [PATCH v2 21/28] vmscan: also shrink slab in memcg pressure Glauber Costa
2013-04-01 7:46 ` Kamezawa Hiroyuki
2013-04-01 8:51 ` Glauber Costa
2013-04-03 10:11 ` Sha Zhengju
2013-04-03 10:43 ` Glauber Costa
2013-04-04 9:35 ` Sha Zhengju
2013-04-05 8:25 ` Glauber Costa
2013-03-29 9:14 ` [PATCH v2 22/28] memcg,list_lru: duplicate LRUs upon kmemcg creation Glauber Costa
2013-04-01 8:05 ` Kamezawa Hiroyuki
2013-04-01 8:22 ` Glauber Costa
2013-03-29 9:14 ` [PATCH v2 23/28] lru: add an element to a memcg list Glauber Costa
2013-04-01 8:18 ` Kamezawa Hiroyuki
2013-04-01 8:29 ` Glauber Costa
2013-03-29 9:14 ` [PATCH v2 24/28] list_lru: also include memcg lists in counts and scans Glauber Costa
2013-03-29 9:14 ` [PATCH v2 25/28] list_lru: per-memcg walks Glauber Costa
2013-03-29 9:14 ` [PATCH v2 26/28] memcg: per-memcg kmem shrinking Glauber Costa
2013-04-01 8:31 ` Kamezawa Hiroyuki
2013-04-01 8:48 ` Glauber Costa
2013-04-01 9:01 ` Kamezawa Hiroyuki
2013-04-01 9:14 ` Glauber Costa
2013-04-01 9:35 ` Kamezawa Hiroyuki
2013-03-29 9:14 ` [PATCH v2 27/28] list_lru: reclaim proportionaly between memcgs and nodes Glauber Costa
2013-03-29 9:14 ` [PATCH v2 28/28] super: targeted memcg reclaim Glauber Costa
2013-04-01 12:38 ` [PATCH v2 00/28] memcg-aware slab shrinking Serge Hallyn
2013-04-01 12:45 ` Glauber Costa
2013-04-01 14:12 ` Serge Hallyn
2013-04-08 8:11 ` Glauber Costa
2013-04-02 4:58 ` Dave Chinner
2013-04-02 7:55 ` Glauber Costa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1364548450-28254-1-git-send-email-glommer@parallels.com \
--to=glommer@parallels.com \
--cc=akpm@linux-foundation.org \
--cc=containers@lists.linux-foundation.org \
--cc=david@fromorbit.com \
--cc=gthelen@google.com \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.cz \
--cc=yinghan@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox