On 11/21/2012 05:02 PM, Fengguang Wu wrote:

On Wed, Nov 21, 2012 at 04:34:40PM +0800, Jaegeuk Hanse wrote:

Cc Fengguang Wu.

On 11/21/2012 04:13 PM, metin d wrote:

  Curious. Added linux-mm list to CC to catch more attention. If you run
echo 1 >/proc/sys/vm/drop_caches does it evict data-1 pages from memory?

I'm guessing it'd evict the entries, but am wondering if we could run any more diagnostics before trying this.

We regularly use a setup where we have two databases; one gets used frequently and the other one about once a month. It seems like the memory manager keeps unused pages in memory at the expense of frequently used database's performance.

My understanding was that under memory pressure from heavily
accessed pages, unused pages would eventually get evicted. Is there
anything else we can try on this host to understand why this is
happening?

We may debug it this way.

1) run 'fadvise data-2 0 0 dontneed' to drop data-2 cached pages
   (please double check via /proc/vmstat whether it does the expected work)

2) run 'page-types -r' with root, to view the page status for the
   remaining pages of data-1

The fadvise tool comes from Andrew Morton's ext3-tools. (source code attached)
Please compile them with options "-Dlinux -I. -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE"

page-types can be found in the kernel source tree tools/vm/page-types.c

Sorry that sounds a bit twisted.. I do have a patch to directly dump
page cache status of a user specified file, however it's not
upstreamed yet.

Hi Fengguang,

Thanks for you detail steps, I think metin can have a try.

        flags    page-count       MB  symbolic-flags            long-symbolic-flags
0x0000000000000000        607699     2373  ___________________________________   
0x0000000100000000        343227     1340  _______________________r___________    reserved

But I have some questions of page-type

Thanks,
Fengguang

On Tue 20-11-12 09:42:42, metin d wrote:

I have two PostgreSQL databases named data-1 and data-2 that sit on the
same machine. Both databases keep 40 GB of data, and the total memory
available on the machine is 68GB.

I started data-1 and data-2, and ran several queries to go over all their
data. Then, I shut down data-1 and kept issuing queries against data-2.
For some reason, the OS still holds on to large parts of data-1's pages
in its page cache, and reserves about 35 GB of RAM to data-2's files. As
a result, my queries on data-2 keep hitting disk.

I'm checking page cache usage with fincore. When I run a table scan query
against data-2, I see that data-2's pages get evicted and put back into
the cache in a round-robin manner. Nothing happens to data-1's pages,
although they haven't been touched for days.

Does anybody know why data-1's pages aren't evicted from the page cache?
I'm open to all kind of suggestions you think it might relate to problem.

  Curious. Added linux-mm list to CC to catch more attention. If you run
echo 1 >/proc/sys/vm/drop_caches
  does it evict data-1 pages from memory?

This is an EC2 m2.4xlarge instance on Amazon with 68 GB of RAM and no
swap space. The kernel version is:

$ uname -r
3.2.28-45.62.amzn1.x86_64
Edit:

and it seems that I use one NUMA instance, if  you think that it can a problem.

$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 70007 MB
node 0 free: 360 MB
node distances:
node   0
   0:  10