Re: [Bug 196729] New: System becomes unresponsive when swapping - Regression since 4.10.x

[Michal Hocko <mhocko@kernel.org>]

On Thu 24-08-17 00:30:40, Steven Haigh wrote:
> On Wednesday, 23 August 2017 11:38:48 PM AEST Michal Hocko wrote:
> > On Tue 22-08-17 15:55:30, Andrew Morton wrote:
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > 
> > > On Tue, 22 Aug 2017 11:17:08 +0000 bugzilla-daemon@bugzilla.kernel.org 
> wrote:
> > [...]
> > 
> > > Sadly I haven't been able to capture this information
> > > 
> > > > fully yet due to said unresponsiveness.
> > 
> > Please try to collect /proc/vmstat in the bacground and provide the
> > collected data. Something like
> > 
> > while true
> > do
> > 	cp /proc/vmstat > vmstat.$(date +%s)
> > 	sleep 1s
> > done
> > 
> > If the system turns out so busy that it won't be able to fork a process
> > or write the output (which you will see by checking timestamps of files
> > and looking for holes) then you can try the attached proggy
> > ./read_vmstat output_file timeout output_size
> > 
> > Note you might need to increase the mlock rlimit to lock everything into
> > memory.
> 
> Thanks Michal,
> 
> I have upgraded PCs since I initially put together this data - however I was 
> able to get strange behaviour by pulling out an 8Gb RAM stick in my new system 
> - leaving it with only 8Gb of RAM.
> 
> All these tests are performed with Fedora 26 and kernel 4.12.8-300.fc26.x86_64
> 
> I have attached 3 files with output.
> 
> 8Gb-noswap.tar.gz contains the output of /proc/vmstat running on 8Gb of RAM 
> with no swap. Under this scenario, I was expecting the OOM reaper to just kill 
> the game when memory allocated became too high for the amount of physical RAM. 
> Interestingly, you'll notice a massive hang in the output before the game is 
> terminated. I didn't see this before.

I have checked few gaps. E.g. vmstat.1503496391 vmstat.1503496451 which
is one minute. The most notable thing is that there are only very few
pagecache pages
			[base]		[diff]
nr_active_file  	1641    	3345
nr_inactive_file        1630    	4787

So there is not much to reclaim without swap. The more important thing
is that we keep reclaiming and refaulting that memory

workingset_activate     5905591 	1616391
workingset_refault      33412538        10302135
pgactivate      	42279686        13219593
pgdeactivate    	48175757        14833350

pgscan_kswapd   	379431778       126407849
pgsteal_kswapd  	49751559        13322930

so we are effectivelly trashing over the very small amount of
reclaimable memory. This is something that we cannot detect right now.
It is even questionable whether the OOM killer would be an appropriate
action. Your system has recovered and then it is always hard to decide
whether a disruptive action is more appropriate. One minute of
unresponsiveness is certainly annoying though. Your system is obviously
under provisioned to load you want to run obviously.

It is quite interesting to see that we do not really have too many
direct reclaimers during this time period
allocstall_normal       30      	1
allocstall_movable      490     	88
pgscan_direct_throttle  0       	0
pgsteal_direct  	24434   	4069
pgscan_direct   	38678   	5868
 
> 8Gb-swap-on-file.tar.gz contains the output of /proc/vmstat still with 8Gb of 
> RAM - but creating a file with swap on the PCIe SSD /swapfile with size 8Gb 
> via:
> 	# dd if=/dev/zero of=/swapfile bs=1G count=8
> 	# mkswap /swapfile
> 	# swapon /swapfile
> 
> Some times (all in UTC+10):
> 23:58:30 - Start loading the saved game
> 23:59:38 - Load ok, all running fine
> 00:00:15 - Load Chrome
> 00:01:00 - Quit the game
> 
> The game seemed to run ok with no real issue - and a lot was swapped to the 
> swap file. I'm wondering if it was purely the speed of the PCIe SSD that 
> caused this appearance - as the creation of the file with dd completed at 
> ~1.4GB/sec.

Swap IO tends to be really scattered and the IO performance is not really
great even on a fast storage AFAIK.
 
Anyway your original report sounded like a regression. Were you able to
run the _same_ workload on an older kernel without these issues?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>