Processes being killed without obvious reason or log entries

[Whoza Ran <marinko.catovic@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2501 bytes --]

hi folks


I have some issue its reasons I am currently unaware of on multiple hosts.

Currently I noticed this when trying to access a postgresql database while
doing `su postgres`. please note that I do not think that this is related
to postgresql, but the kernel.

as soon as I do `su postgres` sometimes, but not always, the su-process
responds immediately with `Killed` and does not su to postgres. this does
not happen every time, sometimes I can turn into the postgres user.
executing anything else in that shell, e.g. psql then results in `Killed`
all the time. I may be hitting some system limits or something I am not
aware of.

but if there were limits hit, like things the oomkiller would handle, this
would be logged to syslog in any case afaik, yet there are absolutely no
logs in syslog, dmesg or the journal. since psql is a perl wrapper script,
it forks lots of processes and does not make it to a forked postgresql
client. so I tried this script: https://pastebin.com/6igR3pmW

as you can observe, even `date` gets killed sometimes. In this short
example at line #22

My best guess now is that there is something wrong with the logging there,
the kernel is vanilla 5.5.2, the OS is debian stretch. the ulimits are not
hit afaik, at least the usual suspects were set to unlimited, yet any log
entry should clarify what's going wrong.

also terminating all processes running as user postgres did not help. afaik
ulimits can not be the cause then, as there are no processes running as
that user. one weird thing: the database server itself, postgresql, can
start perfectly fine.

# ps -elf | grep postgres
0 S postgres   680     1  0  80   0 - 42460 -      Feb07 ?        00:00:41
/usr/lib/postgresql/9.6/bin/postgres [... some args]
1 S postgres   681   680  0  80   0 - 34755 -      Feb07 ?        00:00:02
postgres: logger process
1 S postgres   684   680  0  80   0 - 42518 -      Feb07 ?        00:00:17
postgres: checkpointer process
1 S postgres   685   680  0  80   0 - 42460 -      Feb07 ?        00:00:14
postgres: writer process
1 S postgres   686   680  0  80   0 - 42460 -      Feb07 ?        00:00:15
postgres: wal writer process
1 S postgres   687   680  0  80   0 - 38389 -      Feb07 ?        00:00:12
postgres: stats collector process

what am I missing? could this be some issue with the memory management
killing processes but not logging things?
Also, should that help: strace psql - https://pastebin.com/7w2muHDc which
is being killed as +++ killed by SIGKILL +++

[-- Attachment #2: Type: text/html, Size: 3252 bytes --]