From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B08EAC433DB for ; Fri, 8 Jan 2021 14:39:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0685523884 for ; Fri, 8 Jan 2021 14:39:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0685523884 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0E1416B0130; Fri, 8 Jan 2021 09:39:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 06CB16B0132; Fri, 8 Jan 2021 09:39:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E76266B0134; Fri, 8 Jan 2021 09:39:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0187.hostedemail.com [216.40.44.187]) by kanga.kvack.org (Postfix) with ESMTP id CD6D56B0130 for ; Fri, 8 Jan 2021 09:39:08 -0500 (EST) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8A5254410 for ; Fri, 8 Jan 2021 14:39:08 +0000 (UTC) X-FDA: 77682865176.26.air67_350c5eb274f3 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 633C81804A300 for ; Fri, 8 Jan 2021 14:39:08 +0000 (UTC) X-HE-Tag: air67_350c5eb274f3 X-Filterd-Recvd-Size: 6440 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Fri, 8 Jan 2021 14:39:07 +0000 (UTC) Received: by mail-wr1-f43.google.com with SMTP id d26so9196881wrb.12 for ; Fri, 08 Jan 2021 06:39:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=XOO4wVb3Y58B3RJRLwkxGLbFOdHHzra+zEXjO8FjfrI=; b=Ql/1LeIpfqgWmTYmOAAt44Dlbt9O802bQuS0Bb3JZwlQpGSC1OjV36OX8tc+lPSB3j lPfzFVRdsFAiDAyJqC1ChfVFHCrCmDLIDrRHj5zVllmFGtA7K1NWneSNntRdOq2Hl6pa HRLSfiYvmKi8JMHjc3Se8A/DPE3l8QCJpvJmE22nU0Aec1BBQhGv6Z6r2AX4qqB1xPVc UgnXRX6M3EWLZ4js8tvil0phVcuA987AcgpD59GQmEDwgDTov6x8kwvD4KlQ40tTIIuc lBBwGKjWuBcdm4Qpg8pERsP4JT9cMyDMMfK8PoiKgnEOSYPUoSy/nixyvr838KtWD8ZH iPFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=XOO4wVb3Y58B3RJRLwkxGLbFOdHHzra+zEXjO8FjfrI=; b=SOc+C2ejrI0HlvA+VSdhsj+/kQJTh8tdj88+CoJCZn0+u0XzGq+yQbetUr1VZKpGGZ m4Jg50Lk1dYbZQ6zX7IzFAl0hYsNf5qJA+yKyZGRJgo606sOc7h5PSWsuK6w4Ac5ZACW eEJLYJbfQfIIlUtAUXQDUVIA0wQMC/LF/eutXEdkSeR2D53arRVWzm0Oc23vlycr8GP7 kqKYhbHVQyD9TepdsM6BM0P5ItsxcfqTWZVv8wa1SkY/6oHpZQYNYZgm8iazpgFw10/G n77OZkb/9yiTt2owJF9kqC0WwCm7k3nXZqURMSw1Hnqgr7eSVoXalxiEFywuF6YJsF2u WPdw== X-Gm-Message-State: AOAM532Sc0sBaypy3nukheVvCwlISWQoVF082eOMtGLSh53g8GhFxZc3 J6sLbsbYfF6wMWG34LRWrPI= X-Google-Smtp-Source: ABdhPJweKZ8Sy6xZY9JR/1EpSH6+u+nm+aIQbkTWEFv1MpDKYPLlXOcF7f/eHIye7n7DKDTEZMl6pQ== X-Received: by 2002:adf:eb05:: with SMTP id s5mr3958014wrn.333.1610116746557; Fri, 08 Jan 2021 06:39:06 -0800 (PST) Received: from [192.168.2.27] (39.35.broadband4.iol.cz. [85.71.35.39]) by smtp.gmail.com with ESMTPSA id y2sm11930811wma.6.2021.01.08.06.39.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 08 Jan 2021 06:39:05 -0800 (PST) Subject: Re: Very slow unlockall() To: Michal Hocko Cc: linux-mm@kvack.org, Linux Kernel Mailing List , Mikulas Patocka References: <70885d37-62b7-748b-29df-9e94f3291736@gmail.com> <20210108134140.GA9883@dhcp22.suse.cz> From: Milan Broz Message-ID: Date: Fri, 8 Jan 2021 15:39:04 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <20210108134140.GA9883@dhcp22.suse.cz> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 08/01/2021 14:41, Michal Hocko wrote: > On Wed 06-01-21 16:20:15, Milan Broz wrote: >> Hi, >> >> we use mlockall(MCL_CURRENT | MCL_FUTURE) / munlockall() in cryptsetup code >> and someone tried to use it with hardened memory allocator library. >> >> Execution time was increased to extreme (minutes) and as we found, the problem >> is in munlockall(). >> >> Here is a plain reproducer for the core without any external code - it takes >> unlocking on Fedora rawhide kernel more than 30 seconds! >> I can reproduce it on 5.10 kernels and Linus' git. >> >> The reproducer below tries to mmap large amount memory with PROT_NONE (later never used). >> The real code of course does something more useful but the problem is the same. >> >> #include >> #include >> #include >> #include >> >> int main (int argc, char *argv[]) >> { >> void *p = mmap(NULL, 1UL << 41, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); >> >> if (p == MAP_FAILED) return 1; >> >> if (mlockall(MCL_CURRENT | MCL_FUTURE)) return 1; >> printf("locked\n"); >> >> if (munlockall()) return 1; >> printf("unlocked\n"); >> >> return 0; >> } >> >> In traceback I see that time is spent in munlock_vma_pages_range. >> >> [ 2962.006813] Call Trace: >> [ 2962.006814] ? munlock_vma_pages_range+0xe7/0x4b0 >> [ 2962.006814] ? vma_merge+0xf3/0x3c0 >> [ 2962.006815] ? mlock_fixup+0x111/0x190 >> [ 2962.006815] ? apply_mlockall_flags+0xa7/0x110 >> [ 2962.006816] ? __do_sys_munlockall+0x2e/0x60 >> [ 2962.006816] ? do_syscall_64+0x33/0x40 >> ... >> >> Or with perf, I see >> >> # Overhead Command Shared Object Symbol >> # ........ ....... ................. ..................................... >> # >> 48.18% lock [kernel.kallsyms] [k] lock_is_held_type >> 11.67% lock [kernel.kallsyms] [k] ___might_sleep >> 10.65% lock [kernel.kallsyms] [k] follow_page_mask >> 9.17% lock [kernel.kallsyms] [k] debug_lockdep_rcu_enabled >> 6.73% lock [kernel.kallsyms] [k] munlock_vma_pages_range >> ... >> >> >> Could please anyone check what's wrong here with the memory locking code? >> Running it on my notebook I can effectively DoS the system :) >> >> Original report is https://gitlab.com/cryptsetup/cryptsetup/-/issues/617 >> but this is apparently a kernel issue, just amplified by usage of munlockall(). > > Which kernel version do you see this with? Have older releases worked > better? Hi, I tried 5.10 stable and randomly few kernels I have built on testing VM (5.3 was the oldest), it seems to be very similar run time, so the problem is apparently old...(I can test some specific kernel version if it make any sense). For mainline (reproducer above): With 5.11.0-0.rc2.20210106git36bbbd0e234d.117.fc34.x86_64 (latest Fedora rawhide kernel build - many debug options are on) # time ./lock locked unlocked real 0m32.287s user 0m0.001s sys 0m32.126s Today's Linus git - 5.11.0-rc2+ in my testing x86_64 VM (no extensive kernel debug options): # time ./lock locked unlocked real 0m4.172s user 0m0.000s sys 0m4.172s m.