From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C1FEC433F5 for ; Mon, 10 Oct 2022 12:19:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A28D36B0071; Mon, 10 Oct 2022 08:19:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D6756B0073; Mon, 10 Oct 2022 08:19:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 877B2900002; Mon, 10 Oct 2022 08:19:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7330E6B0071 for ; Mon, 10 Oct 2022 08:19:09 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 455A7408FE for ; Mon, 10 Oct 2022 12:19:09 +0000 (UTC) X-FDA: 80004944418.10.DCA9DA7 Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf11.hostedemail.com (Postfix) with ESMTP id B7A3C40024 for ; Mon, 10 Oct 2022 12:19:08 +0000 (UTC) Received: by mail-lf1-f51.google.com with SMTP id b2so16318561lfp.6 for ; Mon, 10 Oct 2022 05:19:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=KZwwuxlvVfPvUAtn+ynvGscHnlF0P4dALC17cMr1nmc=; b=W/IR+EqjVP5RQJ8g9MlyffuHL6gqARRh7PuVxSEzFu3hX24hbVcyQZxosH084TCS4Q 50WU/dUj0Th5KsONENjYm9oxsqUXdpBujV2f8aKDYxWNoQwHggfax6YFqYBb7vATt+Y+ 1n/w2AGSNPh3O2+PXFQQU8eeoYlUUZ0xD5sblBw180nz1lgTE4mS04ps3J/MTut++vwX G5k5FLEOrccOCM9A9KpAL8eqcw3nur8nEF20GqUoRb3ZtLkZRZ23MsrvevYs+1w6hNdC sCcaVXl4R9060dgSsDmHBPcss8/a6qlu843FbwmbPpNJ88kKg3AAUNHzbZsQpPCAObZN eBjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=KZwwuxlvVfPvUAtn+ynvGscHnlF0P4dALC17cMr1nmc=; b=xBeoz+2QHn6a6uXsJ5uW224prw1TmP6ig9Ac3vtba/vaQH6rniI951xY5kvHjh++Mk 2eEe5UxzvZP3Y2jhoUC9EWKJ3bSy5QfFilT520SIktl4F0D404iSwlRIBAZB2DNxLy2a smev2LtXS+98HXrzjUB4iXo6kqVm00y98MdqhKSWxqCfmD0K5lq4CFPJfJIU6un0q7WD a4ZMsmbGiVAj4hJevgkvvqG7KRV8p78nV0Us0NySP/yOpAP2W9MgClPQ3iZMZmu7OUat 2RsMguVd+PC3jeBYv5c9YCkrQFdqt6gUYvlXpjlDE2rF2Oby8pkcfz4x3VGx0YApvQfK XykQ== X-Gm-Message-State: ACrzQf1mhQ7OD45QJwPLRl7xzqhZbnnFdTNABt0k7C2px7UEAMhfIOFM ijHttbSQCrL0XeKfBOJcsj4= X-Google-Smtp-Source: AMsMyM5LjQmULgtrzhgJh9/ms7P5i/1urOoNxm4Xtv3Yt3h/eVKGVzUMywhot6sprSjK21zl46eu7Q== X-Received: by 2002:ac2:4ecc:0:b0:4a2:2ed2:9400 with SMTP id p12-20020ac24ecc000000b004a22ed29400mr6684979lfr.432.1665404347109; Mon, 10 Oct 2022 05:19:07 -0700 (PDT) Received: from pc636 (host-90-235-26-104.mobileonline.telia.com. [90.235.26.104]) by smtp.gmail.com with ESMTPSA id v8-20020a056512348800b00494a1e875a9sm696343lfr.191.2022.10.10.05.19.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Oct 2022 05:19:06 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Mon, 10 Oct 2022 14:19:03 +0200 To: David Hildenbrand Cc: Uladzislau Rezki , Alexander Potapenko , Andrey Konovalov , "linux-mm@kvack.org" , Andrey Ryabinin , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Subject: Re: KASAN-related VMAP allocation errors in debug kernels with many logical CPUS Message-ID: References: <8aaaeec8-14a1-cdc4-4c77-4878f4979f3e@redhat.com> <9ce8a3a3-8305-31a4-a097-3719861c234e@redhat.com> <6d75325f-a630-5ae3-5162-65f5bb51caf7@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6d75325f-a630-5ae3-5162-65f5bb51caf7@redhat.com> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665404348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KZwwuxlvVfPvUAtn+ynvGscHnlF0P4dALC17cMr1nmc=; b=wxAGbjK52APYOwgJZr8yQYWShi5rMVvTgITkRHARmPOWnWcm8gCa9ugZyyOncvObF4qO2V T+ioRaqj1BNjcf9b45FBkSLNIhCDyVoK4mqFhUmcUGZYHR8BAxTJbXejqM96SRg8yarsoG EiAVRC75E/wqcVSobSaTRRlGXzXNCmA= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="W/IR+Eqj"; spf=pass (imf11.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665404348; a=rsa-sha256; cv=none; b=SPE52m+YtzllMoUAeIR8E/pto+gUxU5IgU9bWuW9zP/jSacjqVj9Yb1W/I16uxiPhWaQgK VyX68Gzygmqn64jvpx1m5hjum74qf2jKkg57Vd/C8lNl4SzfTeCsDlzhGOJnfQlaJCH/gV +ZBS7ayXCGuyb+WPi444VihMRZ9m9x8= Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="W/IR+Eqj"; spf=pass (imf11.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: gstim965tsq1s3nt954ghz589qgjo5go X-Rspamd-Queue-Id: B7A3C40024 X-Rspamd-Server: rspam01 X-HE-Tag: 1665404348-894435 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Oct 10, 2022 at 08:56:55AM +0200, David Hildenbrand wrote: > > > > Maybe try to increase the module-section size to see if it solves the > > > > problem. > > > > > > What would be the easiest way to do that? > > > > > Sorry for late answer. I was trying to reproduce it on my box. What i > > did was trying to load all modules in my system with KASAN_INLINE option: > > > > Thanks! > > > > > #!/bin/bash > > > > # Exclude test_vmalloc.ko > > MODULES_LIST=(`find /lib/modules/$(uname -r) -type f \ > > \( -iname "*.ko" -not -iname "test_vmalloc*" \) | awk -F"/" '{print $NF}' | sed 's/.ko//'`) > > > > function moduleExist(){ > > MODULE="$1" > > if lsmod | grep "$MODULE" &> /dev/null ; then > > return 0 > > else > > return 1 > > fi > > } > > > > i=0 > > > > for module_name in ${MODULES_LIST[@]}; do > > sudo modprobe $module_name > > > > if moduleExist ${module_name}; then > > ((i=i+1)) > > echo "Successfully loaded $module_name counter $i" > > fi > > done > > > > > > as you wrote it looks like it is not easy to reproduce. So i do not see > > any vmap related errors. > > Yeah, it's quite mystery and only seems to trigger on these systems with a > lot of CPUs. > > > > > Returning back to the question. I think you could increase the MODULES_END > > address and shift the FIXADDR_START little forward. See the dump_pagetables.c > > But it might be they are pretty compact and located in the end. So i am not > > sure if there is a room there. > > That's what I was afraid of :) > > > > > Second. It would be good to understand if vmap only fails on allocating for a > > module: > > > > > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > > index dd6cdb201195..53026fdda224 100644 > > --- a/mm/vmalloc.c > > +++ b/mm/vmalloc.c > > @@ -1614,6 +1614,8 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, > > va->va_end = addr + size; > > va->vm = NULL; > > + trace_printk("-> alloc %lu size, align: %lu, vstart: %lu, vend: %lu\n", size, align, vstart, vend); > > + > > spin_lock(&vmap_area_lock); > > > > I'll try grabbing a suitable system again and add some more debugging > output. Might take a while, unfortunately. > Yes that makes sense. Especially to understand if it fails on the MODULES_VADDR - MODULES_END range or somewhere else. According to your trace output it looks like that but it would be good to confirm it by adding some traces. BTW, vmap code is lack of good trace events. Probably it is worth to add some basic ones. -- Uladzislau Rezki