From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28CAEC43334 for ; Wed, 15 Jun 2022 14:02:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0CC66B0071; Wed, 15 Jun 2022 10:02:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BBC06B0072; Wed, 15 Jun 2022 10:02:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 884016B0073; Wed, 15 Jun 2022 10:02:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7A3FA6B0071 for ; Wed, 15 Jun 2022 10:02:37 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 5307E80F56 for ; Wed, 15 Jun 2022 14:02:37 +0000 (UTC) X-FDA: 79580635554.12.D616E9E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id B5281400BD for ; Wed, 15 Jun 2022 14:02:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1655301754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HZS4DJmuD0Oig/XR1uUnL7+FP6te8eTyWXHDcizxyNM=; b=TYQmjwI7UK1joN7rQpmA8oZ9/6Us+KZ8RL1vtR785IS2TiylgNQ5Xf3egiTFxgTrD9vUCO SZ8aPEn+RwlUv22dgpag43JGOFS6/0WN2cdo24Knr+s9qHWuOaJJojFXt23W7mbkQH2Yc5 JOu2wVMa5CLBUiseczsQuEaHdIodm9M= Received: from mail-il1-f197.google.com (mail-il1-f197.google.com [209.85.166.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-6p8arfhGOsGK-3cZdioB8w-1; Wed, 15 Jun 2022 10:02:33 -0400 X-MC-Unique: 6p8arfhGOsGK-3cZdioB8w-1 Received: by mail-il1-f197.google.com with SMTP id j18-20020a056e02219200b002d3aff22b4cso8468612ila.9 for ; Wed, 15 Jun 2022 07:02:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-transfer-encoding; bh=HZS4DJmuD0Oig/XR1uUnL7+FP6te8eTyWXHDcizxyNM=; b=HZwUn0YI54cGmn8yqJMU7NtU4E4Ly1wVBTfIGWFYpTd9I6UASznz2JOgf8zSUv6E+L 7VJx4YeCYzMNqSRmyNZSWKnBm+XnjSoiuJZM+61Ltwu4/YAgzQ0I3Wu4Y7QE1xhbwWrk Fe6YmTprkHe0P0OSjyHsXVJbhPSiwPBrMH8D9bj6oXe9Wo9QlqDVx6xBWX0A3faCjhe8 WZk5EYskz6207+vUiwsizlzrSgV/UoLeBcGvOV0qp2/CZVz6BTULi5XHAEq9xjiGkfn7 879uH+Ybn7vcH4AyDusqzKRyO2GHNrm/nSq68z5lJb6UNAWhpaI9307REk6kBxM3r3Y/ cPYg== X-Gm-Message-State: AJIora/XFQap0TFExFdNGhw6Hxc3S8KcXYPDzZINBvTUd2+Fv3Eu3Frv MwYywzuSGW0F2DovWxO5SUjWWk22sUSKLM08/IbU/yz8odKjxCt1RRJUBEku1W+wq06P8XSVeAf wNmerm09KQu8= X-Received: by 2002:a05:6638:3284:b0:335:b861:a6aa with SMTP id f4-20020a056638328400b00335b861a6aamr3241497jav.19.1655301751869; Wed, 15 Jun 2022 07:02:31 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tvO0iOQCgXp+koCGK1WOiWGDNDUEGhXfoma0D3gy7sUXh9ITqvEwnxJ/rKadWbissgPktcpg== X-Received: by 2002:a05:6638:3284:b0:335:b861:a6aa with SMTP id f4-20020a056638328400b00335b861a6aamr3241471jav.19.1655301751567; Wed, 15 Jun 2022 07:02:31 -0700 (PDT) Received: from redhat.com ([38.15.36.239]) by smtp.gmail.com with ESMTPSA id d29-20020a023f1d000000b0032e5205f4e7sm3011368jaa.4.2022.06.15.07.02.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 07:02:30 -0700 (PDT) Date: Wed, 15 Jun 2022 08:02:28 -0600 From: Alex Williamson To: Yishai Hadas Cc: , jason Gunthorpe , maor Gottlieb , "kvm@vger.kernel.org" , , Subject: Re: Bug report: vfio over kernel 5.19 - mm area Message-ID: <20220615080228.7a5e7552.alex.williamson@redhat.com> In-Reply-To: <3391f2e5-149a-7825-f89e-8bde3c6d555d@nvidia.com> References: <3391f2e5-149a-7825-f89e-8bde3c6d555d@nvidia.com> Organization: Red Hat MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655301755; a=rsa-sha256; cv=none; b=7WccuxFN8zQKITvyem9hUCrfq5T89vD5UjTyDdqF7+nHcQjZNktqUqNNUXaWi65gDMrX5b 6elwZi1jIv4igUWmEkChYDath2McpiEBjCeIHkJ5KqDs+8dF3+XpN/sQvpT9eg3IOnaxeg BcfV2RWOYQ2yes+0kOvHW3ulQIqpdHo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYQmjwI7; spf=none (imf17.hostedemail.com: domain of alex.williamson@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=alex.williamson@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655301755; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HZS4DJmuD0Oig/XR1uUnL7+FP6te8eTyWXHDcizxyNM=; b=pYUYGi+KZUgh7C/Fbc9Dc9MUuWIgmwGdZun37PruOQYY0J7XLvQRIpUftRjtxQB/FX+j8A sEtTpNbGDYvk9BTw4TiK/bgFE0soMD3+JA9EXx7hpzTe2eyBqSVEE0b1h2PwHvMOLbVrg9 HRHffIouKSn6C+/rqox9EFAEtGMXGVs= X-Stat-Signature: ubd4enzs8nn3jbfxaj3mtganhqqhbs9x X-Rspamd-Queue-Id: B5281400BD Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYQmjwI7; spf=none (imf17.hostedemail.com: domain of alex.williamson@redhat.com has no SPF policy when checking 170.10.129.124) smtp.mailfrom=alex.williamson@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1655301754-814968 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 15 Jun 2022 13:52:10 +0300 Yishai Hadas wrote: > Adding some extra relevant people from the MM area. >=20 > On 15/06/2022 13:43, Yishai Hadas wrote: > > Hi All, > > > > Any idea what could cause the below break in 5.19 ? we run QEMU and=20 > > immediately the machine is stuck. > > > > Once I run, echo l > /proc/sysrq-trigger could see the below task=20 > > which seems to be stuck.. > > > > This basic flow worked fine in 5.18. Spent Friday bisecting this and posted this fix: https://lore.kernel.org/all/165490039431.944052.12458624139225785964.stgit@= omen/ I expect you're hotting the same. Thanks, Alex > > > > [1162.056583] NMI backtrace for cpu 4 > > [ 1162.056585] CPU: 4 PID: 1979 Comm: qemu-system-x86 Not tainted=20 > > 5.19.0-rc1 #747 > > [ 1162.056587] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),=20 > > BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014 > > [ 1162.056588] RIP: 0010:pmd_huge+0x0/0x20 > > [ 1162.056592] Code: 49 89 44 24 28 48 8b 47 30 49 89 44 24 30 31 c0=20 > > 41 5c c3 5b b8 01 00 00 00 5d 41 5c c3 cc cc cc cc cc cc cc cc cc cc=20 > > cc cc cc <0f> 1f 44 00 00 31 c0 48 f7 c7 9f ff ff ff 74 0f 81 e7 81 00= =20 > > 00 00 > > [ 1162.056594] RSP: 0018:ffff888146253b38 EFLAGS: 00000202 > > [ 1162.056596] RAX: ffff888101461980 RBX: ffff888146253bc0 RCX:=20 > > 000ffffffffff000 > > [ 1162.056597] RDX: ffff88814fa22000 RSI: 00007f9f68231000 RDI:=20 > > 000000010a6b6067 > > [ 1162.056598] RBP: ffff888111b90dc0 R08: 000000000002f424 R09:=20 > > 0000000000000001 > > [ 1162.056599] R10: ffffffff825c2a40 R11: 0000000000000a08 R12:=20 > > ffff88814fa22a08 > > [ 1162.056600] R13: 000000010a6b6067 R14: 0000000000052202 R15:=20 > > 00007f9f68231000 > > [ 1162.056602] FS:=C2=A0 00007f9f6c228c40(0000) GS:ffff88885f900000(000= 0)=20 > > knlGS:0000000000000000 > > [ 1162.056605] CS:=C2=A0 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 1162.056606] CR2: 00005643994fd0ed CR3: 00000001496da005 CR4:=20 > > 0000000000372ea0 > > [ 1162.056607] DR0: 0000000000000000 DR1: 0000000000000000 DR2:=20 > > 0000000000000000 > > [ 1162.056609] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:=20 > > 0000000000000400 > > [ 1162.056610] Call Trace: > > [ 1162.056611]=C2=A0 > > [ 1162.056611]=C2=A0 follow_page_mask+0x196/0x5e0 > > [ 1162.056615]=C2=A0 __get_user_pages+0x190/0x5d0 > > [ 1162.056617]=C2=A0 ? flush_workqueue_prep_pwqs+0x110/0x110 > > [ 1162.056620]=C2=A0 __gup_longterm_locked+0xaf/0x470 > > [ 1162.056624]=C2=A0 vaddr_get_pfns+0x8e/0x240 [vfio_iommu_type1] > > [ 1162.056628]=C2=A0 ? qi_flush_iotlb+0x83/0xa0 > > [ 1162.056631]=C2=A0 vfio_pin_pages_remote+0x326/0x460 [vfio_iommu_type= 1] > > [ 1162.056634]=C2=A0 vfio_iommu_type1_ioctl+0x421/0x14f0 [vfio_iommu_ty= pe1] > > [ 1162.056638]=C2=A0 __x64_sys_ioctl+0x3e4/0x8e0 > > [ 1162.056641]=C2=A0 do_syscall_64+0x3d/0x90 > > [ 1162.056644]=C2=A0 entry_SYSCALL_64_after_hwframe+0x46/0xb0 > > [ 1162.056646] RIP: 0033:0x7f9f6d14317b > > [ 1162.056648] Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00=20 > > 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00=20 > > 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89= =20 > > 01 48 > > [ 1162.056650] RSP: 002b:00007fff4fca15b8 EFLAGS: 00000246 ORIG_RAX:=20 > > 0000000000000010 > > [ 1162.056652] RAX: ffffffffffffffda RBX: 0000000000000001 RCX:=20 > > 00007f9f6d14317b > > [ 1162.056653] RDX: 00007fff4fca1620 RSI: 0000000000003b71 RDI:=20 > > 000000000000001c > > [ 1162.056654] RBP: 00007fff4fca1650 R08: 0000000000000001 R09:=20 > > 0000000000000000 > > [ 1162.056655] R10: 0000000100000000 R11: 0000000000000246 R12:=20 > > 0000000000000000 > > [ 1162.056656] R13: 0000000000000000 R14: 0000000000000000 R15:=20 > > 0000000000000000 > > [ 1162.056657]=C2=A0 > > > > Yishai > > =20 >=20