From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73446C83F04 for ; Thu, 3 Jul 2025 17:02:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C6EC98E000A; Thu, 3 Jul 2025 13:02:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C1F1B8E0009; Thu, 3 Jul 2025 13:02:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE6DE8E000A; Thu, 3 Jul 2025 13:02:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9937C8E0009 for ; Thu, 3 Jul 2025 13:02:21 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3FD70106771 for ; Thu, 3 Jul 2025 17:02:21 +0000 (UTC) X-FDA: 83623571682.09.21832C5 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf19.hostedemail.com (Postfix) with ESMTP id 2CF471A0018 for ; Thu, 3 Jul 2025 17:02:18 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fF+ubSD9; spf=pass (imf19.hostedemail.com: domain of david.laight.linux@gmail.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=david.laight.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751562139; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pqSUbsPjahwSUAB6AZ/GxcwqbVye734kyCehbMAYMyw=; b=oaKeQceY2sjUsHK24jC9oh2+knaCVXgQp+/IyYiUnR/92EcECiKrTbPIDh5PqxYOEEVu2z HlpC9Ni3xmVn+Zv97ajvFQV6gkN8xlTyPmjcvfIC/b2NE/hFNlSib2JmE/l7gp1DqG3ia0 tbilK85+j5XXB9dT9EAG9RO3gwV6fPA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fF+ubSD9; spf=pass (imf19.hostedemail.com: domain of david.laight.linux@gmail.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=david.laight.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751562139; a=rsa-sha256; cv=none; b=NUPAA6in3Ad9Qsq8EfUxZrPSy5tWMknJ4DBaiX48/EEbaIEGlAAKqv1B7X0Mav48CkRF5B mcm7G4f/PjkPWbaXEKoyeRu6IruvvUHCvoDIWJ20VT+9zdCzUYfRchfxeWG2WNVBIGv83P PCJQoih2UTirLloqmt7VPc1ImBmLfHc= Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-453066fad06so585475e9.2 for ; Thu, 03 Jul 2025 10:02:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1751562138; x=1752166938; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=pqSUbsPjahwSUAB6AZ/GxcwqbVye734kyCehbMAYMyw=; b=fF+ubSD9ca3OoXfKToG2lMU2Tt/MWmXdUNwhspyP+dlwl/iao1OgJVzuUA4Ji/9RZN wjk7wmyf9MMBKEF1R8Kv+LpVIveOPgHrGHr0d5mGN4DSWbNt67tWRBX/8wkmK0CMT0iv CmmebC1rrrwJDnyxTCEyi26l9tZesRLO80jCeIGP3kZmgdrmO8m0Crhi0Qi7vU44ZXmZ SKU/dznYBjjt/dZ0ig/qS3j8xvFI2w41YBwniija7qlzYRFOqh2Sze0ew/ATnmd6Yzsp jkasFfpM4qgheaaN/X3EwF5NUnDlHQ4qfbhjx79R0xjv7b/qabiPYrY/X41qL8tnwZeH mWgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751562138; x=1752166938; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pqSUbsPjahwSUAB6AZ/GxcwqbVye734kyCehbMAYMyw=; b=ehno0gM8xUgnkF3Xb/bLzZeKyXUbYP5F2gL0E0QlAR3SH2qPsxhPwXGYsJ9ljpmgqB +VpW2LCvvpf8hv68ZDhFWjREC3sdwDTJaSmDhxpYdkFggcSz8L8siONcIRduA3GTzFGe T+TfK0OgsdRPvv1NMNQ/lsFrpYsKL0L92t5QgHC1Gl5R2+0p4G5tbUb4sR1jVfbAp41g VnRFg5hNto/Uzj1mAG8UZFwRMRvCGyDD2n8LO896MFXv/Cz0TyId3GNiwNalet0zpwPN dCRlf/GNcVTFRyA+77ws9ItWcQfeNw8UJzc5gBSTqGN1ciMmWdbLIcFRE7RPxfDcqfhe WstQ== X-Forwarded-Encrypted: i=1; AJvYcCXuUlIv1/rKqkoIPyjWkUiyVuCmJ4h/P33jmJV2FV2ZWWot0BXH9K3MLJpY0YMf2BVrYTSl3YApaA==@kvack.org X-Gm-Message-State: AOJu0YwTVEvs1FiUnngKgADq9sZejJhRb/QFbpoQ9KApo0op+fHa2wW+ eyHGtkAlSuhWaO/FOjAhkNr8Pdk429gnnocvsb2nMWNBFdRGnMrZDB9E X-Gm-Gg: ASbGncvk6BPmaHfr1unPNH/smDCrUPASPz9yDsAaO0+aDhjsDbiAhzIrGGFhPfUZbfE rISm5W7vc3ZG4eYpkft3bm7iuVXrVyOlLIffyaiBKLC3DVHgeUph+gQnLbO0MjLK2ls6iRW9Rvr rs18suGCcp4Qd6Z4wJJ88NXrEdGi7Fv1535JgSTxqwb1LsFq+PYogGt7WWHovX28AMedM2kAJwM Tkj6TE9OWPdleWL1RNtDIhCiiFYSe/BdpNkSddhq1omDMfziyHPwygMJ0Us352GNYo30pAmkb+3 pcN/QHdWD3mqHm+avWqsrqyKn08hy1WOfDMAofA3dKe32aHflRaHpSmOTuSM86okOcAeGLrRn++ lHorpToTqLRI+oMVFlA== X-Google-Smtp-Source: AGHT+IHaQgjZrXhRgzZx++9Yi/fzdCBPXdZfk6poOyduS7JH7jm8D3wKUoXwoBb3dw9YTRiJNjYZkA== X-Received: by 2002:a05:6000:4b11:b0:3a3:7ba5:9618 with SMTP id ffacd0b85a97d-3b32e70d7d7mr3507915f8f.29.1751562137307; Thu, 03 Jul 2025 10:02:17 -0700 (PDT) Received: from pumpkin (host-92-21-58-28.as13285.net. [92.21.58.28]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3b4708d0ed9sm279010f8f.38.2025.07.03.10.02.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Jul 2025 10:02:16 -0700 (PDT) Date: Thu, 3 Jul 2025 18:02:13 +0100 From: David Laight To: "Kirill A. Shutemov" Cc: Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Ard Biesheuvel , "Paul E. McKenney" , Josh Poimboeuf , Xiongwei Song , Xin Li , "Mike Rapoport (IBM)" , Brijesh Singh , Michael Roth , Tony Luck , Alexey Kardashevskiy , Alexander Shishkin , Jonathan Corbet , Sohil Mehta , Ingo Molnar , Pawan Gupta , Daniel Sneddon , Kai Huang , Sandipan Das , Breno Leitao , Rick Edgecombe , Alexei Starovoitov , Hou Tao , Juergen Gross , Vegard Nossum , Kees Cook , Eric Biggers , Jason Gunthorpe , "Masami Hiramatsu (Google)" , Andrew Morton , Luis Chamberlain , Yuntao Wang , Rasmus Villemoes , Christophe Leroy , Tejun Heo , Changbin Du , Huang Shijie , Geert Uytterhoeven , Namhyung Kim , Arnaldo Carvalho de Melo , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCHv8 02/17] x86/asm: Introduce inline memcpy and memset Message-ID: <20250703180213.28c0e92e@pumpkin> In-Reply-To: References: <20250701095849.2360685-1-kirill.shutemov@linux.intel.com> <20250701095849.2360685-3-kirill.shutemov@linux.intel.com> <20250703094417.165e5893@pumpkin> <20250703131552.32adf6b8@pumpkin> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; arm-unknown-linux-gnueabihf) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Stat-Signature: hd86umirrigenf7r3yu3s3y5qganhd35 X-Rspamd-Queue-Id: 2CF471A0018 X-Rspamd-Server: rspam11 X-Rspam-User: X-HE-Tag: 1751562138-53986 X-HE-Meta: U2FsdGVkX1+WVl6DNcTtNDfsnDGRZaLS/c7ytO6FYFThitLGd0bt2tB3xfSHYCqxaf3VKr4EOGU11P6RnGIpMHn6TeXufctccsOyzd18SQ4FZgEHwuWGuxXvdQtDQSGKwd+pyNwiQhxJY4wac00uHSPnFeFebrYi0L3pJs+RwjRReIckr/m7AeNoivGSv2TQpT3PWfG2YsPDE0od9E+Rh8RU1hLt9QI2UcOoI66c7zHUtPeRMHNSelV/zx00VS+OK4vMK0hZvtsU6fgh0ZqvzkOFSrDXUxUY0ncE23Bq/Qr1hcVFGL9InQbFrnsD2vTzqcUbIb3t54AUtV0jHCJ5nGSLwMk3lBoBgdUhfJB/bGya+v/7Sr5JVwv3cjEXI/0MnGiMFJhZvh+HDs3CPVPo6i2h36+GkfjwgPZEjYj/kK6Q3Ccaq4TKdcnephvHLfOTQRGOoXs8tCf3kzRWgsrthiQSJrRJlxoZ8hp6/iqqK4DLnOgq2vCULLxODV36sI1oG68y3/phf+tqNQtSJT3n2gPUItnkaHXLkxXab+hidtkMaoQM6hAdygRCoTE7csaRpKbCsVMm/+zWn6mINfjVFlsek+7CdLG3PCMxpppGuHl9F0Lmsh7bhWJp6W9IrPGTpC19eQYEpCQhPGQ12xhBJ1CYor7eCmJGHSiOKTefnx33ShKr00KymlSafXwDukDIZrp5B6mzA/I45JMeBueIfELRYQp0vZjI36EGGgbPUzF41YS/+gBe+q4kvbmAeJPftybgwiK+KgmOqRuKlP3W91BSz4rkGX9XV3ap+gv0WGjQl7agyBCEYip8LyHw0lKB7wTrCe8hX5ytYROxmiAhmD24g3E2fXxq3v7b/aO+pVt+hdWIpOpUtSrNzRYg/ff0s2OalRExa3lVxShta9vzpuvY9YbcSRKlvdONhO8u2Irx3eeJyGs9xko0QzWE8jXtP05osXlb/yZZYfpfQ/a MhPx1lHR 6SK63yot0+2B/9ZVusTWsn/tA+JtMuP4WnH2fzYjDU/RjauIpVKQsRQZ9VKHKKxrBFM3xCCfS+/8jUQ2R53LF22ZRT4phSM9Lm2W/uEF26TjzT5AMi94a6wOAEtKp9hTf6C2XQjXVhIbc/VAjRbxa4MBvGKMVND2043TjUfkVzhrJ32j/BWjIq06NGyVG17YHE6QxmG7iOF10MZ6XSlXZWOUn6AvUd2r8jB2S0Nr415SYA2kO/36QFjQuj5MGUZ6NY/8zaBfC55y3yM7fCM7PbL3CITj2Y9iHDYEiX+O17pQioGNO0Y01rmXGp3z6jvMJZAkjm5UXCkxoXRk4fYbD9wODm7N25OJGQ4uJr9zAQR9ebZZ9i5TLp8+85WFdt69ogeACdl+3Pq/eEzE1t9APEYe4hJq20Fg1zFuvdKwZ/MemOAxf3oEOuHYIgR8Juw/Y/DVtqP861GwcTdYE0W1j5FiSLekFZzkr7o+ScKGXa01EwN1mMSg8L5XL5d+FhXS6eTzOsKSjGl0vTko/NSUJmtsp12rW4OwiRdpyfEJ9AgFvak6ovigknkBE1a6afRAcUyDqyUWFFlPVSZhXJ5LGEvJbLQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 3 Jul 2025 17:10:34 +0300 "Kirill A. Shutemov" wrote: > On Thu, Jul 03, 2025 at 01:15:52PM +0100, David Laight wrote: > > On Thu, 3 Jul 2025 13:39:57 +0300 > > "Kirill A. Shutemov" wrote: > > > > > On Thu, Jul 03, 2025 at 09:44:17AM +0100, David Laight wrote: > > > > On Tue, 1 Jul 2025 12:58:31 +0300 > > > > "Kirill A. Shutemov" wrote: > > > > > > > > > Extract memcpy and memset functions from copy_user_generic() and > > > > > __clear_user(). > > > > > > > > > > They can be used as inline memcpy and memset instead of the GCC builtins > > > > > whenever necessary. LASS requires them to handle text_poke. > > > > > > > > Except they contain the fault handlers so aren't generic calls. > > > > > > That's true. I will add a comment to clarify it. > > > > They need renaming. > > __inline_memcpy/memset_safe()? 'safe' against what :-) They can't be used for user accesses without access_ok() and clac. The get/put_user variants without access_ok() have _unsafe() suffix. > > > ... > > > > > diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S > > > > > index a508e4a8c66a..47b613690f84 100644 > > > > > --- a/arch/x86/lib/clear_page_64.S > > > > > +++ b/arch/x86/lib/clear_page_64.S > > > > > @@ -55,17 +55,26 @@ SYM_FUNC_END(clear_page_erms) > > > > > EXPORT_SYMBOL_GPL(clear_page_erms) > > > > > > > > > > /* > > > > > - * Default clear user-space. > > > > > + * Default memset. > > > > > * Input: > > > > > * rdi destination > > > > > + * rsi scratch > > > > > * rcx count > > > > > - * rax is zero > > > > > + * al is value > > > > > * > > > > > * Output: > > > > > * rcx: uncleared bytes or 0 if successful. > > > > > + * rdx: clobbered > > > > > */ > > > > > SYM_FUNC_START(rep_stos_alternative) > > > > > ANNOTATE_NOENDBR > > > > > + > > > > > + movzbq %al, %rsi > > > > > + movabs $0x0101010101010101, %rax > > > > > + > > > > > + /* RDX:RAX = RAX * RSI */ > > > > > + mulq %rsi > > > > > > > > NAK - you can't do that here. > > > > Neither %rsi nor %rdx can be trashed. > > > > The function has a very explicit calling convention. > > > > > > What calling convention? We change the only caller to confirm to this. > > > > The one that is implicit in: > > > > > > > + asm volatile("1:\n\t" > > > > > + ALT_64("rep stosb", > > > > > + "call rep_stos_alternative", ALT_NOT(X86_FEATURE_FSRM)) > > > > > + "2:\n\t" > > > > > + _ASM_EXTABLE_UA(1b, 2b) > > > > > + : "+c" (len), "+D" (addr), ASM_CALL_CONSTRAINT > > > > > + : "a" ((uint8_t)v) > > > > The called function is only allowed to change the registers that > > 'rep stosb' uses - except it can access (but not change) > > all of %rax - not just %al. > > > > See: https://godbolt.org/z/3fnrT3x9r > > In particular note that 'do_mset' must not change %rax. > > > > This is very specific and is done so that the compiler can use > > all the registers. > > Okay, I see what you are saying. > > > > > It is also almost certainly a waste of time. > > > > Pretty much all the calls will be for a constant 0x00. > > > > Rename it all memzero() ... > > > > > > text_poke_memset() is not limited to zeroing. > > > > But you don't want the overhead of extending the constant > > on all the calls - never mind reserving %rdx to do it. > > Maybe define a function that requires the caller to have > > done the 'dirty work' - so any code that wants memzero() > > just passes zero. > > Or do the multiply in the C code where it will get optimised > > away for constant zero. > > You do get the multiply for the 'rep stosb' case - but that > > is always going to be true unless you complicate things further. > > The patch below seems to do the trick: compiler optimizes out the > multiplication for v == 0. > > It would be nice to avoid it for X86_FEATURE_FSRM, but we cannot use > cpu_feature_enabled() here as depends on > . > > I cannot say I like the result. > > Any suggestions? > > diff --git a/arch/x86/include/asm/string.h b/arch/x86/include/asm/string.h > index becb9ee3bc8a..c7644a6f426b 100644 > --- a/arch/x86/include/asm/string.h > +++ b/arch/x86/include/asm/string.h > @@ -35,16 +35,27 @@ static __always_inline void *__inline_memcpy(void *to, const void *from, size_t > > static __always_inline void *__inline_memset(void *addr, int v, size_t len) > { > + unsigned long val = v; > void *ret = addr; > > + if (IS_ENABLED(CONFIG_X86_64)) { > + /* > + * Fill all bytes by value in byte 0. > + * > + * To be used in rep_stos_alternative()i > + */ > + val &= 0xff; > + val *= 0x0101010101010101; > + } That won't compile for 32bit, and it needs the same thing done. val *= (unsigned long)0x0101010101010101ull; should work. I don't think you need the 'val &= 0xff', just rely on the caller passing a valid value - nothing will break badly if it doesn't. David > + > asm volatile("1:\n\t" > ALT_64("rep stosb", > "call rep_stos_alternative", ALT_NOT(X86_FEATURE_FSRM)) > "2:\n\t" > _ASM_EXTABLE_UA(1b, 2b) > : "+c" (len), "+D" (addr), ASM_CALL_CONSTRAINT > - : "a" (v) > - : "memory", _ASM_SI, _ASM_DX); > + : "a" (val) > + : "memory"); > > return ret + len; > } > diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S > index 47b613690f84..3ef7d796deb3 100644 > --- a/arch/x86/lib/clear_page_64.S > +++ b/arch/x86/lib/clear_page_64.S > @@ -58,23 +58,15 @@ EXPORT_SYMBOL_GPL(clear_page_erms) > * Default memset. > * Input: > * rdi destination > - * rsi scratch > * rcx count > * al is value > * > * Output: > * rcx: uncleared bytes or 0 if successful. > - * rdx: clobbered > */ > SYM_FUNC_START(rep_stos_alternative) > ANNOTATE_NOENDBR > > - movzbq %al, %rsi > - movabs $0x0101010101010101, %rax > - > - /* RDX:RAX = RAX * RSI */ > - mulq %rsi > - > cmpq $64,%rcx > jae .Lunrolled >