From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8EDAC77B7C for ; Thu, 3 Jul 2025 14:10:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 476C46B01A4; Thu, 3 Jul 2025 10:10:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44E246B01A5; Thu, 3 Jul 2025 10:10:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38ABC6B01A6; Thu, 3 Jul 2025 10:10:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2A6916B01A4 for ; Thu, 3 Jul 2025 10:10:53 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DFD77140278 for ; Thu, 3 Jul 2025 14:10:52 +0000 (UTC) X-FDA: 83623139544.15.7AE4DED Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.12]) by imf01.hostedemail.com (Postfix) with ESMTP id 3A1CF40002 for ; Thu, 3 Jul 2025 14:10:50 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fe+rocg8; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf01.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 198.175.65.12) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751551850; a=rsa-sha256; cv=none; b=wl4ZWqut8ERQC9scOZCu3rLwz2Tp1dHOclNGeG/DBkh/PV6d+ScJfmptLHGYPFRQtOeqZA JzqBIUzqMU2l8XSedI09T5/J8VltGR4Bgvl/yHzcH26Hk+XKCHRUhUN1brqa/3e0sUSDdn fAAN41P4OZ4LnvovvikxJLorZXftN4g= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=fe+rocg8; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf01.hostedemail.com: domain of kirill.shutemov@linux.intel.com has no SPF policy when checking 198.175.65.12) smtp.mailfrom=kirill.shutemov@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751551850; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uwXb33Go1X2BFDQheFGlTAQXabBwEu3Oa+VxLX6G3ck=; b=ne1B8kP8ZorDK0FjiV6hLB+m8t3Oiz0aJaAZRHKC/2cIZZ6wa+mOHTvwy0wF5i7vFFWzc7 VUfAGIMrdBoDiE4aMu5JrlLr5CCZ4Mm03sqnokMutnRvBOwSdtGaRfT5y7EIy5ZOoi7s4a ubxWttjjwYn309+TQxK0sw9KztJKM2U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1751551851; x=1783087851; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=UKZDj7Dr6U+nDPNDBDqz64P+s+8aLK1xb7gK+2NOHNc=; b=fe+rocg8zl62vmZOukqI26JN0rvBgbtYsI3Gk2etfs0fKFrygzEYK2Hj sPXdnEacewNeMpsZ5d5yeiu+YgPaQpuTgInIpB9t2jj727bllGe4X41e3 1/ugGhT08rifBtEdd3W72mVNeq1MU4CWJKlYutVaKUsD/FZymnwe9xqz7 J8FprnoDumKpwwTxreYSAEOI+s5PjCbf+HBf/pSuD3fyFu+O0UUMbJDMh 0nWPDMgHkQD0/SWwucuE5FfnVo3YyyncSUT6PKvBAK3fTu4zM59xOPx8H HpYIW/imSb8jR2BAJEt0q/xDytnHOsY1r6pC7llpvosLSYV8TeCBpLdxi w==; X-CSE-ConnectionGUID: IwO1R9YFSsWvo8rBKl12bw== X-CSE-MsgGUID: /DRNr4IvQNGeDXt+t+fXNw== X-IronPort-AV: E=McAfee;i="6800,10657,11483"; a="65334007" X-IronPort-AV: E=Sophos;i="6.16,284,1744095600"; d="scan'208";a="65334007" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2025 07:10:49 -0700 X-CSE-ConnectionGUID: n/f2VhrpRXW5ix8BRRZ06g== X-CSE-MsgGUID: tjOT0cYYRgu2z73konZ4hg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.16,284,1744095600"; d="scan'208";a="153792238" Received: from black.fi.intel.com ([10.237.72.28]) by orviesa010.jf.intel.com with ESMTP; 03 Jul 2025 07:10:36 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id E997D1E0; Thu, 03 Jul 2025 17:10:34 +0300 (EEST) Date: Thu, 3 Jul 2025 17:10:34 +0300 From: "Kirill A. Shutemov" To: David Laight Cc: Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Ard Biesheuvel , "Paul E. McKenney" , Josh Poimboeuf , Xiongwei Song , Xin Li , "Mike Rapoport (IBM)" , Brijesh Singh , Michael Roth , Tony Luck , Alexey Kardashevskiy , Alexander Shishkin , Jonathan Corbet , Sohil Mehta , Ingo Molnar , Pawan Gupta , Daniel Sneddon , Kai Huang , Sandipan Das , Breno Leitao , Rick Edgecombe , Alexei Starovoitov , Hou Tao , Juergen Gross , Vegard Nossum , Kees Cook , Eric Biggers , Jason Gunthorpe , "Masami Hiramatsu (Google)" , Andrew Morton , Luis Chamberlain , Yuntao Wang , Rasmus Villemoes , Christophe Leroy , Tejun Heo , Changbin Du , Huang Shijie , Geert Uytterhoeven , Namhyung Kim , Arnaldo Carvalho de Melo , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-efi@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCHv8 02/17] x86/asm: Introduce inline memcpy and memset Message-ID: References: <20250701095849.2360685-1-kirill.shutemov@linux.intel.com> <20250701095849.2360685-3-kirill.shutemov@linux.intel.com> <20250703094417.165e5893@pumpkin> <20250703131552.32adf6b8@pumpkin> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250703131552.32adf6b8@pumpkin> X-Stat-Signature: c5nikoshydqyesxy6rdwgchbqgbrwskx X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 3A1CF40002 X-Rspam-User: X-HE-Tag: 1751551850-313853 X-HE-Meta: U2FsdGVkX18FAVbWvKE9Zmo3Hsx2C/6jsXqn1R0eddEzI7fah5noLIXjBdaNafOyAT73IyAKNIyVrbXHeDmqe6uyksIsrZKaQN0TQ7neWXoBzxWejMUdiV0Zr6SXtTBsd2UmOs825se2VXjPoZ+CKOcC2BjXi9Ga++VQQkLm/HA3Ew2xSRBWaFqqEibX96ma4L0ASUabp4oOzsu0Ec6xZcm79LHKzCelD+ug92iGPuvHxk0gOz30vmkp8tk6JtHGZkoSLBOvYu5fEERrbQus0D+4MriQ/zRqDxgvwve4CjB6U4V2MBE5Sx38nRdGT+g5YbAvnTunCEI4Y6Iua/i1R/x8sj0B2YQh1celny+sP0mD29gxakJd7Usrp25pnqIL2NlJdVNoc53NgPl4ZGdQMP0i/xldNtweobipr74HjnkwVnjClG2IcFm/Pe4bQtMX10fdXwpDN8yusmvkqDx1Mx/v2USjln/FRz3wzfz16/+pXwcur8D3HLHWWjgpw59RsstV8laojP6sduYjaADh9HLhINElKzvvnwp02ndCEiFrHbPsPcPQm/VNiuKR4YvqlVXpRUfe6dRj+NsZydFg6A1p4OxAr/4L3eYtMUB4oNK5IBMXYz9C8ayTeP5qZ7zm57lULFkRaZoak+ndMMP4l+UmIpWC8+rpy4cCl2EF0fof+6KwIqIJNgGAeEHt4tiynQPDHamxGVLPF5I9NpDBriv0eiDIi962VkkFPe0TDKe6okkiY/hj1uaSB1xYjzgDrlGCKBaloqtVYw9Yqdx3QFyi8H3fc7Bs0/BTDGle9Iu37+B/plZTohzpYpP3RmnKWoiBK/10YJAM1AYCtBCvrQiX/upe9zkw3WlFB66+SRoR6H1/R9UFBBaSvlxK+OTWjYSOuop1rUxebVn2R589zUjftadoYUFZ5mGa8U2z0dsaND3PotsQNw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jul 03, 2025 at 01:15:52PM +0100, David Laight wrote: > On Thu, 3 Jul 2025 13:39:57 +0300 > "Kirill A. Shutemov" wrote: > > > On Thu, Jul 03, 2025 at 09:44:17AM +0100, David Laight wrote: > > > On Tue, 1 Jul 2025 12:58:31 +0300 > > > "Kirill A. Shutemov" wrote: > > > > > > > Extract memcpy and memset functions from copy_user_generic() and > > > > __clear_user(). > > > > > > > > They can be used as inline memcpy and memset instead of the GCC builtins > > > > whenever necessary. LASS requires them to handle text_poke. > > > > > > Except they contain the fault handlers so aren't generic calls. > > > > That's true. I will add a comment to clarify it. > > They need renaming. __inline_memcpy/memset_safe()? > ... > > > > diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S > > > > index a508e4a8c66a..47b613690f84 100644 > > > > --- a/arch/x86/lib/clear_page_64.S > > > > +++ b/arch/x86/lib/clear_page_64.S > > > > @@ -55,17 +55,26 @@ SYM_FUNC_END(clear_page_erms) > > > > EXPORT_SYMBOL_GPL(clear_page_erms) > > > > > > > > /* > > > > - * Default clear user-space. > > > > + * Default memset. > > > > * Input: > > > > * rdi destination > > > > + * rsi scratch > > > > * rcx count > > > > - * rax is zero > > > > + * al is value > > > > * > > > > * Output: > > > > * rcx: uncleared bytes or 0 if successful. > > > > + * rdx: clobbered > > > > */ > > > > SYM_FUNC_START(rep_stos_alternative) > > > > ANNOTATE_NOENDBR > > > > + > > > > + movzbq %al, %rsi > > > > + movabs $0x0101010101010101, %rax > > > > + > > > > + /* RDX:RAX = RAX * RSI */ > > > > + mulq %rsi > > > > > > NAK - you can't do that here. > > > Neither %rsi nor %rdx can be trashed. > > > The function has a very explicit calling convention. > > > > What calling convention? We change the only caller to confirm to this. > > The one that is implicit in: > > > > > + asm volatile("1:\n\t" > > > > + ALT_64("rep stosb", > > > > + "call rep_stos_alternative", ALT_NOT(X86_FEATURE_FSRM)) > > > > + "2:\n\t" > > > > + _ASM_EXTABLE_UA(1b, 2b) > > > > + : "+c" (len), "+D" (addr), ASM_CALL_CONSTRAINT > > > > + : "a" ((uint8_t)v) > > The called function is only allowed to change the registers that > 'rep stosb' uses - except it can access (but not change) > all of %rax - not just %al. > > See: https://godbolt.org/z/3fnrT3x9r > In particular note that 'do_mset' must not change %rax. > > This is very specific and is done so that the compiler can use > all the registers. Okay, I see what you are saying. > > > It is also almost certainly a waste of time. > > > Pretty much all the calls will be for a constant 0x00. > > > Rename it all memzero() ... > > > > text_poke_memset() is not limited to zeroing. > > But you don't want the overhead of extending the constant > on all the calls - never mind reserving %rdx to do it. > Maybe define a function that requires the caller to have > done the 'dirty work' - so any code that wants memzero() > just passes zero. > Or do the multiply in the C code where it will get optimised > away for constant zero. > You do get the multiply for the 'rep stosb' case - but that > is always going to be true unless you complicate things further. The patch below seems to do the trick: compiler optimizes out the multiplication for v == 0. It would be nice to avoid it for X86_FEATURE_FSRM, but we cannot use cpu_feature_enabled() here as depends on . I cannot say I like the result. Any suggestions? diff --git a/arch/x86/include/asm/string.h b/arch/x86/include/asm/string.h index becb9ee3bc8a..c7644a6f426b 100644 --- a/arch/x86/include/asm/string.h +++ b/arch/x86/include/asm/string.h @@ -35,16 +35,27 @@ static __always_inline void *__inline_memcpy(void *to, const void *from, size_t static __always_inline void *__inline_memset(void *addr, int v, size_t len) { + unsigned long val = v; void *ret = addr; + if (IS_ENABLED(CONFIG_X86_64)) { + /* + * Fill all bytes by value in byte 0. + * + * To be used in rep_stos_alternative()i + */ + val &= 0xff; + val *= 0x0101010101010101; + } + asm volatile("1:\n\t" ALT_64("rep stosb", "call rep_stos_alternative", ALT_NOT(X86_FEATURE_FSRM)) "2:\n\t" _ASM_EXTABLE_UA(1b, 2b) : "+c" (len), "+D" (addr), ASM_CALL_CONSTRAINT - : "a" (v) - : "memory", _ASM_SI, _ASM_DX); + : "a" (val) + : "memory"); return ret + len; } diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index 47b613690f84..3ef7d796deb3 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -58,23 +58,15 @@ EXPORT_SYMBOL_GPL(clear_page_erms) * Default memset. * Input: * rdi destination - * rsi scratch * rcx count * al is value * * Output: * rcx: uncleared bytes or 0 if successful. - * rdx: clobbered */ SYM_FUNC_START(rep_stos_alternative) ANNOTATE_NOENDBR - movzbq %al, %rsi - movabs $0x0101010101010101, %rax - - /* RDX:RAX = RAX * RSI */ - mulq %rsi - cmpq $64,%rcx jae .Lunrolled -- Kiryl Shutsemau / Kirill A. Shutemov