From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A56DC32771 for ; Wed, 21 Sep 2022 21:53:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97A8594000B; Wed, 21 Sep 2022 17:53:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 902AE940008; Wed, 21 Sep 2022 17:53:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 707E694000B; Wed, 21 Sep 2022 17:53:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5903E940008 for ; Wed, 21 Sep 2022 17:53:33 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 358EF1212A8 for ; Wed, 21 Sep 2022 21:53:33 +0000 (UTC) X-FDA: 79937444706.19.D3439A1 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf16.hostedemail.com (Postfix) with ESMTP id E353918000B for ; Wed, 21 Sep 2022 21:53:32 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id w10so6108222pll.11 for ; Wed, 21 Sep 2022 14:53:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=NQRKtndsBQVthqZ/KWoSsGVPFqnEuTZgHG0gsudzTz4=; b=5ce3QwpZ49NppFQNZRZZcJZM8WdPKY26XL+i90+YYAVYhw9g5ugHVk/6segFceMuUS Umrk//LVcZ8L03BX85lF1I16W6X7UP3hGV4+2g4JvVGPvMArqFxdCq985NCE4rneI5oB jZpk1Z32geuua2LmATIdzyfUz0a9c5F2EAqaTNyo9S+QvEuaVV2cjU3dxZrRqNJ92gvV WStvSGiM2JhUMcDKUi1mIji6Sbrv3LzojrQfrufyJ79ySD7hS1n9O5p1cbvFTWarKcc1 cm2S3JQSAZ8TZH4/qIh2WPsXLGH0EQz6TNhEURNP4Uy4lhQdl4ud5uUzj3zFbsoy8EYU Sfhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=NQRKtndsBQVthqZ/KWoSsGVPFqnEuTZgHG0gsudzTz4=; b=bkmWspRqLE7L5pRAx9REG32I+2dPMZgsqDGezhrvjpACupAzC0iGN5vlFJWtCvFNVG h3eAn9bMWbtuxtmKEjkYJL4FyC0wyG2Fc33/m+e6gNrHppRldeF7I6gtErmZ4uDTRjkq j/0Yr3IdUY/0SuNvUjIKegVMN1z1TuuSW0KfhMGH5Q0C3cX8udQpUKHnjUtH8A764siq NlQhhAVYvXaBzOG4sKEaeAwaMOOx3ReIZs2Ko/nxxp6gI52EEBNTIhI8zBN1aJ2c8rGs cjas5ZSZEbEsYriZkZ4N8ZpM2tnaCQpN6ebxIS30cEWnhznwelRPBYIIh/wtPZ1cmi2w ZTNQ== X-Gm-Message-State: ACrzQf1wKN+alrokywyPBHLNe+rrsp+awRSDAYgbpcT1iRsSM19UWIH2 hlaWyu7i97SXOaQ9t6y11UTudQ== X-Google-Smtp-Source: AMsMyM7aIgVABcJGRmpYEU74+kJijT4CbG3mh4JZ89HaWzsi9IRKIn8rfo/xoZPJvKDUVvJ3bS9fvg== X-Received: by 2002:a17:90b:3c90:b0:203:bf90:f78b with SMTP id pv16-20020a17090b3c9000b00203bf90f78bmr6997208pjb.138.1663797211990; Wed, 21 Sep 2022 14:53:31 -0700 (PDT) Received: from stillson.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id k7-20020aa79727000000b005484d133127sm2634536pfg.129.2022.09.21.14.53.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 14:53:31 -0700 (PDT) From: Chris Stillson To: Cc: Greentime Hu , Han-Kuan Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , Eric Biederman , Kees Cook , Anup Patel , Atish Patra , Oleg Nesterov , Heinrich Schuchardt , Guo Ren , Conor Dooley , Arnaud Pouliquen , Chris Stillson , Paolo Bonzini , Qinglin Pan , Alexandre Ghiti , Arnd Bergmann , Vincent Chen , Heiko Stuebner , Dao Lu , Jisheng Zhang , Sunil V L , Li Zhengyu , Alexander Graf , Ard Biesheuvel , Tsukasa OI , Yury Norov , "Paul E. McKenney" , Nicolas Saenz Julienne , Mark Rutland , Frederic Weisbecker , Changbin Du , Vitaly Wool , Myrtle Shah , Catalin Marinas , Will Deacon , Mark Brown , Huacai Chen , Alexey Dobriyan , Janosch Frank , Christian Brauner , Colin Cross , Eugene Syromiatnikov , Peter Collingbourne , Andrew Morton , Suren Baghdasaryan , Barret Rhoden , Davidlohr Bueso , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org Subject: [PATCH v12 13/17] riscv: Add vector extension XOR implementation Date: Wed, 21 Sep 2022 14:43:55 -0700 Message-Id: <20220921214439.1491510-13-stillson@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220921214439.1491510-1-stillson@rivosinc.com> References: <20220921214439.1491510-1-stillson@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663797212; a=rsa-sha256; cv=none; b=KQbG9IVdoKgfc218nQM49KHleJ/YuztLngSqnOoCW16/oVTLnrecIBZrx/jEPZ02LNk3ia Y9QkBNXR60YHlqwf9LtDOIipUOEq/5mHIrBjjgVuiIo9Uzr9USNlDgQNC22Vkj4pm7oLgY 2xq054QFDbZ1TQTP4tx7N+KqmLR9lUg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=rivosinc-com.20210112.gappssmtp.com header.s=20210112 header.b=5ce3QwpZ; dmarc=none; spf=pass (imf16.hostedemail.com: domain of stillson@rivosinc.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=stillson@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663797212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NQRKtndsBQVthqZ/KWoSsGVPFqnEuTZgHG0gsudzTz4=; b=vb8ahMupA5e6QmqdEmP2iwET4qYheh3oojvSs8JNah3an9n0NRzeBgYNcgkTSLt2JitxeA gsAjONpOdOspjiw9uftmtja3xvtcz9Rpdi4okEbkXlRK0H9cg3wVCkqrE8lrbfwgWweDMW 1p00VXMFS8v2IbFp0Gq08sNYe+1qwJc= X-Stat-Signature: pd3qo37dzy7gzq4mubrx97ie4861teba X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=rivosinc-com.20210112.gappssmtp.com header.s=20210112 header.b=5ce3QwpZ; dmarc=none; spf=pass (imf16.hostedemail.com: domain of stillson@rivosinc.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=stillson@rivosinc.com X-Rspamd-Queue-Id: E353918000B X-Rspamd-Server: rspam04 X-HE-Tag: 1663797212-131825 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu --- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..d1f2eeb14afb --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_VECTOR +#include +#include + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +static void xor_rvv_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_rvv_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_rvv_end(); +} + +static void xor_rvv_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_rvv_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_rvv_end(); +} + +static void xor_rvv_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_rvv_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_rvv_end(); +} + +static void xor_rvv_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_rvv_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_rvv_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_rvv_2, + .do_3 = xor_rvv_3, + .do_4 = xor_rvv_4, + .do_5 = xor_rvv_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 25d5c9664e57..acd87ac86d24 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -7,3 +7,4 @@ lib-$(CONFIG_MMU) += uaccess.o lib-$(CONFIG_64BIT) += tishift.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_VECTOR) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..3bc059e18171 --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +ENTRY(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +ENTRY(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +ENTRY(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +ENTRY(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) -- 2.25.1