From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E27CC5475B for ; Wed, 6 Mar 2024 19:02:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 300566B009F; Wed, 6 Mar 2024 14:02:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B0BF6B00A0; Wed, 6 Mar 2024 14:02:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 151BE6B00A3; Wed, 6 Mar 2024 14:02:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 012146B009F for ; Wed, 6 Mar 2024 14:01:59 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BF9A5A1251 for ; Wed, 6 Mar 2024 19:01:59 +0000 (UTC) X-FDA: 81867533958.09.06AD705 Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf27.hostedemail.com (Postfix) with ESMTP id C8C8C4001D for ; Wed, 6 Mar 2024 19:01:56 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=X5nQQx6k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709751717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1DzzN/+Bl7F0FgkUNIWenV+wkEKVb5/vjtwB59oUvlM=; b=yYoNviD7cAlUS5ri9DkBOt6M7zbrRRqKQ6fJbwmQEqA+bjbxqqzmztVld04IOk0srXb+su rw82oMmEZLAZRDF9DnDWqR/okoBPsAh7LXLTXmsX0LBaH6ZQb6ELfoSkk1gVpLtbapjsoX 8xIaWkfBgJ+EyFXwatbHInCNoKqdYGA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=X5nQQx6k; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709751717; a=rsa-sha256; cv=none; b=beib/7SjgDAFNK4mqoZm+jn134cA4207ITpZjIz7Wcf1eF7msV/uFnxofBXBIqssZnWRti ECWcUTNbuVch9OGdDbO19OqF/qzU2BwqcrxipudvJOSFkzoIyB4p6LSzTzLBbsdCDBHvJV W31YTRFrbzWe+lR6zSRPzDauz/8mDxk= Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-5134f830488so18408e87.0 for ; Wed, 06 Mar 2024 11:01:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709751715; x=1710356515; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=1DzzN/+Bl7F0FgkUNIWenV+wkEKVb5/vjtwB59oUvlM=; b=X5nQQx6k4dngnA3Sf0eFoRWoj7K9J40wNKuXM5lZpx0N3ySgKwSB2QNMLm3NdF3kwp Aqch1H7WoMnOmPWeQCVGZjOnFb13NaRGwXbUoAGU7mWabjTWQl2pkf5S3P8NCFAFUD5v ORIvP5UNQ0Jm9Fw3yctdFG4qrfdXyG7Ho3VdZHH/5Kkhn8NQsj2+iqeVm/x6Sf9/Hmey fJiQOiCk9R7nMQ7fxZmehw/9LZOlSzVbnpLRW7Bnteizi4w7Q8KDE3hBctU8pzGjSv5L 2tqG9hLJkzLNH9Orx+hMUe0N0vDeMAmtoG3dcw7PlCTzupDSSyqSjOOjf7nA+CnXJu7n PhBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709751715; x=1710356515; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=1DzzN/+Bl7F0FgkUNIWenV+wkEKVb5/vjtwB59oUvlM=; b=Qh61BgJg64AI8CqfYr6YbxQi60t4xhRspYZ3uR0hW0bKbizhlGkhWiV3wt2UqqJu6c Puhll6aqRCyPf1F5cvT61uaNtfSqmAKDdQ4WltaapvN1/2T75qF7zMCjHXKkysMVMAc0 psyytklNmZbq5J3+s4N0AKJFbjvSw5PkjpSSwgzwVzEV5degy8q2Tw1E5b+rwJuQrPtn J+RdT46xGGKnWuIaR4W+BSwhC+s3sEOX8MYoIuV5UYSqCdbSHSuJ62r7VqbOsegpPY+h aWdg4HKRtBfbFhM4darbt4WAsvWq0zLRT0m6eQx5Be/cHkkuwHVjWudTwVI5WFkqj0Fl h7Aw== X-Forwarded-Encrypted: i=1; AJvYcCXaROeILdA4aQ0mn8phI4txekqs2WNQTVqsQ40pgrVgTLvY2hPK7Z+1VoiU16dAySKtZAmAfckMxBb2RygxCk0kt8I= X-Gm-Message-State: AOJu0Yyy3xrHvPZszHi1YE1T3Kq230uu0HU3UO8Dr00ONfWuH+KiXfMW oDA+E0LYOSXQtPXb5zhOiZ0mjacH+EXPCMPVWZMgwt+qyiBEA/01 X-Google-Smtp-Source: AGHT+IHqjfr3rLF8+moPk8plkK0aVWvITPbCAgYR3U3rANvK3Y8nSvNheIi6e/5q5c1oduqCLPaZZw== X-Received: by 2002:a05:6512:e81:b0:513:341a:91ca with SMTP id bi1-20020a0565120e8100b00513341a91camr4280778lfb.63.1709751714464; Wed, 06 Mar 2024 11:01:54 -0800 (PST) Received: from pc636 (host-90-235-19-15.mobileonline.telia.com. [90.235.19.15]) by smtp.gmail.com with ESMTPSA id x11-20020a19e00b000000b005133b381a5csm2046730lfg.90.2024.03.06.11.01.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Mar 2024 11:01:54 -0800 (PST) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Wed, 6 Mar 2024 20:01:51 +0100 To: rulinhuang Cc: urezki@gmail.com, bhe@redhat.com, akpm@linux-foundation.org, colin.king@intel.com, hch@infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, lstoakes@gmail.com, tianyou.li@intel.com, tim.c.chen@intel.com, wangyang.guo@intel.com, zhiguo.zhou@intel.com Subject: Re: [PATCH v7 1/2] mm/vmalloc: Moved macros with no functional change happened Message-ID: References: <20240301155417.1852290-1-rulin.huang@intel.com> <20240301155417.1852290-2-rulin.huang@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240301155417.1852290-2-rulin.huang@intel.com> X-Rspamd-Queue-Id: C8C8C4001D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 8f84crw7n7f6tjge55qezrpp6q97gaa4 X-HE-Tag: 1709751716-361458 X-HE-Meta: U2FsdGVkX19i9IrjE9CN7nk07lCJAW+zbSYP+fYNUO8fOzelqYbE9iAF1BgO4ggkp4f005nxvsPLnSVbFDbizXJZbzYtyuEi51oSBn19jgwh7A1wSe+Z4yk5lqeBDMTVVi/SLbXCmLDoQ3H+VW5y+O4U2cqRAe7ct9BIMunKyrtgFeSn0rrqvJtKY0eCXndGTgciHCucF25Aeqrf7lnolbLBNPAfteWdVlkFawey1euaUzEUagM0L/w3ddeKq2R0EngIXG6r5HuRPyQQiISOtmmxRlbvaW8AtKZMBlwRuU8KfSCXTpODaPGPL/oCMzLPboxKkPsb8Twn4+GA6zRL8xfi8CfqhFoE8ZwMM/vhX9EF6jDxtu3Oadi3bYdChCu4Va1qkL5APkew9gCL6bnOXcQaqFa5R284O54Ubu4E9laYcfigYYNQ+S8VKSKxcZpk4EANAD/F4Rz0In6JZivkZB5q/S3LBfv084sKJryQ7B6vB5AxWyV3Uo0q4+KocCmcPbfX5SfMrFwJSa3HWN0d3MkQnvSgEVzWAoI15NKQY8p49SoIh9dDS9fFd11aCj6N/mkXzUaS6VwE46s0ggLH8CEpejsQ/hSGbyAnf56Y3o/gizEzwLSxNJC6hikuXxfS4hTmCCjwUWiYRrKdJLicyTcZqrj83cmaAIi3smA5BUcYifrWSW0W/idgQ5o8fzQNQnMG9E/cF2mcKhBq/nqgjRegfnrm7QY6saT9eBgPFPT+o5S1Zzo9O7mQc5S8AHFLpj1b+3dZqPzKnGWgFn2RplJIEzM+FG6k4AXOwxbTkZwyQVh3ksy6jovD0Q56DlhUP8YUwtc64xbrS8WJYdjYHHT0jZdjK/tRI6t9sO4TM7c4tswqVBFx4cKQ3xds0xQxmv6IzpBfThDM67C0b4ZizfDhMfxU73ftzetAZNvcRHjFkxB21jvyjqzuKY/fnfTxouw/f9TppM7p+9FtogD vMkXL6l+ /y723xTlLDd1qIrIZTJDsFrd++/Ds3fiMA5p8l1vjGJK74KQSgNyv9f0xPDklFdvcx9NUWFRXy3yMdN8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Mar 01, 2024 at 10:54:16AM -0500, rulinhuang wrote: > Moved data structures and basic helpers related to per cpu kva allocator > up too to along with these macros with no functional change happened. > > Signed-off-by: rulinhuang > --- > V6 -> V7: Adjusted the macros > --- > mm/vmalloc.c | 262 +++++++++++++++++++++++++-------------------------- > 1 file changed, 131 insertions(+), 131 deletions(-) > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > index 25a8df497255..fc027a61c12e 100644 > --- a/mm/vmalloc.c > +++ b/mm/vmalloc.c > @@ -887,6 +887,137 @@ is_vn_id_valid(unsigned int node_id) > return false; > } > > +/* > + * vmap space is limited especially on 32 bit architectures. Ensure there is > + * room for at least 16 percpu vmap blocks per CPU. > + */ > +/* > + * If we had a constant VMALLOC_START and VMALLOC_END, we'd like to be able > + * to #define VMALLOC_SPACE (VMALLOC_END-VMALLOC_START). Guess > + * instead (we just need a rough idea) > + */ > +#if BITS_PER_LONG == 32 > +#define VMALLOC_SPACE (128UL*1024*1024) > +#else > +#define VMALLOC_SPACE (128UL*1024*1024*1024) > +#endif > + > +#define VMALLOC_PAGES (VMALLOC_SPACE / PAGE_SIZE) > +#define VMAP_MAX_ALLOC BITS_PER_LONG /* 256K with 4K pages */ > +#define VMAP_BBMAP_BITS_MAX 1024 /* 4MB with 4K pages */ > +#define VMAP_BBMAP_BITS_MIN (VMAP_MAX_ALLOC*2) > +#define VMAP_MIN(x, y) ((x) < (y) ? (x) : (y)) /* can't use min() */ > +#define VMAP_MAX(x, y) ((x) > (y) ? (x) : (y)) /* can't use max() */ > +#define VMAP_BBMAP_BITS \ > + VMAP_MIN(VMAP_BBMAP_BITS_MAX, \ > + VMAP_MAX(VMAP_BBMAP_BITS_MIN, \ > + VMALLOC_PAGES / roundup_pow_of_two(NR_CPUS) / 16)) > + > +#define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) > + > +/* > + * Purge threshold to prevent overeager purging of fragmented blocks for > + * regular operations: Purge if vb->free is less than 1/4 of the capacity. > + */ > +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) > + > +#define VMAP_RAM 0x1 /* indicates vm_map_ram area*/ > +#define VMAP_BLOCK 0x2 /* mark out the vmap_block sub-type*/ > +#define VMAP_FLAGS_MASK 0x3 > + > +struct vmap_block_queue { > + spinlock_t lock; > + struct list_head free; > + > + /* > + * An xarray requires an extra memory dynamically to > + * be allocated. If it is an issue, we can use rb-tree > + * instead. > + */ > + struct xarray vmap_blocks; > +}; > + > +struct vmap_block { > + spinlock_t lock; > + struct vmap_area *va; > + unsigned long free, dirty; > + DECLARE_BITMAP(used_map, VMAP_BBMAP_BITS); > + unsigned long dirty_min, dirty_max; /*< dirty range */ > + struct list_head free_list; > + struct rcu_head rcu_head; > + struct list_head purge; > +}; > + > +/* Queue of free and dirty vmap blocks, for allocation and flushing purposes */ > +static DEFINE_PER_CPU(struct vmap_block_queue, vmap_block_queue); > + > +/* > + * In order to fast access to any "vmap_block" associated with a > + * specific address, we use a hash. > + * > + * A per-cpu vmap_block_queue is used in both ways, to serialize > + * an access to free block chains among CPUs(alloc path) and it > + * also acts as a vmap_block hash(alloc/free paths). It means we > + * overload it, since we already have the per-cpu array which is > + * used as a hash table. When used as a hash a 'cpu' passed to > + * per_cpu() is not actually a CPU but rather a hash index. > + * > + * A hash function is addr_to_vb_xa() which hashes any address > + * to a specific index(in a hash) it belongs to. This then uses a > + * per_cpu() macro to access an array with generated index. > + * > + * An example: > + * > + * CPU_1 CPU_2 CPU_0 > + * | | | > + * V V V > + * 0 10 20 30 40 50 60 > + * |------|------|------|------|------|------|... > + * CPU0 CPU1 CPU2 CPU0 CPU1 CPU2 > + * > + * - CPU_1 invokes vm_unmap_ram(6), 6 belongs to CPU0 zone, thus > + * it access: CPU0/INDEX0 -> vmap_blocks -> xa_lock; > + * > + * - CPU_2 invokes vm_unmap_ram(11), 11 belongs to CPU1 zone, thus > + * it access: CPU1/INDEX1 -> vmap_blocks -> xa_lock; > + * > + * - CPU_0 invokes vm_unmap_ram(20), 20 belongs to CPU2 zone, thus > + * it access: CPU2/INDEX2 -> vmap_blocks -> xa_lock. > + * > + * This technique almost always avoids lock contention on insert/remove, > + * however xarray spinlocks protect against any contention that remains. > + */ > +static struct xarray * > +addr_to_vb_xa(unsigned long addr) > +{ > + int index = (addr / VMAP_BLOCK_SIZE) % num_possible_cpus(); > + > + return &per_cpu(vmap_block_queue, index).vmap_blocks; > +} > + > +/* > + * We should probably have a fallback mechanism to allocate virtual memory > + * out of partially filled vmap blocks. However vmap block sizing should be > + * fairly reasonable according to the vmalloc size, so it shouldn't be a > + * big problem. > + */ > + > +static unsigned long addr_to_vb_idx(unsigned long addr) > +{ > + addr -= VMALLOC_START & ~(VMAP_BLOCK_SIZE-1); > + addr /= VMAP_BLOCK_SIZE; > + return addr; > +} > + > +static void *vmap_block_vaddr(unsigned long va_start, unsigned long pages_off) > +{ > + unsigned long addr; > + > + addr = va_start + (pages_off << PAGE_SHIFT); > + BUG_ON(addr_to_vb_idx(addr) != addr_to_vb_idx(va_start)); > + return (void *)addr; > +} > + > static __always_inline unsigned long > va_size(struct vmap_area *va) > { > @@ -2327,137 +2458,6 @@ static struct vmap_area *find_unlink_vmap_area(unsigned long addr) > > /*** Per cpu kva allocator ***/ > > -/* > - * vmap space is limited especially on 32 bit architectures. Ensure there is > - * room for at least 16 percpu vmap blocks per CPU. > - */ > -/* > - * If we had a constant VMALLOC_START and VMALLOC_END, we'd like to be able > - * to #define VMALLOC_SPACE (VMALLOC_END-VMALLOC_START). Guess > - * instead (we just need a rough idea) > - */ > -#if BITS_PER_LONG == 32 > -#define VMALLOC_SPACE (128UL*1024*1024) > -#else > -#define VMALLOC_SPACE (128UL*1024*1024*1024) > -#endif > - > -#define VMALLOC_PAGES (VMALLOC_SPACE / PAGE_SIZE) > -#define VMAP_MAX_ALLOC BITS_PER_LONG /* 256K with 4K pages */ > -#define VMAP_BBMAP_BITS_MAX 1024 /* 4MB with 4K pages */ > -#define VMAP_BBMAP_BITS_MIN (VMAP_MAX_ALLOC*2) > -#define VMAP_MIN(x, y) ((x) < (y) ? (x) : (y)) /* can't use min() */ > -#define VMAP_MAX(x, y) ((x) > (y) ? (x) : (y)) /* can't use max() */ > -#define VMAP_BBMAP_BITS \ > - VMAP_MIN(VMAP_BBMAP_BITS_MAX, \ > - VMAP_MAX(VMAP_BBMAP_BITS_MIN, \ > - VMALLOC_PAGES / roundup_pow_of_two(NR_CPUS) / 16)) > - > -#define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) > - > -/* > - * Purge threshold to prevent overeager purging of fragmented blocks for > - * regular operations: Purge if vb->free is less than 1/4 of the capacity. > - */ > -#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) > - > -#define VMAP_RAM 0x1 /* indicates vm_map_ram area*/ > -#define VMAP_BLOCK 0x2 /* mark out the vmap_block sub-type*/ > -#define VMAP_FLAGS_MASK 0x3 > - > -struct vmap_block_queue { > - spinlock_t lock; > - struct list_head free; > - > - /* > - * An xarray requires an extra memory dynamically to > - * be allocated. If it is an issue, we can use rb-tree > - * instead. > - */ > - struct xarray vmap_blocks; > -}; > - > -struct vmap_block { > - spinlock_t lock; > - struct vmap_area *va; > - unsigned long free, dirty; > - DECLARE_BITMAP(used_map, VMAP_BBMAP_BITS); > - unsigned long dirty_min, dirty_max; /*< dirty range */ > - struct list_head free_list; > - struct rcu_head rcu_head; > - struct list_head purge; > -}; > - > -/* Queue of free and dirty vmap blocks, for allocation and flushing purposes */ > -static DEFINE_PER_CPU(struct vmap_block_queue, vmap_block_queue); > - > -/* > - * In order to fast access to any "vmap_block" associated with a > - * specific address, we use a hash. > - * > - * A per-cpu vmap_block_queue is used in both ways, to serialize > - * an access to free block chains among CPUs(alloc path) and it > - * also acts as a vmap_block hash(alloc/free paths). It means we > - * overload it, since we already have the per-cpu array which is > - * used as a hash table. When used as a hash a 'cpu' passed to > - * per_cpu() is not actually a CPU but rather a hash index. > - * > - * A hash function is addr_to_vb_xa() which hashes any address > - * to a specific index(in a hash) it belongs to. This then uses a > - * per_cpu() macro to access an array with generated index. > - * > - * An example: > - * > - * CPU_1 CPU_2 CPU_0 > - * | | | > - * V V V > - * 0 10 20 30 40 50 60 > - * |------|------|------|------|------|------|... > - * CPU0 CPU1 CPU2 CPU0 CPU1 CPU2 > - * > - * - CPU_1 invokes vm_unmap_ram(6), 6 belongs to CPU0 zone, thus > - * it access: CPU0/INDEX0 -> vmap_blocks -> xa_lock; > - * > - * - CPU_2 invokes vm_unmap_ram(11), 11 belongs to CPU1 zone, thus > - * it access: CPU1/INDEX1 -> vmap_blocks -> xa_lock; > - * > - * - CPU_0 invokes vm_unmap_ram(20), 20 belongs to CPU2 zone, thus > - * it access: CPU2/INDEX2 -> vmap_blocks -> xa_lock. > - * > - * This technique almost always avoids lock contention on insert/remove, > - * however xarray spinlocks protect against any contention that remains. > - */ > -static struct xarray * > -addr_to_vb_xa(unsigned long addr) > -{ > - int index = (addr / VMAP_BLOCK_SIZE) % num_possible_cpus(); > - > - return &per_cpu(vmap_block_queue, index).vmap_blocks; > -} > - > -/* > - * We should probably have a fallback mechanism to allocate virtual memory > - * out of partially filled vmap blocks. However vmap block sizing should be > - * fairly reasonable according to the vmalloc size, so it shouldn't be a > - * big problem. > - */ > - > -static unsigned long addr_to_vb_idx(unsigned long addr) > -{ > - addr -= VMALLOC_START & ~(VMAP_BLOCK_SIZE-1); > - addr /= VMAP_BLOCK_SIZE; > - return addr; > -} > - > -static void *vmap_block_vaddr(unsigned long va_start, unsigned long pages_off) > -{ > - unsigned long addr; > - > - addr = va_start + (pages_off << PAGE_SHIFT); > - BUG_ON(addr_to_vb_idx(addr) != addr_to_vb_idx(va_start)); > - return (void *)addr; > -} > - > /** > * new_vmap_block - allocates new vmap_block and occupies 2^order pages in this > * block. Of course pages number can't exceed VMAP_BBMAP_BITS > -- > 2.43.0 > Sorry for the late answer, i also just noticed this email. It was not in my inbox... OK, now you move part of the per-cpu allocator on the top and leave another part down making it split. This is just for the: BUG_ON(va_flags & VMAP_RAM); VMAP_RAM macro. Do we really need this BUG_ON()? -- Uladzislau Rezki