From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AB2BC10F1A for ; Tue, 7 May 2024 13:55:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC6466B008C; Tue, 7 May 2024 09:55:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E769D6B0092; Tue, 7 May 2024 09:55:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D17546B0093; Tue, 7 May 2024 09:55:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B4CFB6B008C for ; Tue, 7 May 2024 09:55:38 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 42AD8C0C34 for ; Tue, 7 May 2024 13:55:38 +0000 (UTC) X-FDA: 82091747556.12.D3B87FC Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf17.hostedemail.com (Postfix) with ESMTP id BE3A34001F for ; Tue, 7 May 2024 13:55:34 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=xfGM7AWT; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Ss+M+pRo; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kM7ddcqE; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="pWJA8U/t"; dmarc=none; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715090135; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kisJu+ayWULyIY3ote/9ApIfabGyrFRB2ZIL5nsy1L8=; b=Ldt4rzEj4hKIhCj7icZ5Az1kNYCdjQp8sOC7eIDRaqgXxaY8ZPDQtAaO1r651kRhtNxY8I k2m5btYM04J65Sm1DTJZLtgOoyaKlMBNCHxdOUFAZew8BsnSP/Ng+31cUNo8+r2jON6DVk Ntzn+BOMBzZw5jYUzm1S3CfcIOyZA50= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715090135; a=rsa-sha256; cv=none; b=rxfBCUCdqUuET7rA1fqREfPv4GOjNTol0T7KWbz2yfuRlvD6t6To+uzsxwV39J1s1LYUfs mACKhxaKnc+Rs36Qm4iAcSY9U26T0Hj+jBF97n3BY1mP7k5ofBcTT0tBWtpR1C1MoS8EhI +tPikpjDETpU8LDXVu1SBaK1QT6srkM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=xfGM7AWT; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Ss+M+pRo; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=kM7ddcqE; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="pWJA8U/t"; dmarc=none; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id B9E5720AE1; Tue, 7 May 2024 13:55:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1715090133; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=kisJu+ayWULyIY3ote/9ApIfabGyrFRB2ZIL5nsy1L8=; b=xfGM7AWTssIrfUkgzEKYPUbFI7tAZWkTB8bxQ1/Hi9kbP0LMvKvuEN3EWWoYoNgU0Z48Ad K117uQ7O1wkvmMDXeR8rVPPcqbAb9fFdwZC3HG1u/rxmzX4NocT7wDiAFkrccHAZGO5jMX qzHMRbvBAL6+UjrrqtuONSsOwaTlJL4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1715090133; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=kisJu+ayWULyIY3ote/9ApIfabGyrFRB2ZIL5nsy1L8=; b=Ss+M+pRoK2m5vgHFu6kByPE4GaGndngCSHHAMLo56BOIeQ4Sz18AZFLLkaWDdFGLHkhgBj BmwH3T5snmuXqECg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1715090130; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=kisJu+ayWULyIY3ote/9ApIfabGyrFRB2ZIL5nsy1L8=; b=kM7ddcqEsJB9crRdxmNBuMoauvRguNZXE5GCt0K1XLR+hO8AVZNah7i1hFGirQwltg5H8Y RpDnyS/NHshyCjkpiL1kIEgzzOMzDgTnkJGPVMuMg97+7cjmQbUJfOT1I2CqB8SJZfXZNN VYZbf2B8n75qaDxhgd+eqpi1iPPmg2M= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1715090130; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=kisJu+ayWULyIY3ote/9ApIfabGyrFRB2ZIL5nsy1L8=; b=pWJA8U/tbTqwJXPN93nm7OYMWoVduQ7bGd2El1C6khv0ML+3jbEM49kl/7pYACEg3s+i0N rwBp7ghcfJ0XsNDA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 9B3B013A2D; Tue, 7 May 2024 13:55:30 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id e+SSJdIyOmbBMgAAD6G6ig (envelope-from ); Tue, 07 May 2024 13:55:30 +0000 Message-ID: <0e8b7482-478e-4efc-ad5f-76d60cf02bfd@suse.cz> Date: Tue, 7 May 2024 15:55:30 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/2] objpool: enable inlining objpool_push() and objpool_pop() operations Content-Language: en-US To: Andrii Nakryiko , linux-trace-kernel@vger.kernel.org, rostedt@goodmis.org, mhiramat@kernel.org Cc: bpf@vger.kernel.org, Matt Wu , "linux-mm@kvack.org" References: <20240424215214.3956041-1-andrii@kernel.org> <20240424215214.3956041-2-andrii@kernel.org> From: Vlastimil Babka Autocrypt: addr=vbabka@suse.cz; keydata= xsFNBFZdmxYBEADsw/SiUSjB0dM+vSh95UkgcHjzEVBlby/Fg+g42O7LAEkCYXi/vvq31JTB KxRWDHX0R2tgpFDXHnzZcQywawu8eSq0LxzxFNYMvtB7sV1pxYwej2qx9B75qW2plBs+7+YB 87tMFA+u+L4Z5xAzIimfLD5EKC56kJ1CsXlM8S/LHcmdD9Ctkn3trYDNnat0eoAcfPIP2OZ+ 9oe9IF/R28zmh0ifLXyJQQz5ofdj4bPf8ecEW0rhcqHfTD8k4yK0xxt3xW+6Exqp9n9bydiy tcSAw/TahjW6yrA+6JhSBv1v2tIm+itQc073zjSX8OFL51qQVzRFr7H2UQG33lw2QrvHRXqD Ot7ViKam7v0Ho9wEWiQOOZlHItOOXFphWb2yq3nzrKe45oWoSgkxKb97MVsQ+q2SYjJRBBH4 8qKhphADYxkIP6yut/eaj9ImvRUZZRi0DTc8xfnvHGTjKbJzC2xpFcY0DQbZzuwsIZ8OPJCc LM4S7mT25NE5kUTG/TKQCk922vRdGVMoLA7dIQrgXnRXtyT61sg8PG4wcfOnuWf8577aXP1x 6mzw3/jh3F+oSBHb/GcLC7mvWreJifUL2gEdssGfXhGWBo6zLS3qhgtwjay0Jl+kza1lo+Cv BB2T79D4WGdDuVa4eOrQ02TxqGN7G0Biz5ZLRSFzQSQwLn8fbwARAQABzSBWbGFzdGltaWwg QmFia2EgPHZiYWJrYUBzdXNlLmN6PsLBlAQTAQoAPgIbAwULCQgHAwUVCgkICwUWAgMBAAIe AQIXgBYhBKlA1DSZLC6OmRA9UCJPp+fMgqZkBQJkBREIBQkRadznAAoJECJPp+fMgqZkNxIQ ALZRqwdUGzqL2aeSavbum/VF/+td+nZfuH0xeWiO2w8mG0+nPd5j9ujYeHcUP1edE7uQrjOC Gs9sm8+W1xYnbClMJTsXiAV88D2btFUdU1mCXURAL9wWZ8Jsmz5ZH2V6AUszvNezsS/VIT87 AmTtj31TLDGwdxaZTSYLwAOOOtyqafOEq+gJB30RxTRE3h3G1zpO7OM9K6ysLdAlwAGYWgJJ V4JqGsQ/lyEtxxFpUCjb5Pztp7cQxhlkil0oBYHkudiG8j1U3DG8iC6rnB4yJaLphKx57NuQ PIY0Bccg+r9gIQ4XeSK2PQhdXdy3UWBr913ZQ9AI2usid3s5vabo4iBvpJNFLgUmxFnr73SJ KsRh/2OBsg1XXF/wRQGBO9vRuJUAbnaIVcmGOUogdBVS9Sun/Sy4GNA++KtFZK95U7J417/J Hub2xV6Ehc7UGW6fIvIQmzJ3zaTEfuriU1P8ayfddrAgZb25JnOW7L1zdYL8rXiezOyYZ8Fm ZyXjzWdO0RpxcUEp6GsJr11Bc4F3aae9OZtwtLL/jxc7y6pUugB00PodgnQ6CMcfR/HjXlae h2VS3zl9+tQWHu6s1R58t5BuMS2FNA58wU/IazImc/ZQA+slDBfhRDGYlExjg19UXWe/gMcl De3P1kxYPgZdGE2eZpRLIbt+rYnqQKy8UxlszsBNBFsZNTUBCACfQfpSsWJZyi+SHoRdVyX5 J6rI7okc4+b571a7RXD5UhS9dlVRVVAtrU9ANSLqPTQKGVxHrqD39XSw8hxK61pw8p90pg4G /N3iuWEvyt+t0SxDDkClnGsDyRhlUyEWYFEoBrrCizbmahOUwqkJbNMfzj5Y7n7OIJOxNRkB IBOjPdF26dMP69BwePQao1M8Acrrex9sAHYjQGyVmReRjVEtv9iG4DoTsnIR3amKVk6si4Ea X/mrapJqSCcBUVYUFH8M7bsm4CSxier5ofy8jTEa/CfvkqpKThTMCQPNZKY7hke5qEq1CBk2 wxhX48ZrJEFf1v3NuV3OimgsF2odzieNABEBAAHCwXwEGAEKACYCGwwWIQSpQNQ0mSwujpkQ PVAiT6fnzIKmZAUCZAUSmwUJDK5EZgAKCRAiT6fnzIKmZOJGEACOKABgo9wJXsbWhGWYO7mD 8R8mUyJHqbvaz+yTLnvRwfe/VwafFfDMx5GYVYzMY9TWpA8psFTKTUIIQmx2scYsRBUwm5VI EurRWKqENcDRjyo+ol59j0FViYysjQQeobXBDDE31t5SBg++veI6tXfpco/UiKEsDswL1WAr tEAZaruo7254TyH+gydURl2wJuzo/aZ7Y7PpqaODbYv727Dvm5eX64HCyyAH0s6sOCyGF5/p eIhrOn24oBf67KtdAN3H9JoFNUVTYJc1VJU3R1JtVdgwEdr+NEciEfYl0O19VpLE/PZxP4wX PWnhf5WjdoNI1Xec+RcJ5p/pSel0jnvBX8L2cmniYnmI883NhtGZsEWj++wyKiS4NranDFlA HdDM3b4lUth1pTtABKQ1YuTvehj7EfoWD3bv9kuGZGPrAeFNiHPdOT7DaXKeHpW9homgtBxj 8aX/UkSvEGJKUEbFL9cVa5tzyialGkSiZJNkWgeHe+jEcfRT6pJZOJidSCdzvJpbdJmm+eED w9XOLH1IIWh7RURU7G1iOfEfmImFeC3cbbS73LQEFGe1urxvIH5K/7vX+FkNcr9ujwWuPE9b 1C2o4i/yZPLXIVy387EjA6GZMqvQUFuSTs/GeBcv0NjIQi8867H3uLjz+mQy63fAitsDwLmR EP+ylKVEKb0Q2A== In-Reply-To: <20240424215214.3956041-2-andrii@kernel.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Action: no action X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BE3A34001F X-Stat-Signature: 3qrqwgr8iq1fp8b8bgrzqwhpbyjhmm8f X-Rspam-User: X-HE-Tag: 1715090134-588328 X-HE-Meta: U2FsdGVkX18lmQjcoM+RsfX2bg0AdVHbFaCUTBFNgjyMwKXAf7ciY4k8MWXmOE+BUfua2oVXY/wwakNXuMvJDmncxNuTYg2pUMarcEItaTcjJnmy/RDlkcSXlPPumErQk0kw8YoWliCNSvwsT5GC786ZzVccet4QnH+AOQm3Oi5AX0ErdXbz2pFl1e6TJAgKVaJ2cKLgsYaVq+Ipx/aIzlYumwMaSqI1c1seMit1wCUR4dOj+GtQSLtr6TWs/nlqameRy6Lr/Nw7tnFqEnVD4VkRrUOsM83FY99daDbm/iRK5lT2SHq8IPtyqX0IdamCM92VbB1bBgIjrpu9QIFULVw5ePcuD/IxCBkuqhgr76NS5mEpvEFtMgUlUlJnaQ7LVJXOU2dQeJv20Gi4OZhOWzzWufipBpCkM+CwExSZbPg67uFS1e38Xw+4XImbLGtIPs/E2tVnT1RtoftoFUvyCbBrhko/D6z7v+vzDLTTweyHYhcU2ALFzL/xIhlLp6841jjsJKiC9RGete/M7BqbL881Wm2fW5tC2rkFJ8zT29VRz4EoGvBNf+fOnnLmyoaYwAvBBGL5vDdt+W19YJE8EdFJZlaR8R3/KYjQBCulyAB3Y22CbMCIIATayiQyxsqmMLkFfdTtBcPRLlysKB7MB0kdoiNqat+MDjig0LiGSyf0MLXeDIIjiFyj1yFd3IVucTGX4p0SM8uNcL+7HBjKQVz5MTyPrbJSSoSJs1Uo/+RgigQ9KRzykV+VrsagGsDRuk2zGo9Krw/x6ofOOIce0naTnF1ZQnKbzpOXW7AKkQnTxg2xRvQf8I0p1VhQTE/c0RWjtQQn83Ko5yCWv0uzF+uanYjewEsIUR2gjcTxnbPWSd+Jek7ukNsj2j69gO9XN8dVOmmQFxpJZQ1WjDroUyZ5kqRSuy8+J4J8sBncFPclgCDgobS6Pae3wjK10K8ZRz2Cqnd0/BNKurVqZX4 vid3hvaU 9TcQ4Tf+Jng9j1p/u5wCyyZQEQA5Ky+QvJ0bse9c4E3s4KffNntkmdLdFGwdN1KqeSWZRxtou70MRPHKN0JuKzMTbKE1KqgQpsQcAsGIrYsNqFhfPbNTJertRPpbqIV3ZJGWhyX1/6JQRXC2BXHX7//73ceYvliBGoGGXvDSuQ1IV9LvboGZ8yD7jO0POC17FIJCkmfNR3aEDjiku2jlx1JMB/8iPKXqCAnlwsweog9Caco8WGVAvPbmibAWQLrELXA6yse97f24MSvm0m2g3VxBHoSiUh+Gm13RW4SWqnK3G2niMwgA6zxTYN5wlyqdlBTR2HlhocSKh382fYT1xP6552Act0c9gBCkJpEpregzqLwBdg7tP2RKr9Q/9gdkMbNZMWNn1SdIdCHTQWCH3MEuR8sRoxdubPbZUftPyDG8MSYBcKiWy0QlkfZbvadJzq7IZX2JVvsrWs6Dw5qPL6nJdtQH5HGJHICqnRHBR39BvKXwCu1/duvSfm7nOgEJbrK1nQX+4u0z4W5co+6ELimYSqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 4/24/24 11:52 PM, Andrii Nakryiko wrote: > objpool_push() and objpool_pop() are very performance-critical functions > and can be called very frequently in kretprobe triggering path. > > As such, it makes sense to allow compiler to inline them completely to > eliminate function calls overhead. Luckily, their logic is quite well > isolated and doesn't have any sprawling dependencies. > > This patch moves both objpool_push() and objpool_pop() into > include/linux/objpool.h and marks them as static inline functions, > enabling inlining. To avoid anyone using internal helpers > (objpool_try_get_slot, objpool_try_add_slot), rename them to use leading > underscores. > > We used kretprobe microbenchmark from BPF selftests (bench trig-kprobe > and trig-kprobe-multi benchmarks) running no-op BPF kretprobe/kretprobe.multi > programs in a tight loop to evaluate the effect. BPF own overhead in > this case is minimal and it mostly stresses the rest of in-kernel > kretprobe infrastructure overhead. Results are in millions of calls per > second. This is not super scientific, but shows the trend nevertheless. > > BEFORE > ====== > kretprobe : 9.794 ± 0.086M/s > kretprobe-multi: 10.219 ± 0.032M/s > > AFTER > ===== > kretprobe : 9.937 ± 0.174M/s (+1.5%) > kretprobe-multi: 10.440 ± 0.108M/s (+2.2%) > > Cc: Matt (Qiang) Wu > Signed-off-by: Andrii Nakryiko Hello, this question is not specific to your patch, but since it's a recent thread, I'll ask it here instead of digging up the original objpool patches. I'm trying to understand how objpool works and if it could be integrated into SLUB, for the LSF/MM discussion next week: https://lore.kernel.org/all/b929d5fb-8e88-4f23-8ec7-6bdaf61f84f9@suse.cz/ > +/* adding object to slot, abort if the slot was already full */ I don't see any actual abort in the code (not in this code nor in the deleted code - it's the same code, just moved for inlining purposes). > +static inline int > +__objpool_try_add_slot(void *obj, struct objpool_head *pool, int cpu) > +{ > + struct objpool_slot *slot = pool->cpu_slots[cpu]; > + uint32_t head, tail; > + > + /* loading tail and head as a local snapshot, tail first */ > + tail = READ_ONCE(slot->tail); > + > + do { > + head = READ_ONCE(slot->head); > + /* fault caught: something must be wrong */ > + WARN_ON_ONCE(tail - head > pool->nr_objs); So this will only WARN if we go over the capacity, but continue and overwrite a pointer that was already there, effectively leaking said object, no? > + } while (!try_cmpxchg_acquire(&slot->tail, &tail, tail + 1)); > + > + /* now the tail position is reserved for the given obj */ > + WRITE_ONCE(slot->entries[tail & slot->mask], obj); > + /* update sequence to make this obj available for pop() */ > + smp_store_release(&slot->last, tail + 1); > + > + return 0; > +} > > /** > * objpool_push() - reclaim the object and return back to objpool > @@ -134,7 +219,19 @@ void *objpool_pop(struct objpool_head *pool); > * return: 0 or error code (it fails only when user tries to push > * the same object multiple times or wrong "objects" into objpool) > */ > -int objpool_push(void *obj, struct objpool_head *pool); > +static inline int objpool_push(void *obj, struct objpool_head *pool) > +{ > + unsigned long flags; > + int rc; > + > + /* disable local irq to avoid preemption & interruption */ > + raw_local_irq_save(flags); > + rc = __objpool_try_add_slot(obj, pool, raw_smp_processor_id()); And IIUC, we could in theory objpool_pop() on one cpu, then later another cpu might do objpool_push() and cause the latter cpu's pool to go over capacity? Is there some implicit requirements of objpool users to take care of having matched cpu for pop and push? Are the current objpool users obeying this requirement? (I can see the selftests do, not sure about the actual users). Or am I missing something? Thanks. > + raw_local_irq_restore(flags); > + > + return rc; > +} > + > > /** > * objpool_drop() - discard the object and deref objpool > diff --git a/lib/objpool.c b/lib/objpool.c > index cfdc02420884..f696308fc026 100644 > --- a/lib/objpool.c > +++ b/lib/objpool.c > @@ -152,106 +152,6 @@ int objpool_init(struct objpool_head *pool, int nr_objs, int object_size, > } > EXPORT_SYMBOL_GPL(objpool_init); > > -/* adding object to slot, abort if the slot was already full */ > -static inline int > -objpool_try_add_slot(void *obj, struct objpool_head *pool, int cpu) > -{ > - struct objpool_slot *slot = pool->cpu_slots[cpu]; > - uint32_t head, tail; > - > - /* loading tail and head as a local snapshot, tail first */ > - tail = READ_ONCE(slot->tail); > - > - do { > - head = READ_ONCE(slot->head); > - /* fault caught: something must be wrong */ > - WARN_ON_ONCE(tail - head > pool->nr_objs); > - } while (!try_cmpxchg_acquire(&slot->tail, &tail, tail + 1)); > - > - /* now the tail position is reserved for the given obj */ > - WRITE_ONCE(slot->entries[tail & slot->mask], obj); > - /* update sequence to make this obj available for pop() */ > - smp_store_release(&slot->last, tail + 1); > - > - return 0; > -} > - > -/* reclaim an object to object pool */ > -int objpool_push(void *obj, struct objpool_head *pool) > -{ > - unsigned long flags; > - int rc; > - > - /* disable local irq to avoid preemption & interruption */ > - raw_local_irq_save(flags); > - rc = objpool_try_add_slot(obj, pool, raw_smp_processor_id()); > - raw_local_irq_restore(flags); > - > - return rc; > -} > -EXPORT_SYMBOL_GPL(objpool_push); > - > -/* try to retrieve object from slot */ > -static inline void *objpool_try_get_slot(struct objpool_head *pool, int cpu) > -{ > - struct objpool_slot *slot = pool->cpu_slots[cpu]; > - /* load head snapshot, other cpus may change it */ > - uint32_t head = smp_load_acquire(&slot->head); > - > - while (head != READ_ONCE(slot->last)) { > - void *obj; > - > - /* > - * data visibility of 'last' and 'head' could be out of > - * order since memory updating of 'last' and 'head' are > - * performed in push() and pop() independently > - * > - * before any retrieving attempts, pop() must guarantee > - * 'last' is behind 'head', that is to say, there must > - * be available objects in slot, which could be ensured > - * by condition 'last != head && last - head <= nr_objs' > - * that is equivalent to 'last - head - 1 < nr_objs' as > - * 'last' and 'head' are both unsigned int32 > - */ > - if (READ_ONCE(slot->last) - head - 1 >= pool->nr_objs) { > - head = READ_ONCE(slot->head); > - continue; > - } > - > - /* obj must be retrieved before moving forward head */ > - obj = READ_ONCE(slot->entries[head & slot->mask]); > - > - /* move head forward to mark it's consumption */ > - if (try_cmpxchg_release(&slot->head, &head, head + 1)) > - return obj; > - } > - > - return NULL; > -} > - > -/* allocate an object from object pool */ > -void *objpool_pop(struct objpool_head *pool) > -{ > - void *obj = NULL; > - unsigned long flags; > - int i, cpu; > - > - /* disable local irq to avoid preemption & interruption */ > - raw_local_irq_save(flags); > - > - cpu = raw_smp_processor_id(); > - for (i = 0; i < num_possible_cpus(); i++) { > - obj = objpool_try_get_slot(pool, cpu); > - if (obj) > - break; > - cpu = cpumask_next_wrap(cpu, cpu_possible_mask, -1, 1); > - } > - raw_local_irq_restore(flags); > - > - return obj; > -} > -EXPORT_SYMBOL_GPL(objpool_pop); > - > /* release whole objpool forcely */ > void objpool_free(struct objpool_head *pool) > {