From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41E1EC433EF for ; Tue, 1 Mar 2022 23:01:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C04358D0002; Tue, 1 Mar 2022 18:01:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B8CCA8D0001; Tue, 1 Mar 2022 18:01:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DEEE8D0002; Tue, 1 Mar 2022 18:01:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 861558D0001 for ; Tue, 1 Mar 2022 18:01:40 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 61E8661F79 for ; Tue, 1 Mar 2022 23:01:40 +0000 (UTC) X-FDA: 79197341160.11.B7FE394 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf14.hostedemail.com (Postfix) with ESMTP id 8348910000C for ; Tue, 1 Mar 2022 23:01:39 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.1.2/8.16.1.2) with ESMTP id 221GjKXj010804; Tue, 1 Mar 2022 15:01:20 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=facebook; bh=13yf4XwVwzTCNKdtvOczb8GHMP1QzVqvinRdeqQMkrM=; b=LCSu8XcZAHQJE960NuyHTenaj2wcQ78txN5hfCGY27FAk2ZFpQSyGajcrm+Tmx/S2X8g G7Gd0fvzyuN80rP1pVZ5z+H92vwQfTQkcRh8uupgplMCtQwjBf87a1MzTC+ObmFF8SkE 084fT1FPghj4ZQ80VJmnlpnVFWKJKHl66xA= Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3ehn3e3qp7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Mar 2022 15:01:20 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F/Xd2Yg2WBT8UY74qE691u6xw6FcM9rcVzaU5G/nfPGKTY3SvwfkgeDaeELniicVAGDluifIpP6mBc2D3k5/BrbMhh7ZyADXgIUrLrehoU5Ll6poP25zlrXTZiK/7n+JWnM2qtegQVMjBOWogBK9oMTQXQF2Wv4oUzW0IXw9xoFAC0+5Tk3SGQVJD2/Voyavnd8EDKgQOsI3P4uqoYTWM8R1KW2wSNFuvtM4eyBg2X5HEWGj2/qYieh/yyzxsOY5RamEluxOw5O/rayp9x+XeT3XbUdJVpQpKlqu598a/g3o7G0yRt4qTBEwhrjwzp0hcZYqzpuDAgmtibJR+XuXvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=13yf4XwVwzTCNKdtvOczb8GHMP1QzVqvinRdeqQMkrM=; b=Lw46SSoHF+Tyvj1PC/qKS7wlUQvUs0P6neiSMUgLh01AHRDzI125O4bsHdq8T6M83PNhMEPCx1iKL6fFR1I3Mi4ipE0616Xtmi1BnEGVFk80Uz7Qmc4Bl58XcvP7ud41W+1wLQZBv26LUdfg8S3ds6Xo7DuTc0UqjxV8iEZM6c6rJVXr3W21Zaa9Wc5VokvWMX4TMgbjbmijJOJpVs6FJ7bwSIasf5irx6dRFmliO2aJ5BY6dSeGRj/wDTilOGfE982WtHu6IorOAX0WFODD0LYfCpQWQtwHmy77OMsy5h6ynF5eQDTKwK2f8hGb30swHdIKjC5G+El2r5fsBMEXQA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fb.com; dmarc=pass action=none header.from=fb.com; dkim=pass header.d=fb.com; arc=none Received: from SA1PR15MB5109.namprd15.prod.outlook.com (2603:10b6:806:1dc::10) by DM5PR1501MB2168.namprd15.prod.outlook.com (2603:10b6:4:a8::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.24; Tue, 1 Mar 2022 23:01:15 +0000 Received: from SA1PR15MB5109.namprd15.prod.outlook.com ([fe80::7da4:795a:91b8:fa54]) by SA1PR15MB5109.namprd15.prod.outlook.com ([fe80::7da4:795a:91b8:fa54%7]) with mapi id 15.20.5017.027; Tue, 1 Mar 2022 23:01:15 +0000 From: Song Liu To: Daniel Borkmann CC: Song Liu , Linux Memory Management List , bpf , Network Development , Alexei Starovoitov , Andrii Nakryiko , Kernel Team , Andrew Morton , Eric Dumazet , Michal Hocko Subject: Re: [PATCH bpf-next 2/2] bpf: flexible size for bpf_prog_pack Thread-Topic: [PATCH bpf-next 2/2] bpf: flexible size for bpf_prog_pack Thread-Index: AQHYHklKCIaff07zEkyT+B5ekpbKEqyMcwwAgACNNQCAAWxqgIAAVccAgByBeQA= Date: Tue, 1 Mar 2022 23:01:15 +0000 Message-ID: References: <20220210064108.1095847-1-song@kernel.org> <20220210064108.1095847-3-song@kernel.org> <34d0ed40-30cf-a1a2-f4eb-fa3d0a55bce8@iogearbox.net> <14B98886-D56E-4FE5-89F6-3A41D35B7105@fb.com> In-Reply-To: <14B98886-D56E-4FE5-89F6-3A41D35B7105@fb.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3693.60.0.1.1) x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 41e3124f-c96c-4cfa-ced4-08d9fbd7637f x-ms-traffictypediagnostic: DM5PR1501MB2168:EE_ x-microsoft-antispam-prvs: x-fb-source: Internal x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: bjd+ZPf+sMypJo5Glai0UTGVVmGJkifiWy5seFH4PARkaMoq4XjYVCar3/1sZEzYCrCWtARAUorcuYWUhuocHvKmy2BCu9p9Z0Urb1/sw0bJqJ0Xxxkh9x+9GSWWLPHA8mwvPtetXqsyxyr5Vc7zWp323N7W4595Xrpnolj3Uli6/Iuoxpk7EAOV+XIOY2GaWip5FEqkwqSd0DLzyyzat491UPih5e7kxXCe+57+AgUZ0iaU/iv6kwsQ9Up1oSTQY+tJlPf7jRUXrTPfBG8L9np/TTGYhLJocRkxYAdfYrqI2iTY5H2RPIncGxCmn4iVhMaXvIhWf7Ld80LephMcKHL4KneuhlsS4ZSyojQRLDy/SB8xAAqtuNIiSd8r9h2sWxdZnMz1Anr4flGEsISSEiCaM52xZqKjwRam9HbASNOWsk9r26hyCI+HaXAwnzoIIzoFqdXcOvSsXsYlJZPZ7PkNtFRO+nVCmgSSiChxPYAFxRAlw3d1ec1VyZUeZODxWf+kSY6IXpGzHjpYQ7J6CEkn8bKPwIfYd6lv/brosi5GR0wLC7W8oMmlF5CMrI5oWmC/TQ9BuEDU8ionIUiMiDKr7NlHVvbu7OfAPqR1OINw/tuKprPDku7/upoxcng8C/szdaXOCknSE9x1/2coBm6eK4hNHpAuc3Uu/MPF0nHLZya2lhqazUFMD+pWobfwwi5IpbdlDb5GGQ96o6yeSsng5sGdoAcigqoZf097O9MsarQL6hBmQfQ0YL2YDvAP x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA1PR15MB5109.namprd15.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(316002)(2906002)(33656002)(36756003)(6512007)(186003)(6916009)(2616005)(6486002)(8936002)(54906003)(7416002)(66556008)(508600001)(86362001)(5660300002)(66476007)(53546011)(66446008)(6506007)(64756008)(66946007)(38100700002)(91956017)(76116006)(4326008)(122000001)(8676002)(83380400001)(38070700005)(71200400001)(45980500001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Gd/tNN531VUO9qTv/Wxj6ydkwHPCqeMZRaF/jfftFL8BVXhGbRYFEWwZ0aYv?= =?us-ascii?Q?9rs1QS9ddjK+s93TtGbAytG7+HcO/N1zV0OKi93RtCe8FhvCi7JF2qWRyNgy?= =?us-ascii?Q?+Jja0qGrA+gSf0TLf0T3S4Ic2Uz+3StvFsEwxaaDgJ+kjhg24AiZsEEMopqJ?= =?us-ascii?Q?aw8MXQSlMz1Elvda0gU8VqiWlwfLG7R5FNpkRvEHoPKrIHhC+0Ov+nhxaHv5?= =?us-ascii?Q?ILOPreDlQKIafOlvvoJF4EcjFkDwUzraiy81f8dqdFAsbo5RlkaxMiwMlH3p?= =?us-ascii?Q?CDJnt4yDqoXrJmxSwMtvBfg/kXZhYzopytTnjEPNnjbhQrJSaSgz+riqSURB?= =?us-ascii?Q?/RDUmLPJeqQmmu1WLMbq2qAE1G65BIMPO6DuIAPRZReoXyc2okrz8pA4U2kj?= =?us-ascii?Q?UkUhSeM2gFmHAonZ3M2OHaezCxxRiy3kfo8lzRhyhaiMfYH3GnJ67GH76AXi?= =?us-ascii?Q?hytpYSasmKX7tdj9+a0LpTqpcPP1v0BqTP62MRby3tVWU2SRit4yv6gMJLG3?= =?us-ascii?Q?4cTVH3HtHsfYcICAN86lfmDIt2CBsfeCUt4fGAcpmjBZHQZ+JglGuJc6Y55z?= =?us-ascii?Q?oXdV9dkOMb3ZKeOPtgxCS2sWdjSJLjuMgU5MrH+nloHksYzUg2SZjapy0npV?= =?us-ascii?Q?z+4KvVWIXRu5oiS3B1zX9JDbBP/gjkraAijtBUEtfVOynlZCQqNhJhJ3AnfC?= =?us-ascii?Q?Rf2SfrCOwwYw9uQSj2cF6oxXl1x2EKVUMjjxVhbMS0XuqRQgBETbsna0OBkf?= =?us-ascii?Q?rAoclhm3ffONhqYRHKAmj+FH2o8agvdPyvW2EN354VsSg/PhD/zmpGFfTJl5?= =?us-ascii?Q?Y3R1bGkF2w79knceLCHA0qbopMo+B1i1hVFkL0RdSZY54/LtHtd/V+cV0TRR?= =?us-ascii?Q?aqf078bsDTGUTdhPo5hYbpby+WjzorniXvShzxG6u5wZeMSUAzYkazIPdDua?= =?us-ascii?Q?X/JTdtdry1qYBNbKZCd3cZ9AW+PzEpWtMhHAJd2UBv9R2BiTDflnjutMGMlT?= =?us-ascii?Q?NPRDrpAFRjf5aJIRMXYDSDxifed+zPfKS3k9pcTVmNPzxPbXdxpNicujNeeC?= =?us-ascii?Q?adeLj8yVS6LfOb1JpEnqHEArPs4pHIp1dYw7atdLZjebjVox6lMRG2S+2cFd?= =?us-ascii?Q?+59bJa52V/9/b9NdM581oM3JHk0sukjjAlrL+IZDMjW6sJNVrw+PrR5nawaa?= =?us-ascii?Q?LqGvkfjCNHUTkQYPIderrjSGhLRoGVJlGciRPdKwRMq7t2dsSJpEBTLQg68B?= =?us-ascii?Q?141FvUGDvD6+2BMnf7LIH2w32DMVIwzNADp3Z8WyRVrKyOo1dpna49onV4x5?= =?us-ascii?Q?v/n3PTxss0WR2ocq4bZGck5JNDte+RalfudvZ0lSraWHmhZJfbj6yg/2JHSn?= =?us-ascii?Q?ykDP+9smTUDkHFrf/DWcfoKaagqRZvimqWhDY9lzmvRbtqpT/Ie4wFABcMcy?= =?us-ascii?Q?KS3TmZ5oqmSMQJinIUfkO81YxSHQJSzaKqyP3ttNhM7JGQnKIiDsFOkd5o9Q?= =?us-ascii?Q?8qRGINTkIqnBMpXs52g4orhNOkAX5OSOvw9j8FSUi/niGpFxilJGZu2a8G5t?= =?us-ascii?Q?saFCoEaXK+Lxf1PFiu/q+zwoOxQM0H2rsxkaflcCQWBYXeRCnFyrgnLQ6dZP?= =?us-ascii?Q?A/MsWOQJGDD2WP9RpkL8mCVfEfgqYXgsS8ZSaamN79DAN6gSPznThr7HtzZB?= =?us-ascii?Q?k7k7bQ=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: fb.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA1PR15MB5109.namprd15.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 41e3124f-c96c-4cfa-ced4-08d9fbd7637f X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Mar 2022 23:01:15.4715 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: nFI7mQerWTkZ7PY6zk7pONAfXBbhRdSwi2OnW70GTRgLdM+7lzPj0/BJhHndX7AjEFp3SIpuU6MG6xLsqQ7IUQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1501MB2168 X-Proofpoint-GUID: 3DD5qtZL6OtL4RhOcQ7NBDkdjJG_apt7 X-Proofpoint-ORIG-GUID: 3DD5qtZL6OtL4RhOcQ7NBDkdjJG_apt7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.816,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-01_07,2022-02-26_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=fb_outbound_notspam policy=fb_outbound score=0 lowpriorityscore=0 phishscore=0 mlxscore=0 spamscore=0 suspectscore=0 bulkscore=0 adultscore=0 mlxlogscore=999 priorityscore=1501 impostorscore=0 clxscore=1011 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000 definitions=main-2203010114 X-FB-Internal: deliver X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8348910000C X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=fb.com header.s=facebook header.b=LCSu8XcZ; dmarc=pass (policy=reject) header.from=fb.com; spf=none (imf14.hostedemail.com: domain of "prvs=405948ca8e=songliubraving@fb.com" has no SPF policy when checking 67.231.145.42) smtp.mailfrom="prvs=405948ca8e=songliubraving@fb.com" X-Stat-Signature: pwsth77fnd4qu36yyxshfjdprphickms X-HE-Tag: 1646175699-916096 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > On Feb 11, 2022, at 11:42 AM, Song Liu wrote: >=20 >=20 >=20 >> On Feb 11, 2022, at 6:35 AM, Daniel Borkmann wrot= e: >>=20 >> On 2/10/22 5:51 PM, Song Liu wrote: >>>> On Feb 10, 2022, at 12:25 AM, Daniel Borkmann w= rote: >>>> On 2/10/22 7:41 AM, Song Liu wrote: >>>>> bpf_prog_pack uses huge pages to reduce pressue on instruction TLB. >>>>> To guarantee allocating huge pages for bpf_prog_pack, it is necessary= to >>>>> allocate memory of size PMD_SIZE * num_online_nodes(). >>>>> On the other hand, if the system doesn't support huge pages, it is mo= re >>>>> efficient to allocate PAGE_SIZE bpf_prog_pack. >>>>> Address different scenarios with more flexible bpf_prog_pack_size(). >>>>> Signed-off-by: Song Liu >>>>> --- >>>>> kernel/bpf/core.c | 47 +++++++++++++++++++++++++++-------------------= - >>>>> 1 file changed, 27 insertions(+), 20 deletions(-) >>>>> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c >>>>> index 42d96549a804..d961a1f07a13 100644 >>>>> --- a/kernel/bpf/core.c >>>>> +++ b/kernel/bpf/core.c >>>>> @@ -814,46 +814,53 @@ int bpf_jit_add_poke_descriptor(struct bpf_prog= *prog, >>>>> * allocator. The prog_pack allocator uses HPAGE_PMD_SIZE page (2MB o= n x86) >>>>> * to host BPF programs. >>>>> */ >>>>> -#ifdef CONFIG_TRANSPARENT_HUGEPAGE >>>>> -#define BPF_PROG_PACK_SIZE HPAGE_PMD_SIZE >>>>> -#else >>>>> -#define BPF_PROG_PACK_SIZE PAGE_SIZE >>>>> -#endif >>>>> #define BPF_PROG_CHUNK_SHIFT 6 >>>>> #define BPF_PROG_CHUNK_SIZE (1 << BPF_PROG_CHUNK_SHIFT) >>>>> #define BPF_PROG_CHUNK_MASK (~(BPF_PROG_CHUNK_SIZE - 1)) >>>>> -#define BPF_PROG_CHUNK_COUNT (BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SI= ZE) >>>>> struct bpf_prog_pack { >>>>> struct list_head list; >>>>> void *ptr; >>>>> - unsigned long bitmap[BITS_TO_LONGS(BPF_PROG_CHUNK_COUNT)]; >>>>> + unsigned long bitmap[]; >>>>> }; >>>>> -#define BPF_PROG_MAX_PACK_PROG_SIZE BPF_PROG_PACK_SIZE >>>>> #define BPF_PROG_SIZE_TO_NBITS(size) (round_up(size, BPF_PROG_CHUNK_S= IZE) / BPF_PROG_CHUNK_SIZE) >>>>> static DEFINE_MUTEX(pack_mutex); >>>>> static LIST_HEAD(pack_list); >>>>> +static inline int bpf_prog_pack_size(void) >>>>> +{ >>>>> + /* If vmap_allow_huge =3D=3D true, use pack size of the smallest >>>>> + * possible vmalloc huge page: PMD_SIZE * num_online_nodes(). >>>>> + * Otherwise, use pack size of PAGE_SIZE. >>>>> + */ >>>>> + return get_vmap_allow_huge() ? PMD_SIZE * num_online_nodes() : PAGE= _SIZE; >>>>> +} >>>>=20 >>>> Imho, this is making too many assumptions about implementation details= . Can't we >>>> just add a new module_alloc*() API instead which internally guarantees= allocating >>>> huge pages when enabled/supported (e.g. with a __weak function as fall= back)? >>> I agree that this is making too many assumptions. But a new module_allo= c_huge() >>> may not work, because we need the caller to know the proper size to ask= for. >>> (Or maybe I misunderstood your suggestion?) >>> How about we introduce something like >>> /* minimal size to get huge pages from vmalloc. If not possible, >>> * return 0 (or -1?) >>> */ >>> int vmalloc_hpage_min_size(void) >>> { >>> return vmap_allow_huge ? PMD_SIZE * num_online_nodes() : 0; >>> } >>=20 >> And that would live inside mm/vmalloc.c and is exported to users ... >=20 > Yeah, this will go to vmalloc.c. >=20 >>=20 >>> /* minimal size to get huge pages from module_alloc */ >>> int module_alloc_hpage_min_size(void) >>> { >>> return vmalloc_hpage_min_size(); >>> } >>=20 >> ... and this one as wrapper in module alloc infra with __weak attr? >=20 > And this goes to some module.c file(s). I am not quite sure whether we > need __weak attr or not.=20 >=20 >>=20 >>> static inline int bpf_prog_pack_size(void) >>> { >>> return module_alloc_hpage_min_size() ? : PAGE_SIZE; >>> } >>=20 >> Could probably work. It's not nice, but at least in the corresponding pl= aces so it's >> not exposed / hard coded inside bpf and assuming implementation details = which could >> potentially break later on. >=20 > I don't really like it either.=20 >=20 > Another way to do this is to test the required size for bpf_prog_pack=20 > in BPF code, something like the following. The pro of this version is=20 > that we don't need changes in vmalloc and module code.=20 Hi Daniel,=20 Do you have further suggestions on this? I personally like the following version best, as all the changes are limited to bpf/core.c.=20 Thanks, Song > diff --git i/kernel/bpf/core.c w/kernel/bpf/core.c > index 44623c9b5bb1..3cfd0f0c93d2 100644 > --- i/kernel/bpf/core.c > +++ w/kernel/bpf/core.c > @@ -814,15 +814,9 @@ int bpf_jit_add_poke_descriptor(struct bpf_prog *pro= g, > * allocator. The prog_pack allocator uses HPAGE_PMD_SIZE page (2MB on x8= 6) > * to host BPF programs. > */ > -#ifdef CONFIG_TRANSPARENT_HUGEPAGE > -#define BPF_PROG_PACK_SIZE HPAGE_PMD_SIZE > -#else > -#define BPF_PROG_PACK_SIZE PAGE_SIZE > -#endif > #define BPF_PROG_CHUNK_SHIFT 6 > #define BPF_PROG_CHUNK_SIZE (1 << BPF_PROG_CHUNK_SHIFT) > #define BPF_PROG_CHUNK_MASK (~(BPF_PROG_CHUNK_SIZE - 1)) > -#define BPF_PROG_CHUNK_COUNT (BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZE= ) >=20 > struct bpf_prog_pack { > struct list_head list; > @@ -830,30 +824,56 @@ struct bpf_prog_pack { > unsigned long bitmap[]; > }; >=20 > -#define BPF_PROG_MAX_PACK_PROG_SIZE BPF_PROG_PACK_SIZE > #define BPF_PROG_SIZE_TO_NBITS(size) (round_up(size, BPF_PROG_CHUNK_SIZ= E) / BPF_PROG_CHUNK_SIZE) >=20 > +static int bpf_prog_pack_size =3D -1; > + > +static inline int bpf_prog_chunk_count(void) > +{ > + WARN_ON_ONCE(bpf_prog_pack_size =3D=3D -1); > + return bpf_prog_pack_size / BPF_PROG_CHUNK_SIZE; > +} > + > static DEFINE_MUTEX(pack_mutex); > static LIST_HEAD(pack_list); >=20 > static struct bpf_prog_pack *alloc_new_pack(void) > { > struct bpf_prog_pack *pack; > + void *ptr; > + int size; >=20 > - pack =3D kzalloc(sizeof(*pack) + BITS_TO_BYTES(BPF_PROG_CHUNK_COU= NT), GFP_KERNEL); > - if (!pack) > + /* Test whether we can get huge pages. If not just use PAGE_SIZE > + * packs. > + */ > + if (bpf_prog_pack_size =3D=3D -1) { > + size =3D PMD_SIZE * num_online_nodes(); > + ptr =3D module_alloc(size); > + if (is_vm_area_hugepages(ptr)) { > + bpf_prog_pack_size =3D size; > + goto got_ptr; > + } else { > + bpf_prog_pack_size =3D PAGE_SIZE; > + vfree(ptr); > + } > + } > + > + ptr =3D module_alloc(bpf_prog_pack_size); > + if (!ptr) > return NULL; > - pack->ptr =3D module_alloc(BPF_PROG_PACK_SIZE); > - if (!pack->ptr) { > - kfree(pack); > +got_ptr: > + pack =3D kzalloc(sizeof(*pack) + BITS_TO_BYTES(bpf_prog_chunk_cou= nt()), GFP_KERNEL); > + if (!pack) { > + vfree(ptr); > return NULL; > } > - bitmap_zero(pack->bitmap, BPF_PROG_PACK_SIZE / BPF_PROG_CHUNK_SIZ= E); > + pack->ptr =3D ptr; > + bitmap_zero(pack->bitmap, bpf_prog_pack_size / BPF_PROG_CHUNK_SIZ= E); > list_add_tail(&pack->list, &pack_list); >=20 > set_vm_flush_reset_perms(pack->ptr); > - set_memory_ro((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE= _SIZE); > - set_memory_x((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE_= SIZE); > + set_memory_ro((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE= _SIZE); > + set_memory_x((unsigned long)pack->ptr, bpf_prog_pack_size / PAGE_= SIZE); > return pack; > } >=20 > @@ -864,7 +884,7 @@ static void *bpf_prog_pack_alloc(u32 size) > unsigned long pos; > void *ptr =3D NULL; >=20 > - if (size > BPF_PROG_MAX_PACK_PROG_SIZE) { > + if (size > bpf_prog_pack_size) { > size =3D round_up(size, PAGE_SIZE); > ptr =3D module_alloc(size); > if (ptr) { > @@ -876,9 +896,9 @@ static void *bpf_prog_pack_alloc(u32 size) > } > mutex_lock(&pack_mutex); > list_for_each_entry(pack, &pack_list, list) { > - pos =3D bitmap_find_next_zero_area(pack->bitmap, BPF_PROG= _CHUNK_COUNT, 0, > + pos =3D bitmap_find_next_zero_area(pack->bitmap, bpf_prog= _chunk_count(), 0, > nbits, 0); > - if (pos < BPF_PROG_CHUNK_COUNT) > + if (pos < bpf_prog_chunk_count()) > goto found_free_area; > } >=20 > @@ -904,12 +924,12 @@ static void bpf_prog_pack_free(struct bpf_binary_he= ader *hdr) > unsigned long pos; > void *pack_ptr; >=20 > - if (hdr->size > BPF_PROG_MAX_PACK_PROG_SIZE) { > + if (hdr->size > bpf_prog_pack_size) { > module_memfree(hdr); > return; > } >=20 > - pack_ptr =3D (void *)((unsigned long)hdr & ~(BPF_PROG_PACK_SIZE -= 1)); > + pack_ptr =3D (void *)((unsigned long)hdr & ~(bpf_prog_pack_size -= 1)); > mutex_lock(&pack_mutex); >=20 > list_for_each_entry(tmp, &pack_list, list) { > @@ -926,8 +946,8 @@ static void bpf_prog_pack_free(struct bpf_binary_head= er *hdr) > pos =3D ((unsigned long)hdr - (unsigned long)pack_ptr) >> BPF_PROG= _CHUNK_SHIFT; >=20 > bitmap_clear(pack->bitmap, pos, nbits); > - if (bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT= , 0, > - BPF_PROG_CHUNK_COUNT, 0) =3D=3D 0)= { > + if (bitmap_find_next_zero_area(pack->bitmap, bpf_prog_chunk_count= (), 0, > + bpf_prog_chunk_count(), 0) =3D=3D = 0) { > list_del(&pack->list); > module_memfree(pack->ptr); > kfree(pack); >=20 >=20