From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17006D591B1 for ; Mon, 18 Nov 2024 20:05:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4EF36B007B; Mon, 18 Nov 2024 15:05:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FE576B0082; Mon, 18 Nov 2024 15:05:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C5E06B008C; Mon, 18 Nov 2024 15:05:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6EDC96B007B for ; Mon, 18 Nov 2024 15:05:15 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0BD7BAC99A for ; Mon, 18 Nov 2024 20:05:15 +0000 (UTC) X-FDA: 82800293814.05.C02A9CE Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by imf14.hostedemail.com (Postfix) with ESMTP id AE022100004 for ; Mon, 18 Nov 2024 20:04:17 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=scylladb.com header.s=google header.b=VvYPf8kR; spf=pass (imf14.hostedemail.com: domain of avi@scylladb.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=avi@scylladb.com; dmarc=pass (policy=reject) header.from=scylladb.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731960164; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fTbmXBLcL3C/FD1sG63IhMYAyfCB0uEzv/2adkt2AHA=; b=oxtGnuNwG2x8vuSAjWf68c4dnto7ttYZR2TljjliX3Ey701q4eoZ5W/QJnkTsnLoITCxJo eLGPI8TO4/2svTsreiJvj6W2y7jo9JICEQhWvJjY8f7zsMXQGQSD2+H8amnxbshGDVmGiV ZKaeAkHUGGoy6AWwAhYmyeb2CJJ+8Lc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731960164; a=rsa-sha256; cv=none; b=oapUI+YzRi60NR019rI0bl7RUYZPkZM8d2v0pUTfWbFp0zYQkt8DKY15nrO8iBrthLiPsA OklhHRXOTDkrKp9Xet30zj3CKaEvsBA2S3zX+v9ZobiNd3HWtfkPqvVUP9hALhTMwbRRq4 FguYtLDhhuNNGv9g4v9AJKKjS3foqJ0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=scylladb.com header.s=google header.b=VvYPf8kR; spf=pass (imf14.hostedemail.com: domain of avi@scylladb.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=avi@scylladb.com; dmarc=pass (policy=reject) header.from=scylladb.com Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-4315839a7c9so784475e9.3 for ; Mon, 18 Nov 2024 12:05:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scylladb.com; s=google; t=1731960311; x=1732565111; darn=kvack.org; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date :message-id:reply-to; bh=fTbmXBLcL3C/FD1sG63IhMYAyfCB0uEzv/2adkt2AHA=; b=VvYPf8kR5MSeQy/Y4Xj4X+kahebOFXcM2Ku+ZzUbBcPILUUGBDtvbr7jwzHN2cPNfr +wdeKH7r72/KDjAz1SwSH8QrbAKh3vf+0HAofBh/sx31wq9sm+oNLcv3/i9eaOTd0pON BnwbYkynS2a6K51zMFhc9kUNmIv9QMXXBfkh2FjiDyhbjKYherhyL9VcTrwO0edlUx9n ZNEBa+RSMh2PpM5g2YsQmCx/bA2QJvblcs1vZTJrdQHjxBfVKfwJwcV3up0xT0qWpgml TW96pgFPvCadEicoth+lW4rWA7n8H6IxNP2J7LFx6rBzXbdEknAT6xVqTSm5shi6GtJs 5wcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731960311; x=1732565111; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=fTbmXBLcL3C/FD1sG63IhMYAyfCB0uEzv/2adkt2AHA=; b=bpIR1gATS7zO8jJmQ/TPx5sd0Ndfcuu4oCIAQpe4BWCB8cF1EtA2ApKKLotr9p8cR2 X790518iisvpB0//Y2Dc+wCJwX+5lcTvgwmdoxYlS//naCk+kUcibIZ5SucmftrFvKoW anfopvrrWmqMTToLV8yfc3YmebkhTJlyrj10lLQHJA/vtcfdWmbD74ZNyi6kolv9oquL iqTZ6I0Ylbqmhuj0afxAnEfnIHaAvirJ62C+byZ6CuOaL4L77DlO+DwlXTYukene7B9C SumWuOLBCu702n3iG5lDWlWrCTyEHQRWuz2NRP/bTxgWkoSWOHvYX592xcauDes+zYeY BgWg== X-Forwarded-Encrypted: i=1; AJvYcCW1NdmpAaxGPHP/KolFmNDHhjle+gghs8d43I4DsjqndpvlYHiM1HErwas3wgmkStFX50NBn66Bjw==@kvack.org X-Gm-Message-State: AOJu0Yxx0hBfJNceuPl6Hlk5jxWrxVQqooYoUlvIlC0iogyUem5Z8yGF YQV4EdZGw6DB8I9V4/bY7OPjsj2RE9CV3h608K6o5QmNYFHcs1zehI7MnD5DOFjqDDQHxc1js0x noA5zAlK3fNmjIj1Ah9BNxccgfM9Zpz1MMDHxd7Cj78q97Q+s5v0traXgHyPMLfIgrf+9ZYYcdZ OmqOxF/x+UVNvLSnfnaAfb3FT4d8pnHCgbhj+y9YrataaLIPwdMNrJEmjqsSsBxPWR0/I2Nz74T Cs2gIpnrd14AnV9HSk+25bSSHnMj1eJUrWmE8JKIFvoFzEhODhqZF0Yge1hOYezdTu6tSnGLelS 3ov0KQ== X-Google-Smtp-Source: AGHT+IGLqt9yheivwxWvr2d4jUe6jjFH35y8SdBbbrIykzKAkVI0BOJA1g2500rJm8nvRRgYQ5QvIA== X-Received: by 2002:a05:600c:1c11:b0:42c:b9c8:2bb0 with SMTP id 5b1f17b1804b1-432df71a6edmr121358405e9.4.1731960311156; Mon, 18 Nov 2024 12:05:11 -0800 (PST) Received: from avi.scylladb.com ([2a09:bac5:3ac4:194b::285:a]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-432dab72141sm166282505e9.1.2024.11.18.12.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Nov 2024 12:05:10 -0800 (PST) Message-ID: <3853f2df4e9b488d705f13a48f15105989fb45d9.camel@scylladb.com> Subject: Re: memfd and transparent hugepages From: Avi Kivity To: David Hildenbrand , linux-mm Date: Mon, 18 Nov 2024 22:05:08 +0200 In-Reply-To: References: <2f7b4534689ec00dc9dd139ec2845c79d1617514.camel@scylladb.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.1 (3.54.1-1.fc41) MIME-Version: 1.0 X-CLOUD-SEC-AV-Sent: true X-CLOUD-SEC-AV-Info: scylladb,google_mail,monitor X-Gm-Spam: 0 X-Gm-Phishy: 0 X-CLOUD-SEC-AV-Sent: true X-CLOUD-SEC-AV-Info: scylla,google_mail,monitor X-Gm-Spam: 0 X-Gm-Phishy: 0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AE022100004 X-Stat-Signature: sdsjh3xx4oxkk81is393qnpzw3ujzksd X-HE-Tag: 1731960257-249060 X-HE-Meta: U2FsdGVkX18RDBIrAhVcncroCfHyR1RVEJ+r+ZI5T5KuwlovgL+0SCOQnlMDrJ2VOgJolNCMY4ib/q95MfnYEJH9gUdV9CUDg0Gd58EUpcPa6K6QEDnjunRFP9RcTlIMKL7PQRt6QXTkgSWae9svwHvTvuSs5xKZMZTy+Y57Z5h9NNfbiu6QHzpuxPk+mJR5pOJUwO5Lqmr4Vg315yAoxsCHVl6OA9Bznnm/q6yEXPSyrbixvtsPMng0DOwrt6dzBKl3rQLzcwbBkGTDMUUG0yPqPlnMTTZRsKMPHkVjxGjKXNYnorZsJIkJuQaKA/AQTHOyTkuSdp+bskl6hVkp3dspnoXS0EIbndZ9RTyvWX6Y5phbLXVv9tE6PfTj3dsQcf/mRS3zGVEr7ML0vx4UiBZZhVrfukWftMAhT5aglYjrPZRPAEQoa5UP+VRrwnNi20vOwm/FhRrKH0sP+F8+QeaqKPgp8B5woVS4GlrSM6xyc47DO0J05NFb3ynx81RgxfbptUuCG2CD8znABJ2XHy0XKfgQLhJzrFhMBhgVku5pCzyqcMBg0WoosJztYhnrjtLhC3NnmQOl5P1QUFrMtfw+CQNL+HCPk5VRrD70XWqlh0oOEsKeZ/xQs45B9t2KXSvfzxJmCCVCfQo6u56Upj/YdJm7yNSq6MLy5eK0MdzGIuWnGnzzlNuIC7U83DLQXULXeoON8uj67rOEtGMkvhnLXqcaNe/wpv49DCqB2+4u1qO1n8LpONB6m5mD30didpw22Fw/oV2oIcowtA/LxjnQziBZSuziHyaNNDQWJ1Aqjzcu/3cPhoP2WQdjj5iY85Jk2U1XxnsSDmyUXsvTNsMpu2P0fEYPht4LUFhCfAzWXAPuF2YrdU1QOUi6JN+jG5eD4eopJPc/iwq7tndENVhtVsYjgeQFKVlD9KmPXf+XnopDr3R4yDB53BRyhRYm/cU9OM+4j9pxDHOytQp 6oHHglG7 7X42udSNSnTOWne857asi2QBvjQgEUrrtKCDbKiPcsNDB2bAPebTI8fQ7j6UNjujohJd/AvSDYToyd2BBa5n9POPzHPicdO9gm+IezcRC/X9C00oMUSOOLhHXMTqwvJMba5cxot2+Ymt/bajWopMUM51nmJgfVDMTbvqrZynSd/xCWUmKqhthW3vI/wiLO3XIh3HclAO5/GAqkq9qySMpJkbUrzm3vnAhSQFnUFObvJ5Ru5Mf22ACstt6MdfLS74DQbfOOy4bRHuChWQd83uvlyyAWyRpkAKy8QQNeX2aMcRIZL5Gp26ifjBP20w6+bB+qp7NYrgNmqEIrdo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.024050, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 2024-11-18 at 20:42 +0100, David Hildenbrand wrote: > On 18.11.24 15:13, Avi Kivity wrote: > > #define _GNU_SOURCE > > #include > > #include > > #include > > #include > >=20 > > int main(int ac, char** av) { > > =C2=A0=C2=A0=C2=A0=C2=A0 size_t memsz =3D (size_t)2048 << 20; > > =C2=A0=C2=A0=C2=A0=C2=A0 int fd =3D memfd_create("memory", MFD_CLOEXEC)= ; > > =C2=A0=C2=A0=C2=A0=C2=A0 ftruncate(fd, memsz); > > =C2=A0=C2=A0=C2=A0=C2=A0 void* p =3D mmap((void*)0x40000000, memsz, PRO= T_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_FILE, fd, 0); > > =C2=A0=C2=A0=C2=A0=C2=A0 madvise(p, memsz, MADV_HUGEPAGE); > > =C2=A0=C2=A0=C2=A0=C2=A0 madvise(p, memsz, MADV_POPULATE_WRITE); > > =C2=A0=C2=A0=C2=A0=C2=A0 madvise(p, memsz, MADV_COLLAPSE); > > =C2=A0=C2=A0=C2=A0=C2=A0 pause(); > > =C2=A0=C2=A0=C2=A0=C2=A0 return 0; > > } >=20 > Hi, >=20 > >=20 > > While memfd is documented as using anonymous pages, AnonHugePages > > shows > > as zero. >=20 > it's an anonymous file, not anonymous memory. memfd is shmem in > diguise. >=20 > >=20 > > If memfd incompatible with transparent hugepages? >=20 > Shmem supports THP. >=20 > But you have one flaws in your program: you have to mmap it > MAP_SHARED.=20 > It's shmem. >=20 > Otherwise during MADV_POPULATE_WRITE you're simply allocating a shmem > page (pageache), and the COW it to map an anonymous page in your page > tables, resulting in a double memory consumption (pagecache+anonymous > memory). khugepaged does currently not operate on such VMAs. >=20 > If you use MAP_SHARED in your example above: >=20 Thanks a lot - this works. In case anyone's interested, I'm using this to provide the equivalent of vmalloc() in my application. It usually uses a large THP arena (consuming all of the machine's memory) to allocate from, but sometimes I need large contiguous memory areas for third-party libraries. I can't easily allocate large contiguous areas, but 128kB allocations are fine. So I allocate a few of those, and use mmap(MAP_FIXED) to stitch together several 128kB allocations into a single large contiguous map.