[PATCH] shmem: add shmem_size option, set filesystem size

[PATCH] shmem: add shmem_size option, set filesystem size

[Gary B. Genett]

Adds a kernel configuration option and command line parameter to
specify the size of the shmem filesystem.  It is currently hard-coded
to 50% of memory.  Users should have the option to set this value as
they see fit.

A specific case where this is necessary would be if the initramfs were
larger than half of the memory, such as a 2.5GB filesystem with 4GB of
memory.  Without this option, this causes a kernel panic.  With this
option, the user may specify something like 75%, which would allow the
filesystem into memory, while still leaving enough resources to run
a functioning system.

This patch creates the SHMEM_SIZE configuration option and the
shmem_size parameter, which may be specified in bytes, human notation
(1G, 100M, etc.) or percentage (75%).  The default remains unchanged.
This patch has no impact unless the values are changed.

Signed-off-by: Gary B. Genett <me@garybgenett.net>
---
 Documentation/admin-guide/kernel-parameters.rst |  1 +
 Documentation/admin-guide/kernel-parameters.txt |  5 +++
 init/Kconfig                                    |  8 ++++
 mm/shmem.c                                      | 57 +++++++++++++++++++++++--
 4 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst
index d05d531b4ec9..33122a6df04f 100644
--- a/Documentation/admin-guide/kernel-parameters.rst
+++ b/Documentation/admin-guide/kernel-parameters.rst
@@ -149,6 +149,7 @@ parameter is applicable::
 	APPARMOR AppArmor support is enabled.
 	SERIAL	Serial support is enabled.
 	SH	SuperH architecture is enabled.
+	SHMEM	Full shmem filesystem is enabled.
 	SMP	The kernel is an SMP kernel.
 	SPARC	Sparc architecture is enabled.
 	SWSUSP	Software suspend (hibernation) is enabled.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index c7ac2f3ac99f..aeac92577569 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -4307,6 +4307,11 @@
 	shapers=	[NET]
 			Maximal number of shapers.

+	shmem_size=	[SHMEM]
+			Size of shmem filesystem.  Can be specified as a plain
+			integer, in memory size notation (such as 1024M or 1G),
+			or as a percentage (50%).
+
 	simeth=		[IA-64]
 	simscsi=

diff --git a/init/Kconfig b/init/Kconfig
index b4daad2bac23..7b07f5c448af 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1537,6 +1537,14 @@ config SHMEM
 	  option replaces shmem and tmpfs with the much simpler ramfs code,
 	  which may be appropriate on small systems without swap.

+config SHMEM_SIZE
+	string "Hard-set the shmem filesystem size"
+	default 0
+	depends on SHMEM
+	help
+	  Size of shmem filesystem.  Can be specified as a plain integer, in
+	  memory size notation (such as 1024M or 1G), or as a percentage (50%).
+
 config AIO
 	bool "Enable AIO support" if EXPERT
 	default y
diff --git a/mm/shmem.c b/mm/shmem.c
index cd570cc79c76..ad6109524ad1 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -95,6 +95,12 @@ static struct vfsmount *shm_mnt;
 /* Symlink up to this size is kmalloc'ed instead of using a swappable page */
 #define SHORT_SYMLINK_LEN 128

+/* Default size of shmem filesystem (defaults to 50% of memory) */
+#define SHMEM_SIZE_DEFAULT ((totalram_pages * PAGE_SIZE) / 2)
+/* Final initialized size of shmem filesystem, and stored parameter value */
+unsigned long shmem_size_final;
+char shmem_size_param[20];
+
 /*
  * shmem_fallocate communicates with shmem_fault or shmem_writepage via
  * inode->i_private (with i_mutex making sure that it has only one user at
@@ -123,16 +129,56 @@ struct shmem_options {
 };

 #ifdef CONFIG_TMPFS
+static void shmem_size_parse(void)
+{
+	char *str, *t;
+
+	sprintf(shmem_size_param, "%s", "0");
+	str = strstr(boot_command_line, "shmem_size=");
+	if (str) {
+		t = strsep(&str, " ");
+		t += 11;
+		sprintf(shmem_size_param, "%s", t);
+	}
+}
+
+static void shmem_size(char *str, char *val)
+{
+	char *rest;
+	unsigned long mem = memparse(val, &rest);
+
+	if (*rest == '%') {
+		mem *= (totalram_pages * PAGE_SIZE);
+		do_div(mem, 100);
+	}
+	if (mem > 0) {
+		pr_info("shmem: found %s value: %s (%ld pages, %ld bytes)\n",
+			str, val, (mem / PAGE_SIZE), mem);
+		shmem_size_final = (mem / PAGE_SIZE);
+	}
+}
+
+static void shmem_size_init(void)
+{
+	char tmp[20];
+
+	sprintf(tmp, "%ld", SHMEM_SIZE_DEFAULT);
+	shmem_size("default", tmp);
+#ifdef CONFIG_SHMEM_SIZE
+	shmem_size("kernel configuration", CONFIG_SHMEM_SIZE);
+#endif
+	shmem_size_parse();
+	shmem_size("kernel parameter", shmem_size_param);
+}
+
 static unsigned long shmem_default_max_blocks(void)
 {
-	return totalram_pages() / 2;
+	return shmem_size_final;
 }

 static unsigned long shmem_default_max_inodes(void)
 {
-	unsigned long nr_pages = totalram_pages();
-
-	return min(nr_pages - totalhigh_pages(), nr_pages / 2);
+	return min(totalram_pages - totalhigh_pages, shmem_size_final);
 }
 #endif

@@ -3887,6 +3933,9 @@ int __init shmem_init(void)
 {
 	int error;

+#ifdef CONFIG_TMPFS
+	shmem_size_init();
+#endif
 	shmem_init_inodecache();

 	error = register_filesystem(&shmem_fs_type);
--
2.15.2

Re: [PATCH] shmem: add shmem_size option, set filesystem size

[Qian Cai]

> On Oct 7, 2019, at 6:32 PM, Gary B. Genett <me@garybgenett.net> wrote:
> 
> A specific case where this is necessary would be if the initramfs were
> larger than half of the memory, such as a 2.5GB filesystem with 4GB of
> memory.  Without this option, this causes a kernel panic.  With this
> option, the user may specify something like 75%, which would allow the
> filesystem into memory, while still leaving enough resources to run
> a functioning system.

Why would anyone need such a large an initramfs?

Re: [PATCH] shmem: add shmem_size option, set filesystem size

[-Gary-]

> > A specific case where this is necessary would be if the initramfs were
> > larger than half of the memory, such as a 2.5GB filesystem with 4GB of
> > memory.  Without this option, this causes a kernel panic.  With this
> > option, the user may specify something like 75%, which would allow the
> > filesystem into memory, while still leaving enough resources to run
> > a functioning system.
> 
> Why would anyone need such a large an initramfs?

Thank you for such a quick response, Qian!  I really appreciate your question and consideration of my submission.

The use case here is a live distribution, where the initramfs is a standard filesystem.  It is common for these types of distributions to unpack root into memory.  Most of them use SquashFS, and resize shmem using a smaller initramfs first.

https://grml.org

In my case, I am unpacking straight from the initramfs to alleviate the need for a separate file and packaging everything in an ISO.  It very well could be that I am the only person who has ever needed this, but it has been a requirement for my distribution.

https://sourceforge.net/projects/gary-os
https://github.com/garybgenett/gary-os

It is also possible that other live distributions are taking the approach they are because of the current inability to load an initrd/initramfs directly into memory without manually resizing the shmem filesystem first.

Since I was in there hacking the feature, anyway, I thought it would be worth submitting.

At the least, it should be a "#define", or something.  Right now it is a hard-coded magic token.

Thoughts?

-- Gary

Re: [PATCH] shmem: add shmem_size option, set filesystem size

[Qian Cai]

> On Oct 7, 2019, at 9:15 PM, -Gary- <me@garybgenett.net> wrote:
> 
> The use case here is a live distribution, where the initramfs is a standard filesystem.  It is common for these types of distributions to unpack root into memory.  Most of them use SquashFS, and resize shmem using a smaller initramfs first.
> 
> https://grml.org
> 
> In my case, I am unpacking straight from the initramfs to alleviate the need for a separate file and packaging everything in an ISO.  It very well could be that I am the only person who has ever needed this, but it has been a requirement for my distribution.
> 
> https://sourceforge.net/projects/gary-os
> https://github.com/garybgenett/gary-os
> 
> It is also possible that other live distributions are taking the approach they are because of the current inability to load an initrd/initramfs directly into memory without manually resizing the shmem filesystem first.
> 
> Since I was in there hacking the feature, anyway, I thought it would be worth submitting.
> 
> At the least, it should be a "#define", or something.  Right now it is a hard-coded magic token.

I am not sure if it worth the complication. Right now, you have the choice of resizing shmem like every body else or increasing your system memory a bit.

If there are more compiling reasons to bump the limit, it is probably more appropriate to make it the default to use something like 75% memory if there isn’t much side-effect.

Re: [PATCH] shmem: add shmem_size option, set filesystem size

[-Gary-]

> > At the least, it should be a "#define", or something.  Right now it is a hard-coded magic token.
>
> I am not sure if it worth the complication. Right now, you have the choice of resizing shmem like every body else or increasing your system memory a bit.
>
> If there are more compiling reasons to bump the limit, it is probably more appropriate to make it the default to use something like 75% memory if there isn’t much side-effect.

Thank you again, Qian.  I appreciate the conversation and coaching!

That makes perfect sense.  I agree with the effort to keep things tidy, efficient and supportable.  With your blessing, I will submit a separate patch that only adds a configuration option to set the number of pages, and does not perform any additional math or steps other than reporting the value.  This will at least make it a defined value rather than a magic token, and will allow any expert users who need it to change the value.

I'm very reluctant, if not outright opposed, to changing the default, regardless of how compelling the case.  The possible side-effects of that change in behavior are unforeseen, and the scope of testing required is infeasible.  I sense that you agree with this.

I hope my subsequent patch is low-impact enough to be accepted.

It means a lot that you took the time to review and comment on my work.  I do greatly appreciate it.  This was my first kernel patch submission, and you made it a good experience.  <^)

-- Gary