[Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[John Stultz]

I'd like to discuss some thoughts on how to address the 2038 issues on
32bit architectures. This is important, as vendors are still producing
lots of 32bit hardware, which may very well have 24+ year lifespans
(think industrial control applications, security systems).  NetBSD and
OpenBSD have recently broken their ABI, converted their time_t to long
long, to properly address this. So I'd like to discuss thoughts on how
Linux can do similar despite our no-breaking-userspace rules, after all,
one way or another (almost) all of Linux's 32bit architectures are
terminally broken past 2038.

Thomas (who I don't think can attend due to other plans) and I have had
some small talks about this and we have different initial preferences on
how to go about solving things, so I'd like to present the pros and cons
of the current options we're stewing on, open the discussion up to other
ideas, and see if there's a consensus on which way to go.

thanks
-john

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Andy Lutomirski]

On Mon, May 5, 2014 at 11:33 AM, John Stultz <john.stultz@linaro.org> wrote:
> I'd like to discuss some thoughts on how to address the 2038 issues on
> 32bit architectures. This is important, as vendors are still producing
> lots of 32bit hardware, which may very well have 24+ year lifespans
> (think industrial control applications, security systems).  NetBSD and
> OpenBSD have recently broken their ABI, converted their time_t to long
> long, to properly address this. So I'd like to discuss thoughts on how
> Linux can do similar despite our no-breaking-userspace rules, after all,
> one way or another (almost) all of Linux's 32bit architectures are
> terminally broken past 2038.
>
> Thomas (who I don't think can attend due to other plans) and I have had
> some small talks about this and we have different initial preferences on
> how to go about solving things, so I'd like to present the pros and cons
> of the current options we're stewing on, open the discussion up to other
> ideas, and see if there's a consensus on which way to go.

I'm interested in this discussion.

It would be nice to try to do something about the seconds, nanoseconds
representation in clock_gettime at the same time.  One way or another,
a lot of programs will continue to expect to see seconds and
nanoseconds, but it's simpler and faster to be able to feed userspace
a nanosecond count.

Also, leap seconds :(

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[josh]

On Mon, May 05, 2014 at 11:33:45AM -0700, John Stultz wrote:
> I'd like to discuss some thoughts on how to address the 2038 issues on
> 32bit architectures. This is important, as vendors are still producing
> lots of 32bit hardware, which may very well have 24+ year lifespans
> (think industrial control applications, security systems).  NetBSD and
> OpenBSD have recently broken their ABI, converted their time_t to long
> long, to properly address this. So I'd like to discuss thoughts on how
> Linux can do similar despite our no-breaking-userspace rules, after all,
> one way or another (almost) all of Linux's 32bit architectures are
> terminally broken past 2038.

I would be interested in this, not just because of time_t itself, but as
a general pattern for "how can we transition away from an old and broken
ABI".  Whether by introducing new system calls, new personalities,
seccomp filters, or other mechanisms, we should have some ways to start
such transitions and to smooth them out.  Sure, we never break
userspace, but that just means we need an appropriate
CONFIG_OLD_AND_BUSTED option for as long as people still need the old
ABI.

- Josh Triplett

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Andy Lutomirski]

On Mon, May 5, 2014 at 1:53 PM,  <josh@joshtriplett.org> wrote:
> On Mon, May 05, 2014 at 11:33:45AM -0700, John Stultz wrote:
>> I'd like to discuss some thoughts on how to address the 2038 issues on
>> 32bit architectures. This is important, as vendors are still producing
>> lots of 32bit hardware, which may very well have 24+ year lifespans
>> (think industrial control applications, security systems).  NetBSD and
>> OpenBSD have recently broken their ABI, converted their time_t to long
>> long, to properly address this. So I'd like to discuss thoughts on how
>> Linux can do similar despite our no-breaking-userspace rules, after all,
>> one way or another (almost) all of Linux's 32bit architectures are
>> terminally broken past 2038.
>
> I would be interested in this, not just because of time_t itself, but as
> a general pattern for "how can we transition away from an old and broken
> ABI".  Whether by introducing new system calls, new personalities,
> seccomp filters, or other mechanisms, we should have some ways to start
> such transitions and to smooth them out.  Sure, we never break
> userspace, but that just means we need an appropriate
> CONFIG_OLD_AND_BUSTED option for as long as people still need the old
> ABI.

I encountered this with CONFIG_COMPAT_VDSO, and it would be nice if
there were a clear, mostly-applicable rule for how this should work.

The current situation is that CONFIG_COMPAT_VDSO's name is silly, and
it's 'default n', but the name is stuck so that old configurations
keep being able to run old software.

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[H. Peter Anvin]

On 05/05/2014 01:53 PM, josh@joshtriplett.org wrote:
> 
> I would be interested in this, not just because of time_t itself, but as
> a general pattern for "how can we transition away from an old and broken
> ABI".  Whether by introducing new system calls, new personalities,
> seccomp filters, or other mechanisms, we should have some ways to start
> such transitions and to smooth them out.  Sure, we never break
> userspace, but that just means we need an appropriate
> CONFIG_OLD_AND_BUSTED option for as long as people still need the old
> ABI.
> 

There is absolutely nothing new here... we have dealt with these kinds
of transitions for most of Linux' existence.

However, time_t is a particularly nontrivial issue, because it is not
just a matter of changing the kernel ABI but a *lot* of user space ABIs
also contain this type.  The kernel/glibc interfaces are pretty well set
up to handle this properly these days, but I have very little hope that
all the user space libraries will properly handle having two versions of
a bunch of APIs with different version numbers.

On the other hand, a *limited* form of compatibility could be had by
changing the windowing by changing int32_t (1901-2038) to uint32_t
(1970-2106).  This is similar to how a lot of legacy applications were
"fixed" for the Y2K problem.

That has its own share of problems, in particular legacy applications
might end up thinking that 2039 comes long before 2037, but there is to
some degree a choice of plague or cholera, as we say in Swedish.

	-hpa

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Josh Triplett]

On Mon, May 05, 2014 at 07:12:54PM -0700, H. Peter Anvin wrote:
> On 05/05/2014 01:53 PM, josh@joshtriplett.org wrote:
> > 
> > I would be interested in this, not just because of time_t itself, but as
> > a general pattern for "how can we transition away from an old and broken
> > ABI".  Whether by introducing new system calls, new personalities,
> > seccomp filters, or other mechanisms, we should have some ways to start
> > such transitions and to smooth them out.  Sure, we never break
> > userspace, but that just means we need an appropriate
> > CONFIG_OLD_AND_BUSTED option for as long as people still need the old
> > ABI.
> > 
> 
> There is absolutely nothing new here... we have dealt with these kinds
> of transitions for most of Linux' existence.
> 
> However, time_t is a particularly nontrivial issue, because it is not
> just a matter of changing the kernel ABI but a *lot* of user space ABIs
> also contain this type.  The kernel/glibc interfaces are pretty well set
> up to handle this properly these days, but I have very little hope that
> all the user space libraries will properly handle having two versions of
> a bunch of APIs with different version numbers.

64-bit off_t support seems like a similar issue; all the kernel
interfaces support 64-bit off_t, and glibc has #defines for
64-bit off_t, but a large number of userspace programs don't bother to
define them.

That seems like the right path to transition time_t as well.

- Josh Triplett

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Theodore Ts'o]

On Mon, May 05, 2014 at 07:21:59PM -0700, Josh Triplett wrote:
> 
> 64-bit off_t support seems like a similar issue; all the kernel
> interfaces support 64-bit off_t, and glibc has #defines for
> 64-bit off_t, but a large number of userspace programs don't bother to
> define them.
> 
> That seems like the right path to transition time_t as well.

It's actually quite different, because there off_t isn't used in
userspace libraries or data structures nearly as much as time_t.  It's
not that terrible if you have some libraries which are compiled with
64-bit loff_t (note the different type) and some userspace programs
which aren't.  But the problems if some libraries want to use a 64-bit
time_t and some programs are still only able to support a 32-bit
time_t are much more trickly.

I suspect the only real way of dealing with the problem for 32-bit
architectures is to consider doing a major version bump for all shared
libraries, and for a while, 32-bit distributions might have to ship
two versions of all of the more popular shared libraries.

It might be easier for x86 to simply say, "suck it up", if you care,
you should use x32 ABI.  However, I believe the original concern was
that there might still be 32-bit x86 chips still out there in 2038,
and x32 won't solve that problem, unfortunately.

OTOH, if the main issue is "industrial control and security systems"
as stated in the original problem description,, I very much doubt they
would need GNOME shared libraries.  :-)

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[John Stultz]

On 05/06/2014 05:57 AM, Theodore Ts'o wrote:
> I suspect the only real way of dealing with the problem for 32-bit
> architectures is to consider doing a major version bump for all shared
> libraries, and for a while, 32-bit distributions might have to ship
> two versions of all of the more popular shared libraries.
...
> OTOH, if the main issue is "industrial control and security systems"
> as stated in the original problem description,, I very much doubt they
> would need GNOME shared libraries.  :-)

Well, there's also things like in-car-infotainment systems (which would
annoy future car collectors should they stop working), which may require
GUI toolkits. But I suspect the most problematic cases will be embedded
systems in the literal sense of "embedded in the wall".

But the point being that I think we'll have to consider all existing
32bit applications as broken after 2038 (of course there may be some
which don't use time a t all, or if they do only use relative offsets -
but as a general interface, the existing Linux 32bit ABI is broken). So
if we do something like an ABI break, while providing a compat
personality for existing applications, duplicated libraries would be
necessary, but that extra cost might help motivate vendors to get their
applications ported to the new ABI.

And of course, while rebuilding for the new ABI might be one of the
easier ways to address things, there's no magic bullet to fix all the
places where applications might be casting or storing time_ts as longs
either internally or on fixed wire or on-disk formats. (So folks who
have been planning their retirement around lucrative Y2K-like
contracting work, will still have things to do!).

thanks
-john

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Arnd Bergmann]

On Tuesday 06 May 2014 10:53:57 John Stultz wrote:
> On 05/06/2014 05:57 AM, Theodore Ts'o wrote:
> > I suspect the only real way of dealing with the problem for 32-bit
> > architectures is to consider doing a major version bump for all shared
> > libraries, and for a while, 32-bit distributions might have to ship
> > two versions of all of the more popular shared libraries.
> ...
> > OTOH, if the main issue is "industrial control and security systems"
> > as stated in the original problem description,, I very much doubt they
> > would need GNOME shared libraries.  :-)
> 
> Well, there's also things like in-car-infotainment systems (which would
> annoy future car collectors should they stop working), which may require
> GUI toolkits. But I suspect the most problematic cases will be embedded
> systems in the literal sense of "embedded in the wall".

Medical equipment (and I assume military, which is often in the same
category) is probably a big thing. I remember we had a big discussion
about a product at a former employer when a customer asked for support
beyond 2038. I think we ended up saying that the hardware is fine but
no supported distro would handle this.

	Arnd

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Theodore Ts'o]

On Tue, May 06, 2014 at 08:20:47PM +0200, Arnd Bergmann wrote:
> Medical equipment (and I assume military, which is often in the same
> category) is probably a big thing. I remember we had a big discussion
> about a product at a former employer when a customer asked for support
> beyond 2038. I think we ended up saying that the hardware is fine but
> no supported distro would handle this.

The question for the kernel summit is what sort of solutions can we
suggest where we as kernel developers could help provide a solution
for this problem?

If the answer is "the kernel is fine (or could be trivially made
fine)" but the problems are all at the glibc and/or distro level, then
it's a problem, but I'm not sure we'd be able to make progress on
solving it in this venue.

					- Ted

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[josh]

On Tue, May 06, 2014 at 04:19:59PM -0400, Theodore Ts'o wrote:
> On Tue, May 06, 2014 at 08:20:47PM +0200, Arnd Bergmann wrote:
> > Medical equipment (and I assume military, which is often in the same
> > category) is probably a big thing. I remember we had a big discussion
> > about a product at a former employer when a customer asked for support
> > beyond 2038. I think we ended up saying that the hardware is fine but
> > no supported distro would handle this.
> 
> The question for the kernel summit is what sort of solutions can we
> suggest where we as kernel developers could help provide a solution
> for this problem?
> 
> If the answer is "the kernel is fine (or could be trivially made
> fine)" but the problems are all at the glibc and/or distro level, then
> it's a problem, but I'm not sure we'd be able to make progress on
> solving it in this venue.

I don't think it's entirely trivial to make the kernel fine.  Two topics
that *do* make sense for Kernel Summit:

- How do we fix existing ABIs while making it minimally painful to
  transition existing userspace to the new ABI?  Should we just
  introduce a replacement for every system call that includes a time_t
  and deprecate the old ones, or should we introduce some more
  systematic migration mechanism?

- Can the kernel do anything to make the transition easier for libc or
  other userspace programs?

One possible approach to that second point that might be worth exploring
independently, though I'd expect it to prove controversial:

- Could we move some of the userspace compatibility code out of the
  kernel and into low-level userspace shipped with the kernel?


- Josh Triplett

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Theodore Ts'o]

On Tue, May 06, 2014 at 01:33:37PM -0700, josh@joshtriplett.org wrote:
> 
> I don't think it's entirely trivial to make the kernel fine.  Two topics
> that *do* make sense for Kernel Summit:
> 
> - How do we fix existing ABIs while making it minimally painful to
>   transition existing userspace to the new ABI?  Should we just
>   introduce a replacement for every system call that includes a time_t
>   and deprecate the old ones, or should we introduce some more
>   systematic migration mechanism?

It might be useful if someone were to do a bit of homework first.  How
many system calls (and there will be some ioctl's too, but system
calls will probably be the hard part) that are using a 32-bit time_t
(including struct timeval and struct timespec) or clock_t?  Can
someone produce a list so we know how many interfaces we're talking
about here?

> - Can the kernel do anything to make the transition easier for libc or
>   other userspace programs?

My experience is that mailing list discussions (like this one) are
better at this kind of brain-storming than the actual face-to-face
summit meeting.  Summit meetings are good for weighing tradeoffs
between different possible solutions, but in general, things go much
more smoothly if someone has done some homework and outlined a few
possible proposals (even if they are strawman proposals) to prime the
discussion.

So I'm not just picking on this topic, but in general --- if we have
topics of the form, "we should bell the cat", the program committee
will be more likely to consider it a good use of everyone's time if
there is evidence that some potential technologies for applying the
bell to the cat (and how to handle minor problems like the cat's teeth
and claws) have been considered raised ahead of time.  (And of course,
if someone sends a patch making the whole discussion moot, that's even
better.  :-)

Cheers,

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[John Stultz]

On 05/06/2014 01:50 PM, Theodore Ts'o wrote:
> On Tue, May 06, 2014 at 01:33:37PM -0700, josh@joshtriplett.org wrote:
>> I don't think it's entirely trivial to make the kernel fine.  Two topics
>> that *do* make sense for Kernel Summit:
>>
>> - How do we fix existing ABIs while making it minimally painful to
>>   transition existing userspace to the new ABI?  Should we just
>>   introduce a replacement for every system call that includes a time_t
>>   and deprecate the old ones, or should we introduce some more
>>   systematic migration mechanism?
> It might be useful if someone were to do a bit of homework first.  How
> many system calls (and there will be some ioctl's too, but system
> calls will probably be the hard part) that are using a 32-bit time_t
> (including struct timeval and struct timespec) or clock_t?  Can
> someone produce a list so we know how many interfaces we're talking
> about here?

Sure. If its preferred I can try to get some mail on the list outlining
my and Thomas' proposals w/ various pros and cons to the list prior to
the discussion. And I can make a point to provide specific data on the #
of syscall and ioctl structures that embed time_t (or embed structures
which contain time_t), etc.


Briefly, my proposal (and forgive me for maybe letting my bias creep in
here a bit. I was hoping to work with Thomas to have a more neutral
presentation by Kernel Summit, so this summary is a little rough and
probably not completely fair to all sides) is to introduce a new ABI for
32bit architectures where time_t is defined as a long long. We could
continue supporting the existing ABI via a personality/compat interface.

Pros:
* This would allow applications to be updated by being recompiled,
letting the compiler provide warnings where time_ts are cast down to a
long, etc.
* This follows in the path that the BSDs have gone, which allows us to
leverage their work, where they have already ensured applications they
package are fixed.

Cons:
* Having yet another new compat layer, and all the various compat
structs just for the new time_t size.
* Distros wanting to support existing ABI would need to have compat
libraries as well (much as they do for i386 or x32 on x86_64) - Although
this extra cost helps motivate folks to migrate away from the "old" ABI,
so might not completely be negative.
* Of course, if we do introduce a new ABI w/ my proposal, there may be
other non-time_t related ABI changes folks would like to make on 32bit
systems. So there could be lots of "riders" to the change which could
make the transition more complex.


Thomas (at least as far as my understanding last we spoke) would prefer
not to introduce a whole new ABI, but to introduce new syscalls in
addition existing syscalls which would provide something like ktime_t
(u64 nanosecond value) as main time type. Eventually deprecating all the
syscalls that use time_t.

Pros:
* Various folks have wanted a u64 nanosecond interface as converting
back and forth from timespecs has overhead in userspace, so this solves
that issue as well.
* Avoids hard ABI break for slow transition.

Cons:
* There are lots of syscalls and ioctls that use time_t (sometimes
deeply embedded in structures) for things like timeouts, etc,  so
creating new syscalls for all of those will be substantial effort.
* We'd also have to generate the same syscalls on 64bit systems, where
it would be of minimal benefit.
* All applications would have to be manually converted from time_t
interfaces to the new u64 interfaces (not just a recompile)
* Auditing that all the interfaces have been replaced and getting
applications to move away from them would be more difficult.


And of course, there is the variant of my proposal which Peter already
raised, which is to preserve time_t as 32bits, but define it as a
unsigned long. Basically doing a "quiet" ABI break.

Pros:
* No size changes, no need for compat interfaces
* Kernel already considers time values prior to 1970 invalid, so we
could easily reuse them from the kernel side.

Cons:
* I worry this would lead to lots of subtle breakage in applications.
Its less easy to audit to see if things are fixed or not.
* Applications would still need to make changes if they want to continue
to support time values prior to 1970 (likely using 64bit values
internally anyway).
* Have to decide if actually change the ABI to an unsigned long or not
for newly compiled applications.



>
>> - Can the kernel do anything to make the transition easier for libc or
>>   other userspace programs?
> My experience is that mailing list discussions (like this one) are
> better at this kind of brain-storming than the actual face-to-face
> summit meeting.  Summit meetings are good for weighing tradeoffs
> between different possible solutions, but in general, things go much
> more smoothly if someone has done some homework and outlined a few
> possible proposals (even if they are strawman proposals) to prime the
> discussion.
>
> So I'm not just picking on this topic, but in general --- if we have
> topics of the form, "we should bell the cat", the program committee
> will be more likely to consider it a good use of everyone's time if
> there is evidence that some potential technologies for applying the
> bell to the cat (and how to handle minor problems like the cat's teeth
> and claws) have been considered raised ahead of time.  (And of course,
> if someone sends a patch making the whole discussion moot, that's even
> better.  :-)

Right. So clearly I don't have all the details ready right this moment,
but I hope providing more detail above gives you a better sense of the
current proposals and some confidence that I'll try to do my homework
and have some discussions on lkml before the Kernel Summit discussion.

thanks
-john

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Theodore Ts'o]

On Tue, May 06, 2014 at 03:06:27PM -0700, John Stultz wrote:
> 
> Sure. If its preferred I can try to get some mail on the list outlining
> my and Thomas' proposals w/ various pros and cons to the list prior to
> the discussion. And I can make a point to provide specific data on the #
> of syscall and ioctl structures that embed time_t (or embed structures
> which contain time_t), etc.

It also might be useful to also consider which interfaces are mediated
by glibc, and which aren't (i.e., ioctl's).  We might be able to
separate what might be doable by negotaitons between glibc and the
kernel, and between applications and the shared library ecosystem,
which would allow a soft/slow cutover, but with all or most of the
compatibility in glibc.

> Thomas (at least as far as my understanding last we spoke) would prefer
> not to introduce a whole new ABI, but to introduce new syscalls in
> addition existing syscalls which would provide something like ktime_t
> (u64 nanosecond value) as main time type. Eventually deprecating all the
> syscalls that use time_t.

One of the questions to evaluate this proposal is how many syscalls
this would take.  And if we the goal is to "avoid a hard ABI break",
then that also means doing something like the Large File Support hack
(i.e., open64, read64, etc.) which is application visible.  Right?
But the problem is unless you get all applications to use these
non-standard, non-POSIX interfaces --- or you need to get them to use
a magic #define, ala the LFS --- and good luck getting all
applications to do make that change, or to get distributions to modify
their build scrpits to include that --- and then it's equivalent to a
hard ABI break, since it means time_t changes size.

> Right. So clearly I don't have all the details ready right this moment,
> but I hope providing more detail above gives you a better sense of the
> current proposals and some confidence that I'll try to do my homework
> and have some discussions on lkml before the Kernel Summit discussion.

Something else to consider is whether we would need to have some glibc
folks present at the kernel summit if we want to discuss it there, and
whether we should also think about also planning some discussions at
the Plumber's Conference --- because there will certainly be userspace
impacts that will need to be taken into account.

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Jonathan Corbet]

On Tue, 6 May 2014 22:07:43 -0400
Theodore Ts'o <tytso@mit.edu> wrote:

> Something else to consider is whether we would need to have some glibc
> folks present at the kernel summit if we want to discuss it there, and
> whether we should also think about also planning some discussions at
> the Plumber's Conference --- because there will certainly be userspace
> impacts that will need to be taken into account.

In fact, a year 2038 miniconf at Plumbers seems like a good idea if the
right people can be pulled into it.  

jon

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[John Stultz]

On 05/06/2014 07:07 PM, Theodore Ts'o wrote:
> On Tue, May 06, 2014 at 03:06:27PM -0700, John Stultz wrote:
>> Right. So clearly I don't have all the details ready right this moment,
>> but I hope providing more detail above gives you a better sense of the
>> current proposals and some confidence that I'll try to do my homework
>> and have some discussions on lkml before the Kernel Summit discussion.
> Something else to consider is whether we would need to have some glibc
> folks present at the kernel summit if we want to discuss it there, and
> whether we should also think about also planning some discussions at
> the Plumber's Conference --- because there will certainly be userspace
> impacts that will need to be taken into account.

Do you have a sense of who might be a good glibc contact for this sort
of thing?

And yes, When I ran into Thomas at lsf-mm, we talked about splitting the
duty here, with me bringing it up at Kernel Summit as Thomas cannot
attend, and since I cannot attend Plumbers this year having Thomas
covering it there.

thanks
-john

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Theodore Ts'o]

On Wed, May 07, 2014 at 10:28:35AM -0700, John Stultz wrote:
> 
> Do you have a sense of who might be a good glibc contact for this sort
> of thing?

I'm not that familiar with the internals of the glibc project post
Uli, but I'd suggest asking Roland McGrath and DaveM to start.

> And yes, When I ran into Thomas at lsf-mm, we talked about splitting the
> duty here, with me bringing it up at Kernel Summit as Thomas cannot
> attend, and since I cannot attend Plumbers this year having Thomas
> covering it there.

Excellent!

					- Ted

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

On Tue, 2014-05-06 at 22:07 -0400, Theodore Ts'o wrote:
[...]
> One of the questions to evaluate this proposal is how many syscalls
> this would take.  And if we the goal is to "avoid a hard ABI break",
> then that also means doing something like the Large File Support hack
> (i.e., open64, read64, etc.) which is application visible.  Right?
> But the problem is unless you get all applications to use these
> non-standard, non-POSIX interfaces --- or you need to get them to use
> a magic #define, ala the LFS --- and good luck getting all
> applications to do make that change,

LFS is far from universally supported by applications, 17 years after it
was standardised.  In fact, many applications recently regressed due to
a broken test for LFS in autoconf <https://bugs.debian.org/742780>.  It
doesn't seem like a good example to follow.

> or to get distributions to modify
> their build scrpits to include that --- and then it's equivalent to a
> hard ABI break, since it means time_t changes size.
[...]

However this is done, almost every library that includes time_t in its
API will change ABI.  I say 'almost' because glibc will probably use
symbol versioning or mangling to maintain binary compatibility, but most
library maintainers won't go to that trouble.

Ben.

-- 
Ben Hutchings
The obvious mathematical breakthrough [to break modern encryption] would be
development of an easy way to factor large prime numbers. - Bill Gates

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Theodore Ts'o]

On Thu, May 08, 2014 at 09:37:06PM +0100, Ben Hutchings wrote:
> 
> LFS is far from universally supported by applications, 17 years after it
> was standardised.  In fact, many applications recently regressed due to
> a broken test for LFS in autoconf <https://bugs.debian.org/742780>.  It
> doesn't seem like a good example to follow.

Yes, that was my point.

> However this is done, almost every library that includes time_t in its
> API will change ABI.  I say 'almost' because glibc will probably use
> symbol versioning or mangling to maintain binary compatibility, but most
> library maintainers won't go to that trouble.

Agreed.  This is why I'm not sure anything other than a hard ABI break
is realistic.  Yes, it's incredibly painful, and the distro's will
probably be very unhappy, but I suspect the alternatives are worse.
The only real question is do we start trying to deal with the pain
now, or in 2020, or in 2030, or 2035, or even worse, in 2037....

Given what what I saw with Y2K, if I was going to participate in a
betting pool on the question, I'd probably put my money down for 2035
or so.  :-/

     	   	       	     	      	 - Ted

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Arnd Bergmann]

On Friday 09 May 2014 11:10:43 Theodore Ts'o wrote:
> On Thu, May 08, 2014 at 09:37:06PM +0100, Ben Hutchings wrote:
> > 
> > LFS is far from universally supported by applications, 17 years after it
> > was standardised.  In fact, many applications recently regressed due to
> > a broken test for LFS in autoconf <https://bugs.debian.org/742780>.  It
> > doesn't seem like a good example to follow.
> 
> Yes, that was my point.
> 
> > However this is done, almost every library that includes time_t in its
> > API will change ABI.  I say 'almost' because glibc will probably use
> > symbol versioning or mangling to maintain binary compatibility, but most
> > library maintainers won't go to that trouble.
> 
> Agreed.  This is why I'm not sure anything other than a hard ABI break
> is realistic.  Yes, it's incredibly painful, and the distro's will
> probably be very unhappy, but I suspect the alternatives are worse.
> The only real question is do we start trying to deal with the pain
> now, or in 2020, or in 2030, or 2035, or even worse, in 2037....
> 
> Given what what I saw with Y2K, if I was going to participate in a
> betting pool on the question, I'd probably put my money down for 2035
> or so.  :-/

I think an important distinction is that the majority of systems that
will be seriously affected are embedded machines, which run a custom
user space anyway.

x86-32 PCs and end-user distros are going to be largely extinct
in a couple of years and replaced by x64-64 or arm64 depending
on who you ask, and arm32 Android phones are going to be
replaced with arm64 hardware shortly after, or they see an ABI
break before then anyway.
The typical embedded machines don't even use glibc, and they
cross-build everything from source.

While I'm sure there will be the odd case where people run into the
2038 problem on existing software, we can for now solve a lot of
problems just by getting the kernel interfaces in place and making
it as easy as possible to build a user space that uses the new types.

	Arnd

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Josh Triplett]

On Fri, May 09, 2014 at 10:39:13PM +0200, Arnd Bergmann wrote:
> On Friday 09 May 2014 11:10:43 Theodore Ts'o wrote:
> > On Thu, May 08, 2014 at 09:37:06PM +0100, Ben Hutchings wrote:
> > > 
> > > LFS is far from universally supported by applications, 17 years after it
> > > was standardised.  In fact, many applications recently regressed due to
> > > a broken test for LFS in autoconf <https://bugs.debian.org/742780>.  It
> > > doesn't seem like a good example to follow.
> > 
> > Yes, that was my point.
> > 
> > > However this is done, almost every library that includes time_t in its
> > > API will change ABI.  I say 'almost' because glibc will probably use
> > > symbol versioning or mangling to maintain binary compatibility, but most
> > > library maintainers won't go to that trouble.
> > 
> > Agreed.  This is why I'm not sure anything other than a hard ABI break
> > is realistic.  Yes, it's incredibly painful, and the distro's will
> > probably be very unhappy, but I suspect the alternatives are worse.
> > The only real question is do we start trying to deal with the pain
> > now, or in 2020, or in 2030, or 2035, or even worse, in 2037....
> > 
> > Given what what I saw with Y2K, if I was going to participate in a
> > betting pool on the question, I'd probably put my money down for 2035
> > or so.  :-/
> 
> I think an important distinction is that the majority of systems that
> will be seriously affected are embedded machines, which run a custom
> user space anyway.
> 
> x86-32 PCs and end-user distros are going to be largely extinct
> in a couple of years and replaced by x64-64 or arm64 depending
> on who you ask, and arm32 Android phones are going to be
> replaced with arm64 hardware shortly after, or they see an ABI
> break before then anyway.
> The typical embedded machines don't even use glibc, and they
> cross-build everything from source.

In particular, even systems that want some of the properties of 32-bit
on 64-bit hardware can use x32; the concern is with new systems that
don't support 64-bit at all.  Hence why we need to solve the problem
*today*, so that the devices we're building in the next few years will
survive 2038.

- Josh Triplett

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[H. Peter Anvin]

On 05/09/2014 03:33 PM, Josh Triplett wrote:
>>
>> I think an important distinction is that the majority of systems that
>> will be seriously affected are embedded machines, which run a custom
>> user space anyway.
>>
>> x86-32 PCs and end-user distros are going to be largely extinct
>> in a couple of years and replaced by x64-64 or arm64 depending
>> on who you ask, and arm32 Android phones are going to be
>> replaced with arm64 hardware shortly after, or they see an ABI
>> break before then anyway.
>> The typical embedded machines don't even use glibc, and they
>> cross-build everything from source.
> 
> In particular, even systems that want some of the properties of 32-bit
> on 64-bit hardware can use x32; the concern is with new systems that
> don't support 64-bit at all.  Hence why we need to solve the problem
> *today*, so that the devices we're building in the next few years will
> survive 2038.
> 

I used to think 32-bit devices would be extinct by the mid-2020s.  It is
now obvious that not only will that be wrong, it will be wrong in the
most dramatic way possible... simply because all the places where we
currently have $0.25 8-bit microcontrollers running trivial operating
systems we'll have $0.25 32-bit microcontrollers and a fair chunk of
them will run Linux.  As we're getting to the point where the most
expensive part of the microcontroller is the package, there is simply no
reason to not have a powerful CPU with a real OS and minimize the amount
of time spent programming the damned thing.

Not to mention that the Internet of Things is going to mean many of them
are going to want to be Internet-connected.

	-hpa

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Rafael J. Wysocki]

On Friday, May 09, 2014 05:16:34 PM H. Peter Anvin wrote:
> On 05/09/2014 03:33 PM, Josh Triplett wrote:
> >>
> >> I think an important distinction is that the majority of systems that
> >> will be seriously affected are embedded machines, which run a custom
> >> user space anyway.
> >>
> >> x86-32 PCs and end-user distros are going to be largely extinct
> >> in a couple of years and replaced by x64-64 or arm64 depending
> >> on who you ask, and arm32 Android phones are going to be
> >> replaced with arm64 hardware shortly after, or they see an ABI
> >> break before then anyway.
> >> The typical embedded machines don't even use glibc, and they
> >> cross-build everything from source.
> > 
> > In particular, even systems that want some of the properties of 32-bit
> > on 64-bit hardware can use x32; the concern is with new systems that
> > don't support 64-bit at all.  Hence why we need to solve the problem
> > *today*, so that the devices we're building in the next few years will
> > survive 2038.
> > 
> 
> I used to think 32-bit devices would be extinct by the mid-2020s.  It is
> now obvious that not only will that be wrong, it will be wrong in the
> most dramatic way possible... simply because all the places where we
> currently have $0.25 8-bit microcontrollers running trivial operating
> systems we'll have $0.25 32-bit microcontrollers and a fair chunk of
> them will run Linux.  As we're getting to the point where the most
> expensive part of the microcontroller is the package, there is simply no
> reason to not have a powerful CPU with a real OS and minimize the amount
> of time spent programming the damned thing.
> 
> Not to mention that the Internet of Things is going to mean many of them
> are going to want to be Internet-connected.

Yeah.  And there's a good chance that some of them will still be kind of
doing something in 2038.

Rafael

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Grant Likely]

On Fri, 09 May 2014 17:16:34 -0700, "H. Peter Anvin" <hpa@zytor.com> wrote:
> On 05/09/2014 03:33 PM, Josh Triplett wrote:
> >>
> >> I think an important distinction is that the majority of systems that
> >> will be seriously affected are embedded machines, which run a custom
> >> user space anyway.
> >>
> >> x86-32 PCs and end-user distros are going to be largely extinct
> >> in a couple of years and replaced by x64-64 or arm64 depending
> >> on who you ask, and arm32 Android phones are going to be
> >> replaced with arm64 hardware shortly after, or they see an ABI
> >> break before then anyway.
> >> The typical embedded machines don't even use glibc, and they
> >> cross-build everything from source.
> > 
> > In particular, even systems that want some of the properties of 32-bit
> > on 64-bit hardware can use x32; the concern is with new systems that
> > don't support 64-bit at all.  Hence why we need to solve the problem
> > *today*, so that the devices we're building in the next few years will
> > survive 2038.
> > 
> 
> I used to think 32-bit devices would be extinct by the mid-2020s.  It is
> now obvious that not only will that be wrong, it will be wrong in the
> most dramatic way possible... simply because all the places where we
> currently have $0.25 8-bit microcontrollers running trivial operating
> systems we'll have $0.25 32-bit microcontrollers and a fair chunk of
> them will run Linux.  As we're getting to the point where the most
> expensive part of the microcontroller is the package, there is simply no
> reason to not have a powerful CPU with a real OS and minimize the amount
> of time spent programming the damned thing.
> 
> Not to mention that the Internet of Things is going to mean many of them
> are going to want to be Internet-connected.

And yet still, once deployed will never get updated. :-(

g.

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[H. Peter Anvin]

On 05/15/2014 05:18 AM, Grant Likely wrote:
> 
> And yet still, once deployed will never get updated. :-(
> 

That is a whole separate problem.  I'm not quite convinced that it will
work that way... maybe ubiquitous upgrades will be a solved problem by then.

	-hpa

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Jason Cooper]

On Thu, May 15, 2014 at 10:20:24AM -0700, H. Peter Anvin wrote:
> On 05/15/2014 05:18 AM, Grant Likely wrote:
> > 
> > And yet still, once deployed will never get updated. :-(
> > 
> 
> That is a whole separate problem.  I'm not quite convinced that it will
> work that way... maybe ubiquitous upgrades will be a solved problem by then.

With the infrastructure to support verification, which brings in key
management.  Otherwise, it'll be an easy attack surface.  I'm not
optimistic...

The alternative is to make the devices so trivially disposable that
outdated/vulnerable devices are rejected from the network.  The user
then orders a replacement.

thx,

Jason.

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Andy Lutomirski]

Here's a suggestion.  I don't know whether it's at all practical.
What if we added a mode in which all structs passed into and out of
the kernel use the 64-bit variant *even for 32-bit code*?  Think of it
as the opposite of x32.

This has a big benefit: there's no whole new ABI to maintain.  The
downside is that every single use of unsigned long on the uapi side
will have to change to __u64, etc.  This might be doable
automatically.  Another downside is that some structs may be laid out
differently in the 32-bit C ABI as compared to the 64-bit C ABI, even
given the same primitive type sizes.

Pointers would be tricky.  Hmm.  Could we get gcc and clang to add
__attribute__((64_bit_layout)) if needed?

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Daniel Phillips]

On 05/06/2014 01:19 PM, Theodore Ts'o wrote:
> On Tue, May 06, 2014 at 08:20:47PM +0200, Arnd Bergmann wrote:
>> Medical equipment (and I assume military, which is often in the same
>> category) is probably a big thing. I remember we had a big discussion
>> about a product at a former employer when a customer asked for support
>> beyond 2038. I think we ended up saying that the hardware is fine but
>> no supported distro would handle this.
> The question for the kernel summit is what sort of solutions can we
> suggest where we as kernel developers could help provide a solution
> for this problem?
>
> If the answer is "the kernel is fine (or could be trivially made
> fine)" but the problems are all at the glibc and/or distro level, then
> it's a problem, but I'm not sure we'd be able to make progress on
> solving it in this venue.
>

Hi Ted,

At minimum, the collective kernel voice has a great soapbox to stand on, 
it would not make sense to waste that, and no doubt there will always be 
more that could be done on the kernel side. For example, the kernel has 
a definitive vantage point to witness all 32 bit timestamps coming in 
and going out with the sign bit set. Perhaps some logging mechanism 
would be helpful in setting up audit systems to use in sandbox testing 
and triage on Jan 1, 2038, with a view to clawing back the sign bit.

Regards,

Daniel

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Luck, Tony]

> and going out with the sign bit set. Perhaps some logging mechanism 
> would be helpful in setting up audit systems to use in sandbox testing 
> and triage on Jan 1, 2038, with a view to clawing back the sign bit.

We can't wait till Jan 2038 to commit a change - programs want to
work with dates in the future - allegedly the first Y2038 bugs began
appearing in 2008 when banks issuing 30 year mortgages were unable
to print the correct date for final payment.

-Tony

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Daniel Phillips]

On 05/06/2014 02:56 PM, Luck, Tony wrote:
>> and going out with the sign bit set. Perhaps some logging mechanism
>> would be helpful in setting up audit systems to use in sandbox testing
>> and triage on Jan 1, 2038, with a view to clawing back the sign bit.
> We can't wait till Jan 2038 to commit a change - programs want to
> work with dates in the future - allegedly the first Y2038 bugs began
> appearing in 2008 when banks issuing 30 year mortgages were unable
> to print the correct date for final payment.
> -Tony
To clarify, sandbox testing can and should start any time. The "triage" 
part attempts to answer the question: what about bad behavior that 
somehow slips through the net and manifests only on the day of 
reckoning? The idea is to provide admins the ability to monitor their 
systems and spot issues at source as opposed to letting things go 
silently wrong and potentially cause problems far downstream. Another 
way of saying it: what way is there to be sure that the problematic 
interfaces are no longer being used, other than monitoring them? It's 
clear this is the responsible thing to do, what is less clear is whether 
the task could benefit from kernel help. Say, CONFIG_Y2038_PARANOIA.

Regards,

Daniel

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Grant Likely]

On Tue, 6 May 2014 08:57:41 -0400, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, May 05, 2014 at 07:21:59PM -0700, Josh Triplett wrote:
> > 
> > 64-bit off_t support seems like a similar issue; all the kernel
> > interfaces support 64-bit off_t, and glibc has #defines for
> > 64-bit off_t, but a large number of userspace programs don't bother to
> > define them.
> > 
> > That seems like the right path to transition time_t as well.
> 
> It's actually quite different, because there off_t isn't used in
> userspace libraries or data structures nearly as much as time_t.  It's
> not that terrible if you have some libraries which are compiled with
> 64-bit loff_t (note the different type) and some userspace programs
> which aren't.  But the problems if some libraries want to use a 64-bit
> time_t and some programs are still only able to support a 32-bit
> time_t are much more trickly.
> 
> I suspect the only real way of dealing with the problem for 32-bit
> architectures is to consider doing a major version bump for all shared
> libraries, and for a while, 32-bit distributions might have to ship
> two versions of all of the more popular shared libraries.
> 
> It might be easier for x86 to simply say, "suck it up", if you care,
> you should use x32 ABI.  However, I believe the original concern was
> that there might still be 32-bit x86 chips still out there in 2038,
> and x32 won't solve that problem, unfortunately.

And not just x86. There will be a very long tail of 32-bit embedded platforms
that will continue to be produced in huge quantities because they are
cheap. Tim's cereal box freebie Linux machine isn't far fetched.

g.

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[H. Peter Anvin]

On 05/05/2014 07:21 PM, Josh Triplett wrote:
> 
> 64-bit off_t support seems like a similar issue; all the kernel
> interfaces support 64-bit off_t, and glibc has #defines for
> 64-bit off_t, but a large number of userspace programs don't bother to
> define them.
> 
> That seems like the right path to transition time_t as well.
> 

The transition to a 64-bit off_t was still enormously complex, which
involved defining entirely new types and APIs which were then optionally
#defined to the traditional names.  Hence the _FILE_OFFSET_BITS opt-in.

For time_t it is even worse.

	-hpa

Re: [Ksummit-discuss] [CORE TOPIC] Dealing with 2038

[Li Zefan]

On 2014/5/6 2:33, John Stultz wrote:
> I'd like to discuss some thoughts on how to address the 2038 issues on
> 32bit architectures. This is important, as vendors are still producing
> lots of 32bit hardware, which may very well have 24+ year lifespans
> (think industrial control applications, security systems).

This is definitely true for some of our products, so we're aware of this
issue and concerned about it.

> NetBSD and
> OpenBSD have recently broken their ABI, converted their time_t to long
> long, to properly address this. So I'd like to discuss thoughts on how
> Linux can do similar despite our no-breaking-userspace rules, after all,
> one way or another (almost) all of Linux's 32bit architectures are
> terminally broken past 2038.
> 
> Thomas (who I don't think can attend due to other plans) and I have had
> some small talks about this and we have different initial preferences on
> how to go about solving things, so I'd like to present the pros and cons
> of the current options we're stewing on, open the discussion up to other
> ideas, and see if there's a consensus on which way to go.
> 
> thanks
> -john