arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <>
Subject Re: [DISCUSS][Format] Time Interval Changes
Date Tue, 02 Apr 2019 17:02:59 GMT
Since there were some mentions of leap seconds:

I think the intent of the timedelta/duration type should be to express
the difference between UNIX timestamps (from second to nanosecond
resolution), which don't include leap seconds. We use the
timedelta64[ns] type in pandas for example, which is a
nanosecond-resolution difference of UNIX timestamps.

On Tue, Apr 2, 2019 at 10:05 AM Jacques Nadeau <> wrote:
> >
> > I could go either way, it has some benefits for forward compatibility I
> > suppose, but on the other hand YAGNI, if you feel strongly, I'm ok
> > including it.  However, the more optional fields we have for a specific
> > enum value, makes me lean more towards a new type instead of just an enum.
> >
> I'm okay with skipping for now. Appreciate the focus on only what we
> actually need.
> > Could you elaborate on defining standard arithmetic conversions between
> > time-delta/duration in seconds and other time unit (days, months, years) as
> > part of the standard/format, I'm still not sure I understand what the
> > use-case is here.
> >
> Here goes nothing...
> Seems like there are two options for durations:
> 1) they aren't related to any other type
> 2) they have a relationship to timestamps and dates.
> If 1, then the only thing I could understand is real world duration how
> seconds are defined (and fractions thereof). E.g. [1] :D. In this
> situation, there is no way to express any unit of time of higher
> granularity than a second (e.g. days) since it is up to application
> implementer to define the relationship. This severely limits the
> expressiveness of the concept. (I can't ever use something TimeUnit.DAYS)
> and stops the ability to cover the existing interval YEAR_MONTH type I
> believe (since it has a resolution of months).
> If 2, then we must define the canonical value of ts + duration, otherwise
> duration are somewhat meaningless, thus the proposed translation chart
> (which causes its own oddities depending on the resolution of the time type
> you are adding to).
> That being said, having started to remember previous discussions on this,
> I'm most inclined to simply pick #1 and ignore the need for anything more.
> The curiousness of interval math in database systems underscores the fact
> that it apparently doesn't matter that much. In most cases, today + 3
> months is close enough to today + 90 days for government work.
> Let's +2 a patch and get it merged quickly so we never have to think about
> this again :)
> [1]  "the duration of 9,192,631,770 periods
> <> of the radiation corresponding to
> the transition between the two hyperfine levels
> <> of the ground state of
> the caesium-133 <> atom" (at a
> temperature of 0 K <>)
> >

View raw message