How to parse ISO8601 dates with linux date command

Question

I'm trying to use the date command to generate a file timestamp that the date command itself can interpret. However, the date command does not seem to like its own output, and I am not sure how to work around this. Case in point:

sh-4.2$ date
Fri Jan  3 14:22:19 PST 2014
sh-4.2$ date +%Y%m%dT%H%M
20140103T1422
sh-4.2$ date -d "20140103T1422"
Thu Jan  2 23:22:00 PST 2014

date appears to be interpreting the string with an offset of 15 hours. Are there any known workarounds for this?

Edit: this is not an issue of display:

sh-4.2$ date +%s
1388791096
sh-4.2$ date +%Y%m%dT%H%M
20140103T1518
sh-4.2$ date -d 20140103T1518 +%s
1388737080
sh-4.2$ python
Python 3.3.3 (default, Nov 26 2013, 13:33:18) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 1388737080 - 1388791096
-54016
>>> 54016/3600
15.004444444444445
>>> 

It's still off by 15 hours when displayed as a unix timestamp.

EDIT #1

Maybe I should pose this question a little differently. Say I have a list of ISO8601 basic timestamps of the form:

• YYYYMMDDThhmm
• YYYYMMDDThhmmss

What is the simplest way to convert them to the corresponding Unix timestamps?

For example:

- 20140103T1422   = 1388787720
- 20140103T142233 = 1388787753

Answer 1

The coreutils info docs says that ISO 8601 "extended format" is supported.

You'll need to add hyphens, colons, and a +%z to make it work.

$ date +"%Y-%m-%dT%H:%M:%S%z"
2014-01-03T16:08:23-0800
$ date -d 2014-01-03T16:08:23-0800
Fri Jan  3 16:08:23 PST 2014

---

To answer your second part of the question...

Since the date format only contains numbers and symbols, you could replace each symbol with a unique letter, e.g. using tr

$ ts="$(date +"%Y-%m-%dT%H:%M:%S%z" | tr -- '-:+' 'hcp')"; echo "$ts"
2014h01h03T16c18c04h0800
$ date -d "$(echo "$ts" | tr -- 'hcp' '-:+')"
Fri Jan  3 16:18:04 PST 2014

Or you could parse it using the T and the - or + as separators, e.g. using shell ${var%word} and ${var#word} expansion

$ ts="$(date +"%Y%m%dT%H%M%S%z")"; echo "$ts"
20140103T162228-0800
$ date=${ts%T*}; time=${ts#*T}
etc.    

or using bash regular expression matching

$ ts="$(date +"%Y%m%dT%H%M%S%z")"; echo "$ts"
20140103T165611-0800
$ [[ "$ts" =~ (.*)(..)(..)T(..)(..)(..)(.....) ]]
$ match=("${BASH_REMATCH[@]}")
$ Y=${match[1]}; m=${match[2]}; d=${match[3]}; H=${match[4]}; M=${match[5]}; S=${match[6]}; z=${match[7]}
$ date -d "$Y-$m-$d"T"$H:$M:$S$z"
Fri Jan  3 16:56:11 PST 2014

or Perl, Python, etc. etc.

Answer 2

You ask for "known workarounds." Here is a simple one:

$ date -d "$(echo 20140103T1422 | sed 's/T/ /')"
Fri Jan  3 14:22:00 PST 2014

This uses sed to replace "T" with a space. The result is a format that date understands.

If we add seconds onto the ISO8601 date, then date requires more changes:

$ date -d "$(echo 20140103T142211 | sed -r 's/(.*)T(..)(..)(..)/\1 \2:\3:\4/')"
Fri Jan  3 14:22:11 PST 2014

In the above, sed replaces the "T" with a space and also separates HHMMSS into HH:MM:SS.

Answer 3

GNU coreutils have only supported ISO 8601 dates as input since version 8.13 (released on 2011-09-08). You must be using an older version.

Under older versions, you need to replace the T by a space. Otherwise it is interpreted as a US military time zone.

Even under recent versions, only the fully punctuated form is recognized, not the basic format with only digits and a T in the middle.

# Given a possibly abbreviated ISO date $iso_date...
date_part=${iso_date%%T*}
if [ "$date_part" != "$iso_date" ]; then
  time_part=${abbreviated_iso_date#*T}
  case ${iso_date#*T} in
    [!0-9]*) :;;
    [0-9]|[0-9][0-9]) time_part=${time_part}:00;;
    *)
      hour=${time_part%${time_part#??}}
      minute=${time_part%${time_part#????}}; minute=${minute#??}
      time_part=${hour}:${minute}:${time_part#????};;
  esac
else
  time_part=
fi
date -d "$date_part $time_part"

Answer 4

I did notice this note in the man page for date.

DATE STRING
      The --date=STRING is a mostly free format human readable date string
      such as "Sun, 29 Feb 2004 16:21:42 -0800"  or  "2004-02-29
      16:21:42"  or  even  "next Thursday".  A date string may contain 
      items indicating calendar date, time of day, time zone, day of
      week, relative time, relative date, and numbers.  An empty string 
      indicates the beginning of the day.  The date  string  format
      is more complex than is easily documented here but is fully described 
      in the info documentation.

It isn't conclusive but it doesn't explicitly show a time format string that includes the T as you're attempting, for [ISO 8601]. As @Gilles answer indicated, the support of ISO 8601 in GNU CoreUtils is relatively new.

Re-formatting the string

You can use Perl to reformulate your string.

Example:

$ date -d "$(perl -pe 's/(.*)T(\d{2})(\d{2})(\d{2})/$1 $2:$3:$4/' \
    <<<"20140103T142233")"
Fri Jan  3 14:22:33 EST 2014

You can make this handle both strings that include seconds and those that do not.

20140103T1422:

$ date -d "$(perl -pe 's/^(.*)T(\d{2})(\d{2})(\d{2})$/$1 $2:$3:$4/ || \
     s/^(.*)T(\d{2})(\d{2})$/$1 $2:$3:00/' <<<"20140103T1422")"
Fri Jan  3 14:22:00 EST 2014

20140103T142233:

$ date -d "$(perl -pe 's/^(.*)T(\d{2})(\d{2})(\d{2})$/$1 $2:$3:$4/ || \
     s/^(.*)T(\d{2})(\d{2})$/$1 $2:$3:00/' <<<"20140103T142233")"
Fri Jan  3 14:22:33 EST 2014

Answer 5

According to the man page of date, the format that you output is not the same as what date expects as input. This is what the man page says:

date [-u|--utc|--universal] [MMDDhhmm[[CC]YY][.ss]]

So you could do it like this:

# date +%m%d%H%M%Y
010402052014
# date 010402052014
Sat Jan  4 02:05:00 EAT 2014

Because in the variables that are used to define the output string, +%m%d%H%M%Y would be equal to what it expects as input.