drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Rogers <prog...@mapr.com>
Subject Implicit columns and maps
Date Mon, 09 Oct 2017 17:45:15 GMT
Hi All,

Drill provides a set of “implicit” columns to describe files: filename, suffix, fan and
filepath. Drill also provides an open-ended set of partition columns: dir0, dir1, dir2, etc.

Not all readers support the above: some do and some don’t.

Drill semantics seem to treat these as semi-reserved words when a reader supports implicit
columns. If a table has a “suffix” column, then Drill will treat “suffix” as an implicit
column, ignoring the table column. If the user wants that table column, they can use a session
option to temporarily rename the implicit column. A bit odd, perhaps, but it is our solution.

What is our desired behavior, however, if the user asks for a column that includes an implicit
column as a prefix: “suffix.a”? Clearly, here, “suffix” is a map (i.e. structure)
and “a” is a field within that map. Since the implicit “suffix” is never a map, should
we:

1) Assume that, here, “suffix” is a map column projected from the table?
2) Issue an error?
3) Ignore the “.a” part and just return “suffix” as an implicit column?
4) Something else?

The code is murky on this point because JSON is implemented far differently than text files
and so on. Each has its own rules. Do we need consistency of behavior, or is reader-specific
behavior the expected design?

Thanks,

- Paul

Mime
View raw message