drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefán Baxter <ste...@activitystream.com>
Subject Continued Avro Frustration
Date Fri, 01 Apr 2016 12:43:44 GMT
Hi,

Is it at all possible that we are the only company trying to use Avro with
Drill to some serious extent?

We continue to coma across all sorts of embarrassing shortcomings like the
one we are dealing with now where a schema change exception is thrown even
when working with a single Avro file (that has the same schema).

Can a non project member call for a discussion on this topic and the level
of support that is offered for Avro in Drill?

My discussion topics would be:

   - Strange schema validation that ... :
   ... currently fails on single file
   ... prevents dirX variables to work
   ... would require Drill to scan all Avro files to establish schema (even
   when pruning would be used)
   ... would ALWAY fail for old queries if the an old Avro file, containing
   the original fields, was removed and could not be scanned
   ... does not rhyme with the "eliminate ETL" and "Evolving Schema" goals
   of Drill

   - Simple union types do not work to declare nullable fields

   - Drill can not read Parquet that is created by parquet-mr-avro

   - What is the intention for Avro in Drill
   - Should we select to use some other format to buffer/badge data before
   creating a Parquet file for it?

   - The culture here regarding talking about boring/hard topics like this
   - Where serious complaints/issues are met with silence
   - I know full well that my frustration shines through here and that it
   not helping but this Drill+Avro mess is really getting too much for us to
   handle

Look forward do discuss this here or during the next hangout.

Regards,
 -Stefán (or ... mr. old & frustrated)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message