drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefán Baxter (JIRA) <j...@apache.org>
Subject [jira] [Commented] (DRILL-4548) Avro fails on schema changes even when selecting from a single file (Union related)
Date Mon, 28 Mar 2016 22:34:25 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215014#comment-15215014

Stefán Baxter commented on DRILL-4548:

I removed all the union ( ["null" .. ) settings and ended up adding default values to all
fields (empty strings and 0) which fixed this.

The Avro plugin is not handling simple Unions correctly

> Avro fails on schema changes even when selecting from a single file (Union related)
> -----------------------------------------------------------------------------------
>                 Key: DRILL-4548
>                 URL: https://issues.apache.org/jira/browse/DRILL-4548
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Avro
>    Affects Versions: 1.6.0, 1.7.0
>            Reporter: Stefán Baxter
> Hi,
> I have reworked/refactored our Avro based logging system trying to make the whole Drill
+ Avro->Parquet experience a bit more agreeable.
> Long story short I'm getting this error when selecting form multiple Avro files even
though these files share the EXCACT same schema:
> Error: UNSUPPORTED_OPERATION ERROR: Hash aggregate does not support schema changes
> Fragment 0:0
> [Error Id: 00d49aa2-5564-497e-a330-e852d5889beb on swift:31010] (state=,code=0)
> We are using union types but only to allow for null values as seems to be supported by
drill as per this comment in the Drill code: 
> // currently supporting only nullable union (optional fields) like ["null", "some-type"].
> This happens for a very simple group_by + count(*) query that only uses two fields in
Avro and neither one of them uses a Union construct so and both of them contain string values
in every case.
> I now think this has nothing to do with the union types since the query uses only simple
string, unless there is a full schema validation done on the content of the files rather then
the identical Avro schema embedded in both files.

This message was sent by Atlassian JIRA

View raw message