avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clément MATHIEU (JIRA) <j...@apache.org>
Subject [jira] [Created] (AVRO-1787) Add support of directories & globs to concat
Date Thu, 21 Jan 2016 20:44:39 GMT
Clément MATHIEU created AVRO-1787:

             Summary: Add support of directories & globs to concat
                 Key: AVRO-1787
                 URL: https://issues.apache.org/jira/browse/AVRO-1787
             Project: Avro
          Issue Type: Bug
          Components: java
    Affects Versions: 1.7.7
            Reporter: Clément MATHIEU

While answering [this question|http://stackoverflow.com/questions/34856838/concat-avro-files-using-avro-tools/34899425#34899425]
on stack overflow, I noticed that the concat tool does not support directories or glob patterns
as "input-file".

 OP's use case seems very valid but accepting only files as input push the burden on users.

I think that it makes sense to also accept directories as input like cat does. Supporting
glob patterns seems also useful if one want to concatenate the output of a MR job (in this
case passing the directory won't work because of the {{_SUCESS}} file).

I have quickly patched {{concat}} to support files, directories and glob patterns. Support
of glob patterns has been added to {{cat}} since it seems useful too.

This change should not introduce regression for users. But any feedback about glob patterns
is welcome.

This message was sent by Atlassian JIRA

View raw message