hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Phillips (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-11751) Parquet scan fails when directory contains _SUCCESS or _logs
Date Thu, 26 Mar 2015 01:31:52 GMT
Steven Phillips created HADOOP-11751:

             Summary: Parquet scan fails when directory contains _SUCCESS or _logs
                 Key: HADOOP-11751
                 URL: https://issues.apache.org/jira/browse/HADOOP-11751
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Steven Phillips
            Assignee: Steven Phillips

_SUCCESS and _logs are often created from map reduce jobs, and typically ignored. This is
commonly done using the OutputFilesFilter in hadoop.

The new FooterGatherer class, which is used to read parquet footers in parallel, does not
use this filter. So it attempts to read these as parquet files and fails.

The fix is to use the DrillPathFilter, which extends OutputFilesFilter, in the FooterGather

This message was sent by Atlassian JIRA

View raw message