spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang-Chi Hsieh (JIRA)" <>
Subject [jira] [Created] (SPARK-19082) The config ignoreCorruptFiles doesn't work for Parquet
Date Thu, 05 Jan 2017 04:47:58 GMT
Liang-Chi Hsieh created SPARK-19082:

             Summary: The config ignoreCorruptFiles doesn't work for Parquet
                 Key: SPARK-19082
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Liang-Chi Hsieh

We have a config {{spark.sql.files.ignoreCorruptFiles}} which can be used to ignore corrupt
files when reading files in SQL. Currently the {{ignoreCorruptFiles}} config has two issues
and can't work for Parquet:

1. We only ignore corrupt files in {{FileScanRDD}} . Actually, we begin to read those files
as early as inferring data schema from the files. For corrupt files, we can't read the schema
and fail the program. A related issue reported at

2. In {{FileScanRDD}}, we assume that we only begin to read the files when starting to consume
the iterator. However, it is possibly the files are read before that. In this case, {{ignoreCorruptFiles}}
config doesn't work too.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message