spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-17858) Provide option for Spark SQL to skip corrupt files
Date Mon, 10 Oct 2016 22:52:20 GMT

     [ https://issues.apache.org/jira/browse/SPARK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shixiong Zhu updated SPARK-17858:
---------------------------------
    Description: 
In Spark 2.0, corrupt files will fail a SQL query. However, the user may just want to skip
corrupt files and still run the query.

Another painful thing is the current exception doesn't contain the paths of corrupt files,
makes the user hard to fix their files.

Note: In Spark 1.6, Spark SQL always skip corrupt files because of SPARK-17850.

  was:
In Spark 2.0, corrupt files will fail a job. However, the user may just want to skip corrupt
files and still run the query.

Another painful thing is the current exception doesn't contain the paths of corrupt files,
makes the user hard to fix their files.


> Provide option for Spark SQL to skip corrupt files
> --------------------------------------------------
>
>                 Key: SPARK-17858
>                 URL: https://issues.apache.org/jira/browse/SPARK-17858
>             Project: Spark
>          Issue Type: Improvement
>            Reporter: Shixiong Zhu
>
> In Spark 2.0, corrupt files will fail a SQL query. However, the user may just want to
skip corrupt files and still run the query.
> Another painful thing is the current exception doesn't contain the paths of corrupt files,
makes the user hard to fix their files.
> Note: In Spark 1.6, Spark SQL always skip corrupt files because of SPARK-17850.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message