spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles
Date Wed, 23 May 2018 05:12:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon updated SPARK-22366:
---------------------------------
    Description: 
There's an existing flag "spark.sql.files.ignoreCorruptFiles" that will quietly ignore attempted
reads from files that have been corrupted, but it still allows the query to fail on missing
files. Being able to ignore missing files too is useful in some replication scenarios.

We should add a "spark.sql.files.ignoreMissingFiles" to fill out the functionality.

  was:
+underlined text+There's an existing flag "spark.sql.files.ignoreCorruptFiles" that will quietly
ignore attempted reads from files that have been corrupted, but it still allows the query
to fail on missing files. Being able to ignore missing files too is useful in some replication
scenarios.

We should add a "spark.sql.files.ignoreMissingFiles" to fill out the functionality.


> Support ignoreMissingFiles flag parallel to ignoreCorruptFiles
> --------------------------------------------------------------
>
>                 Key: SPARK-22366
>                 URL: https://issues.apache.org/jira/browse/SPARK-22366
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Jose Torres
>            Assignee: Jose Torres
>            Priority: Minor
>             Fix For: 2.3.0
>
>
> There's an existing flag "spark.sql.files.ignoreCorruptFiles" that will quietly ignore
attempted reads from files that have been corrupted, but it still allows the query to fail
on missing files. Being able to ignore missing files too is useful in some replication scenarios.
> We should add a "spark.sql.files.ignoreMissingFiles" to fill out the functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message