spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kousuke Saruta (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-19913) Log warning rather than throw AnalysisException when output is partitioned although format is memory, console or foreach
Date Sat, 11 Mar 2017 00:01:04 GMT
Kousuke Saruta created SPARK-19913:
--------------------------------------

             Summary: Log warning rather than throw AnalysisException when output is partitioned
although format is memory, console or foreach
                 Key: SPARK-19913
                 URL: https://issues.apache.org/jira/browse/SPARK-19913
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.2.0
            Reporter: Kousuke Saruta
            Priority: Minor


When batches are executed with memory, console or foreach format, `assertNotPartitioned` will
check whether output is not partitioned and throw AnalysisException in case it is.

But I wonder it's better to log warning rather than throw the exception because partitioning
does not affect output for those formats but also does not bring any negative impacts.

Also, this assertion is not applied when the format is `console`. I think in this case too,
we should assert that .

By fixing them, we can easily switch the format to memory or console for debug purposes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message