spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <>
Subject [jira] [Commented] (SPARK-20703) Add an operator for writing data out
Date Thu, 13 Jul 2017 11:32:00 GMT


Steve Loughran commented on SPARK-20703:

Regarding a patch for this, what do people suggest as a good policy for failures

FNFE: log@debug, return 0
IOE: log@info, full stack @debug, return 0

or treat FNFE the same as any other IOE (which is most likely to be a transient network failure,
given by this point we know that the user has connectivity, auth and write access to the bucket)?

# I would like the option of printing that full stack, so that if the time comes to debug
it, saying "edit this log setting" is enough to make problems visible in production.
# Yet I don't want to log the full stack as a default as that can lead to panic, and it also
tends to mean when there are real problems in the code people may just paste that stack in,
rather than one for the real problem.

> Add an operator for writing data out
> ------------------------------------
>                 Key: SPARK-20703
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Reynold Xin
>            Assignee: Liang-Chi Hsieh
>             Fix For: 2.3.0
> We should add an operator for writing data out. Right now in the explain plan / UI there
is no way to tell whether a query is writing data out, and also there is no way to associate
metrics with data writes. It'd be tremendously valuable to do this for adding metrics and
for visibility.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message