spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng Lian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-8118) Turn off noisy log output produced by Parquet 1.7.0
Date Tue, 31 May 2016 16:48:13 GMT

    [ https://issues.apache.org/jira/browse/SPARK-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308083#comment-15308083
] 

Cheng Lian commented on SPARK-8118:
-----------------------------------

Yea, unfortunately at last we found that due to a few Parquet side bugs, it is pretty hard
to override Parquet logging in user applications. See PARQUET-401. This will probably be fixed
in a clean way once we upgrade to parquet-mr 1.9 (which hasn't been released yet).

> Turn off noisy log output produced by Parquet 1.7.0
> ---------------------------------------------------
>
>                 Key: SPARK-8118
>                 URL: https://issues.apache.org/jira/browse/SPARK-8118
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 1.4.1, 1.5.0
>            Reporter: Cheng Lian
>            Assignee: Cheng Lian
>            Priority: Minor
>             Fix For: 1.5.0
>
>
> Parquet 1.7.0 renames package name to "org.apache.parquet", need to adjust {{ParquetRelation.enableLogForwarding}}
accordingly to avoid noisy log output.
> A better approach than simply muting these log lines is to redirect Parquet logs via
SLF4J, so that we can handle them consistently. In general these logs are very useful. Esp.
when used to diagnosing Parquet memory issue and filter push-down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message