spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Platon Potapov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-6569) Kafka directInputStream logs what appear to be incorrect warnings
Date Thu, 02 Apr 2015 17:27:53 GMT

    [ https://issues.apache.org/jira/browse/SPARK-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392992#comment-14392992
] 

Platon Potapov commented on SPARK-6569:
---------------------------------------

If we talk about log level reduction only, then no, I don't feel strongly about the change.

But I would suggest not logging anything at all. Cody's rationale behind the warning is for
empty input batches to not go unnoticed. I feel that is a responsibility of a monitoring agent
external to the spark job - an agent knowing the specifics of the use case the job is being
run in, and hence having the capacity to decide whether an empty batch is an abnormal situation
(in my case, it isn't).

If that's not an option - well, please close the ticket then.


> Kafka directInputStream logs what appear to be incorrect warnings
> -----------------------------------------------------------------
>
>                 Key: SPARK-6569
>                 URL: https://issues.apache.org/jira/browse/SPARK-6569
>             Project: Spark
>          Issue Type: Bug
>          Components: Streaming
>    Affects Versions: 1.3.0
>         Environment: Spark 1.3.0
>            Reporter: Platon Potapov
>            Priority: Minor
>
> During what appears to be normal operation of streaming from a Kafka topic, the following
log records are observed, logged periodically:
> {code}
> [Stage 391:==========================================>              (3 + 0) / 4]
> 2015-03-27 12:49:54 WARN KafkaRDD: Beginning offset ${part.fromOffset} is the same as
ending offset skipping raw 0
> 2015-03-27 12:49:54 WARN KafkaRDD: Beginning offset ${part.fromOffset} is the same as
ending offset skipping raw 0
> 2015-03-27 12:49:54 WARN KafkaRDD: Beginning offset ${part.fromOffset} is the same as
ending offset skipping raw 0
> {code}
> * the part.fromOffset placeholder is not correctly substituted to a value
> * is the condition really mandates a warning being logged?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message