spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bjkonglu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-26135) Structured Streaming reporting metrics programmatically using asynchronous APIs can't get all queries metrics
Date Wed, 21 Nov 2018 06:08:00 GMT
bjkonglu created SPARK-26135:
--------------------------------

             Summary: Structured Streaming reporting metrics programmatically using asynchronous
APIs can't get all queries metrics
                 Key: SPARK-26135
                 URL: https://issues.apache.org/jira/browse/SPARK-26135
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.3.1
         Environment: h3.  

 
            Reporter: bjkonglu


h3. Background

 When I use Structured Streaming handle real-time data, I also want to know the streaming
application metrics, for example prcessedRowsPerSecond、inputRowsPerSeconds etc. So I report
metrics programmatically using asynchronous APIs.
{code:java}
val spark: SparkSession = ...

spark.streams.addListener(new StreamingQueryListener() {
    override def onQueryStarted(queryStarted: QueryStartedEvent): Unit = {
        println("Query started: " + queryStarted.id)
    }
    override def onQueryTerminated(queryTerminated: QueryTerminatedEvent): Unit = {
        println("Query terminated: " + queryTerminated.id)
    }
    override def onQueryProgress(queryProgress: QueryProgressEvent): Unit = {
        println("Query made progress: " + queryProgress.progress)
    }
})
{code}
h3. Questions

  When the streaming application has a single query, asynchronous APIs work well. But when
the streaming application has many queries, asynchronous APIs can't report metrics exactly,
some queries can report well, some queries report delay and metrics number lower. 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message