spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HeartSaVioR <...@git.apache.org>
Subject [GitHub] spark pull request #21622: [SPARK-24637][SS] Add metrics regarding state and...
Date Tue, 26 Jun 2018 21:17:20 GMT
Github user HeartSaVioR commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21622#discussion_r198300792
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetricsReporter.scala
---
    @@ -39,6 +42,23 @@ class MetricsReporter(
       registerGauge("processingRate-total", _.processedRowsPerSecond, 0.0)
       registerGauge("latency", _.durationMs.get("triggerExecution").longValue(), 0L)
     
    +  private val timestampFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
// ISO8601
    +  timestampFormat.setTimeZone(DateTimeUtils.getTimeZone("UTC"))
    +
    +  registerGauge("eventTime-watermark",
    +    s => convertStringDateToMillis(s.eventTime.get("watermark")), 0L)
    --- End diff --
    
    1. will address
    2. We don't know whether the map will be empty when calling `registerGauge`, and once
we register the metric, `getValue` in Gauge is called from Dropwizard so I'm not sure we can
control whether reporting the value or not.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message