ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmytro Sen <d...@hortonworks.com>
Subject Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Date Fri, 26 Feb 2016 11:21:01 GMT


> On Фев. 26, 2016, 8:03 д.п., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java,
line 74
> > <https://reviews.apache.org/r/43955/diff/3-4/?file=1270988#file1270988line74>
> >
> >     What extra locking if using ConcurrentMap, is this necessary?
> 
> Dmytro Sen wrote:
>     TimelineMetricHolder extends ConcurrentSkipListMap<String, TimelineMetricWrapper>,
so it's thread-safe only for read/assign TimelineMetricWrapper by metric name, but not for
metric values wrapped by TimelineMetricWrapper. Multiple threads are still allowed to get
reference to the same TimelineMetricWrapper and simultaneously call TimelineMetricWrapper.putMetric()
or (TimelineMetricWrapper.putMetric() + TimelineMetricWrapper.getTimelineMetric())

TimelineMetricHolder is thread-safe by ConcurrentSkipListMap and TimelineMetricWrapper is
thread-safe by using synchronized methods


- Dmytro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120846
-----------------------------------------------------------


On Фев. 26, 2016, 6:55 д.п., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
> 
> (Updated Фев. 26, 2016, 6:55 д.п.)
> 
> 
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
> 
> 
> Bugs: AMBARI-15100
>     https://issues.apache.org/jira/browse/AMBARI-15100
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Reported on ambari user group:
> 
> 2016-02-16 18:04:24,510 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373))
- java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373))
- java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN  impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373))
- java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51))
- Thread Thread[timeline,5,main] threw an Error.  Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.HashMap.resize(HashMap.java:703)
>        at java.util.HashMap.putVal(HashMap.java:662)
>        at java.util.HashMap.put(HashMap.java:611)
>        at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
>        at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
>        at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
>        at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
>        at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
>        at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
>        at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
>        at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
>        at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO  util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status
-1 Message: HaltException
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java
15bd5f4 
>   ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java
ad98525 
> 
> Diff: https://reviews.apache.org/r/43955/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> mvn clean test
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message