hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6508) Incorrect values for metrics with CompositeContext
Date Wed, 27 Jan 2010 05:32:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805355#action_12805355
] 

Amareshwari Sriramadasu commented on HADOOP-6508:
-------------------------------------------------

Analyzing the following JobTrackerMetricsInst code  with CompositeContext as the MetricsContext:
{code}
    MetricsContext context = MetricsUtil.getContext("mapred");
    metricsRecord = MetricsUtil.createRecord(context, "jobtracker");
    metricsRecord.setTag("sessionId", sessionId);
    context.registerUpdater(this);
{code}

Details on each line of code:
{code}
    MetricsContext context = MetricsUtil.getContext("mapred");
{code}
This code creates a CompositeContext(CC), which creates all its sub-contexts and calls startsMonitoring
on all the
subcontext. Thus there are as many threads(monitoring) as the number of sub-contexts. Here,
each thread calls
doUpdates() followed by emitRecords() in the configured periods.

{code}
    metricsRecord = MetricsUtil.createRecord(context, "jobtracker");
{code}
This code creates a MetricsRecord for CompositeContext, which is a Proxy which has a delegator
for all the sub-records. This record invokes
every method call on all its sub-records.

{code}
    context.registerUpdater(this);
{code}
This code registers JobTracker as the updater for all the sub-contexts.

Putting above relation pictorially (see the attached png): JobTracker (JT) has CompositeContext(CC)
and ProxyMetricsRecord (PR). CC starts
Context1(C1) and Context2(C2) 's timer threads. C1 has MetricsRecord (R1) and C2 has MetricsRecord(R2).
PR delagates
all method calls on it to R1 and R2. C1and C2 register JobTracker as the updater.

Both C1 and C2 call JT.doUpdates at specified periods. The code flow for doUpdates from C1
or C2:
  1. Set/Incr methods on ProxyRecord are delegated to both R1 and R2. On the JobTracker code
these calls are
synchronized.
  2. MetricsRecord.update() boils down to R1.update() and R2.update() irrespective of whether
it is from C1 or C2. This
call is not synchronized on JT.

Moreover, MetricsRecord javadoc clearly says: 
<em>Different threads should  *not* use the same MetricsRecord instance at the same
time. </em>.  So, the problem here is that the ProxyRecord is shared between two threads
without synchronization. There is possibility for race if the above updates are not synchronized.
I think this could be the most likely cause for seeing incorrect values with CompositeContext.


> Incorrect values for metrics with CompositeContext
> --------------------------------------------------
>
>                 Key: HADOOP-6508
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6508
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.0
>            Reporter: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: CompositeContext.png
>
>
> In our clusters, when we use CompositeContext with two contexts, second context gets
wrong values.
> This problem is consistent on 500 (and above) node cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message