hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2938) Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
Date Fri, 12 Dec 2014 07:30:14 GMT

    [ https://issues.apache.org/jira/browse/YARN-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243824#comment-14243824
] 

Varun Saxena commented on YARN-2938:
------------------------------------

bq.  Is the newly adde entries in {{findbugs-exclude.xml}} fixable? Would you mind sharing
the detail findbugs reports about these issues?

Sure [~zjshen]. Kindly find the details as under :
-  *VO_VOLATILE_INCREMENT* in {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode}}
{{numContainers}} in SchedulerNode is volatile. And this variable was being incremented and
decremented which Findbugs complained about. But increment and decrement happens inside synchronized
methods so should not be an issue.
{code}
 public synchronized void allocateContainer(RMContainer rmContainer) {
    Container container = rmContainer.getContainer();
    deductAvailableResource(container.getResource());
    ++numContainers;
   ....
}
...
 private synchronized void updateResource(Container container) {
    addAvailableResource(container.getResource());
    --numContainers;
  }
{code}
-  *VO_VOLATILE_INCREMENT* in *org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue*
Exactly same as issue above. {{numContainers}} incremented and decremented only inside synchronized
methods.
{code}
  synchronized void allocateResource(Resource clusterResource, 
      Resource resource, Set<String> nodeLabels) {
    Resources.addTo(usedResources, resource);
    ......
    ++numContainers;
    CSQueueUtils.updateQueueStatistics(resourceCalculator, this, getParent(),
        clusterResource, minimumAllocation);
  }
  ...
 protected synchronized void releaseResource(Resource clusterResource,
      Resource resource, Set<String> nodeLabels) {
    // Update queue metrics
    Resources.subtractFrom(usedResources, resource);
    ....
    CSQueueUtils.updateQueueStatistics(resourceCalculator, this, getParent(),
        clusterResource, minimumAllocation);
    --numContainers;
  }
{code}
-  *VO_VOLATILE_INCREMENT* in *org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue*
Exactly same as issue above. {{numApplications}} incremented and decremented only inside synchronized
methods.
{code}
private synchronized void addApplication(ApplicationId applicationId,
      String user) {

    ++numApplications;

    LOG.info("Application added -" +
        " appId: " + applicationId + 
        " user: " + user + 
        " leaf-queue of parent: " + getQueueName() + 
        " #applications: " + getNumApplications());
  }
  ......
  private synchronized void removeApplication(ApplicationId applicationId, 
      String user) {
    
    --numApplications;

    LOG.info("Application removed -" +
        " appId: " + applicationId + 
        " user: " + user + 
        " leaf-queue of parent: " + getQueueName() + 
        " #applications: " + getNumApplications());
  }
{code}



- For above 3, refer to [False Positive VO_VOLATILE_INCREMENT when synchronized|http://sourceforge.net/p/findbugs/bugs/1032/]

- *RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE* in {{org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler}}.

Findbugs is complaining about null check for {{containerReports}} in below piece of code.
{code}
 public synchronized void recoverContainersOnNode(
      List<NMContainerStatus> containerReports, RMNode nm) {
    if (!rmContext.isWorkPreservingRecoveryEnabled()
        || containerReports == null
        || (containerReports != null && containerReports.isEmpty())) {
      return;
    }
    ....
}
{code}
Looking closely at the code flow, function above is called on *NODE_ADDED* event in {{CapacityScheduler}},
{{FifoScheduler}} and {{FairScheduler}}
{code:title=CapacityScheduler.java|borderStyle=solid}
 public void handle(SchedulerEvent event) {
    switch(event.getType()) {
    case NODE_ADDED:
    {
      NodeAddedSchedulerEvent nodeAddedEvent = (NodeAddedSchedulerEvent)event;
      addNode(nodeAddedEvent.getAddedRMNode());
      recoverContainersOnNode(nodeAddedEvent.getContainerReports(),
        nodeAddedEvent.getAddedRMNode());
    }
   ...
}
{code}
And this NodeAddedSchedulerEvent is created in below piece of code. In the code below, {{containers}}
will be written to {{NodeAddedSchedulerEvent#containerReports}}. As can be seen below, containers
can actually be null in some cases.
{code:title=RMNodeImpl.java|borderStyle=solid}
public static class AddNodeTransition implements
      SingleArcTransition<RMNodeImpl, RMNodeEvent> {

    @Override
    public void transition(RMNodeImpl rmNode, RMNodeEvent event) {
      // Inform the scheduler
      RMNodeStartedEvent startEvent = (RMNodeStartedEvent) event;
      List<NMContainerStatus> containers = null;

      String host = rmNode.nodeId.getHost();
      if (rmNode.context.getInactiveRMNodes().containsKey(host)) {
        ....
      } else {
        ClusterMetrics.getMetrics().incrNumActiveNodes();
        containers = startEvent.getNMContainerStatuses();
        if (containers != null && !containers.isEmpty()) {
          for (NMContainerStatus container : containers) {
            if (container.getContainerState() == ContainerState.RUNNING) {
              rmNode.launchedContainers.add(container.getContainerId());
            }
          }
        }
      }
      ....
      rmNode.context.getDispatcher().getEventHandler()
        .handle(new NodeAddedSchedulerEvent(rmNode, containers));
     ....
    }
  }
{code}


> Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
> --------------------------------------------------------------------------------------------------
>
>                 Key: YARN-2938
>                 URL: https://issues.apache.org/jira/browse/YARN-2938
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>             Fix For: 2.7.0
>
>         Attachments: YARN-2938.001.patch, YARN-2938.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message