ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Beerbower" <tbeerbo...@hortonworks.com>
Subject Re: Review Request 22941: API call to /hosts to get information about 10 hosts takes more than 30 seconds on 2K-node cluster
Date Tue, 24 Jun 2014 23:38:49 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22941/#review46593
-----------------------------------------------------------


I like the idea but have some issues with the patch.  Could we use this idea and make it much
simpler? Something like ...

QueryImpl...

  /**
   * Query the cluster controller for the top level resources.
   */
  private void queryForResources()
      throws UnsupportedPropertyException,
      SystemException,
      NoSuchResourceException,
      NoSuchParentResourceException {

    Set<Resource> providerResourceSet = new HashSet<Resource>();
    Resource.Type resourceType    = getResourceDefinition().getType();
    Predicate     queryPredicate  = createPredicate(getKeyValueMap(), processUserPredicate(userPredicate));

    // must occur after processing user predicate and prior to creating request
    finalizeProperties();

    Request       request     = createRequest();
    Set<Resource> resourceSet = new LinkedHashSet<Resource>();

    for (Resource queryResource : doQuery(resourceType, request, queryPredicate)) {
      providerResourceSet.add(queryResource);
    }

    // If there is a page request and the predicate doesn't contain any properties that
    // haven't been set yet then go ahead and get just a single page of resources ...
    if (pageRequest != null &&
        !clusterController.predicateRequiresPopulation(resourceType, queryPredicate)) {
      PageResponse pageResponse = clusterController.getPage(resourceType,
          providerResourceSet, request, queryPredicate, pageRequest, sortRequest);

      // build a new set
      for (Resource r : pageResponse.getIterable()) {
        resourceSet.add(r);
      }
    } else {
      resourceSet.addAll(providerResourceSet);
    }

    queryResults.put(null, new QueryResult(
        request, queryPredicate, userPredicate, getKeyValueMap(), resourceSet));

    clusterController.populateResources(resourceType, resourceSet, request, queryPredicate);
    queryForSubResources();
  }


ClusterController ...

  /**
   * Check to see if the given predicate contains any properties to be populated
   * by the providers for the given type.
   *
   * @param type       the resource type
   * @param predicate  the predicate
   *
   * @return true if the given predicate contains any properties to be populated
   *         by the providers for the given type
   */
  public boolean predicateRequiresPopulation(Resource.Type type, Predicate predicate);



ClusterControllerImpl ...

  @Override
  public boolean predicateRequiresPopulation(Type type, Predicate predicate) {
    List<PropertyProvider> propertyProviders = ensurePropertyProviders(type);
    Request request = PropertyHelper.getReadRequest();

    for (PropertyProvider propertyProvider : propertyProviders) {
      if (providesRequestProperties(propertyProvider, request, predicate)) {
        return true;
      }
    }
    return false;
  }

- Tom Beerbower


On June 24, 2014, 9:35 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22941/
> -----------------------------------------------------------
> 
> (Updated June 24, 2014, 9:35 p.m.)
> 
> 
> Review request for Ambari, Sid Wagle and Tom Beerbower.
> 
> 
> Bugs: AMBARI-6263
>     https://issues.apache.org/jira/browse/AMBARI-6263
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> On a large cluster, calls to /hosts are taking more than 30 seconds.
> This severely impacts the responsiveness of the UI for the Hosts page (it takes >
30 seconds to filter, navigate through paged hosts, etc).
> The times shown are latency only (does not include download time).
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java 63426cb

>   ambari-server/src/main/java/org/apache/ambari/server/api/services/BaseRequest.java
b7893c5 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/ganglia/GangliaPropertyProvider.java
4d623c1 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/ganglia/GangliaReportPropertyProvider.java
35b156c 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractProviderModule.java
711e64e 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterControllerImpl.java
3f8f317 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ResourceImpl.java
15fb961 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackDefinedPropertyProvider.java
51c7565 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/jmx/JMXPropertyProvider.java
a4dc2b2 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/nagios/NagiosPropertyProvider.java
ce1dc2f 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/spi/ClusterController.java
5e3f844 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java 13e7c77

>   ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java df97944

>   ambari-server/src/main/java/org/apache/ambari/server/utils/JaxbMapKeyListAdapter.java
4589948 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/JaxbMapKeyMapAdapter.java
8419d73 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/JaxbMapKeyValAdapter.java
e3531cd 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/ganglia/GangliaPropertyProviderTest.java
4dc95a3 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/ganglia/GangliaReportPropertyProviderTest.java
1ae17b3 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterControllerImplTest.java
1d0eef4 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDefinedPropertyProviderTest.java
2a086ae 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/jmx/JMXPropertyProviderTest.java
ac7314e 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/nagios/NagiosPropertyProviderTest.java
92d3c40 
> 
> Diff: https://reviews.apache.org/r/22941/diff/
> 
> 
> Testing
> -------
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ....................................... SUCCESS [1.632s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.303s]
> [INFO] Ambari Web ........................................ SUCCESS [22.921s]
> [INFO] Ambari Views ...................................... SUCCESS [1.546s]
> [INFO] Ambari Server ..................................... SUCCESS [14:21.956s]
> [INFO] Ambari Agent ...................................... SUCCESS [6.150s]
> [INFO] Ambari Client ..................................... SUCCESS [0.035s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.291s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [8.195s]
> [INFO] Ambari Shell ...................................... SUCCESS [4.434s]
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message