ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmytro Sen" <d...@hortonworks.com>
Subject Re: Review Request 22941: API call to /hosts to get information about 10 hosts takes more than 30 seconds on 2K-node cluster
Date Thu, 26 Jun 2014 11:35:45 GMT


> On June 24, 2014, 11:38 p.m., Tom Beerbower wrote:
> > I like the idea but have some issues with the patch.  Could we use this idea and
make it much simpler? Something like ...
> > 
> > QueryImpl...
> > 
> >   /**
> >    * Query the cluster controller for the top level resources.
> >    */
> >   private void queryForResources()
> >       throws UnsupportedPropertyException,
> >       SystemException,
> >       NoSuchResourceException,
> >       NoSuchParentResourceException {
> > 
> >     Set<Resource> providerResourceSet = new HashSet<Resource>();
> >     Resource.Type resourceType    = getResourceDefinition().getType();
> >     Predicate     queryPredicate  = createPredicate(getKeyValueMap(), processUserPredicate(userPredicate));
> > 
> >     // must occur after processing user predicate and prior to creating request
> >     finalizeProperties();
> > 
> >     Request       request     = createRequest();
> >     Set<Resource> resourceSet = new LinkedHashSet<Resource>();
> > 
> >     for (Resource queryResource : doQuery(resourceType, request, queryPredicate))
{
> >       providerResourceSet.add(queryResource);
> >     }
> > 
> >     // If there is a page request and the predicate doesn't contain any properties
that
> >     // haven't been set yet then go ahead and get just a single page of resources
...
> >     if (pageRequest != null &&
> >         !clusterController.predicateRequiresPopulation(resourceType, queryPredicate))
{
> >       PageResponse pageResponse = clusterController.getPage(resourceType,
> >           providerResourceSet, request, queryPredicate, pageRequest, sortRequest);
> > 
> >       // build a new set
> >       for (Resource r : pageResponse.getIterable()) {
> >         resourceSet.add(r);
> >       }
> >     } else {
> >       resourceSet.addAll(providerResourceSet);
> >     }
> > 
> >     queryResults.put(null, new QueryResult(
> >         request, queryPredicate, userPredicate, getKeyValueMap(), resourceSet));
> > 
> >     clusterController.populateResources(resourceType, resourceSet, request, queryPredicate);
> >     queryForSubResources();
> >   }
> > 
> > 
> > ClusterController ...
> > 
> >   /**
> >    * Check to see if the given predicate contains any properties to be populated
> >    * by the providers for the given type.
> >    *
> >    * @param type       the resource type
> >    * @param predicate  the predicate
> >    *
> >    * @return true if the given predicate contains any properties to be populated
> >    *         by the providers for the given type
> >    */
> >   public boolean predicateRequiresPopulation(Resource.Type type, Predicate predicate);
> > 
> > 
> > 
> > ClusterControllerImpl ...
> > 
> >   @Override
> >   public boolean predicateRequiresPopulation(Type type, Predicate predicate) {
> >     List<PropertyProvider> propertyProviders = ensurePropertyProviders(type);
> >     Request request = PropertyHelper.getReadRequest();
> > 
> >     for (PropertyProvider propertyProvider : propertyProviders) {
> >       if (providesRequestProperties(propertyProvider, request, predicate)) {
> >         return true;
> >       }
> >     }
> >     return false;
> >   }
> 
> Dmytro Sen wrote:
>     It won't work . Before providing response we call getPage() one more time, it have
to add to response correct "itemTotal" : "x", how many resources were populated, should correct
be number of resources populated.

works. will upload patch in 15 minutes


- Dmytro


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22941/#review46593
-----------------------------------------------------------


On June 24, 2014, 9:35 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22941/
> -----------------------------------------------------------
> 
> (Updated June 24, 2014, 9:35 p.m.)
> 
> 
> Review request for Ambari, Sid Wagle and Tom Beerbower.
> 
> 
> Bugs: AMBARI-6263
>     https://issues.apache.org/jira/browse/AMBARI-6263
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> On a large cluster, calls to /hosts are taking more than 30 seconds.
> This severely impacts the responsiveness of the UI for the Hosts page (it takes >
30 seconds to filter, navigate through paged hosts, etc).
> The times shown are latency only (does not include download time).
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java 63426cb

>   ambari-server/src/main/java/org/apache/ambari/server/api/services/BaseRequest.java
b7893c5 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/ganglia/GangliaPropertyProvider.java
4d623c1 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/ganglia/GangliaReportPropertyProvider.java
35b156c 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractProviderModule.java
711e64e 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterControllerImpl.java
3f8f317 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ResourceImpl.java
15fb961 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackDefinedPropertyProvider.java
51c7565 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/jmx/JMXPropertyProvider.java
a4dc2b2 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/nagios/NagiosPropertyProvider.java
ce1dc2f 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/spi/ClusterController.java
5e3f844 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java 13e7c77

>   ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java df97944

>   ambari-server/src/main/java/org/apache/ambari/server/utils/JaxbMapKeyListAdapter.java
4589948 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/JaxbMapKeyMapAdapter.java
8419d73 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/JaxbMapKeyValAdapter.java
e3531cd 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/ganglia/GangliaPropertyProviderTest.java
4dc95a3 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/ganglia/GangliaReportPropertyProviderTest.java
1ae17b3 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterControllerImplTest.java
1d0eef4 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackDefinedPropertyProviderTest.java
2a086ae 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/jmx/JMXPropertyProviderTest.java
ac7314e 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/nagios/NagiosPropertyProviderTest.java
92d3c40 
> 
> Diff: https://reviews.apache.org/r/22941/diff/
> 
> 
> Testing
> -------
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Ambari Main ....................................... SUCCESS [1.632s]
> [INFO] Apache Ambari Project POM ......................... SUCCESS [0.303s]
> [INFO] Ambari Web ........................................ SUCCESS [22.921s]
> [INFO] Ambari Views ...................................... SUCCESS [1.546s]
> [INFO] Ambari Server ..................................... SUCCESS [14:21.956s]
> [INFO] Ambari Agent ...................................... SUCCESS [6.150s]
> [INFO] Ambari Client ..................................... SUCCESS [0.035s]
> [INFO] Ambari Python Client .............................. SUCCESS [0.291s]
> [INFO] Ambari Groovy Client .............................. SUCCESS [8.195s]
> [INFO] Ambari Shell ...................................... SUCCESS [4.434s]
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message