ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-20819) LogSearch Integration should limit requests to portal for missing components
Date Sat, 22 Apr 2017 03:39:04 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-20819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979735#comment-15979735
] 

Hadoop QA commented on AMBARI-20819:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12864550/AMBARI-20819.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in ambari-server:

                  org.apache.ambari.server.state.UpgradeHelperTest
                  org.apache.ambari.server.checks.DatabaseConsistencyCheckHelperTest

Console output: https://builds.apache.org/job/Ambari-trunk-test-patch/11454//console

This message is automatically generated.

> LogSearch Integration should limit requests to portal for missing components
> ----------------------------------------------------------------------------
>
>                 Key: AMBARI-20819
>                 URL: https://issues.apache.org/jira/browse/AMBARI-20819
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 2.5.0
>            Reporter: Robert Nettleton
>            Assignee: Robert Nettleton
>            Priority: Critical
>             Fix For: 2.5.1
>
>         Attachments: AMBARI-20819.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> The LogSearch Integration layer in Ambari typically caches logging metadata for each
host/component combination in a given cluster. This cache is configurable, and is typically
a long-lived cache, since the logging metadata is generally not likely to change.
> This issue was found during a test of LogSearch at a large scale, with a cluster of over
800 nodes. 
> it was discovered that the LogSearch Integration layer continues to make repeated requests
for logging metadata for components that are not properly configured in LogSearch.
> The LogSearch portal logs show the following repeated requests on the "Perf" cluster:
> {code}
> {"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-280","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210778","log_message":"Solr
query will be processed: q=*:*&start=0&rows=1&fq=host:perf\\-d\\-86.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
> {"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-273","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210797","log_message":"Solr
query will be processed: q=*:*&start=0&rows=1&fq=host:perf\\-k\\-24.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
> {"level":"INFO","file":"SolrDaoBase.java","thread_name":"qtp1018937824-275","line_number":61,"logger_name":"org.apache.ambari.logsearch.dao.SolrDaoBase","logtime":"1487951210810","log_message":"Solr
query will be processed: q=*:*&start=0&rows=1&fq=host:perf\\-d\\-88.c.pramod\\-thangali.internal&fq=type:hst_agent&fq=logtime:[*+TO+*]&sort=logtime+desc,seq_num+desc"}
> {code}
> The SmartSense component, managed outside of Ambari, includes logging component IDs the
SmartSense stack definition, but does not include the proper metadata to allow LogSearch Portal
to properly handle the SmartSense logs.  
> In the current implementation of the integration layer, since the SmartSense components
are never found, the integration will repeatedly check the cache, fail that check, and attempt
a remote call to obtain the metadata.  This in turn can cause LogSearch to execute repeated,
and un-necessary, queries against the Solr database, which could potentially be a problem
in larger deployments.  
> The LogSearch Integration in Ambari should be modified to have a maximum number of failed
attempts to tolerate.  Having passed a maximum number of failed attempts to obtain the logging
metadata for a given component, the integration code should refrain from further attempts
to contact the LogSearch portal for this component, and log a warning to indicate that the
requested component is not properly handled by LogSearch.  
> This will probably need to be fixed in:
> {code}org.apache.ambari.server.controller.logging.LogSearchDataRetrievalService{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message