ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cabir Zounaidou" <czounai...@hortonworks.com>
Subject Re: Review Request 27802: Rest api's for host component logs.
Date Mon, 10 Nov 2014 19:05:44 GMT


> On Nov. 10, 2014, 3:48 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostComponentLogResourceProvider.java,
line 347
> > <https://reviews.apache.org/r/27802/diff/2/?file=756639#file756639line347>
> >
> >     If I understand this code correctly, it retrieves the log file in chunks of
length with an offset. However, if the LEVEL doesn't match and log entry in this chunk, no
data is returned.
> >     
> >     Is this the behavior we want; offset=10000,length=5000,level=WARNING might return
no data, but 15000/5000 suddenly does.

Yes, you are correct, filtering will be done on the content retrieved using offset/length
and we only support maximum length of 5K.  This implementation is not optimal and it implemented
this way because, i don't see WebHDFS supporting search feature.  If this has to a proper
search, then we have to download the complete file to do searching and sometimes this may
consume huge resource if the HDFS file is huge (say 2GB or more).


- Cabir


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27802/#review60602
-----------------------------------------------------------


On Nov. 10, 2014, 9:04 a.m., Cabir Zounaidou wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27802/
> -----------------------------------------------------------
> 
> (Updated Nov. 10, 2014, 9:04 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev
Konar, Nate Cole, Sid Wagle, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-4083
>     https://issues.apache.org/jira/browse/AMBARI-4083
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The host component log implements the following two rest api's.
> 
> 1. To retrieve the log entries. 
>   /clusters/{cluster}/hosts/{hostname}/host_components/{component}/logs
>   This api will retrieve the logs if available for the host component from HDFS.  It
can fetch only maximum of 5120 bytes.  The window can be adjusted using the query parameters
'offset' and 'length'.  It also provides simple filtering using 'level' query parameter.
>   Sample response will look like below:
>   {
>   "href" : "http://c6501.ambari.apache.org:8080/api/v1/clusters/cl1/hosts/c6503.ambari.apache.org/host_components/HBASE_CLIENT/logs",
>   "items" : [
>     {
>       "href" : "http://c6501.ambari.apache.org:8080/api/v1/clusters/cl1/hosts/c6503.ambari.apache.org/host_components/HBASE_CLIENT/logs/HBASE_CLIENT",
>       "length" : 5120,
>       "level" : null,
>       "offset" : 153920,
>       "size" : 159040,
>       "HostComponentLog" : {
>         "cluster_name" : "cl1",
>         "component_name" : "HBASE_CLIENT",
>         "entries" : [
>           {
>             "timestamp" : "2014-11-01 17:46:38,456",
>             "level" : "DEBUG",
>             "thread" : "main-EventThread",
>             "message" : "master.SplitLogManager: task not yet acquired /hbase-unsecure/splitWAL/WALs%2Fc6503.ambari.apache.org%2C60020%2C1414856039721-splitting%2Fc6503.ambari.apache.org%252C60020%252C1414856039721.1414856049592.meta
ver = 0"
>           },
>           {
>             "timestamp" : "2014-11-01 17:46:38,459",
>             "level" : "DEBUG",
>             "thread" : "main-EventThread",
>             "message" : "master.SplitLogManager: put up splitlog task at znode /hbase-unsecure/splitWAL/WALs%2Fc6503.ambari.apache.org%2C60020%2C1414856039721-splitting%2Fc6503.ambari.apache.org%252C60020%252C1414856039721.1414856050193.meta"
>           },
>           {
>             "timestamp" : "2014-11-01 17:46:38,460",
>             "level" : "DEBUG",
>             "thread" : "main-EventThread",
>             "message" : "master.SplitLogManager: put up splitlog task at znode /hbase-unsecure/splitWAL/WALs%2Fc6503.ambari.apache.org%2C60020%2C1414856039721-splitting%2Fc6503.ambari.apache.org%252C60020%252C1414856039721.1414856105560.meta"
>           },
>           :
>           :
>         }]
>       }
>     ]
>    }
> 2. To download the log file from HDFS use the following the api
>   /clusters/{cluster}/hosts/{hostname}/host_components/{component}/logs?download=true
>   The response will automatically download the file with the HDFS file name.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/AgentCommand.java e2f013d

>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java e99e39f

>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java 56b4f18

>   ambari-server/src/main/java/org/apache/ambari/server/agent/LogConfigCommand.java PRE-CREATION

>   ambari-server/src/main/java/org/apache/ambari/server/agent/RegistrationResponse.java
8a24560 
>   ambari-server/src/main/java/org/apache/ambari/server/api/resources/HostComponentLogResourceDefinition.java
PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/api/resources/HostComponentResourceDefinition.java
6dc9e2d 
>   ambari-server/src/main/java/org/apache/ambari/server/api/resources/ResourceInstanceFactoryImpl.java
9ad37ec 
>   ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
bb4c569 
>   ambari-server/src/main/java/org/apache/ambari/server/api/services/HostComponentService.java
4990ad7 
>   ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
4f53544 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/HostComponentLogResponse.java
PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
ae20f56 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostComponentLogResourceProvider.java
PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/spi/Resource.java dbac906

>   ambari-server/src/main/java/org/apache/ambari/server/state/LogConfig.java PRE-CREATION

>   ambari-server/src/main/java/org/apache/ambari/server/state/LogConfigHandler.java PRE-CREATION

>   ambari-server/src/main/java/org/apache/ambari/server/state/LogDefinition.java PRE-CREATION

>   ambari-server/src/main/java/org/apache/ambari/server/state/LogEntry.java PRE-CREATION

>   ambari-server/src/main/resources/key_properties.json c1a6636 
>   ambari-server/src/main/resources/log_handler_config.json PRE-CREATION 
>   ambari-server/src/main/resources/properties.json 36cff96 
>   ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
e7b946d 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostComponentLogResourceProviderTest.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/27802/diff/
> 
> 
> Testing
> -------
> 
> 1. Ran the unit tests.
> 2. Manually ran the retrieve host component logs without any query parameters.
> 3. Manually ran the retrieve host component logs with offset and length.
> 4. Manually ran the retrieve host component logs with level filter.
> 5. Manually ran the retrieve host component logs with webHDFS disabled.
> 6. Manually ran the download host component log file with webHDFS enabled.
> 7. Manually ran the download host component log file with webHDFS disabled.
> 
> 
> Thanks,
> 
> Cabir Zounaidou
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message