ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cabir Zounaidou" <czounai...@hortonworks.com>
Subject Re: Review Request 27396: Pushing component logs to HDFS from ambari-agent
Date Tue, 04 Nov 2014 22:27:54 GMT


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> >

Thanks for the review.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/config.py, lines 27-49
> > <https://reviews.apache.org/r/27396/diff/2/?file=746521#file746521line27>
> >
> >     You should be using an AmbariConfig instance and create a section in that file
to handle setting all these.  AmbariConfig is the "external property file" for things like
this.

Thanks for the suggestion.  I will move this to AmbariConfig.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py, lines 71-75
> > <https://reviews.apache.org/r/27396/diff/2/?file=746522#file746522line71>
> >
> >     We may have an HdfsExecute common library for things like this

Will move to common library.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines 52-53
> > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line52>
> >
> >     This style is generally deprecated for the string.format() method.

Fixed locally.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines 166-173
> > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line166>
> >
> >     Use the "with" syntax.

Fixed locally.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines 178-180
> > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line178>
> >
> >     No error handling here can leave an open file.  Use this style and the file
won't be left hanging.
> >     
> >     with open(self.__metadata_path, 'w') as file:
> >       file.write(...)

Fixed locally.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/util.py, lines 30-33
> > <https://reviews.apache.org/r/27396/diff/2/?file=746524#file746524line30>
> >
> >     All you have is statics.  This doesn't need to be in a class, can just be loghandler/util
with methods in it.

Fixed.


> On Nov. 3, 2014, 8:45 p.m., Nate Cole wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py, lines 183-189
> > <https://reviews.apache.org/r/27396/diff/2/?file=746523#file746523line183>
> >
> >     See comment about AmbariConfig

Will use AmbariConfig


- Cabir


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/#review59598
-----------------------------------------------------------


On Nov. 4, 2014, 10:11 p.m., Cabir Zounaidou wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27396/
> -----------------------------------------------------------
> 
> (Updated Nov. 4, 2014, 10:11 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev
Konar, Nate Cole, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-1522
>     https://issues.apache.org/jira/browse/AMBARI-1522
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The idea is to run a scheduler which computes diffs in a component log and pushes the
diff (only if available) to HDFS at fixed interval.
> 
> 1. Each component will have its own monitor
> 2. The apscheduler threadpool controls the monitor execution.
> 3. If the log directory is not available if the component (or service) is not running,
it will skip the processing.
> 4. It saves the last read line index and last modified time for next iteration
> 5. It uses HDFS shell utility to push log patch to HDFS.
> 6. Right now, the component log directory is configured in a json file.  In the next
iteration, it will try to automatically detected from the stack config.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
>   ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/27396/diff/
> 
> 
> Testing
> -------
> 
> - Ran the ambari-agent with the patch and verified
> 1. Started ambari-agent with the patch 
> 2. The scheduler started successfully
> 3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell
utility.
> - Ran the tests successfully
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Ambari Agent 1.3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO]
> :
> ----------------------------------------------------------------------
> Ran 324 tests in 8.282s
> 
> OK
> :
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 14.759s
> [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
> [INFO] Final Memory: 10M/4079M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Cabir Zounaidou
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message