ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cabir Zounaidou" <czounai...@hortonworks.com>
Subject Re: Review Request 27396: Pushing component logs to HDFS from ambari-agent
Date Tue, 04 Nov 2014 22:21:04 GMT


> On Nov. 3, 2014, 8:53 p.m., Jonathan Hurley wrote:
> > What happens when this agent code is installed in a cluster that's not running HDFS
(GlusterFS for example)?

Presently, the command will fail and will not push the log files to HDFS.  For log to work,
we need HDFS running.


The LogHandlerCommand on the ambari-server side can skip pushing the configs if HDFS service
is not available.  In that case, no jobs will be scheduled.


> On Nov. 3, 2014, 8:53 p.m., Jonathan Hurley wrote:
> > ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py, line 78
> > <https://reviews.apache.org/r/27396/diff/2-3/?file=746522#file746522line78>
> >
> >     command_args (spelling)

Fixed in the latest patch.


- Cabir


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/#review59638
-----------------------------------------------------------


On Nov. 4, 2014, 10:11 p.m., Cabir Zounaidou wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27396/
> -----------------------------------------------------------
> 
> (Updated Nov. 4, 2014, 10:11 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev
Konar, Nate Cole, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-1522
>     https://issues.apache.org/jira/browse/AMBARI-1522
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> The idea is to run a scheduler which computes diffs in a component log and pushes the
diff (only if available) to HDFS at fixed interval.
> 
> 1. Each component will have its own monitor
> 2. The apscheduler threadpool controls the monitor execution.
> 3. If the log directory is not available if the component (or service) is not running,
it will skip the processing.
> 4. It saves the last read line index and last modified time for next iteration
> 5. It uses HDFS shell utility to push log patch to HDFS.
> 6. Right now, the component log directory is configured in a json file.  In the next
iteration, it will try to automatically detected from the stack config.
> 
> 
> Diffs
> -----
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
>   ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION 
>   ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION 
>   ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/27396/diff/
> 
> 
> Testing
> -------
> 
> - Ran the ambari-agent with the patch and verified
> 1. Started ambari-agent with the patch 
> 2. The scheduler started successfully
> 3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell
utility.
> - Ran the tests successfully
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Building Ambari Agent 1.3.0-SNAPSHOT
> [INFO] ------------------------------------------------------------------------
> [INFO]
> :
> ----------------------------------------------------------------------
> Ran 324 tests in 8.282s
> 
> OK
> :
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 14.759s
> [INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
> [INFO] Final Memory: 10M/4079M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Cabir Zounaidou
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message