ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cabir Zounaidou" <czounai...@hortonworks.com>
Subject Re: Review Request 27396: Pushing component logs to HDFS from ambari-agent
Date Fri, 07 Nov 2014 02:11:50 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/
-----------------------------------------------------------

(Updated Nov. 7, 2014, 2:11 a.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev Konar,
Nate Cole, and Yusaku Sako.


Changes
-------

Nate, updated as per your comments.  Can you please review?


Bugs: AMBARI-1522
    https://issues.apache.org/jira/browse/AMBARI-1522


Repository: ambari


Description
-------

The idea is to run a scheduler which computes diffs in a component log and pushes the diff
(only if available) to HDFS at fixed interval.

1. Each component will have its own monitor
2. The apscheduler threadpool controls the monitor execution.
3. If the log directory is not available if the component (or service) is not running, it
will skip the processing.
4. It saves the last read line index and last modified time for next iteration
5. It uses HDFS shell utility to push log patch to HDFS.
6. Right now, the component log directory is configured in a json file.  In the next iteration,
it will try to automatically detected from the stack config.


Diffs
-----

  ambari-agent/src/main/python/ambari_agent/AmbariConfig.py ca2e80c 
  ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
  ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION 
  ambari-agent/src/test/python/ambari_agent/TestHdfsApi.py PRE-CREATION 
  ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION 
  ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION


Diff: https://reviews.apache.org/r/27396/diff/


Testing (updated)
-------

- Ran the ambari-agent with the patch and verified
1. Started ambari-agent with the patch 
2. The scheduler started successfully
3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell utility.
- Ran the tests successfully
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Ambari Agent 1.3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
:
----------------------------------------------------------------------
Ran 324 tests in 8.282s

OK
:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.759s
[INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
[INFO] Final Memory: 10M/4079M
[INFO] ------------------------------------------------------------------------


Thanks,

Cabir Zounaidou


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message