Mailing-List: contact dev-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Content-Type: multipart/alternative;
 boundary="===============4622494254139985104=="
MIME-Version: 1.0
Subject: Review Request 27396: Pushing component logs to HDFS from
 ambari-agent
From: "Cabir Zounaidou" <czounaidou@hortonworks.com>
To: "Mahadev Konar" <mahadev@apache.org>,
 "John Speidel" <jspeidel@hortonworks.com>,
 "Alejandro Fernandez" <afernandez@hortonworks.com>,
 "Yusaku Sako" <yusaku@hortonworks.com>,
 "Jonathan Hurley" <jhurley@hortonworks.com>,
 "Nate Cole" <ncole@hortonworks.com>
Cc: "Ambari" <dev@ambari.apache.org>,
 "Cabir Zounaidou" <czounaidou@hortonworks.com>
Date: Fri, 31 Oct 2014 00:00:35 -0000
Message-ID: <20141031000035.7143.53588@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Cabir Zounaidou" <noreply@reviews.apache.org>
Reply-To: "Cabir Zounaidou" <czounaidou@hortonworks.com>

--===============4622494254139985104==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27396/
-----------------------------------------------------------

Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, John Speidel, Mahadev Konar, Nate Cole, and Yusaku Sako.


Bugs: AMBARI-1522
    https://issues.apache.org/jira/browse/AMBARI-1522


Repository: ambari


Description
-------

The idea is to run a scheduler which computes diffs in a component log and pushes the diff (only if available) to HDFS at fixed interval.

1. Each component will have its own monitor
2. The apscheduler threadpool controls the monitor execution.
3. If the log directory is not available if the component (or service) is not running, it will skip the processing.
4. It saves the last read line index and last modified time for next iteration
5. It uses HDFS shell utility to push log patch to HDFS.
6. Right now, the component log directory is configured in a json file.  In the next iteration, it will try to automatically detected from the stack config.


Diffs
-----

  ambari-agent/src/main/python/ambari_agent/Controller.py dc3a1cf 
  ambari-agent/src/main/python/ambari_agent/LogSchedulerHandler.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/__init__.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/config.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/hdfsapi.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/monitor.py PRE-CREATION 
  ambari-agent/src/main/python/ambari_agent/loghandler/util.py PRE-CREATION 
  ambari-agent/src/test/python/ambari_agent/TestLogHandler.py PRE-CREATION 
  ambari-agent/src/test/python/ambari_agent/dummy_files/log_handler_config.json PRE-CREATION 

Diff: https://reviews.apache.org/r/27396/diff/


Testing
-------

- Ran the ambari-agent with the patch and verified
1. Started ambari-agent with the patch 
2. The scheduler started successfully
3. The logs files are getting pushed to HDFS successfully. Verified using HDFS shell utility.
- Ran the tests successfully
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Ambari Agent 1.3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
:
----------------------------------------------------------------------
Ran 324 tests in 8.282s

OK
:
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.759s
[INFO] Finished at: Thu Oct 30 14:58:21 PDT 2014
[INFO] Final Memory: 10M/4079M
[INFO] ------------------------------------------------------------------------


Thanks,

Cabir Zounaidou


--===============4622494254139985104==--