Actually the problem is not simple. Based on these problems, there are three companies working on them:
- Loggly:
  http://loggly.com/

  Loggly is a part of the Amazon Marketplace:
  https://aws.amazon.com/solution-providers/isv/loggly

- Pageduty:
  https://papertrailapp.com/
 
  How to do it:
  http://help.papertrailapp.com/discussions/questions/1767-logging-to-heroku-papertrail-addon-from-an-app-on-external-server

 http://help.papertrailapp.com/kb/how-it-works/permanent-log-archives

http://help.papertrailapp.com/kb/analytics/log-analytics-with-hadoop-and-hive
 

- Splunk, with its Cloud platform Splunk Storm: 
  https://www.splunkstorm.com/


With any of these applications, you can collect permanently or based on schedule all generated logs from Amazon EC2, which is the base service for EMR and the others.

Best wishes.



2013/4/19 Mark Kerzner <mark.kerzner@shmsoft.com>
So you are saying, the problem is very simple. Just before you destroy the cluster, simply collect the logs to S3. Anyway, I only need them after I have completed with a specific computation. So I don't need any special requirements.

In regular permanent clusters, is there something that allows you to see all logs in one place?

Thank you,
Mark


On Thu, Apr 18, 2013 at 11:51 PM, Marcos Luis Ortiz Valmaseda <marcosluis2186@gmail.com> wrote:
When you destroy an EC2 instance, the correct behavior is to erase all data. 
Why don't you create a service to collect the logs directly to a S3 bucket in real-time or in a batch of 5 mins? 


2013/4/18 Mark Kerzner <mark.kerzner@shmsoft.com>
Hi,

my clusters are on EC2, and they disappear after the cluster's instances are destroyed. What is the best practice to collect the logs for later storage?

EC2 does exactly that with their EMR, how do they do it?

Thank you,
Mark



--
Marcos Ortiz Valmaseda,
Data-Driven Product Manager at PDVSA
Blog: http://dataddict.wordpress.com/
LinkedIn: http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186




--
Marcos Ortiz Valmaseda,
Data-Driven Product Manager at PDVSA
Blog: http://dataddict.wordpress.com/
LinkedIn: http://www.linkedin.com/in/marcosluis2186
Twitter: @marcosluis2186