Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D1F53108CC for ; Fri, 26 Apr 2013 00:09:45 +0000 (UTC) Received: (qmail 7792 invoked by uid 500); 26 Apr 2013 00:09:40 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 7611 invoked by uid 500); 26 Apr 2013 00:09:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 7604 invoked by uid 99); 26 Apr 2013 00:09:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Apr 2013 00:09:40 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: encountered temporary error during SPF processing of domain of mark.kerzner@shmsoft.com) Received: from [74.125.82.181] (HELO mail-we0-f181.google.com) (74.125.82.181) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Apr 2013 00:09:35 +0000 Received: by mail-we0-f181.google.com with SMTP id m1so3095490wea.26 for ; Thu, 25 Apr 2013 17:08:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:x-originating-ip:in-reply-to:references :date:message-id:subject:from:to:content-type:x-gm-message-state; bh=5/O4YPm+RYLYoG9BTtfxGqbDQaBkBXM64Ghq61qLQt4=; b=CJ/2zuHpG0LWRoEnfE0LCZOMy9EtR5OSjGwhQBh3V/c2KKuVAqGnr8LDXX13aeAhKC 22+sWlouMJOSqpjh0jwXrsEuHDvVNbP5o1iVxMRqrR6myAfX6tnOEahopkYKZFfoOFZI sTxH5pqRy6+o/JFno3ngI/CYy7sAAQvCFXv+cJQi0nGIusxuAoB23r7S89AsG7YcsJBr AKmFCmet17g4g0kdV6akjPDl0QExcK0hUjA5wvQc7CrYmjEBJrM/BfZIU8UIN9edAb+I XQsZ4B7zCHethaCb6i06a3fNnkuIaKe16D1OiNi9A/VtiMdDIA6YSrCmNS3inJ+fVJoe 22aA== MIME-Version: 1.0 X-Received: by 10.180.91.131 with SMTP id ce3mr790573wib.5.1366934933804; Thu, 25 Apr 2013 17:08:53 -0700 (PDT) Received: by 10.194.81.198 with HTTP; Thu, 25 Apr 2013 17:08:53 -0700 (PDT) X-Originating-IP: [98.198.171.77] In-Reply-To: References: Date: Thu, 25 Apr 2013 19:08:53 -0500 Message-ID: Subject: Re: Best way to collect Hadoop logs across cluster From: Mark Kerzner To: Hadoop User Content-Type: multipart/alternative; boundary=f46d04388e552927d204db38574b X-Gm-Message-State: ALoCoQnrToronbO2LVa0jRsBIMD+4+Mu7HIOX4/5JDT2XNWkMK6/UpSo4kYU9IgkpBTidqLk/gJH X-Virus-Checked: Checked by ClamAV on apache.org --f46d04388e552927d204db38574b Content-Type: text/plain; charset=ISO-8859-1 Thank you for all the advice, it was indeed very useful. Mark On Thu, Apr 18, 2013 at 11:44 PM, Roman Shaposhnik wrote: > On Thu, Apr 18, 2013 at 9:23 PM, Mark Kerzner > wrote: > > Hi, > > > > my clusters are on EC2, and they disappear after the cluster's instances > are > > destroyed. What is the best practice to collect the logs for later > storage? > > > > EC2 does exactly that with their EMR, how do they do it? > > Apache Flume could be extremely useful for this purpose. You > can even configure it to deposit log data in realtime into > S3. > > Thanks, > Roman. > --f46d04388e552927d204db38574b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thank you for all the advice, it was indeed very useful.
Mark


On Thu, Apr 18, 2013 at 11:44 PM, Roman Shaposhnik <rvs@apache.org= > wrote:
On T= hu, Apr 18, 2013 at 9:23 PM, Mark Kerzner <mark.kerzner@shmsoft.com> wrote:
> Hi,
>
> my clusters are on EC2, and they disappear after the cluster's ins= tances are
> destroyed. What is the best practice to collect the logs for later sto= rage?
>
> EC2 does exactly that with their EMR, how do they do it?

Apache Flume could be extremely useful for this purpose. You can even configure it to deposit log data in realtime into
S3.

Thanks,
Roman.

--f46d04388e552927d204db38574b--