Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of greycat.na.kor@gmail.com
 designates 209.85.218.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=YiyfoAYyTR5+89sErHkw8nlV62M96QPIVK9PbEufhwSzZWSqWea7f7H8bJBS9a3Mzl
         z+emKM5joetYhTt6POvJO/8GL5p91n8tEW+aMzaHkateSb8vCCdPM/lBa3rybNXcQf7y
         bGGTSpT1nGAVvUA/k3zPJjeRJNJApZ0jzfYwE=
MIME-Version: 1.0
In-Reply-To: <6acdaf2a0902250559j6ff97e5cq989c135ebfcfaab9@mail.gmail.com>
References: <6acdaf2a0902250559j6ff97e5cq989c135ebfcfaab9@mail.gmail.com>
Date: Wed, 25 Feb 2009 21:13:45 +0300
Message-ID: <240e377b0902251013l6d905317y5119c1d23dd1ad70@mail.gmail.com>
Subject: Re: Using Hadoop for near real-time processing of log data
From: Mikhail Yakshin <greycat.na.kor@gmail.com>
To: core-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi,

> Is anyone using Hadoop as more of a near/almost real-time processing
> of log data for their systems to aggregate stats, etc?

We do, although "near realtime" is pretty relative subject and your
mileage may vary. For example, startups / shutdowns of Hadoop jobs are
pretty expensive and it could take anything from 5-10 seconds up to
several minutes to get the job started and almost same thing goes for
job finalization. Generally, if your "near realtime" would tolerate
3-4-5 minutes lag, it's possible to use Hadoop.

-- 
WBR, Mikhail Yakshin