Mailing-List: contact chukwa-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=h6ff0VNV8+mzC3mB5gKXakw928RweotOVPTtT5fWQQHLQYF0OXCj2DS1d2/Swpuj
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Wed, 17 Mar 2010 10:54:32 -0700
Subject: Re: Hbase over Chukwa demux
From: Eric Yang <eyang@yahoo-inc.com>
To: <chukwa-user@hadoop.apache.org>
Message-ID: <C7C66568.726B%eyang@yahoo-inc.com>
Thread-Topic: Hbase over Chukwa demux
Thread-Index: AcrF+uW3EYZp4YsPeEmxNQleeEkASg==
In-Reply-To: <1703587b1003171000x63f3ad7av37e9975b5347a2fe@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Hi Oded,

Current Chukwa Demux uses one reducer per record type for output.  It
depends on your data model.  It may be worth while to look into running
multiple reducer per recordtype, if your data has a lot of record for a
single data type.  I think the conf.setNumReduceTasks is specified in
org.apache.hadoop.chukwa.extraction.demux.Demux.java.  You can set more if
you don't use ChukwaRecord after demux.

The current demux needs some major update to improve, and patches are
welcome.  :)

Regards,
Eric

On 3/17/10 10:00 AM, "Oded Rosen" <oded@legolas-media.com> wrote:

> I work with a hadoop cluster with tons of new data each day.
> The data is flowing into hadoop from outside servers, using chukwa.
>=20
> Chukwa has a tool called demux, a builtin mapred job.
> Chukwa users may write their own map & reduce classes for this demux, wit=
h the
> only limitation that the input & output types are chukwa records - I cann=
ot
> use HBase's TableMap, TableReduce.
> In order to write data to hbase during this mapred job, I can only use th=
e
> table.put & table.commit, which work on one hbase raw only (aren't they?)=
.
> This raised serious latency issues, as writing=A0thousands=A0of records to hb=
ase
> this way every 5 minutes is not effective and really s-l-o-w.
> Even if I'll move the hbase writing from the map phase to the reduce phas=
e,
> the same rows should be updated, so moving the ".put" to the reducer seem=
s
> does not suppose to change anything.
>=20
> I would like to write straight to hbase from the chukwa demuxer, and not =
to
> have another job that reads the chukwa output and write it to hbase.
> The target is to have this data as fast as I can in hbase.
>=20
> Is there a way to write=A0effectively=A0to hbase without TableReduce? Have I =
got
> something wrong?
> is there someone using Chukwa that managed to do this thing?
>=20
>=20
> Thanks in advance for any kind of help,