Mailing-List: contact chukwa-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: chukwa-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=YUGLgGJL2NerSk9rn0Xi8VbiXPhZx0uicULbLIlSBbZVnxrwCaa+2k7A3f4xTUzu
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Tue, 16 Mar 2010 21:08:37 -0700
Subject: Re: How to set up HDFS -> MySQL from trunk?
From: Eric Yang <eyang@yahoo-inc.com>
To: <chukwa-user@hadoop.apache.org>
Message-ID: <C7C5A3D5.721D%eyang@yahoo-inc.com>
Thread-Topic: How to set up HDFS -> MySQL from trunk?
Thread-Index: AcrFh4SiOeY1K2fKFUKczmIMjklJfA==
In-Reply-To: <c9118d4c1003161456x3a61563bp1e3c6769ab379f76@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Chukwa use case is probably not effected by the decision on MAPREDUCE-1126.
Chukwa key is composed of Long (time partition), String (primary key), Long
(timestamp).  The value is composed of Avro blob.  I like to try out using
Avro to serialize the comparator, but it makes no difference in Chukwa use
case because it is likely that I have to write my own comparator to begin
with for Tfile.  I agree with Chris Douglas and Time White said, the Avro
serializing comparator should be optional.

I like Tim's example:

Schema keySchema =3D ...
AvroGenericData.setMapOutputKeySchema(job, keySchema);

Hope this helps.

Regards,
Eric

On 3/16/10 2:56 PM, "Jeff Hammerbacher" <hammer@cloudera.com> wrote:

> Hey Eric,
>=20
> Could you chime in on MAPREDUCE-815 with your potential use case? We're
> currently blocked on other issues, but getting more use cases on the tabl=
e
> will be helpful.
>=20
> Thanks,
> Jeff
>=20
> On Mon, Mar 15, 2010 at 7:41 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
>> Hi Kirk,
>>=20
>> The Avro + Tfile plan depends on
>> https://issues.apache.org/jira/browse/MAPREDUCE-815. =A0The work can start
>> once Avro Input/Out format patch is included in a release build of Hadoo=
p.
>> Hence, I would project to complete this migration would be at least six
>> months from Avro Mapreduce ready. =A0It's a fair big chunk of work, and it
>> would be great if people want to pitch in to build aggregator piece to
>> control the workflow. =A0See https://issues.apache.org/jira/browse/CHUKWA-=
444
>> for reference.
>>=20
>> Regards,
>> Eric
>>=20
>> On 3/15/10 3:03 PM, "Kirk True" <kirk@mustardgrain.com> wrote:
>>=20
>>> Hi Eric,
>>>=20
>>> Any notion as to the ETA for completion of the migration?
>>>=20
>>> Thanks,
>>> Kirk
>>>=20
>>> Eric Yang wrote:
>>>>=20
>>>> Hi Kirk,
>>>>=20
>>>> I am working on a design which removes MySQL from Chukwa. =A0I am making=
 this
>>>> departure from MySQL because MDL framework was for prototype purpose. =
=A0It
>>>> will not scale in production system where Chukwa could be host on larg=
e
>>>> hadoop cluster. =A0HICC will serve data directly from HDFS in the future=
.
>>>>=20
>>>> Meanwhile, the dbAdmin.sh from Chukwa 0.3 is still compatible with tru=
nk
>>>> version of Chukwa. =A0You can load ChukwaRecords using
>>>> org.apache.hadoop.chukwa.dataloader.MetricDataLoader class or mdl.sh f=
rom
>>>> Chukwa 0.3.
>>>>=20
>>>> MetricDataLoader class will be mark as deprecated, and it will not be
>>>> supported once we make transition to Avro + Tfile.
>>>>=20
>>>> Regards,
>>>> Eric
>>>>=20
>>>> On 3/15/10 11:56 AM, "Kirk True" <kirk@mustardgrain.com>
>>>> <mailto:kirk@mustardgrain.com> =A0wrote:
>>>>=20
>>>>=20
>>>>=20
>>>>>=20
>>>>> Hi all,
>>>>>=20
>>>>> I recently switched to trunk as I was experiencing a lot of issues wi=
th
>>>>> 0.3.0. In 0.3.0, there was a dbAdmin.sh script that would run and try=
 to
>>>>> stick data in MySQL from HDFS. However, that script is gone and when =
I
>>>>> run the system as built from trunk, nothing is ever populated in the
>>>>> database. Where are the instructions for setting up the HDFS -> MySQL
>>>>> data migration for HICC?
>>>>>=20
>>>>> Thanks,
>>>>> Kirk
>>>>>=20
>>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>=20
>>=20
>=20
>=20