Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of akumarb2010@gmail.com
 designates 209.85.216.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <SNT149-W8151557956FEDF6E140763D0AF0@phx.gbl>
References: 
 <CAHdEuM24Cz6=NWo1tAUNUogE+-7io1DONYbVx3aj8M8=RdMZcQ@mail.gmail.com>
	<SNT149-W8151557956FEDF6E140763D0AF0@phx.gbl>
Date: Fri, 31 Jan 2014 08:27:27 +0530
Message-ID: 
 <CAHdEuM3p7DOWxnAHwQCt3p==Kbp1NZCNa8Me1rULYEmOErwWUw@mail.gmail.com>
Subject: Re: shifting sequenceFileOutput format to Avro format
From: AnilKumar B <akumarb2010@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a113a96ca90c13404f13b5562

--001a113a96ca90c13404f13b5562
Content-Type: text/plain; charset=ISO-8859-1

Thanks Yong.

Thanks & Regards,
B Anil Kumar.


On Fri, Jan 31, 2014 at 12:44 AM, java8964 <java8964@hotmail.com> wrote:

> In avro, you need to think about a schema to match your data. Avor's
> schema is very flexible and should be able to store all kinds of data.
>
> If you have a Json string, you have 2 options to generate the Avro schema
> for it:
>
> 1) Use "type: string" to store the whole Json string into Avro. This will
> be easiest, but you have to parse the data later when you use it.
> 2) Use Avro schema to match your json data, using matching structure from
> avro for your data, like 'record, array, map' etc.
>
> Yong
>
> ------------------------------
> Date: Fri, 31 Jan 2014 00:13:59 +0530
> Subject: shifting sequenceFileOutput format to Avro format
> From: akumarb2010@gmail.com
> To: user@hadoop.apache.org
>
>
> Hi,
>
> As of now in my jobs, I am using SequenceFileOutputFormat and I am
> emitting custom java objects as MR output.
>
> Now I am planning to emit it in avro format, I went through  few blogs but
> still have following doubts.
>
> 1) My current custom Writable objects has nested json format as
> toString(), So when I shift to avro format, should I just emit json string
> in avro format, instead of writable custom object?
>
> 2) If so, how can I create schema? My json string is nested and will have
> random key/value pairs.
>
> 3) Or can I still emit as custom objects?
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>

--001a113a96ca90c13404f13b5562
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks Yong.</div><div class=3D"gmail_extra"><br clear=3D"=
all"><div>Thanks &amp; Regards,<br>B Anil Kumar.<br></div>
<br><br><div class=3D"gmail_quote">On Fri, Jan 31, 2014 at 12:44 AM, java89=
64 <span dir=3D"ltr">&lt;<a href=3D"mailto:java8964@hotmail.com" target=3D"=
_blank">java8964@hotmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"=
gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-=
left:1ex">


<div><div dir=3D"ltr">In avro, you need to think about a schema to match yo=
ur data. Avor&#39;s schema is very flexible and should be able to store all=
 kinds of data.<div><br></div><div>If you have a Json string, you have 2 op=
tions to generate the Avro schema for it:</div>
<div><br></div><div>1) Use &quot;type: string&quot; to store the whole Json=
 string into Avro. This will be easiest, but you have to parse the data lat=
er when you use it.</div><div>2) Use Avro schema to match your json data, u=
sing matching structure from avro for your data, like &#39;record, array, m=
ap&#39; etc.</div>
<div><br></div><div>Yong<br><br><div><hr>Date: Fri, 31 Jan 2014 00:13:59 +0=
530<br>Subject: shifting sequenceFileOutput format to Avro format<br>From: =
<a href=3D"mailto:akumarb2010@gmail.com" target=3D"_blank">akumarb2010@gmai=
l.com</a><br>
To: <a href=3D"mailto:user@hadoop.apache.org" target=3D"_blank">user@hadoop=
.apache.org</a><div><div class=3D"h5"><br><br><div dir=3D"ltr">Hi,<div><br>=
</div><div>As of now in my jobs, I am using SequenceFileOutputFormat and I =
am emitting custom java objects as MR output.</div>
<div><br></div><div>Now I am planning to emit it in avro format, I went thr=
ough =A0few blogs but still have following doubts.</div>
<div><br></div><div>1) My current custom Writable objects has nested json f=
ormat as toString(), So when I shift to avro format, should I just emit jso=
n string in avro format, instead of writable custom object?=A0</div><div>

<br></div><div>2) If so, how can I create schema? My json string is nested =
and will have random key/value pairs.</div><div><br></div><div>3) Or can I =
still emit as custom objects?=A0</div><div><br></div><div><br></div><div>

<br clear=3D"all"><div>Thanks &amp; Regards,<br>B Anil Kumar.<br></div>
</div></div></div></div></div></div> 		 	   		  </div></div>
</blockquote></div><br></div>

--001a113a96ca90c13404f13b5562--