Mailing-List: contact user-help@orc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@orc.apache.org
MIME-Version: 1.0
In-Reply-To: <CAHbYR5qth-STx6uoQfMAbSDQZXMwvOrh4CiVaTPPh6k_kzeY1g@mail.gmail.com>
References: <CAHbYR5qth-STx6uoQfMAbSDQZXMwvOrh4CiVaTPPh6k_kzeY1g@mail.gmail.com>
From: "Owen O'Malley" <omalley@apache.org>
Date: Mon, 20 Feb 2017 09:50:37 -0800
Message-ID: <CAHfHakEXYojWde+4mcfFWuBuWAHVONnG8bmjWH3hTe4V9mJu1w@mail.gmail.com>
Subject: Re: Converting json record to ORC.
To: user@orc.apache.org
Cc: gopalv@apache.org
Content-Type: multipart/alternative; boundary=001a114dccfeafce420548f9e6f2
archived-at: Mon, 20 Feb 2017 17:50:41 -0000

--001a114dccfeafce420548f9e6f2
Content-Type: text/plain; charset=UTF-8

A few of us have written hacky ones, but we should have an official one
that is more robust. Mine was in this pull request
https://github.com/apache/orc/pull/43/commits/48a9f3443062bfaee4b684e49b137106bbfe9947#diff-efa8880e64e22de68f1e34c2f1d5b538
where I was converting the github archives data to ORC for benchmarking.

I've created a jira https://issues.apache.org/jira/browse/ORC-150 for
adding one.

.. Owen


On Sun, Feb 19, 2017 at 11:14 PM, Piyush Mukati (Data Platform) <
piyush.mukati@flipkart.com> wrote:

> Hi,
> we have a use case where our MR job have to read from old json (data where
> each line is a json with fixed schema) and ORC files. The output of the job
> will be in ORC file.
>
> I tried some approaches.
>
> 1)  Hcatalog but it was not having support for reading from multiple
> tables as of now. Json data don't have hive tables too.
>
>  2) With the help of hive ORC lib and serde.
> But unable to pass orc Struct through shuffle phase. As they don't
> implement writable.(I am creating ORCStruct in mapper)
>
> 3) Currently I am checking org.apache.orc.mapreduce apis. everything is
> good here. I have to convert exiting json record to Orcstruct.
> This looks a common use-case. Writing a converter myself look like
> reinventing.
>
> Hoping if anyone in community aware of any utils which can help me in
> converting json to ORCStruct. Any other suggestion is well come.
>
> Thanks
>
>

--001a114dccfeafce420548f9e6f2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">A few of us have written hacky ones, but we should have an=
 official one that is more robust. Mine was in this pull request=C2=A0<a hr=
ef=3D"https://github.com/apache/orc/pull/43/commits/48a9f3443062bfaee4b684e=
49b137106bbfe9947#diff-efa8880e64e22de68f1e34c2f1d5b538">https://github.com=
/apache/orc/pull/43/commits/48a9f3443062bfaee4b684e49b137106bbfe9947#diff-e=
fa8880e64e22de68f1e34c2f1d5b538</a> where I was converting the github archi=
ves data to ORC for benchmarking.<div><br></div><div>I&#39;ve created a jir=
a=C2=A0<a href=3D"https://issues.apache.org/jira/browse/ORC-150">https://is=
sues.apache.org/jira/browse/ORC-150</a> for adding one.</div><div><br></div=
><div>.. Owen<br><div><br></div></div></div><div class=3D"gmail_extra"><br>=
<div class=3D"gmail_quote">On Sun, Feb 19, 2017 at 11:14 PM, Piyush Mukati =
(Data Platform) <span dir=3D"ltr">&lt;<a href=3D"mailto:piyush.mukati@flipk=
art.com" target=3D"_blank">piyush.mukati@flipkart.com</a>&gt;</span> wrote:=
<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><font face=3D"arial, he=
lvetica, sans-serif">Hi,<br>we have a use case where our MR job have to rea=
d from old json (data where each line is a json with fixed schema) and ORC =
files. The output of the job will be in ORC file.<br><br></font><div><font =
face=3D"arial, helvetica, sans-serif">I tried some approaches.<br><br></fon=
t></div><div><font face=3D"arial, helvetica, sans-serif">1) =C2=A0Hcatalog =
but it was not having support for reading from multiple tables as of now. J=
son data don&#39;t have hive tables too.</font><div><font face=3D"arial, he=
lvetica, sans-serif"><br></font></div><div><font face=3D"arial, helvetica, =
sans-serif">=C2=A02) With the help of hive ORC lib and serde. <br>But unabl=
e to pass orc Struct through shuffle phase. As they don&#39;t implement wri=
table.(I am creating ORCStruct in mapper)<br><br>3) Currently I am checking=
</font><span style=3D"background-color:rgb(255,255,255)"><font face=3D"aria=
l, helvetica, sans-serif">=C2=A0<span style=3D"color:rgb(0,0,0)">org.apache=
.orc.</span><span style=3D"color:rgb(0,0,0)">mapred<wbr>uce=C2=A0</span><fo=
nt color=3D"#000000">apis. everything is good here. I have to convert exiti=
ng json record to Orcstruct. </font><br><font color=3D"#000000">This looks =
a common use-case. Writing=C2=A0a converter myself look like reinventing. <=
br><br>Hoping=C2=A0if anyone in community aware of any utils which can help=
 me in converting json to ORCStruct<span style=3D"font-size:12px">. </span>=
Any other=C2=A0suggestion is well come.=C2=A0<br><br>Thanks<br><span style=
=3D"font-size:9pt">=C2=A0</span></font></font></span></div></div></div>
</blockquote></div><br></div>

--001a114dccfeafce420548f9e6f2--