Return-Path: X-Original-To: apmail-orc-user-archive@minotaur.apache.org Delivered-To: apmail-orc-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F2DF418DDF for ; Wed, 13 Jan 2016 13:48:39 +0000 (UTC) Received: (qmail 52660 invoked by uid 500); 13 Jan 2016 13:48:39 -0000 Delivered-To: apmail-orc-user-archive@orc.apache.org Received: (qmail 52622 invoked by uid 500); 13 Jan 2016 13:48:39 -0000 Mailing-List: contact user-help@orc.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@orc.apache.org Delivered-To: mailing list user@orc.apache.org Received: (qmail 52612 invoked by uid 99); 13 Jan 2016 13:48:39 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2016 13:48:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id EE322180425 for ; Wed, 13 Jan 2016 13:48:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id VfP-vJ3WlA8C for ; Wed, 13 Jan 2016 13:48:34 +0000 (UTC) Received: from mail-wm0-f46.google.com (mail-wm0-f46.google.com [74.125.82.46]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 601892306B for ; Wed, 13 Jan 2016 13:48:33 +0000 (UTC) Received: by mail-wm0-f46.google.com with SMTP id f206so372672375wmf.0 for ; Wed, 13 Jan 2016 05:48:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=rlTYs02XEd3ZbyPgorHTtEgthbcedl17oVJaI38R+vU=; b=kqJrrEbGTVvdJsHh/5rTcuL1Qmcr6eJsf8FDrfhbTaGkva+HXRBF57m4Oh/ytsRyH/ 7b9Ihagn5JxX1fq87sWiIZ2qrrwE+f0pmzTEUtxwPXRLr+wZYhgcXn7CVSR3Ol9xaFLM n8mm47QqaZNRr1Rs7hUsy4NwI90DCYKp802QrsXuv8pC6pWv00kB9Ta2C0H8uIUKRtKK 0YTQ8/qvGRBtMB+Ad81vKOIEGFUWWIrb136Zs78OOQhhi9g2xxfcYDhBcbR10+B70P03 8CVeFChHC2Toz6jq27n08YyzTEk52ybojMle+GWWOUrwrv/1aZAFe+hWgTVUvH2+prY1 U/jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=rlTYs02XEd3ZbyPgorHTtEgthbcedl17oVJaI38R+vU=; b=ExeM7gybdWIwPAMlG13BJ8wLaw6Uhmn0Oxt+e8sbHXbmW1/5+oTePwfFOPlRU/YAIZ 3Xm5nia3cDFGMDrQx6PG4Ycj7FKvY40Hn3+ZMrZvSs3RiWYUhpIvc1nIhi4bmF1FqYTc aUWzrDot7kVz/1c1+BGdAzv1nOY2n9FlYTn+qr4P/5DXJB8BxSK6QJC1OIxYXD5u3NhK 1BfsfJKLTiIsPw6Ick8JoMIy832WZSq2m6/50tzBCX7vJbj94bkiGiCt8vMKj+5du6NK oHPd349cXwkEAAcoXGqQYIdJd3MHp3+AdSccpsZZ6KUcspXPJp2jNgjDh3XAqLLg4mRQ jGGA== X-Gm-Message-State: ALoCoQmPpO8OKPLbd2rC/CNP1bhkakJTZ2ONWF9mD9kOQ8lCKQEUWOskHgVYSjPvJr1kPztdXRLvA8ExpnCFUW9OHSBFpgPvtg== MIME-Version: 1.0 X-Received: by 10.28.218.81 with SMTP id r78mr26820974wmg.91.1452692912017; Wed, 13 Jan 2016 05:48:32 -0800 (PST) Received: by 10.28.64.194 with HTTP; Wed, 13 Jan 2016 05:48:31 -0800 (PST) In-Reply-To: <201601080738.u087cfgc009623@d28av03.in.ibm.com> References: <201601080738.u087cfgc009623@d28av03.in.ibm.com> Date: Wed, 13 Jan 2016 14:48:31 +0100 Message-ID: Subject: Re: Writing ORC files without HDFS From: =?UTF-8?Q?Istv=C3=A1n?= To: user@orc.apache.org Content-Type: multipart/alternative; boundary=001a1145ae1efd229b0529376c9a --001a1145ae1efd229b0529376c9a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ravi, one final question: which package should I import into my project? I found this as the best candidate: org.apache.hive hive-exec 1.2.0 wrote: > Hello, > > You can write ORC-files on local-filesystem, by getting the local > "FileSystem" object, as "FileSystem.getLocal(conf)". Pl. find below the > simple-example given below & see if it works for your requirement. > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > public class orcw { > > private static Configuration conf =3D new Configuration(); > public static Writer writer ; > > public static class OrcRow > { > int col1 ; > String col2 ; > String col3 ; > > OrcRow(int a, String b, String c) { > this.col1 =3D a ; > this.col2 =3D b ; > this.col3 =3D c ; > } > } > > public static void main(String[] args) throws IOException, > > InterruptedException, ClassNotFoundException { > > String path =3D "/tmp/orcfile1"; > > try { > > conf =3D new Configuration(); > FileSystem fs =3D FileSystem.getLocal(conf); > > ObjectInspector ObjInspector =3D > ObjectInspectorFactory.getReflectionObjectInspector(OrcRow.class, > ObjectInspectorFactory.ObjectInspectorOptions.JAVA); > writer =3D OrcFile.createWriter(new Path(path), > OrcFile.writerOptions(conf).inspector(ObjInspector).stripeSize(100000).bu= fferSize(10000).compress(CompressionKind.ZLIB).version(OrcFile.Version.V_0_= 12)); > > writer.addRow(new OrcRow(1,"hello","orcFile")) ; > writer.addRow(new OrcRow(2,"hello2","orcFile2")) ; > > writer.close(); > } > catch (Exception e) > { > e.printStackTrace(); > } > } > } > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Thanks, > Ravi > > > > From: Istv=C3=A1n > To: user@orc.apache.org > Date: 01/08/2016 01:53 AM > Subject: Writing ORC files without HDFS > ------------------------------ > > > > Hi all, > > I am working on a project that requires me to write ORC files locally on = a > non-HDFS location. I was wondering if there is any project doing somethin= g > similar, but I guess there is none, after spending some time on Google. > > I think what needs to get done is to re-implement the ORC Writer, sort of > similar to the following but leaving out Hadoop: > > > *https://github.com/apache/hive/blob/master/orc/src/java/org/apache/orc/i= mpl/WriterImpl.java* > > > Am I on the right track implementing this? > > Let me know if you have any suggestions or links in the subject. > > Thank you very much, > Istvan > > -- > the sun shines for all > > > > --=20 the sun shines for all --001a1145ae1efd229b0529376c9a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Ravi,

one final question: which package= should I import into my project?

I found this as = the best candidate:

<dependency>
<= div> <groupId>org.a= pache.hive</groupId>
<artifactId>hive-exec</artifactId>
<version>1.2.0</vers= ion>
</dependency

Thanks,
Istvan

On Fri, Jan 8, 2016 at 8:38 AM, Ravi Tatapudi &l= t;ravi_tatapu= di@in.ibm.com> wrote:
Hello,

You can write ORC-files on local-filesystem, by getting the local "FileSystem" object, as "FileSystem.get= Local(conf)". Pl. find below the simple-example given below & see if it works for your requirement.

=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3Dpublic class orcw {

=C2=A0 =C2=A0 =C2=A0 =C2=A0 private static Configuration conf =3D new Configuration();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 public static Writer writer ;

p= ublic static class OrcRow
{<= /font>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = int col1 ;
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 String col2 ;
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 String col3 ;

=C2=A0 =C2=A0 =C2= =A0 =C2=A0 OrcRow(int a, String b, String c) {
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 this.col1 =C2=A0=3D a ;
=C2=A0 =C2=A0= =C2=A0 =C2=A0 this.col2 =C2=A0=3D b ;
=C2=A0 =C2=A0= =C2=A0 =C2=A0 this.col3 =C2=A0=3D c ;
=C2=A0 =C2=A0= =C2=A0 =C2=A0 }
}
public static void main(String[] = args) throws IOException,

=C2= =A0 =C2=A0 InterruptedException, ClassNotFoundException {

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 String path =3D "/tmp/orcfile1";

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 try {

=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 conf =3D new Configuration();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 FileSystem fs =3D FileSystem.getLocal(conf);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ObjectInspector ObjInspector =3D ObjectInspectorFactory.getReflectionObject= Inspector(OrcRow.class, ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 writer =3D OrcFile.createWriter(new Path(path), OrcFile.writerOptions(conf)= .inspector(ObjInspector).stripeSize(100000).bufferSize(10000).compress(Comp= ressionKind.ZLIB).version(OrcFile.Version.V_0_12));

=C2=A0 =C2=A0 =C2=A0 =C2=A0 writer.addRow(new OrcRow(1,"hello","orcFile")) ;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 writer.addRow(new OrcRow(2,"hello2","orcFile2")) ;

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 writer.close();
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 catch (Exception e)
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 {
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 e.printStackTrace();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 }
= }
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D

Thanks,=
=C2=A0Ravi



<= font size=3D"1" color=3D"#5f5f5f" face=3D"sans-serif">From: =C2=A0 =C2=A0 = =C2=A0 =C2=A0Istv=C3=A1n <leccine@gmail.com>
To: =C2=A0 = =C2=A0 =C2=A0 =C2=A0user@orc.apache.org
Date: =C2=A0 =C2=A0 =C2=A0 =C2=A001/08/2016 01:53 AM
Subject: =C2=A0= =C2=A0 =C2=A0 =C2=A0Writing ORC files without HDFS



Hi all,

I am working on a project that requires me to write ORC files locally on a non-HDFS location. I was wondering if there is any proje= ct doing something similar, but I guess there is none, after spending some time on Google.

I think what needs to get do= ne is to re-implement the ORC Writer, sort of similar to the following but leaving out Hadoop:=

https://github.com/apache/hive/blob/master/orc/src/java/org/= apache/orc/impl/WriterImpl.java

Am I= on the right track implementing this?=C2=A0

Let me know if you have any suggestions or links in the subject.

Thank you very much,
Istvan

--
the sun shines for all






--
the sun shines for all


--001a1145ae1efd229b0529376c9a--