Return-Path: X-Original-To: apmail-orc-user-archive@minotaur.apache.org Delivered-To: apmail-orc-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9E9C0184B1 for ; Thu, 14 Jan 2016 10:29:11 +0000 (UTC) Received: (qmail 50959 invoked by uid 500); 14 Jan 2016 10:29:11 -0000 Delivered-To: apmail-orc-user-archive@orc.apache.org Received: (qmail 50921 invoked by uid 500); 14 Jan 2016 10:29:11 -0000 Mailing-List: contact user-help@orc.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@orc.apache.org Delivered-To: mailing list user@orc.apache.org Received: (qmail 50911 invoked by uid 99); 14 Jan 2016 10:29:11 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2016 10:29:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 17A511A08ED for ; Thu, 14 Jan 2016 10:29:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.447 X-Spam-Level: ** X-Spam-Status: No, score=2.447 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, MSGID_FROM_MTA_HEADER=0.001, RP_MATCHES_RCVD=-0.554, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Ioc_UGZbY5Ux for ; Thu, 14 Jan 2016 10:29:00 +0000 (UTC) Received: from e28smtp06.in.ibm.com (e28smtp06.in.ibm.com [125.16.236.6]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 72A3F43A97 for ; Thu, 14 Jan 2016 10:28:58 +0000 (UTC) Received: from localhost by e28smtp06.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 14 Jan 2016 15:49:01 +0530 Received: from d28relay04.in.ibm.com (9.184.220.61) by e28smtp06.in.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 14 Jan 2016 15:48:35 +0530 X-IBM-Helo: d28relay04.in.ibm.com X-IBM-MailFrom: ravi_tatapudi@in.ibm.com X-IBM-RcptTo: user@orc.apache.org Received: from d28av04.in.ibm.com (d28av04.in.ibm.com [9.184.220.66]) by d28relay04.in.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u0EAHnEQ2032054 for ; Thu, 14 Jan 2016 15:47:50 +0530 Received: from d28av04.in.ibm.com (localhost [127.0.0.1]) by d28av04.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u0EAHnQ8019593 for ; Thu, 14 Jan 2016 15:47:49 +0530 Received: from d50lp31.co.us.ibm.com (d50lp31.boulder.ibm.com [9.17.249.32]) by d28av04.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u0EAHcKc018856 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Thu, 14 Jan 2016 15:47:42 +0530 Message-Id: <201601141017.u0EAHcKc018856@d28av04.in.ibm.com> Received: from localhost by d50lp31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 14 Jan 2016 00:47:08 -0700 Received: from smtp.notes.na.collabserv.com (192.155.248.73) by d50lp31.co.us.ibm.com (192.168.2.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256/256) Thu, 14 Jan 2016 00:47:06 -0700 X-IBM-Helo: smtp.notes.na.collabserv.com X-IBM-MailFrom: ravi_tatapudi@in.ibm.com X-IBM-RcptTo: user@orc.apache.org Received: from /spool/local by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Thu, 14 Jan 2016 07:47:04 -0000 Received: from us1a3-smtp03.a3.dal06.isc4sb.com (10.106.154.94) by smtp.notes.na.collabserv.com (10.106.227.90) with smtp.notes.na.collabserv.com ESMTP; Thu, 14 Jan 2016 07:47:03 -0000 Received: from us1a3-mail103.a3.dal06.isc4sb.com ([10.146.21.193]) by us1a3-smtp03.a3.dal06.isc4sb.com with ESMTP id 2016011407474610-59049 ; Thu, 14 Jan 2016 07:47:46 +0000 In-Reply-To: To: user@orc.apache.org Subject: Re: Writing ORC files without HDFS From: "Ravi Tatapudi" Date: Thu, 14 Jan 2016 13:17:01 +0530 References: <201601080738.u087cfgc009623@d28av03.in.ibm.com><201601110947.u0B9l0vT020375@d28av02.in.ibm.com> MIME-Version: 1.0 X-KeepSent: F6250DF1:658668FA-65257F3A:002AA987; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1SHF211 December 19, 2013 X-LLNOutbound: False X-Disclaimed: 34399 X-TNEFEvaluated: 1 Content-Type: multipart/alternative; boundary="=_alternative 002AAF6365257F3A_=" x-cbid: 16011410-0021-0000-0000-000009891A6C X-IBM-ISS-SpamDetectors: Score=0.371236; BY=0.041251; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.371236; ST=0; TS=0; UL=0; ISC= X-IBM-ISS-DetailInfo: BY=3.00004786; HX=3.00000239; KW=3.00000007; PH=3.00000004; SC=3.00000133; SDB=6.00645048; UDB=6.00290911; UTC=2016-01-14 07:47:04 x-cbparentid: 16011407-0598-0000-0000-000005F744C1 X-TM-AS-MML: disable --=_alternative 002AAF6365257F3A_= Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="ISO-8859-1" Oh ok. Thanks for the info. Regards, Ravi From: Istv=E1n To: user@orc.apache.org Date: 01/13/2016 08:23 PM Subject: Re: Writing ORC files without HDFS Hi Ravi, I got the code working here: https://github.com/StreamBright/orcdemo/blob/master/src/main/java/org/strea= mbright/orcdemo/App.java It seems the OrcFile.createWriter takes a path on the local filesystem and = there is no need for FileSystem.getLocal. Regards, Istvan On Mon, Jan 11, 2016 at 10:46 AM, Ravi Tatapudi =20 wrote: Yes. I think, including the below example-code, to ORC-documentation would = be useful (for test purposes...etc). Regards, Ravi From: Lefty Leverenz To: user@orc.apache.org Date: 01/11/2016 03:08 PM Subject: Re: Writing ORC files without HDFS Should this be included in the ORC documentation? -- Lefty On Fri, Jan 8, 2016 at 2:33 PM, Istv=E1n wrote: Hi Ravi, Excellent response, thank you very much, this is exactly I was looking=20 for! Best regards, Istvan On Fri, Jan 8, 2016 at 8:38 AM, Ravi Tatapudi = wrote: Hello, You can write ORC-files on local-filesystem, by getting the local=20 "FileSystem" object, as "FileSystem.getLocal(conf)". Pl. find below the=20 simple-example given below & see if it works for your requirement. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D public class orcw { private static Configuration conf =3D new Configuration(); public static Writer writer ; public static class OrcRow { int col1 ; String col2 ; String col3 ; OrcRow(int a, String b, String c) { this.col1 =3D a ;=20 this.col2 =3D b ;=20 this.col3 =3D c ;=20 } } public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException { String path =3D "/tmp/orcfile1"; try { conf =3D new Configuration(); FileSystem fs =3D FileSystem.getLocal(conf); ObjectInspector ObjInspector =3D=20 ObjectInspectorFactory.getReflectionObjectInspector(OrcRow.class,=20 ObjectInspectorFactory.ObjectInspectorOptions.JAVA); writer =3D OrcFile.createWriter(new Path(path),=20 OrcFile.writerOptions(conf).inspector(ObjInspector).stripeSize(100000).buff= erSize(10000).compress(CompressionKind.ZLIB).version(OrcFile.Version.V=5F0= =5F12)); writer.addRow(new OrcRow(1,"hello","orcFile")) ; writer.addRow(new OrcRow(2,"hello2","orcFile2")) ; writer.close(); }=20 catch (Exception e) { e.printStackTrace(); } } } =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Thanks, Ravi From: Istv=E1n To: user@orc.apache.org Date: 01/08/2016 01:53 AM Subject: Writing ORC files without HDFS Hi all, I am working on a project that requires me to write ORC files locally on a = non-HDFS location. I was wondering if there is any project doing something = similar, but I guess there is none, after spending some time on Google. I think what needs to get done is to re-implement the ORC Writer, sort of=20 similar to the following but leaving out Hadoop: https://github.com/apache/hive/blob/master/orc/src/java/org/apache/orc/impl= /WriterImpl.java Am I on the right track implementing this?=20 Let me know if you have any suggestions or links in the subject. Thank you very much, Istvan --=20 the sun shines for all --=20 the sun shines for all --=20 the sun shines for all --=_alternative 002AAF6365257F3A_= Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset="ISO-8859-1" Oh ok. Thanks for the info.
Regards,
Ravi



From:        Istv=E1n <leccine@gmail.= com>
To: &n= bsp;      user@orc.apache.org<= br>Date:     &= nbsp;  01/13/2016 08:23 PM<= br>Subject:        Re: Writing ORC files without HDFS



Hi = Ravi,

I got the code working here:
<= br>https:= //github.com/StreamBright/orcdemo/blob/master/src/main/java/org/streambrigh= t/orcdemo/App.java

It seems the O= rcFile.createWriter takes a path on the local filesystem and there is no need for FileSystem.getLocal.

Regards,
Istvan
On Mon, Jan 11, 2016 at 10:46 AM, Ravi Tatapudi <ravi=5Ftatapudi@in.ibm.com> wrote:
Yes. I think, including= the below example-code, to ORC-documentation would be useful (for test purposes...etc).

Regards,
Rav= i




From:        Lefty Leverenz <leftyleverenz@gmail.= com>
To:        = ;user@orc.apache.org=
Date:     =    01/11/2016 03:08 PM
Subje= ct:        Re: Writing ORC files without HDFS




Should this be included in the ORC documenta= tion?

-- Lefty

On Fri, Jan 8, 2016 at 2:33 PM, Istv=E1n <<= /font>leccine@gmail.com> wrote:
Hi Ravi,

Excellent response, thank you very much, this is = exactly I was looking for!

Best regards,
Istvan

On Fri, Jan 8, 2016 at 8:38 AM, = Ravi Tatapudi <ravi=5Ftatapudi@in.ibm.com> wrote:
Hello,

You can w= rite ORC-files on local-filesystem, by getting the local "FileSystem&q= uot; object, as "FileSystem.getLocal(conf)". Pl. find below the simple= -example given below & see if it works for your requirement.

=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
public clas= s orcw {

        private static Configuration co= nf =3D new Configuration();
        public static Wr= iter writer ;

public static class OrcRow
{
     = ;   int col1 ;
        String col2 ;
  =       String col3 ;

        OrcRo= w(int a, String b, String c) {
        this.col1 &nb= sp;=3D a ;
        this.col2  =3D b ;
&nbs= p;       this.col3  =3D c ;
      &n= bsp; }
}

public static void main(String[] args) throws IOExceptio= n,

    InterruptedException, ClassNotFoundException {
<= br>            String path =3D "/tmp/orc= file1";

            try {

=             conf =3D new Configuration();
=         FileSystem fs =3D FileSystem.getLocal(conf);
            ObjectInspector ObjInspector =3D ObjectInspectorFactory.getReflectionObjectInspector(OrcRow.class, Objec= tInspectorFactory.ObjectInspectorOptions.JAVA);
      &nb= sp;     writer =3D OrcFile.createWriter(new Path(path), OrcFile.writerOptions(conf).inspector(ObjInspector).stripeSize(= 100000).bufferSize(10000).compress(CompressionKind.ZLIB).version(OrcFile.Ve= rsion.V=5F0=5F12));

        writer.addRow(new Or= cRow(1,"hello","orcFile")) ;
        writer.addRow(new OrcRow(2,"hello2&qu= ot;,"orcFile2")) ;

            writer.close();
 = ;           }
        &nbs= p;   catch (Exception e)
            = {
                    = e.printStackTrace();
            }
&nbs= p;   }
}
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D

Thanks,
 Ravi


From: &= nbsp;      Istv=E1n <leccine@gmail.com>
To:        user@orc.apache.org
Date:        = 01/08/2016 01:53 AM
Subje= ct:        Wr= iting ORC files without HDFS




Hi all,

I am working on a project that requir= es me to write ORC files locally on a non-HDFS location. I was wondering if there is any project doing something similar, but I guess there is none, after spending some time on Google.
=
I think what needs to get done is to re-implement the ORC Writer, sort of similar to the following but leaving out Hadoop:

https://github.com/apache/hive/blob/maste= r/orc/src/java/org/apache/orc/impl/WriterImpl.java

Am I on the right track implementing this? 

Let me= know if you have any suggestions or links in the subject.

Thank you= very much,
Istvan

--
the sun shines for all



<= br>

--
the sun shines for all







--
the s= un shines for all



--=_alternative 002AAF6365257F3A_=--