orc-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From István <lecc...@gmail.com>
Subject Re: Writing ORC files without HDFS
Date Wed, 13 Jan 2016 13:44:45 GMT
I have created a small project with this code, if I can add it to the
documentation I would.

Regards,
Istvan

On Mon, Jan 11, 2016 at 10:38 AM, Lefty Leverenz <leftyleverenz@gmail.com>
wrote:

> Should this be included in the ORC documentation?
>
> -- Lefty
>
> On Fri, Jan 8, 2016 at 2:33 PM, István <leccine@gmail.com> wrote:
>
>> Hi Ravi,
>>
>> Excellent response, thank you very much, this is exactly I was looking
>> for!
>>
>> Best regards,
>> Istvan
>>
>> On Fri, Jan 8, 2016 at 8:38 AM, Ravi Tatapudi <ravi_tatapudi@in.ibm.com>
>> wrote:
>>
>>> Hello,
>>>
>>> You can write ORC-files on local-filesystem, by getting the local
>>> "FileSystem" object, as "FileSystem.getLocal(conf)". Pl. find below the
>>> simple-example given below & see if it works for your requirement.
>>>
>>> =================================================
>>> public class orcw {
>>>
>>>         private static Configuration conf = new Configuration();
>>>         public static Writer writer ;
>>>
>>> public static class OrcRow
>>> {
>>>         int col1 ;
>>>         String col2 ;
>>>         String col3 ;
>>>
>>>         OrcRow(int a, String b, String c) {
>>>         this.col1  = a ;
>>>         this.col2  = b ;
>>>         this.col3  = c ;
>>>         }
>>> }
>>>
>>> public static void main(String[] args) throws IOException,
>>>
>>>     InterruptedException, ClassNotFoundException {
>>>
>>>             String path = "/tmp/orcfile1";
>>>
>>>             try {
>>>
>>>             conf = new Configuration();
>>>         FileSystem fs = FileSystem.getLocal(conf);
>>>
>>>             ObjectInspector ObjInspector =
>>> ObjectInspectorFactory.getReflectionObjectInspector(OrcRow.class,
>>> ObjectInspectorFactory.ObjectInspectorOptions.JAVA);
>>>             writer = OrcFile.createWriter(new Path(path),
>>> OrcFile.writerOptions(conf).inspector(ObjInspector).stripeSize(100000).bufferSize(10000).compress(CompressionKind.ZLIB).version(OrcFile.Version.V_0_12));
>>>
>>>         writer.addRow(new OrcRow(1,"hello","orcFile")) ;
>>>         writer.addRow(new OrcRow(2,"hello2","orcFile2")) ;
>>>
>>>             writer.close();
>>>             }
>>>             catch (Exception e)
>>>             {
>>>                     e.printStackTrace();
>>>             }
>>>     }
>>> }
>>> =================================================
>>>
>>> Thanks,
>>>  Ravi
>>>
>>>
>>>
>>> From:        István <leccine@gmail.com>
>>> To:        user@orc.apache.org
>>> Date:        01/08/2016 01:53 AM
>>> Subject:        Writing ORC files without HDFS
>>> ------------------------------
>>>
>>>
>>>
>>> Hi all,
>>>
>>> I am working on a project that requires me to write ORC files locally on
>>> a non-HDFS location. I was wondering if there is any project doing
>>> something similar, but I guess there is none, after spending some time on
>>> Google.
>>>
>>> I think what needs to get done is to re-implement the ORC Writer, sort
>>> of similar to the following but leaving out Hadoop:
>>>
>>>
>>> *https://github.com/apache/hive/blob/master/orc/src/java/org/apache/orc/impl/WriterImpl.java*
>>> <https://github.com/apache/hive/blob/master/orc/src/java/org/apache/orc/impl/WriterImpl.java>
>>>
>>> Am I on the right track implementing this?
>>>
>>> Let me know if you have any suggestions or links in the subject.
>>>
>>> Thank you very much,
>>> Istvan
>>>
>>> --
>>> the sun shines for all
>>>
>>>
>>>
>>>
>>
>>
>> --
>> the sun shines for all
>>
>>
>>
>


-- 
the sun shines for all

Mime
View raw message