orc-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Ruchovets <oruchov...@gmail.com>
Subject Re: convert avro to orc
Date Fri, 08 Dec 2017 07:43:53 GMT
Hello Owen.
   That is interesting.
>From your experience will it support hive external / managed table.
My Idea was to prepare ORC object ( without HIVE ) and after that register
it as external Hive table. Motivation is to prevent hive schema maintenance

Thanks
Oleg.

On Thu, Dec 7, 2017 at 2:55 AM, Owen O'Malley <owen.omalley@gmail.com>
wrote:

> It would be a nice addition to the conversion tools. A first pass of
> converting Avro schemas to ORC would be pretty easy with:
>
> boolean -> boolean
> int -> int
> long -> long
> float -> float
> double -> double
> bytes -> binary
> string -> string
> enum -> string
> fixed -> binary
> map<X> -> map<string,X>
> array<X> -> array<X>
> record<X,Y,Z> -> struct<X,Y,Z>
> union<X,Y,Z> -> union<X,Y,Z>
>
> with special handling for union<null,X> -> X
>
> In terms of the conversion, you would just need to extend ConvertTool to
> create RecordReaders for Avro. There are already examples of JSON and CSV.
>
> .. Owen
>
>
> On Mon, Dec 4, 2017 at 11:31 PM, Oleg Ruchovets <oruchovets@gmail.com>
> wrote:
>
>> Hello.
>>     I wonder if there Utility to convert AVRO to ORC similar JSON to ORC
>> ?
>>
>> Background of what I am doing:
>>    I am reading SQL data using NIFI. NIFI returns data in AVRO format. I
>> want to store this data on s3 in ORC format and use it for hive external
>> table. for that, I need to convert AVRO to ORC and derive hive schema. NIFI
>> has component AVRO to ORC but it supports older version of HIVE and ORC.
>>
>> So the question how to convert AVRO to ORC and derive hive schema. I
>> really like Utility that you guys build for JSON. it has both conversions
>> to ORC and HIVE schema extraction.  What is the way to achieve the same in
>> case of AVRO format?
>>
>> Thanks
>> Oleg.
>>
>
>

Mime
View raw message