hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Goden Yao <goden...@apache.org>
Subject PXF question with HAWQInputFormat to migrate data 1.x -> 2.x
Date Sun, 25 Sep 2016 20:27:05 GMT
+ dev mailing list , modified the title.

Hi Kyle.

Based on your description, your scenario is (as I understand):
1. HAWQ 1.x cluster installed.
2. HAWQ 2.x cluster installed in the same nodes
3. Data migration (ETL) from HAWQ 1.x files to HAWQ 2.x using PXF (from 2.x
installation)

Is that correct?
So you want to develop a custom PXF plugin that can read HAWQ 1.x parquet
data as external tables on HDFS then Insert into new HAWQ 2.x native table?

According to 1.3 doc:
http://hdb.docs.pivotal.io/131/topics/HAWQInputFormatforMapReduce.html#hawqinputformatexample


1) To use *HAWQInputFormat, *it'll require you also run HAWQ 1.x (as it
requires database URL to access metadata), so this mean you need to run 1.x
and 2.x side by side. In theory , it should be doable, but configuration
wise, no one has tried this.

2) If you run hawq side by side, this means PXF will run side by side as
well - have to make sure there's no conflicts in ports or ambiguity of
which version PXF you are invoking.

That's all I can think of for now.
-Goden

On Fri, Sep 23, 2016 at 12:10 PM Kyle Dunn <kdunn@pivotal.io> wrote:

> Glad to hear Resolver is the only other piece - should work out nicely.
>
> So I'm looking at bolting on HAWQInputFormat to PXF (which actually looks
> quite straightforward) and I just want to ensure as many column types are
> supported as possible. This is motivated by needing to be able to read
> orphaned HAWQ 1.x files with PXF in HDB/HAWQ 2.x. Will make "in-place"
> upgrades much simpler.
>
> Here is the list of datatypes HAWQInputFormat supports, and the potential
> mapping to PXF types:
>
> [image: pasted1]
>
>
>
> On Fri, Sep 23, 2016 at 12:51 PM Goden Yao <godenyao@apache.org> wrote:
>
>> Thanks for the wishes.
>> Are you talking about developing a new plugin (a new data source).
>> Mapping data type has 2 parts:
>> 1. what pxf recognized from HAWQ
>> this is
>> https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/io/DataType.java
>>
>> 2. what plugins recognize and want to convert to HAWQ type. (Resolver)
>> sample:
>> https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveResolver.java
>>
>>
>> basically, 1 provides a type list, and 2 select from that list to see
>> which data type should be converted to hawq recognized type.
>>
>> If you're developing a new plugin with a new type mapping in HAWQ, you
>> need to do both 1 and 2.
>>
>> Which specific primitive type you need which is not on the list?
>> BTW, you can also mail dev mailing list so answers will be archived in
>> public for everyone :)
>>
>> -Goden
>>
>>
>> On Fri, Sep 23, 2016 at 11:43 AM Kyle Dunn <kdunn@pivotal.io> wrote:
>>
>>> Hey Goden -
>>>
>>> I'm looking at extending PXF for a new data source and noticed only a
>>> subset of the HAWQ-supported primitive datatypes are implemented in PXF. Is
>>> this as trivial as mapping a type to the corresponding OID in
>>> "api/io/DataType.java" or is there something more I'm missing?
>>>
>>> Hope the new adventure is starting well.
>>>
>>>
>>> -Kyle
>>> --
>>> *Kyle Dunn | Data Engineering | Pivotal*
>>> Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io
>>>
>> --
> *Kyle Dunn | Data Engineering | Pivotal*
> Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io
>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message