hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lei Chang <chang.lei...@gmail.com>
Subject Re: PXF question with HAWQInputFormat to migrate data 1.x -> 2.x
Date Mon, 26 Sep 2016 00:05:14 GMT

I think it might be possible to use HAWQ 1.x MR Inputformt to develop a 2.0 pxf plugin. Then
we  do not need to run 2 versions together.


On Mon, Sep 26, 2016 at 4:27 AM +0800, "Goden Yao" <godenyao@apache.org> wrote:

+ dev mailing list , modified the title.
Hi Kyle.
Based on your description, your scenario is (as I understand):1. HAWQ 1.x cluster installed.2.
HAWQ 2.x cluster installed in the same nodes3. Data migration (ETL) from HAWQ 1.x files to
HAWQ 2.x using PXF (from 2.x installation)
Is that correct?So you want to develop a custom PXF plugin that can read HAWQ 1.x parquet
data as external tables on HDFS then Insert into new HAWQ 2.x native table?
According to 1.3 doc:http://hdb.docs.pivotal.io/131/topics/HAWQInputFormatforMapReduce.html#hawqinputformatexample 

1) To use HAWQInputFormat, it'll require you also run HAWQ 1.x (as it requires database URL
to access metadata), so this mean you need to run 1.x and 2.x side by side. In theory , it
should be doable, but configuration wise, no one has tried this.
2) If you run hawq side by side, this means PXF will run side by side as well - have to make
sure there's no conflicts in ports or ambiguity of which version PXF you are invoking.
That's all I can think of for now.-Goden
On Fri, Sep 23, 2016 at 12:10 PM Kyle Dunn <kdunn@pivotal.io> wrote:
Glad to hear Resolver is the only other piece - should work out nicely.
So I'm looking at bolting on HAWQInputFormat to PXF (which actually looks quite straightforward)
and I just want to ensure as many column types are supported as possible. This is motivated
by needing to be able to read orphaned HAWQ 1.x files with PXF in HDB/HAWQ 2.x. Will make
"in-place" upgrades much simpler.
Here is the list of datatypes HAWQInputFormat supports, and the potential mapping to PXF types:

On Fri, Sep 23, 2016 at 12:51 PM Goden Yao <godenyao@apache.org> wrote:
Thanks for the wishes.Are you talking about developing a new plugin (a new data source). Mapping
data type has 2 parts:1. what pxf recognized from HAWQthis is https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-api/src/main/java/org/apache/hawq/pxf/api/io/DataType.java 2.
what plugins recognize and want to convert to HAWQ type. (Resolver)sample: https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hive/src/main/java/org/apache/hawq/pxf/plugins/hive/HiveResolver.java 
basically, 1 provides a type list, and 2 select from that list to see which data type should
be converted to hawq recognized type. 
If you're developing a new plugin with a new type mapping in HAWQ, you need to do both 1 and
Which specific primitive type you need which is not on the list?BTW, you can also mail dev
mailing list so answers will be archived in public for everyone :)

On Fri, Sep 23, 2016 at 11:43 AM Kyle Dunn <kdunn@pivotal.io> wrote:
Hey Goden -
I'm looking at extending PXF for a new data source and noticed only a subset of the HAWQ-supported
primitive datatypes are implemented in PXF. Is this as trivial as mapping a type to the corresponding
OID in "api/io/DataType.java" or is there something more I'm missing?
Hope the new adventure is starting well.

Kyle Dunn | Data Engineering | PivotalDirect: 303.905.3171 | Email: kdunn@pivotal.io
Kyle Dunn | Data Engineering | PivotalDirect: 303.905.3171 | Email: kdunn@pivotal.io

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message