Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
Date: Mon, 26 Sep 2016 00:05:14 +0000 (UTC)
From: Lei Chang <chang.lei.cn@gmail.com>
To: dev@hawq.incubator.apache.org, Kyle Dunn <kdunn@pivotal.io>,
	 <dev@hawq.incubator.apache.org>
Message-ID: <F734237186262828.CDE159D2-C29F-429F-B78D-F6D11EFE1929@mail.outlook.com>
In-Reply-To: <CABJ=BRr1Xe0BBPcTktQ4zj_QMA3-C8ppi7YZYit-gj5SF9WNeA@mail.gmail.com>
References: <CADvk=GbGKmkGcTzKNeJEgOO8u98ug9hOYmbVtjGn_mN_8_c1_g@mail.gmail.com>
 <CABJ=BRrGw1auDvs6W7tb9pMXAvD7LMhmRpwOHnoU+17SuEu7tg@mail.gmail.com>
 <CADvk=GZBLF-sZ9BwGxOmc_FpULQpuW1jL0SaM2BrCvWpvAFaiA@mail.gmail.com>
 <CABJ=BRr1Xe0BBPcTktQ4zj_QMA3-C8ppi7YZYit-gj5SF9WNeA@mail.gmail.com>
Subject: Re: PXF question with HAWQInputFormat to migrate data 1.x -> 2.x
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_5685_240881826.1474848314295"
archived-at: Mon, 26 Sep 2016 00:05:34 -0000

------=_Part_5685_240881826.1474848314295
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable


I think it might be possible to use HAWQ 1.x MR Inputformt to develop a 2.0=
 pxf plugin. Then we =C2=A0do not need to run 2 versions together.

Cheers
Lei


On Mon, Sep 26, 2016 at 4:27 AM +0800, "Goden Yao" <godenyao@apache.org> wr=
ote:


+ dev mailing list , modified the title.
Hi Kyle.
Based on your description, your scenario is (as I understand):1. HAWQ 1.x c=
luster installed.2. HAWQ 2.x cluster installed in the same nodes3. Data mig=
ration (ETL) from HAWQ 1.x files to HAWQ 2.x using PXF (from 2.x installati=
on)
Is that correct?So you want to develop a custom PXF plugin that can read HA=
WQ 1.x parquet data as external tables on HDFS then Insert into new HAWQ 2.=
x native table?
According to 1.3 doc:http://hdb.docs.pivotal.io/131/topics/HAWQInputFormatf=
orMapReduce.html#hawqinputformatexample=C2=A0

1) To use=C2=A0HAWQInputFormat, it'll require you also run HAWQ 1.x (as it =
requires database URL to access metadata), so this mean you need to run 1.x=
 and 2.x side by side. In theory , it should be doable, but configuration w=
ise, no one has tried this.
2) If you run hawq side by side, this means PXF will run side by side as we=
ll - have to make sure there's no conflicts in ports or ambiguity of which =
version PXF you are invoking.
That's all I can think of for now.-Goden
On Fri, Sep 23, 2016 at 12:10 PM Kyle Dunn <kdunn@pivotal.io> wrote:
Glad to hear Resolver is the only other piece - should work out nicely.
So I'm looking at bolting on HAWQInputFormat to PXF (which actually looks q=
uite straightforward) and I just want to ensure as many column types are su=
pported as possible. This is motivated by needing to be able to read orphan=
ed HAWQ 1.x files with PXF in HDB/HAWQ 2.x. Will make "in-place" upgrades m=
uch simpler.
Here is the list of datatypes HAWQInputFormat supports, and the potential m=
apping to PXF types:


On Fri, Sep 23, 2016 at 12:51 PM Goden Yao <godenyao@apache.org> wrote:
Thanks for the wishes.Are you talking about developing a new plugin (a new =
data source).=C2=A0Mapping data type has 2 parts:1. what pxf recognized fro=
m HAWQthis is=C2=A0https://github.com/apache/incubator-hawq/blob/master/pxf=
/pxf-api/src/main/java/org/apache/hawq/pxf/api/io/DataType.java=C2=A02. wha=
t plugins recognize and want to convert to HAWQ type. (Resolver)sample:=C2=
=A0https://github.com/apache/incubator-hawq/blob/master/pxf/pxf-hive/src/ma=
in/java/org/apache/hawq/pxf/plugins/hive/HiveResolver.java=C2=A0
basically, 1 provides a type list, and 2 select from that list to see which=
 data type should be converted to hawq recognized type.=C2=A0
If you're developing a new plugin with a new type mapping in HAWQ, you need=
 to do both 1 and 2.=C2=A0
Which specific primitive type you need which is not on the list?BTW, you ca=
n also mail dev mailing list so answers will be archived in public for ever=
yone :)
-Goden

On Fri, Sep 23, 2016 at 11:43 AM Kyle Dunn <kdunn@pivotal.io> wrote:
Hey Goden -
I'm looking at extending PXF for a new data source and noticed only a subse=
t of the HAWQ-supported primitive datatypes are implemented in PXF. Is this=
 as trivial as mapping a type to the corresponding OID in "api/io/DataType.=
java" or is there something more I'm missing?
Hope the new adventure is starting well.

-Kyle--=20
Kyle Dunn | Data Engineering | PivotalDirect:=C2=A0303.905.3171=C2=A0| Emai=
l:=C2=A0kdunn@pivotal.io
--=20
Kyle Dunn | Data Engineering | PivotalDirect:=C2=A0303.905.3171=C2=A0| Emai=
l:=C2=A0kdunn@pivotal.io


------=_Part_5685_240881826.1474848314295--