hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Dunn <kd...@pivotal.io>
Subject Re: PXF plugin help
Date Wed, 16 Nov 2016 20:20:03 GMT
Just closing the loop here. This was an oversight on my part related to v1
vs v2 MapReduce APIs (thanks +Andrew Thorbur <athorburn@pivotal.io> for
tracking this down).

I've opened https://issues.apache.org/jira/browse/HAWQ-1161 to help
minimize others from getting bit by oversights similar.

The latest commit <https://github.com/kdunn926/incubator-hawq/pull/1/files>
shows preliminary success reading native HAWQ 1.x data (AO only, for now)
with a HAWQ 2 PXF plugin.


-Kyle

On Tue, Nov 15, 2016 at 2:01 PM Kyle Dunn <kdunn@pivotal.io> wrote:

> Hello,
>
> I'm working on a new PXF plugin to read HAWQ 1.x data using
> HAWQInputFormat (aka hawq-mapreduce). The APIs seems to align relatively
> well but I'm having issues with the Accessor implementation.
>
> The pattern needed (I think) seems to follow well with plugins using a
> RecordReader class (e.g. PXF-JSON). Here is a PR for tracking the code
> under development: https://github.com/kdunn926/incubator-hawq/pull/1
>
> The issue I'm having is more Java-specific than PXF but maybe someone can
> see something obvious I'm missing. The current Fragmenter instantiates fine
> but the Accessor class is failing to compile:
>
> :pxf-hawqinputformat1x:compileJava
> /Users/kdunn/repos/incubator-hawq/pxf/pxf-hawqinputformat1x/src/main/java/org/apache/hawq/pxf/plugins/hawqinputformat/HAWQInputFormatAccessor.java:55:
> error: *HAWQInputFormatAccessor is not abstract and does not override
> abstract method getReader(JobConf,InputSplit) in HdfsSplittableDataAccessor*
> public class HAWQInputFormatAccessor extends HdfsSplittableDataAccessor {
>        ^
> /Users/kdunn/repos/incubator-hawq/pxf/pxf-hawqinputformat1x/src/main/java/org/apache/hawq/pxf/plugins/hawqinputformat/HAWQInputFormatAccessor.java:116:
> error: method does not override or implement a method from a supertype
>     @Override
>     ^
> 2 errors
> :pxf-hawqinputformat1x:compileJava FAILED
>
> The signature for *getReader()* in the derived class (
> HAWQInputFormatAccessor) is identical to the one declared abstract in the
> base class (HdfsSplittableDataAccessor) so I can't wrap my head around
> what's going on. Anyone have any ideas?
>
>
>
> Thanks,
> Kyle
> --
> *Kyle Dunn | Data Engineering | Pivotal*
> Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io
>
-- 
*Kyle Dunn | Data Engineering | Pivotal*
Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message