hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivram Mani <shivram.m...@gmail.com>
Subject Re: PXF plugin help
Date Sat, 19 Nov 2016 19:04:50 GMT
Kyle, thanks for bring this up to our notice. Switching our APIs to use the
new Mapreduce APIs has been on our roadmap for our while. If you have the
bandwidth, feel free to submit a PR for this. Otherwise, we will try
prioritizing this down the line.

On Wed, Nov 16, 2016 at 12:20 PM, Kyle Dunn <kdunn@pivotal.io> wrote:

> Just closing the loop here. This was an oversight on my part related to v1
> vs v2 MapReduce APIs (thanks +Andrew Thorbur <athorburn@pivotal.io> for
> tracking this down).
>
> I've opened https://issues.apache.org/jira/browse/HAWQ-1161 to help
> minimize others from getting bit by oversights similar.
>
> The latest commit <https://github.com/kdunn926/incubator-hawq/pull/1/files
> >
> shows preliminary success reading native HAWQ 1.x data (AO only, for now)
> with a HAWQ 2 PXF plugin.
>
>
> -Kyle
>
> On Tue, Nov 15, 2016 at 2:01 PM Kyle Dunn <kdunn@pivotal.io> wrote:
>
> > Hello,
> >
> > I'm working on a new PXF plugin to read HAWQ 1.x data using
> > HAWQInputFormat (aka hawq-mapreduce). The APIs seems to align relatively
> > well but I'm having issues with the Accessor implementation.
> >
> > The pattern needed (I think) seems to follow well with plugins using a
> > RecordReader class (e.g. PXF-JSON). Here is a PR for tracking the code
> > under development: https://github.com/kdunn926/incubator-hawq/pull/1
> >
> > The issue I'm having is more Java-specific than PXF but maybe someone can
> > see something obvious I'm missing. The current Fragmenter instantiates
> fine
> > but the Accessor class is failing to compile:
> >
> > :pxf-hawqinputformat1x:compileJava
> > /Users/kdunn/repos/incubator-hawq/pxf/pxf-hawqinputformat1x/src/main/
> java/org/apache/hawq/pxf/plugins/hawqinputformat/
> HAWQInputFormatAccessor.java:55:
> > error: *HAWQInputFormatAccessor is not abstract and does not override
> > abstract method getReader(JobConf,InputSplit) in
> HdfsSplittableDataAccessor*
> > public class HAWQInputFormatAccessor extends HdfsSplittableDataAccessor {
> >        ^
> > /Users/kdunn/repos/incubator-hawq/pxf/pxf-hawqinputformat1x/src/main/
> java/org/apache/hawq/pxf/plugins/hawqinputformat/
> HAWQInputFormatAccessor.java:116:
> > error: method does not override or implement a method from a supertype
> >     @Override
> >     ^
> > 2 errors
> > :pxf-hawqinputformat1x:compileJava FAILED
> >
> > The signature for *getReader()* in the derived class (
> > HAWQInputFormatAccessor) is identical to the one declared abstract in the
> > base class (HdfsSplittableDataAccessor) so I can't wrap my head around
> > what's going on. Anyone have any ideas?
> >
> >
> >
> > Thanks,
> > Kyle
> > --
> > *Kyle Dunn | Data Engineering | Pivotal*
> > Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io
> >
> --
> *Kyle Dunn | Data Engineering | Pivotal*
> Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io
>



-- 
shivram mani

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message