flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: HBase 0.98 addon for Flink 0.8
Date Mon, 03 Nov 2014 09:59:24 GMT
The problem is that I also removed the GenericTableOutputFormat because
there is an incompatibility between hadoop1 and hadoop2 for class
TaskAttemptContext and TaskAttemptContextImpl..
then it would be nice if the user doesn't have to worry about passing
pact.hbase.jtkey and pact.job.id parameters..
I think it is probably a good idea to remove hadoop1 compatibility and keep
enable HBase addon only for hadoop2 (as before) and decide how to mange
those 2 parameters..

On Mon, Nov 3, 2014 at 10:19 AM, Stephan Ewen <sewen@apache.org> wrote:

> It is fine to remove it, in my opinion.
>
> On Mon, Nov 3, 2014 at 10:11 AM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
> > That is one class I removed because it was using the deprecated API
> > GenericDataSink..I can restore them but the it will be a good idea to
> > remove those warning (also because from what I understood the Record APIs
> > are going to be removed).
> >
> > On Mon, Nov 3, 2014 at 9:51 AM, Fabian Hueske <fhueske@apache.org>
> wrote:
> >
> > > I'm not familiar with the HBase connector code, but are you maybe
> looking
> > > for the GenericTableOutputFormat?
> > >
> > > 2014-11-03 9:44 GMT+01:00 Flavio Pompermaier <pompermaier@okkam.it>:
> > >
> > > > | was trying to modify the example setting hbaseDs.output(new
> > > > HBaseOutputFormat()); but I can't see any HBaseOutputFormat
> > class..maybe
> > > we
> > > > shall use another class?
> > > >
> > > > On Mon, Nov 3, 2014 at 9:39 AM, Flavio Pompermaier <
> > pompermaier@okkam.it
> > > >
> > > > wrote:
> > > >
> > > > > Maybe that's something I could add to the HBase example and that
> > could
> > > be
> > > > > better documented in the Wiki.
> > > > >
> > > > > Since we're talking about the wiki..I was looking at the Java API
(
> > > > >
> > > >
> > >
> >
> http://flink.incubator.apache.org/docs/0.6-incubating/java_api_guide.html)
> > > > > and the link to the KMeans example is not working (where it says
> For
> > a
> > > > > complete example program, have a look at KMeans Algorithm).
> > > > >
> > > > > Best,
> > > > > Flavio
> > > > >
> > > > >
> > > > > On Mon, Nov 3, 2014 at 9:12 AM, Flavio Pompermaier <
> > > pompermaier@okkam.it
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Ah ok, perfect! That was the reason why I removed it :)
> > > > >>
> > > > >> On Mon, Nov 3, 2014 at 9:10 AM, Stephan Ewen <sewen@apache.org>
> > > wrote:
> > > > >>
> > > > >>> You do not really need a HBase data sink. You can call
> > > > >>> "DataSet.output(new
> > > > >>> HBaseOutputFormat())"
> > > > >>>
> > > > >>> Stephan
> > > > >>> Am 02.11.2014 23:05 schrieb "Flavio Pompermaier" <
> > > pompermaier@okkam.it
> > > > >:
> > > > >>>
> > > > >>> > Just one last thing..I removed the HbaseDataSink because
I
> think
> > it
> > > > was
> > > > >>> > using the old APIs..can someone help me in updating
that class?
> > > > >>> >
> > > > >>> > On Sun, Nov 2, 2014 at 10:55 AM, Flavio Pompermaier
<
> > > > >>> pompermaier@okkam.it>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > Indeed this time the build has been successful
:)
> > > > >>> > >
> > > > >>> > > On Sun, Nov 2, 2014 at 10:29 AM, Fabian Hueske
<
> > > fhueske@apache.org
> > > > >
> > > > >>> > wrote:
> > > > >>> > >
> > > > >>> > >> You can also setup Travis to build your own
Github
> > repositories
> > > by
> > > > >>> > linking
> > > > >>> > >> it to your Github account. That way Travis
can build all
> your
> > > > >>> branches
> > > > >>> > >> (and
> > > > >>> > >> you can also trigger rebuilds if something
fails).
> > > > >>> > >> Not sure if we can manually trigger retrigger
builds on the
> > > Apache
> > > > >>> > >> repository.
> > > > >>> > >>
> > > > >>> > >> Support for Hadoop 1 and 2 is indeed a very
good addition
> :-)
> > > > >>> > >>
> > > > >>> > >> For the discusion about the PR itself, I would
need a bit
> more
> > > > time
> > > > >>> to
> > > > >>> > >> become more familiar with HBase. I do also
not have a HBase
> > > setup
> > > > >>> > >> available
> > > > >>> > >> here.
> > > > >>> > >> Maybe somebody else of the community who was
involved with a
> > > > >>> previous
> > > > >>> > >> version of the HBase connector could comment
on your
> question.
> > > > >>> > >>
> > > > >>> > >> Best, Fabian
> > > > >>> > >>
> > > > >>> > >> 2014-11-02 9:57 GMT+01:00 Flavio Pompermaier
<
> > > > pompermaier@okkam.it
> > > > >>> >:
> > > > >>> > >>
> > > > >>> > >> > As suggestes by Fabian I moved the discussion
on this
> > mailing
> > > > >>> list.
> > > > >>> > >> >
> > > > >>> > >> > I think that what is still to be discussed
is how  to
> > > retrigger
> > > > >>> the
> > > > >>> > >> build
> > > > >>> > >> > on Travis (I don't have an account) and
if the PR can be
> > > > >>> integrated.
> > > > >>> > >> >
> > > > >>> > >> > Maybe what I can do is to move the HBase
example in the
> test
> > > > >>> package
> > > > >>> > >> (right
> > > > >>> > >> > now I left it in the main folder) so it
will force Travis
> to
> > > > >>> rebuild.
> > > > >>> > >> > I'll do it within a couple of hours.
> > > > >>> > >> >
> > > > >>> > >> > Another thing I forgot to say is that
the hbase extension
> is
> > > now
> > > > >>> > >> compatible
> > > > >>> > >> > with both hadoop 1 and 2.
> > > > >>> > >> >
> > > > >>> > >> > Best,
> > > > >>> > >> > Flavio
> > > > >>> > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message