crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongjin Lee <dongjin.lee...@gmail.com>
Subject Re: Compilation Fails with Hadoop 2.5.1 and HBase 0.99.0
Date Mon, 29 Sep 2014 07:12:40 GMT
Mr. Whitacre.

In fact, there is no critical reason that I have to use hbase 0.99.0. I
tried it just because it is the first version that does not support hadoop
1.x. However, In my opinion, The conversion from KeyValue to Cell will be
permanent from hbase 1.0 and on, so this problem would be better to be
fixed.

But as I am a newbie in Crunch, I thought it would be better to ask for
more experienced people. If you don't mind about adding conditional
compilation on hbase submodule, then I would like to report this issue on
JIRA and submit patch. Is it okay?

Looking forward for your response.
Thank you very much.

- Dongjin

On Mon, Sep 29, 2014 at 11:43 AM, Micah Whitacre <mkwhit@gmail.com> wrote:

> A KeyValue is an implementation of the Cell interface.  From my experience
> the Cell API has a similar API but doesn't have some of the methods which
> KeyValue has and those KeyValue methods are deprecated.  Typically you have
> to use the Cell + CellUtil API to get the same functionality as you have
> with just KeyValue.
>
> On Sun, Sep 28, 2014 at 8:47 PM, Josh Wills <jwills@cloudera.com> wrote:
>
> > I don't mind conditional compilation on the assumption that we would keep
> > it scoped to the hbase module; it seems like a lot of changes are coming
> > down the pike in HBase land. Is Cell a 1:1 mapping of the functions of
> > KeyValue?
> >
> > J
> >
> > On Sun, Sep 28, 2014 at 3:18 PM, Micah Whitacre <mkwhit@gmail.com>
> wrote:
> >
> > > Dongjin,
> > > I'm not sure we currently have any plans for setting Crunch up to do
> > > conditional compilation but if there is enough need it could be
> explored.
> > >
> > > With regard to upgrading to HBase 0.99, is there a specific need for
> that
> > > version to be supported?  Since HBase treats "odd" releases[1] as
> > > development, I wouldn't anticipate we would target releasing a version
> > for
> > > it.
> > >
> > > [1] - http://hbase.apache.org/book/upgrading.html
> > >
> > > On Sat, Sep 27, 2014 at 10:33 PM, Dongjin Lee <
> dongjin.lee.kr@gmail.com>
> > > wrote:
> > >
> > > > Hello. I am a developer who is working with hadoop 2.5.1.
> > > > I tried to build crunch 0.11.0 with hbase 0.99.0, but it failed with
> > > > following messages:
> > > >
> > > > [ERROR]
> > > >
> > > >
> > >
> >
> /home/dongjinleekr/workspace/hadoop/crunch/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HFileInputFormat.java:[133,32]
> > > > error: cannot find symbol
> > > > [ERROR]
> > > >   location: class KeyValue
> > > >
> > > >
> > >
> >
> /home/dongjinleekr/workspace/hadoop/crunch/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HFileInputFormat.java:[146,33]
> > > > error: incompatible types
> > > > [ERROR]   required: KeyValue
> > > >   found:    Cell
> > > >
> > > >
> > >
> >
> /home/dongjinleekr/workspace/hadoop/crunch/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HFileReaderFactory.java:[60,37]
> > > > error: incompatible types
> > > > [ERROR]   required: KeyValue
> > > >   found:    Cell
> > > >
> > > >
> > >
> >
> /home/dongjinleekr/workspace/hadoop/crunch/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HFileReaderFactory.java:[73,36]
> > > > error: incompatible types
> > > > [ERROR]   required: KeyValue
> > > >   found:    Cell
> > > >
> > > >
> > >
> >
> /home/dongjinleekr/workspace/hadoop/crunch/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HFileUtils.java:[422,28]
> > > > error: cannot find symbol
> > > > [INFO] 5 errors
> > > >
> > > > I found that this problem occurs by following issue:
> > > > https://issues.apache.org/jira/browse/HBASE-11805 By this update,
> > > KeyValue
> > > > class was replaced with Cell interface. It seems to have been fixed.
> > > >
> > > > If there is any problem, I would like to take the work. However, I
> have
> > > one
> > > > question: Is there any plan to apply conditional compilation to
> Crunch,
> > > > like Giraph? It uses maven munge plugin to achieve this[^1]. If there
> > > is, I
> > > > think munge plugin and symbols should be applied first.
> > > >
> > > > Any guidance would be truly appreciated.
> > > > Thanks in advance.
> > > >
> > > > [^1]: I already fixed giraph's compilation error problem, which
> occurs
> > > with
> > > > hadoop 2.5.x. (https://issues.apache.org/jira/browse/GIRAPH-947)
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > >
> > > > *Oracle Server Technologies Group.So interested in massive-scale
> > machine
> > > > learning.facebook: www.facebook.com/dongjin.lee.kr
> > > > <http://www.facebook.com/dongjin.lee.kr>linkedin:
> > > > kr.linkedin.com/in/dongjinleekr
> > > > <http://kr.linkedin.com/in/dongjinleekr>github:
> > > > <http://goog_969573159>github.com/dongjinleekr
> > > > <http://github.com/dongjinleekr>twitter:
> www.twitter.com/dongjinleekr
> > > > <http://www.twitter.com/dongjinleekr>*
> > > >
> > >
> >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>



-- 
*Dongjin Lee*


*Oracle Server Technologies Group.So interested in massive-scale machine
learning.facebook: www.facebook.com/dongjin.lee.kr
<http://www.facebook.com/dongjin.lee.kr>linkedin:
kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>github:
<http://goog_969573159>github.com/dongjinleekr
<http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
<http://www.twitter.com/dongjinleekr>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message