hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Håvard Wahl Kongsgård <haavard.kongsga...@gmail.com>
Subject Re: pipes(pydoop) and hbase classpath
Date Wed, 15 Aug 2012 08:01:16 GMT
Hi, needed to add this as well


<property>
<name>hbase.mapred.tablecolumns</name>
<value>col_fam:name</value>
</property>

-Håvard


On Wed, Aug 15, 2012 at 9:42 AM, Håvard Wahl Kongsgård
<haavard.kongsgaard@gmail.com> wrote:
> Hi, my job config is
>
> <property>
> <name>mapred.input.format.class</name>
> <value>org.apache.hadoop.hbase.mapred.TableInputFormat</value>
> </property>
>
> <property>
>   <name>hadoop.pipes.java.recordreader</name>
>   <value>true</value>
> </property>
>
>
> Exception in thread "main" java.lang.RuntimeException: Error in
> configuring object
>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>         at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>         at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:596)
>         at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:977)
>         at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:969)
>         at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1248)
>         at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
>         at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
>         at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>         ... 17 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hbase.mapred.TableInputFormat.configure(TableInputFormat.java:51)
>
>
> should I included the col names? according to the api it's deprecated?
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapred/TableInputFormat.html
>
>
> -Håvard
>
>
> On Tue, Aug 14, 2012 at 11:17 PM, Harsh J <harsh@cloudera.com> wrote:
>> Hi,
>>
>> Per:
>>
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>> org.apache.hadoop.mapred.InputFormat
>>
>> Pydoop seems to be expecting you to pass it an old API class for
>> InputFormat/etc. but you've passed in the newer class. I am unsure
>> what part of your code exactly may be at fault since I do not have
>> access to it, but you probably want to use the deprecated
>> org.apache.hadoop.hbase.mapred.* package classes such as
>> org.apache.hadoop.hbase.mapred.TableInputFormat, and not the
>> org.apache.hadoop.hbase.mapreduce.* classes, as you are using at the
>> moment.
>>
>> HTH!
>>
>> On Wed, Aug 15, 2012 at 2:39 AM, Håvard Wahl Kongsgård
>> <haavard.kongsgaard@gmail.com> wrote:
>>> Hi, I'am trying to read hbase key-values with pipes(pydoop). As hadoop
>>> is unable to find the hbase jar files. I get
>>>
>>> Exception in thread "main" java.lang.RuntimeException:
>>> java.lang.RuntimeException: class
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>>> org.apache.hadoop.mapred.InputFormat
>>>
>>> have added export
>>> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar to my
>>> hadoop-env.sh
>>>
>>> According to the doc from cloudera,
>>> https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-UsingMapReducewithHBase
>>> TableMapReduceUtil.addDependencyJars(job); can be used as an
>>> alternative. But is that possible with pipes?
>>>
>>> -Håvard
>>
>>
>>
>> --
>> Harsh J
>
>
>
> --
> Håvard Wahl Kongsgård
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.security-review.net/

Mime
View raw message