hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Håvard Wahl Kongsgård <haavard.kongsga...@gmail.com>
Subject Re: pipes(pydoop) and hbase classpath
Date Wed, 15 Aug 2012 09:31:41 GMT
however, when run hadoop pipes -conf myconf_job.conf -input
name_of_table -output /tmp/out

I don't get any error, hadoop just stalls with

12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.5-cdh3u4--1, built on 05/07/2012
21:08 GMT
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:host.name=kongs1
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_31
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.31/jre
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun//lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-41-server
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.home=/usr/lib/hadoop-0.20
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/home/havard/d/graph
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=180000
watcher=hconnection
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:2181
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Socket connection
established to localhost/127.0.0.1:2181, initiating session
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Session establishment
complete on server localhost/127.0.0.1:2181, sessionid =
0x139266be8b90004, negotiated timeout = 40000


-Håvard


On Wed, Aug 15, 2012 at 10:01 AM, Håvard Wahl Kongsgård
<haavard.kongsgaard@gmail.com> wrote:
> Hi, needed to add this as well
>
>
> <property>
> <name>hbase.mapred.tablecolumns</name>
> <value>col_fam:name</value>
> </property>
>
> -Håvard
>
>
> On Wed, Aug 15, 2012 at 9:42 AM, Håvard Wahl Kongsgård
> <haavard.kongsgaard@gmail.com> wrote:
>> Hi, my job config is
>>
>> <property>
>> <name>mapred.input.format.class</name>
>> <value>org.apache.hadoop.hbase.mapred.TableInputFormat</value>
>> </property>
>>
>> <property>
>>   <name>hadoop.pipes.java.recordreader</name>
>>   <value>true</value>
>> </property>
>>
>>
>> Exception in thread "main" java.lang.RuntimeException: Error in
>> configuring object
>>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>>         at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>>         at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>         at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:596)
>>         at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:977)
>>         at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:969)
>>         at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
>>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
>>         at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
>>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:807)
>>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1248)
>>         at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
>>         at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
>>         at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
>> Caused by: java.lang.reflect.InvocationTargetException
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>>         ... 17 more
>> Caused by: java.lang.NullPointerException
>>         at org.apache.hadoop.hbase.mapred.TableInputFormat.configure(TableInputFormat.java:51)
>>
>>
>> should I included the col names? according to the api it's deprecated?
>> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapred/TableInputFormat.html
>>
>>
>> -Håvard
>>
>>
>> On Tue, Aug 14, 2012 at 11:17 PM, Harsh J <harsh@cloudera.com> wrote:
>>> Hi,
>>>
>>> Per:
>>>
>>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>>> org.apache.hadoop.mapred.InputFormat
>>>
>>> Pydoop seems to be expecting you to pass it an old API class for
>>> InputFormat/etc. but you've passed in the newer class. I am unsure
>>> what part of your code exactly may be at fault since I do not have
>>> access to it, but you probably want to use the deprecated
>>> org.apache.hadoop.hbase.mapred.* package classes such as
>>> org.apache.hadoop.hbase.mapred.TableInputFormat, and not the
>>> org.apache.hadoop.hbase.mapreduce.* classes, as you are using at the
>>> moment.
>>>
>>> HTH!
>>>
>>> On Wed, Aug 15, 2012 at 2:39 AM, Håvard Wahl Kongsgård
>>> <haavard.kongsgaard@gmail.com> wrote:
>>>> Hi, I'am trying to read hbase key-values with pipes(pydoop). As hadoop
>>>> is unable to find the hbase jar files. I get
>>>>
>>>> Exception in thread "main" java.lang.RuntimeException:
>>>> java.lang.RuntimeException: class
>>>> org.apache.hadoop.hbase.mapreduce.TableInputFormat not
>>>> org.apache.hadoop.mapred.InputFormat
>>>>
>>>> have added export
>>>> HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar to my
>>>> hadoop-env.sh
>>>>
>>>> According to the doc from cloudera,
>>>> https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-UsingMapReducewithHBase
>>>> TableMapReduceUtil.addDependencyJars(job); can be used as an
>>>> alternative. But is that possible with pipes?
>>>>
>>>> -Håvard
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.security-review.net/



-- 
Håvard Wahl Kongsgård
Faculty of Medicine &
Department of Mathematical Sciences
NTNU

http://havard.security-review.net/

Mime
View raw message