pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <dai...@gmail.com>
Subject Re: Strange problem with Pig 0.7.0 and Hadoop 0.20.2 and Failed to create DataStorage
Date Fri, 10 Dec 2010 08:20:14 GMT
Looks like hadoop client jar does not match the version of server side. Are 
you using hadoop 0.20.2 from Apache?

Daniel

-----Original Message----- 
From: felix gao
Sent: Thursday, December 09, 2010 5:48 PM
To: pig-user@hadoop.apache.org
Subject: Strange problem with Pig 0.7.0 and Hadoop 0.20.2 and Failed to 
create DataStorage

I kept seening Failed to create DataStroage error when try to run pig

$ java -cp pig-0.7.0-core.jar:$HADOOP_CONF_DIR org.apache.pig.Main -x
mapreduce
10/12/09 20:35:31 INFO pig.Main: Logging error messages to:
/home/testpig/pig-0.7.0/pig_1291944931735.log
2010-12-09 20:35:31,997 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: hdfs://localhost:8020
2010-12-09 20:35:32,333 [main] ERROR org.apache.pig.Main - ERROR 2999:
Unexpected internal error. Failed to create DataStorage

$ cat pig_1291944931735.log
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage

java.lang.RuntimeException: Failed to create DataStorage
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:216)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:126)
at org.apache.pig.impl.PigContext.connect(PigContext.java:184)
at org.apache.pig.PigServer.<init>(PigServer.java:184)
at org.apache.pig.PigServer.<init>(PigServer.java:173)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:54)
at org.apache.pig.Main.main(Main.java:354)
Caused by: java.io.IOException: Call to localhost/127.0.0.1:8020 failed on
local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at
org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 8 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)

if I ran java -cp pig-0.7.0-core.jar org.apache.pig.Main -x mapreduce
command, I can atleast see the grunt shell.

However, when using hadoop commands
$ hadoop fs -ls
Found 1 items
-rw-r--r--   1 testpig supergroup     454557 2010-12-09 19:31
/user/testpig/access_log.2010-08-30-23-01.lzo

everything seems to be fine connecting to hdfs.

My environment have the following settings
PIG_HOME=/home/testpig/pig-0.7.0
HADOOP_HOME=/usr/lib/hadoop-0.20    (cloudera distribution)
HADOOP_CONF_DIR=/usr/lib/hadoop-0.20/conf
JAVA_HOME=/usr/java/default

pig-env.sh have the following setting
export PIG_OPTS="$PIG_OPTS
-Djava.library.path=$HADOOP_HOME/lib/native/Linux-amd64-64"
export
PIG_CLASSPATH=$PIG_CLASSPATH:/home/testpig/hadoop-lzo.jar:/home/testpig/elephant-bird.jar:/home/testpig/elephant-bird/lib/*
export PIG_HADOOP_VERSION=20


What is going on there?

Thanks a lot.

Felix 


Mime
View raw message