Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABoksgYzPs_UcHEwjUfHUvQhmbR4xvYWm2Qbjz_KOJZFdzYcjg@mail.gmail.com>
References: 
 <CABoksgYzPs_UcHEwjUfHUvQhmbR4xvYWm2Qbjz_KOJZFdzYcjg@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Sat, 31 Aug 2013 21:42:27 +0530
Message-ID: 
 <CAOcnVr0EbzaAjcJAi3Bosjjws2SVXL3-rn051CC_oeU5Xhm-aw@mail.gmail.com>
Subject: Re: InvalidProtocolBufferException while submitting crunch job to
 cluster
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Your cluster is using HDFS HA, and therefore requires a little more
configs than just fs.defaultFS/etc..

You need to use the right set of cluster client configs. If you don't
have them at /etc/hadoop/conf and /etc/hbase/conf on your cluster edge
node to pull from, try asking your cluster administrator for a
configuration set, and place their parent directories on your
application's classpath.

The first error deals with perhaps you also including a guava
dependency in your project, which is different than the one
transitively pulled in by hadoop-client via crunch. You should be able
to use guava libs without needing an explicit dependency, and it would
be the right needed version.

The second error deals with your MR submission failing, cause the JT
is using a staging directory over a HDFS HA, which uses a "logical"
name of "bdatadev". A logical HA name needs other configs (typically
in the hdfs-site.xml) that tell it which are the actual physical NNs
under it - configs that you're missing here.

On Sat, Aug 31, 2013 at 1:34 AM, Narlin M <hpnole@gmail.com> wrote:
> I am getting following exception while trying to submit a crunch pipeline
> job to a remote hadoop cluster:
>
> Exception in thread "main" java.lang.RuntimeException: Cannot create job
> output directory /tmp/crunch-324987940
> at
> org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:344)
> at org.apache.crunch.impl.mr.MRPipeline.<init>(MRPipeline.java:125)
> at test.CrunchTest.setup(CrunchTest.java:98)
> at test.CrunchTest.main(CrunchTest.java:367)
> Caused by: java.io.IOException: Failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag.; Host Details : local host is:
> "NARLIN/127.0.0.1"; destination host is: "<server_address>":50070;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759)
> at org.apache.hadoop.ipc.Client.call(Client.java:1164)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:425)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1943)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:523)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1799)
> at
> org.apache.crunch.impl.mr.MRPipeline.createTempDirectory(MRPipeline.java:342)
> ... 3 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
> message end-group tag did not match expected tag.
> at
> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:73)
> at
> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:213)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
> at
> org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985)
> at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:882)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:813)
> 0    [Thread-3] WARN  org.apache.hadoop.util.ShutdownHookManager  -
> ShutdownHook 'ClientFinalizer' failed, java.lang.NoSuchMethodError:
> com.google.common.collect.LinkedListMultimap.values()Ljava/util/List;
> java.lang.NoSuchMethodError:
> com.google.common.collect.LinkedListMultimap.values()Ljava/util/List;
> at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:135)
> at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:672)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:539)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2308)
> at
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2324)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
>
> Google search on this error yielded solutions that asked to confirm that
> /etc/hosts file contained the entry for NARLIN which it does in my case.
>
> Here's the code that I am using to set up the MRPipeline:
>
> Configuration conf = HBaseConfiguration.create();
>
> conf.set("fs.defaultFS", "hdfs://<server_address>:50070");
> conf.set("mapred.job.tracker", "<server_address>:50030");
>
> System.out.println("Hadoop configuration created.");
> System.out.println("Initializing crunch pipeline ...");
>
> conf.set("mapred.jar", "<path_to_jar_file>");
>
> pipeline = new MRPipeline(getClass(), "crunchjobtest", conf);
>
> Has anyone faced this issue before and knows how to resolve it/point out if
> I am missing anything?
>
> Thanks for the help.


-- 
Harsh J