Mailing-List: contact dev-help@samza.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@samza.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAHcwxUTnNB_uScYjA56vrr09PKED97aad=7BXAq0uF1SWEMBew@mail.gmail.com>
References: 
 <CAHcwxUQRtb9QH9cwOAPAZMNJL3tAaCGrL5trL443-b0kLQ_D_A@mail.gmail.com>
	<CAFvExu1L-Y9qY1wpAwGBRZdTpAB4QfrTQ72ia9CZJ-cR3PB1Fg@mail.gmail.com>
	<CAHcwxUTnNB_uScYjA56vrr09PKED97aad=7BXAq0uF1SWEMBew@mail.gmail.com>
Date: Wed, 6 Jan 2016 12:18:39 -0800
Message-ID: 
 <CAFvExu0SAz34WasQtwe8=rentxix0DdfBDbcCx3vC+BQqExMyQ@mail.gmail.com>
Subject: Re: hello-samza doesn't run on CDH5.4
From: Yi Pan <nickpan47@gmail.com>
To: dev@samza.apache.org
Content-Type: multipart/alternative; boundary=001a1140bfd84e60140528b00fc5

--001a1140bfd84e60140528b00fc5
Content-Type: text/plain; charset=UTF-8

Token expiration is still a problem w/o Kerberos in YARN 2.6.0 client. But
the exception stack trace from that bug is also different from yours. I am
suspecting there is some incompatible dependency libraries between CDH5.4
distro and Samza that caused your serialization problems here.

On Wed, Jan 6, 2016 at 11:50 AM, Stanislav Los <stanislav@magnetic.com>
wrote:

> Hi Yi,
>
> Our cluster doesn't have Kerberos security in place, so I don't know how it
> could be related.
> CDH5.4 runs on YARN 2.6.0.
> I compiled Samza with YARN 2.6.0 and YARN 2.6.1.
> In both cases I get the same error.
> Should token expiration be a problem for the cluster without Kerberos?
>
> Regards.
>
> On Wed, Jan 6, 2016 at 2:39 PM, Yi Pan <nickpan47@gmail.com> wrote:
>
> > Hi, Stanislav,
> >
> > Could it be related to SAMZA-727? Your exception stack trace is different
> > though. Besides that, Samza 0.10 requires minimum YARN 2.6.1 now, due to
> a
> > token expiration issue in YARN 2.6.0 client lib.
> >
> > -Yi
> >
> > On Wed, Jan 6, 2016 at 7:52 AM, Stanislav Los <stanislav@magnetic.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Any ideas why Samza can't start a container while running hello-samza
> on
> > > CDH5.4?
> > >
> > > It runs fine on local grid. But when running on cluster I can see it
> > talks
> > > fine to YARN's  ResourceManager and Kafka, get's resources, but fails
> to
> > > start container. At first, I thought it's because CDH5.4 uses YARN
> 2.6.0,
> > > while Samza uses 2.6.1, so I recompiled Samza with YARN 2.6.0 and
> > > re-assambled hello-samza. No luck so far. Also, I thought it's because
> of
> > > different protobuf version, but I checked, it's the same in CDH and
> > Samza.
> > >
> > > Followed
> > > http://samza.apache.org/learn/tutorials/0.10/deploy-samza-to-CDH.html
> to
> > > deploy.
> > >
> > > See full logs:
> > > Samza Deploy Script output http://pastebin.com/0K8HVMzt
> > > YARN logs http://pastebin.com/UvNp6240
> > >
> > > An error is:
> > >
> > > java version "1.7.0_80"
> > >
> > > Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
> > >
> > > Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
> > >
> > > com.google.protobuf.InvalidProtocolBufferException: Protocol message
> > > contained an invalid tag (zero).
> > >
> > > at
> > >
> > >
> >
> com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:89)
> > >
> > > at
> > com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:108)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1078)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.<init>(YarnSecurityTokenProtos.java:1042)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1138)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto$1.parsePartialFrom(YarnSecurityTokenProtos.java:1133)
> > >
> > > at
> > >
> > >
> >
> com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
> > >
> > > at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
> > >
> > > at
> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
> > >
> > > at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$AMRMTokenIdentifierProto.parseFrom(YarnSecurityTokenProtos.java:1307)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.yarn.security.AMRMTokenIdentifier.readFields(AMRMTokenIdentifier.java:81)
> > >
> > > at
> > org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:142)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:181)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > Exception in thread "Container Allocator Thread"
> > > org.apache.samza.SamzaException: IO Exception when writing credentials
> to
> > > output buffer
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.startContainer(ContainerUtil.java:190)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerUtil.runContainer(ContainerUtil.java:119)
> > >
> > > at
> > >
> > >
> >
> org.apache.samza.job.yarn.ContainerAllocator.run(ContainerAllocator.java:65)
> > >
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > Regards,
> > > Stan
> > >
> >
>

--001a1140bfd84e60140528b00fc5--