Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of vseetharam@gmail.com designates
 209.85.192.53 as permitted sender)
MIME-Version: 1.0
Sender: vseetharam@gmail.com
In-Reply-To: 
 <CAJQpoBSDWGxdcOYnDTbtHfVzaG5LGkwzrt6KQGbknASqN7obDw@mail.gmail.com>
References: 
 <CAJQpoBQ+r7JAmBpvoju+p6i4a-=ktqjUU=Wo8XFpWNPZWxgthg@mail.gmail.com>
	<CAFfn_Rr05fS-+JXJgySyyRym8snKejKUH=FR226nphTGPU_h-Q@mail.gmail.com>
	<CAJQpoBSDWGxdcOYnDTbtHfVzaG5LGkwzrt6KQGbknASqN7obDw@mail.gmail.com>
Date: Fri, 4 Jul 2014 11:45:08 +0530
Message-ID: 
 <CAFfn_RohBHwHU9_pcb=c15oRb0yzn-xvRL_zPuX5rjj_fHx0yA@mail.gmail.com>
Subject: Re: Regarding Falcon+Oozie+Hcat table replication
From: Seetharam Venkatesh <venkatesh@innerzeal.com>
To: "dev@falcon.incubator.apache.org" <dev@falcon.incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3bfde1698e004fd580c8e

--001a11c3bfde1698e004fd580c8e
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I think I know this issue. Sorry did not see the error first time around.
This is documented but since 0.5 is not out yet, is in the source. You
should follow the instructions below and this error should go away.

   * Oozie 4.x with Hadoop-2.x
Replication jobs are submitted to oozie on the destination cluster. Oozie
runs a table export job
on RM on source cluster. Oozie server on the target cluster must be
configured with source hadoop
configs else jobs fail with errors on secure and non-secure clusters as
below:
<verbatim>
org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not
found for ApplicationAttempt appattempt_1395965672651_0010_000002
</verbatim>

Make sure all oozie servers that falcon talks to has the hadoop configs
configured in oozie-site.xml
<verbatim>
<property>
      <name>oozie.service.HadoopAccessorService.hadoop.configurations</name=
>

<value>*=3D/etc/hadoop/conf,arpit-new-falcon-1.cs1cloud.internal:8020=3D/et=
c/hadoop-1,arpit-new-falcon-1.cs1cloud.internal:8032=3D/etc/hadoop-1,arpit-=
new-falcon-2.cs1cloud.internal:8020=3D/etc/hadoop-2,arpit-new-falcon-2.cs1c=
loud.internal:8032=3D/etc/hadoop-2,arpit-new-falcon-5.cs1cloud.internal:802=
0=3D/etc/hadoop-3,arpit-new-falcon-5.cs1cloud.internal:8032=3D/etc/hadoop-3=
</value>
      <description>
          Comma separated AUTHORITY=3DHADOOP_CONF_DIR, where AUTHORITY is t=
he
HOST:PORT of
          the Hadoop service (JobTracker, HDFS). The wildcard '*'
configuration is
          used when there is no exact match for an authority. The
HADOOP_CONF_DIR contains
          the relevant Hadoop *-site.xml files. If the path is relative is
looked within
          the Oozie configuration directory; though the path can be
absolute (i.e. to point
          to Hadoop client conf/ directories in the local filesystem.
      </description>
    </property>
</verbatim>


On Thu, Jul 3, 2014 at 8:23 AM, John Yu <johnyu0520@gmail.com> wrote:

> Hey Seetharam,
>
> Thanks for your suggestion.  So I have done the following:
>
> 1. I've put hive-site.xml into oozie/conf/hadoop-conf
> 2. I've made sure  hive.metastore.execute.setugi is true (it is true by
> default actually)
> 3. I've copied the source cluster hadoop xml's into oozie/conf/hadoop-con=
f
> per
>
> https://git-wip-us.apache.org/repos/asf?p=3Dincubator-falcon.git;a=3Dblob=
;f=3Ddocs/src/site/twiki/HiveIntegration.twiki;h=3D2af5a6b8351c09c60d735314=
70b959e801f19b97;hb=3DHEAD
>
> so far it is still complaining for the same error.
>
> I am thinking on running through the hive integration documentation page
> from scratch again to see if it changes things.
>
>
> Thanks,
> John
>
>
> 2014-06-30 10:05 GMT-07:00 Seetharam Venkatesh <venkatesh@innerzeal.com>:
>
> > I'm not sure if you have the latest sandbox but you'd need to add
> > hive-site.xml in oozie and have hive.metastore.execute.setugi set to tr=
ue
> > for this to work.
> >
> > This has been fixed in the next patch release.
> >
> >
> > On Fri, Jun 27, 2014 at 11:50 PM, John Yu <johnyu0520@gmail.com> wrote:
> >
> > > Hello Everyone,
> > >
> > > I am trying to get a Hcat based feed replication going between
> clusters,
> > > and is running into the below error (full stack trace and related
> > > information will be provided below)
> > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception
> > while
> > > registering
> > >
> > > org.apache.hadoop.security.token.SecretManager$InvalidToken: Password
> > > not found for ApplicationAttempt appattempt_1403829424249_0002_000001
> > >
> > >
> > > Wondering if someone can provide insights into what might be happenin=
g
> > > here, or where to start trying to resolve this issue.  Any pointers
> would
> > > be greatly appreciated.
> > >
> > > Thanks!
> > > John
> > >
> > >
> > > --- System ---
> > > Hortonworks Sandbox 2.1
> > > Hadoop 2.4.0
> > > Hive 0.13.0
> > > Falcon 0.5
> > > Oozie 4.0.0
> > >
> > > The setup is between two HDP 2.1 sandboxes hosted on different boxes
> > >
> > >
> > > --- Hive site.xml ---
> > >
> > > (are these the only two properties I have to worry about?  )
> > >
> > >     <property>
> > >
> > >       <name>hive.security.authorization.enabled</name>
> > >
> > >       <value>false (originally true) </value>
> > >
> > >     </property>
> > >
> > >     <property>
> > >
> > >       <name>hive.server2.enable.doAs</name>
> > >
> > >       <value>true (originally false) </value>
> > >
> > >     </property>
> > >
> > > ---Error Stacktrace
> > >
> > > 2014-06-26 17:52:58,418 WARN [main] org.apache.hadoop.ipc.Client:
> > > Exception encountered while connecting to the server :
> > >
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Se=
cretManager$InvalidToken):
> > > Password not found for ApplicationAttempt
> > > appattempt_1403829424249_0002_000001
> > > 2014-06-26 17:52:58,425 ERROR [main]
> > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception
> > > while registering
> > > org.apache.hadoop.security.token.SecretManager$InvalidToken: Password
> > > not found for ApplicationAttempt appattempt_1403829424249_0002_000001
> > >         at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > >         at
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc=
cessorImpl.java:57)
> > >         at
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst=
ructorAccessorImpl.java:45)
> > >         at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > >         at
> > >
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> > >         at
> > >
> >
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:1=
04)
> > >         at
> > >
> >
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie=
ntImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java=
:109)
> > >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method=
)
> > >         at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:57)
> > >         at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:43)
> > >         at java.lang.reflect.Method.invoke(Method.java:606)
> > >         at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc=
ationHandler.java:190)
> > >         at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH=
andler.java:103)
> > >         at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown
> > Source)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunica=
tor.java:155)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommu=
nicator.java:116)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(R=
MContainerAllocator.java:213)
> > >         at
> > >
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.s=
erviceStart(MRAppMaster.java:817)
> > >         at
> > >
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> > >         at
> > >
> >
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.=
java:120)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.j=
ava:1075)
> > >         at
> > >
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:146=
0)
> > >         at java.security.AccessController.doPrivileged(Native Method)
> > >         at javax.security.auth.Subject.doAs(Subject.java:415)
> > >         at
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation=
.java:1557)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAp=
pMaster.java:1456)
> > >         at
> > >
> >
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1389=
)
> > > Caused by:
> > >
> >
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.Se=
cretManager$InvalidToken):
> > > Password not found for ApplicationAttempt
> > > appattempt_1403829424249_0002_000001
> > >         at org.apache.hadoop.ipc.Client.call(Client.java:1410)
> > >         at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> > >         at
> > >
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.=
java:206)
> > >         at com.sun.proxy.$Proxy34.registerApplicationMaster(Unknown
> > Source)
> > >         at
> > >
> >
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie=
ntImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java=
:106)
> > >
> > >
> > >
> > > --
> > > =E4=BD=99=E5=AE=88=E4=B8=AD  John Yu (Yu, Shoou-Jong)
> > > Mobile: 650-691-3314
> > >
> >
> >
> >
> > --
> > Regards,
> > Venkatesh
> >
> > =E2=80=9CPerfection (in design) is achieved not when there is nothing m=
ore to
> add,
> > but rather when there is nothing more to take away.=E2=80=9D
> > - Antoine de Saint-Exup=C3=A9ry
> >
>
>
>
> --
> =E4=BD=99=E5=AE=88=E4=B8=AD  John Yu (Yu, Shoou-Jong)
> Mobile: 650-691-3314
>


--=20
Regards,
Venkatesh

=E2=80=9CPerfection (in design) is achieved not when there is nothing more =
to add,
but rather when there is nothing more to take away.=E2=80=9D
- Antoine de Saint-Exup=C3=A9ry

--001a11c3bfde1698e004fd580c8e--