falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Murali Ramasami (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FALCON-2084) HCatReplicationTest are failing in secure mode
Date Sat, 16 Jul 2016 17:03:20 GMT

     [ https://issues.apache.org/jira/browse/FALCON-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Murali Ramasami updated FALCON-2084:
------------------------------------
    Description: 
HCatReplicationTest are failing in secure mode with the following error in the application
logs:

{noformat}
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw
exception, java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:523)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
        at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:334)
        at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:311)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
        at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1554)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3208)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3227)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
        ... 19 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1552)
        ... 25 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided.
Most recent failure: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
{noformat}

Test case Name:
HCatReplicationTest.oneSourceOneTarget

Console log of test case
{noformat}
2016-07-12 08:50:09,236 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,237 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Try 50 of 50 (InstanceUtil:643)
2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@1 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@2 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@3 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@4 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@5 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@6 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@7 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@8 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@9 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@10 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@11 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@12 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@13 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)

{noformat}


Co-ordinator info:
{noformat}
http://qelog.hortonworks.com/log/nat-r6-ryrs-falcon-multi/test-logs/Falcon-Multicluster/captured_logs/oozie_logs/nat-r6-ryrs-falcon-multi-9.openstacklocal/0000037-160712052909267-oozie-oozi-C-info.log

Coordinator Job ID : 0000037-160712052909267-oozie-oozi-C
--------------------------------------------------------------------------------
Job Name : FALCON_FEED_REPLICATION_Acdd50630-37cacd58_Acdd50630-d3d8d630
App Path : hdfs://nat-r6-ryrs-falcon-multi-9.openstacklocal:8020/tmp/fs/falcon/workflows/feed/Acdd50630-37cacd58/70ca9590153960affad34071740d6f6a_1468311662316/REPLICATION/Acdd50630-d3d8d630/coordinator.xml
Status : KILLED
User : hrt_qa
Started : Fri Jan 01 20:00:00 UTC 2010
EndTime : Thu Jan 01 00:00:00 UTC 2099
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Job ID                                   Status     Ext ID                               
   Err Code   Created                        Nominal Time                  
----------------------------------------------------------------------------------------------------------------------------------------------------------------
0000037-160712052909267-oozie-oozi-C@1   KILLED     0000039-160712052909267-oozie-oozi-W 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 20:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@2   KILLED     0000040-160712052909267-oozie-oozi-W 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 21:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@3   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 22:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@4   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 23:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@5   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 00:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@6   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 01:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@7   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 02:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@8   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 03:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@9   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 04:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@10  KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 05:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@11  KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 06:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@12  KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 07:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@13  KILLED     null                                 
   null       Tue Jul 12 08:24:30 UTC 2016   Sat Jan 02 08:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@14  KILLED     null                                 
   null       Tue Jul 12 08:24:30 UTC 2016   Sat Jan 02 09:00:00 UTC 2010  
----------------------------------------------------------------------------------------------------------------------------------------------------------------
{noformat}


0000039-160712052909267-oozie-oozi-W 

{noformat}

Actions
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Job ID                                                                           Status  
  Ext ID                                   Ext Status      Err Code  
----------------------------------------------------------------------------------------------------------------------------------------------------------------
0000039-160712052909267-oozie-oozi-W@:start:                                     OK      
  -                                        OK              null      
0000039-160712052909267-oozie-oozi-W@pre-processing                              OK      
  job_1468298287018_0041                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@table-export                                OK      
  job_1468296204243_0145                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@replication                                 OK      
  job_1468298287018_0043                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@table-import                                ERROR   
  job_1468298287018_0045                   FAILED/KILLED   JA018     
0000039-160712052909267-oozie-oozi-W@failed-post-processing                      OK      
  job_1468298287018_0046                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@fail                                        OK      
  -                                        OK              E0729     
----------------------------------------------------------------------------------------------------------------------------------------------------------------

{noformat}


  was:
HCatReplicationTest are failing in secure mode with the following error in the application
logs:

{noformat}
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw
exception, java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:523)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
        at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:334)
        at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:311)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
        at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1554)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3208)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3227)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
        ... 19 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1552)
        ... 25 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided.
Most recent failure: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
{noformat}

Test case Name:
HCatReplicationTest.oneSourceOneTarget

Console log of test case
{noformat}
2016-07-12 08:50:09,236 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,237 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Try 50 of 50 (InstanceUtil:643)
2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@1 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@2 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@3 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@4 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@5 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@6 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@7 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@8 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@9 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@10 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@11 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@12 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@13 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)

{noformat}


Co-ordinator info:
{noformat}
http://qelog.hortonworks.com/log/nat-r6-ryrs-falcon-multi/test-logs/Falcon-Multicluster/captured_logs/oozie_logs/nat-r6-ryrs-falcon-multi-9.openstacklocal/0000037-160712052909267-oozie-oozi-C-info.log

Coordinator Job ID : 0000037-160712052909267-oozie-oozi-C
--------------------------------------------------------------------------------
Job Name : FALCON_FEED_REPLICATION_Acdd50630-37cacd58_Acdd50630-d3d8d630
App Path : hdfs://nat-r6-ryrs-falcon-multi-9.openstacklocal:8020/tmp/fs/falcon/workflows/feed/Acdd50630-37cacd58/70ca9590153960affad34071740d6f6a_1468311662316/REPLICATION/Acdd50630-d3d8d630/coordinator.xml
Status : KILLED
User : hrt_qa
Started : Fri Jan 01 20:00:00 UTC 2010
EndTime : Thu Jan 01 00:00:00 UTC 2099
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Job ID                                   Status     Ext ID                               
   Err Code   Created                        Nominal Time                  
----------------------------------------------------------------------------------------------------------------------------------------------------------------
0000037-160712052909267-oozie-oozi-C@1   KILLED     0000039-160712052909267-oozie-oozi-W 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 20:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@2   KILLED     0000040-160712052909267-oozie-oozi-W 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 21:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@3   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 22:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@4   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 23:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@5   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 00:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@6   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 01:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@7   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 02:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@8   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 03:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@9   KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 04:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@10  KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 05:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@11  KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 06:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@12  KILLED     null                                 
   null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 07:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@13  KILLED     null                                 
   null       Tue Jul 12 08:24:30 UTC 2016   Sat Jan 02 08:00:00 UTC 2010  
0000037-160712052909267-oozie-oozi-C@14  KILLED     null                                 
   null       Tue Jul 12 08:24:30 UTC 2016   Sat Jan 02 09:00:00 UTC 2010  
----------------------------------------------------------------------------------------------------------------------------------------------------------------
{noformat}


0000039-160712052909267-oozie-oozi-W 

{noformat}

Actions
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Job ID                                                                           Status  
  Ext ID                                   Ext Status      Err Code  
----------------------------------------------------------------------------------------------------------------------------------------------------------------
0000039-160712052909267-oozie-oozi-W@:start:                                     OK      
  -                                        OK              null      
0000039-160712052909267-oozie-oozi-W@pre-processing                              OK      
  job_1468298287018_0041                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@table-export                                OK      
  job_1468296204243_0145                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@replication                                 OK      
  job_1468298287018_0043                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@table-import                                ERROR   
  job_1468298287018_0045                   FAILED/KILLED   JA018     
0000039-160712052909267-oozie-oozi-W@failed-post-processing                      OK      
  job_1468298287018_0046                   SUCCEEDED       null      
0000039-160712052909267-oozie-oozi-W@fail                                        OK      
  -                                        OK              E0729     
----------------------------------------------------------------------------------------------------------------------------------------------------------------

{noformat}

Attached the failed application log with this jira. 



> HCatReplicationTest are failing in secure mode
> ----------------------------------------------
>
>                 Key: FALCON-2084
>                 URL: https://issues.apache.org/jira/browse/FALCON-2084
>             Project: Falcon
>          Issue Type: Bug
>          Components: replication
>    Affects Versions: trunk
>            Reporter: Murali Ramasami
>            Priority: Critical
>             Fix For: trunk
>
>
> HCatReplicationTest are failing in secure mode with the following error in the application
logs:
> {noformat}
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main()
threw exception, java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>         at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:523)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
>         at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:334)
>         at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:311)
>         at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
>         at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1554)
>         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:89)
>         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:135)
>         at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:107)
>         at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3208)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3227)
>         at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
>         ... 19 more
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>         at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1552)
>         ... 25 more
> Caused by: MetaException(message:Could not connect to meta store using any of the URIs
provided. Most recent failure: org.apache.thrift.transport.TTransportException
>         at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> {noformat}
> Test case Name:
> HCatReplicationTest.oneSourceOneTarget
> Console log of test case
> {noformat}
> 2016-07-12 08:50:09,236 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,237 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Try 50 of 50
(InstanceUtil:643)
> 2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@1 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@2 status is KILLED on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,280 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@3 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@4 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@5 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@6 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@7 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@8 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@9 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@10 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@11 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@12 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@13 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> 2016-07-12 08:50:45,281 INFO  - [pool-34-thread-1:oneSourceOneTarget] ~ Coordinator Action
0000037-160712052909267-oozie-oozi-C@14 status is WAITING on oozie http://nat-r6-ryrs-falcon-multi-10.openstacklocal:11000/oozie/
(InstanceUtil:653)
> {noformat}
> Co-ordinator info:
> {noformat}
> http://qelog.hortonworks.com/log/nat-r6-ryrs-falcon-multi/test-logs/Falcon-Multicluster/captured_logs/oozie_logs/nat-r6-ryrs-falcon-multi-9.openstacklocal/0000037-160712052909267-oozie-oozi-C-info.log
> Coordinator Job ID : 0000037-160712052909267-oozie-oozi-C
> --------------------------------------------------------------------------------
> Job Name : FALCON_FEED_REPLICATION_Acdd50630-37cacd58_Acdd50630-d3d8d630
> App Path : hdfs://nat-r6-ryrs-falcon-multi-9.openstacklocal:8020/tmp/fs/falcon/workflows/feed/Acdd50630-37cacd58/70ca9590153960affad34071740d6f6a_1468311662316/REPLICATION/Acdd50630-d3d8d630/coordinator.xml
> Status : KILLED
> User : hrt_qa
> Started : Fri Jan 01 20:00:00 UTC 2010
> EndTime : Thu Jan 01 00:00:00 UTC 2099
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
> Job ID                                   Status     Ext ID                          
        Err Code   Created                        Nominal Time                  
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
> 0000037-160712052909267-oozie-oozi-C@1   KILLED     0000039-160712052909267-oozie-oozi-W
    null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 20:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@2   KILLED     0000040-160712052909267-oozie-oozi-W
    null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 21:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@3   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 22:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@4   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Fri Jan 01 23:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@5   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 00:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@6   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 01:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@7   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 02:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@8   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 03:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@9   KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 04:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@10  KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 05:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@11  KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 06:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@12  KILLED     null                            
        null       Tue Jul 12 08:21:04 UTC 2016   Sat Jan 02 07:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@13  KILLED     null                            
        null       Tue Jul 12 08:24:30 UTC 2016   Sat Jan 02 08:00:00 UTC 2010  
> 0000037-160712052909267-oozie-oozi-C@14  KILLED     null                            
        null       Tue Jul 12 08:24:30 UTC 2016   Sat Jan 02 09:00:00 UTC 2010  
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
> {noformat}
> 0000039-160712052909267-oozie-oozi-W 
> {noformat}
> Actions
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
> Job ID                                                                           Status
    Ext ID                                   Ext Status      Err Code  
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
> 0000039-160712052909267-oozie-oozi-W@:start:                                     OK 
       -                                        OK              null      
> 0000039-160712052909267-oozie-oozi-W@pre-processing                              OK 
       job_1468298287018_0041                   SUCCEEDED       null      
> 0000039-160712052909267-oozie-oozi-W@table-export                                OK 
       job_1468296204243_0145                   SUCCEEDED       null      
> 0000039-160712052909267-oozie-oozi-W@replication                                 OK 
       job_1468298287018_0043                   SUCCEEDED       null      
> 0000039-160712052909267-oozie-oozi-W@table-import                                ERROR
     job_1468298287018_0045                   FAILED/KILLED   JA018     
> 0000039-160712052909267-oozie-oozi-W@failed-post-processing                      OK 
       job_1468298287018_0046                   SUCCEEDED       null      
> 0000039-160712052909267-oozie-oozi-W@fail                                        OK 
       -                                        OK              E0729     
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Mime
View raw message