hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-624) PutBlock fails with Unexpected Storage Container Exception
Date Fri, 12 Oct 2018 00:16:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647208#comment-16647208 ] 

Arpit Agarwal commented on HDDS-624:
------------------------------------

I think I see the problem. MetadataStoreBuilder starts off with a null configuration object unless the caller initializes it via {{setConfiguration}}. 

ContainerCache.getDB doesn't do so. Hence the user's configuration is always ignored and RocksDB statistics are always enabled even when _ozone.metastore.rocksdb.statistics_ is _OFF_.

HDDS-636 will fix this indirectly by changing the default, but we should fix this also.

> PutBlock fails with Unexpected Storage Container Exception
> ----------------------------------------------------------
>
>                 Key: HDDS-624
>                 URL: https://issues.apache.org/jira/browse/HDDS-624
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>            Reporter: Namit Maheshwari
>            Priority: Major
>
> As per HDDS-622, Datanodes were shutting down while running MR jobs due to issue in RocksDBStore. To avoid that failure set the property _ozone.metastore.rocksdb.statistics_ to _OFF_ in ozone-site.xml
> Now running Mapreduce job fails with below error
> {code:java}
> -bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /tmp/mr_jobs/input/ o3://bucket2.volume2/mr_jobAA
> 18/10/11 00:14:41 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:14:42 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:14:42 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:14:43 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:14:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> 18/10/11 00:14:43 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1539208750583_0005
> 18/10/11 00:14:43 INFO input.FileInputFormat: Total input files to process : 1
> 18/10/11 00:14:43 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 18/10/11 00:14:43 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9]
> 18/10/11 00:14:44 INFO mapreduce.JobSubmitter: number of splits:1
> 18/10/11 00:14:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1539208750583_0005
> 18/10/11 00:14:44 INFO mapreduce.JobSubmitter: Executing with tokens: []
> 18/10/11 00:14:44 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:14:44 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml
> 18/10/11 00:14:44 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:14:44 INFO impl.YarnClientImpl: Submitted application application_1539208750583_0005
> 18/10/11 00:14:45 INFO mapreduce.Job: The url to track the job: http://ctr-e138-1518143905142-510793-01-000005.hwx.site:8088/proxy/application_1539208750583_0005/
> 18/10/11 00:14:45 INFO mapreduce.Job: Running job: job_1539208750583_0005
> 18/10/11 00:14:53 INFO mapreduce.Job: Job job_1539208750583_0005 running in uber mode : false
> 18/10/11 00:14:53 INFO mapreduce.Job: map 0% reduce 0%
> 18/10/11 00:15:00 INFO mapreduce.Job: map 100% reduce 0%
> 18/10/11 00:15:10 INFO mapreduce.Job: map 100% reduce 67%
> 18/10/11 00:15:11 INFO mapreduce.Job: Task Id : attempt_1539208750583_0005_r_000000_0, Status : FAILED
> Error: java.io.IOException: Unexpected Storage Container Exception: java.io.IOException: Failed to command cmdType: PutBlock
> traceID: "df0ed956-fa4d-40ef-a7f2-ec0b6160b41b"
> containerID: 2
> datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f"
> putBlock {
> blockData {
> blockID {
> containerID: 2
> localID: 100874119214399488
> }
> metadata {
> key: "TYPE"
> value: "KEY"
> }
> chunks {
> chunkName: "f24fa36171bda3113584cb01dc12a871_stream_84157b3a-654d-4e3d-8455-fbf85321a306_chunk_1"
> offset: 0
> len: 5017
> }
> }
> }
> at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:171)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:699)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:502)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:531)
> at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:57)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:551)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: java.io.IOException: Failed to command cmdType: PutBlock
> traceID: "df0ed956-fa4d-40ef-a7f2-ec0b6160b41b"
> containerID: 2
> datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f"
> putBlock {
> blockData {
> blockID {
> containerID: 2
> localID: 100874119214399488
> }
> metadata {
> key: "TYPE"
> value: "KEY"
> }
> chunks {
> chunkName: "f24fa36171bda3113584cb01dc12a871_stream_84157b3a-654d-4e3d-8455-fbf85321a306_chunk_1"
> offset: 0
> len: 5017
> }
> }
> }
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:103)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.putBlock(ContainerProtocolCalls.java:158)
> at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:166)
> ... 15 more
> Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.protocol.StateMachineException: java.util.concurrent.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:101)
> ... 17 more
> Caused by: org.apache.ratis.protocol.StateMachineException: java.util.concurrent.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
> at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> 18/10/11 00:15:12 INFO mapreduce.Job: map 100% reduce 0%
> 18/10/11 00:15:19 INFO mapreduce.Job: Task Id : attempt_1539208750583_0005_r_000000_1, Status : FAILED
> Error: java.io.IOException: Unexpected Storage Container Exception: java.io.IOException: Failed to command cmdType: PutBlock
> traceID: "742dce10-f784-416a-88de-daff81c3a225"
> containerID: 3
> datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f"
> putBlock {
> blockData {
> blockID {
> containerID: 3
> localID: 100874119883325441
> }
> metadata {
> key: "TYPE"
> value: "KEY"
> }
> chunks {
> chunkName: "b689c0eb73234bfa6b71d7d6740a0710_stream_521fb062-c983-49df-8f28-b4c7ef82acdd_chunk_1"
> offset: 0
> len: 5017
> }
> }
> }
> at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:171)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:699)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:502)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:531)
> at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:57)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:551)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: java.io.IOException: Failed to command cmdType: PutBlock
> traceID: "742dce10-f784-416a-88de-daff81c3a225"
> containerID: 3
> datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f"
> putBlock {
> blockData {
> blockID {
> containerID: 3
> localID: 100874119883325441
> }
> metadata {
> key: "TYPE"
> value: "KEY"
> }
> chunks {
> chunkName: "b689c0eb73234bfa6b71d7d6740a0710_stream_521fb062-c983-49df-8f28-b4c7ef82acdd_chunk_1"
> offset: 0
> len: 5017
> }
> }
> }
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:103)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.putBlock(ContainerProtocolCalls.java:158)
> at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:166)
> ... 15 more
> Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.protocol.StateMachineException: java.util.concurrent.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:101)
> ... 17 more
> Caused by: org.apache.ratis.protocol.StateMachineException: java.util.concurrent.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
> at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> 18/10/11 00:15:29 INFO mapreduce.Job: Task Id : attempt_1539208750583_0005_r_000000_2, Status : FAILED
> Error: java.io.IOException: Unexpected Storage Container Exception: java.io.IOException: Failed to command cmdType: PutBlock
> traceID: "b2b86eba-70e3-40a4-af8c-6807bb63438f"
> containerID: 4
> datanodeUuid: "bc3d3379-d410-4a2f-bf45-644d5cf70b15"
> putBlock {
> blockData {
> blockID {
> containerID: 4
> localID: 100874120439201794
> }
> metadata {
> key: "TYPE"
> value: "KEY"
> }
> chunks {
> chunkName: "de92ed9710182fb99f1283fb5ed712bd_stream_6241ecb3-1fc6-4f30-bbdd-c094f2c3ee9d_chunk_1"
> offset: 0
> len: 5017
> }
> }
> }
> at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:171)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:699)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:502)
> at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:531)
> at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:57)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
> at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:551)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: java.io.IOException: Failed to command cmdType: PutBlock
> traceID: "b2b86eba-70e3-40a4-af8c-6807bb63438f"
> containerID: 4
> datanodeUuid: "bc3d3379-d410-4a2f-bf45-644d5cf70b15"
> putBlock {
> blockData {
> blockID {
> containerID: 4
> localID: 100874120439201794
> }
> metadata {
> key: "TYPE"
> value: "KEY"
> }
> chunks {
> chunkName: "de92ed9710182fb99f1283fb5ed712bd_stream_6241ecb3-1fc6-4f30-bbdd-c094f2c3ee9d_chunk_1"
> offset: 0
> len: 5017
> }
> }
> }
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:103)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.putBlock(ContainerProtocolCalls.java:158)
> at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:166)
> ... 15 more
> Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.protocol.StateMachineException: java.util.concurrent.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
> at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:101)
> ... 17 more
> Caused by: org.apache.ratis.protocol.StateMachineException: java.util.concurrent.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/management/ObjectName;
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
> at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
> 18/10/11 00:15:39 INFO mapreduce.Job: map 100% reduce 100%
> 18/10/11 00:15:39 INFO mapreduce.Job: Job job_1539208750583_0005 failed with state FAILED due to: Task failed task_1539208750583_0005_r_000000
> Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 killedReduces: 0
> 18/10/11 00:15:39 INFO conf.Configuration: Removed undeclared tags:
> 18/10/11 00:15:39 INFO mapreduce.Job: Counters: 45
> File System Counters
> FILE: Number of bytes read=0
> FILE: Number of bytes written=266140
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=215876
> HDFS: Number of bytes written=0
> HDFS: Number of read operations=2
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=0
> O3: Number of bytes read=0
> O3: Number of bytes written=0
> O3: Number of read operations=0
> O3: Number of large read operations=0
> O3: Number of write operations=0
> Job Counters
> Failed reduce tasks=4
> Launched map tasks=1
> Launched reduce tasks=4
> Rack-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=18436
> Total time spent by all reduces in occupied slots (ms)=233304
> Total time spent by all map tasks (ms)=4609
> Total time spent by all reduce tasks (ms)=29163
> Total vcore-milliseconds taken by all map tasks=4609
> Total vcore-milliseconds taken by all reduce tasks=29163
> Total megabyte-milliseconds taken by all map tasks=18878464
> Total megabyte-milliseconds taken by all reduce tasks=238903296
> Map-Reduce Framework
> Map input records=716
> Map output records=32019
> Map output bytes=343475
> Map output materialized bytes=6332
> Input split bytes=121
> Combine input records=32019
> Combine output records=461
> Spilled Records=461
> Failed Shuffles=0
> Merged Map outputs=0
> GC time elapsed (ms)=87
> CPU time spent (ms)=3370
> Physical memory (bytes) snapshot=2525511680
> Virtual memory (bytes) snapshot=5400559616
> Total committed heap usage (bytes)=2711617536
> Peak Map Physical memory (bytes)=2525511680
> Peak Map Virtual memory (bytes)=5400559616
> File Input Format Counters
> Bytes Read=215755
> 18/10/11 00:15:39 INFO conf.Configuration: Removed undeclared tags:
> -bash-4.2$
> {code}
> SCM logs
> {code:java}
> 2018-10-11 00:12:13,119 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: c14ab941-71ac-4179-899e-a5f9ed04b5f9 Registered.
> 2018-10-11 00:12:13,123 INFO org.apache.hadoop.hdds.scm.server.SCMChillModeManager: SCM in chill mode. 1 DataNodes registered, 1 required.
> 2018-10-11 00:12:13,123 INFO org.apache.hadoop.hdds.scm.server.SCMChillModeManager: SCM exiting chill mode.
> 2018-10-11 00:12:13,363 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 96f8fa78-413e-4350-a8ff-6cbdaa16ba7f Registered.
> 2018-10-11 00:12:13,716 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: ad72f259-a17f-4961-ada5-530f8d9daf49 Registered.
> 2018-10-11 00:12:15,682 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: a5ddc709-dbbe-4101-8c23-50bc9c1655b9 Registered.
> 2018-10-11 00:12:16,270 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: cff7e449-f2ba-46e1-9b76-db177ca246a4 Registered.
> 2018-10-11 00:12:16,939 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: bc3d3379-d410-4a2f-bf45-644d5cf70b15 Registered.
> 2018-10-11 00:12:18,581 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 9f4a4ee6-8651-425c-8314-efcb4363afc4 Registered.
> 2018-10-11 00:12:19,836 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 97f3cb73-736f-425e-9017-d99cd9c12356 Registered.
> 2018-10-11 00:15:04,148 INFO org.apache.hadoop.hdds.scm.pipelines.ratis.RatisManagerImpl: Allocating a new ratis pipeline of size: 3 id: pipelineId=3fcf62c2-1159-4269-9018-5a478db25c4a
> 2018-10-11 00:15:06,239 INFO org.apache.hadoop.hdds.scm.pipelines.ratis.RatisManagerImpl: Allocating a new ratis pipeline of size: 3 id: pipelineId=57002d7d-b866-4f3c-87e9-2a985840020d
> 2018-10-11 00:15:10,176 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: Object type container id 2 op create new stage complete
> 2018-10-11 00:15:18,493 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: Object type container id 3 op create new stage complete
> 2018-10-11 00:15:28,080 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: Object type container id 4 op create new stage complete
> 2018-10-11 00:15:37,426 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: Object type container id 5 op create new stage complete
> {code}
> OM logs
> {code:java}
> 2018-10-11 00:14:43,161 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:43,181 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:51,947 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:51,964 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:52,029 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:52,033 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:52,042 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:52,045 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:58,819 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_m_000000_0
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:14:58,833 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_m_000000_0/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,043 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0/part-r-00000
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,051 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0/part-r-00000/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,055 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,060 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,063 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,066 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,096 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,098 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:04,107 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:10,877 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_0
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,094 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1/part-r-00000
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,103 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1/part-r-00000/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,106 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,109 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,111 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,113 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:17,147 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:18,817 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_1
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,564 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2/part-r-00000
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,576 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2/part-r-00000/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,578 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,580 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,583 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,585 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:25,625 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:29,121 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_2
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:35,996 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3/part-r-00000
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:36,005 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3/part-r-00000/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:36,008 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:36,010 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:36,012 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:36,015 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3/
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:36,047 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:37,728 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/_temporary/attempt_1539208750583_0005_r_000000_3
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> 2018-10-11 00:15:37,783 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary
> org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
> at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:368)
> at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:881)
> at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396)
> at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message