From hdfs-issues-return-237868-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Oct 12 07:11:18 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6C808180660 for ; Fri, 12 Oct 2018 07:11:16 +0200 (CEST) Received: (qmail 10024 invoked by uid 500); 12 Oct 2018 05:11:15 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 10011 invoked by uid 99); 12 Oct 2018 05:11:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Oct 2018 05:11:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 72A63180711 for ; Fri, 12 Oct 2018 05:11:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -108.187 X-Spam-Level: X-Spam-Status: No, score=-108.187 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URI_HEX=1.313, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id YO03R8eT1-RM for ; Fri, 12 Oct 2018 05:11:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id B0C355F3EE for ; Fri, 12 Oct 2018 05:11:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D6FA2E02BE for ; Fri, 12 Oct 2018 05:11:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 955DD24DBC for ; Fri, 12 Oct 2018 05:11:00 +0000 (UTC) Date: Fri, 12 Oct 2018 05:11:00 +0000 (UTC) From: "Bharat Viswanadham (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDDS-624) PutBlock fails with Unexpected Storage Container Exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDDS-624?page=3Dcom.atlassian.j= ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D166474= 83#comment-16647483 ]=20 Bharat Viswanadham commented on HDDS-624: ----------------------------------------- Thank You [~arpitagarwal] for fixing this issue. *Few minor comments:* 1. Container Cache:=C2=A0 @param conf - Ozone Configuration.=C2=A0 (I think it is okay to have Hadoop= Configuration also) 2. TestContainerCache.java: Configuration conf; must be private. This is from checkstyle=C2=A0plugin ru= n. These can be fixed during commit also. =C2=A0 =C2=A0 > PutBlock fails with Unexpected Storage Container Exception > ---------------------------------------------------------- > > Key: HDDS-624 > URL: https://issues.apache.org/jira/browse/HDDS-624 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Reporter: Namit Maheshwari > Assignee: Arpit Agarwal > Priority: Major > Attachments: HDDS-624.01.patch > > > As per HDDS-622, Datanodes were shutting down while running MR jobs due t= o issue in RocksDBStore. To avoid that failure set the property _ozone.meta= store.rocksdb.statistics_ to _OFF_ in ozone-site.xml > Now running Mapreduce job fails with below error > {code:java} > -bash-4.2$ /usr/hdp/current/hadoop-client/bin/hadoop jar /usr/hdp/current= /hadoop-mapreduce-client/hadoop-mapreduce-examples.jar wordcount /tmp/mr_jo= bs/input/ o3://bucket2.volume2/mr_jobAA > 18/10/11 00:14:41 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:14:42 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:14:42 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:14:43 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:14:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing = over to rm2 > 18/10/11 00:14:43 INFO mapreduce.JobResourceUploader: Disabling Erasure C= oding for path: /user/hdfs/.staging/job_1539208750583_0005 > 18/10/11 00:14:43 INFO input.FileInputFormat: Total input files to proces= s : 1 > 18/10/11 00:14:43 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library > 18/10/11 00:14:43 INFO lzo.LzoCodec: Successfully loaded & initialized na= tive-lzo library [hadoop-lzo rev 5d6248d8d690f8456469979213ab2e9993bfa2e9] > 18/10/11 00:14:44 INFO mapreduce.JobSubmitter: number of splits:1 > 18/10/11 00:14:44 INFO mapreduce.JobSubmitter: Submitting tokens for job:= job_1539208750583_0005 > 18/10/11 00:14:44 INFO mapreduce.JobSubmitter: Executing with tokens: [] > 18/10/11 00:14:44 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:14:44 INFO conf.Configuration: found resource resource-types.= xml at file:/etc/hadoop/3.0.3.0-63/0/resource-types.xml > 18/10/11 00:14:44 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:14:44 INFO impl.YarnClientImpl: Submitted application applica= tion_1539208750583_0005 > 18/10/11 00:14:45 INFO mapreduce.Job: The url to track the job: http://ct= r-e138-1518143905142-510793-01-000005.hwx.site:8088/proxy/application_15392= 08750583_0005/ > 18/10/11 00:14:45 INFO mapreduce.Job: Running job: job_1539208750583_0005 > 18/10/11 00:14:53 INFO mapreduce.Job: Job job_1539208750583_0005 running = in uber mode : false > 18/10/11 00:14:53 INFO mapreduce.Job: map 0% reduce 0% > 18/10/11 00:15:00 INFO mapreduce.Job: map 100% reduce 0% > 18/10/11 00:15:10 INFO mapreduce.Job: map 100% reduce 67% > 18/10/11 00:15:11 INFO mapreduce.Job: Task Id : attempt_1539208750583_000= 5_r_000000_0, Status : FAILED > Error: java.io.IOException: Unexpected Storage Container Exception: java.= io.IOException: Failed to command cmdType: PutBlock > traceID: "df0ed956-fa4d-40ef-a7f2-ec0b6160b41b" > containerID: 2 > datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f" > putBlock { > blockData { > blockID { > containerID: 2 > localID: 100874119214399488 > } > metadata { > key: "TYPE" > value: "KEY" > } > chunks { > chunkName: "f24fa36171bda3113584cb01dc12a871_stream_84157b3a-654d-4e3d-84= 55-fbf85321a306_chunk_1" > offset: 0 > len: 5017 > } > } > } > at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutput= Stream.java:171) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputSt= reamEntry.close(ChunkGroupOutputStream.java:699) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOr= Close(ChunkGroupOutputStream.java:502) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGr= oupOutputStream.java:531) > at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStre= am.java:57) > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutp= utStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:= 101) > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWrit= er.close(TextOutputFormat.java:106) > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(Redu= ceTask.java:551) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.io.IOException: Failed to command cmdType: PutBlock > traceID: "df0ed956-fa4d-40ef-a7f2-ec0b6160b41b" > containerID: 2 > datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f" > putBlock { > blockData { > blockID { > containerID: 2 > localID: 100874119214399488 > } > metadata { > key: "TYPE" > value: "KEY" > } > chunks { > chunkName: "f24fa36171bda3113584cb01dc12a871_stream_84157b3a-654d-4e3d-84= 55-fbf85321a306_chunk_1" > offset: 0 > len: 5017 > } > } > } > at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientS= pi.java:103) > at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.putBlock(Con= tainerProtocolCalls.java:158) > at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutput= Stream.java:166) > ... 15 more > Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.prot= ocol.StateMachineException: java.util.concurrent.CompletionException: java.= lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Lja= va/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/m= anagement/ObjectName; > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.jav= a:357) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895= ) > at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientS= pi.java:101) > ... 17 more > Caused by: org.apache.ratis.protocol.StateMachineException: java.util.con= current.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop= .metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/ut= il/Map;Ljava/lang/Object;)Ljavax/management/ObjectName; > at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFutu= re.java:273) > at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFu= ture.java:280) > at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFutu= re.java:1592) > at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFut= ure.java:1582) > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:= 1056) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.jav= a:157) > 18/10/11 00:15:12 INFO mapreduce.Job: map 100% reduce 0% > 18/10/11 00:15:19 INFO mapreduce.Job: Task Id : attempt_1539208750583_000= 5_r_000000_1, Status : FAILED > Error: java.io.IOException: Unexpected Storage Container Exception: java.= io.IOException: Failed to command cmdType: PutBlock > traceID: "742dce10-f784-416a-88de-daff81c3a225" > containerID: 3 > datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f" > putBlock { > blockData { > blockID { > containerID: 3 > localID: 100874119883325441 > } > metadata { > key: "TYPE" > value: "KEY" > } > chunks { > chunkName: "b689c0eb73234bfa6b71d7d6740a0710_stream_521fb062-c983-49df-8f= 28-b4c7ef82acdd_chunk_1" > offset: 0 > len: 5017 > } > } > } > at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutput= Stream.java:171) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputSt= reamEntry.close(ChunkGroupOutputStream.java:699) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOr= Close(ChunkGroupOutputStream.java:502) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGr= oupOutputStream.java:531) > at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStre= am.java:57) > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutp= utStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:= 101) > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWrit= er.close(TextOutputFormat.java:106) > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(Redu= ceTask.java:551) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.io.IOException: Failed to command cmdType: PutBlock > traceID: "742dce10-f784-416a-88de-daff81c3a225" > containerID: 3 > datanodeUuid: "96f8fa78-413e-4350-a8ff-6cbdaa16ba7f" > putBlock { > blockData { > blockID { > containerID: 3 > localID: 100874119883325441 > } > metadata { > key: "TYPE" > value: "KEY" > } > chunks { > chunkName: "b689c0eb73234bfa6b71d7d6740a0710_stream_521fb062-c983-49df-8f= 28-b4c7ef82acdd_chunk_1" > offset: 0 > len: 5017 > } > } > } > at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientS= pi.java:103) > at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.putBlock(Con= tainerProtocolCalls.java:158) > at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutput= Stream.java:166) > ... 15 more > Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.prot= ocol.StateMachineException: java.util.concurrent.CompletionException: java.= lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Lja= va/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/m= anagement/ObjectName; > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.jav= a:357) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895= ) > at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientS= pi.java:101) > ... 17 more > Caused by: org.apache.ratis.protocol.StateMachineException: java.util.con= current.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop= .metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/ut= il/Map;Ljava/lang/Object;)Ljavax/management/ObjectName; > at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFutu= re.java:273) > at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFu= ture.java:280) > at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFutu= re.java:1592) > at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFut= ure.java:1582) > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:= 1056) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.jav= a:157) > 18/10/11 00:15:29 INFO mapreduce.Job: Task Id : attempt_1539208750583_000= 5_r_000000_2, Status : FAILED > Error: java.io.IOException: Unexpected Storage Container Exception: java.= io.IOException: Failed to command cmdType: PutBlock > traceID: "b2b86eba-70e3-40a4-af8c-6807bb63438f" > containerID: 4 > datanodeUuid: "bc3d3379-d410-4a2f-bf45-644d5cf70b15" > putBlock { > blockData { > blockID { > containerID: 4 > localID: 100874120439201794 > } > metadata { > key: "TYPE" > value: "KEY" > } > chunks { > chunkName: "de92ed9710182fb99f1283fb5ed712bd_stream_6241ecb3-1fc6-4f30-bb= dd-c094f2c3ee9d_chunk_1" > offset: 0 > len: 5017 > } > } > } > at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutput= Stream.java:171) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputSt= reamEntry.close(ChunkGroupOutputStream.java:699) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOr= Close(ChunkGroupOutputStream.java:502) > at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGr= oupOutputStream.java:531) > at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStre= am.java:57) > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutp= utStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:= 101) > at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWrit= er.close(TextOutputFormat.java:106) > at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(Redu= ceTask.java:551) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:630) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) > Caused by: java.io.IOException: Failed to command cmdType: PutBlock > traceID: "b2b86eba-70e3-40a4-af8c-6807bb63438f" > containerID: 4 > datanodeUuid: "bc3d3379-d410-4a2f-bf45-644d5cf70b15" > putBlock { > blockData { > blockID { > containerID: 4 > localID: 100874120439201794 > } > metadata { > key: "TYPE" > value: "KEY" > } > chunks { > chunkName: "de92ed9710182fb99f1283fb5ed712bd_stream_6241ecb3-1fc6-4f30-bb= dd-c094f2c3ee9d_chunk_1" > offset: 0 > len: 5017 > } > } > } > at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientS= pi.java:103) > at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.putBlock(Con= tainerProtocolCalls.java:158) > at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutput= Stream.java:166) > ... 15 more > Caused by: java.util.concurrent.ExecutionException: org.apache.ratis.prot= ocol.StateMachineException: java.util.concurrent.CompletionException: java.= lang.NoSuchMethodError: org.apache.hadoop.metrics2.util.MBeans.register(Lja= va/lang/String;Ljava/lang/String;Ljava/util/Map;Ljava/lang/Object;)Ljavax/m= anagement/ObjectName; > at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.jav= a:357) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895= ) > at org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientS= pi.java:101) > ... 17 more > Caused by: org.apache.ratis.protocol.StateMachineException: java.util.con= current.CompletionException: java.lang.NoSuchMethodError: org.apache.hadoop= .metrics2.util.MBeans.register(Ljava/lang/String;Ljava/lang/String;Ljava/ut= il/Map;Ljava/lang/Object;)Ljavax/management/ObjectName; > at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFutu= re.java:273) > at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFu= ture.java:280) > at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFutu= re.java:1592) > at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFut= ure.java:1582) > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) > at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:= 1056) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) > at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.jav= a:157) > 18/10/11 00:15:39 INFO mapreduce.Job: map 100% reduce 100% > 18/10/11 00:15:39 INFO mapreduce.Job: Job job_1539208750583_0005 failed w= ith state FAILED due to: Task failed task_1539208750583_0005_r_000000 > Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 kil= ledReduces: 0 > 18/10/11 00:15:39 INFO conf.Configuration: Removed undeclared tags: > 18/10/11 00:15:39 INFO mapreduce.Job: Counters: 45 > File System Counters > FILE: Number of bytes read=3D0 > FILE: Number of bytes written=3D266140 > FILE: Number of read operations=3D0 > FILE: Number of large read operations=3D0 > FILE: Number of write operations=3D0 > HDFS: Number of bytes read=3D215876 > HDFS: Number of bytes written=3D0 > HDFS: Number of read operations=3D2 > HDFS: Number of large read operations=3D0 > HDFS: Number of write operations=3D0 > O3: Number of bytes read=3D0 > O3: Number of bytes written=3D0 > O3: Number of read operations=3D0 > O3: Number of large read operations=3D0 > O3: Number of write operations=3D0 > Job Counters > Failed reduce tasks=3D4 > Launched map tasks=3D1 > Launched reduce tasks=3D4 > Rack-local map tasks=3D1 > Total time spent by all maps in occupied slots (ms)=3D18436 > Total time spent by all reduces in occupied slots (ms)=3D233304 > Total time spent by all map tasks (ms)=3D4609 > Total time spent by all reduce tasks (ms)=3D29163 > Total vcore-milliseconds taken by all map tasks=3D4609 > Total vcore-milliseconds taken by all reduce tasks=3D29163 > Total megabyte-milliseconds taken by all map tasks=3D18878464 > Total megabyte-milliseconds taken by all reduce tasks=3D238903296 > Map-Reduce Framework > Map input records=3D716 > Map output records=3D32019 > Map output bytes=3D343475 > Map output materialized bytes=3D6332 > Input split bytes=3D121 > Combine input records=3D32019 > Combine output records=3D461 > Spilled Records=3D461 > Failed Shuffles=3D0 > Merged Map outputs=3D0 > GC time elapsed (ms)=3D87 > CPU time spent (ms)=3D3370 > Physical memory (bytes) snapshot=3D2525511680 > Virtual memory (bytes) snapshot=3D5400559616 > Total committed heap usage (bytes)=3D2711617536 > Peak Map Physical memory (bytes)=3D2525511680 > Peak Map Virtual memory (bytes)=3D5400559616 > File Input Format Counters > Bytes Read=3D215755 > 18/10/11 00:15:39 INFO conf.Configuration: Removed undeclared tags: > -bash-4.2$ > {code} > SCM logs > {code:java} > 2018-10-11 00:12:13,119 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: c14ab941-71ac-4179-899e-a5f9ed04b5f9 Registered. > 2018-10-11 00:12:13,123 INFO org.apache.hadoop.hdds.scm.server.SCMChillMo= deManager: SCM in chill mode. 1 DataNodes registered, 1 required. > 2018-10-11 00:12:13,123 INFO org.apache.hadoop.hdds.scm.server.SCMChillMo= deManager: SCM exiting chill mode. > 2018-10-11 00:12:13,363 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: 96f8fa78-413e-4350-a8ff-6cbdaa16ba7f Registered. > 2018-10-11 00:12:13,716 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: ad72f259-a17f-4961-ada5-530f8d9daf49 Registered. > 2018-10-11 00:12:15,682 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: a5ddc709-dbbe-4101-8c23-50bc9c1655b9 Registered. > 2018-10-11 00:12:16,270 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: cff7e449-f2ba-46e1-9b76-db177ca246a4 Registered. > 2018-10-11 00:12:16,939 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: bc3d3379-d410-4a2f-bf45-644d5cf70b15 Registered. > 2018-10-11 00:12:18,581 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: 9f4a4ee6-8651-425c-8314-efcb4363afc4 Registered. > 2018-10-11 00:12:19,836 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManag= er: Data node with ID: 97f3cb73-736f-425e-9017-d99cd9c12356 Registered. > 2018-10-11 00:15:04,148 INFO org.apache.hadoop.hdds.scm.pipelines.ratis.R= atisManagerImpl: Allocating a new ratis pipeline of size: 3 id: pipelineId= =3D3fcf62c2-1159-4269-9018-5a478db25c4a > 2018-10-11 00:15:06,239 INFO org.apache.hadoop.hdds.scm.pipelines.ratis.R= atisManagerImpl: Allocating a new ratis pipeline of size: 3 id: pipelineId= =3D57002d7d-b866-4f3c-87e9-2a985840020d > 2018-10-11 00:15:10,176 INFO org.apache.hadoop.hdds.scm.server.SCMClientP= rotocolServer: Object type container id 2 op create new stage complete > 2018-10-11 00:15:18,493 INFO org.apache.hadoop.hdds.scm.server.SCMClientP= rotocolServer: Object type container id 3 op create new stage complete > 2018-10-11 00:15:28,080 INFO org.apache.hadoop.hdds.scm.server.SCMClientP= rotocolServer: Object type container id 4 op create new stage complete > 2018-10-11 00:15:37,426 INFO org.apache.hadoop.hdds.scm.server.SCMClientP= rotocolServer: Object type container id 5 op create new stage complete > {code} > OM logs > {code:java} > 2018-10-11 00:14:43,161 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:43,181 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:51,947 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:51,964 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:52,029 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:52,033 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:52,042 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:52,045 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:58,819 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_m_000000_0 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:14:58,833 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_m_000000_0/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,043 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0/part-r-00000 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,051 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0/part-r-00000/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,055 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,060 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,063 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,066 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,096 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,098 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:04,107 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:10,877 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_0 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,094 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1/part-r-00000 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,103 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1/part-r-00000/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,106 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,109 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,111 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,113 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:17,147 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:18,817 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_1 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,564 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2/part-r-00000 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,576 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2/part-r-00000/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,578 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,580 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,583 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,585 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:25,625 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:29,121 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_2 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:35,996 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3/part-r-00000 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:36,005 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3/part-r-00000/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:36,008 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:36,010 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:36,012 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:36,015 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3/ > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:36,047 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:37,728 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary/1/= _temporary/attempt_1539208750583_0005_r_000000_3 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-10-11 00:15:37,783 ERROR org.apache.hadoop.ozone.om.KeyManagerImpl: = Get key failed for volume:volume2 bucket:bucket2 key:mr_jobAA/_temporary > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.jav= a:368) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:88= 1) > at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTrans= latorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:396) > at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$Ozon= eManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:38315) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call= (ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat= ion.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org