From issues-return-334616-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Thu Feb 22 01:37:08 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9631418061A for ; Thu, 22 Feb 2018 01:37:07 +0100 (CET) Received: (qmail 20673 invoked by uid 500); 22 Feb 2018 00:37:06 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 20662 invoked by uid 99); 22 Feb 2018 00:37:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Feb 2018 00:37:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E0A6F1805BD for ; Thu, 22 Feb 2018 00:37:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.511 X-Spam-Level: X-Spam-Status: No, score=-109.511 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 5oFwUWsD_rfY for ; Thu, 22 Feb 2018 00:37:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 933A05F58D for ; Thu, 22 Feb 2018 00:37:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 038CBE031B for ; Thu, 22 Feb 2018 00:37:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 6F9DA21096 for ; Thu, 22 Feb 2018 00:37:00 +0000 (UTC) Date: Thu, 22 Feb 2018 00:37:00 +0000 (UTC) From: "Hadoop QA (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-20035) .TestQuotaStatusRPCs.testQuotaStatusFromMaster failed with NPEs and RuntimeExceptions MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-20035?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D163= 72263#comment-16372263 ]=20 Hadoop QA commented on HBASE-20035: ----------------------------------- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s= {color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m = 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} = 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {colo= r} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0= m 0s{color} | {color:green} The patch does not contain any @author tags. {= color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green}= 0m 0s{color} | {color:green} The patch appears to include 1 new or modif= ied test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}= 3m 4s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0= m 38s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}= 0m 54s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}= 4m 31s{color} | {color:green} branch has no errors when building our shad= ed downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0= m 25s{color} | {color:green} branch-2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}= 3m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0= m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m = 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}= 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green}= 0m 0s{color} | {color:green} The patch has no whitespace issues. {color}= | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}= 3m 37s{color} | {color:green} patch has no errors when building our shade= d downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green= } 13m 10s{color} | {color:green} Patch does not cause any errors with Hadoo= p 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0= m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}108m = 3s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green}= 0m 22s{color} | {color:green} The patch does not generate ASF License war= nings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}135m 41s{colo= r} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=3D17.05.0-ce Server=3D17.05.0-ce Image:yetus/hbase:9f2f2d= b | | JIRA Issue | HBASE-20035 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/1291143= 6/HBASE-20035.001.branch-2.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars = hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7d48724cb4f4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:= 48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build= /component/dev-support/hbase-personality.sh | | git revision | branch-2 / baec532aa2 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a= 5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11605= /testReport/ | | Max. process+thread count | 5504 (vs. ulimit of 10000) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/1160= 5/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > .TestQuotaStatusRPCs.testQuotaStatusFromMaster failed with NPEs and Runti= meExceptions > -------------------------------------------------------------------------= ------------ > > Key: HBASE-20035 > URL: https://issues.apache.org/jira/browse/HBASE-20035 > Project: HBase > Issue Type: Sub-task > Reporter: stack > Assignee: Josh Elser > Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-20035.001.branch-2.patch > > > It failed the nightly. > Says this... > Error Message > Waiting timed out after [30,000] msec > Stacktrace > java.lang.AssertionError: Waiting timed out after [30,000] msec > =09at org.apache.hadoop.hbase.quotas.TestQuotaStatusRPCs.testQuotaStatusF= romMaster(TestQuotaStatusRPCs.java:267) > ... but looking in log I see following: > Odd thing is the test is run three times and it failed all three times fo= r same reason. > [ERROR] Failures:=20 > [ERROR] org.apache.hadoop.hbase.quotas.TestQuotaStatusRPCs.testQuotaStatu= sFromMaster(org.apache.hadoop.hbase.quotas.TestQuotaStatusRPCs) > [ERROR] Run 1: TestQuotaStatusRPCs.testQuotaStatusFromMaster:267 Waitin= g timed out after [30,000] msec > [ERROR] Run 2: TestQuotaStatusRPCs.testQuotaStatusFromMaster:267 Waitin= g timed out after [30,000] msec > [ERROR] Run 3: TestQuotaStatusRPCs.testQuotaStatusFromMaster:267 Waitin= g timed out after [30,000] msec > If you go to build artifacts you can download full -output.txt log. I see= stuff like this which might be ok.... > {code} > 2018-02-21 01:29:59,546 INFO [StoreCloserThread-testQuotaStatusFromMaste= r4,0,1519176558800.1dbd00f38915cd276410065f85140b26.-1] regionserver.HStore= (930): Closed f1 > 2018-02-21 01:29:59,551 ERROR [master/ad51e354307e:0.Chore.2] hbase.Sched= uledChore(189): Caught error > java.lang.RuntimeException: java.util.concurrent.RejectedExecutionExcepti= on: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$Queu= eingFuture@79ec2ef9 rejected from java.util.concurrent.ThreadPoolExecutor@5= 198a326[Terminated, pool size =3D 0, active threads =3D 0, queued tasks =3D= 0, completed tasks =3D 142] > at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetr= ies(RpcRetryingCallerImpl.java:200) > at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java= :269) > at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner= .java:437) > at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(Clien= tScanner.java:312) > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java= :597) > at org.apache.hadoop.hbase.quotas.QuotaRetriever.next(QuotaRetriever.ja= va:106) > at org.apache.hadoop.hbase.quotas.QuotaRetriever$Iter.(QuotaRetri= ever.java:125) > at org.apache.hadoop.hbase.quotas.QuotaRetriever.iterator(QuotaRetrieve= r.java:117) > at org.apache.hadoop.hbase.quotas.QuotaObserverChore.fetchAllTablesWith= QuotasDefined(QuotaObserverChore.java:458) > at org.apache.hadoop.hbase.quotas.QuotaObserverChore._chore(QuotaObserv= erChore.java:148) > at org.apache.hadoop.hbase.quotas.QuotaObserverChore.chore(QuotaObserve= rChore.java:136) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask= .access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask= .run(ScheduledThreadPoolExecutor.java:294) > at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$Jitter= edRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:11= 1) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.concurrent.RejectedExecutionException: Task org.apac= he.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@79ec2e= f9 rejected from java.util.concurrent.ThreadPoolExecutor@5198a326[Terminate= d, pool size =3D 0, active threads =3D 0, queued tasks =3D 0, completed tas= ks =3D 142] > at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutio= n(ThreadPoolExecutor.java:2063) > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.ja= va:830) > at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.j= ava:1379) > at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit= (ResultBoundedCompletionService.java:171) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsF= orCurrentReplica(ScannerCallableWithReplicas.java:320) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(Scan= nerCallableWithReplicas.java:182) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(Scan= nerCallableWithReplicas.java:58) > at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetr= ies(RpcRetryingCallerImpl.java:192) > ... 19 more > {code} > .. but then this which looks less ok: > {code} > 2018-02-21 01:30:24,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3D4,qu= eue=3D0,port=3D39067] ipc.CallRunner(141): callId: 16 service: RegionServer= StatusService methodName: ReportRegionSpaceUse size: 109 connection: 172.17= .0.2:57185 deadline: 1519176634044 > org.apache.hadoop.hbase.PleaseHoldException: Master is initializing > at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java= :2741) > at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionSpaceUs= e(MasterRpcServices.java:2135) > at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatus= Protos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusPro= tos.java:11053) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :304) > 2018-02-21 01:30:24,046 ERROR [regionserver/ad51e354307e:0.Chore.1] hbase= .ScheduledChore(189): Caught error > java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundExcept= ion: hbase:quota > at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner= .java:55) > at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.fetchSnapsho= tsFromQuotaTable(SpaceQuotaRefresherChore.java:151) > at org.apache.hadoop.hbase.quotas.SpaceQuotaRefresherChore.chore(SpaceQ= uotaRefresherChore.java:84) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask= .access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask= .run(ScheduledThreadPoolExecutor.java:294) > at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$Jitter= edRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:11= 1) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hbase.TableNotFoundException: hbase:quota > at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion= InMeta(ConnectionImplementation.java:842) > at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion= (ConnectionImplementation.java:733) > at org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingCluste= rConnection.locateRegion(ConnectionUtils.java:131) > at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.get= RegionLocations(RpcRetryingCallerWithReadReplicas.java:325) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(Scan= nerCallableWithReplicas.java:153) > at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(Scan= nerCallableWithReplicas.java:58) > at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetr= ies(RpcRetryingCallerImpl.java:192) > at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java= :269) > at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner= .java:437) > at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(Clien= tScanner.java:312) > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java= :597) > at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner= .java:53) > ... 11 more > 2018-02-21 01:30:24,179 DEBUG [Time-limited test-EventThread] zookeeper.Z= KWatcher(443): master:39067-0x161b5fb62d10000, quorum=3Dlocalhost:49877, ba= seZNode=3D/hbase Received ZooKeeper Event, type=3DNodeChildrenChanged, stat= e=3DSyncConnected, path=3D/hbase/namespace > 2018-02-21 01:30:24,604 INFO [PEWorker-13] procedure2.ProcedureExecutor(= 1247): Finished pid=3D6, state=3DSUCCESS; CreateNamespaceProcedure, namespa= ce=3Dhbase in 1.0980sec > 2018-02-21 01:30:24,772 DEBUG [Time-limited test-EventThread] zookeeper.Z= KWatcher(443): master:39067-0x161b5fb62d10000, quorum=3Dlocalhost:49877, ba= seZNode=3D/hbase Received ZooKeeper Event, type=3DNodeDataChanged, state=3D= SyncConnected, path=3D/hbase/namespace/default > 2018-02-21 01:30:24,830 DEBUG [Time-limited test-EventThread] zookeeper.Z= KWatcher(443): master:39067-0x161b5fb62d10000, quorum=3Dlocalhost:49877, ba= seZNode=3D/hbase Received ZooKeeper Event, type=3DNodeDataChanged, state=3D= SyncConnected, path=3D/hbase/namespace/hbase > 2018-02-21 01:30:24,831 INFO [M:0;ad51e354307e:39067] master.HMaster(927= ): Master has completed initialization 17.276sec > 2018-02-21 01:30:24,832 INFO [M:0;ad51e354307e:39067] quotas.MasterQuota= Manager(87): Quota table not found. Creating... > 2018-02-21 01:30:24,833 INFO [M:0;ad51e354307e:39067] master.HMaster(178= 3): Client=3Dnull/null create 'hbase:quota', {NAME =3D> 'q', VERSIONS =3D> = '1', EVICT_BLOCKS_ON_CLOSE =3D> 'false', NEW_VERSION_BEHAVIOR =3D> 'false',= KEEP_DELETED_CELLS =3D> 'FALSE', CACHE_DATA_ON_WRITE =3D> 'false', DATA_BL= OCK_ENCODING =3D> 'NONE', TTL =3D> 'FOREVER', MIN_VERSIONS =3D> '0', REPLIC= ATION_SCOPE =3D> '0', BLOOMFILTER =3D> 'ROW', CACHE_INDEX_ON_WRITE =3D> 'fa= lse', IN_MEMORY =3D> 'false', CACHE_BLOOMS_ON_WRITE =3D> 'false', PREFETCH_= BLOCKS_ON_OPEN =3D> 'false', COMPRESSION =3D> 'NONE', BLOCKCACHE =3D> 'true= ', BLOCKSIZE =3D> '65536'}, {NAME =3D> 'u', VERSIONS =3D> '1', EVICT_BLOCKS= _ON_CLOSE =3D> 'false', NEW_VERSION_BEHAVIOR =3D> 'false', KEEP_DELETED_CEL= LS =3D> 'FALSE', CACHE_DATA_ON_WRITE =3D> 'false', DATA_BLOCK_ENCODING =3D>= 'NONE', TTL =3D> 'FOREVER', MIN_VERSIONS =3D> '0', REPLICATION_SCOPE =3D> = '0', BLOOMFILTER =3D> 'ROW', CACHE_INDEX_ON_WRITE =3D> 'false', IN_MEMORY = =3D> 'false', CACHE_BLOOMS_ON_WRITE =3D> 'false', PREFETCH_BLOCKS_ON_OPEN = =3D> 'false', COMPRESSION =3D> 'NONE', BLOCKCACHE =3D> 'true', BLOCKSIZE = =3D> '65536'} > 2018-02-21 01:30:24,979 ERROR [RpcServer.default.FPBQ.Fifo.handler=3D4,qu= eue=3D0,port=3D39067] ipc.RpcServer(464): Unexpected throwable object > java.lang.NullPointerException > at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionSpaceUs= e(MasterRpcServices.java:2142) > at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatus= Protos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusPro= tos.java:11053) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :304) > 2018-02-21 01:30:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3D4,qu= eue=3D0,port=3D39067] ipc.CallRunner(141): callId: 18 service: RegionServer= StatusService methodName: ReportRegionSpaceUse size: 109 connection: 172.17= .0.2:57185 deadline: 1519176634979 > java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:465) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :304) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionSpaceUs= e(MasterRpcServices.java:2142) > at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatus= Protos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusPro= tos.java:11053) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) > ... 3 more > 2018-02-21 01:30:24,982 DEBUG [regionserver/ad51e354307e:0.Chore.1] regio= nserver.HRegionServer(1245): Failed to report region sizes to Master. This = will be retried. > java.io.IOException: java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:465) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :304) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionSpaceUs= e(MasterRpcServices.java:2142) > at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatus= Protos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusPro= tos.java:11053) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) > ... 3 more > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method= ) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruc= torAccessorImpl.java:62) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegating= ConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateExc= eption(RemoteWithExtrasException.java:100) > at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteEx= ception(RemoteWithExtrasException.java:90) > at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOException= OfException(ProtobufUtil.java:358) > at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.getRemoteExcept= ion(ProtobufUtil.java:335) > at org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionSizes= ForQuotas(HRegionServer.java:1224) > at org.apache.hadoop.hbase.quotas.FileSystemUtilizationChore.reportRegi= onSizesToMaster(FileSystemUtilizationChore.java:187) > at org.apache.hadoop.hbase.quotas.FileSystemUtilizationChore.chore(File= SystemUtilizationChore.java:136) > at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:5= 11) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask= .access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask= .run(ScheduledThreadPoolExecutor.java:294) > at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$Jitter= edRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:11= 1) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.= IOException): java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:465) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java= :304) > Caused by: java.lang.NullPointerException > at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionSpaceUs= e(MasterRpcServices.java:2142) > at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatus= Protos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusPro= tos.java:11053) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) > ... 3 more > {code} > You might have some input mighty [~elserj] ? Thanks sir. It failed only t= his once. I could up the timeouts from 30 seconds because that allowed but = my guess it he NPE does us in. Thanks sir. -- This message was sent by Atlassian JIRA (v7.6.3#76005)