Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 60C05200AF7 for ; Tue, 14 Jun 2016 19:35:29 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5F577160A5F; Tue, 14 Jun 2016 17:35:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AF89B160A06 for ; Tue, 14 Jun 2016 19:35:28 +0200 (CEST) Received: (qmail 81989 invoked by uid 500); 14 Jun 2016 17:35:27 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 81939 invoked by uid 99); 14 Jun 2016 17:35:27 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2016 17:35:27 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 603A82C1F61 for ; Tue, 14 Jun 2016 17:35:27 +0000 (UTC) Date: Tue, 14 Jun 2016 17:35:27 +0000 (UTC) From: "ramkrishna.s.vasudevan (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-15999) NPE in MemstoreCompactor MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 14 Jun 2016 17:35:29 -0000 [ https://issues.apache.org/jira/browse/HBASE-15999?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15999: ------------------------------------------- Status: Open (was: Patch Available) > NPE in MemstoreCompactor > ------------------------ > > Key: HBASE-15999 > URL: https://issues.apache.org/jira/browse/HBASE-15999 > Project: HBase > Issue Type: Sub-task > Affects Versions: 2.0.0 > Reporter: ramkrishna.s.vasudevan > Assignee: ramkrishna.s.vasudevan > Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15999.patch, HBASE-15999_1.patch > > > In the INMEMORY_COMPACTION_POOL ThreadPoolExecutor for every thread the c= urrent thread name is also appended. Since we are using a pool the name get= s appended and we end up in names like this > {code} > "B.defaultRpcServer.handler=3D89,queue=3D9,port=3D16041-inmemoryCompactio= ns-1465492533442-inmemoryCompactions-1465492548754-inmemoryCompactions-1465= 492548913-inmemoryCompactions-1465492549625-inmemoryCompactions-14654925499= 56-inmemoryCompactions-1465492567040-inmemoryCompactions-1465492567160-inme= moryCompactions-1465492578465-inmemoryCompactions-1465492578707-inmemoryCom= pactions-1465492579292-inmemoryCompactions-1465492579357-inmemoryCompaction= s-1465492579786-inmemoryCompactions-1465492580059-inmemoryCompactions-14654= 92589975-inmemoryCompactions-1465492590192-inmemoryCompactions-146549259048= 4-inmemoryCompactions-1465492591144-inmemoryCompactions-1465492592603-inmem= oryCompactions-1465492592799-inmemoryCompactions-1465492597106-inmemoryComp= actions-1465492602925-inmemoryCompactions-1465492606620-inmemoryCompactions= -1465492651478-inmemoryCompactions-1465492653460-inmemoryCompactions-146549= 2677020-inmemoryCompactions-1465492680857-inmemoryCompactions-1465492681989= -inmemoryCompactions-1465492721818-inmemoryCompactions-1465492723562-inmemo= ryCompactions-1465492724801-inmemoryCompactions-1465492726665-inmemoryCompa= ctions-1465492745750-inmemoryCompactions-1465492745964-inmemoryCompactions-= 1465492746578-inmemoryCompactions-1465492756867-inmemoryCompactions-1465492= 764727-inmemoryCompactions-1465492766944-inmemoryCompactions-1465492767098-= inmemoryCompactions-1465492785298-inmemoryCompactions-1465492788334-inmemor= yCompactions-1465492795954-inmemoryCompactions-1465493047265-inmemoryCompac= tions-1465493091530-inmemoryCompactions-1465493185684" #6006 daemon prio=3D= 5 os_prio=3D0 tid=3D0x000000000daa6800 nid=3D0x454a runnable [0x00007f50fd0= b9000] > {code} > As we were surprised to see why so many threads are getting created as An= oop pointed out the pool size is 10 and there is no setting for the thread = to die, the reason for this issue is that we have an multi threaded issue i= n MemstoreCompactor. The memstoreCompactor has the StoreScanner and Memstor= eScanner as the state variable and every time we just instantiate a new one= when a new inmemory flush request comes. Finally we try to release the res= ource where the scanner is nullified and closed. But the instance would hav= e already been updated or nullified by another thread when there are multip= le requests. So this causes an NPE in releaseResources. > {code} > Exception in thread "B.defaultRpcServer.handler=3D76,queue=3D6,port=3D160= 41-inmemoryCompactions-1465906554314" java.lang.NullPointerException > at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.release= Resources(MemStoreCompactor.java:108) > at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.doCompa= ction(MemStoreCompactor.java:144) > at org.apache.hadoop.hbase.regionserver.MemStoreCompactor.startCo= mpaction(MemStoreCompactor.java:88) > at org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushI= nMemory(CompactingMemStore.java:287) > at org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemo= ryFlushRunnable.run(CompactingMemStore.java:356) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx= ecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE= xecutor.java:617) > {code} > So every time the thread dies and a new is created which I found from the= stack traces by adding some logs. So this is why we were creating lot of t= hreads and the name was simply getting appened. Now from this we can see th= at adding the Handler name is fine but the main issue is the NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)