Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 77EB8200C53 for ; Tue, 11 Apr 2017 16:03:52 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 76629160B89; Tue, 11 Apr 2017 14:03:52 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 981E5160B9B for ; Tue, 11 Apr 2017 16:03:51 +0200 (CEST) Received: (qmail 72749 invoked by uid 500); 11 Apr 2017 14:03:50 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 72738 invoked by uid 99); 11 Apr 2017 14:03:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Apr 2017 14:03:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 52AFF181069 for ; Tue, 11 Apr 2017 14:03:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id a8YOJ0gY-LUn for ; Tue, 11 Apr 2017 14:03:44 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 9448860E04 for ; Tue, 11 Apr 2017 14:03:43 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 65CA5E0D50 for ; Tue, 11 Apr 2017 14:03:42 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id C3C6F2406E for ; Tue, 11 Apr 2017 14:03:41 +0000 (UTC) Date: Tue, 11 Apr 2017 14:03:41 +0000 (UTC) From: "Raman Ch (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-17901) HBase region server stops because of a failure during memstore flush MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 11 Apr 2017 14:03:52 -0000 [ https://issues.apache.org/jira/browse/HBASE-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raman Ch updated HBASE-17901: ----------------------------- Environment: Ubuntu 14.04.5 LTS HBase Version 1.2.2, revision=1 Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) was: Ubuntu 14.04.5 LTS HBase Version 1.2.2, revision=1 Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) > HBase region server stops because of a failure during memstore flush > -------------------------------------------------------------------- > > Key: HBASE-17901 > URL: https://issues.apache.org/jira/browse/HBASE-17901 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 1.2.2 > Environment: Ubuntu 14.04.5 LTS > HBase Version 1.2.2, revision=1 > Java(TM) SE Runtime Environment (build 1.8.0_60-b27) > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) > Reporter: Raman Ch > > Once per several days region server fails to flush a memstore and stops. > April, 8: > {code} > 2017-04-08 00:10:57,737 WARN [MemStoreFlusher.1] regionserver.HStore: Failed flushing store file, retrying num=9 > java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column actually smaller than the previous column: > at org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:117) > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:464) > at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529) > at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:119) > at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:74) > at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:915) > at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2271) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2375) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2105) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2067) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1958) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1884) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:244) > at java.lang.Thread.run(Thread.java:745) > 2017-04-08 00:10:57,737 FATAL [MemStoreFlusher.1] regionserver.HRegionServer: ABORTING region server datanode13.webmeup.com,16020,1491573320653: Replay of WAL required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: di_ordinal_tmp,gov.ok.data/browse?page=2&category=Natural%20Resources&limitTo=datasets&tags=ed,1489764397211.9d7ca11018672c4aace7f30c8f4253f3. > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2428) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2105) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2067) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1958) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1884) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$600(MemStoreFlusher.java:75) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:244) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: ScanWildcardColumnTracker.checkColumn ran into a column actually smaller than the previous column: > at org.apache.hadoop.hbase.regionserver.ScanWildcardColumnTracker.checkVersions(ScanWildcardColumnTracker.java:117) > at org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:464) > at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529) > at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:119) > at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:74) > at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:915) > at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2271) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2375) > ... 9 more > {code} > After region server restart it functioned properly for a couple of days. > April, 10: > {code} > 2017-04-10 22:36:32,147 WARN [MemStoreFlusher.0] regionserver.HStore: Failed flushing store file, retrying num=9 > java.io.IOException: Non-increasing Bloom keys: de.tina-eicke.blog/category/garten/\x09h after de.uina-eicke.blog/category/fruehling/\x09h > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:936) > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:969) > at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:125) > at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:74) > at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:915) > at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2271) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2375) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2105) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2067) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1958) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1884) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) > at java.lang.Thread.run(Thread.java:745) > 2017-04-10 22:36:32,147 FATAL [MemStoreFlusher.0] regionserver.HRegionServer: ABORTING region server datanode13.webmeup.com,16020,1491828707088: Replay of WAL required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: di_ordinal_tmp,de.thschroeer/lmo/lmo.php?action=results&file=archiv/BLW2-2013.l98&endtab=8&st=8&tabtype=2\x09hw,1489764397211.b07eaba657affc2ba29f84b59c672836. > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2428) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2105) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2067) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1958) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1884) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) > at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Non-increasing Bloom keys: de.tina-eicke.blog/category/garten/\x09h after de.uina-eicke.blog/category/fruehling/\x09h > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:936) > at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:969) > at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:125) > at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:74) > at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:915) > at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2271) > at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2375) > ... 9 more > {code} > Table description: > {code} > 'di_ordinal_tmp', {TABLE_ATTRIBUTES => {DURABILITY => 'ASYNC_WAL', MAX_FILESIZE => '8589934592'}, {NAME => 'di', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => '10368000 SECONDS (120 DAYS)', COMPRESSION => 'GZ', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0', METADATA => {'COMPRESSION_COMPACT' => 'GZ'}} > {code} > The table is being populated only using put operations. There has never been any bulk loading into this table. -- This message was sent by Atlassian JIRA (v6.3.15#6346)