Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D67A9EF0E for ; Tue, 15 Jan 2013 20:14:17 +0000 (UTC) Received: (qmail 94624 invoked by uid 500); 15 Jan 2013 20:14:14 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 94585 invoked by uid 500); 15 Jan 2013 20:14:14 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 94528 invoked by uid 99); 15 Jan 2013 20:14:14 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jan 2013 20:14:14 +0000 Date: Tue, 15 Jan 2013 20:14:14 +0000 (UTC) From: "Eric Newton (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-962) Failed to do close consistency check for tablet MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554268#comment-13554268 ] Eric Newton commented on ACCUMULO-962: -------------------------------------- I just reproduced the problem on a single-host instance: # create a table with 10K splits # offline the table # online the table Failed to find metadata entry for 2;20bac710cb2962ce;20b780346dc5da68 found 2;214e3bcd35a85cbc > Failed to do close consistency check for tablet > ----------------------------------------------- > > Key: ACCUMULO-962 > URL: https://issues.apache.org/jira/browse/ACCUMULO-962 > Project: Accumulo > Issue Type: Bug > Components: tserver > Affects Versions: 1.5.0 > Environment: Small cluster: CentOS 5.7, Cloudera Hadoop 3 update 5 > Reporter: Josh Elser > Assignee: Eric Newton > Priority: Minor > > After updating Accumulo trunk and deploying to a small cluster, I tend to find the following table problem is shown on the monitor. It seems to be a false positive, as I can ignore it and everything goes according to plan. > Some context from the TServer which logged the exception: > {noformat} > 2013-01-13 00:39:36,692 [tabletserver.MinorCompactor] DEBUG: Begin minor compaction /accumulo/tables/7/t-001gnfo/F001gqrl.rf_tmp 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 > 2013-01-13 00:39:36,713 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: UpSess null 9,549 in 0.153s, at=[0 1 0.01 138] ft=0.103s(pt=0.002s lt=0.051s ct=0.050s) > 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: Failures: 7, first extent 7;0f6c8b439581063c;0f5c28f5c28f5c4 successful commits: 0 > 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,714 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,732 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,782 [tabletserver.TabletServer] DEBUG: UpSess null 4,703 in 0.101s, at=[0 0 0.00 138] ft=0.070s(pt=0.002s lt=0.033s ct=0.035s) > 2013-01-13 00:39:36,782 [tabletserver.TabletServer] DEBUG: Failures: 7, first extent 7;0f6c8b439581063c;0f5c28f5c28f5c4 successful commits: 0 > 2013-01-13 00:39:36,826 [tabletserver.TabletServer] DEBUG: UpSess null 9,513 in 0.127s, at=[0 0 0.00 138] ft=0.088s(pt=0.003s lt=0.052s ct=0.033s) > 2013-01-13 00:39:36,826 [tabletserver.TabletServer] DEBUG: Failures: 7, first extent 7;0f6c8b439581063c;0f5c28f5c28f5c4 successful commits: 0 > 2013-01-13 00:39:36,863 [tabletserver.TabletServer] DEBUG: UpSess null 4,688 in 0.071s, at=[0 0 0.00 136] ft=0.048s(pt=0.003s lt=0.022s ct=0.023s) > 2013-01-13 00:39:36,863 [tabletserver.TabletServer] DEBUG: Failures: 5, first extent 7;0f8d4fdf3b645a34;0f7ced916872b038 successful commits: 0 > 2013-01-13 00:39:36,868 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,868 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,868 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,879 [tabletserver.TabletServer] DEBUG: Got unloadTablet message from user: !SYSTEM > 2013-01-13 00:39:36,886 [tabletserver.Compactor] DEBUG: Compaction 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 25,236 read | 25,236 written | 141,775 entries/sec | 0.178 secs > 2013-01-13 00:39:36,888 [tabletserver.Tablet] DEBUG: Logs for memory compacted: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 10.10.32.121+9997/ea6c3aab-02ac-4599-b50e-b00df41887ed > 2013-01-13 00:39:36,888 [tabletserver.Tablet] DEBUG: Logs for memory compacted: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 10.10.32.121+9997/d1aaf814-6209-46f3-8997-36541ea0dcc2 > 2013-01-13 00:39:36,888 [tabletserver.Tablet] DEBUG: Logs for current memory: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 10.10.32.121+9997/ea6c3aab-02ac-4599-b50e-b00df41887ed > 2013-01-13 00:39:36,895 [log.TabletServerLogger] DEBUG: wrote MinC finish 11312: writeTime:3ms > 2013-01-13 00:39:36,895 [tabletserver.Tablet] TABLET_HIST: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 MinC [memory] -> /t-001gnfo/F001gqrl.rf > 2013-01-13 00:39:36,895 [tabletserver.Tablet] DEBUG: MinC finish lock 0.00 secs 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 > 2013-01-13 00:39:36,895 [tabletserver.NativeMap] DEBUG: Deallocating native map 0x00002aaab8f40540 > 2013-01-13 00:39:36,903 [tabletserver.Tablet] DEBUG: completeClose(saveState=true completeClose=true) 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 > 2013-01-13 00:39:36,904 [tabletserver.NativeMap] DEBUG: Allocated native map 0x00002aaabc85e210 > 2013-01-13 00:39:36,904 [tabletserver.MinorCompactor] DEBUG: Begin minor compaction /accumulo/tables/7/t-001gnfo/F001gqrm.rf_tmp 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 > 2013-01-13 00:39:36,919 [tabletserver.LargestFirstMemoryManager] DEBUG: BEFORE compactionThreshold = 0.851 maxObserved = 731,712,351 > 2013-01-13 00:39:36,919 [tabletserver.LargestFirstMemoryManager] DEBUG: AFTER compactionThreshold = 0.851 > 2013-01-13 00:39:36,935 [tabletserver.Compactor] DEBUG: Compaction 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 138 read | 138 written | 69,000 entries/sec | 0.002 secs > 2013-01-13 00:39:36,938 [tabletserver.Tablet] DEBUG: Logs for memory compacted: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 10.10.32.121+9997/ea6c3aab-02ac-4599-b50e-b00df41887ed > 2013-01-13 00:39:36,948 [log.TabletServerLogger] DEBUG: wrote MinC finish 11314: writeTime:0ms > 2013-01-13 00:39:36,948 [tabletserver.Tablet] TABLET_HIST: 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 MinC [memory] -> /t-001gnfo/F001gqrm.rf > 2013-01-13 00:39:36,949 [tabletserver.Tablet] DEBUG: MinC finish lock 0.00 secs 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 > 2013-01-13 00:39:36,949 [tabletserver.NativeMap] DEBUG: Deallocating native map 0x00002aaabd00fd10 > 2013-01-13 00:39:36,979 [tabletserver.Tablet] ERROR: Failed to do close consistency check for tablet 7;3de353f7ced916e4;3dd2f1a9fbe76ce8 > java.lang.RuntimeException: Unexpected row 7;3df3b645a1cac0e expected 7;3de353f7ced916e4 > at org.apache.accumulo.server.util.MetadataTable.getFileAndLogEntries(MetadataTable.java:838) > at org.apache.accumulo.server.tabletserver.Tablet.closeConsistencyCheck(Tablet.java:2761) > at org.apache.accumulo.server.tabletserver.Tablet.completeClose(Tablet.java:2714) > at org.apache.accumulo.server.tabletserver.Tablet.close(Tablet.java:2592) > at org.apache.accumulo.server.tabletserver.TabletServer$UnloadTabletHandler.run(TabletServer.java:2350) > at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) > at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47) > at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) > at java.lang.Thread.run(Thread.java:662) > {noformat} > I can make the entire collection of logs available too if necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira