Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE19E11E3E for ; Mon, 7 Jul 2014 06:28:34 +0000 (UTC) Received: (qmail 58815 invoked by uid 500); 7 Jul 2014 06:28:34 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 58771 invoked by uid 500); 7 Jul 2014 06:28:34 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 58754 invoked by uid 99); 7 Jul 2014 06:28:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jul 2014 06:28:34 +0000 Date: Mon, 7 Jul 2014 06:28:34 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-11315) Keeping MVCC for configurable longer time MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-11315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053376#comment-14053376 ] Hudson commented on HBASE-11315: -------------------------------- FAILURE: Integrated in HBase-1.0 #16 (See [https://builds.apache.org/job/HBase-1.0/16/]) HBase-11315: Keeping MVCC for configurable longer time (jeffreyz: rev cc4f54770bc9b517f561a6fefcca087ba15c79c2) * hbase-common/src/test/java/org/apache/hadoop/hbase/codec/TestCellCodec.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConsistencyControl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeCell.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/row/data/TestRowDataDifferentTimestamps.java * hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestByteRangeWithKVSerialization.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodedDataBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestPayloadCarryingRpcController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * hbase-protocol/src/main/protobuf/WAL.proto * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java > Keeping MVCC for configurable longer time > ------------------------------------------ > > Key: HBASE-11315 > URL: https://issues.apache.org/jira/browse/HBASE-11315 > Project: HBase > Issue Type: Improvement > Affects Versions: 0.99.0 > Reporter: Jeffrey Zhong > Assignee: Jeffrey Zhong > Attachments: hbase-11315-v1.patch, hbase-11315.patch > > > After hbase-8763, we need keep mvcc number longer in hfile so that it can be used to order changes during writes. For example, the known put,delete,put,... scenario, cross region server scan, out of order puts(in recovery case). > Current thinking is that we make the retention period configurable(below we're using 1 day to explain). During major compaction, we check hfile's creation time if a hfile creation time is older than 1 day then all mvcc of KVs in that hfile will be removed. If a hfile is created within 1 day, then all mvccs of KVs in that hfile will be kept. > In case there are time clock skew, we can firstly sort hfiles based on its seqId in ascending order and find the first hfile's creation time stamp less than 1 day. Then mvcc of all hfiles before the found file will be removed during compaction. -- This message was sent by Atlassian JIRA (v6.2#6252)