hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10378) Divide HLog interface into User and Implementor specific interfaces
Date Tue, 18 Nov 2014 21:49:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216857#comment-14216857
] 

Hudson commented on HBASE-10378:
--------------------------------

FAILURE: Integrated in HBase-TRUNK #5787 (See [https://builds.apache.org/job/HBase-TRUNK/5787/])
    HBASE-10378 Refactor write-ahead-log implementation -- ADDEDNUM (stack: rev b94e6738d920cb5bf297214883e0cf29522597b9)
* hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSourceImpl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/HLogSplitterHandler.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationHLogReaderManager.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationHLogReaderManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALCoprocessorEnvironment.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestReadOldRootAndMetaEdits.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SecureProtobufLogWriter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
* hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithEncryption.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/InstrumentedSequenceFileLogWriter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationKillMasterRSCompressed.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/WALObserver.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/CompressionContext.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ReaderBase.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterFileSystem.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
* hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestMetricsHLogSource.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SampleRegionWALObserver.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/HbaseObjectWritableFor96Migration.java
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
* hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* hbase-protocol/src/main/protobuf/RegionServerStatus.proto
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
* hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon
* hbase-shell/src/main/ruby/shell.rb
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/DroppedSnapshotException.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* hbase-common/src/main/resources/hbase-default.xml
* bin/hbase.cmd
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSecureHLog.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/SplitLogWorkerCoordination.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
* src/main/docbkx/troubleshooting.xml
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java
* src/main/docbkx/ops_mgt.xml
* hbase-shell/src/main/ruby/shell/commands/hlog_roll.rb
* hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ChainWALEntryFilter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplitCompressed.java
* hbase-protocol/src/main/protobuf/ZooKeeper.proto
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java
* hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
* hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsEditsReplaySource.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RegionServerCoprocessorRpcChannel.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseWALObserver.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestSecureWALReplay.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/SystemTableWALEntryFilter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCellCodec.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
* bin/hbase
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/HLogLink.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileRefresherChore.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogTask.java
* hbase-protocol/src/main/protobuf/WAL.proto
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALPlayer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
* src/main/docbkx/performance.xml
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogReaderOnSecureHLog.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
* hbase-protocol/src/main/protobuf/Admin.proto
* hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterFileSystemSource.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollPeriod.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinorCompaction.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultyHLog.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
* hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSource.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WriterBase.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
* src/main/docbkx/book.xml
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetaLogRoller.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java


> Divide HLog interface into User and Implementor specific interfaces
> -------------------------------------------------------------------
>
>                 Key: HBASE-10378
>                 URL: https://issues.apache.org/jira/browse/HBASE-10378
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>            Reporter: Himanshu Vashishtha
>            Assignee: Sean Busbey
>             Fix For: 2.0.0
>
>         Attachments: 10378-1.patch, 10378-2.patch, HBASE-10378.3.patch.txt, HBASE-10378.4.patch.txt,
HBASE-10378.5.patch.txt
>
>
> HBASE-5937 introduces the HLog interface as a first step to support multiple WAL implementations.
This interface is a good start, but has some limitations/drawbacks in its current state, such
as:
> 1) There is no clear distinction b/w User and Implementor APIs, and it provides APIs
both for WAL users (append, sync, etc) and also WAL implementors (Reader/Writer interfaces,
etc). There are APIs which are very much implementation specific (getFileNum, etc) and a user
such as a RegionServer shouldn't know about it.
> 2) There are about 14 methods in FSHLog which are not present in HLog interface but are
used at several places in the unit test code. These tests typecast HLog to FSHLog, which makes
it very difficult to test multiple WAL implementations without doing some ugly checks.
> I'd like to propose some changes in HLog interface that would ease the multi WAL story:
> 1) Have two interfaces WAL and WALService. WAL provides APIs for implementors. WALService
provides APIs for users (such as RegionServer).
> 2) A skeleton implementation of the above two interface as the base class for other WAL
implementations (AbstractWAL). It provides required fields for all subclasses (fs, conf, log
dir, etc). Make a minimal set of test only methods and add this set in AbstractWAL.
> 3) HLogFactory returns a WALService reference when creating a WAL instance; if a user
need to access impl specific APIs (there are unit tests which get WAL from a HRegionServer
and then call impl specific APIs), use AbstractWAL type casting,
> 4) Make TestHLog abstract and let all implementors provide their respective test class
which extends TestHLog (TestFSHLog, for example).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message