hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-4908) HBase cluster test tool (port from 0.89-fb)
Date Fri, 02 Dec 2011 10:09:41 GMT

     [ https://issues.apache.org/jira/browse/HBASE-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Phabricator updated HBASE-4908:
-------------------------------

    Attachment: D549.3.patch

mbautin updated the revision "[jira] [HBASE-4908] HBase cluster test tool (port from 0.89-fb)".
Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

  Debugged and significantly improved RestartMetaTest and its dependency ProcessBasedLocalHBaseCluster
to the point of actually being able to run a local multi-process test of an HBase cluster
that kills the region server with the .META. region and splits the logs. Discovered that our
log splitting does not work correctly with RawLocalFileSystem or checksum-enabled local filesystem,
and added a fix to the hack in SequenceFileLogReader to only apply reflection logic to DFSInputStream.

  Still need to clean up the code, re-run unit tests, and re-test HBaseClusterTest on a live
cluster.

REVISION DETAIL
  https://reviews.facebook.net/D549

AFFECTED FILES
  pom.xml
  src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
  src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
  src/main/java/org/apache/hadoop/hbase/HConstants.java
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
  src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
  src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
  src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
  src/main/java/org/apache/hadoop/hbase/util/HMerge.java
  src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
  src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
  src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
  src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
  src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
  src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
  src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
  src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
  src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java
  src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
  src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
  src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java
  src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
  src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
  src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
  src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
  src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
  src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java

                
> HBase cluster test tool (port from 0.89-fb)
> -------------------------------------------
>
>                 Key: HBASE-4908
>                 URL: https://issues.apache.org/jira/browse/HBASE-4908
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>         Attachments: D549.1.patch, D549.2.patch, D549.3.patch
>
>
> Porting one of our HBase cluster test tools (a single-process multi-threaded load generator
and verifier) from 0.89-fb to trunk.
> I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has
some features that I have not tried yet (some kind of a kill test, and some way to run HBase
as multiple processes on one machine).
> The main utility of this piece of code for us has been the HBaseClusterTest command-line
tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node
dev cluster testing, e.g.:
> hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test
-read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP
> I will be using this code to load-test the delta encoding patch and making fixes, but
I am submitting the patch for early feedback. I will probably try out its other functionality
and comment on how it works.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message