hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: [VOTE] The 2nd hbase 0.94.17 release candidate is available for download
Date Tue, 25 Feb 2014 13:34:56 GMT
I reran the tests just for read using --presplit 6 and enought lines to
create 7 regions. Ran more than 12h for each release.
0.94.16 => 367 random get / second.
0.94.17 => 372 random get / second.

I initially had more than 400 gets/seconds for 0.94.16. But I'm not getting
that anymore.

So it's all +1 for me.

JM

2014-02-23 13:54 GMT-05:00 Jean-Marc Spaggiari <jean-marc@spaggiari.org>:

> Hey, I'm not getting where the 15% slowdown is coming too neither. I will
> re-run the tests. I run the randomread on a 2M lines table written with
> RandomWrite. It creates  3 region table so I did not used presplit. I run
> the balancer after that. Since they are 3 nodes, whereever the regions are
> assigned, HDFS blocks are local. So I'm not running major compaction. But I
> can if required. I did not run it with MR. Only with 1 single client. I
> will have the MR in the scope for the next release.
>
> For MAX_FILESIZE, I will first retry on my previous cluster with a
> previous version and will com back if I'm able to get a difference in the
> behaviors. But at the end, what I see in 0.94.17 is still good, and even if
> different that what I think I saw before, it's better. More what I will
> have expected.
>
> I'm rerunning the RandomRead with presplit and twice the rows. Will take
> about 24h to get the results...
>
>
> 2014-02-23 12:39 GMT-05:00 Ted Yu <yuzhihong@gmail.com>:
>
> To make the read comparison fair, I think '--presplit=' should be used.
>>
>> Cheers
>>
>>
>> On Sun, Feb 23, 2014 at 11:15 AM, lars hofhansl <larsh@apache.org> wrote:
>>
>> > Thanks for the thorough testing.
>> >
>> >
>> > Can you elaborate "page is split according to the MAX_FILESIZE parameter
>> > until all regions are according to it"?
>> > As for the 15% slowdown in the random read test... I have looked through
>> > the issues and did not see anything warranting that, it's also nothing I
>> > had observed in my testing.
>> >
>> > Did you run this with M/R? HBASE-10501 will reduce the default number of
>> > regions per RegionServer (would be as high as 19 per table before even
>> with
>> > little data), so maybe you're seeing a bit less parallelism (for small
>> data
>> > sets).
>> >
>> >
>> > -- Lars
>> >
>> >
>> >
>> > ________________________________
>> >  From: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
>> > To: dev <dev@hbase.apache.org>
>> > Sent: Sunday, February 23, 2014 5:47 AM
>> > Subject: Re: [VOTE] The 2nd hbase 0.94.17 release candidate is available
>> > for download
>> >
>> >
>> > Haha ;) Yes. UI. Under the page details in the WebUI, In previous
>> release
>> > (maybe not 0.94.16 because I did not test it) when clicking split in the
>> > WebUI, table was split in half. Now, page is split according to the
>> > MAX_FILESIZE
>> > parameter until all regions are according to it.
>> >
>> > I'm not 100% sure about it so might be wrong. But I'm not sure I have
>> seen
>> > that before.
>> >
>> >
>> >
>> > 2014-02-23 8:32 GMT-05:00 Ted Yu <yuzhihong@gmail.com>:
>> >
>> > > bq. Click split under UA
>> > >
>> > > I guess you meant 'under UI'
>> > >
>> > > Cheers
>> > >
>> > > On Feb 23, 2014, at 4:47 AM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org>
>> > > wrote:
>> > >
>> > > > Tested, tl.tr: +1 ;)
>> > > >
>> > > > All tests are done with Oracle JDK 1.7.0_45 + Hadoop 1.2.1 (Except
>> for
>> > > the
>> > > > 0.96.1 test where Hadoop 2.2.0 is used)
>> > > >
>> > > > Checked the jars, the signature, the CHANGES.TXT file, the
>> > documentation
>> > > =>
>> > > > Passed.
>> > > > Checked logs, UI, HBCK, test suite => Passed
>> > > > Run all Performances tests => Passed. 0.94.17 performances just
a
>> bit
>> > > under
>> > > > 0.94.16. RandomReadTest 15% slower with 0.94.17. Will retest it.
>> Will
>> > > post
>> > > > all the numbers in detail in my blog later today.
>> > > >
>> > > > Only concern is again the IntegrationTestLoadAndVerify result where
>> I
>> > > don't
>> > > > have the exptected result, but it's the same thing with 0.94.16 and
>> > > 0.96.1
>> > > > too. With difference Hadoop version.
>> > > >
>> > > >
>> > > > 0.94.16 + Hadoop 1.2.1:
>> > > > 14/02/21 21:22:19 INFO mapred.JobClient:
>> > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters
>> > > > 14/02/21 21:22:19 INFO mapred.JobClient:     ROWS_WRITTEN=0
>> > > > 14/02/21 21:22:19 INFO mapred.JobClient:
>> REFERENCES_CHECKED=9855584
>> > > >
>> > > > 0.94.17 + Hadoop 1.2.1:
>> > > > 14/02/21 21:31:28 INFO mapred.JobClient:
>> > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters
>> > > > 14/02/21 21:31:28 INFO mapred.JobClient:     ROWS_WRITTEN=0
>> > > > 14/02/21 21:31:28 INFO mapred.JobClient:
>> REFERENCES_CHECKED=9855909
>> > > >
>> > > > 0.96.1 + Hadoop 2.2.0:
>> > > > org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify$Counters
>> > > > REFERENCES_CHECKED=9855588
>> > > >
>> > > > Create table, put, scan, compact => Passed.
>> > > > Offline merged all regions into a single one, hbck, split => Passed.
>> > > > Rolling restart => Failed once because of jar push issue (my bad),
>> > > retried
>> > > > => Passed.
>> > > > Ran all tests (LoadTestTool, IntegrationTestLoadAndVerify,
>> > > > HLogPerformanceEvaluation, IntegrationTestBigLinkedList) on a 4
>> nodes
>> > > > cluster => Passed.
>> > > > When we click split in the UA, seems to trigger major compaction,
>> which
>> > > was
>> > > > not the case before (Not sure in which version this has been
>> > introduced)
>> > > >
>> > > > Summary:
>> > > > - IntgrationTestLoadAndVerify results still "strange";
>> > > > - Split behaviour in the UA seem the have changed.
>> > > > - +1
>> > > >
>> > > > JM
>> > > > PS: Go Canada Go!
>> > > >
>> > > > 2014-02-21 10:01 GMT-05:00 Ted Yu <yuzhihong@gmail.com>:
>> > > >
>> > > >> +1 based on the following:
>> > > >>
>> > > >> 1. Ran test suite using JDK 1.7 - pass
>> > > >> 2. compiled Phoenix with locally built 0.94.17 - pass
>> > > >> 3. checked the documentation
>> > > >>
>> > > >> Cheers
>> > > >>
>> > > >>
>> > > >> On Thu, Feb 20, 2014 at 11:18 PM, lars hofhansl <larsh@apache.org>
>> > > wrote:
>> > > >>
>> > > >>> +1
>> > > >>>
>> > > >>> - checked documentation and tarball
>> > > >>>
>> > > >>> - Ran in local and distributes mode
>> > > >>> - checked the UI pages
>> > > >>>
>> > > >>> - loaded 100m KeyValues
>> > > >>> - tested scan performance
>> > > >>>
>> > > >>> - converted between FAST_DIFF and NONE encoding
>> > > >>> - tested flushes and compactions
>> > > >>>
>> > > >>>
>> > > >>> -- Lars
>> > > >>>
>> > > >>>
>> > > >>> ________________________________
>> > > >>> From: lars hofhansl <larsh@apache.org>
>> > > >>> To: hbase-dev <dev@hbase.apache.org>
>> > > >>> Sent: Tuesday, February 18, 2014 8:31 PM
>> > > >>> Subject: [VOTE] The 2nd hbase 0.94.17 release candidate is
>> available
>> > > for
>> > > >>> download
>> > > >>>
>> > > >>>
>> > > >>> The 2nd 0.94.17 RC is available for download at
>> > > >>> http://people.apache.org/~larsh/hbase-0.94.17-rc1/
>> > > >>> Signed with my code signing key: C7CFE328
>> > > >>>
>> > > >>> HBase 0.94.17 is a bug fix release with 35 fixes:
>> > > >>>    [HBASE-7963] - HBase VerifyReplication not working when
>> security
>> > > >>> enabled
>> > > >>>    [HBASE-10249] - TestReplicationSyncUpTool fails because
>> failover
>> > > >> takes
>> > > >>> too long
>> > > >>>    [HBASE-10274] - MiniZookeeperCluster should close ZKDatabase
>> when
>> > > >>> shutdown ZooKeeperServers
>> > > >>>    [HBASE-10319] - HLog should roll periodically to allow
DN
>> > > >> decommission
>> > > >>> to eventually complete.
>> > > >>>    [HBASE-10320] - Avoid ArrayList.iterator()
>> ExplicitColumnTracker
>> > > >>>    [HBASE-10335] - AuthFailedException in zookeeper may block
>> > > >> replication
>> > > >>> forever
>> > > >>>    [HBASE-10340] - [BACKPORT] HBASE-9892 Add info port to
>> ServerName
>> > to
>> > > >>> support multi instances in a node
>> > > >>>    [HBASE-10363] - [0.94] TestInputSampler and
>> TestInputSamplerTool
>> > > fail
>> > > >>> under hadoop 2.0/23 profiles.
>> > > >>>    [HBASE-10371] - Compaction creates empty hfile, then selects
>> this
>> > > >> file
>> > > >>> for compaction and creates empty hfile and over again
>> > > >>>    [HBASE-10383] - Secure Bulk Load for 'completebulkload'
fails
>> for
>> > > >>> version 0.94.15
>> > > >>>    [HBASE-10400] - [hbck] Continue if region dir missing on
region
>> > > merge
>> > > >>> attempt
>> > > >>>    [HBASE-10401] - [hbck] perform overlap group merges in
parallel
>> > > >>>    [HBASE-10448] - ZKUtil create and watch methods don't set
>> watch in
>> > > >>> some cases
>> > > >>>    [HBASE-10470] - Import generates huge log file while importing
>> > large
>> > > >>> amounts of data
>> > > >>>    [HBASE-10481] - API Compatibility JDiff script does not
>> properly
>> > > >>> handle arguments in reverse order
>> > > >>>    [HBASE-10482] - ReplicationSyncUp doesn't clean up its
ZK,
>> needed
>> > > for
>> > > >>> tests
>> > > >>>    [HBASE-10485] - PrefixFilter#filterKeyValue() should perform
>> > > >> filtering
>> > > >>> on row key
>> > > >>>    [HBASE-10489] - TestImportExport fails in 0.94 with Hadoop2
>> > > >>>    [HBASE-10493] - InclusiveStopFilter#filterKeyValue() should
>> > perform
>> > > >>> filtering on row key
>> > > >>>    [HBASE-10501] - Improve
>> IncreasingToUpperBoundRegionSplitPolicy to
>> > > >>> avoid too many regions
>> > > >>>    [HBASE-10505] - Import.filterKv does not call
>> Filter.filterRowKey
>> > > >>>    [HBASE-10506] - Fail-fast if client connection is lost
before
>> the
>> > > >> real
>> > > >>> call be executed in RPC layer
>> > > >>>    [HBASE-10508] - Backport HBASE-10365 'HBaseFsck should
clean up
>> > > >>> connection properly when repair is completed' to 0.94 and
0.96
>> > > >>>    [HBASE-10539] - HRegion.addAndGetGlobalMemstoreSize returns
>> > previous
>> > > >>> size
>> > > >>>    [HBASE-10545] - RS Hangs waiting on region to close on
>> shutdown;
>> > has
>> > > >>> to timeout before can go down
>> > > >>>    [HBASE-10546] - Two scanner objects are open for each hbase
map
>> > task
>> > > >>> but only one scanner object is closed
>> > > >>>    [HBASE-10551] - Change local mode back to one RS thread
by
>> default
>> > > >>>    [HBASE-10552] -
>> > > >> HFilePerformanceEvaluation.GaussianRandomReadBenchmark
>> > > >>> fails sometimes.
>> > > >>>    [HBASE-10555] - Backport HBASE-8519 to 0.94, Backup master
will
>> > > never
>> > > >>> come up if primary master dies during initialization
>> > > >>>    [HBASE-10562] - Fix TestMultiTableInputFormat for Hadoop
2 in
>> 0.94
>> > > >>>    [HBASE-10212] - New rpc metric: number of active handler
>> > > >>>    [HBASE-10423] - Report back the message of split or rollback
>> > failure
>> > > >>> to the master
>> > > >>>    [HBASE-10457] - Print corrupted file information in
>> SnapshotInfo
>> > > tool
>> > > >>> without -file option
>> > > >>>    [HBASE-10473] - Add utility for adorning http Context
>> > > >>>    [HBASE-10480] - TestLogRollPeriod#testWithEdits may fail
due to
>> > > >>> insufficient waiting
>> > > >>>
>> > > >>> Notable in this release is HBASE-10501, which changes how
regions
>> are
>> > > >>> distributed for an initially empty table.
>> > > >>>
>> > > >>> The list of changes is also available here:
>> > > >>
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12325845
>> > > >>>
>> > > >>> Here're the jenkins runs for this RC:
>> > > >>> https://builds.apache.org/job/HBase-0.94.17/15/ and
>> > > >>> https://builds.apache.org/job/HBase-0.94.17-security/5/
>> > > >>>
>> > > >>> Please try out the RC, check out the doc, take it for a spin,
etc,
>> > and
>> > > >>> vote +1/-1 by EOD February 24th on whether we should release
this
>> as
>> > > >>> 0.94.17.
>> > > >>>
>> > > >>> As before, I realize that release testing is like pulling
teeth,
>> but
>> > > >>> please give this is quick run, so we can get these bug fixes
out
>> > there.
>> > > >>>
>> > > >>> Thanks.
>> > > >>>
>> > > >>> -- Lars
>> > > >>
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message