hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qiang Tian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
Date Thu, 23 Oct 2014 09:33:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181168#comment-14181168
] 

Qiang Tian commented on HBASE-11368:
------------------------------------

initial YCSB test:

Env:
---
hadoop 2.2.0
YCSB 1.0.4(Andrew's branch)
3 nodes, 1 master, 2 RS  //ignore cluster details since just to evaluate the new lock

Steps:
---
Followed Andrew's steps(see http://search-hadoop.com/m/DHED4hl7pC/)
the seed table has 3 CFs, pre-split to 20 regions
load 1 million rows to CF 'f1', using workloada
run 3 iterations for workloadc and workloada respectively. the parameter in each run:
bq. -p columnfamily=f1 -p operationcount=1000000 -s -threads 10


Results:
---
0.98.5:
workload c:
[READ], AverageLatency(us), 496.225811
[READ], AverageLatency(us), 510.206831
[READ], AverageLatency(us), 501.256123

workload a:
[READ], AverageLatency(us), 676.4527555821747
[READ], AverageLatency(us), 622.5544771452717
[READ], AverageLatency(us), 628.1365657163067


0.98.5+patch:
workload c:
[READ], AverageLatency(us), 536.334437
[READ], AverageLatency(us), 508.405555
[READ], AverageLatency(us), 491.416182


workload a:
[READ], AverageLatency(us), 640.3625218319231
[READ], AverageLatency(us), 642.9719823488798
[READ], AverageLatency(us), 631.7491770928287

looks little performance penalty.

I also ran PE in the cluster, since the test table has only 1 CF, the new lock is actually
not used. interestingly, with the patch the performance is even a bit better...

> Multi-column family BulkLoad fails if compactions go on too long
> ----------------------------------------------------------------
>
>                 Key: HBASE-11368
>                 URL: https://issues.apache.org/jira/browse/HBASE-11368
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Qiang Tian
>         Attachments: hbase-11368-0.98.5.patch
>
>
> Compactions take a read lock.  If a multi-column family region, before bulk loading,
we want to take a write lock on the region.  If the compaction takes too long, the bulk load
fails.
> Various recipes include:
> + Making smaller regions (lame)
> + [~victorunique] suggests major compacting just before bulk loading over in HBASE-10882
as a work around.
> Does the compaction need a read lock for that long?  Does the bulk load need a full write
lock when multiple column families?  Can we fail more gracefully at least?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message