hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Qiang Tian (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
Date Sat, 25 Oct 2014 10:26:34 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Qiang Tian updated HBASE-11368:
-------------------------------
    Attachment: key_stacktrace_hbase10882.TXT

Hi [~stack],
Sorry for confusing. let me explain from scratch:
1)the root cause of problem - HRegion#lock.
>From the stacktrace in HBASE-10882(also see key_stacktrace_hbase10882.TXT attached), 
the event sequence is: 
1.1)the compaction acquires the readlock of HRegion#lock, 
1.2)the bulkload try to acquire the writelock of HRegion#lock if there are multiple CFs. it
has to wait for compaction to release the readlock.
1.3)scanners try to acquire the readlock of HRegion#lock. they have to wait for the bulkload
to release the writelock.
so both bulkload and scanners are blocked on HRegion#lock by compaction.

2)what is HRegion#lock used for?
Investigation on the HRegion#lock shows, it is originally designed to protect region close
ONLY. if someone, such as region split, wants to close the region, it needs to wait for others
release the readlock.  
Then HBASE-4552 used the lock to solve the multi-CF bulkload consistency issue. now we see
it is too heavy.

3)can we not use HRegion#lock in bulkload?
the answer is yes. 
Internally, HStore#DefaultStoreFileManager#storefiles keeps track of the on-disk HFiles for
a CF. we have below steps for the bulkload:
3.1)moves HFiles directly to region directory
3.2)add them into the {{storefiles}} list
3.3)notify StoreScanner that the HFile list is changed, which is done by resetting the StoreScanner#heap
to null. this forces existing StoreScanner instances to reinitialize based on new the HFiles
seen on disk in next scan/read request.
the step 3.2 and 3.3 is synchronized by HStore#lock. so we have CF level scan-bulkload consistency.
 
To achieve multi-CF scan-bulkload consistency, if we do not use HRegion#lock, we still need
another region level lock --- a RegionScanner is composed of multiple StoreScanner, a StoreScanner(a
CF scanner) is composed of a MemStoreScanner and multiple StoreFileScanner.

the RegionScannerImpl#sortheap(and joinedHeap) is just the entry point of multiple StoreScanners.
to have multi-CF consistency, we need synchronization here - a lock is needed, but it is used
only between scan and bulkload.



Regarding the code change you referenced, performance_improvement_verification_98.5.patch
is to simulate the event sequence described in #1, for testing purpose only.

currently I use 98.5 for test since it is stable and easy to evaluate the effect of the change.
thanks.









> Multi-column family BulkLoad fails if compactions go on too long
> ----------------------------------------------------------------
>
>                 Key: HBASE-11368
>                 URL: https://issues.apache.org/jira/browse/HBASE-11368
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Qiang Tian
>         Attachments: hbase-11368-0.98.5.patch, key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch
>
>
> Compactions take a read lock.  If a multi-column family region, before bulk loading,
we want to take a write lock on the region.  If the compaction takes too long, the bulk load
fails.
> Various recipes include:
> + Making smaller regions (lame)
> + [~victorunique] suggests major compacting just before bulk loading over in HBASE-10882
as a work around.
> Does the compaction need a read lock for that long?  Does the bulk load need a full write
lock when multiple column families?  Can we fail more gracefully at least?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message