hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Tom <work2m...@gmail.com>
Subject hbase regions behaving badly
Date Wed, 11 Nov 2015 04:09:51 GMT
I am on CDH 5.4.8 hbase 1.0

I am migrating to a new cluster and the data is sourced from CDH 5.3 hbase
0.98.6

I took snapshots from the old cluster exported them to the new cluster - no
errors

cloned the table, flush, major_compact and balanced the regions

the configs are practically identical to the old cluster



hbck and fsck all check out with no inconsistencies found



yet when we try to load data via thrift the regions start splitting at an
outrageous pace.

Eventually the regions become offline starting a snowball affect.  The
region exits  are forcing the data to start transferring to other regions,
with that creates large GC activity and multiple regions in transition
before thrift can no longer execute any writes.  This usually happens after
a few records.   How we put is that we perform rowkey scans before puts to
ensure where we last left off.



Thrift eventually stops writing due to regions being offline and no longer
serving.  The regions eventually stop splitting and all the GC eventually
complete.  Once things settle back down, I am able to perform hbck and fsck
and it again comes back with no inconsistencies



I need ideas as to what is causing this


-- 
Abraham Tom
Email:   work2much@gmail.com
Phone:  415-515-3621

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message