hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-1477) Contention on META stalls mapreduce job execution
Date Tue, 02 Jun 2009 19:39:08 GMT
Contention on META stalls mapreduce job execution
-------------------------------------------------

                 Key: HBASE-1477
                 URL: https://issues.apache.org/jira/browse/HBASE-1477
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: Andrew Purtell


>From  Jeremy Pinkham up on hbase-users@: 

bq. A typical mapper in the job takes several minutes, how many minutes depends on whether
I use the the region partitioner and how many I let run concurrently... it's been anywhere
from 2 minutes with no partitioner and small concurrency (5 mappers) to 8 minutes with the
region partitioner and high concurrency (150 mappers).  This seems to directly correlate with
how long it takes to do a simple count of .META. while each job is running (2 seconds to 1
minute)

bq. I was able to get past this issue affecting my data load by reorganizing some of my workflow
and data structures to force the ordering of keys without the region partitioner.  Those changes
appear to have side stepped the problem for me as I can now load from 100+ mappers without
seeing the degradation that I was seeing with 40 when using the partitioner (and getting some
sweet numbers in the requests column of the UI).  It's still an interesting scaling situation
with the region partitioner, but I'm good to go without it.

I have seen this also in the form of freezing of master UI during high load, where the UI
comes back as soon as load is reduced. When I thread dump it looks like all IPC handlers on
the region server hosting .META. are busy. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message