hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakhi Khatwani <rakhi.khatw...@gmail.com>
Subject Re: Region Servers going down frequently
Date Wed, 08 Apr 2009 05:15:01 GMT
Hi Andy,
             Thanks for your suggesstion.
But i was wondering how could we seperate HDFS storage from Mapred
Computations. as mapred uses the same master/slave configuration as HDFS.

did you mean using a set of instances as slaves and another set of instances
as regionservers.??

Thanks in Advance,
Rakhi


On Tue, Apr 7, 2009 at 11:06 PM, Andrew Purtell <apurtell@apache.org> wrote:

>
> Hi Rakhi,
>
> The "cannot obtain block" error is actually a HDFS problem. Most
> likely this block was lost by HDFS during a period of excessive
> load. Usually the first sign you are using insufficient
> resources for your load is filesystem issues such as these. To
> address the problems I recommend you do two things at once.
>
> 1) The minimum usable instance type for HBase (and Hadoop) is
> large in my opinion. The basic rule of thumb for HBase and
> Hadoop daemons is you must allocate 1GB of heap/RAM and one
> CPU (or vcpu) thread for each daemon. You can search the
> hbase-user@ archives for previous discussion on this topic.
>
> 2) Allocate more instances to spread the load on DFS.
>
> On EC2 I recommend running storage such as HDFS/HBase on one set
> of instances and mapreduce computations on another set. Hadoop
> and HBase daemons are sensitive to thread starvation problems.
>
> Hope this helps,
>
>   - Andy
>
> > From: Rakhi Khatwani
> > Subject: Region Servers going down frequently
> > Date: Tuesday, April 7, 2009, 2:45 AM
> > Hi,
> >       I have a 20 node cluster on ec2(small instance).... i
> > have a set of tables which store huge amount of data (tried
> > wid 10,000 rows... more to be added).... but during my map
> > reduce jobs, some of the region servers shut
> > down thereby causing data loss, stop in my program
> > execution and infact one of my tables got damaged. when ever
> > i scan the table, i get the could not obtain block error.
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message