Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of imyousuf@gmail.com designates
 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=x3dZkGYnOzKqozuRpjWoyUzs6S9ysjRFhzJJ/AUgYgurl343djXS365RlkG5kbxwOF
         4++btV+d3glk+jSU611G92s3AIVaETU5ivCO8SHCw32Nay3zEXZ3xW40QV+2bH2YW9Pm
         QnwdwxP85jkxf5+fhjahJoN0wIUyRD8lFCFq4=
MIME-Version: 1.0
In-Reply-To: <4BBA2938.4030404@apache.org>
References: <u2p7bfdc29a1004050952pc298ae71u6f6596cf3dff2218@mail.gmail.com>
	 <8D66B74984F9564BBB25C3C67D630F2D6671FDA1@SC-MBXC1.TheFacebook.com>
	 <4BBA2938.4030404@apache.org>
Date: Tue, 6 Apr 2010 06:09:03 +0545
Message-ID: <n2y7bfdc29a1004051724h3e85ac7br4dd89002adab1742@mail.gmail.com>
Subject: Re: About test/production server configuration
From: Imran M Yousuf <imyousuf@gmail.com>
To: hbase-user@hadoop.apache.org
Cc: Jonathan Gray <jgray@facebook.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 6, 2010 at 12:02 AM, Patrick Hunt <phunt@apache.org> wrote:
> The ZK servers are sensitive to disk (io) latency. I just troubleshot an
> issue last week where a user was seeing 80second (second!) latencies.
> Turns out they were running zk server, namenode, tasktracker, and hbase
> region server all on the same box, that box had a single spindle for all
> io activity and was at 100% utilization for long periods of time. If
> you want decent ZK API latencies (<100ms) you really need to ensure that
> there's at least a separate spindle available for the ZK transaction logs=
.
>

Great insight and info! So that means if ZK is using a separate
spindle the rest can reside in the same spindle, at least in your
case, nice!

Thanks for the info, noting it.

Imran

> Patrick
>
> On 04/05/2010 11:11 AM, Jonathan Gray wrote:
>>
>> Imran,
>>
>> It's impossible to give good advice on cluster size and hardware
>> configuration without some idea of the requirements.
>>
>> How much data? =A0How will the data be queried? =A0What kind of load do
>> you expect? =A0You are going to be doing offline batch/MapReduce,
>> online random access, as well as search all from the same nodes?
>> This can be dangerous.
>>
>> I would strongly recommend against putting Hadoop+HBase on the same
>> nodes as something like Solr, unless you have dedicated disks for
>> each. =A0Also, don't forget about ZooKeeper which you definitely will
>> need separate nodes/disks for if you will be co-locating so many
>> other things.
>>
>> JG
>>
>>> -----Original Message----- From: Imran M Yousuf
>>> [mailto:imyousuf@gmail.com] Sent: Monday, April 05, 2010 9:52 AM
>>> To: hbase-user@hadoop.apache.org Subject: About test/production
>>> server configuration
>>>
>>> Hi,
>>>
>>> We are a startup who have decided to use HBase purely because we
>>> want to take advantage of HDFS based reliability, redundancy,
>>> MapReduce and BigTable. For that we are thinking to go for a test
>>> environment with 5 servers and production environment with 10
>>> servers in both case the Hadoop cluster will be used for HBase +
>>> MapReduce + Solr Index.
>>>
>>> Firstly, I would like some opinion on whether 10 servers is a good
>>> number for all 3 purposes or not. Secondly what kind of test
>>> environment is currently in use in different organizations.
>>> Thirdly, I would like to learn some server configuration and
>>> purchase price (with purchase location if possible).
>>>
>>> Waiting eagerly for some feedback.
>>>
>>> Thank you,
>>>
>>> -- Imran M Yousuf Entrepreneur& =A0Software Engineer Smart IT
>>> Engineering Dhaka, Bangladesh Email: imran@smartitengineering.com
>>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile:
>>> +880-1711402557
>


--=20
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557