Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <4AF9A305.5040506@apache.org>
Date: Tue, 10 Nov 2009 09:29:41 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: zookeeper-user@hadoop.apache.org
Subject: Re: ZK on EC2
References: <C71A2C11.28294%mahadev@yahoo-inc.com>
 <OF1B89DB71.49CD1CC9-ON8825766A.000199CF-8825766A.0001EC66@us.ibm.com>
 	<c7d45fc70911091624o5fd1c37am9f7455f333dd7c75@mail.gmail.com>
 	<4AF8B9DF.50002@apache.org>
 <c7d45fc70911091719u4ca6f12cl56f5ab79fbf45624@mail.gmail.com>
 	<4AF8FE63.4060905@apache.org>
 <c7d45fc70911100915t3142c4bfm951feb76d6fdf036@mail.gmail.com>
In-Reply-To: <c7d45fc70911100915t3142c4bfm951feb76d6fdf036@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Ok, good. Based on the comparison of perf numbers, and Ted's experience 
with large instances on ec2 running zk, this makes sense to me. A large 
is about half (very roughly) the horsepower of what I was using for my 
tests. Ted mentioned that he hasn't seen issues on ec2 running with 
large instances and that correlates to these numbers (again, this is all 
rough back of the envelope type stuff but good enough imo).

Anyone have a small that they could run the same cpu/disk/network tests? 
  I'd be interested to see how that stacks up.

Ted, could you provide your configuration information for the cluster 
(incl the client timeout you use), if you're willing I'd be happy to put 
this up on the wiki for others interested in running in EC2.

Thanks!

Patrick

Ted Dunning wrote:
> I only have one large instance live.  My impression from previously is that
> between host bandwidth is generally about what you saw.  We have been able
> to sustain 20-30MB/s into EC2 to a single node which should be harder than
> moving data between nodes.  I have heard rumors that others were able to get
> double what I got for incoming transfer.
> 
> On Mon, Nov 9, 2009 at 9:47 PM, Patrick Hunt <phunt@apache.org> wrote:
> 
>> Could you test networking - scping data between hosts? (I was seeing
>> 64.1MB/s for a 512mb file - the one created by dd, random data)
>>
> 
> 
>