hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiamin Lu <jiamin....@googlemail.com>
Subject Re: How many real cores in large type instances on EC2 ?
Date Sat, 26 Nov 2011 15:30:24 GMT
Hi, all

Thanks for the help that I got from this mailing list, the link given by
Harsh is very helpful, thank you very much. Here is my conclusion.
I think Amazon didn't explain this question well, and some concepts are
also confused.

First, the core that I understand is a processor chip inside a multi-core
CPU, but in Amazon, a core means a physical CPU which contains several
ECUs. That's why they say the cores in High-CPU Medium Instance contain 2.5
ECU each. I agree to Rebert, that the ECU is equal to the concept of
processing power, and the power of each ECU is similar with a 1.0 ~1.2 GHz
2007 Opteron or 2007 Xeon Processor.

This is really confuse, especially when people find the EC2 small instance
contains a 2.6GHz cpu, and the large instance only contains two 2.0GHz cpu,
though Amazon claims that the large one should be 4 times as powerful as
the small one. The reason here is that the 2.6GHz cpu in small instance is
a dual-core processor, contains 2 ECU, and it only can use the half power
of it, i.e., one ECU. But in the large instance, we can use the full power
of these processors, i.e. four ECUs together.

Another confusing thing is that in my own computer, which has a 6-core
processor, I can see clearly that it has 6 items listed in /proc/cpuinfo,
and I can see the usage of each core in top window. But in EC2, I can only
see 2 physical cpu information there, even though it's true has 4
cores(ECU) in total. The only difference is that the speed of each core is
800MHz in my computer, and it's 2004MHz in EC2 instance.

I did some experiment by running 4 similar processes in parallel on one EC2
large instance, then I find that each process can almost evenly take 50%
power of each cpu. I think that's means we do can fully use these 4 ECUs to
process tasks independently.

Thanks again, and hope this conclusion can help the other people who are
also interested about EC2 instances, which I think is really good place for
building a temporary large scale cluster.

On Fri, Nov 25, 2011 at 9:56 AM, Robert Hafner <tedivm@tedivm.com> wrote:

> You're looking at two different things. The number of cores is one thing,
> and the power of each core another. Number of cores is pretty straight
> forward, and the power is relative to that of a small instance. You've got
> the answer right there in your email- there are two virtual cores
> of two compute units each, so you will only see two cores. Those cores are
> twice as powerful as the typical core (hence the two compute units), but
> they are still only going to show two cores.
> This does have a bit of an effect on processing jobs. If you need more CPU
> power behind your tasks than a large instance will be nice, but if your
> bottleneck is something else (memory, disk i/o, s3 access, etc) then you'll
> probably be optimizing for the wrong problem. In my experience it's better
> to have four small instances running in a job than one large instance, even
> though the cost is equivalent. If you're using spot instances- which are a
> huge budget saver- then the small instances make even more sense, since
> lost machines will only result in one lost task instead of four.
> Robert
> On Nov 24, 2011, at 7:40 AM , Jiamin Lu wrote:
> Hi, all
> I am using the Amazon EC2, with their large instances.
> Amazon claims these large type instances have 4 EC2 Compute units (2
> virtual cores with 2 EC2 Compute Units each).
> But according to my observation, it seems like they only have two cores.
> I checked the /proc/cpuinfo, which shows there are only two processors,
> I also used the *top* command, and it also says only two cpu there.
> Can someone tell me actually how many cores are contained inside these
> large instances?
> Did I misunderstand these terms that Amazon talks about ??
> Thanks
> Jiamin Lu

Jiamin Lu
Fernuniversität Hagen
D-58084 Hagen

Phone: +49-2331-987-4276
Email: Jiamin.Luu@gmail.com

View raw message