hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@yahoo-inc.com>
Subject Re: NameNode memory usage and 32 vs. 64 bit JVMs
Date Mon, 10 Nov 2008 20:00:12 GMT
On 11/10/08 1:30 AM, "Aaron Kimball" <aaron@cloudera.com> wrote:
> It sounds like you think the 64- and 32-bit environments are effectively
> interchangable. May I ask why are you using both? The 64bit environment
> gives you access to more memory; do you see faster performance for the TT's
> in 32-bit mode? Do you get bit by library compatibility bugs that others
> should watch out for in running a dual-mode Hadoop environment?

Some random thoughts on our mixed environment:

A) The vast majority of user provided (legacy) code is 32-bit.  Since you
can't mix 64 and 32 bit objects at link or runtime, it just makes sense for
us to run TTs, etc, by default as 32-bit to give us the most bang for our

B) In the case of the data node, the memory usage is small enough that the
64-bit JVM isn't needed.

C) Since we currently run HOD, it should be possible for users to switch
their bit-ness and I think we have a handful of users that do.  We'll
probably lose this capability when we go back to a static job tracker. :(

D) For streaming jobs, the bit-ness of the JVM is irrelevant.  32-bit is
better due to the smaller footprint since streaming jobs eat memory like it
was candy. :)

E) We load the 64-bit and 32-bit versions of libraries on our nodes, thus
allowing us to move our bit-ness whenever we like.  This makes for a fat
image (so no RAM disk for the OS for us!), but given the streaming VM
issues, it works out mostly in our favor anyway.

    In general, 64-bit code runs slower than 32-bit code.  So unless one
needs to access more memory or has external dependencies (JNIs, whatever),
32-bit for your Java environment is the way to go. The name node and maybe a
static job tracker are the potential problem children here and places where
I suspect most people will be using the 64-bit JVM.

View raw message