accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Gresock <jgres...@gmail.com>
Subject Accumulo advice
Date Tue, 10 Dec 2013 18:08:38 GMT
I'm forwarding this email chain to the user group, since it was so helpful
to our Accumulo cluster setup.  The original post is at the bottom.

Thanks to Josh Elser!

 To answer your question:
> - We'll have probably several times more read traffic than write.  Bulk
> ingest will be more rare than a generally slow ingest rate and a high
> read rate.
>

Ok -- Then I'd probably say to reduce the in-memory maps to 1G and give
more to tserver Java heap, and ultimately have more you can throw at the
caches


 As for monitoring the cache hit rate, what exactly am I looking for?  I
> guess I want as high a hit rate in the cache as possible?
>

There should be some graphs on the landing page of the Accumulo monitor at
the bottom for the cache hit rates (I think). High (near 100%) would be
desirable. For all of your reads, you want to hit those cached indexes
instead of having to re-read them. Having some data blocks be cached as
well is nice, but you don't have near enough ram to get a really good hit
rate.


 One more question -- do you think it will be easy to upgrade to 1.5
> without disrupting the data?  Do you know if 1.5 requires specific
> versions of zk and hadoop?
>

IIRC, there aren't any special upgrade procedures -- 1.5 should pick up any
dangling things from 1.4 but a clean shut down should alleviate most of
this. The biggest gain is that 1.5 does away with the loggers, so you can
completely remove them from your equation and give that memory to the
tserver.

As long as you're not using a 0.20 release, you should be fine for Hadoop.
I'd recommend Apache Hadoop 1.2.1 or even 2.2.0 if you don't mind making
the the Hadoop2 jump. If anything, you might have to recompile Accumulo
with the version of Hadoop you're targeting, but this is easy (we have the
options built into the build process and we can follow through on this
later if necessary). Zookeeper doesn't matter, the latest on their 3.3 or
3.4 line should both be fine.

>
>
> On Tue, Dec 10, 2013 at 11:11 AM, Josh Elser <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com>> wrote:
>
>     Hi Joe!
>
>     Let's see -- You can probably set ZooKeeper to have an Xmx of 2G,
>     but I highly doubt you'll see them even rise up to 1G resident. That
>     should give you enough head room to throw 4G at the master, the
>     namenode, and the snn. I wouldn't be as worried about making sure
>     you have free memory for the OS (as you're not running on baremetal
>     anyways), but that may be something to keep an eye on (and would be
>     easy for you to give those master nodes an extra GB or two.
>
>     For the workers, obligatory "_please_ switch to 1.5". It will make
>     your life easier -- trust me. The memory allocations, namely the
>     caching, is going to be dependent on your workload -- write once,
>     read many? heavy write, light read? somewhere in the middle?
>
>     Each slave has ~150GB of space then? I would guess that 4G to the DN
>     is a little high. You could probably back that down to 2G and watch
>     the usage.
>
>     As far as using native maps by default, it depends on your
>     installation method. When you start up Accumulo, you'll see a
>     message near the top of the tserver log file that says if it loaded
>     the native library. If you don't see this, you're not using them.
>
>     tserver.walog.max.size: The default here should be fine (1G, I
>     think). The only way this is going to affect you is if you have
>     frequent failures, so you likely shouldn't have to worry about it.
>
>     logger.sort.buffer.size: Again, should only be used in the face of
>     failure. The default should be fine.
>
>     tserver.cache.index.size and tserver.cache.data.size: These are both
>     calculated from the amount of Java heap you give the tserver. The
>     index cache is a good idea, and the data cache is a nice to have,
>     but not necessary. The heavier the read load, the better it is to
>     try to increase these while lowering the inmemory maps. A reduced
>     logger sort buffer is likely an attempt to try to keep the loggers
>     from running out of memory but, if you're not killing processes,
>     this shouldn't be a big deal.
>
>     Maybe a good starting point is to take 1/4 of your JVM heap and give
>     that to the index and data cache. Then, of that amount, give at
>     least 512M to the index cache, and the rest to the data cache.
>     That's a pretty big guess though. This also isn't anything that's
>     going to break your system if you screw it up. Use the monitor and
>     see what kind of cache hit/miss rates you're getting and go from there.
>
>     I would probably make the tserver's heap be 4G and give 2G to the
>     native maps. A 4G native map is pretty big for the "small" VMs
>     you're running on. Again, though, that's dependent on your workload.
>
>     Take care!
>
>     - Josh
>
>     On 12/10/13, 9:52 AM, jgresock@gmail.com <mailto:jgresock@gmail.com>
>
>     wrote:
>
>         Hi Josh,
>
>         I found a post of yours on the low side that helped me
>         understand some basic Accumulo memory settings:
>         http://apache-accumulo.__1065345.n5.nabble.com/__
> Tabletserver-message-quot-__Running-low-on-memory-quot-__td6373.html
>
>         <http://apache-accumulo.1065345.n5.nabble.com/
> Tabletserver-message-quot-Running-low-on-memory-quot-td6373.html>
>
>         I was wondering if you could give me some advice on setting up a
>         cluster.  We have 3 "master" VMs with 6G RAM and 4 cores, and 8
>         "worker" VMs with 16G RAM and 8 cores.  The workers have a total
>         of 1.2T of disk space among them.  We don't plan on running any
>         MR jobs, we're just using Accumulo as a data store.
>
>         My current plan is:
>         master-1 (6G RAM): Accumulo master (2G), zookeeper (not sure how
>         to set mem)
>         master-2 (6G RAM): Namenode (2G), zookeeper
>         master-3 (6G RAM): Secondary namenode (2G), zookeeper
>         worker-1-8 (16G RAM): Datanode (4G), tserver (6G total), logger
> (2G)
>
>         For tservers, I was going to set -Xmx2g in the TSERVER_OPTS, and
>         set tserver.memory.maps.max to 4G.
>
>         My main questions are:
>         1) Does this sound like an effective setup for a cluster?
>         2) I assume we're by default using native maps, since we didn't
>         explicitly specify otherwise .. is this true?
>         3) I don't really know how to configure the remaining settings,
>         like tserver.cache.index.size, tserver.cache.data.size,
>         tserver.walog.max.size, and logger.sort.buffer.size.  Any
>         recommendations for our cluster size would be greatly appreciated!
>
>         I think what I'm not sure of, with regard to #3, is that we see
>         that some of the settings in our accumulo-site.xml have been set
>         lower than the defaults for the ones I mentioned.  I just don't
>         know what they should be.
>
>         Anyway, sorry for the brain dump.  Feel free to refer me to
>         someone else, too.
>
>         Thanks!
>         Joe
>
>
>
>
>         _______________________________________
>         Sent from http://apache-accumulo.__1065345.n5.nabble.com
>
>         <http://apache-accumulo.1065345.n5.nabble.com>
>
>

-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*

Mime
View raw message