accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlmar...@comcast.net
Subject Re: Configuring batch writers
Date Fri, 15 Jul 2016 18:33:58 GMT
The batch writer has several knobs (latency time, memory buffer, etc) that you can tune to
meet your requirements. The values for those settings will depend on a lot of variables, to
include: 

- number of tablet servers 
- size of mutations 
- desired latency 
- memory buffer 
- configuration settings on the table(s) and tablet servers. 

Suggest picking a starting point and see how it works for you, such as 

threads - equal to the number of tablet servers (unless you have a really large number of
tablet servers) 
buffer - 100MB 
latency - 10 seconds 

If you are hitting a wall with those settings, you could increase the buffer and latency and/or
change some settings on the server side that have to do with the write ahead logs. 

----- Original Message -----

From: "Jamie Johnson" <jej2003@gmail.com> 
To: user@accumulo.apache.org 
Sent: Friday, July 15, 2016 2:16:40 PM 
Subject: Configuring batch writers 

Is there any documentation that outlines reasonable settings for batch writers given a known
ingest rate? For instance if I have a source that is producing in the neighborhood of 15MB
of mutations per second, what would a reasonable configuration for the batch writer be to
handle an ingest at this rate? What are reasonable rules of thumb to follow to ensure that
the writers don't block, etc? 


Mime
View raw message