accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jimmy Lin <jimmys.em...@gmail.com>
Subject Increasing Ingest Rate
Date Thu, 04 Apr 2013 18:01:29 GMT
Hello,
I am fairly new to Accumulo and am trying to figure out what is preventing
my system from ingesting data at a faster rate. We have 15 nodes running a
simple Java program that reads and writes to Accumulo and then indexes some
data into Solr. The rate of ingest is not scaling linearly with the number
of nodes that we start up. I have tried increasing several parameters
including:
- limit of file descriptors in linux
- max zookeeper connections
- tserver.memory.maps.max
- tserver_opts memory size
- tserver.mutation_queue.max
- tserver.scan.files.open.max
- tserver.walog.max.size
- tserver.cache.data.size
- tserver.cache.index.size
- hdfs setting for xceivers
No matter what changes we make, we cannot get the ingest rate to go over
100k entries/s and about 6 Mb/s. I know Accumulo should be able to ingest
faster than this.
Thanks in advance,

Jimmy Lin

Mime
View raw message