Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which
 is an MX secondary for dev@accumulo.apache.org)
Message-ID: <5548E68E.4020900@gmail.com>
Date: Tue, 05 May 2015 11:49:34 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Macintosh/20140602)
MIME-Version: 1.0
To: dev@accumulo.apache.org
Subject: Re: Ingest speed
References: <1430818553916-14005.post@n5.nabble.com>
 <5548CF37.2050308@gmail.com> <1430839974510-14013.post@n5.nabble.com>
In-Reply-To: <1430839974510-14013.post@n5.nabble.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On a single node, you can easily achieve 10s of thousands of key-value 
inserts per second. Depending on how many columns are in each row, 600 a 
second is rather slow :)

Your loop looks good. Using a single BatchWriter and letting it amortize 
sending data from your client to the servers will be the most efficient.

If the JSON parsing is the slowest part, you could consider a single 
thread reading the file and provide the line to a thread pool, parse the 
line and add it to some concurrent data structure. You could have a 
consumer on that data structure reading each parsed object and sending 
it to Accumulo.

Alternatively, this is where MapReduce is a clear win as it's very good 
at parallelizing these types of problems. You could use the 
FileInputFormat and the AccumuloOutputFormat to accomplish this task.

Andrea Leoni wrote:
> Thank you for your answer.
> Today i tried to create a big command file and push it to shell (about 300k
> insert per file). As you said it is too slow for me (about 600 inserted
> row/sec)
>
> I'm on Accumulo by just one week. I'm a noob but i'm learning.
>
> Actually my app has to store a large number of data.
>
> The row is the timestamp and the family/qualif are the column... I catch my
> data from a JSON file, so my app scan it for new records, parse it and once
> for record create a mutation and push it on Accumulo with batchWriter...
>
> Maybe I wrong something that can increase the speed of my inserts.
>
> Actually I:
>
> LOOP
> 1) read a json line
> 2) parse it
> 3) create a mutation
> 4) put in this mutation the line's information
> 5) use batchWriter to insert mutation in Accumulo
> END LOOP
>
> Is it all right? I now that point 1) and 2) are slow but it's necessary and
> i use the fastest json parser i've found online.
>
> Thank you so much again!
> (and sorry again for my bad english!)
>
>
>
> -----
> Andrea Leoni
> Italy
> Computer Engineering
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/Ingest-speed-tp14005p14013.html
> Sent from the Developers mailing list archive at Nabble.com.