accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Slacum <wilhelm.von.cl...@accumulo.net>
Subject Re: Mapreduce output format killing tablet servers
Date Wed, 25 Jun 2014 18:27:01 GMT
I had a similar thread going on and am currently rummaging through the
batch writer code (as well as pontificating on how the tablet server
handles multiple write clients for the tablet).

What is your ingest skew like? Is it uniform? How quickly do splits occur?
I've seen, at relatively low scale, doing "live" ingest become problematic.

Have you looked into using file output? One of our committers, Cory, has a
library that can handle writing to multiple tables/files. You can peek
here: https://github.com/calrissian/accumulo-recipes (doing a `find . -name
'Group*'` will give you the classes you need). I had to do some massaging
to get them to work properly and am happy to share what I had to do if this
becomes a route you're interested in.


On Wed, Jun 25, 2014 at 2:10 PM, Sean Busbey <busbey@cloudera.com> wrote:

> What version of Accumulo?
>
> What version of Hadoop?
>
> What does your server memory and per-role allocation look like?
>
> Can you paste the tserver debug log?
>
>
>
> On Wed, Jun 25, 2014 at 1:01 PM, Jacob Rust <jrust@clearedgeit.com> wrote:
>
>> I am trying to create an inverted text index for a table using accumulo
>> input/output format in a java mapreduce program.  When the job reaches the
>> reduce phase and creates the table / tries to write to it the tablet
>> servers begin to die.
>>
>> Now when I do a start-all.sh the tablet servers start for about a minute
>> and then die again. Any idea as to why the mapreduce job is killing the
>> tablet servers and/or how to bring the tablet servers back up without
>> failing?
>>
>> This is on a 12 node cluster with low quality hardware.
>>
>> The java code I am running is here http://pastebin.com/ti7Qz19m
>>
>>  The log files on each tablet server only display the startup
>> information, no errors. The log files on the master server show these
>> errors http://pastebin.com/LymiTfB7
>>
>>
>>
>>
>> --
>> Jacob Rust
>> Software Intern
>>
>
>
>
> --
> Sean
>

Mime
View raw message