accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
Date Tue, 17 Jan 2017 14:48:16 GMT
Matt,

Using the merge command can be a little tricky.  Imagine your splits look
like this for whatever table 1 is using 'scan -t accumulo.metadata -b 1; -e
2;  -c ~tab'

1;a ~tab:~pr []    \x00
1;b ~tab:~pr []    \x01a
1;c ~tab:~pr []    \x01b
1;d ~tab:~pr []    \x01c
1;e ~tab:~pr []    \x01d
1;f ~tab:~pr []    \x01e
1;g ~tab:~pr []    \x01f
1;h ~tab:~pr []    \x01g
1;i ~tab:~pr []    \x01h
1;j ~tab:~pr []    \x01i
1;k ~tab:~pr []    \x01j
1< ~tab:~pr []    \x01k

Merging will take the entire range you specify, and put all the splits into
the last split.  Say you want to merge split c and d into e.  You might try
this command, but it would be wrong

'merge -t test1 -b c -e e'

because the -b is exclusive.  That command would merge d into e but skip
c.  A correct command would be something like

'merge -t test1 -b b\x00 -e e'

After this would end up with the following splits

1;a ~tab:~pr []    \x00
1;b ~tab:~pr []    \x01a
1;e ~tab:~pr []    \x01b
1;f ~tab:~pr []    \x01e
1;g ~tab:~pr []    \x01f
1;h ~tab:~pr []    \x01g
1;i ~tab:~pr []    \x01h
1;j ~tab:~pr []    \x01i
1;k ~tab:~pr []    \x01j
1< ~tab:~pr []    \x01k

where the begin range is a value after the ~tab:~pr value for the range you
want to start with.

Also, merges can be really slow.  Much of this comes from having to look at
every rfile and doing a special "chop" compaction to pull out what doesn't
belong in the merged range.  This can be avoided if you do a full major
compaction on the ranges you are going to merge.  This get each split down
to 1 rfile and no "chop" compaction is done.

Suggest you start slow merging one split into another to get a feel for how
it goes and how long it takes.

Good luck.

Mike

On Mon, Jan 16, 2017 at 10:57 PM, Mike Drob <mdrob@mdrob.com> wrote:

> I can't find this in the docs, but IIRC the merge command can take a
> start/end range for what to merge. So the best option might be to try it on
> a smaller slice and see what happens. At a guess, queries won't block but
> indexing will.
>
> Mike
>
> On Mon, Jan 16, 2017 at 5:23 PM, Dickson, Matt MR <
> matt.dickson@defence.gov.au> wrote:
>
>> *UNOFFICIAL*
>> That looks like a great option.  Before using it, whats the cost/impact
>> of running this on a massive table in a system with other large bulk
>> ingests/queries running?  In the past when I have used that (which was in
>> 2013 so things may have changed) all ingests were blocked and it took days
>> to complete.
>>
>> With 1.07T tablets to work on this may take some time?
>>
>>
>> ------------------------------
>> *From:* Mike Drob [mailto:mdrob@mdrob.com]
>> *Sent:* Tuesday, 17 January 2017 09:37
>> *To:* user@accumulo.apache.org
>> *Subject:* Re: Merging smaller/empty tablets [SEC=UNOFFICIAL]
>>
>> http://accumulo.apache.org/1.8/accumulo_user_manual.html#_merging_tablets
>>
>> In order to merge small tablets, you can ask Accumulo to merge sections
>> of a table smaller than a given size.
>>
>> root@myinstance> merge -t myTable -s 100M
>>
>>
>>
>> On Mon, Jan 16, 2017 at 4:31 PM, Dickson, Matt MR <
>> matt.dickson@defence.gov.au> wrote:
>>
>>> *UNOFFICIAL*
>>> I have a table that has evolved to have 1.07T tablets and I fairly
>>> confident a large portion of these are now empty or very small.  I'd like
>>> to merge smaller tablets and delete empty tablets, is there a smart way to
>>> do this?
>>>
>>> My thought was to query the metadata table for all tablets under a
>>> certain size for the table and then merge these tablets.
>>>
>>> Is the first number in thevalue the size of the tablet, ie
>>>
>>> >  scan -b 1xk -e 1xk\xff -c file
>>> 1xk;34234 file:hdfs//name/accumulo/tables/1xk/t-er23423/M423432.rf
>>> []    *213134*,234234
>>>
>>> Also, are there any side effects of this that I need to be aware of when
>>> doing this on a massive table?
>>>
>>> Thanks in advance,
>>> Matt
>>>
>>
>>
>

Mime
View raw message