accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Massimilian Mattetti" <MASSI...@il.ibm.com>
Subject Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1
Date Thu, 23 Mar 2017 15:31:30 GMT
The tserver.compaction.major.thread.files.open.max=10 and I have a maximum 
of 30 files per tablet (table.file.max=30). What property are you 
referring to for the max open files?
Thanks,
Max 




From:   "Marc P." <marc.parisi@gmail.com>
To:     user@accumulo.apache.org
Date:   23/03/2017 16:37
Subject:        Re: tserver.compaction.*.concurrent.max behavior in 
Accumulo 1.8.1



Max,
  Additionally, what are your max open files? If you are under heavy 
ingest you may exceed this with compactions if they cannot keep up. 
Additionally, how many files per compaction are you allowing? 

On Thu, Mar 23, 2017 at 10:27 AM, Michael Wall <mjwall@gmail.com> wrote:
Max,

On you 3 node cluster, how many tables are you ingesting into?  How many 
tablets are in each table?  Are the tablets equally spread amongst the 3 
tablet servers?

Mike

On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti <MASSIMIL@il.ibm.com
> wrote:
With the configuration I presented before the concurrent major compactions 
are never more than 3 per tablet server while the minor are under the 4 
per node. Can one of the other configurations be the cause of this 
behavior?

Regards,
Max



From:        Dave Marion <dlmarion@comcast.net>
To:        user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
Date:        23/03/2017 14:55
Subject:        Re: tserver.compaction.*.concurrent.max behavior in 
Accumulo 1.8.1



Can you explain more what you mean by "My problem is that both the minor 
and major compactions do not overcome their default max values?" I have 
done some testing with 1.8.1 and specifically modifying 
tserver.compaction.major.concurrent.max to a higher number and I have seen 
it take effect.

On March 23, 2017 at 7:54 AM Massimilian Mattetti <MASSIMIL@il.ibm.com> 
wrote:

Hi All,

I am running some heavy ingestion process on a 3 nodes cluster of Accumulo 
1.8.1, using the following configuration:

table.compaction.minor.logs.threshold=10
table.durability=flush
table.file.max=30

tserver.wal.blocksize=2G
tserver.walog.max.size=4G
tserver.mutation.queue.max=2M
tserver.memory.maps.max=4G
tserver.compaction.minor.concurrent.max=50
tserver.compaction.major.concurrent.max=8

My problem is that both the minor and major compactions do not overcome 
their default max values. I checked the config from the shell and it looks 
fine to me:

default    | tserver.compaction.major.concurrent.max ................ | 3
system    |                     @override 
........................................... | 8

default    | tserver.compaction.minor.concurrent.max ............... | 4
system    |                     @override 
........................................... | 50

Is something changed from 1.8.0? I haven't seen such behavior with the 
previous version. 
Thanks.

Regards,
Max









Mime
View raw message