accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <busbey+li...@cloudera.com>
Subject Re: Persistent WAL files on 1.5
Date Sun, 30 Mar 2014 14:42:42 GMT
Hi Chris!

If you look at the "master" page on the monitor, does it show a large
number of entries in memory?

If you look at the "garbage collector" page on the monitor, what does it
report for the last cycle?

Can you upload the log file(s) for the GC and a tablet server somewhere?

-Sean
On Mar 30, 2014 9:09 AM, "Chris Bennight" <chris@slowcar.net> wrote:

> Cluster is a 5 node VM based  accumulo 1.5 / cdh 4.5 instance.
>  Replication factor of 2.
>
> It's a dev instance, so nothing critical (though I would like to not loose
> data there as it represents a week or so to re-ingest & process)
>
> It recently ran out of space during in ingest, so I cleared out some
> tables which were no longer being used.    I didn't recover much of the
> free space, and really the total usage  (~6TB seemed  much higher than the
> number of entries (~50Billion) - knowing that none of the entries were
> especially large)
>
> -bash-4.1$ hadoop fs -du -h /accumulo/
> 0            /accumulo/instance_id
> 58.5K     /accumulo/lib
> 3.5G      /accumulo/recovery
> 118.2G  /accumulo/tables
> 0           /accumulo/version
> 2.5T      /accumulo/wal
>
> -bash-4.1$ hadoop fs -du -h /accumulo/wal/
> 495.2G     /accumulo/wal/10.10.10.51:9997
> 541.3G     /accumulo/wal/10.10.10.52:9997
> 515.7G     /accumulo/wal/10.10.10.53:9997
> 474.3G     /accumulo/wal/10.10.10.54:9997
> 562.5G     /accumulo/wal/10.10.10.55:9997
>
>
> As I mentioned, it's a dev cluster so it's entirely possible some wierd
> confluence of events happened previously to cause this - what I'm more
> concerned about is how to I recover that space.   I'm not worried at this
> point about any information that might be in the WAL files.
>
> Accumulo itself has been restarted a few times for various reasons.
>
> The only notable log entry is in the tserver log file are the
> [tabletserver.TabletServer] WARN : Running low on memory
> occuring ~ 15 times a second.   Tserver memory settings don't seem to
> impact this  (8GB allocated to tservers, bloom filters are on, as are block
> cache (2GB), index cache (1GB), native memory maps (1GB)
>
> Otherwise I don't see anything out of norm in the master, monitor, gc, or
> tracer files (on master)
>
>
>

Mime
View raw message