hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: 答复: flushing + compactions after config change
Date Fri, 28 Jun 2013 16:31:25 GMT
On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria <viral.bajaria@gmail.com> wrote:
> Hey JD,
> Thanks for the clarification. I also came across a previous thread which
> sort of talks about a similar problem.
> http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3CCAGpTDNfWNRsNqV7n3wgjE-iCHZPx-CXn1TBchgwRPOhgcoS+bw@mail.gmail.com%3E
> I guess my problem is also similar to the fact that my writes are well
> distributed and at a given time I could be writing to a lot of regions.
> Some of the regions receive very little data but since the flush algorithm
> choose at random what to flush when "too many hlogs" is hit, it will flush

It's not random, it picks the region with the most data in its memstores.

> a region with less than 10mb of data causing too many small files. This
> in-turn causes compaction storms where even though major compactions is
> disabled, some of the minor get upgraded to major and that's when things
> start getting worse.

I doubt that it's the fact that it's a major compaction that it's
making everything worse. When a minor gets promoted into a major it's
because we're already going to compact all the files, so we might as
well get rid of some deletes at the same time. They are all getting
selected because the files are within the selection ratio. I would not
focus on this to resolve your problem.

> My compaction queues are still the same and so I doubt I will be coming out
> of this storm without bumping up max hlogs for now. Reducing regions per
> server is one option but then I will be wasting my resources since the
> servers at current load are at < 30% CPU and < 25% RAM. Maybe I can bump up
> heap space and give more memory to the the memstore. Sorry, I am just
> thinking out loud.

I haven't been closely following this thread, but have you posted a
log snippet somewhere? It's usually much more telling and we eliminate
a few levels of interpretation. Make sure it's at DEBUG, and that you
grab a few hours of activity. Get the GC log for the same time as
well. Drop this on a web server or pastebin if it fits.



View raw message