hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikael Sitruk <mikael.sit...@gmail.com>
Subject Re: manual trigger of major compaction does not seems to work on 0.92
Date Fri, 24 Feb 2012 09:32:00 GMT
I triggered several time major_compaction between 22:30 and the time i
wrote the original email 00:58
>From the log of one of the severs between those time no major compaction
was running a major_compaction ended at 22:23 on the same server but this
is because the number of file was high (after a load phase) and not
manually triggered.
On the system major_compaction are disabled.

>From the logs:
1. 22:23:42 --> 22:23:48 - single major compaction runs
2. 22:23:48 --> 23:44:27 - minor compaction runs
3. 23:44:27 --> 23:44:32 - single major compaction runs
5. 23:44:32 --> 01:13:49 - minor compaction are run
5. 01:13:49 the cycle of major compaction began to run (heavily working
only from 01:40:).
-------------------------------
*below a grep of compaction (ends, with major and minor to illustrate the
above)*

2012-02-23 22:23:48,272 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 1 file(s) in PI of
TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d82840e84eb870a6.
into 7400a662b5924d3baf3ac520b1eb7ad9, size=49.0m; total size for store is
49.0m
2012-02-23 22:25:38,477 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef.
into 77c43127e38846f78e7ada302d399732, size=540.8m; total size for store is
2.1g
2012-02-23 22:27:41,930 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000003951999,1329399349764.f55cfda6067ad1425ba68d20d82c945b.
into f9c31ac78b3d4feb848b1e8d78975e34, size=532.0m; total size for store is
2.1g
2012-02-23 22:29:43,260 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000010971999,1329399349802.15720106584fe32f58287b6174970f31.
into af2472f30a9f4c558323a3268f98183c, size=517.9m; total size for store is
2.6g
2012-02-23 22:30:19,476 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731.
into 3ac255c439ab4c518f4a938d542c9220, size=335.4m; total size for store is
2.1g
2012-02-23 22:31:15,051 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000008501999,1329399349789.fa0ac64d8ac9050f4acf2b0deb6e2dc7.
into b8c733a27d02468a82f8a2c4d84db4ff, size=523.0m; total size for store is
2.1g
2012-02-23 22:32:33,430 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000004991999,1329399349770.46a2a16d484faacfcda36bc52bb218ba.
into 5344911081424de69299128412d41573, size=538.0m; total size for store is
2.2g
2012-02-23 22:33:28,882 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000004081999,1329399349765.a5ca0a6a0a5561af5b6964e458615493.
into d15c1e5b089d46beb2c4d01c796367a7, size=520.7m; total size for store is
2.1g
2012-02-23 22:34:46,714 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 3 file(s) in EVENT of
TC,00000000000007591999,1329399349784.711b032969e1ae6cce8e16bb9b
...
..
..
2012-02-23 23:44:32,215 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 1 file(s) in PI of
TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206.
into 2c701bd613314198889d8b3d79b82992, size=40.8m; total size for store is
40.8m
2012-02-23 23:45:00,838 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000005251999,1329399349771.50df7d847b5e36be9982b47eb3234ad4.
into 701d49ca010a41169ca495917b4ce3d4, size=1.3g; total size for store is
2.1g
2012-02-23 23:47:12,191 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000002261999,1329399349751.ffb8f2f3d9d5a23aee280359793557ae.
into d2b8336a36b1451d9a7d38c6bf3e79ee, size=1.2g; total size for store is
2.1g
2012-02-23 23:49:28,440 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000011881999,1329399349806.81f2bb0de10974b617521b62b9b85f09.
into 19208d50fe8543a682e3351c60801eb9, size=1.2g; total size for store is
2.1g
2012-02-23 23:52:12,523 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000000051999,1329399349734.e7ba4a35ee8524fbf4e6f9f9ba845206.
into 639ffb3d8c434363aac0d007b6f78ce9, size=1.5g; total size for store is
2.1g
2012-02-23 23:56:14,058 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000005901999,1329399349775.48aa14a1feea4793ec68bfe076570257.
into 3e2f293a33f14612b8fa1ad570799feb, size=1.6g; total size for store is
2.1g
2012-02-23 23:59:08,211 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000010451999,1329399349799.38739d7acb5f10198540bcfa0d4b5aef.
into dd7f92dd786d4e3ea9cb733cb3a6d2f8, size=1.6g; total size for store is
2.1g
...
...
...
...
2012-02-24 01:17:15,611 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 6 file(s) in EVENT of
TC,00000000000012401999,1329399349808.03a3d34ac2c0c7d09619
8032a151120f. into eff7b5bda7bf4653a9f229c1b40e0e4f, size=1.3g; total size
for store is 2.1g
2012-02-24 01:17:51,057 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000000441999,1329399349737.02354040d2650db1d59de0e01b
0edd1d. into ac1d5e5f1ab746229bad223f56534054, size=329.8m; total size for
store is 2.2g
2012-02-24 01:20:40,454 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 5 file(s) in EVENT of
TC,00000000000012921999,1329399349811.8dcc952ff57e87c6d828
40e84eb870a6. into ae82b7832eed4bafb9e64ed45574831c, size=978.9m; total
size for store is 2.5g
2012-02-24 01:21:13,635 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000007981999,1329399349786.7e26a6440421112a0a15ed78ac
33b168. into 94aa6f0328834b0896a05cda6a6a9a0f, size=309.1m; total size for
store is 2.1g
2012-02-24 01:22:50,578 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000002391999,1329399349752.05a0d235efa8135618628e5124
dc9507. into 0ca4a3b8221c42f992649431e7182a6d, size=329.6m; total size for
store is 2.2g
2012-02-24 01:25:03,055 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 5 file(s) in EVENT of
TC,00000000000001221999,1329399349743.56d16361d17dce747fb0
23ab15ded16b. into 47e9f68efb8a475dbce7db2c08b46f9c, size=1.0g; total size
for store is 2.5g
2012-02-24 01:25:38,876 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000001091999,1329399349742.5907e62062b4edc83033b4fda0
fc105f. into d5148c6b4dbc4d3c8339f151b987e51a, size=326.4m; total size for
store is 2.2g
2012-02-24 01:26:10,242 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000005381999,1329399349772.15d2f2fb1ad338ab3b5e57077d
874601. into bacf547a2b4f4627b630a18b88ec8ec6, size=306.5m; total size for
store is 2.1g
2012-02-24 01:27:47,690 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000010581999,1329399349800.8144106ed55b1571a720e46569
af8ea6. into 827d75a8b3144ee3a7e53dae5fdecf3a, size=296.5m; total size for
store is 2.1g
2012-02-24 01:28:23,907 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000006681999,1329399349779.9e0dc98e734715e66702419dc2
1e23ed. into 4a93423a056645d7a6f869f66c76fd0e, size=341.4m; total size for
store is 2.1g
2012-02-24 01:29:11,192 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000012661999,1329399349810.76715703e8bf5f3f960564834f
d9417d. into 3ef0ae62a13a4ed98cd772eeb1e55fd0, size=438.1m; total size for
store is 2.1g
2012-02-24 01:29:49,930 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000011101999,1329399349802.a63a43695fdcc0be243cfe3ea6
263981. into 2a542d20b2ed49b5964777c97f5f2a5b, size=355.2m; total size for
store is 2.1g
2012-02-24 01:33:08,747 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 5 file(s) in EVENT of
TC,00000000000008761999,1329399349791.7c0b5c9e5c98e9e39ae7
a3afcfaad4d9. into 93fb6ab4de7a4672aceed0e23cc9effc, size=1.2g; total size
for store is 2.1g
2012-02-24 01:35:21,194 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 5 file(s) in EVENT of
TC,00000000000006811999,1329399349780.6051d2a9230460194d41
f6249d087f6f. into 3adba15b5ae04bd1a18a18afeb7cf107, size=1.2g; total size
for store is 2.1g
2012-02-24 01:37:08,338 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000011621999,1329399349805.4d2bb3d00b0fa089163d6a0491dedadb.
into 26ac704d826a495080e93f07b3c6e24d, size=395.6m; total size for store is
2.6g
2012-02-24 01:37:51,454 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000012011999,1329399349807.99d15c956a3897276d0fdeaa33611417.
into 28aac65938bf493187f84e7019ca126d, size=393.8m; total size for store is
2.1g
2012-02-24 01:38:33,930 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000004861999,1329399349769.dce64ca59ca897a34aa6416de4f32dea.
into 5dda42912eb64701b20639065a7612cb, size=393.6m; total size for store is
2.1g
2012-02-24 01:39:23,057 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of 4 file(s) in EVENT of
TC,00000000000010841999,1329399349801.c7f801818067718327858c10b0f3ded2.
into 39ce298723bc4f66a5bfe1f0729143d4, size=468.0m; total size for store is
2.2g
2012-02-24 01:40:16,369 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in EVENT of
TC,00000000000001611999,1329399349746.eb6f8b45363b6d3f254d4b6d3dab3617.
into 7821cb65510840afaff12c1e008b28c4, size=491.1m; total size for store is
2.1g
2012-02-24 01:42:18,657 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in EVENT of
TC,00000000000002651999,1329399349754.5c75673b41370540e3c758e937181651.
into a7a12651ff3d414bb722945157686c11, size=505.1m; total size for store is
2.1g
2012-02-24 01:43:10,404 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in EVENT of
TC,00000000000004211999,1329399349766.041d7f3c86c4d5657c6eeded7584fad8.
into 7b8d8075a4d04873a50e517431935e03, size=482.2m; total size for store is
2.1g
2012-02-24 01:44:03,388 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in EVENT of
TC,00000000000011751999,1329399349805.21ac7408f58fa1b022b6a38b2f197915.
into f2906f916c16442a89161fdf9551bc0c, size=507.9m; total size for store is
2.1g
2012-02-24 01:44:59,108 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in EVENT of
TC,00000000000010061999,1329399349797.27a2f37d5d6ac48c860cca5251be5731.
into 60defcf94fce43488fffc619c509a9d3, size=524.2m; total size for store is
2.1g
2012-02-24 01:46:17,683 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 4 file(s) in EVENT of
TC,00000000000011491999,1329399349804.c613c26eff42730acd1f
..
..
..
..


Mikael.S


On Fri, Feb 24, 2012 at 2:05 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> Can you show us some logs that exhibit this behavior?
>
> Thx,
>
> J-D
>
> On Thu, Feb 23, 2012 at 2:58 PM, Mikael Sitruk <mikael.sitruk@gmail.com>
> wrote:
> > Hi
> >
> > Before filling a jira, I  would like to check this with you.
> > It seems that the major compaction behavior changed between 0.90.x and
> 0.92
> > (not taking about all the nice stuff you added there...)
> >
> > Effectively when i trigger a major compaction (via hbase shell or ui)
> when
> > other compactions (minor) are running (because i had a lot of storefiles
> > and a big compactionQueueSize), the major compaction is not executed.
> While
> > in 0.90.X major_compact command was executed immediately after any
> current
> > compaction.
> >
> >
> > Mikael.S
>



-- 
Mikael.S

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message