hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: Flushs and compactions
Date Thu, 06 Aug 2015 16:12:02 GMT
>From the logs it looks like the flushes requested the compactions as
expected once the min number of files for a minor compaction was reached,
but I don't see any compaction running afterwards:

2015-08-06 09:21:57,465 INFO
 [StoreOpener-3d02eb59261771da84824c44e6d2ca40-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1,200000; off-peak ratio 5,000000; throttle point
2684354560; major period 604800000, major jitter 0,500000, min locality to
compact 0,000000.
...
2015-08-06 09:21:57,854 INFO
 [rs(localhost,33882,1438867297154)-flush-proc-pool5-thread-1]
regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
in 77ms, sequenceid=6, compaction requested=false
...
2015-08-06 09:21:58,097 INFO
 [rs(localhost,33882,1438867297154)-flush-proc-pool6-thread-1]
regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
in 12ms, sequenceid=11, compaction requested=false
...
2015-08-06 09:21:58,318 INFO
 [rs(localhost,33882,1438867297154)-flush-proc-pool7-thread-1]
regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
in 9ms, sequenceid=16, compaction requested=true
... no compactions here ...
2015-08-06 10:31:37,420 INFO  [LruBlockCacheStatsExecutor]
hfile.LruBlockCache: totalSize=1.61 MB, freeSize=1.53 GB, max=1.53 GB,
blockCount=1, accesses=1, hits=0, hitRatio=0, cachingAccesses=1,
cachingHits=0, cachingHitsRatio=0,evictions=419, evicted=0,
evictedPerRun=0.0

Are you sure you have the correct log4j config? I will try to reproduce
this and I let you know JMS.

esteban.



--
Cloudera, Inc.


On Thu, Aug 6, 2015 at 7:35 AM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:

> Nothing on the logs...
>
> http://pastebin.com/4U6Fmxt9
>
> Running standalone. Kept the server running for 1h after I tried, still no
> compactions.
>
> 2015-08-06 0:52 GMT-04:00 ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com>:
>
> > Any log saying skipping the compaction?  I need to check the code here.
> > Is the compaction thread picking up at a later point of time?
> >
> > Regards
> > Ram
> >
> > On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > I tried that with 1.0.1 but I think it's the same with 1.1.0.
> > >
> > > How to reproduce:
> > >
> > > create 'testflush', 'f1'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > >
> > > Result:
> > > jmspaggiari@t430s:~/hbase-1.0.0$ tree
> > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > └── testflush
> > >     └── 6546495c8df1e1b0a5a5a11d5df23e44
> > >         ├── f1
> > >         │   ├── 195e2b5af8e842f89440927a12c38c25
> > >         │   ├── 266ef04cefe944249d72791a5741a64e
> > >         │   ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> > >         │   ├── 365dbdc33f324f67bf466965cc6ad9e8
> > >         │   ├── 47a47d405bce4bf0a261a16a1ba5694f
> > >         │   ├── 54b7a073289b459292d0eff447d413a5
> > >         │   ├── 55fafb3c30fa4067a3bab54823478cdd
> > >         │   ├── 923ddff172a542b6a5ba6f3467983efa
> > >         │   ├── c69127daa4d34fe8b832e1fafea42713
> > >         │   ├── da4fe34334824f6ea9f5a12dfb93cab9
> > >         │   └── f3a15c326535420cb68d505ce9798d40
> > >         └── recovered.edits
> > >             └── 2.seqid
> > >
> > > 4 directories, 12 files
> > >
> > > I can test the same on 1.1.0 if you want, just let me know.
> > >
> > > 2015-08-05 13:18 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:
> > >
> > > > Can you provide some more detail please ?
> > > >
> > > > Release of hbase
> > > > region server log (snippet) showing that compaction is skipped
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > Quick question here.
> > > > >
> > > > > Compactions seems to be triggered when we flush automatically, but
> > not
> > > > when
> > > > > we flush manually.
> > > > >
> > > > > for 1 to 10
> > > > >   put into a table
> > > > >   flush the table
> > > > > end of for
> > > > >
> > > > > Will create 10 files, and will not trigger any compaction.
> > > > >
> > > > > Should we also compact after manual flushs?
> > > > >
> > > > > JM
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message