hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Flushs and compactions
Date Thu, 06 Aug 2015 19:25:19 GMT
With DEBUG level: http://pastebin.com/arvD95bA

2015-08-06 12:14 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:

> JMS:
> Having log level at DEBUG would give us more clue.
>
> FYI
>
> On Thu, Aug 6, 2015 at 9:12 AM, Esteban Gutierrez <esteban@cloudera.com>
> wrote:
>
> > From the logs it looks like the flushes requested the compactions as
> > expected once the min number of files for a minor compaction was reached,
> > but I don't see any compaction running afterwards:
> >
> > 2015-08-06 09:21:57,465 INFO
> >  [StoreOpener-3d02eb59261771da84824c44e6d2ca40-1]
> > compactions.CompactionConfiguration: size [134217728,
> 9223372036854775807);
> > files [3, 10); ratio 1,200000; off-peak ratio 5,000000; throttle point
> > 2684354560; major period 604800000, major jitter 0,500000, min locality
> to
> > compact 0,000000.
> > ...
> > 2015-08-06 09:21:57,854 INFO
> >  [rs(localhost,33882,1438867297154)-flush-proc-pool5-thread-1]
> > regionserver.HRegion: Finished memstore flush of ~160 B/160,
> currentsize=0
> > B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> > in 77ms, sequenceid=6, compaction requested=false
> > ...
> > 2015-08-06 09:21:58,097 INFO
> >  [rs(localhost,33882,1438867297154)-flush-proc-pool6-thread-1]
> > regionserver.HRegion: Finished memstore flush of ~160 B/160,
> currentsize=0
> > B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> > in 12ms, sequenceid=11, compaction requested=false
> > ...
> > 2015-08-06 09:21:58,318 INFO
> >  [rs(localhost,33882,1438867297154)-flush-proc-pool7-thread-1]
> > regionserver.HRegion: Finished memstore flush of ~160 B/160,
> currentsize=0
> > B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> > in 9ms, sequenceid=16, compaction requested=true
> > ... no compactions here ...
> > 2015-08-06 10:31:37,420 INFO  [LruBlockCacheStatsExecutor]
> > hfile.LruBlockCache: totalSize=1.61 MB, freeSize=1.53 GB, max=1.53 GB,
> > blockCount=1, accesses=1, hits=0, hitRatio=0, cachingAccesses=1,
> > cachingHits=0, cachingHitsRatio=0,evictions=419, evicted=0,
> > evictedPerRun=0.0
> >
> > Are you sure you have the correct log4j config? I will try to reproduce
> > this and I let you know JMS.
> >
> > esteban.
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Thu, Aug 6, 2015 at 7:35 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org
> > > wrote:
> >
> > > Nothing on the logs...
> > >
> > > http://pastebin.com/4U6Fmxt9
> > >
> > > Running standalone. Kept the server running for 1h after I tried, still
> > no
> > > compactions.
> > >
> > > 2015-08-06 0:52 GMT-04:00 ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com>:
> > >
> > > > Any log saying skipping the compaction?  I need to check the code
> here.
> > > > Is the compaction thread picking up at a later point of time?
> > > >
> > > > Regards
> > > > Ram
> > > >
> > > > On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > I tried that with 1.0.1 but I think it's the same with 1.1.0.
> > > > >
> > > > > How to reproduce:
> > > > >
> > > > > create 'testflush', 'f1'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > >
> > > > > Result:
> > > > > jmspaggiari@t430s:~/hbase-1.0.0$ tree
> > > > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > > > └── testflush
> > > > >     └── 6546495c8df1e1b0a5a5a11d5df23e44
> > > > >         ├── f1
> > > > >         │   ├── 195e2b5af8e842f89440927a12c38c25
> > > > >         │   ├── 266ef04cefe944249d72791a5741a64e
> > > > >         │   ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> > > > >         │   ├── 365dbdc33f324f67bf466965cc6ad9e8
> > > > >         │   ├── 47a47d405bce4bf0a261a16a1ba5694f
> > > > >         │   ├── 54b7a073289b459292d0eff447d413a5
> > > > >         │   ├── 55fafb3c30fa4067a3bab54823478cdd
> > > > >         │   ├── 923ddff172a542b6a5ba6f3467983efa
> > > > >         │   ├── c69127daa4d34fe8b832e1fafea42713
> > > > >         │   ├── da4fe34334824f6ea9f5a12dfb93cab9
> > > > >         │   └── f3a15c326535420cb68d505ce9798d40
> > > > >         └── recovered.edits
> > > > >             └── 2.seqid
> > > > >
> > > > > 4 directories, 12 files
> > > > >
> > > > > I can test the same on 1.1.0 if you want, just let me know.
> > > > >
> > > > > 2015-08-05 13:18 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:
> > > > >
> > > > > > Can you provide some more detail please ?
> > > > > >
> > > > > > Release of hbase
> > > > > > region server log (snippet) showing that compaction is skipped
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > > > > > jean-marc@spaggiari.org> wrote:
> > > > > >
> > > > > > > Quick question here.
> > > > > > >
> > > > > > > Compactions seems to be triggered when we flush automatically,
> > but
> > > > not
> > > > > > when
> > > > > > > we flush manually.
> > > > > > >
> > > > > > > for 1 to 10
> > > > > > >   put into a table
> > > > > > >   flush the table
> > > > > > > end of for
> > > > > > >
> > > > > > > Will create 10 files, and will not trigger any compaction.
> > > > > > >
> > > > > > > Should we also compact after manual flushs?
> > > > > > >
> > > > > > > JM
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message