hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bing Jiang <jiangbinglo...@gmail.com>
Subject Re: delete rows without writing HLog may be appear in the future?
Date Thu, 22 Nov 2012 06:50:25 GMT
If I set hbase.hregion.preclose.flush.size to zero, can HBase guarantee
that when HRegionServer quit, whether It will execute the last flush.
As Lars says, the issue refers to memstore flush, I have check that the
default value of ‘hbase.hregion.preclose.flush.size’ is 1024 * 1024 * 5.
I think if under the circumstance that we cannot bear with the low
performance of delete enable hlog, set ‘hbase.hregion.preclose.flush.size’
to zero, is it another choice?
Thanks.

----
Bing

2012/11/22 lars hofhansl <lhofhansl@yahoo.com>

> Hi Bing,
>
> I think you are referring to a memstore flush.
> The HLog represents the set of changes that are in the memstore (in ram)
> but not in an HFile on disk, yet.
>
>
> I am pretty sure there is no flaw in the flush/compaction logic when it
> comes to deletes.
>
>
> If you do not write the deletes to the WAL and the RS crashes it is
> expected that deletes there were not flushed to disk are lost.
>
> (And there's also HBASE-6059, which in some case resurfaces deleted data
> even when it was flushed to the WAL).
>
>
> -- Lars
>
>
> ________________________________
>  From: Bing Jiang <jiangbinglover@gmail.com>
> To: user@hbase.apache.org
> Sent: Wednesday, November 21, 2012 8:36 PM
> Subject: Re: delete rows without writing HLog may be appear in the future?
>
> I think when compaction is intrigued, if the records has already flushed
> into hdfs, whether it is worthless to retain the Hlog before that
> timestamp.
> In other ways, for example, some rows are deleted, then it executes a
> compaction, at the same time , the rows do not exist. So the hlog before
> the timestamp of compaction is not useful, and we can drop these unused
> wal.
> This is view of my own,  please correct me if wrong.
> ---
> Bing
>
>
> 2012/11/22 ramkrishna vasudevan <ramkrishna.s.vasudevan@gmail.com>
>
> > Sorry Bing.. am not much clear as what you suggest
> > 'One idea occurs to me why not check or restore wal when compaction
> > executes. If it does, hbase can drop some unused hlog'.
> >
> > Could you be more clear?  Are you trying to read the WAL while compaction
> > is going on?
> >
> > Regards
> > Ram
> >
> > On Thu, Nov 22, 2012 at 9:23 AM, Bing Jiang <jiangbinglover@gmail.com
> > >wrote:
> >
> > > In our hbase cluster, I test if delete records with hlog or without.
> > > Attachment is my my test.
> > > The result of test can testify why I make a decision of delete rows
> > > without hlog .
> > >
> > >
> > >
> > > 2012/11/22 Bing Jiang <jiangbinglover@gmail.com>
> > >
> > >> Thanks for all your suggestion and talk.
> > >> One idea occurs to me why not check or restore wal when compaction
> > >> executes. If it does, hbase can drop some unused hlog, I think that
> > will be
> > >> effective to the issue.
> > >> please correct me if I am wrong.
> > >>
> > >> ---Bing
> > >>
> > >> 2012/11/22 lars hofhansl <lhofhansl@yahoo.com>
> > >>
> > >>> I have it on my list of things to do to allow deferred WAL flush as
a
> > >>> per operation option (right now it's a CF option).
> > >>> You really do not want to do anything with the WAL off. If you use
> > >>> deferred flush there is still a chance that this might happen (the
RS
> > could
> > >>> die in the few seconds after a Delete before it is flushed to the
> > WAL), but
> > >>> it should be a rare occurrance.
> > >>>
> > >>>
> > >>> -- Lars
> > >>>
> > >>>
> > >>>
> > >>> ________________________________
> > >>>  From: Bing Jiang <jiangbinglover@gmail.com>
> > >>> To: user@hbase.apache.org
> > >>> Sent: Wednesday, November 21, 2012 7:20 AM
> > >>> Subject: Re: delete rows without writing HLog may be appear in the
> > >>> future?
> > >>>
> > >>> we need to confirm that put must be safe,but deletes must be quick
> and
> > >>> low-latency.
> > >>> On Nov 21, 2012 11:10 PM, "Michael Segel" <michael_segel@hotmail.com
> >
> > >>> wrote:
> > >>>
> > >>> > Some time later?
> > >>> >
> > >>> > Time of course is relative, so I have to ask what occurred between
> > the
> > >>> > write and the delete?
> > >>> > How much time? Did you have any compactions in between the write
> and
> > >>> the
> > >>> > delete?
> > >>> >
> > >>> > Why are you not consistent in your use of the WAL ?
> > >>> >
> > >>> >
> > >>> > On Nov 21, 2012, at 6:37 AM, Bing Jiang <jiangbinglover@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > hi,all.
> > >>> > > I want to describe a phenomenon that happens to our hbase
> cluster.
> > >>> > > I use puts(List<Put>) to insert many records with writing
hlog
> > >>> enable,
> > >>> > > and some time later I delete all of these records with writing
> hlog
> > >>> > disable.
> > >>> > > When one week later, i scan the table, I found some records
I
> have
> > >>> delete
> > >>> > > reappear again.
> > >>> > > It is an interesting case. In my opinion, if we delete data
> without
> > >>> > enable
> > >>> > > writing hlog, when regionserver fails, the log will replay
in
> > another
> > >>> > > regionserver.
> > >>> > > Can anyone tell me if I persist on deleting records without
> enable
> > >>> > writing
> > >>> > > hlog, is there a way to prevent these records from reappearing
> > again
> > >>> some
> > >>> > > time later?
> > >>> > >
> > >>> > > Cheers!
> > >>> > > --
> > >>> > > Bing Jiang
> > >>> > > weibo: http://weibo.com/jiangbinglover
> > >>> > > BLOG: http://blog.sina.com.cn/jiangbinglover
> > >>> > > BLOG: http://www.binospace.com
> > >>> > > National Research Center for Intelligent Computing Systems
> > >>> > > Institute of Computing technology
> > >>> > > Graduate University of Chinese Academy of Science
> > >>> >
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Bing Jiang
> > >> Tel:(86)134-2619-1361
> > >>
> > >> weibo: http://weibo.com/jiangbinglover
> > >> BLOG: http://blog.sina.com.cn/jiangbinglover
> > >> National Research Center for Intelligent Computing Systems
> > >> Institute of Computing technology
> > >> Graduate University of Chinese Academy of Science
> > >>
> > >>
> > >
> > >
> > > --
> > > Bing Jiang
> > > Tel:(86)134-2619-1361
> > > weibo: http://weibo.com/jiangbinglover
> > > BLOG: http://blog.sina.com.cn/jiangbinglover
> > > National Research Center for Intelligent Computing Systems
> > > Institute of Computing technology
> > > Graduate University of Chinese Academy of Science
> > >
> > >
> >
>
>
>
> --
> Bing Jiang
> Tel:(86)134-2619-1361
> weibo: http://weibo.com/jiangbinglover
> BLOG: http://blog.sina.com.cn/jiangbinglover
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>



-- 
Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message