hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
Date Fri, 26 Apr 2013 05:30:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642615#comment-13642615
] 

Devaraj Das commented on HBASE-5930:
------------------------------------

bq. I'd like to open a separate issue on the queuing of flush requests
After some deliberations and code browsing, it seems like we don't need to do this. The reasons
being:

1. We already have the emergency flush procedure in place (MemStoreFlusher.flushRegion with
an emergencyFlush boolean argument exists that is called when there is an urgent need to flush).

2. In the normal steady state of affairs, entries from the queue are processed asynchronously
anyway. Flushes are done by a thread and there is no guarantee when the thread picks a flush
request up. By adding a delay, this situation is not worsened much, and especially in combination
with (1) above.

3. To address the issue raised, I was thinking that when a new flush request comes without
a delay, we could remove an existing delayed-flush-request entry that was queued before, and
queue a new one without a delay. However, there is a complication here - there is logic that
queues up entries with delayed flushes in _MemStoreFlusher.flushRegion(final FlushRegionEntry)_.
That would be impacted if I did what I said.

All in all, this doesn't seem like an issue.. [~enis] also ack'ed this in our offline discussions.
                
> Limits the amount of time an edit can live in the memstore.
> -----------------------------------------------------------
>
>                 Key: HBASE-5930
>                 URL: https://issues.apache.org/jira/browse/HBASE-5930
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Lars Hofhansl
>            Assignee: Devaraj Das
>             Fix For: 0.98.0, 0.94.8, 0.95.1
>
>         Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch,
5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch,
hbase-5930-test-execution.log
>
>
> A colleague of mine ran into an interesting issue.
> He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores
memory.
> Two weeks later he a had problem with the HDFS cluster, which caused the region servers
to abort. He found that his data was lost. Looking at the log we found that the Memstores
were not flushed at all during these two weeks.
> Should we have an option to flush memstores periodically. There are obvious downsides
to this, like many small storefiles, etc.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message