hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: StoreScanner created for memstore flush should be bothered about updated readers?
Date Fri, 31 Jan 2014 05:41:53 GMT
>> The scanner stack is only reset if the set of HFiles for this store
changes, i.e. a compaction or a concurrent flush (when using multithreaded
flushing). It seems that would relatively rare.
In our test scenario this happens.  While trying to find out the root cause
for HBASE-10443, hit this issue. It is not directly related to the flush
scenario but found this issue while debugging it.
I was not trying to improve the performance here, but the fact that we
updating the kv heap does make the flush to read those Hfiles on a
StoreScanner.next() call and it is expensive.

Regards
Ram





On Fri, Jan 31, 2014 at 11:02 AM, lars hofhansl <larsh@apache.org> wrote:

> From what I found is that the main performance detriment comes from the
> fact that we need to take a lock for each next/peek call of the
> StoreScanner. Even when those are uncontended (which they are in 99.9% of
> the cases) the memory read/writes barriers are expensive.
>
> I doubt you'll see much improvement from this. The scanner stack is only
> reset if the set of HFiles for this store changes, i.e. a compaction or a
> concurrent flush (when using multithreaded flushing). It seems that would
> relatively rare.
>
> If anything we could a class like StoreScanner that does not need to
> synchronize any of its calls, but even there, the flush is asynchronous to
> any user action (unless we're blocked on the number of store files, in
> which case there bigger problem anyway).
>
>
> Did you see a specific issue?
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: ramkrishna vasudevan <ramkrishna.s.vasudevan@gmail.com>
> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
> Cc:
> Sent: Thursday, January 30, 2014 11:48 AM
> Subject: StoreScanner created for memstore flush should be bothered about
> updated readers?
>
> Hi All
>
> In case of flush we create a memstore flusher which in turn creates a
> StoreScanner backed by a Single ton MemstoreScanner.
>
> But this scanner also registers for any updates in the reader in the
> HStore.  Is this needed?
> If this happens then any update on the reader may nullify the current heap
> and the entire Scanner Stack is reset, but this time with the other
> scanners for all the files that satisfies the last top key.  So the flush
> that happens on the memstore holds the storefile scanners also in the heap
> that was recreated but originally the intention was to create a scanner on
> the memstore alone.
>
> Am i missing something here?  Or what i observed is right?  If so, then I
> feel that this step can be avoided.
>
> Regards
> Ram
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message