accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-2889) Batch metadata table updates for new walogs
Date Mon, 30 Jun 2014 16:48:24 GMT


Josh Elser commented on ACCUMULO-2889:

bq. We could just add all the hosted tablets to the log when it is new (common), and when
new tablets are hosted (rare).

Could also try to be a little smarter and keep some basic measure of "recently written to"
tablets and update the log entry there instead of all tablets. That would help the write-once,
read-many case that's common.

bq. This would cause unnecessary recoveries in the case where a tablet is flushed or isn't
otherwise using the log.

I'm not entirely clear on what performing log recovery for a tablet using a WAL that doesn't
have any updates for that tablet entails. Would we just be sorting and re-reading that WAL
to not find any updates that need to be replayed? Do you think we would incur a noticeable
penalty for doing this?

> Batch metadata table updates for new walogs
> -------------------------------------------
>                 Key: ACCUMULO-2889
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>    Affects Versions: 1.5.1, 1.6.0
>            Reporter: Jonathan Park
>            Assignee: Jonathan Park
>         Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch, accumulo-2889-withpatch.png,
> Currently, when we update the Metadata table with new loggers, we will update the metadata
for each tablet serially. We could optimize this to instead use a batchwriter to send all
metadata updates for all tablets in a batch.
> A few special cases include:
> - What if the !METADATA tablet was included in the batch?
> - What about the root tablet?
> Benefit:
> In one of our clusters, we're experiencing particularly slow HDFS operations leading
to large oscillations in ingest performance. We haven't isolated the cause in HDFS but when
we profile the tservers, we noticed that they were waiting for metadata table operations to
complete. This would target the waiting.
> Potential downsides:
> Given the existing locking scheme, it looks like we may have to lock a tablet for slightly
longer (we'll lock for the duration of the batch).

This message was sent by Atlassian JIRA

View raw message