hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19486) Periodically ensure records are not buffered too long by BufferedMutator
Date Fri, 29 Dec 2017 17:42:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306411#comment-16306411

Hudson commented on HBASE-19486:

FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4307 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4307/])
HBASE-19486 Periodically ensure records are not buffered too long by (chia7712: rev 5a1c36f70ac52e6f4e85f11ea0602d46b4861ac0)
* (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorParams.java
* (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestBufferedMutatorParams.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java

>  Periodically ensure records are not buffered too long by BufferedMutator
> -------------------------------------------------------------------------
>                 Key: HBASE-19486
>                 URL: https://issues.apache.org/jira/browse/HBASE-19486
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>             Fix For: 2.0.0-beta-1
>         Attachments: HBASE-19486-20171212-2117.patch, HBASE-19486-20171218-1229.patch,
HBASE-19486-20171218-1300.patch, HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
HBASE-19486-20171219-1122-trigger-qa-run.patch, HBASE-19486-20171220-1612-trigger-qa-run.patch,
HBASE-19486-20171220-2228-trigger-qa-run.patch, HBASE-19486-20171223-1438-trigger-qa-run.patch,
HBASE-19486-20171223-1728-trigger-qa-run.patch, HBASE-19486-20171223-2222-trigger-qa-run.patch,
HBASE-19486-20171224-1101-trigger-qa-run.patch, HBASE-19486-20171224-1602.patch, HBASE-19486-branch-1.v0.patch,
HBASE-19486-branch-1.v1.patch, HBASE-19486.v0.patch
> I'm working on several projects where we are doing stream / event type processing instead
of batch type processing. We mostly use Apache Flink and Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a BufferedMutator
this all works fine. The buffer fills up at a predictable rate and we can make sure it flushes
several times per second into HBase by tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times because the
source is in reality a batch job that puts records into Kafka, sometimes because it is the
"predictable in production" application in our testing environment (where only the dev triggers
a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to automatically
flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against the master
(i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of the client
in the same (as close as possible) way.

This message was sent by Atlassian JIRA

View raw message