hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19486) Periodically ensure records are not buffered too long by BufferedMutator
Date Fri, 29 Dec 2017 17:42:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306411#comment-16306411
] 

Hudson commented on HBASE-19486:
--------------------------------

FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4307 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4307/])
HBASE-19486 Periodically ensure records are not buffered too long by (chia7712: rev 5a1c36f70ac52e6f4e85f11ea0602d46b4861ac0)
* (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorParams.java
* (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestBufferedMutatorParams.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutator.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionConfiguration.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java


>  Periodically ensure records are not buffered too long by BufferedMutator
> -------------------------------------------------------------------------
>
>                 Key: HBASE-19486
>                 URL: https://issues.apache.org/jira/browse/HBASE-19486
>             Project: HBase
>          Issue Type: Improvement
>          Components: Client
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>             Fix For: 2.0.0-beta-1
>
>         Attachments: HBASE-19486-20171212-2117.patch, HBASE-19486-20171218-1229.patch,
HBASE-19486-20171218-1300.patch, HBASE-19486-20171219-0933.patch, HBASE-19486-20171219-1026.patch,
HBASE-19486-20171219-1122-trigger-qa-run.patch, HBASE-19486-20171220-1612-trigger-qa-run.patch,
HBASE-19486-20171220-2228-trigger-qa-run.patch, HBASE-19486-20171223-1438-trigger-qa-run.patch,
HBASE-19486-20171223-1728-trigger-qa-run.patch, HBASE-19486-20171223-2222-trigger-qa-run.patch,
HBASE-19486-20171224-1101-trigger-qa-run.patch, HBASE-19486-20171224-1602.patch, HBASE-19486-branch-1.v0.patch,
HBASE-19486-branch-1.v1.patch, HBASE-19486.v0.patch
>
>
> I'm working on several projects where we are doing stream / event type processing instead
of batch type processing. We mostly use Apache Flink and Apache Beam for these projects.
> When we ingest a continuous stream of events and feed that into HBase via a BufferedMutator
this all works fine. The buffer fills up at a predictable rate and we can make sure it flushes
several times per second into HBase by tuning the buffer size.
> We also have situations where the event rate is unpredictable. Some times because the
source is in reality a batch job that puts records into Kafka, sometimes because it is the
"predictable in production" application in our testing environment (where only the dev triggers
a handful of events).
> For these kinds of use cases we need a way to 'force' the BufferedMutator to automatically
flush any records in the buffer even if the buffer is not full.
> I'll put up a pull request with a proposed implementation for review against the master
(i.e. 3.0.0).
> When approved I would like to backport this to the 1.x and 2.x versions of the client
in the same (as close as possible) way.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message