hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Lavati (Jira)" <>
Subject [jira] [Updated] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
Date Fri, 15 Nov 2019 09:16:00 GMT


David Lavati updated HIVE-21146:
    Attachment: HIVE-21146.2.patch

> Enforce TransactionBatch size=1 for blob stores
> -----------------------------------------------
>                 Key: HIVE-21146
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 3.0.0
>            Reporter: Eugene Koifman
>            Assignee: David Lavati
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.2.patch,
>          Time Spent: 20m
>  Remaining Estimate: 0h
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N transactions
can be opened at once and the data in all of them will be written to the same delta_x_y directory
where each transaction in the batch can be committed/aborted independently.  The implementation
relies on {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is available
on HDFS but is often implemented as no-op in Blob store backed {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an error if
{{builder.transactionBatchSize > 1}} and the target table/partitions are backed by something
that doesn't support {{hflush()}}.

This message was sent by Atlassian Jira

View raw message