hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Lavati (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
Date Fri, 15 Nov 2019 09:16:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

David Lavati updated HIVE-21146:
--------------------------------
    Attachment: HIVE-21146.2.patch

> Enforce TransactionBatch size=1 for blob stores
> -----------------------------------------------
>
>                 Key: HIVE-21146
>                 URL: https://issues.apache.org/jira/browse/HIVE-21146
>             Project: Hive
>          Issue Type: Bug
>          Components: Streaming, Transactions
>    Affects Versions: 3.0.0
>            Reporter: Eugene Koifman
>            Assignee: David Lavati
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.2.patch, HIVE-21146.2.patch,
HIVE-21146.patch
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N transactions
can be opened at once and the data in all of them will be written to the same delta_x_y directory
where each transaction in the batch can be committed/aborted independently.  The implementation
relies on {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is available
on HDFS but is often implemented as no-op in Blob store backed {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an error if
{{builder.transactionBatchSize > 1}} and the target table/partitions are backed by something
that doesn't support {{hflush()}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message