hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11006) Ozone: support setting chunk size in streaming API
Date Tue, 04 Apr 2017 04:25:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954546#comment-15954546

Anu Engineer commented on HDFS-11006:

from this page - https://developers.google.com/protocol-buffers/docs/techniques
Protocol Buffers are not designed to handle large messages. 
As a general rule of thumb, if you are dealing in messages larger than a megabyte each, 
it may be time to consider an alternate strategy. 

So I would think that anything more than 1 MB, we should be chunking automatically or just
use 1 MB as the hard coded value as PB documentation suggests.

> Ozone: support setting chunk size in streaming API
> --------------------------------------------------
>                 Key: HDFS-11006
>                 URL: https://issues.apache.org/jira/browse/HDFS-11006
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Anu Engineer
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11006-HDFS-7240.001.patch, HDFS-11006-HDFS-7240.002.patch
> Right now we have a hard coded chunk size, we should either have this read from config
or the user should be able to pass this to ChunkInputStream

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message