hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1640) Transmission of large files in compressed format will save the network traffic and can improve the data transfer time.
Date Thu, 24 Feb 2011 16:03:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12998909#comment-12998909
] 

Uma Maheswara Rao G commented on HDFS-1640:
-------------------------------------------

Hi Harsh,

*Option 1*: Give a system level configuration, which will be applicable for all the files.
This approach has the drawback as you mentioned.

*Option 2*: Provide an option at file level to decide whether a file should be transmitted
between nodes in noramal way or compressed way.
          
 My suggestion is , we can provide both options to handle this scenarios.




> Transmission of large files in compressed format will save the network traffic and can
improve the data transfer time.
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1640
>                 URL: https://issues.apache.org/jira/browse/HDFS-1640
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>            Reporter: Uma Maheswara Rao G
>
> *In Write scenario:*
> DFSClient can Compress the data when transmitting it over the network, Data Nodes can
forward the same compressed data to other Data Nodes in pipeline and as well as it can decompress
that data and write on to their local disks. 
> *In Read Scenario:*
>  Data Node can compress the data when transmitting it over the network. DFSClient can
decompress it and write on to the local stream.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message