hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1640) Transmission of large files in compressed format will save the network traffic and can improve the data transfer time.
Date Thu, 30 Jun 2011 05:14:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057618#comment-13057618
] 

Uma Maheswara Rao G commented on HDFS-1640:
-------------------------------------------

Hi Harsh,
 Thanks for the comments,

{quote}
@Option 2 – This can be done at the writer level itself. Much too expensive maintaining
a per-file property for compression and implementing the same to be guaranteed as well.
{quote}
  I agree with you.

bq. Besides, compressing on the fly for tx would still invoke the same amount of IO. No savings,
right?
  Here the amount of data we are transferring will become very less. In our test, we noticed
that we are  constrained by network IO. With this feature alone we could achieve good performance
results. 
  



> Transmission of large files in compressed format will save the network traffic and can
improve the data transfer time.
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1640
>                 URL: https://issues.apache.org/jira/browse/HDFS-1640
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node, hdfs client
>            Reporter: Uma Maheswara Rao G
>
> *In Write scenario:*
> DFSClient can Compress the data when transmitting it over the network, Data Nodes can
forward the same compressed data to other Data Nodes in pipeline and as well as it can decompress
that data and write on to their local disks. 
> *In Read Scenario:*
>  Data Node can compress the data when transmitting it over the network. DFSClient can
decompress it and write on to the local stream.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message