hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Debarshi Basak <>
Subject Re: Compressed data storage in HDFS - Error
Date Wed, 06 Jun 2012 10:33:26 GMT
<font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"> Compression
is an overhead when you have a CPU intensive job<br><br><br>Debarshi Basak<br>Tata
Consultancy Services<br>Mailto:<br>Website:<br>____________________________________________<br>Experience
certainty.	IT Services<br>			Business Solutions<br>			Outsourcing<br>____________________________________________<br><br><font
color="#990099">-----Bejoy Ks <> wrote: -----</></font><div><blockquote
style="border-left: 2px solid black; padding-right: 0px; padding-left: 5px; margin-left: 5px;
margin-right: 0px;">To: "" &lt;;<br>From:
Bejoy Ks &lt;;<br>Date: 06/06/2012 03:37PM<br>Subject:
Re: Compressed data storage in HDFS - Error<br><br><div style="color: rgb(0,
0, 0); background-color: rgb(255, 255, 255); font-family: verdana,helvetica,sans-serif; font-size:
10pt;"><div><span><br></span></div><div>Hi Sreenath</div><div><br></div><div>Output
compression is more useful on storage level, when a larger file is compressed it saves on
hdfs blocks and there by the cluster become more scalable in terms of number of files.&nbsp;</div><div><br></div><div>Yes
lzo libraries needs to be there in all task tracker nodes as well the node that hosts the
hive client.</div><div><br></div><div>Regards</div><div>Bejoy
KS<br></div><div><br></div><div></div><div style="font-family:
verdana,helvetica,sans-serif; font-size: 10pt;"> <div style="font-family: times new
roman,new york,times,serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial"
size="2"> <hr size="1">  <b><span style="font-weight: bold;">From:</span></b>
Sreenath Menon &lt;;<br> <b><span style="font-weight:
bold;">To:</span></b>; Bejoy Ks &lt;;
<br> <b><span style="font-weight: bold;">Sent:</span></b> Wednesday,
June 6, 2012 3:25 PM<br> <b><span style="font-weight: bold;">Subject:</span></b>
Re: Compressed data storage in HDFS - Error<br> </font> </div> <br>
<!--Notes ACF
<meta http-equiv="x-dns-prefetch-control" content="off">--><div id="yiv802454005">Hi
Bejoy<br>I would like to make this clear.<br>There is no gain on processing throughput/time
on compressing the data stored in HDFS (not talking about intermediate compression)...wright??<br>And
do I need to add the <span>lzo libraries in Hadoop_Home/lib/native for all the nodes
(including the slave nodes)??<br>
</div><!--Notes ACF
<meta http-equiv="x-dns-prefetch-control" content="on">--><br><br> </div>
</div>  </div></blockquote></div><div></div></font><p>=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you</p>


View raw message