hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hequn cheng <chenghe...@gmail.com>
Subject why can FSDataInputStream.read() only read 2^17 bytes in hadoop2.0?
Date Fri, 07 Mar 2014 05:32:34 GMT
Hi~
First, i use FileSystem to open a file in hdfs.
         FSDataInputStream m_dis = fs.open(...);

Second, read the data in m_dis to a byte array.
          byte[] inputdata = new byte[m_dis.available()];
 //m_dis.available = 47185920
          m_dis.read(inputdata, 0, 20 * 1024 * 768 * 3);

the value returned by m_dis.read() is 131072(2^17), so the data after
131072 is missing. It seems that FSDataInputStream use short to manage it's
data which confused me a lot. The same code run well in hadoop1.2.1.

thank you~

Mime
View raw message