hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Binglin Chang <decst...@gmail.com>
Subject Re: why can FSDataInputStream.read() only read 2^17 bytes in hadoop2.0?
Date Fri, 07 Mar 2014 05:48:04 GMT
the semantic of read does not guarantee read as much as possible. you need
to call read() many times or use readFully


On Fri, Mar 7, 2014 at 1:32 PM, hequn cheng <chenghequn@gmail.com> wrote:

> Hi~
> First, i use FileSystem to open a file in hdfs.
>          FSDataInputStream m_dis = fs.open(...);
>
> Second, read the data in m_dis to a byte array.
>           byte[] inputdata = new byte[m_dis.available()];
>  //m_dis.available = 47185920
>           m_dis.read(inputdata, 0, 20 * 1024 * 768 * 3);
>
> the value returned by m_dis.read() is 131072(2^17), so the data after
> 131072 is missing. It seems that FSDataInputStream use short to manage it's
> data which confused me a lot. The same code run well in hadoop1.2.1.
>
> thank you~
>

Mime
View raw message