hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Porter <gmpor...@gmail.com>
Subject Re: Faster alternative to FSDataInputStream
Date Tue, 18 Aug 2009 20:12:15 GMT
You could try changing your buffer size from 1KB to a much higher
number (like 64MB).

i.e.,

  byte[] buf = new byte[1024*1024*64];

-George

On Tue, Aug 18, 2009 at 9:00 AM, Ananth T.
Sarathy<ananth.t.sarathy@gmail.com> wrote:
> I am trying to download binary files stored in Hadoop but there is like a 2
> minute wait on a 20mb file when I try to execute the in.read(buf).
>
> is there a better way to be doing this?
>
>    private void pipe(InputStream in, OutputStream out) throws IOException
>    {    System.out.println(System.currentTimeMillis()+" Starting to Pipe
> Data");
>        byte[] buf = new byte[1024];
>        int read = 0;
>        while ((read = in.read(buf)) >= 0)
>        {
>            out.write(buf, 0, read);
>            System.out.println(System.currentTimeMillis()+" Piping Data");
>        }
>        out.flush();
>        System.out.println(System.currentTimeMillis()+" Finished Piping
> Data");
>
>    }
>
> public void readFile(String fileToRead, OutputStream out)
>            throws IOException
>    {
>        System.out.println(System.currentTimeMillis()+" Start Read File");
>        Path inFile = new Path(fileToRead);
>        System.out.println(System.currentTimeMillis()+" Set Path");
>        // Validate the input/output paths before reading/writing.
>
>        if (!fs.exists(inFile))
>        {
>            throw new HadoopFileException("Specified file  " + fileToRead
>                    + " not found.");
>        }
>        if (!fs.isFile(inFile))
>        {
>            throw new HadoopFileException("Specified file  " + fileToRead
>                    + " not found.");
>        }
>        // Open inFile for reading.
>        System.out.println(System.currentTimeMillis()+" Opening Data
> Stream");
>        FSDataInputStream in = fs.open(inFile);
>
>        System.out.println(System.currentTimeMillis()+" Opened Data
> Stream");
>        // Open outFile for writing.
>
>        // Read from input stream and write to output stream until EOF.
>        pipe(in, out);
>
>        // Close the streams when done.
>        out.close();
>        in.close();
>    }
> Ananth T Sarathy
>

Mime
View raw message