hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: How can I copy a local disk file, and output it as bytesWritable
Date Tue, 16 Mar 2010 19:31:30 GMT
You can use multiple methods to create input/output streams.  One is to just
copy the content of the file into a buffer.  You can use Hadoop's IOUtils
(org.apache.hadoop.io.IOUtils):

    BytesWritable value = new BytesWritable();
    FileInputStream istream = new FileInputStream(file);
    byte[] contents = new byte[(int) file.length()];
    try {
      IOUtils.readFully(istream, contents, 0, contents.length);
      value.set(contents, 0, contents.length);
    } catch(Exception e) {
      e.printStackTrace(System.err);
    } finally {
      istream.close();
      contents = null;
    }

Alex K


On Tue, Mar 16, 2010 at 11:52 AM, welman Lu <welmanwenzi@gmail.com> wrote:

> Hi, everyone!
>
> The problem what I met is that, I want to transform a local disk file into
> bytesWritable to output.
>
> Now, all I found is only can use FileSystem.copyFromLocalFile to copy a
> file from local disk to HDFS.
>
> The first problem is that, will this file be distributed naturally?
> And the other problem is that, I can't find methods can read the file into
> bytesWritable.
>
> Can anybody tell me what kind of class that I should use?
> Thank you very much!
>
> Best Regards
> Jiamin Lu
>

Mime
View raw message