hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liu, Yi A" <yi.a....@intel.com>
Subject RE: Appending to HDFS file
Date Thu, 28 Aug 2014 06:35:51 GMT
Right, please use FileSystem#append

From: Stanley Shi [mailto:sshi@pivotal.io]
Sent: Thursday, August 28, 2014 2:18 PM
To: user@hadoop.apache.org
Subject: Re: Appending to HDFS file

You should not use this method:
FSDataOutputStream fp = fs.create(pt, true)

Here's the java doc for this "create" method:

  /**
   * Create an FSDataOutputStream at the indicated Path.
   * @param f the file to create
   * @param overwrite if a file with this name already exists, then if true,
   *   the file will be overwritten, and if false an exception will be thrown.
   */
  public FSDataOutputStream create(Path f, boolean overwrite)
      throws IOException {
    return create(f, overwrite,
                  getConf().getInt("io.file.buffer.size", 4096),
                  getDefaultReplication(f),
                  getDefaultBlockSize(f));
  }

On Wed, Aug 27, 2014 at 2:12 PM, rab ra <rabmdu@gmail.com<mailto:rabmdu@gmail.com>>
wrote:

hello

Here is d code snippet, I use to append

def outFile = "${outputFile}.txt"

Path pt = new Path("${hdfsName}/${dir}/${outFile}")

def fs = org.apache.hadoop.fs.FileSystem.get(configuration);

FSDataOutputStream fp = fs.create(pt, true)

fp << "${key} ${value}\n"
On 27 Aug 2014 09:46, "Stanley Shi" <sshi@pivotal.io<mailto:sshi@pivotal.io>>
wrote:
would you please past the code in the loop?

On Sat, Aug 23, 2014 at 2:47 PM, rab ra <rabmdu@gmail.com<mailto:rabmdu@gmail.com>>
wrote:

Hi

By default, it is true in hadoop 2.4.1. Nevertheless, I have set it to true explicitly in
hdfs-site.xml. Still, I am not able to achieve append.

Regards
On 23 Aug 2014 11:20, "Jagat Singh" <jagatsingh@gmail.com<mailto:jagatsingh@gmail.com>>
wrote:
What is value of dfs.support.append in hdfs-site.cml

https://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml



On Sat, Aug 23, 2014 at 1:41 AM, rab ra <rabmdu@gmail.com<mailto:rabmdu@gmail.com>>
wrote:
Hello,

I am currently using Hadoop 2.4.1.I am running a MR job using hadoop streaming utility.

The executable needs to write large amount of information in a file. However, this write is
not done in single attempt. The file needs to be appended with streams of information generated.

In the code, inside a loop, I open a file in hdfs, appends some information. This is not working
and I see only the last write.

How do I accomplish append operation in hadoop? Can anyone share a pointer to me?




regards
Bala




--
Regards,
Stanley Shi,
[http://www.gopivotal.com/files/media/logos/pivotal-logo-email-signature.png]



--
Regards,
Stanley Shi,
[http://www.gopivotal.com/files/media/logos/pivotal-logo-email-signature.png]
Mime
View raw message