Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EE5F311C6D for ; Wed, 27 Aug 2014 06:13:08 +0000 (UTC) Received: (qmail 1906 invoked by uid 500); 27 Aug 2014 06:13:00 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 1793 invoked by uid 500); 27 Aug 2014 06:13:00 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 1076 invoked by uid 99); 27 Aug 2014 06:12:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 06:12:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rabmdu@gmail.com designates 209.85.213.54 as permitted sender) Received: from [209.85.213.54] (HELO mail-yh0-f54.google.com) (209.85.213.54) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Aug 2014 06:12:34 +0000 Received: by mail-yh0-f54.google.com with SMTP id v1so13159315yhn.13 for ; Tue, 26 Aug 2014 23:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bjbe3vrbGlvUhS1UblN3UdUaL/J0P1p2N4SgTsC9PGk=; b=hAqmCE+GEThcgt7Q99O56MIslCw5Zsg2OANRD2zTARaL27bg4c2wIXI0K/l43ucUmt zbMb9vI+MSYLbueziS8qxI4lkmzEFYtQxdRTGfWSq3VcvtXmzhtuLS4Xh3zpFBsMFVu0 hsfiMImisrm7uPHHJmbJtm2Y1kDjP6y3d3xC4m4/VDHO1wn/nSXNVWJTzZNuehUGnRgo HDQkzkStLjfKXmKXnDlJYELEsOCevvfow5IUAw+3j+MqlK00ty0LerVUTC+Y5NMGK9Fo TG7zVLYWH6esROiEgoZZNUt7iP19eILJ7VrRErpmR67tYjrBKOml44Zb/ANZeq+ELwtQ yEjA== MIME-Version: 1.0 X-Received: by 10.220.74.195 with SMTP id v3mr27421205vcj.23.1409119952896; Tue, 26 Aug 2014 23:12:32 -0700 (PDT) Received: by 10.220.210.199 with HTTP; Tue, 26 Aug 2014 23:12:32 -0700 (PDT) Received: by 10.220.210.199 with HTTP; Tue, 26 Aug 2014 23:12:32 -0700 (PDT) In-Reply-To: References: Date: Wed, 27 Aug 2014 11:42:32 +0530 Message-ID: Subject: Re: Appending to HDFS file From: rab ra To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e01634bf03d08760501964e31 X-Virus-Checked: Checked by ClamAV on apache.org --089e01634bf03d08760501964e31 Content-Type: text/plain; charset=UTF-8 hello Here is d code snippet, I use to append def outFile = "${outputFile}.txt" Path pt = new Path("${hdfsName}/${dir}/${outFile}") def fs = org.apache.hadoop.fs.FileSystem.get(configuration); FSDataOutputStream fp = fs.create(pt, true) fp << "${key} ${value}\n" On 27 Aug 2014 09:46, "Stanley Shi" wrote: > would you please past the code in the loop? > > > On Sat, Aug 23, 2014 at 2:47 PM, rab ra wrote: > >> Hi >> >> By default, it is true in hadoop 2.4.1. Nevertheless, I have set it to >> true explicitly in hdfs-site.xml. Still, I am not able to achieve append. >> >> Regards >> On 23 Aug 2014 11:20, "Jagat Singh" wrote: >> >>> What is value of dfs.support.append in hdfs-site.cml >>> >>> >>> https://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml >>> >>> >>> >>> >>> On Sat, Aug 23, 2014 at 1:41 AM, rab ra wrote: >>> >>>> Hello, >>>> >>>> I am currently using Hadoop 2.4.1.I am running a MR job using hadoop >>>> streaming utility. >>>> >>>> The executable needs to write large amount of information in a file. >>>> However, this write is not done in single attempt. The file needs to be >>>> appended with streams of information generated. >>>> >>>> In the code, inside a loop, I open a file in hdfs, appends some >>>> information. This is not working and I see only the last write. >>>> >>>> How do I accomplish append operation in hadoop? Can anyone share a >>>> pointer to me? >>>> >>>> >>>> >>>> >>>> regards >>>> Bala >>>> >>> >>> > > > -- > Regards, > *Stanley Shi,* > > --089e01634bf03d08760501964e31 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


hello

Here is d code snippet, I use to append

def outFile =3D "${outputFile}.txt"

Path pt =3D new Path("${hdfsName}/${dir}/${outFile}")

def fs =3D org.apache.hadoop.fs.FileSystem.get(configuration);

FSDataOutputStream fp =3D fs.create(pt, true)

fp << "${key} ${value}\n"

On 27 Aug 2014 09:46, "Stanley Shi" &l= t;sshi@pivotal.io> wrote:
would you please past the code in the loop?=C2=A0


On Sat, Aug 23, = 2014 at 2:47 PM, rab ra <rabmdu@gmail.com> wrote:

Hi

By default, it is true in hadoop 2.4.1. Nevertheless, I have set it to t= rue explicitly in hdfs-site.xml. Still, I am not able to achieve append.

Regards

On 23 Aug 2014 11:20, "Jagat Singh" &l= t;jagatsingh@gmai= l.com> wrote:
What is value of=C2=A0dfs.support.append in hdfs-site.cml

<= span style=3D"line-height:17.280000686645508px">



On Sat, Aug 23, 2014 at 1:41 AM, rab ra = <rabmdu@gmail.com> wrote:
Hello,

I am currently using Hadoop 2.4.= 1.I am running a MR job using hadoop streaming utility.=C2=A0
The executable needs to write large amount of information in a = file. However, this write is not done in single attempt. The file needs to = be appended with streams of information generated.=C2=A0

In the code, inside a loop, I open a file in hdfs, appe= nds some information. This is not working and I see only the last write.=C2= =A0

How do I accomplish append operation in hadoop= ? Can anyone share a pointer to me?




regards
Bala




--
=
Regards,
Stanley Shi,

--089e01634bf03d08760501964e31--