Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of xelllee@aim.com designates
 205.188.109.204 as permitted sender)
To: user@hadoop.apache.org
Subject: OPENFORWRITE  Files issue
MIME-Version: 1.0
From: Xiao Li <xelllee@aim.com>
Content-Type: multipart/alternative;
 boundary="--------MB_8D0F671B3DD8650_C6C_161DB_webmail-m223.sysops.aol.com"
Message-Id: <8D0F671B3C5B890-C6C-5AAF@webmail-m223.sysops.aol.com>
Date: Wed, 12 Feb 2014 20:26:07 -0500 (EST)

This is a multi-part message in MIME format.
----------MB_8D0F671B3DD8650_C6C_161DB_webmail-m223.sysops.aol.com
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="us-ascii"


Say I have a text file on hdfs in "OPENFORWRITE, HEALTHY" status. some proc=
ess is appending to it.=20


It has 4 lines in it.


hadoop fs -cat /file | wc -l=20
4


However when I do a wordcount on this file, only first line is visible to t=
he mapreduce. Similar in hive when i do "select count(*) from filetable" =
=3D 1


If I do "hadoop cp /file /file2", then it works as expected.(file2 is close=
d, file is still open)


wordcount would see 5 lines in the input directory(1 from opened file, 4 fr=
om copied file), hive will return 5.


I am wondering if there is anything related to TextInputFormat?


I am using CDH 4.4.0


Thanks.


Xiao Li


----------MB_8D0F671B3DD8650_C6C_161DB_webmail-m223.sysops.aol.com
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset="us-ascii"

<font color=3D'black' size=3D'2' face=3D'arial'>
<div><br>
</div>

<div>Say I have a text file on hdfs in "OPENFORWRITE, HEALTHY" status. some=
 process is appending to it.&nbsp;</div>

<div><br>
</div>

<div>It has 4 lines in it.</div>

<div><br>
</div>

<div>hadoop fs -cat /file | wc -l&nbsp;</div>

<div>4</div>

<div><br>
</div>

<div>However when I do a wordcount on this file, only first line is visible=
 to the mapreduce. Similar in hive when i do "select count(*) from filetabl=
e" =3D 1</div>

<div><br>
</div>

<div>If I do "hadoop cp /file /file2", then it works as expected.(file2 is =
closed, file is still open)</div>

<div><br>
</div>

<div>wordcount would see 5 lines in the input directory(1 from opened file,=
 4 from copied file), hive will return 5.</div>

<div><br>
</div>

<div>I am wondering if there is anything related to&nbsp;<span style=3D"fon=
t-family: Helvetica, Arial, sans-serif; font-size: 10pt;">TextInputFormat?<=
/span></div>

<div><br>
</div>

<div>I am using CDH 4.4.0</div>

<div><br>
</div>

<div>Thanks.</div>
<br>

<div style=3D"clear:both">Xiao Li<br>
<br>
</div>
</font>
----------MB_8D0F671B3DD8650_C6C_161DB_webmail-m223.sysops.aol.com--