Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of ed.serrano@gmail.com designates
 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <E1D4637F-ECCA-4E48-96B6-249D9E40A751@trifacta.com>
References: <E1D4637F-ECCA-4E48-96B6-249D9E40A751@trifacta.com>
Date: Fri, 14 Jun 2013 11:49:50 -0500
Message-ID: 
 <CAEdwYjPhycwa0sjwjnLxGOhC0ishNqQ5Nmr-bFO0+NJq05Cpkg@mail.gmail.com>
Subject: Re: webhdfs read error after successful pig job
From: Ed Serrano <ed.serrano@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e013a062410d27f04df2009e9

--089e013a062410d27f04df2009e9
Content-Type: text/plain; charset=ISO-8859-1

You might want to investigate if your issue is aways on the same node.


On Fri, Jun 14, 2013 at 11:43 AM, Adam Silberstein <adam@trifacta.com>wrote:

> Hi,
> I'm having some trouble with webhdfs read after running a Pig job that
> completed successfully.
>
> Here are some details:
>
> -I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this
> (0.10.0 I think)
>
> -The Pig job writes out about 10 files.  I'm programmatically attempting
> to read each of these with webhdfs soon after pig notifies me the job is
> complete.  The reads often all succeed.  And even in the failure case, most
> of the reads still succeed, but one may fail.
>
> -I wondered if I was facing a race condition where Pig was reporting
> success before the file was truly ready to read.  However, when I run
> WebHDFS read with curl even hours later, the request hangs.  In contrast, I
> can run 'cat' from the DFS command line and the file is output correctly.
>
> -I ran fsck over the problem file and it report back totally normal.
>
> -I looked at the namenode to see why my curl request hangs.  I get this
> error:
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:ubuntu (auth:SIMPLE)
> cause:java.io.IOException: Could not reach the block containing the data.
> Please try again
> (I'm guessing the permissions aren't really the important thing here, the
> underlying cause of not reaching the block seems more reasonable).
>
> -I have a 4 node cluster with replication set to 1.
>
>
> If anyone has seen this, has diagnostic tips, or best of all, a solution,
> please let me know!
>
> Thanks,
> Adam
>
>
>


-- 

-------------------------------------
*Ed Serrano*
Mobile: 972-897-5443

--089e013a062410d27f04df2009e9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">You might want to investigate if your issue is aways on th=
e same node.</div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_qu=
ote">On Fri, Jun 14, 2013 at 11:43 AM, Adam Silberstein <span dir=3D"ltr">&=
lt;<a href=3D"mailto:adam@trifacta.com" target=3D"_blank">adam@trifacta.com=
</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi,<br>
I&#39;m having some trouble with webhdfs read after running a Pig job that =
completed successfully.<br>
<br>
Here are some details:<br>
<br>
-I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this (0.=
10.0 I think)<br>
<br>
-The Pig job writes out about 10 files. =A0I&#39;m programmatically attempt=
ing to read each of these with webhdfs soon after pig notifies me the job i=
s complete. =A0The reads often all succeed. =A0And even in the failure case=
, most of the reads still succeed, but one may fail.<br>

<br>
-I wondered if I was facing a race condition where Pig was reporting succes=
s before the file was truly ready to read. =A0However, when I run WebHDFS r=
ead with curl even hours later, the request hangs. =A0In contrast, I can ru=
n &#39;cat&#39; from the DFS command line and the file is output correctly.=
<br>

<br>
-I ran fsck over the problem file and it report back totally normal.<br>
<br>
-I looked at the namenode to see why my curl request hangs. =A0I get this e=
rror:<br>
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionExc=
eption as:ubuntu (auth:SIMPLE) cause:java.io.IOException: Could not reach t=
he block containing the data. Please try again<br>
(I&#39;m guessing the permissions aren&#39;t really the important thing her=
e, the underlying cause of not reaching the block seems more reasonable).<b=
r>
<br>
-I have a 4 node cluster with replication set to 1.<br>
<br>
<br>
If anyone has seen this, has diagnostic tips, or best of all, a solution, p=
lease let me know!<br>
<br>
Thanks,<br>
Adam<br>
<br>
<br>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><blockquote =
style=3D"margin:0 0 0 40px;border:none;padding:0px"></blockquote>----------=
---------------------------<div><b>Ed Serrano</b></div><div>Mobile: 972-897=
-5443</div>

</div>

--089e013a062410d27f04df2009e9--