Mailing-List: contact user-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@beam.apache.org
MIME-Version: 1.0
References: <F160988E-257A-4161-B6C5-D0CDBEC7FBBE@quartic.ai>
 <3fe0ebda-ede5-5a37-40e5-be1569cbb48f@nanthrax.net> <1A616CC2-C4D3-4714-BF35-B6895F5C421D@quartic.ai>
 <AAA4B850-FA6E-47F6-ADCD-806E7C9DA7E2@quartic.ai> <9db64c63-0af0-a179-84c5-a8894d38b5f9@nanthrax.net>
 <89D7196D-6DD5-4FD4-BEAF-17C683B520E7@quartic.ai>
In-Reply-To: <89D7196D-6DD5-4FD4-BEAF-17C683B520E7@quartic.ai>
From: Aviem Zur <aviemzur@gmail.com>
Date: Thu, 10 Aug 2017 07:13:33 +0000
Message-ID: <CADa_fpCG232OQKo_mSE_oUNxj5_sAmMn7SD6-OcG33HGkDdXmw@mail.gmail.com>
Subject: Re: Spark job hangs up at Evaluating ParMultiDo(ParseInput)
To: "user@beam.apache.org" <user@beam.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c1244a2d6be8a055660efe4"
archived-at: Thu, 10 Aug 2017 07:13:57 -0000

--94eb2c1244a2d6be8a055660efe4
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Jayaraman,

Thanks for reaching out.
We run Beam using Spark runner daily on a yarn cluster.

It appears that in many of the logs you sent there is hanging when
connecting to certain servers on certain ports, could this be a network
issue or an issue with your Spark setup?

Could you please share which version of Beam you are running?

On Thu, Aug 3, 2017 at 12:18 PM Sathish Jayaraman <sathish@quartic.ai>
wrote:

> Hi,
>
> Thanks for trying it out.
>
> I was running the job in local single node setup. I also spawn a
> HDInsights cluster in Azure platform just to test the WordCount program.
> Its the same result there too, stuck at the Evaluating ParMultiDo step. I=
t
> runs fine in mvn compile exec, but when bundled into jar & submitted via
> spark-submit there is no result. If there is no support from Beam to run =
on
> top of Spark then I have to write Spark native code which is what I am
> doing currently.
>
> Regards,
> Sathish. J
>
>
> On 03-Aug-2017, at 2:34 PM, Jean-Baptiste Onofr=C3=A9 <jb@nanthrax.net> w=
rote:
>
> nanthrax.net <jb@nanthrax.net>
>
>
>

--94eb2c1244a2d6be8a055660efe4
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Jayaraman,<div><br></div><div>Thanks for reaching out.<=
/div><div>We run Beam using Spark runner daily on a yarn cluster.</div><div=
><br></div><div>It appears that in many of the logs you sent there is hangi=
ng when connecting to certain servers on certain ports, could this be a net=
work issue or an issue with your Spark setup?</div><div><br></div><div>Coul=
d you please share which version of Beam you are running?</div></div><br><d=
iv class=3D"gmail_quote"><div dir=3D"ltr">On Thu, Aug 3, 2017 at 12:18 PM S=
athish Jayaraman &lt;<a href=3D"mailto:sathish@quartic.ai">sathish@quartic.=
ai</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margi=
n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


<div style=3D"word-wrap:break-word">
Hi,
<div><br>
</div>
<div>Thanks for trying it out.=C2=A0</div>
<div><br>
</div>
<div>I was running the job in local single node setup. I also spawn a HDIns=
ights cluster in Azure platform just to test the WordCount program. Its the=
 same result there too, stuck at the=C2=A0Evaluating ParMultiDo step. It ru=
ns fine in mvn compile exec,
 but when bundled into jar &amp; submitted via spark-submit there is no res=
ult. If there is no support from Beam to run on top of Spark then I have to=
 write Spark native code which is what I am doing currently.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Sathish. J</div>
<div><br>
</div>
<div><br>
<div>
<blockquote type=3D"cite">
<div>On 03-Aug-2017, at 2:34 PM, Jean-Baptiste Onofr=C3=A9 &lt;<a href=3D"m=
ailto:jb@nanthrax.net" target=3D"_blank">jb@nanthrax.net</a>&gt; wrote:</di=
v>
<br class=3D"m_7691728598030553732Apple-interchange-newline">
<div>
<blockquote type=3D"cite" style=3D"font-family:Helvetica;font-size:12px;fon=
t-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:n=
ormal;text-align:start;text-indent:0px;text-transform:none;white-space:norm=
al;word-spacing:0px">
<blockquote type=3D"cite">
<blockquote type=3D"cite"><a href=3D"mailto:jb@nanthrax.net" target=3D"_bla=
nk">nanthrax.net</a><span class=3D"m_7691728598030553732Apple-converted-spa=
ce">=C2=A0</span></blockquote>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br>
</div>
</div>

</blockquote></div>

--94eb2c1244a2d6be8a055660efe4--