Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <1512139390968-0.post@n6.nabble.com>
References: <CAO=2E=vmmXpCjQMbosUq7Xq8vxyAZzSvf7sEBx6a0HpXBB2_rA@mail.gmail.com>
 <1512139390968-0.post@n6.nabble.com>
From: =?UTF-8?B?SnVhbiBSb2Ryw61ndWV6IEhvcnRhbMOh?= <juan.rodriguez.hortala@gmail.com>
Date: Tue, 5 Dec 2017 18:19:51 -0800
Message-ID: <CAO=2E=v6QLnLfWjFn5Quydq3TSpk4-h3LAxCabj+NJFHzzvA2w@mail.gmail.com>
Subject: Re: Error running ignite in YARN
To: user@ignite.apache.org
Content-Type: multipart/alternative; boundary="f403045fad682e6338055fa2963d"
archived-at: Wed, 06 Dec 2017 02:20:03 -0000

--f403045fad682e6338055fa2963d
Content-Type: text/plain; charset="UTF-8"

Hi Ilya,

Thanks a lot for your help, I'll try and the test the patch.

Regarding how I run this, I wait for the cluster to enter in WAITING state
and then I ssh to the master node as described in
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-ssh.html,
and then I just run the commands there. Before running the command I put
the files in HDFS. We cannot use a bootstrap action here because they run
before HDFS starts, but another option is using a EMR step for this.

For the node manager logs, I follow
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
to have access to the resource manager UI, and then I see the nodes that
are involved with the YARN application. Then I ssh to some of the slave
nodes involved with the job from the EMR master, and check the node manager
logs in /var/log/hadoop-yarn. Also we could configure yarn like suggested
in https://slider.incubator.apache.org/docs/getting_started.html to have 1
hour of timeout before cleaning up containers, using
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html
to configure that on EMR. The container files are then available in the EMR
slaves at /mnt/yarn/


I'll tell you how it goes when I have tested the patch.

Thanks a lot!

Juan


On Fri, Dec 1, 2017 at 6:43 AM, ilya.kasnacheev <ilya.kasnacheev@gmail.com>
wrote:

> Hello once more!
>
> I have prepared a patch. I think it won't be hard for you to try it out
> since you are accustomized to building Ignite. Please try it, tell if it
> solves the current issue.  ignite-7080.patch
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> t1312/ignite-7080.patch>
>
> I was trying to set Ignite up on AWS EMR today, without much success. I
> wonder if you could provide me with some directions.
>
> > IGNITE_YARN_JAR=/mnt/ignite/apache-ignite-2.3.0-src/
> modules/yarn/target/ignite-yarn-2.3.0.jar
> >  yarn jar ${IGNITE_YARN_JAR} ${IGNITE_YARN_JAR}
> > /mnt/ignite/ignite_yarn.properties
>
> Can you tell me what were you using to launch this command? I have tried
> executing "Custom JAR" step on EMR cluster, after copying Ignite properties
> and JAR to /mnt on master manually. So I was informed that an application
> was started, and is now running, but no indication about execution results.
> I suspect it hung with the same problem that you outlined (or earlier), but
> I haven't found a way to view application logs, neither "node manager" logs
> that you are referring to. Can you please point me where I can watch
> intermediate execution results?
>
> Also there's two data access questions. First, Ignite YARN jar expects two
> files on local FS, the jar itself, and properties file. Is there a way to
> make sure that these files are on file system, at predictable locations,
> when starting EMR cluster? To be able to add "Custom JAR" step to cluster
> initialization? E.g. specify an s3 bucket to be mounted to master node?
>
> Another question is that you are specifying ignite zip and ignite config
> are
> on hdfs. How would you put these files to hdfs when cluster is created? Or,
> for that matter, after it is created? I haven't found any access to HDFS in
> EMR console, neither any instructions on how to access it.
>
> Maybe we should change this process in order to use Ignite YARN with EMR
> without extensive boilerplate?
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

--f403045fad682e6338055fa2963d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Ilya,=C2=A0<div><br></div><div>Thanks a lot for your he=
lp, I&#39;ll try and the test the patch.=C2=A0</div><div><br></div><div>Reg=
arding how I run this, I wait for the cluster to enter in WAITING state and=
 then I ssh to the master node as described in <a href=3D"https://docs.aws.=
amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-ssh.html">htt=
ps://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node=
-ssh.html</a>, and then I just run the commands there. Before running the c=
ommand I put the files in HDFS. We cannot use a bootstrap action here becau=
se they run before HDFS starts, but another option is using a EMR step for =
this.=C2=A0</div><div><br></div><div>For the node manager logs, I follow=C2=
=A0<a href=3D"https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-we=
b-interfaces.html">https://docs.aws.amazon.com/emr/latest/ManagementGuide/e=
mr-web-interfaces.html</a> to have access to the resource manager UI, and t=
hen I see the nodes that are involved with the YARN application. Then I ssh=
 to some of the slave nodes involved with the job from the EMR master, and =
check the node manager logs in=C2=A0/var/log/hadoop-yarn. Also we could con=
figure yarn like suggested in=C2=A0<a href=3D"https://slider.incubator.apac=
he.org/docs/getting_started.html">https://slider.incubator.apache.org/docs/=
getting_started.html</a> to have 1 hour of timeout before cleaning up conta=
iners, using=C2=A0<a href=3D"https://docs.aws.amazon.com/emr/latest/Release=
Guide/emr-configure-apps.html">https://docs.aws.amazon.com/emr/latest/Relea=
seGuide/emr-configure-apps.html</a> to configure that on EMR. The container=
 files are then available in the EMR slaves at=C2=A0/mnt/yarn/<br></div><di=
v><br></div><div><br></div><div>I&#39;ll tell you how it goes when I have t=
ested the patch.=C2=A0</div><div><br></div><div>Thanks a lot!</div><div><br=
></div><div>Juan=C2=A0</div><div><br></div></div><div class=3D"gmail_extra"=
><br><div class=3D"gmail_quote">On Fri, Dec 1, 2017 at 6:43 AM, ilya.kasnac=
heev <span dir=3D"ltr">&lt;<a href=3D"mailto:ilya.kasnacheev@gmail.com" tar=
get=3D"_blank">ilya.kasnacheev@gmail.com</a>&gt;</span> wrote:<br><blockquo=
te class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc so=
lid;padding-left:1ex">Hello once more!<br>
<br>
I have prepared a patch. I think it won&#39;t be hard for you to try it out=
<br>
since you are accustomized to building Ignite. Please try it, tell if it<br=
>
solves the current issue.=C2=A0 ignite-7080.patch<br>
&lt;<a href=3D"http://apache-ignite-users.70518.x6.nabble.com/file/t1312/ig=
nite-7080.patch" rel=3D"noreferrer" target=3D"_blank">http://apache-ignite-=
users.<wbr>70518.x6.nabble.com/file/<wbr>t1312/ignite-7080.patch</a>&gt;<br=
>
<br>
I was trying to set Ignite up on AWS EMR today, without much success. I<br>
wonder if you could provide me with some directions.<br>
<span class=3D""><br>
&gt; IGNITE_YARN_JAR=3D/mnt/ignite/<wbr>apache-ignite-2.3.0-src/<wbr>module=
s/yarn/target/ignite-<wbr>yarn-2.3.0.jar<br>
&gt;=C2=A0 yarn jar ${IGNITE_YARN_JAR} ${IGNITE_YARN_JAR}<br>
&gt; /mnt/ignite/ignite_yarn.<wbr>properties<br>
<br>
</span>Can you tell me what were you using to launch this command? I have t=
ried<br>
executing &quot;Custom JAR&quot; step on EMR cluster, after copying Ignite =
properties<br>
and JAR to /mnt on master manually. So I was informed that an application<b=
r>
was started, and is now running, but no indication about execution results.=
<br>
I suspect it hung with the same problem that you outlined (or earlier), but=
<br>
I haven&#39;t found a way to view application logs, neither &quot;node mana=
ger&quot; logs<br>
that you are referring to. Can you please point me where I can watch<br>
intermediate execution results?<br>
<br>
Also there&#39;s two data access questions. First, Ignite YARN jar expects =
two<br>
files on local FS, the jar itself, and properties file. Is there a way to<b=
r>
make sure that these files are on file system, at predictable locations,<br=
>
when starting EMR cluster? To be able to add &quot;Custom JAR&quot; step to=
 cluster<br>
initialization? E.g. specify an s3 bucket to be mounted to master node?<br>
<br>
Another question is that you are specifying ignite zip and ignite config ar=
e<br>
on hdfs. How would you put these files to hdfs when cluster is created? Or,=
<br>
for that matter, after it is created? I haven&#39;t found any access to HDF=
S in<br>
EMR console, neither any instructions on how to access it.<br>
<br>
Maybe we should change this process in order to use Ignite YARN with EMR<br=
>
without extensive boilerplate?<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
Regards,<br>
<br>
<br>
<br>
--<br>
Sent from: <a href=3D"http://apache-ignite-users.70518.x6.nabble.com/" rel=
=3D"noreferrer" target=3D"_blank">http://apache-ignite-users.<wbr>70518.x6.=
nabble.com/</a><br>
</div></div></blockquote></div><br></div>

--f403045fad682e6338055fa2963d--