ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Rodríguez Hortalá <juan.rodriguez.hort...@gmail.com>
Subject Re: Error running ignite in YARN
Date Wed, 06 Dec 2017 02:19:51 GMT
Hi Ilya,

Thanks a lot for your help, I'll try and the test the patch.

Regarding how I run this, I wait for the cluster to enter in WAITING state
and then I ssh to the master node as described in
and then I just run the commands there. Before running the command I put
the files in HDFS. We cannot use a bootstrap action here because they run
before HDFS starts, but another option is using a EMR step for this.

For the node manager logs, I follow
to have access to the resource manager UI, and then I see the nodes that
are involved with the YARN application. Then I ssh to some of the slave
nodes involved with the job from the EMR master, and check the node manager
logs in /var/log/hadoop-yarn. Also we could configure yarn like suggested
in https://slider.incubator.apache.org/docs/getting_started.html to have 1
hour of timeout before cleaning up containers, using
to configure that on EMR. The container files are then available in the EMR
slaves at /mnt/yarn/

I'll tell you how it goes when I have tested the patch.

Thanks a lot!


On Fri, Dec 1, 2017 at 6:43 AM, ilya.kasnacheev <ilya.kasnacheev@gmail.com>

> Hello once more!
> I have prepared a patch. I think it won't be hard for you to try it out
> since you are accustomized to building Ignite. Please try it, tell if it
> solves the current issue.  ignite-7080.patch
> <http://apache-ignite-users.70518.x6.nabble.com/file/
> t1312/ignite-7080.patch>
> I was trying to set Ignite up on AWS EMR today, without much success. I
> wonder if you could provide me with some directions.
> > IGNITE_YARN_JAR=/mnt/ignite/apache-ignite-2.3.0-src/
> modules/yarn/target/ignite-yarn-2.3.0.jar
> > /mnt/ignite/ignite_yarn.properties
> Can you tell me what were you using to launch this command? I have tried
> executing "Custom JAR" step on EMR cluster, after copying Ignite properties
> and JAR to /mnt on master manually. So I was informed that an application
> was started, and is now running, but no indication about execution results.
> I suspect it hung with the same problem that you outlined (or earlier), but
> I haven't found a way to view application logs, neither "node manager" logs
> that you are referring to. Can you please point me where I can watch
> intermediate execution results?
> Also there's two data access questions. First, Ignite YARN jar expects two
> files on local FS, the jar itself, and properties file. Is there a way to
> make sure that these files are on file system, at predictable locations,
> when starting EMR cluster? To be able to add "Custom JAR" step to cluster
> initialization? E.g. specify an s3 bucket to be mounted to master node?
> Another question is that you are specifying ignite zip and ignite config
> are
> on hdfs. How would you put these files to hdfs when cluster is created? Or,
> for that matter, after it is created? I haven't found any access to HDFS in
> EMR console, neither any instructions on how to access it.
> Maybe we should change this process in order to use Ignite YARN with EMR
> without extensive boilerplate?
> Regards,
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

View raw message