Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 93DCF200B25 for ; Wed, 8 Jun 2016 19:21:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 92BA4160A2E; Wed, 8 Jun 2016 17:21:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 41882160A0E for ; Wed, 8 Jun 2016 19:21:57 +0200 (CEST) Received: (qmail 94736 invoked by uid 500); 8 Jun 2016 17:21:56 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 94726 invoked by uid 99); 8 Jun 2016 17:21:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2016 17:21:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id B4E77C0552 for ; Wed, 8 Jun 2016 17:21:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.18 X-Spam-Level: * X-Spam-Status: No, score=1.18 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=kayak.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eqG8ZHz67OzA for ; Wed, 8 Jun 2016 17:21:53 +0000 (UTC) Received: from mail-qk0-f174.google.com (mail-qk0-f174.google.com [209.85.220.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E56C85FB12 for ; Wed, 8 Jun 2016 17:21:52 +0000 (UTC) Received: by mail-qk0-f174.google.com with SMTP id s186so7762540qkc.1 for ; Wed, 08 Jun 2016 10:21:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kayak.com; s=google; h=from:message-id:mime-version:subject:date:references:to:in-reply-to; bh=nEU8l9EIi/Ffpw7qIXFmZyHwblXh3vwohjW2EHQASu4=; b=a1z/u18XSZK8h6P9/7DFnj/Z0orPouHCgYIWwPSDuih/rzpIXqZo7DNz0k8YkdH5Kx sy3/YESo9ugHuFdDZp8wkpipvg2JsXPUzy49sjoO3YywYLLi/JKuFOYs+GFVQW7qvo5E nvVbowghj/WPFrdCKUf7QTjDA4NkfyUeqOXU0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :references:to:in-reply-to; bh=nEU8l9EIi/Ffpw7qIXFmZyHwblXh3vwohjW2EHQASu4=; b=HaYBchuI3u1aWiDW05jV0by1wcZ2kRpaGsFTKty/DssjYOUbuP111f25T/2gd35n9O aYILpB2aY9zkNpk+GtFkKNSHjL5F7w8em7a0O5mpTh8JbeHUBPSMwvsY851cEShYIjsO oDp7LueLadmKUjhWXGgRIFC4hfd3y7ZSR5XYp2H5NSY6/4JzMUIfNAdRRsJzS34fMeDB HtOtgI201sdGSfILRb/0/xVMOxnU5+Or3NThJOHZIlGuHoLDv1GFvWQwEbpKMpPOsh6z LDxWcV+f2GNGtZW24ZAWAk83yro2MMEG/KNYM+tjh4nXcM13zNKsAza4BHxqFIeaB2Qm 7/KA== X-Gm-Message-State: ALyK8tLI4LE6pFet0ZoGwSWhyaSCOwKZeSwUHbiITwsB9OVDxszPD58d5Q7qj4gOA5kIFyiM X-Received: by 10.55.154.136 with SMTP id c130mr5715875qke.79.1465406511974; Wed, 08 Jun 2016 10:21:51 -0700 (PDT) Received: from ma-lt-hongmei.ma.runwaynine.com (50-207-13-186-static.hfc.comcastbusiness.net. [50.207.13.186]) by smtp.gmail.com with ESMTPSA id 2sm551822qgx.21.2016.06.08.10.21.50 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 08 Jun 2016 10:21:51 -0700 (PDT) From: Hongmei Zong Content-Type: multipart/alternative; boundary="Apple-Mail=_B4478DE0-1697-49A6-A068-20741EDC7514" Message-Id: <85DD3D60-FC12-4F03-B194-897F8DA3C915@kayak.com> Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Ignite for Spark on YARN Deployment Date: Wed, 8 Jun 2016 13:22:09 -0400 References: <1465225139653-5465.post@n6.nabble.com> <784BCAB2-D27F-42BC-BBD2-FE654F8A4235@gridgain.com> To: user@ignite.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3124) archived-at: Wed, 08 Jun 2016 17:21:58 -0000 --Apple-Mail=_B4478DE0-1697-49A6-A068-20741EDC7514 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Nikolai, Thank you very much for your solution! I would like to double check with = you about my address1=E2=80=A6... ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", = "your_address2:47500..47510", >=20 > "your_address3:47500..47510")); your_address1:47500..47510,your_address2:47500..47510 and = your_address3:47500..47510 are the YARN master_host address, right? Is that necessary to install Ignite on these three YARN master_host? I = guess the answer is =E2=80=9CNO=E2=80=9D. Is the the master_port should be 47500 or 47510? Sorry for these questions, I am new to Ignite and YARN framework. Thanks again!!! Hongmei > On Jun 8, 2016, at 12:30 PM, Nikolai Tikhonov = wrote: >=20 > Hi Hongmei Zong! >=20 > Client node which started from IgniteContext can't to find server = nodes. By default ignite integration with YARN uses = TcpDiscoveryVmIpFinder (if you don't use another ip finder in your = configuration). > In the case you should set the ip finder in IgniteContext. The = following code snippet shows how do it on Java: >=20 > IgniteConfiguration cfg =3D new IgniteConfiguration(); >=20 > TcpDiscoverySpi spi =3D new TcpDiscoverySpi(); >=20 > TcpDiscoveryVmIpFinder ipFinder =3D new TcpDiscoveryVmIpFinder(); > ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", = "your_address2:47500..47510", > "your_address3:47500..47510")); >=20 > spi.setIpFinder(ipFinder); >=20 > cfg.setDiscoverySpi(spi); >=20 > On Wed, Jun 8, 2016 at 6:31 PM, Hongmei Zong > wrote: > Hi Denis, >=20 > I tried testing Ignite as the following steps: >=20 > Background information:=20 > 1. Our Spark is running on YARN deployment; There are three Master = hosts and many Worker nodes and three client nodes in the Spark = clusters. > 2. I installed Ignite on one of the client node and can launch = Ignite-shell locally on this client node. Since for testing purpose, I = did not install Ignite on any master nodes or worker nodes. > 3. I logged into the client node which Ignite was installed and = launched Ignite YARN application using the following command: >=20 > hadoop jar = /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar = /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar = /u/hongmei/apache-ignite/config/cluster.properties >=20 > I open the UI console for the Ignite YARN application, and it shows = that Ignite YARN is running, Containers, CPU Cores, Memory are all = allocated by YARN. screenshot is as the following=20 >=20 > application_1464374946035_26956 = hongmei ignition YARN default Wed Jun 8 = 10:21:04 -0400 2016 N/A RUNNING UNDEFINED 12 12 = 34816 9.6 3.8=09 > ApplicationMaster = >=20 > 4. I open an other terminal log into the same client node and run the = Ignite for Spark Shell command as the following: > The Spark shell started >=20 > /usr/bin/spark-shell --jars = /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-igni= te/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ign= ite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite= -log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignit= e-log4j/log4j-1.2.17.jar --packages = org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0= >=20 > The Spark shell launched successfully and I use these two commands to = import the ignite spark files: > import org.apache.ignite.spark._ > import org.apache.ignite.configuration._ >=20 > Next I create an instance of Ignite context as the following syntax: > val ic =3D new IgniteContext[Integer, Integer](sc, () =3D> new = IgniteConfiguration()) >=20 > I got the following message: So I stuck at this point. >=20 > 16/06/08 10:28:27 WARN TcpDiscoverySpi: IP finder returned empty = addresses list. Please check IP finder configuration and make sure = multicast works on your network. Will retry every 2 secs. >=20 > So I stuck at this point. >=20 > The other scenario when I run command listed as Step 4. The Spark = shell can not be launched successfully with the status of =E2=80=9CAccept=E2= =80=9D, but never get chance to run. >=20 > Any good suggestions?????? Is there anything wrong with my test = procedure?? > Thanks in advance!=20 >=20 > Mei >=20 >=20 >> On Jun 7, 2016, at 10:26 AM, Denis Magda > wrote: >>=20 >> Hi, >>=20 >> I=E2=80=99m not an expert in this area however have you tried to = specify a Spark master like the following documentation says? >> = https://apacheignite-fs.readme.io/docs/testing-integration-with-spark-shel= l#working-with-spark-shell = >>=20 >> If you did try please share the full logs, someone from the community = will respond. >>=20 >> =E2=80=94 >> Denis >>=20 >>> On Jun 6, 2016, at 5:58 PM, Hongmei Zong > wrote: >>>=20 >>> Hi there, >>>=20 >>> I would like to use "Ignite for Spark" to save the states of Spark = jobs in >>> memory and those states can be used for later jobs. For Shared = Deployment, >>> the document only offer two ways to deploy Ignite cluster. First is = the >>> standalone deployment, second is MESOS deployment. But Our Spark = clusters >>> are running on YARN. My question is: is it possible to run Ignite = for Spark >>> on YARN deployment??? >>>=20 >>> I downloaded and installed Ignite on my machine. Next, I referenced = the link >>> below for YARN Deployment. >>> http://apacheignite.gridgain.org/docs/yarn-deployment = >>>=20 >>> I created the cluster.properties file and ran the application using = the >>> command: hadoop jar >>> = /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar >>> = /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar >>> /u/hongmei/apache-ignite/config/cluster.properties=20 >>>=20 >>> Form the YARN console, The YARN ignite application works ok. It = shows >>> running, and 16 containers are allocated for the ignite application. >>>=20 >>> After this step, what should I do in order to run Spark with Ignite = on YARN >>> deployment?? >>>=20 >>> Many Thanks!!! >>>=20 >>> Mei >>>=20 >>>=20 >>>=20 >>> -- >>> View this message in context: = http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark-on-YARN-De= ployment-tp5465.html = >>> Sent from the Apache Ignite Users mailing list archive at = Nabble.com. >>=20 >=20 >=20 --Apple-Mail=_B4478DE0-1697-49A6-A068-20741EDC7514 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi Nikolai,

Thank you very much for your solution! I would like to double = check with you about my address1=E2=80=A6...

ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", "your_address2:47500..47510",
    "your_address3:47500..47510"));
your_address1:47500..47510,your_address2:47500..47510 = and your_address3:47500..47510 are the YARN master_host address, = right?

Is that necessary to install Ignite on these = three YARN master_host? I guess the answer = is =E2=80=9CNO=E2=80=9D.
Is = the the master_port should be 47500 or 47510?

Sorry = for these questions, I am new to Ignite and YARN = framework.
Thanks = again!!!

Hongmei


On Jun 8, 2016, at 12:30 PM, Nikolai Tikhonov <ntikhonov@apache.org> wrote:

Hi Hongmei Zong!

Client node which = started from IgniteContext can't to find server nodes. By default ignite = integration with YARN uses TcpDiscoveryVmIpFinder (if you don't use = another ip finder in your configuration).
In the = case you should set the ip finder in IgniteContext. The following code = snippet shows how do it on Java:

IgniteConfiguration cfg =3D new =
IgniteConfiguration();

TcpDiscoverySpi=
 spi =3D new TcpDiscoverySpi();

TcpDiscoveryVmIpFinder ipFinder =3D new =
TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("your_address1:47500..47510", "your_address2:47500..47510",
    "your_address3:47500..47510"));

spi.setIpFinder(ipFinder);

cfg.setDiscoverySpi(spi);

On Wed, = Jun 8, 2016 at 6:31 PM, Hongmei Zong <hongmei@kayak.com> wrote:
Hi Denis,

I tried testing Ignite as the = following steps:

Background = information: 
1. Our Spark is running on YARN = deployment; There are three Master hosts and many Worker nodes and three = client nodes in the Spark clusters.
2. I installed Ignite on one of the = client node and can launch Ignite-shell locally on this client node. = Since for testing purpose, I did not install Ignite on any master nodes = or worker nodes.
3. I logged into the client node = which Ignite was installed and launched Ignite YARN application using = the following command:

hadoop jar = /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar = /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar = /u/hongmei/apache-ignite/config/cluster.properties

I open the UI = console for the Ignite YARN application, and it shows that Ignite YARN = is running, Containers, CPU Cores, Memory are all allocated by YARN. = screenshot is as the following 

application_1464374946035_26956hongmeiignitionYARNdefaultWed = Jun 8 10:21:04 -0400 2016N/ARUNNINGUNDEFINED1212348169.63.8
ApplicationMaster

4. I open an other = terminal log into the same client node and run the Ignite for Spark = Shell command as the following:
The Spark shell = started

/usr/bin/spark-shell = --jars /u/hongmei/apache-ignite/libs/ignite-core-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-spark/ignite-spark-1.6.0.jar,/u/hongmei/apache-ignite/libs/cache-api-1.0.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/ignite-log4j-1.6.0.jar,/u/hongmei/apache-ignite/libs/optional/ignite-log4j/log4j-1.2.17.jar --packages = org.apache.ignite:ignite-spark:1.6.0,org.apache.ignite:ignite-spring:1.6.0=

The Spark shell = launched successfully and I use these two commands = to import the ignite spark files:
import org.apache.ignite.spark._
import org.apache.ignite.configuration._

Next I create an instance of Ignite context as the following = syntax:
val ic =3D new IgniteContext[Integer, Integer](sc, () =3D> new IgniteConfiguration())

I got the following = message: So I stuck at this point.

16/06/08 10:28:27 WARN TcpDiscoverySpi: IP = finder returned empty addresses list. Please check IP finder = configuration and make sure multicast works on your network. Will retry = every 2 secs.

So I stuck at this point.

The other scenario when I run = command listed as Step 4. The Spark shell can not be launched = successfully with the status of =E2=80=9CAccept=E2=80=9D, but never = get chance to run.

Any good suggestions?????? Is there anything wrong with my = test procedure??
Thanks in advance! 

Mei


On Jun 7, 2016, at 10:26 AM, Denis Magda <dmagda@gridgain.com> wrote:

Hi,

I=E2=80=99m = not an expert in this area however have you tried to specify a Spark = master like the following documentation says?
https://apacheignite-fs.readme.io/docs/testing-integration-with= -spark-shell#working-with-spark-shell

If = you did try please share the full logs, someone from the community will = respond.

=E2=80=94
Denis

On Jun 6, = 2016, at 5:58 PM, Hongmei Zong <hongmei@kayak.com> wrote:

Hi there,

I would = like to use "Ignite for Spark" to save the states of Spark jobs in
memory and those states can be used for later jobs. For = Shared Deployment,
the document only offer two ways to = deploy Ignite cluster. First is the
standalone deployment, = second is MESOS deployment. But Our Spark clusters
are = running on YARN. My question is: is it possible to run Ignite for = Spark
on YARN deployment???

I = downloaded and installed Ignite on my machine. Next, I referenced the = link
below for YARN Deployment.
http://apacheignite.gridgain.org/docs/yarn-deployment

I created the cluster.properties file and ran = the application using the
command: =    hadoop jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-= 1.6.0.jar
/u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-= 1.6.0.jar
/u/hongmei/apache-ignite/config/cluster.properties=

Form the YARN console, The YARN ignite = application works ok. It shows
running, and 16 containers = are allocated for the ignite application.

After this step, what should I do in order to run Spark with = Ignite on YARN
deployment??

Many Thanks!!!

Mei



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-for-Spark= -on-YARN-Deployment-tp5465.html
Sent from the Apache = Ignite Users mailing list archive at Nabble.com.




= --Apple-Mail=_B4478DE0-1697-49A6-A068-20741EDC7514--