ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy Setrakyan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-2736) custom Ignite Configuration (not xml file) is not used by spark executors
Date Tue, 01 Mar 2016 15:31:18 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dmitriy Setrakyan updated IGNITE-2736:
--------------------------------------
    Assignee: Ivan Veselovsky

> custom Ignite Configuration (not xml file) is not used by spark executors
> -------------------------------------------------------------------------
>
>                 Key: IGNITE-2736
>                 URL: https://issues.apache.org/jira/browse/IGNITE-2736
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Luca Rea
>            Assignee: Ivan Veselovsky
>              Labels: community
>             Fix For: 1.6
>
>
> Hi,
> I have launched an Ignite Cluster inside YARN and I use spark-shell from a client to
attach to existing cluster cluster in client mode, connection between client and cluster doesn't
support multicast so I've tried to use a custom config like below:
> {code}
> import org.apache.ignite.spark._
> import org.apache.ignite.configuration._
> import org.apache.ignite._
> import org.apache.ignite.spi.discovery.tcp._
> val spi = new TcpDiscoverySpi();
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm._
> val ipFinder = new TcpDiscoveryVmIpFinder();
> import java.util.Arrays
> ipFinder.setAddresses(Arrays.asList("172.16.24.48:47500", "172.16.24.49:47500", "172.16.24.50:47500",
"172.16.24.51:47500", "172.16.24.52:47500", "172.16.24.53:47500"));
> spi.setIpFinder(ipFinder);
> val cfg = new IgniteConfiguration() with Serializable;
> cfg.setGridName("ignite-cluster");
> cfg.setDiscoverySpi(spi);
> val cacheCfg = new CacheConfiguration("myCache");
> import org.apache.ignite.cache._
> cacheCfg.setCacheMode(CacheMode.PARTITIONED);
> cacheCfg.setBackups(1);
> cfg.setCacheConfiguration(cacheCfg);
> class cfg extends Serializable;
> val ic = new IgniteContext[Integer, Integer](sc, () => cfg )
> val sharedRdd = ic.fromCache("example")
> val x = sqlContext.sparkContext.parallelize(1 to 10000, 10).map(i => (new Integer(i),
new Integer(i)))
> sharedRdd.savePairs(x)
> {code}
> when I run the last command it freeze waiting to connect to the cluster, in fact it seems
that in this way the spark execurors don't use the above configuration nor load the file default-config.xml
but use some hardocoded configuration with only multicast enabled.
> The workaround is to use a acustom xml configuration file and copy it into the config
ignite path of all all spark nodes the run:
> {code}
> import org.apache.ignite.spark._
> val ic = new IgniteContext[Integer, Integer](sc, "config/custom-config.xml")
> {code}
> custom-config.xml:
> {code}
> <beans xmlns="http://www.springframework.org/schema/beans"
>        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>        xsi:schemaLocation="
>        http://www.springframework.org/schema/beans
>        http://www.springframework.org/schema/beans/spring-beans.xsd">
>     <!--
>         Alter configuration below as needed.
>     -->
> <bean class="org.apache.ignite.configuration.IgniteConfiguration">
>   <property name="clientMode" value="true"/>
>   <property name="discoverySpi">
>     <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>       <property name="ipFinder">
>         <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
>           <property name="addresses">
>             <list>
>               <value>172.16.24.48:47500</value>
>               <value>172.16.24.49:47500</value>
>               <value>172.16.24.50:47500</value>
>               <value>172.16.24.51:47500</value>
>               <value>172.16.24.52:47500</value>
>               <value>172.16.24.53:47500</value>
>             </list>
>           </property>
>         </bean>
>       </property>
>     </bean>
>   </property>
> </bean>
> </beans>
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message