ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hongmei Zong <hong...@kayak.com>
Subject Re: Ignite for Spark on YARN Deployment
Date Thu, 09 Jun 2016 14:08:39 GMT
Hi nikolai,

Thank you very much for prompt reply!

I did not find the ignite-config.xml file under my ignite home directory(
 /u/hongmei/apache-ignite/  ).

I find a "default-config.xml" at the path:
/u/hongmei/apache-ignite/config/default-config.xml

<?xml version="1.0" encoding="UTF-8"?>


<!--

  Licensed to the Apache Software Foundation (ASF) under one or more

  contributor license agreements.  See the NOTICE file distributed with

  this work for additional information regarding copyright ownership.

  The ASF licenses this file to You under the Apache License, Version 2.0

  (the "License"); you may not use this file except in compliance with

  the License.  You may obtain a copy of the License at


       http://www.apache.org/licenses/LICENSE-2.0


  Unless required by applicable law or agreed to in writing, software

  distributed under the License is distributed on an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->


<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="

       http://www.springframework.org/schema/beans

       http://www.springframework.org/schema/beans/spring-beans.xsd">

    <!--

        Alter configuration below as needed.

    -->

        <bean id="grid.cfg" class=
"org.apache.ignite.configuration.IgniteConfiguration">

                <property name="discoverySpi">

                        <bean class=
"org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">

                                <property name="ipFinder">

                                        <bean class=
"org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">

                                                <property name="addresses">

                                                        <list>

                                                                <!--

                                                                 Explicitly
specifying address of a local node to let it start

                                                                and operate
normally even if there is no more nodes in the cluster.

                                                                You can
also optionally specify an individual port or port range.

                                                                -->

                                                                <!--


<value>1.2.3.4</value>

                                                                -->

                                                                <!--

                                                                        IP
Address and optional port range of a remote node.

                                                                        You
can also optionally specify an individual port and don't set

                                                                        the
port range at all.

                                                                -->

                                                                <value>
c5hdpe001.c5.runwaynine.com:47500..47509</value>

                                                                <value>
c5hdpe002.c5.runwaynine.com:47500..47509</value>

                                                                <value>
c5hdpe003.c5.runwaynine.com:47500..47509</value>


                                                        </list>

                                                </property>

                                        </bean>

                                </property>

                        </bean>

                </property>

        </bean>

</beans>



The "cluster.properties" at the path:
/u/hongmei/apache-ignite/config/cluster.properties

# The number of nodes in the cluster.

IGNITE_NODE_COUNT=2


# The number of CPU Cores for each Apache Ignite node.

IGNITE_RUN_CPU_PER_NODE=1


# The number of Megabytes of RAM for each Apache Ignite node.

IGNITE_MEMORY_PER_NODE=2048


# The version ignite which will be run on nodes.

IGNITE_VERSION=1.6.0


# The hdfs directory which will be used for saving Apache Ignite
disbributives.

IGNITE_RELEASES_DIR=/user/hongmei/ignite-yarn


#The directory which will be used for saving Apache Ignite
distributives(copy .jar file to it).

IGNITE_WORKING_DIR=/user/hongmei/ignite/workdir


#The hdfs path to Apache Ignite config file.

#IGNITE_XML_CONFIG=/user/yarn/ignite/

IGNITE_XML_CONFIG=/user/hongmei/ignite/


#The hdfs path to libs which will be added to classpath.

IGNITE_USERS_LIBS=/user/hongmei/ignite/libs/


#The constraint on slave hosts.

#IGNITE_HOSTENAME_CONSTRAINT=


IGNITE_LOCAL_WORK_DIR=/u/hongmei/apache-ignite



Thank you very much!!!!

Hongmei





On Thu, Jun 9, 2016 at 9:43 AM, Nikolai Tikhonov <ntikhonov@apache.org>
wrote:

> It seems that your ignite configuration invalid. Could you share ignite-config.xml
> and /u/hongmei/apache-ignite/config/cluster.properties?
>
> On Thu, Jun 9, 2016 at 4:35 PM, Hongmei Zong <hongmei@kayak.com> wrote:
>
>> Thank you very much Nikolai !
>>
>> I found another issue regarding my Ignite YARN Integration:
>>
>> I  run Ignite YARN application on one client server machine 'c5hdpe001',
>> screenshot as the following:
>> Then I log onto one of the container node and the log is as the following:
>>
>> [hongmei@c5hdpe001 apache-ignite]$ hadoop jar
>> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
>> /u/hongmei/apache-ignite/libs/optional/ignite-yarn/ignite-yarn-1.6.0.jar
>> /u/hongmei/apache-ignite/config/cluster.properties
>>
>> WARNING: Use "yarn jar" to launch YARN applications.
>>
>> 16/06/09 08:55:20 INFO impl.TimelineClientImpl: Timeline service address:
>> http://c5hdp003.c5.runwaynine.com:8188/ws/v1/timeline/
>>
>> 16/06/09 08:55:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing
>> over to rm2
>>
>> 16/06/09 08:55:24 INFO impl.YarnClientImpl: Submitted application
>> application_1464374946035_29511
>>
>> Jun 09, 2016 8:55:24 AM org.apache.ignite.yarn.IgniteYarnClient main
>>
>> INFO: Submitted application. Application id:
>> application_1464374946035_29511
>>
>> Jun 09, 2016 8:55:27 AM org.apache.ignite.yarn.IgniteYarnClient main
>> *INFO: Application application_1464374946035_29511 is RUNNING.*
>>
>>
>> Then I log onto one of the container node and the log is as the
>> following: "It failed to start grid".
>> Any good suggestion? Thank you very much!!!
>>
>>
>> class org.apache.ignite.IgniteException: Failed to instantiate Spring XML
>> application context
>> [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/,
>> err=Line 1 in XML document from URL
>> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/]
>> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
>> 1; columnNumber: 1; Content is not allowed in prolog.]
>>         at
>> org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:904)
>>         at org.apache.ignite.Ignition.start(Ignition.java:350)
>>         at
>> org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
>> *Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
>> instantiate Spring XML application context
>> [springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/,
>> err=Line 1 in XML document from URL
>> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/]
>> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
>> 1; columnNumber: 1; Content is not allowed in prolog.]*
>>         at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:391)
>>         at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:104)
>>         at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.loadConfigurations(IgniteSpringHelperImpl.java:98)
>>         at
>> org.apache.ignite.internal.IgnitionEx.loadConfigurations(IgnitionEx.java:606)
>>         at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:807)
>>         at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
>>         at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:586)
>>         at
>> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:556)
>>         at org.apache.ignite.Ignition.start(Ignition.java:347)
>>         ... 1 more
>> *Caused by:
>> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line
>> 1 in XML document from URL
>> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/]
>> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
>> 1; columnNumber: 1; Content is not allowed in prolog.*
>>         at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:398)
>>         at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:335)
>>         at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:303)
>>         at
>> org.apache.ignite.internal.util.spring.IgniteSpringHelperImpl.applicationContext(IgniteSpringHelperImpl.java:379)
>>         ... 9 more
>> *Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber:
>> 1; Content is not allowed in prolog.*
>>         at
>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
>>         at
>> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1436)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:999)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
>>         at
>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
>>         at
>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
>>         at
>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
>>         at
>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
>>         at
>> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
>>         at
>> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
>>         at
>> org.springframework.beans.factory.xml.DefaultDocumentLoader.loadDocument(DefaultDocumentLoader.java:76)
>>         at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadDocument(XmlBeanDefinitionReader.java:428)
>>         at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:390)
>>         ... 12 more
>> *Failed to start grid: Failed to instantiate Spring XML application
>> context *[springUrl=file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/,
>> err=Line 1 in XML document from URL
>> [file:/disk/12/hadoop/yarn/local/usercache/hongmei/appcache/application_1464374946035_27403/container_e24_1464374946035_27403_01_110493/./ignite-config.xml/]
>> is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber:
>> 1; columnNumber: 1; Content is not allowed in prolog.]
>>
>>
>> Hongmei
>>
>>
>> On Thu, Jun 9, 2016 at 6:04 AM, Nikolai Tikhonov <ntikhonov@apache.org>
>> wrote:
>>
>>> *your_address1:47500..47510,your_address2:47500..47510
>>>> and your_address3:47500..47510 are the YARN master_host address, right?*
>>>>
>>>
>>> No, this addresses hosts on which deploy YARN cluster. For example, you
>>> have YARN cluster which contains two servers: 10.0.0.1 and 10.0.0.2. In
>>> this case you will have the following configuration:
>>>
>>> ipFinder.setAddresses(Arrays.asList("10.0.0.1:47500..47510",
>>> "10.0.0.2:47500..47510"));
>>>
>>>
>>
>

Mime
View raw message