ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Veselovsky (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-483) Simplify Hadoop "ignition"
Date Fri, 13 Mar 2015 19:26:38 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ivan Veselovsky updated IGNITE-483:
-----------------------------------
    Description: 
Currently in Ignite we have setup-hadoop script + java class to replace configs and create
symlinks to Ignite libraries.
This setup script and Ignite launcher script have many dependencies on the Hadoop distribution
layout (files, configs, etc.)

I suggest to simplify this:
1) It seems to me that *no* symlink and/or library copying needed at all. Instead, to make
hadoop client work via Ignite, we can use the following wrapper around the default hadoop
client launcher script:
file "hadoop-ignited" :

{code}
# Ignite home is needed to allow Ignite libraries to find the logger config (it is resolved
relative to the Ignite home):
export IGNITE_HOME=/home/ignite/ignite-hadoop-1.0.0-RC3-SNAPSHOT

# Add necessary Ignite libraries to the Hadoop client classpath:
export HADOOP_CLASSPATH=${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar

hadoop --config ${IGNITE_HOME}/ignite-conf "${@}"
{code}

where  {code}${IGNITE_HOME}/ignite-conf{code} is folder inside IGNITE distribution where the
2 custom configs are located: core-site.xml, mapred-site.xml .

(If the user wants only Ignite MapRed engine, but does not need Ignite filesystem, he symlinks
the default mapred-site.xml in this directory.)

This way, in order to use fully pre-configured Hadoop cluster the user only needs to add "hadoop-ignited"
script into PATH.
After that instead of running hadoop jar examples.jar pi 10 10 , use can run hadoop-ignited
jar examples.jar pi 10 10 to run the sample on fully ignited Hadoop.

2) The only thing that is still needs attention is how the Ignite node should find the Hadoop
libraries. My suggestion is to provide 3 versions of config scripts depending on the supported
layouts, like "layout-apache", "layout-cloudera", and "layout-bigtop-hortonworks" (I suppose,
the latter 2 are identical) . After that ask use after Ignite archive unzipping write 1 line
into startup script, like LAYOUT=cloudera, and that's it -- the corresponding env script will
be picked up to set up env variables needed by the user's Hadoop distribution.

TODO: will this approach work with Hive, etc.?

  was:
Currently in Ignite we have setup-hadoop script + java class to replace configs and create
symlinks to Ignite libraries.
This setup script and Ignite launcher script have many dependencies on the Hadoop distribution
layout (files, configs, etc.)

I suggest to simplify this:
1) It seems to me that *no* symlink and/or library copying needed at all. Instead, to make
hadoop client work via Ignite, we can use the following wrapper around the default hadoop
client launcher script:
file "hadoop-ignited" :

{code}
# Ignite home is needed to allow Ignite libraries to find the logger config (it is resolved
relative to the Ignite home):
export IGNITE_HOME=/home/ignite/ignite-hadoop-1.0.0-RC3-SNAPSHOT

# Add necessary Ignite libraries to the Hadoop client classpath:
export HADOOP_CLASSPATH=${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar

hadoop --config ${IGNITE_HOME}/ignite-conf "${@}"
{code}

where  ${IGNITE_HOME}/ignite-conf is folder inside IGNITE distribution where the 2 custom
configs are located: core-site.xml, mapred-site.xml .

(If the user wants only Ignite MapRed engine, but does not need Ignite filesystem, he symlinks
the default mapred-site.xml in this directory.)

This way, in order to use fully pre-configured Hadoop cluster the user only needs to add "hadoop-ignited"
script into PATH.
After that instead of running hadoop jar examples.jar pi 10 10 , use can run hadoop-ignited
jar examples.jar pi 10 10 to run the sample on fully ignited Hadoop.

2) The only thing that is still needs attention is how the Ignite node should find the Hadoop
libraries. My suggestion is to provide 3 versions of config scripts depending on the supported
layouts, like "layout-apache", "layout-cloudera", and "layout-bigtop-hortonworks" (I suppose,
the latter 2 are identical) . After that ask use after Ignite archive unzipping write 1 line
into startup script, like LAYOUT=cloudera, and that's it -- the corresponding env script will
be picked up to set up env variables needed by the user's Hadoop distribution.

TODO: will this approach work with Hive, etc.?


> Simplify Hadoop "ignition" 
> ---------------------------
>
>                 Key: IGNITE-483
>                 URL: https://issues.apache.org/jira/browse/IGNITE-483
>             Project: Ignite
>          Issue Type: Task
>            Reporter: Ivan Veselovsky
>            Assignee: Ivan Veselovsky
>
> Currently in Ignite we have setup-hadoop script + java class to replace configs and create
symlinks to Ignite libraries.
> This setup script and Ignite launcher script have many dependencies on the Hadoop distribution
layout (files, configs, etc.)
> I suggest to simplify this:
> 1) It seems to me that *no* symlink and/or library copying needed at all. Instead, to
make hadoop client work via Ignite, we can use the following wrapper around the default hadoop
client launcher script:
> file "hadoop-ignited" :
> {code}
> # Ignite home is needed to allow Ignite libraries to find the logger config (it is resolved
relative to the Ignite home):
> export IGNITE_HOME=/home/ignite/ignite-hadoop-1.0.0-RC3-SNAPSHOT
> # Add necessary Ignite libraries to the Hadoop client classpath:
> export HADOOP_CLASSPATH=${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar
> hadoop --config ${IGNITE_HOME}/ignite-conf "${@}"
> {code}
> where  {code}${IGNITE_HOME}/ignite-conf{code} is folder inside IGNITE distribution where
the 2 custom configs are located: core-site.xml, mapred-site.xml .
> (If the user wants only Ignite MapRed engine, but does not need Ignite filesystem, he
symlinks the default mapred-site.xml in this directory.)
> This way, in order to use fully pre-configured Hadoop cluster the user only needs to
add "hadoop-ignited" script into PATH.
> After that instead of running hadoop jar examples.jar pi 10 10 , use can run hadoop-ignited
jar examples.jar pi 10 10 to run the sample on fully ignited Hadoop.
> 2) The only thing that is still needs attention is how the Ignite node should find the
Hadoop libraries. My suggestion is to provide 3 versions of config scripts depending on the
supported layouts, like "layout-apache", "layout-cloudera", and "layout-bigtop-hortonworks"
(I suppose, the latter 2 are identical) . After that ask use after Ignite archive unzipping
write 1 line into startup script, like LAYOUT=cloudera, and that's it -- the corresponding
env script will be picked up to set up env variables needed by the user's Hadoop distribution.
> TODO: will this approach work with Hive, etc.?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message