ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Veselovsky (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (IGNITE-483) Simplify Hadoop "ignition"
Date Fri, 13 Mar 2015 19:23:39 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ivan Veselovsky reassigned IGNITE-483:

    Assignee: Ivan Veselovsky

> Simplify Hadoop "ignition" 
> ---------------------------
>                 Key: IGNITE-483
>                 URL: https://issues.apache.org/jira/browse/IGNITE-483
>             Project: Ignite
>          Issue Type: Task
>            Reporter: Ivan Veselovsky
>            Assignee: Ivan Veselovsky
> Currently in Ignite we have setup-hadoop script + java class to replace configs and create
symlinks to Ignite libraries.
> This setup script and Ignite launcher script have many dependencies on the Hadoop distribution
layout (files, configs, etc.)
> I suggest to simplify this:
> 1) It seems to me that *no* symlink and/or library copying needed at all. Instead, to
make hadoop client work via Ignite, we can use the following wrapper around the default hadoop
client launcher script:
> file "hadoop-ignited" :
> -----------------------------------------------
> # Ignite home is needed to allow Ignite libraries to find the logger config (it is resolved
relative to the Ignite home):
> export IGNITE_HOME=/home/ignite/ignite-hadoop-1.0.0-RC3-SNAPSHOT
> # Add necessary Ignite libraries to the Hadoop client classpath:
> export HADOOP_CLASSPATH=${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar
> hadoop --config ${IGNITE_HOME}/ignite-conf "${@}"
> -----------------------------------------------
> where  ${IGNITE_HOME}/ignite-conf is folder inside IGNITE distribution where the 2 custom
configs are located: core-site.xml, mapred-site.xml .
> (If the user wants only Ignite MapRed engine, but does not need Ignite filesystem, he
symlinks the default mapred-site.xml in this directory.)
> This way, in order to use fully pre-configured Hadoop cluster the user only needs to
add "hadoop-ignited" script into PATH.
> 2) The only thing that is still needs attention is how the Ignite node should find the
Hadoop libraries. My suggestion is to provide 3 versions of config scripts depending on the
supported layouts, like "layout-apache", "layout-cloudera", and "layout-bigtop-hortonworks"
(I suppose, the latter 2 are identical) . After that ask use after Ignite archive unzipping
write 1 line into startup script, like LAYOUT=cloudera, and that's it -- the corresponding
env script will be picked up to set up env variables needed by the user's Hadoop distribution.
> TODO: will this approach work with Hive, etc.?

This message was sent by Atlassian JIRA

View raw message