incubator-ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vitthal \"Suhas\" Gogate" <gog...@hortonworks.com>
Subject Re: stack definition and component plugin
Date Tue, 17 Jan 2012 06:51:31 GMT
About Stack definition:

-- Look at the example stack at
"./controller/src/main/resources/org/apache/ambari/stacks/puppet1-0.json".
It has global variables defined that can be substituted throughout the
stack configuration. Ambari follows ruby template language.

-- There are few global variables are that are hidden or by default defined
by Ambari such as
    -- names of master hosts  e.g. ambari_namenode_host,
ambari_jobtracker_host (format is  ambari_<rolename>_host).
    -- ambari_cluster_name

     These default global variables are derived by ambari from cluster
definition and they would be documented for Ambari users.

About component plugins
   -- They are not user level plugins, they are supposed to be added by
service providers or component developers.  When you talk about
scalability, you mean execution performance or in terms of adding new
component to Ambari? How many new Hadoop components you think would be
added over next decade excluding ones already in the stack?  1, 10, 100,
1000?

About synchronization of services,
  -- As we talked once, I would plus one for avoiding any synchronization
while deploying and configuring. We should mandate services to tolerate if
dependent services are not running. In the existing Hadoop stack do you
know what services need synchronization other than  NN/DN. I heard from few
HDFS folks that it is a bug if DN goes down when NN is not up and can be
fixed very easily (if not already).

I would suggest we should have Ambari Meet-up  to talk about some of the
issues..

--Suhas

On Mon, Jan 16, 2012 at 10:08 PM, Eric Yang <eric818@gmail.com> wrote:

> Hi all,
>
> The current hierarchical stack definition is confusing to me.
> Supposedly, the definition can be expanded and flatten configuration
> key value pair, and an example of defining namenode url and get
> inherited in hbase configuration would look like this:
>
> {
>  ...
>  "components": {
>    "hdfs": {
>      "roles": {
>        "namenode": { /* override one value on the namenode */
>          "hadoop/hdfs-site": {
>            "dfs.https.enable": "true",
>            "fs.default.name": "hdfs://${namenode}:${port}/"
>          }
>        }
>      }
>    },
>    "hbase": {
>      "roles": {
>        "region-server": {
>          "hbase/hbase-site": {
>            "hbase.rootdir":
> "${components.hdfs.namenode.hadoop/hdfs-site.fs.default.name}/hbase"
>        }
>      }
>    }
>  }
> }
>
> hbase.rootdir is a key for hbase-site.xml, and it should contain key
> of "fs.default.name" plus additional path for hbase to store data.  In
> my interpretation of the macro would look like
> ${components.hdfs.namenode.hadoop/hdfs-site.fs.default.name}/hbase.
> This seems like a utterly awkward method of describing inheritance.
> Why don't we use a flat name space to remove additional logistics
> imposed by Ambari.  I agree that the syntax is fully accurate, but it
> is a larger headache to maintain this hierarchical structure.
>
> The second problem is the component plugin architecture sounds good in
> theory.  I see some scalability issues with this approach.  Each
> component describes the components that it depends on.  This could
> interfere with introducing new components.  i.e. Mapreduce component
> depends on HDFS.  A new component is introduced, and name HBase.  Now,
> the Mapreduce component needs to update it's dependency to HDFS and
> HBase and ZooKeeper.  For introducing new component, there is a lot of
> plugins updates to make the new version work.  The plugin writer also
> needs to make theoretical assumption that if components X is installed
> do Y, otherwise do Z.  Conditional assumption in plugin introduces
> uncertainty and corner cases into the deployment system.  The number
> of permutation can greatly exceed the logics that is required to
> handle by the plugin.
>
> Instead of using plugin architecture to manage deployment, it would be
> safer to use a scripting approach to enable power administrator to
> deploy a stack of software by writing shell script like script to
> accomplish the deployment tasks.  The recipes scripts can be shared by
> the community to automate software stack deployment.  This will ensure
> the scope of Ambari deployment is focus on cross nodes orchestration
> without having to build bell and whistles which does not scale well in
> the long term.
>
> What do you guys think?
>
> regards,
> Eric
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message