Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <227946364.1223918444366.JavaMail.jira@brutus>
Date: Mon, 13 Oct 2008 10:20:44 -0700 (PDT)
From: "Kai Mosebach (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3999) Need to add host capabilites /
 abilities
In-Reply-To: <248855554.1219398166106.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639117#action_12639117 ] 

Kai Mosebach commented on HADOOP-3999:
--------------------------------------

Thanks a lot for your comment!

Regarding 1.) im just implementing some sort of plugin system which allows us to load arbitrary plugin classes that have to implement the CapabilityPlugin class. Its working with Maps so the plugins are quite free in what the put into it as results.
This is necessary since many benchmarks are available under a non-apache license only (i.e. scimark2) and in this way they can still be used. Furthermore i think is makes sense to define which "key(s)" from the CapabilitiyPlugin are supposed to be your relevant keys for your scheduler (i.e. the capability.performance.dhrystone value combined w/ capability.performance.diskwrite and the capability.hardware.memory might be interesting, other combinations for others - a good default setting is important here but should be tweakable - at least for testing). The plugin system should also be able to handle shell scripts/tools since some benchmarks (i/o etc) are nearly impossible in java.
Furthermore this system can also hold software info as well as other at the same time. it will have aging (since we dont want to do some (.i.e. performance) tests on every start) and serialization of the data.

I assume this system fits into other domains (beside sw/hw) as well.

Regarding 2.) I see this danger as well ... anyway i think it still makes a lot of sence if you can assume you have a special tool onsite you can use (as we have - using a lot of biological add ons - which you dont want to reinvent ;). 
Further down the road, if we see superclouds that need to handle multiple customers with different needs / specs / service levels we also should be able to differ between nodes (i call it individualized nodes at this point)
Looking at smaller setups / test setups with a lot of heterogeneousity (as we have here) we could be better of, if we can make the scheduler stop using machines for workload which are needed otherwise.
Regarding the "work-near-the-data", not only the scheduler has to know about specs of the nodes, also the dfs could make use of it (actually should prefer fast IO machines eventually)

For friends I often use the metaphor : different people are living in the cloud, i.e. workers, scientists, housewifes. so why give mathematical problems to the housewife and ironing jobs to the scientists?

Regarding 3.) (and 2) maybe the performance system is - in the beginning - more usable for core-developers and performance tweakers than for my biologist neighbors who just were forced to develop in java.


> Need to add host capabilites / abilities
> ----------------------------------------
>
>                 Key: HADOOP-3999
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3999
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: metrics
>         Environment: Any
>            Reporter: Kai Mosebach
>
> The MapReduce paradigma is limited to run MapReduce jobs with the lowest common factor of all nodes in the cluster.
> On the one hand this is wanted (cloud computing, throw simple jobs in, nevermind who does it)
> On the other hand this is limiting the possibilities quite a lot, for instance if you had data which could/needs to be fed to a 3rd party interface like Mathlab, R, BioConductor you could solve a lot more jobs via hadoop.
> Furthermore it could be interesting to know about the OS, the architecture, the performance of the node in relation to the rest of the cluster. (Performance ranking)
> i.e. if i'd know about a sub cluster of very computing performant nodes or a sub cluster of very fast disk-io nodes, the job tracker could select these nodes regarding a so called job profile (i.e. my job is a heavy computing job / heavy disk-io job), which can usually be estimated by a developer before.
> To achieve this, node capabilities could be introduced and stored in the DFS, giving you
> a1.) basic information about each node (OS, ARCH)
> a2.) more sophisticated infos (additional software, path to software, version). 
> a3.) PKI collected about the node (disc-io, cpu power, memory)
> a4.) network throughput to neighbor hosts, which might allow generating a network performance map over the cluster
> This would allow you to
> b1.) generate jobs that have a profile (computing intensive, disk io intensive, net io intensive)
> b2.) generate jobs that have software dependencies (run on Linux only, run on nodes with MathLab only)
> b3.) generate a performance map of the cluster (sub clusters of fast disk nodes, sub clusters of fast CPU nodes, network-speed-relation-map between nodes)
> From step b3) you could then even acquire statistical information which could again be fed into the DFS Namenode to see if we could store data on fast disk subclusters only (that might need to be a tool outside of hadoop core though)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.