hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Mosebach (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3999) Need to add host capabilites / abilities
Date Mon, 13 Oct 2008 17:20:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639117#action_12639117

Kai Mosebach commented on HADOOP-3999:

Thanks a lot for your comment!

Regarding 1.) im just implementing some sort of plugin system which allows us to load arbitrary
plugin classes that have to implement the CapabilityPlugin class. Its working with Maps so
the plugins are quite free in what the put into it as results.
This is necessary since many benchmarks are available under a non-apache license only (i.e.
scimark2) and in this way they can still be used. Furthermore i think is makes sense to define
which "key(s)" from the CapabilitiyPlugin are supposed to be your relevant keys for your scheduler
(i.e. the capability.performance.dhrystone value combined w/ capability.performance.diskwrite
and the capability.hardware.memory might be interesting, other combinations for others - a
good default setting is important here but should be tweakable - at least for testing). The
plugin system should also be able to handle shell scripts/tools since some benchmarks (i/o
etc) are nearly impossible in java.
Furthermore this system can also hold software info as well as other at the same time. it
will have aging (since we dont want to do some (.i.e. performance) tests on every start) and
serialization of the data.

I assume this system fits into other domains (beside sw/hw) as well.

Regarding 2.) I see this danger as well ... anyway i think it still makes a lot of sence if
you can assume you have a special tool onsite you can use (as we have - using a lot of biological
add ons - which you dont want to reinvent ;). 
Further down the road, if we see superclouds that need to handle multiple customers with different
needs / specs / service levels we also should be able to differ between nodes (i call it individualized
nodes at this point)
Looking at smaller setups / test setups with a lot of heterogeneousity (as we have here) we
could be better of, if we can make the scheduler stop using machines for workload which are
needed otherwise.
Regarding the "work-near-the-data", not only the scheduler has to know about specs of the
nodes, also the dfs could make use of it (actually should prefer fast IO machines eventually)

For friends I often use the metaphor : different people are living in the cloud, i.e. workers,
scientists, housewifes. so why give mathematical problems to the housewife and ironing jobs
to the scientists?

Regarding 3.) (and 2) maybe the performance system is - in the beginning - more usable for
core-developers and performance tweakers than for my biologist neighbors who just were forced
to develop in java.

> Need to add host capabilites / abilities
> ----------------------------------------
>                 Key: HADOOP-3999
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3999
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: metrics
>         Environment: Any
>            Reporter: Kai Mosebach
> The MapReduce paradigma is limited to run MapReduce jobs with the lowest common factor
of all nodes in the cluster.
> On the one hand this is wanted (cloud computing, throw simple jobs in, nevermind who
does it)
> On the other hand this is limiting the possibilities quite a lot, for instance if you
had data which could/needs to be fed to a 3rd party interface like Mathlab, R, BioConductor
you could solve a lot more jobs via hadoop.
> Furthermore it could be interesting to know about the OS, the architecture, the performance
of the node in relation to the rest of the cluster. (Performance ranking)
> i.e. if i'd know about a sub cluster of very computing performant nodes or a sub cluster
of very fast disk-io nodes, the job tracker could select these nodes regarding a so called
job profile (i.e. my job is a heavy computing job / heavy disk-io job), which can usually
be estimated by a developer before.
> To achieve this, node capabilities could be introduced and stored in the DFS, giving
> a1.) basic information about each node (OS, ARCH)
> a2.) more sophisticated infos (additional software, path to software, version). 
> a3.) PKI collected about the node (disc-io, cpu power, memory)
> a4.) network throughput to neighbor hosts, which might allow generating a network performance
map over the cluster
> This would allow you to
> b1.) generate jobs that have a profile (computing intensive, disk io intensive, net io
> b2.) generate jobs that have software dependencies (run on Linux only, run on nodes with
MathLab only)
> b3.) generate a performance map of the cluster (sub clusters of fast disk nodes, sub
clusters of fast CPU nodes, network-speed-relation-map between nodes)
> From step b3) you could then even acquire statistical information which could again be
fed into the DFS Namenode to see if we could store data on fast disk subclusters only (that
might need to be a tool outside of hadoop core though)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message