incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jerome Boulon <jbou...@netflix.com>
Subject Re: Rolling out incremental changes to Chukwa cluster?
Date Mon, 24 May 2010 19:35:41 GMT
Not sure if this is public yet but Hadoop config was supposed to be able to
load addition files from the main config file (similar the the addRessource
but built-in).
/Jerome.

On 5/24/10 12:24 PM, "Ariel Rabkin" <asrabkin@gmail.com> wrote:

> Changes to mdl.xml need only happen on machines running the demux
> manager, not on the entire cluster. So that should ease migration.
> 
> Breaking mdl.xml into smaller pieces seems like a good idea; my
> instinct would be to modify the demux manager to read all files in
> conf starting with mdl, so you can have mdl-core, mdl-site, or what
> have you.  But I think we should move implementation discussion either
> to JIRA or to chukwa-dev@hadoop.apache.org.
> 
> --Ari
> 
> On Mon, May 24, 2010 at 12:04 PM, Kirk True <kirk@mustardgrain.com> wrote:
>> Hi all,
>> 
>> I'm interested in this too. Is there any objection to creating a mechanism
>> and/or naming convention that would allow certain files (like mdl.xml) to be
>> split into data-type specific files? If not, any tips on how to go about it?
>> :)
>> 
>> Thanks,
>> Kirk
>> 
>> On 5/19/10 2:27 PM, William Bajzek wrote:
>>> 
>>> Hi all,
>>> I am looking for a good way to roll out updates to my client's chukwa
>>> deployment. Over time, they will be incrementally adding new Demuxes and the
>>> associated configuration and schema (they are using Mysql and do not intend
>>> to change). Is there a recommended way of doing this? It looks like as it
>>> is, one would have to add the new configuration information to each node,
>>> run schema change scripts, and then bounce the cluster. I didn't see a way
>>> of adding schema incrementally, so I assume we will either need to do it in
>>> individual scripts and then add them to the main database_create_tables.sql
>>> script, or else come up with some mechanism for breaking that script up into
>>> manageable chunks.
>>> 
>>> We're thinking of reusing chukwa's mechanism for sending commands out to
>>> the nodes on the cluster to push out the config changes, then bounce Chukwa
>>> and Hadoop on each of the nodes, and finally add the new JARs to hadoop.
>>> 
>>> I think the other outstanding question is, how do we make sure we're not
>>> interrupting a running job when we stop and start the cluster?
>>> 
>>> Thanks,
>>> -William Bajzek
>>> 
>>> 
>> 
> 
> 


Mime
View raw message