incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: Rolling out incremental changes to Chukwa cluster?
Date Mon, 24 May 2010 19:24:03 GMT
Changes to mdl.xml need only happen on machines running the demux
manager, not on the entire cluster. So that should ease migration.

Breaking mdl.xml into smaller pieces seems like a good idea; my
instinct would be to modify the demux manager to read all files in
conf starting with mdl, so you can have mdl-core, mdl-site, or what
have you.  But I think we should move implementation discussion either
to JIRA or to chukwa-dev@hadoop.apache.org.

--Ari

On Mon, May 24, 2010 at 12:04 PM, Kirk True <kirk@mustardgrain.com> wrote:
> Hi all,
>
> I'm interested in this too. Is there any objection to creating a mechanism
> and/or naming convention that would allow certain files (like mdl.xml) to be
> split into data-type specific files? If not, any tips on how to go about it?
> :)
>
> Thanks,
> Kirk
>
> On 5/19/10 2:27 PM, William Bajzek wrote:
>>
>> Hi all,
>> I am looking for a good way to roll out updates to my client's chukwa
>> deployment. Over time, they will be incrementally adding new Demuxes and the
>> associated configuration and schema (they are using Mysql and do not intend
>> to change). Is there a recommended way of doing this? It looks like as it
>> is, one would have to add the new configuration information to each node,
>> run schema change scripts, and then bounce the cluster. I didn't see a way
>> of adding schema incrementally, so I assume we will either need to do it in
>> individual scripts and then add them to the main database_create_tables.sql
>> script, or else come up with some mechanism for breaking that script up into
>> manageable chunks.
>>
>> We're thinking of reusing chukwa's mechanism for sending commands out to
>> the nodes on the cluster to push out the config changes, then bounce Chukwa
>> and Hadoop on each of the nodes, and finally add the new JARs to hadoop.
>>
>> I think the other outstanding question is, how do we make sure we're not
>> interrupting a running job when we stop and start the cluster?
>>
>> Thanks,
>> -William Bajzek
>>
>>
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message