Mailing-List: contact dev-help@storm.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@storm.incubator.apache.org
From: dashengju <git@git.apache.org>
To: dev@storm.incubator.apache.org
Reply-To: dev@storm.incubator.apache.org
Message-ID: <git-pr-38-incubator-storm@git.apache.org>
Subject: [GitHub] incubator-storm pull request: STORM-200 Proposal for
 Multilang's M...
Content-Type: text/plain
Date: Fri, 21 Feb 2014 06:16:27 +0000 (UTC)

GitHub user dashengju opened a pull request:

    https://github.com/apache/incubator-storm/pull/38

    STORM-200 Proposal for Multilang's Metrics feature

    for https://issues.apache.org/jira/browse/STORM-200
    
    Storm 0.9.0.1 exposes a metrics interface to report summary statistics across the full topology. We can build our own metric, and build metrics consumer to use those statistics. 
    
    But when we use Multilang(ie. Python), we can not use this feature. So we want to summit a proposal for multilang's metrics. 
    
    The specifics of the proposal: 
    1. The main idea is: when user want to add a metric statistics in multilang(python) bolt， 
        a) he need first create a metric object and register in ShellBolt's sub-class, 
        b) then update the metric in Python bolt process through RPC call. 
    
    2. In Metrics API: 
        a) extend IMetric interface add a method for RPC call：
                       public void updateMetricFromRPC(List<Object> params); 
        b) modify IMetric implements, to support updateMetricFromRPC; 
    
    3. In ShellBolt， 
        a) we have a Map<String, IMetric> to hold user's registered metrics object； 
        b) we have a method registerMetric(String name, T metric) for user to register their metrics object; 
        c) we have a method handleMetrics(Map action) to handle RPC call from Python process； 
    
    4) In Multilang protocol: add a command "metrics" for shell process to make RPC call. The protocol is: {"command":"metrics", "name":"metric-registerd-name", "params":["param-1", param-2]} 
    
    5) In storm.py：add rpcMetrics(name, params), user can update remote metric through this RPC call. 
    
    --------ExamplePythonBolt.java----------------------------------------------------------------------------------------------------
    public class ExamplePythonBolt extends ShellBolt implements IRichBolt {
    	private static final long serialVersionUID = 1999209252187463355L;
    
    	private TopologyConfig topologyConfig;
    	
    	public ExamplePythonBolt(TopologyConfig config) {
    		super("python", "ExampleBolt.py", config.topologyName);
    		topologyConfig = config;
    	}
    	
        public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
        	super.prepare(stormConf, context, collector);
        	
        	CountMetric cMetric = new CountMetric();                         //generate a metric object
        	context.registerMetric("PythonBoltCount", cMetric, 120);  //register metric object to context
        	this.registerMetric("PythonBoltCount", cMetric);                //register metric object to ShellBolt
        }
    
    	public void declareOutputFields(OutputFieldsDeclarer declarer) {
    		declarer.declare(new Fields("LogRecord"));
    	}
    
    	public Map<String, Object> getComponentConfiguration() {
    		return null;
    	}
    }
    
    --------ExampleBolt.py---------------------------------------------------------------------------------------------------
    class ExampleBolt(storm.BasicBolt):
        def __init__(self, boltParams):
            #from here, you can get params from java
            pass
        
        def initialize(self, stormconf, context):
            pass
            
        def process(self, tup):
            try:         
                
                MTLOGGER.info("TestAction", "handle the json %s" % (logjson))  
                storm.emit([tup[0]])
                
                storm.rpcMetrics("PythonBoltCount", [1])        //update your PythonBoltCount metric with param list "[1]"
            except Exception,tx:
                //handle error


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dashengju/incubator-storm master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-storm/pull/38.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #38
    
----
commit edadde30420e5b2cd3e3d248a6a2268f8dedc22f
Author: JuDasheng <judasheng@meituan.com>
Date:   2014-01-28T06:47:30Z

    modify metrics system to support shellbolt's metrics

commit 52a4fbbd670cfac3a9174bd6c7ad63d3a2282fa8
Author: JuDasheng <judasheng@meituan.com>
Date:   2014-02-21T06:02:28Z

    modify for metrics

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---