hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aman <aman_d...@hotmail.com>
Subject Re: Please help with hadoop configuration parameter set and get
Date Fri, 17 Dec 2010 21:10:09 GMT

Pei 
You cannot change configuration in Mapper method and see it in main method.
There are 2 ways to do this
1. Have your mapper store data in HDFS 
2. Use counters

I recommend using the 2nd approach. What you can do is set a counter in the
Mapper, something like 
context.getCounter("MyCounter").increment(1);
and then extract this counter in the main method. I had this class that
emits out all counters which I then load into a DB. You can tweak this class
to get the value of the counter you want. 

 public class JobMetadata {
    Job job = null;

    public JobMetadata(Job job) {
        this.job = job;
    }
    public String getJobMetadata() throws IOException {
        /**
         * Get Job Name, Id and URL
         */
        String jobName = job.getJobName();
        String trackingUrl = job.getTrackingURL();


        /**
         * Get Job Counter
         */
        Counters jobCounters = job.getCounters();
        int totalCounters = jobCounters.countCounters();
        CounterGroup counterGroup = null;
        Counter counter = null;

        String counterGroupName = null;
        String counterName = null;
        long counterValue = 0;

        Iterator itGroups = jobCounters.iterator();
        StringBuffer sb = new StringBuffer();
        while (itGroups.hasNext()) {
            counterGroup = (CounterGroup) itGroups.next();
            counterGroupName = counterGroup.getDisplayName();

            Iterator itCounter = counterGroup.iterator();
            while (itCounter.hasNext()) {
                counter = (Counter) itCounter.next();
                counterName = counter.getDisplayName();
                counterValue = counter.getValue();
                sb.append(jobName + "\t" + trackingUrl + "\t" +
counterGroupName
                        + "\t" + counterName + "\t" + counterValue + "\n");
            }
        }
        return sb.toString();
    }
}


Peng, Wei wrote:
> 
> Hi,
> 
>  
> 
> I am a newbie of hadoop.
> 
> Today I was struggling with a hadoop problem for several hours.
> 
>  
> 
> I initialize a parameter by setting job configuration in main.
> 
> E.g. Configuration con = new Configuration();
> 
> con.set("test", "1");
> 
> Job job = new Job(con);
> 
>  
> 
> Then in the mapper class, I want to set "test" to "2". I did it by 
> 
> context.getConfiguration().set("test","2");
> 
>  
> 
> Finally in the main method, after the job is finished, I check the
> "test" again by
> 
> job.getConfiguration().get("test");
> 
>  
> 
> However, the value of "test" is still "1".
> 
>  
> 
> The reason why I want to change the parameter inside Mapper class is
> that I want to determine when to stop an iteration in the main method.
> For example, for doing breadth-first search, when there is no new nodes
> are added for further expansion, the searching iteration should stop. 
> 
>  
> 
> Your help will be deeply appreciated. Thank you
> 
>  
> 
> Wei
> 
> 
> 

-- 
View this message in context: http://lucene.472066.n3.nabble.com/Please-help-with-hadoop-configuration-parameter-set-and-get-tp2103303p2107316.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Mime
View raw message