hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: Simple MapReduce logic using Java API
Date Tue, 31 Mar 2015 18:21:45 GMT
What is the reason of using the queue?
"job.getConfiguration().set("mapred.job.queue.name", "exp_dsa");"

Is your mapper or reducer even been called?

Try adding the override annotation to the map/reduce methods as below:

@Override
 public void map(Object key, Text value, Context context) throws
IOException, InterruptedException {

Regards,
Shahab

On Tue, Mar 31, 2015 at 3:26 AM, bradford li <bradfordli@gmail.com> wrote:

> I'm not sure why my Mapper and Reducer have no output. The logic behind my
> code is, given a file of UUIDs (new line separated), I want to use
> `globStatus` to display all the paths to all potential files that the UUID
> might be in. Open and read the file. Each file contains 1-n lines of JSON.
> The UUID is in `event_header.event_id` in the JSON.
>
> Right now the MapReduce job runs without errors. However, something is
> wrong because I dont have any output. I'm not sure how to debug MapReduce
> jobs as well. If someone could provide me a source that would be awesome!
> The expected output from this program should be
>
>     UUID_1 1
>     UUID_2 1
>     UUID_3 1
>     UUID_4 1
>     ...
>     ...
>     UUID_n 1
>
> In my logic, the output file should be the UUIDs with a 1 next to them
> because upon found, 1 is written, if not found 0 is written. They should be
> all 1's because I pulled the UUIDs from the source.
>
> My Reducer currently does not do anything except I just wanted to see if I
> could get some simple logic working. There are most likely bugs in my code
> as I dont know have a easy way to debug MapReduce jobs
>
> Driver:
>
>     public class SearchUUID {
>
>         public static void main(String[] args) throws Exception {
>             Configuration conf = new Configuration();
>             Job job = Job.getInstance(conf, "UUID Search");
>             job.getConfiguration().set("mapred.job.queue.name",
> "exp_dsa");
>             job.setJarByClass(SearchUUID.class);
>             job.setMapperClass(UUIDMapper.class);
>             job.setReducerClass(UUIDReducer.class);
>             job.setOutputKeyClass(Text.class);
>             job.setOutputValueClass(Text.class);
>             FileInputFormat.addInputPath(job, new Path(args[0]));
>             FileOutputFormat.setOutputPath(job, new Path(args[1]));
>             System.exit(job.waitForCompletion(true) ? 0 : 1);
>         }
>     }
>
>
> UUIDMapper:
>
>     public class UUIDMapper extends Mapper<Object, Text, Text, Text> {
>         public void map(Object key, Text value, Context context) throws
> IOException, InterruptedException {
>
>             try {
>                 Text one = new Text("1");
>                 Text zero = new Text("0");
>
>                 FileSystem fs = FileSystem.get(new Configuration());
>                 FileStatus[] paths = fs.globStatus(new
> Path("/data/path/to/file/d_20150330-1650"));
>                 for (FileStatus path : paths) {
>                     BufferedReader br = new BufferedReader(new
> InputStreamReader(fs.open(path.getPath())));
>                     String json_string = br.readLine();
>                     while (json_string != null) {
>                         JsonElement jelement = new
> JsonParser().parse(json_string);
>                         JsonObject jsonObject = jelement.getAsJsonObject();
>                         jsonObject =
> jsonObject.getAsJsonObject("event_header");
>                         jsonObject =
> jsonObject.getAsJsonObject("event_id");
>
>                         if
> (value.toString().equals(jsonObject.getAsString())) {
>                             System.out.println(value.toString() +
> "slkdjfksajflkjsfdkljsadfk;ljasklfjklasjfklsadl;sjdf");
>                             context.write(value, one);
>                         } else {
>                             context.write(value, zero);
>                         }
>
>                         json_string = br.readLine();
>                     }
>                 }
>             } catch (IOException failed) {
>             }
>         }
>     }
>
>
> Reducer:
>
>     public class UUIDReducer extends Reducer<Text, Text, Text, Text>{
>
>         public void reduce(Text key, Text value, Context context) throws
> IOException, InterruptedException{
>             context.write(key, value);
>         }
>     }
>
>

Mime
View raw message