hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bradford li <bradfor...@gmail.com>
Subject Simple MapReduce logic using Java API
Date Tue, 31 Mar 2015 07:26:58 GMT
I'm not sure why my Mapper and Reducer have no output. The logic behind my
code is, given a file of UUIDs (new line separated), I want to use
`globStatus` to display all the paths to all potential files that the UUID
might be in. Open and read the file. Each file contains 1-n lines of JSON.
The UUID is in `event_header.event_id` in the JSON.

Right now the MapReduce job runs without errors. However, something is
wrong because I dont have any output. I'm not sure how to debug MapReduce
jobs as well. If someone could provide me a source that would be awesome!
The expected output from this program should be

    UUID_1 1
    UUID_2 1
    UUID_3 1
    UUID_4 1
    ...
    ...
    UUID_n 1

In my logic, the output file should be the UUIDs with a 1 next to them
because upon found, 1 is written, if not found 0 is written. They should be
all 1's because I pulled the UUIDs from the source.

My Reducer currently does not do anything except I just wanted to see if I
could get some simple logic working. There are most likely bugs in my code
as I dont know have a easy way to debug MapReduce jobs

Driver:

    public class SearchUUID {

        public static void main(String[] args) throws Exception {
            Configuration conf = new Configuration();
            Job job = Job.getInstance(conf, "UUID Search");
            job.getConfiguration().set("mapred.job.queue.name", "exp_dsa");
            job.setJarByClass(SearchUUID.class);
            job.setMapperClass(UUIDMapper.class);
            job.setReducerClass(UUIDReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(Text.class);
            FileInputFormat.addInputPath(job, new Path(args[0]));
            FileOutputFormat.setOutputPath(job, new Path(args[1]));
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    }


UUIDMapper:

    public class UUIDMapper extends Mapper<Object, Text, Text, Text> {
        public void map(Object key, Text value, Context context) throws
IOException, InterruptedException {

            try {
                Text one = new Text("1");
                Text zero = new Text("0");

                FileSystem fs = FileSystem.get(new Configuration());
                FileStatus[] paths = fs.globStatus(new
Path("/data/path/to/file/d_20150330-1650"));
                for (FileStatus path : paths) {
                    BufferedReader br = new BufferedReader(new
InputStreamReader(fs.open(path.getPath())));
                    String json_string = br.readLine();
                    while (json_string != null) {
                        JsonElement jelement = new
JsonParser().parse(json_string);
                        JsonObject jsonObject = jelement.getAsJsonObject();
                        jsonObject =
jsonObject.getAsJsonObject("event_header");
                        jsonObject = jsonObject.getAsJsonObject("event_id");

                        if
(value.toString().equals(jsonObject.getAsString())) {
                            System.out.println(value.toString() +
"slkdjfksajflkjsfdkljsadfk;ljasklfjklasjfklsadl;sjdf");
                            context.write(value, one);
                        } else {
                            context.write(value, zero);
                        }

                        json_string = br.readLine();
                    }
                }
            } catch (IOException failed) {
            }
        }
    }


Reducer:

    public class UUIDReducer extends Reducer<Text, Text, Text, Text>{

        public void reduce(Text key, Text value, Context context) throws
IOException, InterruptedException{
            context.write(key, value);
        }
    }

Mime
View raw message