hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xeon Mailinglist <xeonmailingl...@gmail.com>
Subject How can I create a SequenceFiles with `org.apache.hadoop.io.Text`?
Date Wed, 06 Jan 2016 15:38:22 GMT
Hi,

This is a snippet of a Gridmix code available in Hadop MapReduce V1, but I
have the following question.

They set `org.apache.hadoop.mapred.SequenceFileInputFormat` and
`org.apache.hadoop.mapred.SequenceFileOutputFormat` as the `inFormat` and
`outFormat` respectively, and it also has `org.apache.hadoop.io.Text` as
`outKey` and `outValue`. For me, this seems that this example accepts Text
files as sequence files. How can I create a SequenceFiles with
`org.apache.hadoop.io.Text`?


```
    WEBDATASCAN("webdataScan") {
    public void addJob(int numReducers, boolean mapoutputCompressed,
        boolean outputCompressed, Size size, JobControl gridmix) {
      final String prop = String.format("webdataScan.%sJobs.inputFiles",
size);
      final String indir = getInputDirsFor(prop,
size.defaultPath(VARCOMPSEQ));
      final String outdir = addTSSuffix("perf-out/webdata-scan-out-dir-" +
size);
      StringBuffer sb = new StringBuffer();
      sb.append("-keepmap 0.2 ");
      sb.append("-keepred 5 ");
      sb.append("-inFormat org.apache.hadoop.mapred.SequenceFileInputFormat
");
      sb.append("-outFormat
org.apache.hadoop.mapred.SequenceFileOutputFormat ");
      sb.append("-outKey org.apache.hadoop.io.Text ");
      sb.append("-outValue org.apache.hadoop.io.Text ");
      sb.append("-indir ").append(indir).append(" ");
      sb.append("-outdir ").append(outdir).append(" ");
      sb.append("-r ").append(numReducers);

      String[] args = sb.toString().split(" ");
      clearDir(outdir);
      try {
        JobConf jobconf = GenericMRLoadJobCreator.createJob(
            args, mapoutputCompressed, outputCompressed);
        jobconf.setJobName("GridmixWebdatascan." + size);
        Job job = new Job(jobconf);
        gridmix.addJob(job);
      } catch (Exception ex) {
        System.out.println(ex.getStackTrace());
      }
    }
    }
```

Mime
View raw message