Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: xeon Mailinglist <xeonmailinglist@gmail.com>
Date: Wed, 6 Jan 2016 15:38:22 +0000
Message-ID: 
 <CACgT_v+_5ui74dVudtLWfwSY_9qV5NguCuNaYxZj+_jDMms9tw@mail.gmail.com>
Subject: How can I create a SequenceFiles with `org.apache.hadoop.io.Text`?
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=e89a8f234413b0d0b10528ac2610

--e89a8f234413b0d0b10528ac2610
Content-Type: text/plain; charset=UTF-8

Hi,

This is a snippet of a Gridmix code available in Hadop MapReduce V1, but I
have the following question.

They set `org.apache.hadoop.mapred.SequenceFileInputFormat` and
`org.apache.hadoop.mapred.SequenceFileOutputFormat` as the `inFormat` and
`outFormat` respectively, and it also has `org.apache.hadoop.io.Text` as
`outKey` and `outValue`. For me, this seems that this example accepts Text
files as sequence files. How can I create a SequenceFiles with
`org.apache.hadoop.io.Text`?


```
    WEBDATASCAN("webdataScan") {
    public void addJob(int numReducers, boolean mapoutputCompressed,
        boolean outputCompressed, Size size, JobControl gridmix) {
      final String prop = String.format("webdataScan.%sJobs.inputFiles",
size);
      final String indir = getInputDirsFor(prop,
size.defaultPath(VARCOMPSEQ));
      final String outdir = addTSSuffix("perf-out/webdata-scan-out-dir-" +
size);
      StringBuffer sb = new StringBuffer();
      sb.append("-keepmap 0.2 ");
      sb.append("-keepred 5 ");
      sb.append("-inFormat org.apache.hadoop.mapred.SequenceFileInputFormat
");
      sb.append("-outFormat
org.apache.hadoop.mapred.SequenceFileOutputFormat ");
      sb.append("-outKey org.apache.hadoop.io.Text ");
      sb.append("-outValue org.apache.hadoop.io.Text ");
      sb.append("-indir ").append(indir).append(" ");
      sb.append("-outdir ").append(outdir).append(" ");
      sb.append("-r ").append(numReducers);

      String[] args = sb.toString().split(" ");
      clearDir(outdir);
      try {
        JobConf jobconf = GenericMRLoadJobCreator.createJob(
            args, mapoutputCompressed, outputCompressed);
        jobconf.setJobName("GridmixWebdatascan." + size);
        Job job = new Job(jobconf);
        gridmix.addJob(job);
      } catch (Exception ex) {
        System.out.println(ex.getStackTrace());
      }
    }
    }
```

--e89a8f234413b0d0b10528ac2610
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi,<br><br></div>This is a snippet of a Gridmix =
code available in Hadop MapReduce V1, but I have the following question.<br=
><br></div>They set `org.apache.hadoop.mapred.SequenceFileInputFormat` and =
`org.apache.hadoop.mapred.SequenceFileOutputFormat` as the `inFormat` and `=
outFormat` respectively, and it also has `org.apache.hadoop.io.Text` as `ou=
tKey` and `outValue`. For me, this seems that this example accepts Text fil=
es as sequence files. How can I create a SequenceFiles with `org.apache.had=
oop.io.Text`?<div><div><div><div><br><br>```<br>=C2=A0=C2=A0=C2=A0 WEBDATAS=
CAN(&quot;webdataScan&quot;) {<br>=C2=A0=C2=A0=C2=A0 public void addJob(int=
 numReducers, boolean mapoutputCompressed,<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 boolean outputCompressed, Size size, JobControl gridmix) {<=
br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 final String prop =3D String.format(&quot=
;webdataScan.%sJobs.inputFiles&quot;, size);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 final String indir =3D getInputDirsFor(prop, size.defaultPath(VARCOMPSE=
Q));<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 final String outdir =3D addTSSuffix(=
&quot;perf-out/webdata-scan-out-dir-&quot; + size);<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 StringBuffer sb =3D new StringBuffer();<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 sb.append(&quot;-keepmap 0.2 &quot;);<br>=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 sb.append(&quot;-keepred 5 &quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 sb.append(&quot;-inFormat org.apache.hadoop.mapred.SequenceFileInput=
Format &quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sb.append(&quot;-outForma=
t org.apache.hadoop.mapred.SequenceFileOutputFormat &quot;);<br>=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 sb.append(&quot;-outKey org.apache.hadoop.io.Text &qu=
ot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sb.append(&quot;-outValue org.apach=
e.hadoop.io.Text &quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sb.append(&quot=
;-indir &quot;).append(indir).append(&quot; &quot;);<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 sb.append(&quot;-outdir &quot;).append(outdir).append(&quot; &=
quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sb.append(&quot;-r &quot;).append=
(numReducers);<br><br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 String[] args =3D sb.t=
oString().split(&quot; &quot;);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 clearDir(=
outdir);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try {<br>=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 JobConf jobconf =3D GenericMRLoadJobCreator.createJob=
(<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 arg=
s, mapoutputCompressed, outputCompressed);<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 jobconf.setJobName(&quot;GridmixWebdatascan.&quot; + size);=
<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Job job =3D new Job(jobconf)=
;<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gridmix.addJob(job);<br>=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 } catch (Exception ex) {<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 System.out.println(ex.getStackTrace());<br>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 }<br>=C2=A0=C2=A0=C2=A0 }<br>=C2=A0=C2=A0=C2=A0 }<=
br>```<br></div></div></div></div></div>

--e89a8f234413b0d0b10528ac2610--