Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: message received from 54.164.171.186
 which is an MX secondary for user@hadoop.apache.org)
Message-ID: <553EABC3.7070109@gmail.com>
Date: Mon, 27 Apr 2015 22:36:03 +0100
From: xeonmailinglist-gmail <xeonmailinglist@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.6.0
MIME-Version: 1.0
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: how to merge sequence files?
Content-Type: multipart/alternative;
 boundary="------------040700080903000601010901"

This is a multi-part message in MIME format.
--------------040700080903000601010901
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

I have several directories that contain several files of 
|SequenceFileOutputFormat| with |org.apache.hadoop.io.Text| as key and 
value. I want to merge all these files into one.

I have looked to the join example [1], but it is not working. How I 
merge SequenceFiles?

|+ /root/Programs/hadoop/bin/hadoop jar /root/Programs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar join -inFormat org.apache.hadoop.mapred.SequenceFileInputFormat -outFormat org.apache.hadoop.mapred.SequenceFileOutputFormat -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text /wiki-output,/wiki-output2 /aggregate-output
15/04/27 17:34:47 INFO client.RMProxy: Connecting to ResourceManager at /172.16.100.11:8040
java.lang.ClassCastException: class org.apache.hadoop.mapred.SequenceFileOutputFormat
         at java.lang.Class.asSubclass(Class.java:3168)
         at org.apache.hadoop.examples.Join.run(Join.java:112)
         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
         at org.apache.hadoop.examples.Join.main(Join.java:173)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:622)
         at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
         at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
         at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:622)
         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
|

—

​

--------------040700080903000601010901
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=utf-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div markdown-here-wrapper-content-modified="true" style=""
      class="markdown-here-wrapper" data-md-url="Thunderbird">
      <p style="margin: 1.2em 0px ! important;">Hi,</p>
      <p style="margin: 1.2em 0px ! important;">I have several
        directories that contain several files of <code
          style="font-size: 0.85em; font-family:
          Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
          padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
          rgb(234, 234, 234); background-color: rgb(248, 248, 248);
          border-radius: 3px; display: inline;">SequenceFileOutputFormat</code>
        with <code style="font-size: 0.85em; font-family:
          Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em;
          padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid
          rgb(234, 234, 234); background-color: rgb(248, 248, 248);
          border-radius: 3px; display: inline;">org.apache.hadoop.io.Text</code>
        as key and value. I want to merge all these files into one.</p>
      <p style="margin: 1.2em 0px ! important;">I have looked to the
        join example [1], but it is not working. How I merge
        SequenceFiles?</p>
      <pre style="font-size: 0.85em; font-family: Consolas,Inconsolata,Courier,monospace;font-size: 1em; line-height: 1.2em;margin: 1.2em 0px;"><code style="font-size: 0.85em; font-family: Consolas,Inconsolata,Courier,monospace;margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px; display: inline;white-space: pre; overflow: auto; border-radius: 3px; border: 1px solid rgb(204, 204, 204); padding: 0.5em 0.7em; display: block ! important;">+ /root/Programs/hadoop/bin/hadoop jar /root/Programs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar join -inFormat org.apache.hadoop.mapred.SequenceFileInputFormat -outFormat org.apache.hadoop.mapred.SequenceFileOutputFormat -outKey org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text /wiki-output,/wiki-output2 /aggregate-output
15/04/27 17:34:47 INFO client.RMProxy: Connecting to ResourceManager at /172.16.100.11:8040
java.lang.ClassCastException: class org.apache.hadoop.mapred.SequenceFileOutputFormat
        at java.lang.Class.asSubclass(Class.java:3168)
        at org.apache.hadoop.examples.Join.run(Join.java:112)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.Join.main(Join.java:173)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
</code></pre>
      <p style="margin: 1.2em 0px ! important;"> —</p>
      <div
title="MDH:SSBoYXZlIHNldmVyYWwgZGlyZWN0b3JpZXMgdGhhdCBjb250YWluwqAgc2V2ZXJhbCBmaWxlcyBvZiBgU2VxdWVuY2VGaWxlT3V0cHV0Rm9ybWF0YCB3aXRoIGBvcmcuYXBhY2hlLmhhZG9vcC5pby5U
ZXh0YCBhcyBrZXkgYW5kIHZhbHVlLiBJIHdhbnQgdG8gbWVyZ2UgYWxsIHRoZXNlIGZpbGVzIGlu
dG8gb25lLjxicj48YnI+SSBoYXZlIGxvb2tlZCB0byB0aGUgam9pbiBleGFtcGxlIFsxXSwgYnV0
IGl0IGlzIG5vdCB3b3JraW5nLiBIb3cgSSBtZXJnZSBTZXF1ZW5jZUZpbGVzPzxicj48YnI+YGBg
PGJyPisgL3Jvb3QvUHJvZ3JhbXMvaGFkb29wL2Jpbi9oYWRvb3AgamFyIC9yb290L1Byb2dyYW1z
L2hhZG9vcC9zaGFyZS9oYWRvb3AvbWFwcmVkdWNlL2hhZG9vcC1tYXByZWR1Y2UtZXhhbXBsZXMt
Mi42LjAuamFyIGpvaW4gLWluRm9ybWF0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5TZXF1ZW5j
ZUZpbGVJbnB1dEZvcm1hdCAtb3V0Rm9ybWF0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5TZXF1
ZW5jZUZpbGVPdXRwdXRGb3JtYXQgLW91dEtleSBvcmcuYXBhY2hlLmhhZG9vcC5pby5UZXh0IC1v
dXRWYWx1ZSBvcmcuYXBhY2hlLmhhZG9vcC5pby5UZXh0IC93aWtpLW91dHB1dCwvd2lraS1vdXRw
dXQyIC9hZ2dyZWdhdGUtb3V0cHV0PGJyPjE1LzA0LzI3IDE3OjM0OjQ3IElORk8gY2xpZW50LlJN
UHJveHk6IENvbm5lY3RpbmcgdG8gUmVzb3VyY2VNYW5hZ2VyIGF0IC8xNzIuMTYuMTAwLjExOjgw
NDA8YnI+amF2YS5sYW5nLkNsYXNzQ2FzdEV4Y2VwdGlvbjogY2xhc3Mgb3JnLmFwYWNoZS5oYWRv
b3AubWFwcmVkLlNlcXVlbmNlRmlsZU91dHB1dEZvcm1hdDxicj7CoMKgwqDCoMKgwqDCoCBhdCBq
YXZhLmxhbmcuQ2xhc3MuYXNTdWJjbGFzcyhDbGFzcy5qYXZhOjMxNjgpPGJyPsKgwqDCoMKgwqDC
oMKgIGF0IG9yZy5hcGFjaGUuaGFkb29wLmV4YW1wbGVzLkpvaW4ucnVuKEpvaW4uamF2YToxMTIp
PGJyPsKgwqDCoMKgwqDCoMKgIGF0IG9yZy5hcGFjaGUuaGFkb29wLnV0aWwuVG9vbFJ1bm5lci5y
dW4oVG9vbFJ1bm5lci5qYXZhOjcwKTxicj7CoMKgwqDCoMKgwqDCoCBhdCBvcmcuYXBhY2hlLmhh
ZG9vcC5leGFtcGxlcy5Kb2luLm1haW4oSm9pbi5qYXZhOjE3Myk8YnI+wqDCoMKgwqDCoMKgwqAg
YXQgc3VuLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmludm9rZTAoTmF0aXZlIE1l
dGhvZCk8YnI+wqDCoMKgwqDCoMKgwqAgYXQgc3VuLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNz
b3JJbXBsLmludm9rZShOYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YTo1Nyk8YnI+wqDCoMKg
wqDCoMKgwqAgYXQgc3VuLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5pbnZv
a2UoRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhOjQzKTxicj7CoMKgwqDCoMKgwqDC
oCBhdCBqYXZhLmxhbmcucmVmbGVjdC5NZXRob2QuaW52b2tlKE1ldGhvZC5qYXZhOjYyMik8YnI+
wqDCoMKgwqDCoMKgwqAgYXQgb3JnLmFwYWNoZS5oYWRvb3AudXRpbC5Qcm9ncmFtRHJpdmVyJFBy
b2dyYW1EZXNjcmlwdGlvbi5pbnZva2UoUHJvZ3JhbURyaXZlci5qYXZhOjcxKTxicj7CoMKgwqDC
oMKgwqDCoCBhdCBvcmcuYXBhY2hlLmhhZG9vcC51dGlsLlByb2dyYW1Ecml2ZXIucnVuKFByb2dy
YW1Ecml2ZXIuamF2YToxNDQpPGJyPsKgwqDCoMKgwqDCoMKgIGF0IG9yZy5hcGFjaGUuaGFkb29w
LmV4YW1wbGVzLkV4YW1wbGVEcml2ZXIubWFpbihFeGFtcGxlRHJpdmVyLmphdmE6NzQpPGJyPsKg
wqDCoMKgwqDCoMKgIGF0IHN1bi5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5pbnZv
a2UwKE5hdGl2ZSBNZXRob2QpPGJyPsKgwqDCoMKgwqDCoMKgIGF0IHN1bi5yZWZsZWN0Lk5hdGl2
ZU1ldGhvZEFjY2Vzc29ySW1wbC5pbnZva2UoTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmE6
NTcpPGJyPsKgwqDCoMKgwqDCoMKgIGF0IHN1bi5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nl
c3NvckltcGwuaW52b2tlKERlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YTo0Myk8YnI+
wqDCoMKgwqDCoMKgwqAgYXQgamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kLmludm9rZShNZXRob2Qu
amF2YTo2MjIpPGJyPsKgwqDCoMKgwqDCoMKgIGF0IG9yZy5hcGFjaGUuaGFkb29wLnV0aWwuUnVu
SmFyLnJ1bihSdW5KYXIuamF2YToyMjEpPGJyPsKgwqDCoMKgwqDCoMKgIGF0IG9yZy5hcGFjaGUu
aGFkb29wLnV0aWwuUnVuSmFyLm1haW4oUnVuSmFyLmphdmE6MTM2KTxicj48YnI+YGBgPGJyPjxi
        cj48YnI+wqAtLQ=="
style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0;">​</div>
    </div>
  </body>
</html>

--------------040700080903000601010901--