Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B649417A0C for ; Sun, 5 Apr 2015 13:57:08 +0000 (UTC) Received: (qmail 70279 invoked by uid 500); 5 Apr 2015 13:57:03 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 70171 invoked by uid 500); 5 Apr 2015 13:57:03 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 70161 invoked by uid 99); 5 Apr 2015 13:57:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Apr 2015 13:57:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of shahab.yunus@gmail.com designates 209.85.215.47 as permitted sender) Received: from [209.85.215.47] (HELO mail-la0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Apr 2015 13:56:37 +0000 Received: by layy10 with SMTP id y10so6524411lay.0 for ; Sun, 05 Apr 2015 06:55:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=F992HAf/4qmN1kySVHs7U743QAlkPqEBt9+HMuAeOMQ=; b=sl5GOXSVI/Ymnn3ndgBhR1GPWnrm7BzzpFfnOyXnw3989oxMIxV8shwwv8XNA3BvhL l+odCuqGZlvmSojPAB3Binr11BlaIinR2/huOriCAWupepF5odVrvoGO1pywgyVn2Px9 BzDWiulwbN6epkbL6Ghk5ZLHuaGbqq9HkuspYJPwU3C5azX67/haH/K/Vw15rAb5kjGr E2wI/QGt59oNYRcGUGVlBlX4vOQ3IXuuwHzHKH7/tMEHPHhdh+SXe3tsZrty21gxMjHu gkA3i8u7MWbkNds/0Q8ke1lnPtKfiQ4etivdFxyXE4LJtBrJKPOLt9BN1lds+nSblVVk Y0qQ== MIME-Version: 1.0 X-Received: by 10.152.206.75 with SMTP id lm11mr3787322lac.41.1428242105542; Sun, 05 Apr 2015 06:55:05 -0700 (PDT) Received: by 10.25.213.75 with HTTP; Sun, 5 Apr 2015 06:55:05 -0700 (PDT) In-Reply-To: <55213C93.4010109@gmail.com> References: <55213C93.4010109@gmail.com> Date: Sun, 5 Apr 2015 09:55:05 -0400 Message-ID: Subject: Re: compress data in hadoop From: Shahab Yunus To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a1133bdc65aaf530512fa8717 X-Virus-Checked: Checked by ClamAV on apache.org --001a1133bdc65aaf530512fa8717 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Your package seems different. Have you tried the following package and class? org.apache.hadoop.io.compress.BZip2Codec Regards, Shahab On Sun, Apr 5, 2015 at 9:45 AM, xeonmailinglist-gmail < xeonmailinglist@gmail.com> wrote: > Hi, > > I have run the command [1] to create compressed data from my Sequence > files that are in the /user/root/out1 dir, but I got the error [2]. How I > compress data in hadoop? > > [1] > > hadoop jar ./share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar -D mapredu= ce.job.reduces=3D30 -D mapred.output.compress=3Dtrue -D mapred.output.compr= ession.codec=3Dcom.hadoop.io.compression.BZip2Codec -D mapreduce.output.fil= eoutputformat.compress.type=3DBLOCK -mapper /bin/cat -reducer /bin/cat -i= nput /user/root/out1 -output /user/root/outcompressed > > [2] > > 15/04/05 09:41:32 INFO mapreduce.Job: Task Id : attempt_1428165800289_001= 7_r_000004_0, Status : FAILED > Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.i= o.compression.BZip2Codec was not found. > at org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass= (FileOutputFormat.java:100) > at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutp= utFormat.java:126) > at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.= (ReduceTask.java:484) > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:= 414) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfo= rmation.java:1628) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.lang.ClassNotFoundException: Class com.hadoop.io.compress= ion.BZip2Codec not found > at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.= java:1980) > at org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass= (FileOutputFormat.java:97) > ... 9 more > > =E2=80=8B > > -- > -- > > Thanks, > > --001a1133bdc65aaf530512fa8717 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Your package seems different.=C2=A0

Hav= e you tried the following package and class?
org.apache.hadoop.io.compress.BZip2Cod= ec=C2=A0

Regards,
Shahab

On Sun, Ap= r 5, 2015 at 9:45 AM, xeonmailinglist-gmail <xeonmailinglist@gma= il.com> wrote:
=20 =20 =20

Hi,

I have run the command [1] to create compressed data from my Sequence files that are in the /user/root/out1 dir, but I got the error [2]. How I compress data in hadoop?

[1]

hadoop jar ./=
share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar -D mapreduce.job.reduces=
=3D30 -D mapred.output.compress=3Dtrue -D mapred.output.compression.codec=
=3Dcom.hadoop.io.compression.BZip2Codec -D mapreduce.output.fileoutputforma=
t.compress.type=3DBLOCK  -mapper /bin/cat -reducer /bin/cat  -input /user/r=
oot/out1 -output /user/root/outcompressed

[2]

15/04/05 09:4=
1:32 INFO mapreduce.Job: Task Id : attempt_1428165800289_0017_r_000004_0, S=
tatus : FAILED
Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.io.=
compression.BZip2Codec was not found.
    at org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(F=
ileOutputFormat.java:100)
    at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutput=
Format.java:126)
    at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init=
>(ReduceTask.java:484)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:41=
4)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform=
ation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.io.compressio=
n.BZip2Codec not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.ja=
va:1980)
    at org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(F=
ileOutputFormat.java:97)
    ... 9 more
=E2=80=8B
--=20
--

Thanks,

--001a1133bdc65aaf530512fa8717--