hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "J. Rottinghuis" <jrottingh...@gmail.com>
Subject Re: test lzo problem in hadoop
Date Fri, 02 Aug 2013 02:23:48 GMT
In Hadoop 2.0 some of the classes have changed from an abstract class to an
interface.
You'll have to compile again. In addition, you need to use a version of
hadoop-lzo that is compatible with Hadoop 2.0 (Yarn).

See: https://github.com/twitter/hadoop-lzo/issues/56
and the announcement of a newer version 0.4.17 that solves this problem:
http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201306.mbox/%3CCADbBEnt1iiVbZFqQFupEUJ6VVdm5NitVvXqrfuzqugaOyMz+0g@mail.gmail.com%3E

Cheers,

Joep


On Thu, Aug 1, 2013 at 6:44 PM, ch huang <justlooks@gmail.com> wrote:

> i use yarn ,and i commented the following option and error is different
>
>
> vi /etc/hadoop/conf/mapred-site.xml
> <!--
>         <property>
>                 <name>mapred.job.tracker</name>
>                 <value>CH22:8088</value>
>         </property>
> -->
>
>
> #  hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.15.jar
> com.hadoop.compression.lzo.DistributedLzoIndexer /alex/ttt.lzo
> 13/08/02 09:25:51 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 13/08/02 09:25:51 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
> 13/08/02 09:25:52 INFO lzo.DistributedLzoIndexer: Adding LZO file
> /alex/ttt.lzo to indexing list (no index currently exists)
> 13/08/02 09:25:52 WARN conf.Configuration: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 13/08/02 09:25:52 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 13/08/02 09:25:52 WARN conf.Configuration: slave.host.name is deprecated.
> Instead, use dfs.datanode.hostname
> 13/08/02 09:25:52 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 13/08/02 09:25:52 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/08/02 09:25:53 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 13/08/02 09:25:53 INFO mapred.JobClient: Running job:
> job_local180628093_0001
> 13/08/02 09:25:53 INFO mapred.LocalJobRunner: OutputCommitter is
> com.hadoop.mapreduce.LzoIndexOutputFormat$1
> 13/08/02 09:25:53 INFO mapred.LocalJobRunner: Waiting for map tasks
> 13/08/02 09:25:53 INFO mapred.LocalJobRunner: Starting task:
> attempt_local180628093_0001_m_000000_0
> 13/08/02 09:25:53 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 13/08/02 09:25:53 INFO util.ProcessTree: setsid exited with exit code 0
> 13/08/02 09:25:53 INFO mapred.Task:  Using ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@338e18a3
> 13/08/02 09:25:53 INFO mapred.MapTask: Processing split:
> hdfs://CH22:9000/alex/ttt.lzo:0+306
> 13/08/02 09:25:53 INFO mapred.LocalJobRunner: Map task executor complete.
> 13/08/02 09:25:53 WARN mapred.LocalJobRunner: job_local180628093_0001
> java.lang.Exception: java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was
> expected
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
>         at
> com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:47)
>         at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:478)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:671)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:266)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> 13/08/02 09:25:54 INFO mapred.JobClient:  map 0% reduce 0%
> 13/08/02 09:25:54 INFO mapred.JobClient: Job complete:
> job_local180628093_0001
> 13/08/02 09:25:54 INFO mapred.JobClient: Counters: 0
>
> On Mon, Jul 22, 2013 at 6:07 PM, Sandeep Nemuri <nhsandeep6@gmail.com>wrote:
>
>> Try  this Ccommand
>> hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar
>> com.hadoop.compression.lzo.LzoIndexer /user/sample.txt.lzo
>>
>>
>> On Mon, Jul 22, 2013 at 2:08 PM, ch huang <justlooks@gmail.com> wrote:
>>
>>> anyone can help?
>>>
>>> # sudo -u hdfs hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.15.jar
>>> com.hadoop.compression.lzo.DistributedLzoIndexer
>>> /alex/test_lzo/sqoop-1.99.2-bin-hadoop200.tar.gz.lzo
>>> 13/07/22 16:33:50 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
>>> 13/07/22 16:33:50 INFO lzo.LzoCodec: Successfully loaded & initialized
>>> native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
>>> 13/07/22 16:33:50 INFO lzo.DistributedLzoIndexer: Adding LZO file
>>> /alex/test_lzo/sqoop-1.99.2-bin-hadoop200.tar.gz.lzo to indexing list (no
>>> index currently exists)
>>> 13/07/22 16:33:50 ERROR security.UserGroupInformation:
>>> PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException:
>>> Failed on local exception:
>>> com.google.protobuf.InvalidProtocolBufferException: Protocol message
>>> end-group tag did not match expected tag.; Host Details : local host is:
>>> "CH22/192.168.10.22"; destination host is: "CH22":8088;
>>> Exception in thread "main" java.io.IOException: Failed on local
>>> exception: com.google.protobuf.InvalidProtocolBufferException: Protocol
>>> message end-group tag did not match expected tag.; Host Details : local
>>> host is: "CH22/192.168.10.22"; destination host is: "CH22":8088;
>>>         at
>>> org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
>>>         at org.apache.hadoop.ipc.Client.call(Client.java:1229)
>>>         at
>>> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
>>>         at org.apache.hadoop.mapred.$Proxy10.getStagingAreaDir(Unknown
>>> Source)
>>>         at
>>> org.apache.hadoop.mapred.JobClient.getStagingAreaDir(JobClient.java:1324)
>>>         at
>>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:102)
>>>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
>>>         at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>         at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>         at
>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>         at
>>> com.hadoop.compression.lzo.DistributedLzoIndexer.run(DistributedLzoIndexer.java:111)
>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>         at
>>> com.hadoop.compression.lzo.DistributedLzoIndexer.main(DistributedLzoIndexer.java:115)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
>>> message end-group tag did not match expected tag.
>>>         at
>>> com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:73)
>>>         at
>>> com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
>>>         at
>>> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:213)
>>>         at
>>> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
>>>         at
>>> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
>>>         at
>>> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
>>>         at
>>> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
>>>         at
>>> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
>>>         at
>>> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
>>>         at
>>> org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985)
>>>         at
>>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:938)
>>>         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:836)
>>>
>>
>>
>>
>> --
>> --Regards
>>   Sandeep Nemuri
>>
>
>

Mime
View raw message