pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tobias Schlottke (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (PIG-3231) Problems with pig (TRUNK, 0.11) after upgrading to CDH4.2(yarn) using avro input
Date Thu, 20 Jun 2013 09:23:20 GMT

     [ https://issues.apache.org/jira/browse/PIG-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tobias Schlottke resolved PIG-3231.
-----------------------------------

    Resolution: Cannot Reproduce
    
> Problems with pig (TRUNK, 0.11) after upgrading to CDH4.2(yarn) using avro input
> --------------------------------------------------------------------------------
>
>                 Key: PIG-3231
>                 URL: https://issues.apache.org/jira/browse/PIG-3231
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.11
>         Environment: CDH4.2, yarn, avro
>            Reporter: Tobias Schlottke
>
> Hi there,
> we've got a strange issue after switching to a new cluster with cdh4.2 (from cdh3):
> Pig seems to create temporary avro files for its map reduce jobs, which it either deletes
or never creates.
> Pig fails with the "no error returned by hadoop"-message, but in nn-logs I found something
interesting.
> The actual exception from nn-log is:
> a
> {code}
> 2013-03-01 12:59:30,858 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020,
call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.1.28:37814: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/metrigo/event_logger/compact_log/2013/01/14/_temporary/1/_temporary/attempt_1362133122980_0017_m_000007_0/part-m-00007.avro
File does not exist. Holder DFSClient_attempt_1362133122980_0017_m_000007_0_1992466008_1 does
not have any open files.
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/metrigo/event_logger/compact_log/2013/01/14/_temporary/1/_temporary/attempt_1362133122980_0017_m_000007_0/part-m-00007.avro
File does not exist. Holder DFSClient_attempt_1362133122980_0017_m_000007_0_1992466008_1 does
not have any open files.
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2396)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2387)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2183)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:481)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
>         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1695)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1691)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1689)
> {code}
> Please note that we're analyzing a bunch of files (~200 files, we're using glob matchers),
some of them are small.
> We made it work once without the small files.
> *Update*
> I found the following exception deep in the logs that seems to make the job fail:
> {code}
> 2013-03-03 19:51:06,169 ERROR [main] org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:metrigo (auth:SIMPLE) cause:java.io.IOException: org.apache.avro.AvroRuntimeException:
java.io.IOException: Filesystem closed
> 2013-03-03 19:51:06,170 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running
child : java.io.IOException: org.apache.avro.AvroRuntimeException: java.io.IOException: Filesystem
closed
>         at org.apache.pig.piggybank.storage.avro.AvroStorage.getNext(AvroStorage.java:357)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
>         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:526)
>         at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> Caused by: org.apache.avro.AvroRuntimeException: java.io.IOException: Filesystem closed
>         at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:275)
>         at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:197)
>         at org.apache.pig.piggybank.storage.avro.PigAvroRecordReader.nextKeyValue(PigAvroRecordReader.java:180)
>         at org.apache.pig.piggybank.storage.avro.AvroStorage.getNext(AvroStorage.java:352)
>         ... 12 more
> Caused by: java.io.IOException: Filesystem closed
>         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:552)
>         at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:648)
>         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:706)
>         at java.io.DataInputStream.read(DataInputStream.java:149)
>         at org.apache.pig.piggybank.storage.avro.AvroStorageInputStream.read(AvroStorageInputStream.java:43)
>         at org.apache.avro.file.DataFileReader$SeekableInputStream.read(DataFileReader.java:210)
>         at org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw(BinaryDecoder.java:835)
>         at org.apache.avro.io.BinaryDecoder.isEnd(BinaryDecoder.java:440)
>         at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:261)
>         ... 15 more
> {code}
> Any Idea on how to find the reason for this?
> Best,
> Tobias

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message