incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pob <peterob...@gmail.com>
Subject Re: pig + hadoop
Date Wed, 20 Apr 2011 10:33:10 GMT
my false,

ignore last post.


2011/4/20 pob <peterob333@gmail.com>

> Hi,
>
> everything works fine with cassandra 0.7.5, but when I tried with 0.7.3
> another errors showed up, but task finished with success whats strange.....
>
>
> 2011-04-20 11:45:40,674 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> from attempt_201104201139_0004_m_000000_3: Error: java.lang.ClassNotF
> oundException: org.apache.thrift.TException
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:247)
>         at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:426)
>         at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:456)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java:153)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:105)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
>
> 2011-04-20 11:45:43,629 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> from attempt_201104201139_0004_m_000001_3: org.apache.pig.backend.exe
> cutionengine.ExecException: ERROR 2044: The type null cannot be collected
> as a Key type
>         at
> org.apache.pig.backend.hadoop.HDataType.getWritableComparableTypes(HDataType.java:143)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:105)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:238)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> 2011-04-20 11:42:49,498 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> from attempt_201104201139_0001_m_000000_1: Error: java.lang.ClassNotF
> oundException: org.apache.commons.lang.ArrayUtils
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>         at
> org.apache.cassandra.utils.ByteBufferUtil.<clinit>(ByteBufferUtil.java:75)
>         at
> org.apache.cassandra.hadoop.pig.CassandraStorage.<clinit>(Unknown Source)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:247)
>         at
> org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:426)
>         at
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:456)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java:153)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:105)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
>
>
>
> 2011/4/20 Jeremy Hanna <jeremy.hanna1234@gmail.com>
>
>> Just as an example:
>>
>>  <property>
>>    <name>cassandra.thrift.address</name>
>>    <value>10.12.34.56</value>
>>  </property>
>>  <property>
>>    <name>cassandra.thrift.port</name>
>>    <value>9160</value>
>>  </property>
>>  <property>
>>    <name>cassandra.partitioner.class</name>
>>    <value>org.apache.cassandra.dht.RandomPartitioner</value>
>>  </property>
>>
>>
>> On Apr 19, 2011, at 10:28 PM, Jeremy Hanna wrote:
>>
>> > oh yeah - that's what's going on.  what I do is on the machine that I
>> run the pig script from, I set the PIG_CONF variable to my HADOOP_HOME/conf
>> directory and in my mapred-site.xml file found there, I set the three
>> variables.
>> >
>> > I don't use environment variables when I run against a cluster.
>> >
>> > On Apr 19, 2011, at 9:54 PM, Jeffrey Wang wrote:
>> >
>> >> Did you set PIG_RPC_PORT in your hadoop-env.sh? I was seeing this error
>> for a while before I added that.
>> >>
>> >> -Jeffrey
>> >>
>> >> From: pob [mailto:peterob333@gmail.com]
>> >> Sent: Tuesday, April 19, 2011 6:42 PM
>> >> To: user@cassandra.apache.org
>> >> Subject: Re: pig + hadoop
>> >>
>> >> Hey Aaron,
>> >>
>> >> I read it, and all of 3 env variables was exported. The results are
>> same.
>> >>
>> >> Best,
>> >> P
>> >>
>> >> 2011/4/20 aaron morton <aaron@thelastpickle.com>
>> >> Am guessing but here goes. Looks like the cassandra RPC port is not
>> set, did you follow these steps in contrib/pig/README.txt
>> >>
>> >> Finally, set the following as environment variables (uppercase,
>> >> underscored), or as Hadoop configuration variables (lowercase, dotted):
>> >> * PIG_RPC_PORT or cassandra.thrift.port : the port thrift is listening
>> on
>> >> * PIG_INITIAL_ADDRESS or cassandra.thrift.address : initial address to
>> connect to
>> >> * PIG_PARTITIONER or cassandra.partitioner.class : cluster partitioner
>> >>
>> >> Hope that helps.
>> >> Aaron
>> >>
>> >>
>> >> On 20 Apr 2011, at 11:28, pob wrote:
>> >>
>> >>
>> >> Hello,
>> >>
>> >> I did cluster configuration by
>> http://wiki.apache.org/cassandra/HadoopSupport. When I run pig
>> example-script.pig
>> >> -x local, everything is fine and i get correct results.
>> >>
>> >> Problem is occurring with -x mapreduce
>> >>
>> >> Im getting those errors :>
>> >>
>> >>
>> >> 2011-04-20 01:24:21,791 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStats - ERROR:
>> java.lang.NumberFormatException: null
>> >> 2011-04-20 01:24:21,792 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>> >> 2011-04-20 01:24:21,793 [main] INFO
>>  org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>> >>
>> >> Input(s):
>> >> Failed to read data from "cassandra://Keyspace1/Standard1"
>> >>
>> >> Output(s):
>> >> Failed to produce result in
>> "hdfs://ip:54310/tmp/temp-1383865669/tmp-1895601791"
>> >>
>> >> Counters:
>> >> Total records written : 0
>> >> Total bytes written : 0
>> >> Spillable Memory Manager spill count : 0
>> >> Total bags proactively spilled: 0
>> >> Total records proactively spilled: 0
>> >>
>> >> Job DAG:
>> >> job_201104200056_0005   ->      null,
>> >> null    ->      null,
>> >> null
>> >>
>> >>
>> >> 2011-04-20 01:24:21,793 [main] INFO
>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Failed!
>> >> 2011-04-20 01:24:21,803 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>> ERROR 1066: Unable to open iterator for alias topnames. Backend error :
>> java.lang.NumberFormatException: null
>> >>
>> >>
>> >>
>> >> ====
>> >> thats from jobtasks web management - error  from task directly:
>> >>
>> >> java.lang.RuntimeException: java.lang.NumberFormatException: null
>> >> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:123)
>> >> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:176)
>> >> at
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
>> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >> Caused by: java.lang.NumberFormatException: null
>> >> at java.lang.Integer.parseInt(Integer.java:417)
>> >> at java.lang.Integer.parseInt(Integer.java:499)
>> >> at
>> org.apache.cassandra.hadoop.ConfigHelper.getRpcPort(ConfigHelper.java:233)
>> >> at
>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:105)
>> >> ... 5 more
>> >>
>> >>
>> >>
>> >> Any suggestions where should be problem?
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >
>>
>>
>

Mime
View raw message