cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pob <peterob...@gmail.com>
Subject Re: pig + hadoop
Date Wed, 20 Apr 2011 02:17:20 GMT
and one more thing...

2011-04-20 04:09:23,412 INFO org.apache.hadoop.mapred.TaskTracker:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201104200406_0001/attempt_201104200406_0001_m_000002_0/output/file.out
in any of the configured local directories


2011/4/20 pob <peterob333@gmail.com>

> Thats from jobtracker:
>
>
> 2011-04-20 03:36:39,519 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing rack-local task task_201104200331_0002_m_000000
> 2011-04-20 03:36:42,521 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> from attempt_201104200331_0002_m_000000_3: java.lang.NumberFormatException:
> null
>         at java.lang.Integer.parseInt(Integer.java:417)
>         at java.lang.Integer.parseInt(Integer.java:499)
>         at
> org.apache.cassandra.hadoop.ConfigHelper.getRpcPort(ConfigHelper.java:250)
>         at
> org.apache.cassandra.hadoop.pig.CassandraStorage.setConnectionInformation(Unknown
> Source)
>         at
> org.apache.cassandra.hadoop.pig.CassandraStorage.setLocation(Unknown Source)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.mergeSplitSpecificConf(PigInputFormat.java:133)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputFormat.java:111)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> and tasktracker
>
> 2011-04-20 03:33:10,942 INFO org.apache.hadoop.mapred.TaskTracker:  Using
> MemoryCalculatorPlugin :
> org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@3c1fc1a6
> 2011-04-20 03:33:10,945 WARN org.apache.hadoop.mapred.TaskTracker:
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> disabled.
> 2011-04-20 03:33:10,946 INFO org.apache.hadoop.mapred.IndexCache:
> IndexCache created with max memory = 10485760
> 2011-04-20 03:33:11,069 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201104200331_0001_m_000000_1 task's
> state:UNASSIGNED
> 2011-04-20 03:33:11,072 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to launch : attempt_201104200331_0001_m_000000_1
> 2011-04-20 03:33:11,072 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201104200331_0001_m_000000_1
> 2011-04-20 03:33:11,986 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201104200331_0001_m_-926908110
> 2011-04-20 03:33:11,986 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_201104200331_0001_m_-926908110 spawned.
> 2011-04-20 03:33:12,400 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201104200331_0001_m_-926908110 given task:
> attempt_201104200331_0001_m_000000_1
> 2011-04-20 03:33:12,895 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201104200331_0001_m_000000_1 0.0%
> 2011-04-20 03:33:12,918 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201104200331_0001_m_-926908110 exited. Number of tasks it ran: 0
> 2011-04-20 03:33:15,919 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_201104200331_0001_m_000000_1 done; removing files.
> 2011-04-20 03:33:15,920 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2011-04-20 03:33:38,090 INFO org.apache.hadoop.mapred.TaskTracker: Received
> 'KillJobAction' for job: job_201104200331_0001
> 2011-04-20 03:36:32,199 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_201104200331_0002_m_000000_2 task's
> state:UNASSIGNED
> 2011-04-20 03:36:32,199 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to launch : attempt_201104200331_0002_m_000000_2
> 2011-04-20 03:36:32,199 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
> attempt_201104200331_0002_m_000000_2
> 2011-04-20 03:36:32,813 INFO org.apache.hadoop.mapred.JvmManager: In
> JvmRunner constructed JVM ID: jvm_201104200331_0002_m_-134007035
> 2011-04-20 03:36:32,814 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_201104200331_0002_m_-134007035 spawned.
> 2011-04-20 03:36:33,214 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> ID: jvm_201104200331_0002_m_-134007035 given task:
> attempt_201104200331_0002_m_000000_2
> 2011-04-20 03:36:33,711 INFO org.apache.hadoop.mapred.TaskTracker:
> attempt_201104200331_0002_m_000000_2 0.0%
> 2011-04-20 03:36:33,731 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_201104200331_0002_m_-134007035 exited. Number of tasks it ran: 0
> 2011-04-20 03:36:36,732 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_201104200331_0002_m_000000_2 done; removing files.
> 2011-04-20 03:36:36,733 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2011-04-20 03:36:50,210 INFO org.apache.hadoop.mapred.TaskTracker: Received
> 'KillJobAction' for job: job_201104200331_0002
>
>
>
>
> 2011/4/20 pob <peterob333@gmail.com>
>
>> ad2. it works with -x local , so there cant be issue with
>> pig->DB(Cassandra).
>>
>> im using pig-0.8 from official site + hadoop-0.20.2 from offic. site.
>>
>>
>> thx
>>
>>
>> 2011/4/20 aaron morton <aaron@thelastpickle.com>
>>
>>> Am guessing but here goes. Looks like the cassandra RPC port is not set,
>>> did you follow these steps in contrib/pig/README.txt
>>>
>>> Finally, set the following as environment variables (uppercase,
>>> underscored), or as Hadoop configuration variables (lowercase, dotted):
>>> * PIG_RPC_PORT or cassandra.thrift.port : the port thrift is listening
>>> on
>>> * PIG_INITIAL_ADDRESS or cassandra.thrift.address : initial address to
>>> connect to
>>> * PIG_PARTITIONER or cassandra.partitioner.class : cluster partitioner
>>>
>>> Hope that helps.
>>> Aaron
>>>
>>>
>>> On 20 Apr 2011, at 11:28, pob wrote:
>>>
>>> Hello,
>>>
>>> I did cluster configuration by
>>> http://wiki.apache.org/cassandra/HadoopSupport. When I run
>>> pig example-script.pig
>>> -x local, everything is fine and i get correct results.
>>>
>>> Problem is occurring with -x mapreduce
>>>
>>> Im getting those errors :>
>>>
>>>
>>> 2011-04-20 01:24:21,791 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStats - ERROR:
>>> java.lang.NumberFormatException: null
>>> 2011-04-20 01:24:21,792 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>>> 2011-04-20 01:24:21,793 [main] INFO
>>>  org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>>
>>> Input(s):
>>> Failed to read data from "cassandra://Keyspace1/Standard1"
>>>
>>> Output(s):
>>> Failed to produce result in "
>>> hdfs://ip:54310/tmp/temp-1383865669/tmp-1895601791"
>>>
>>> Counters:
>>> Total records written : 0
>>> Total bytes written : 0
>>> Spillable Memory Manager spill count : 0
>>> Total bags proactively spilled: 0
>>> Total records proactively spilled: 0
>>>
>>> Job DAG:
>>> job_201104200056_0005   ->      null,
>>> null    ->      null,
>>> null
>>>
>>>
>>> 2011-04-20 01:24:21,793 [main] INFO
>>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>>> - Failed!
>>> 2011-04-20 01:24:21,803 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>>> ERROR 1066: Unable to open iterator for alias topnames. Backend error :
>>> java.lang.NumberFormatException: null
>>>
>>>
>>>
>>> ====
>>> thats from jobtasks web management - error  from task directly:
>>>
>>> java.lang.RuntimeException: java.lang.NumberFormatException: null
>>> at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:123)
>>>  at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:176)
>>> at
>>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> Caused by: java.lang.NumberFormatException: null
>>> at java.lang.Integer.parseInt(Integer.java:417)
>>>  at java.lang.Integer.parseInt(Integer.java:499)
>>> at
>>> org.apache.cassandra.hadoop.ConfigHelper.getRpcPort(ConfigHelper.java:233)
>>>  at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:105)
>>> ... 5 more
>>>
>>>
>>>
>>> Any suggestions where should be problem?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>

Mime
View raw message