ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Evans Ye (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-909) Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
Date Fri, 15 May 2015 18:11:00 GMT
Evans Ye created IGNITE-909:
-------------------------------

             Summary: Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
                 Key: IGNITE-909
                 URL: https://issues.apache.org/jira/browse/IGNITE-909
             Project: Ignite
          Issue Type: Task
          Components: hadoop
    Affects Versions: sprint-4
            Reporter: Evans Ye


Currently ignite does not support to run pig jobs on top of it.  A test script like the following:
{code}
grunt> A = load '/passwd';
grunt> B = foreach A generate $0;
grunt> dump B;
{code}

can lead to a failure stack trace showing below:
{code}
[13:33:49,455][ERROR][Hadoop-task-16820899-f25d-40ee-acc7-7ec0b86f4c27_10-MAP-0-0-#165%null%][HadoopRunnableTask]
Task execution failed.
class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException:
null
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:102)
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:50)
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:107)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Context.getInputSplit(HadoopV2Context.java:93)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getInputSplit(WrappedMapper.java:76)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:202)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:84)
        ... 8 more
{code}

Implementing the TODO section in HadoopV2Context should get pig jobs to function.

{code}
    @Override public InputSplit getInputSplit() {
        if (inputSplit == null) {
            HadoopInputSplit split = ctx.taskInfo().inputSplit();

            if (split == null)
                return null;

            if (split instanceof HadoopFileBlock) {
                HadoopFileBlock fileBlock = (HadoopFileBlock)split;

                inputSplit = new FileSplit(new Path(fileBlock.file()), fileBlock.start(),
fileBlock.length(), null);
            }
            else if (split instanceof HadoopExternalSplit)
                throw new UnsupportedOperationException(); // TODO
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message