Mailing-List: contact dev-help@ignite.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.incubator.apache.org
Date: Fri, 15 May 2015 18:11:00 +0000 (UTC)
From: "Evans Ye (JIRA)" <jira@apache.org>
To: dev@ignite.incubator.apache.org
Message-ID: <JIRA.12830252.1431713451000.129277.1431713460474@Atlassian.JIRA>
In-Reply-To: <JIRA.12830252.1431713451000@Atlassian.JIRA>
References: <JIRA.12830252.1431713451000@Atlassian.JIRA>
 <JIRA.12830252.1431713451667@arcas>
Subject: [jira] [Created] (IGNITE-909) Support HadoopExternalSplit in
 HadoopV2Context to get pig jobs to function
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Evans Ye created IGNITE-909:
-------------------------------

             Summary: Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
                 Key: IGNITE-909
                 URL: https://issues.apache.org/jira/browse/IGNITE-909
             Project: Ignite
          Issue Type: Task
          Components: hadoop
    Affects Versions: sprint-4
            Reporter: Evans Ye


Currently ignite does not support to run pig jobs on top of it.  A test script like the following:
{code}
grunt> A = load '/passwd';
grunt> B = foreach A generate $0;
grunt> dump B;
{code}

can lead to a failure stack trace showing below:
{code}
[13:33:49,455][ERROR][Hadoop-task-16820899-f25d-40ee-acc7-7ec0b86f4c27_10-MAP-0-0-#165%null%][HadoopRunnableTask] Task execution failed.
class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException: null
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:102)
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:50)
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
        at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:107)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Context.getInputSplit(HadoopV2Context.java:93)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getInputSplit(WrappedMapper.java:76)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:202)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
        at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:84)
        ... 8 more
{code}

Implementing the TODO section in HadoopV2Context should get pig jobs to function.

{code}
    @Override public InputSplit getInputSplit() {
        if (inputSplit == null) {
            HadoopInputSplit split = ctx.taskInfo().inputSplit();

            if (split == null)
                return null;

            if (split instanceof HadoopFileBlock) {
                HadoopFileBlock fileBlock = (HadoopFileBlock)split;

                inputSplit = new FileSplit(new Path(fileBlock.file()), fileBlock.start(), fileBlock.length(), null);
            }
            else if (split instanceof HadoopExternalSplit)
                throw new UnsupportedOperationException(); // TODO
{code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)