ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Evans Ye (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-909) Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
Date Sat, 23 May 2015 13:40:17 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557340#comment-14557340
] 

Evans Ye commented on IGNITE-909:
---------------------------------

That would be awesome, thanks!

> Support HadoopExternalSplit in HadoopV2Context to get pig jobs to function
> --------------------------------------------------------------------------
>
>                 Key: IGNITE-909
>                 URL: https://issues.apache.org/jira/browse/IGNITE-909
>             Project: Ignite
>          Issue Type: Task
>          Components: hadoop
>    Affects Versions: sprint-4
>            Reporter: Evans Ye
>            Assignee: Evans Ye
>             Fix For: sprint-5
>
>         Attachments: IGNITE-909.patch
>
>
> Currently ignite does not support to run pig jobs on top of it.  A test script like the
following:
> {code}
> grunt> A = load '/passwd';
> grunt> B = foreach A generate $0;
> grunt> dump B;
> {code}
> can lead to a failure stack trace showing below:
> {code}
> [13:33:49,455][ERROR][Hadoop-task-16820899-f25d-40ee-acc7-7ec0b86f4c27_10-MAP-0-0-#165%null%][HadoopRunnableTask]
Task execution failed.
> class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException:
null
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:102)
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:50)
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
>         at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:107)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Context.getInputSplit(HadoopV2Context.java:93)
>         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getInputSplit(WrappedMapper.java:76)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.setup(PigGenericMapBase.java:202)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
>         at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:84)
>         ... 8 more
> {code}
> Implementing the TODO section in HadoopV2Context should get pig jobs to function.
> {code}
>     @Override public InputSplit getInputSplit() {
>         if (inputSplit == null) {
>             HadoopInputSplit split = ctx.taskInfo().inputSplit();
>             if (split == null)
>                 return null;
>             if (split instanceof HadoopFileBlock) {
>                 HadoopFileBlock fileBlock = (HadoopFileBlock)split;
>                 inputSplit = new FileSplit(new Path(fileBlock.file()), fileBlock.start(),
fileBlock.length(), null);
>             }
>             else if (split instanceof HadoopExternalSplit)
>                 throw new UnsupportedOperationException(); // TODO
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message