hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akira Ajisaka (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6459) native task crashes when merging spilled file on ppc64
Date Tue, 16 May 2017 20:28:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013055#comment-16013055
] 

Akira Ajisaka commented on MAPREDUCE-6459:
------------------------------------------

The 2nd patch makes sense to me. Hi [~Tao Jie] and [~Ayappan], have you confirmed the 2nd
patch works as expected?

> native task crashes when merging spilled file on ppc64
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-6459
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6459
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>         Environment: Linux version 2.6.32-431.el6.ppc64
>            Reporter: Tao Jie
>            Assignee: Tao Jie
>         Attachments: MAPREDUCE-6459.001.patch, MAPREDUCE-6459.002.patch, ppc64_error.txt
>
>
> when running native task on ppc64´╝îmerging spilled files fails since we could not deserialize
local spill file correctly.
> Function readVLong in WritableUtils.h and Buffers.h, we try to compare a char with a
number and convert a char to int64_t. It does not work correctly on ppc64 since char definition
is different between ppc64 and x86 platform. On x86 platform char is defined as signed number
while on ppc64 char is unsigned. As a result, we write EOF marker [-1, -1] at the end of spill
partition, but deserialize chars as [255, 255].



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message