hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen, Haifeng" <haifeng.c...@intel.com>
Subject Some mappers are much slower than others in reading data from HDFS
Date Tue, 08 Jan 2013 03:07:22 GMT
Dear sir,
I encountered a strange problem that all the mappers on some nodes are much slower than the
mappers on other nodes as following some times (not always). I didn't see any reasons why
they should slow down in this pattern.

000013(MAP on node4): --------(8.115)
000014(MAP on node4): --------(8.570)
000011(MAP on node4): --------(8.5)
000016(MAP on node4): --------(8.344)
000010(MAP on node4): --------(8.585)
000015(MAP on node4): --------(8.179)
000017(MAP on node4): --------(8.445)
000012(MAP on node4): --------(8.312)
000018(MAP on node2):  ---(3.367)
000020(MAP on node2):  ---(3.335)
000019(MAP on node2):  ---(3.320)
000023(MAP on node2):  ---(3.91)
000022(MAP on node2):  ---(3.371)
000021(MAP on node2):  ---(3.458)
000004(MAP on node3):   -------------------(19.624)
000007(MAP on node3):   -------------------(19.92)
000005(MAP on node3):   --------------------(20.613)
000008(MAP on node3):   --------------------(20.316)
000003(MAP on node3):   --------------------(20.574)
000006(MAP on node3):   --------------------(20.654)
000002(MAP on node3):   -------------------(19.935)
000009(MAP on node3):   --------------------(20.489)
000025(MAP on node2):      --(2.877)
000026(MAP on node2):      ---(3.112)
000027(MAP on node2):      --(2.959)
000024(MAP on node2):      --(2.845)
000029(MAP on node2):       --(2.863)
000028(MAP on node2):       --(2.933)
000031(MAP on node2):          --(2.596)
000030(RED on node2):          -------------(13.378)

The testing is as following:
I have a 4 nodes cluster and all of them has the same hardware and software configurations.
One node acts as name node and yarn resource manager.  Other three nodes act as both data
node and yarn node manager.

The test input file is around 7GB file on the HDFS cluster and the replication number is 3.
(This means that each data node has a copy of every block of the file)

The mapper did nothing and didn't write out any records:

public static class KeyMapper
       extends Mapper<Object, Text, Text, Text>{
    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {


So this mapper logically is reading and iterating through its splits of data and then finish
the job.
I didn't see any factors in the above configurations that will cause the above phenomenon.
I turned on the debug log for each mapper task and it also showed that all the mapper's DFSClient
read data from its local data node.

Can any experts help give some hints for this? I attached the log and client code for analysis.


View raw message