Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Thu, 1 Aug 2013 15:05:53 +0000 (UTC)
From: "Stanislav Barton (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12658209.1374048957788.152019.1375369553535@arcas>
In-Reply-To: <JIRA.12658209.1374048957788@arcas>
References: <JIRA.12658209.1374048957788@arcas>
Subject: [jira] [Commented] (MAPREDUCE-5399) Large number of map tasks cause
 slow sort at reduce phase, invariant to amount of data to sort
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/MAPREDUCE-5399?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
13726500#comment-13726500 ]=20

Stanislav Barton commented on MAPREDUCE-5399:
---------------------------------------------

In the proposed patch, I have replaced the constructor that allowed passing=
 no Configuration object, then looked for all usages of the removed constru=
ctor and fixed the call by adding the Configuration object already known fr=
om the context. To my opinion, if the code compiles and the tests pass, it =
should be good to go, since the new constructor is not backwards compatible=
.
               =20
> Large number of map tasks cause slow sort at reduce phase, invariant to a=
mount of data to sort
> -------------------------------------------------------------------------=
---------------------
>
>                 Key: MAPREDUCE-5399
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5399
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1, mrv2
>    Affects Versions: 1.1.0, 2.0.2-alpha
>            Reporter: Stanislav Barton
>            Assignee: Stanislav Barton
>            Priority: Critical
>         Attachments: MAPREDUCE-5399.patch
>
>
> We are using hadoop-2.0.0+1357-1.cdh4.3.0.p0.21 with MRv1. After upgrade =
from 4.1.2 to 4.3.0, I have noticed some performance deterioration in our M=
R job in the Reduce phase. The MR job has usually 10 000 map tasks (10 000 =
files on input each about 100MB) and 6 000 reducers (one reducer per table =
region). I was trying to figure out what at which phase the slow down appea=
rs (firstly I suspected that the slow gathering of the 10000 map output fil=
es is the culprit) and found out that the problem is not reading the map ou=
tput (the shuffle) but the sort/merge phase that follows - the last and act=
ual reduce phase is fast. I have tried to up the io.sort.factor because I t=
hought the lots of small files are being merged on disk, but again upping t=
hat to 1000 didnt do any difference. I have then printed the stack trace an=
d found out that the problem is initialization of the org.apache.hadoop.map=
red.IFileInputStream namely the creation of the Configuration object which =
is not propagated along from earlier context, see the stack trace:
> Thread 13332: (state =3D IN_NATIVE)
>  - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=3D0 (C=
ompiled frame; information may be imprecise)
>  - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=3D2, li=
ne=3D228 (Compiled frame)
>  - java.io.File.exists() @bci=3D20, line=3D733 (Compiled frame)
>  - sun.misc.URLClassPath$FileLoader.getResource(java.lang.String, boolean=
) @bci=3D136, line=3D999 (Compiled frame)
>  - sun.misc.URLClassPath$FileLoader.findResource(java.lang.String, boolea=
n) @bci=3D3, line=3D966 (Compiled frame)
>  - sun.misc.URLClassPath.findResource(java.lang.String, boolean) @bci=3D1=
7, line=3D146 (Compiled frame)
>  - java.net.URLClassLoader$2.run() @bci=3D12, line=3D385 (Compiled frame)
>  - java.security.AccessController.doPrivileged(java.security.PrivilegedAc=
tion, java.security.AccessControlContext) @bci=3D0 (Compiled frame)
>  - java.net.URLClassLoader.findResource(java.lang.String) @bci=3D13, line=
=3D382 (Compiled frame)
>  - java.lang.ClassLoader.getResource(java.lang.String) @bci=3D30, line=3D=
1002 (Compiled frame)
>  - java.lang.ClassLoader.getResourceAsStream(java.lang.String) @bci=3D2, =
line=3D1192 (Compiled frame)
>  - javax.xml.parsers.SecuritySupport$4.run() @bci=3D26, line=3D96 (Compil=
ed frame)
>  - java.security.AccessController.doPrivileged(java.security.PrivilegedAc=
tion) @bci=3D0 (Compiled frame)
>  - javax.xml.parsers.SecuritySupport.getResourceAsStream(java.lang.ClassL=
oader, java.lang.String) @bci=3D10, line=3D89 (Compiled frame)
>  - javax.xml.parsers.FactoryFinder.findJarServiceProvider(java.lang.Strin=
g) @bci=3D38, line=3D250 (Interpreted frame)
>  - javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.Strin=
g) @bci=3D273, line=3D223 (Interpreted frame)
>  - javax.xml.parsers.DocumentBuilderFactory.newInstance() @bci=3D4, line=
=3D123 (Compiled frame)
>  - org.apache.hadoop.conf.Configuration.loadResource(java.util.Properties=
, org.apache.hadoop.conf.Configuration$Resource, boolean) @bci=3D16, line=
=3D1890 (Compiled frame)
>  - org.apache.hadoop.conf.Configuration.loadResources(java.util.Propertie=
s, java.util.ArrayList, boolean) @bci=3D49, line=3D1867 (Compiled frame)
>  - org.apache.hadoop.conf.Configuration.getProps() @bci=3D43, line=3D1785=
 (Compiled frame)
>  - org.apache.hadoop.conf.Configuration.get(java.lang.String) @bci=3D35, =
line=3D712 (Compiled frame)
>  - org.apache.hadoop.conf.Configuration.getTrimmed(java.lang.String) @bci=
=3D2, line=3D731 (Compiled frame)
>  - org.apache.hadoop.conf.Configuration.getBoolean(java.lang.String, bool=
ean) @bci=3D2, line=3D1047 (Interpreted frame)
>  - org.apache.hadoop.mapred.IFileInputStream.<init>(java.io.InputStream, =
long, org.apache.hadoop.conf.Configuration) @bci=3D111, line=3D93 (Interpre=
ted frame)
>  - org.apache.hadoop.mapred.IFile$Reader.<init>(org.apache.hadoop.conf.Co=
nfiguration, org.apache.hadoop.fs.FSDataInputStream, long, org.apache.hadoo=
p.io.compress.CompressionCodec, org.apache.hadoop.mapred.Counters$Counter) =
@bci=3D60, line=3D303 (Interpreted frame)
>  - org.apache.hadoop.mapred.IFile$InMemoryReader.<init>(org.apache.hadoop=
.mapred.RamManager, org.apache.hadoop.mapred.TaskAttemptID, byte[], int, in=
t) @bci=3D11, line=3D480 (Interpreted frame)
>  - org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createInMemorySegment=
s(java.util.List, long) @bci=3D133, line=3D2416 (Interpreted frame)
>  - org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createKVIterator() @b=
ci=3D669, line=3D2530 (Interpreted frame)
>  - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobCo=
nf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=3D513, line=3D425 =
(Interpreted frame)
>  - org.apache.hadoop.mapred.Child$4.run() @bci=3D29, line=3D268 (Interpre=
ted frame)
>  - java.security.AccessController.doPrivileged(java.security.PrivilegedEx=
ceptionAction, java.security.AccessControlContext) @bci=3D0 (Interpreted fr=
ame)
>  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.sec=
urity.PrivilegedExceptionAction) @bci=3D42, line=3D396 (Interpreted frame)
>  - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.Pri=
vilegedExceptionAction) @bci=3D14, line=3D1408 (Interpreted frame)
>  - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=3D776, li=
ne=3D262 (Interpreted frame)
> A blank configuration object is created at IFileInputStream. I have made =
a test and found out that this operation costs about 10-15ms depending on t=
he load on the system, because it goes to the local FS to load the properti=
es!!! This is to my opinion a bug since in the context the configuration (o=
f the job) is known and could be reused at that point. My problem (and ever=
y others who has big number of reducer and mapper tasks) is that for 10K ma=
p taks it does 10000 x 15 =3D 150 seconds just to find out that there is no=
thing to sort. The overhead should be normally zero.=20
> At this moment, the 10-15ms problem is amplified by 6 000 reducers so the=
 bottom line is that my reduce phase is at least 1.6 hours longer than it s=
hould be.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira