Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAMQQebApbYocR0-1LWHKKKcvfO9L9=WnHhmnz4+emOEijg7ZPQ@mail.gmail.com>
References: 
 <CAMQQebApbYocR0-1LWHKKKcvfO9L9=WnHhmnz4+emOEijg7ZPQ@mail.gmail.com>
From: Josh Wills <josh.wills@gmail.com>
Date: Mon, 23 Nov 2015 17:45:18 -0800
Message-ID: 
 <CANb5z2+6Q+OkBTYkev9WV36og_cNymmp36F9u-CbmidpmX9sgQ@mail.gmail.com>
Subject: Re: CrunchJobHooks.CompletionHook Inefficiency on S3NativeFileSystem
To: "user@crunch.apache.org" <user@crunch.apache.org>
Content-Type: multipart/alternative; boundary=089e013a04a89889e005253f7fda

--089e013a04a89889e005253f7fda
Content-Type: text/plain; charset=UTF-8

(I don't know the answer to this, but as I also now run Crunch on top of
S3, I'm interested in a solution.)

On Mon, Nov 23, 2015 at 5:22 PM, Jeff Quinn <jeff@nuna.com> wrote:

> Hey All,
>
> We have run in to a pretty frustrating inefficiency inside of
> the CrunchJobHooks.CompletionHook#handleMultiPaths.
>
> This method loops over all of the partial output files and moves them to
> their ultimate destination directories,
> calling org.apache.hadoop.fs.FileSystem#rename(org.apache.hadoop.fs.Path,
> org.apache.hadoop.fs.Path) on each partial output in a loop.
>
> This is no problem when the org.apache.hadoop.fs.FileSystem in question is
> HDFS where #rename is a cheap operation, but when an implementation such
> as S3NativeFileSystem is used it is extremely inefficient, as each
> iteration through the loop makes a single blocking S3 API call, and this
> loop can be extremely long when there are many thousands of partial output
> files.
>
> Has anyone dealt with this before / have any ideas to work around?
>
> Thanks!
>
> Jeff
>
>
>
> *DISCLAIMER:* The contents of this email, including any attachments, may
> contain information that is confidential, proprietary in nature, protected
> health information (PHI), or otherwise protected by law from disclosure,
> and is solely for the use of the intended recipient(s). If you are not the
> intended recipient, you are hereby notified that any use, disclosure or
> copying of this email, including any attachments, is unauthorized and
> strictly prohibited. If you have received this email in error, please
> notify the sender of this email. Please delete this and all copies of this
> email from your system. Any opinions either expressed or implied in this
> email and all attachments, are those of its author only, and do not
> necessarily reflect those of Nuna Health, Inc.

--089e013a04a89889e005253f7fda
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">(I don&#39;t know the answer to this, but as I also now ru=
n Crunch on top of S3, I&#39;m interested in a solution.)</div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Nov 23, 2015 at 5:2=
2 PM, Jeff Quinn <span dir=3D"ltr">&lt;<a href=3D"mailto:jeff@nuna.com" tar=
get=3D"_blank">jeff@nuna.com</a>&gt;</span> wrote:<br><blockquote class=3D"=
gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-=
left:1ex"><div dir=3D"ltr">Hey All,<div><br></div><div>We have run in to a =
pretty frustrating inefficiency inside of the=C2=A0CrunchJobHooks.Completio=
nHook#handleMultiPaths.=C2=A0</div><div><br></div><div>This method loops ov=
er all of the partial output files and moves them to their ultimate destina=
tion directories, calling=C2=A0org.apache.hadoop.fs.FileSystem#rename(org.a=
pache.hadoop.fs.Path, org.apache.hadoop.fs.Path) on each partial output in =
a loop.</div><div><br></div><div>This is no problem when the org.apache.had=
oop.fs.FileSystem in question is HDFS where #rename is a cheap operation, b=
ut when an implementation such as=C2=A0S3NativeFileSystem is used it is ext=
remely inefficient, as each iteration through the loop makes a single block=
ing S3 API call, and this loop can be extremely long when there are many th=
ousands of partial output files.</div><div><br></div><div>Has anyone dealt =
with this before / have any ideas to work around?=C2=A0</div><div><br></div=
><div>Thanks!</div><div><br></div><div>Jeff=C2=A0</div><div><br></div><div>=
<br></div></div>

<br>
<font size=3D"2" color=3D"#808080"><b style=3D"font-family:Calibri,sans-ser=
if;background-color:rgb(255,255,255)">DISCLAIMER:</b><span style=3D"font-fa=
mily:Calibri,sans-serif;background-color:rgb(255,255,255)">=C2=A0The conten=
ts of this email, including any attachments, may contain information that i=
s confidential, proprietary in nature, protected health information (PHI), =
or otherwise protected by law from disclosure, and is solely for the use of=
 the intended recipient(s). If you are not the intended recipient, you are =
hereby notified that any use, disclosure or copying of this email, includin=
g any attachments, is unauthorized and strictly prohibited. If you have rec=
eived this email in error, please notify the sender of this email. Please d=
elete this and all copies of this email from your system. Any opinions eith=
er expressed or implied in this email and all attachments, are those of its=
 author only, and do not necessarily reflect those of Nuna Health, Inc.</sp=
an></font></blockquote></div><br></div>

--089e013a04a89889e005253f7fda--