hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Gollakota <pradeep...@gmail.com>
Subject Re: Custom FileInputFormat.class
Date Mon, 01 Dec 2014 17:12:08 GMT
Can you expand on your use case a little bit please? It may be that you're
duplicating functionality.

You can take a look at the CombineFileInputFormat for inspiration. If this
is indeed taking a long time, one cheap to implement thing you can do is to
parallelize the calls to get block locations.

Another question to ask yourself is whether it is worth it to optimize this
portion. In many use cases, (certainly mine), the bottleneck is the running
job itself. So the launch overhead is comparatively minimal.

Hope this helps.

On Mon Dec 01 2014 at 8:38:30 AM 胡斐 <hufei68@gmail.com> wrote:

> Hi,
> I want to custom FileInputFormat.class. In order to determine which host
> the specific part of a file belongs to, I need to open the file in HDFS and
> read some information. It will take me nearly 500ms to open a file and get
> the information I need. But now I have thousands of files to deal with, so
> it would be a long time if I deal with all of them as the above.
> Is there any better solution to reduce the time when the number of files
> is large?
> Thanks in advance!
> Fei

View raw message