hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "neovazjr@gmail.com" <neova...@gmail.com>
Subject native s3 file system - file split on S3 native file system and custom filesystem store
Date Wed, 25 Nov 2009 18:09:06 GMT
For security reasons I am required to conform and use a different S3 library
that I am provided to access S3 data. If I write an adapter against the
native file system store class to access S3 using my own library, do I still
get the same benefits that I would get for using the default file system
store , i.e. jets3t native file system store? My motivation here is to
exploit hadoop's capability to compute and generate file splits , so that I
can parallelize the work across different mappers for a single S3 file. I
believe this is quite different from the norm, as splits are generally used
in HDFS and supports larger files (where in this case the max is 5GB) and
that most approaches that I've heard requires the uploading of the data from
S3 to HDFS prior to processing - I am currently reading and writing straight
to S3, similar to EMR. What I have just pointed out may be completely
infeasible - I have looked through parts of the hadoop library but haven't
completely grasped how file split could interact with S3 input stream. There
are two questions here that may be totally unrelated, but thanks for
reading.

Clarence

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message