hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Prakash <ravi...@ymail.com>
Subject Re: Seek behavior difference between NativeS3FsInputStream and DFSInputStream
Date Mon, 03 Nov 2014 06:32:05 GMT

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Venkata!

Please feel free to open a JIRA and upload a patch. You might also try
the new s3a implementation (instead of s3n), but there's a chance that
the behavior will be the same.

Cheers
Ravi


On 10/31/14 03:23, Ravuri, Venkata Puneet wrote:
> Hi,
>
> I noticed a difference in behavior while seeking a given file present
in S3 using NativeS3FileSystem$NativeS3FsInputStream and the file
present in HDFS using DFSInputStream.
>
> If we seek to the end of the file incase of NativeS3FsInputStream, it
fails with exception "java.io.EOFException: Attempted to seek or read
past the end of the file".
> That is because a getObject request is issued on the S3 object with
range start as value of length of file.
>
> The end of file case is being handled safely in DFSInputStream.
> Shouldn't NativeS3FsInputStream also have similar checks?
>
> This issue is causing errors when Hive is trying to read S3 files.
> Please advise.
>
>
> Thanks and Regards,
> Puneet
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJUVyFiAAoJEGunL/HJl4XeUC0QALP6qRMf1I4CeM5hMsPfWg3n
4aZNC36XM6IrxMCNDTGwUOsYCl/O0D0q537AbZLPg0LqbeC/FYq+4zkU/z4X9hWH
BTd7UJJ9KtxD4hnFsBMxh0es8p8VfY6CrbPt13vyWukoJvMgBEHSg5QlIEYXE2SA
dvvXnjYyYCi88M5YpAsUSAteUHkCoY7iw3OjhNd3ipYyr2raCqdo1Tg6guXfRER/
hsPOjSvZYSFmUy10x94ZR5/a9/9Ar1Tm/VtNgLpGK1cG/ElK17lWLzsUKgFF5EnU
FWmXfqadvZMdKJupoqx50hebx0OwXQD+qBulaxSvEIu5LM2PmeZZIPLawoRmDj5R
4APvIMfUthEZDQc7Fe1IOHcQr9gmnVFFX9I4Ao9OZnRTdjkh2xg9wN5qePVCrmI2
69frsb61nGp6emqdUNb7Ov3GtEDZdRfl3vpfFWf3XUtExhxQZwIFAbspRTzAYnrS
D9HGbRLqklKc/+VnvUEhoh+x8BcTEz4vvnNimmKbnTLJLMusTUM5VtzWawo5xHvR
bdfqSnvXlZra6nU54r1D+e5K+Jo3xWzq0y3z+3RbQp0Y9Aqct0KH31pphypdh+z2
qSSHy5/1FxKyK5rSot0mHKZ+WTa9H1ZYM5+52CbccnoS8yraUZjFkXaWPKjw0gGx
9CXbhmMrKD8Ap2DqPMgz
=bqce
-----END PGP SIGNATURE-----


Mime
View raw message