Mailing-List: contact user-help@ant.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Ant Users List" <user@ant.apache.org>
Received-SPF: neutral (nike.apache.org: 216.145.54.171 is neither permitted
 nor denied by domain of fastier@yahoo-inc.com)
From: Frank Astier <fastier@yahoo-inc.com>
To: "user@ant.apache.org" <user@ant.apache.org>
Date: Mon, 19 Dec 2011 10:25:40 -0800
Subject: Question about using Tar with Hadoop files
Thread-Topic: Question about using Tar with Hadoop files
Thread-Index: Acy+e5xV5xFrAXUv1kCVu0vAKY+SiA==
Message-ID: <CB14C1A4.C88C%fastier@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-Entourage/13.10.0.110428
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CB14C1A4C88Cfastieryahooinccom_"
MIME-Version: 1.0

--_000_CB14C1A4C88Cfastieryahooinccom_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Hi -

I=92m trying to use the Apache Tar package (1.8.2) for a Java program that =
tars large files in Hadoop. I am currently failing on a file that=92s 17 GB=
 long. Note that this code works without any problem for smaller files. I=
=92m tarring smaller HDFS files all day long without any problem. It fails =
only when I have to tar that 17 GB file. I have a hard time making sense of=
 the error message, after looking at source code for 3 days now... The exac=
t file size at the time of the error is: 17456999265 bytes. The exception I=
=92m seeing is:

12/19/11 5:54 PM [BDM.main] EXCEPTION request to write '65535' bytes exceed=
s size in header of '277130081' bytes
12/19/11 5:54 PM [BDM.main] EXCEPTION org.apache.tools.tar.TarOutputStream.=
write(TarOutputStream.java:238)
12/19/11 5:54 PM [BDM.main] EXCEPTION com.yahoo.ads.ngdstone.tpbdm.HDFSTar.=
archive(HDFSTar.java:149)

My code is:

           TarEntry entry =3D new TarEntry(p.getName());
           Path absolutePath =3D p.isAbsolute() ? p : new Path(baseDir, p);=
 // HDFS Path
           FileStatus fileStatus =3D fs.getFileStatus(absolutePath); // HDF=
S fileStatus
           entry.setNames(fileStatus.getOwner(), fileStatus.getGroup());
           entry.setUserName(user);
           entry.setGroupName(group);
            entry.setName(name);
            entry.setSize(fileStatus.getLen());
            entry.setMode(Integer.parseInt("0100" + permissions, 8));
            out.putNextEntry(entry); // out =3D TarOutputStream

            if (fileStatus.getLen() > 0) {

                InputStream in =3D fs.open(absolutePath); // large file in =
HDFS

                try {

                    ++nEntries;

                    int bytesRead =3D in.read(buf);

                    while (bytesRead >=3D 0) {
                        out.write(buf, 0, bytesRead);
                        bytesRead =3D in.read(buf);
                    }

                } finally {
                    in.close();
                }
            }

            out.closeEntry();

Any idea? Am I missing anything in the way I=92m setting up the TarOutputSt=
ream or TarEntry? Or does tar have implicit limits that are never going to =
work for multi-gigabytes size files?

Thanks!

Frank

--_000_CB14C1A4C88Cfastieryahooinccom_--