commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jessy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COMPRESS-194) Unable to create a TAR file that contains an entry which is >8 GB in size.
Date Mon, 09 Jul 2012 09:56:34 GMT

    [ https://issues.apache.org/jira/browse/COMPRESS-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409297#comment-13409297
] 

jessy commented on COMPRESS-194:
--------------------------------

Stefan,

I tried to setBigNumberMode to tarOutputStream.BIGNUMBER_STAR [ =1 ]. 
However, i found that tarring completes without any exceptions, and shows the tar archive
size as 9GB(which is my input of  9GB content file).
But ,when i tried to unzip( windows ),  I was unsuccessful in opening this tar. Got the error
dialog "The archive is corrupt” 
When I tried to open the tar thru windows right click menu , I found my content  file size
is 0 bytes. I have taken care to close all streams . 
Could you kindly look at my below code snippet and suggest what might be the rootcause for
this. 


import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.apache.log4j.Logger;

/**
* The Class TarArchiver.
*/
public class TarArchiver {
                
                /** The Constant TWO_KILO_BYTES. */
                private static final int TWO_KILO_BYTES = 2048;
                
                /** The tar output stream. */
                private TarArchiveOutputStream tarOutputStream;
                
                /** The file output stream. */
                private FileOutputStream fileOutputStream;
                
                /** The Constant EOF. */
                private static final int EOF = -1;
                
                /** The Constant LOGGER. */
                private static final Logger LOGGER = Logger.getLogger(TarArchiver.class);
                                             
                /**
                * Instantiates a new   tar archiver.
                *
                * @param directory the directory
                * @param tarName the tar name
                * @throws IOException Signals that an I/O exception has occurred.
                * @throws CustomException the custom exception
                */
                public void archive(String directory, String tarName) throws IOException,
CustomException {
                                tarOutputStream = new TarArchiveOutputStream(new FileOutputStream(tarName));
                                tarOutputStream.setBigNumberMode(tarOutputStream.BIGNUMBER_STAR);
                                String path = "";
                                try {
                                                archiveDir(directory, path);
                                }
                                finally {
                                                tarOutputStream.close();
                                }
                }
                
                /**
                * Archive dir.
                *
                * @param directory the directory
                * @param path the path
                * @throws CustomException the custom exception
                */
                private void archiveDir(String directory, String path) throws CustomException
{

                                byte[] dataBuffer = new byte[TWO_KILO_BYTES];
                                fileOutputStream = null;
                                File tarDir = null;
                                try {
                                                tarDir = new File(directory);
                                                String files[] = tarDir.list();

                                                for (int count = 0; count < files.length;
count++) {
                                                                File f = new File(tarDir,
files[count]);
                                                                createTarArchive(dataBuffer,
path, f);
                                                }
                                                tarOutputStream.flush();
                                }
                                catch (FileNotFoundException e) {
                                                LOGGER.error("FileNotFoundException to archive
: " + tarDir, e);
                                                throw new CustomException(e.getMessage(),
e);
                                }
                                catch (IOException e) {
                                                LOGGER.error("IOException during archiving
: " + tarDir, e);
                                                throw new CustomException(e.getMessage(),
e);
                                }
                }
                
                /**
                * Creates the tar archive.
                *
                * @param dataBuffer the data buffer
                * @param path the path
                * @param file the file
                * @throws IOException Signals that an I/O exception has occurred.
                */
                private void createTarArchive(byte[] dataBuffer, String path, File file) throws
IOException {
                                {
                                                FileInputStream fileInputStream = new FileInputStream(file);
                                                try {
                                                                int bytesIn = 0;
                                                                String string = path + file.getName();
                                                                TarArchiveEntry archiveEntry
= new TarArchiveEntry(file, string);
                                                                tarOutputStream.putArchiveEntry(archiveEntry);
                                                                bytesIn = fileInputStream.read(dataBuffer);
                                                                while (bytesIn != EOF) {
                                                                                tarOutputStream.write(dataBuffer,
0, bytesIn);
                                                                                bytesIn =
fileInputStream.read(dataBuffer);
                                                                }
                                                                tarOutputStream.closeArchiveEntry();
                                                }
                                                catch (RuntimeException ex) {
                                                                closeInputStream(fileInputStream);
                                                                closeOutputStream();
                                                }
                                                finally {
                                                                closeInputStream(fileInputStream);
                                                }
                                }
                }


                private void closeOutputStream() {
                                closeOutputStream(fileOutputStream);
                                closeOutputStream(tarOutputStream);
                }


                private void closeOutputStream(OutputStream os) {
                                try {
                                                if (os != null) {
                                                                os.close();
                                                }
                                }
                                catch (IOException e) {
                                                LOGGER.error("Failed to close Output Stream.",
e);
                                }
                }


                private void closeInputStream(InputStream is) {
                                try {
                                                if (is != null) {
                                                                is.close();
                                                }
                                }
                                catch (IOException e) {
                                                LOGGER.error("Failed to close Input Stream.",
e);
                                }
                }


Jes

                
> Unable to create a TAR file that contains an entry which is >8 GB in size. 
> ---------------------------------------------------------------------------
>
>                 Key: COMPRESS-194
>                 URL: https://issues.apache.org/jira/browse/COMPRESS-194
>             Project: Commons Compress
>          Issue Type: Bug
>          Components: Archivers
>    Affects Versions: 1.4.1
>         Environment: I am using win xp and Red HatLinux 5.1v, but this should be platform
independent.
>            Reporter: jessy
>            Priority: Critical
>             Fix For: 1.4
>
>
> The common-compress-1.4.1 api says it supports unlimited file sizes for tar and untar
operations. [http://ant.apache.org/antlibs/compress/]
> Only Untar operations on a file > 8GB is fixed,
> But creating a tar on a file >8GB is not working.
> When I try to do  " tarOutputStream.putArchiveEntry(archiveEntry); " 
> i get the below exception. 
> java.lang.RuntimeException: entry size '9633985364' is too big ( > 8589934591 )
> 	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumber(TarArchiveOutputStream.java:572)
> 	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumbers(TarArchiveOutputStream.java:557)
> 	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:297)
> Looking for a fix .. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message