commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew <li...@serff.net>
Subject Re: [fielupload] how much big size file can be handled?
Date Tue, 03 Oct 2006 22:42:47 GMT
Ok, so I'm playing with the streaming now and that seems to be working, 
but it seems like the file takes forever to write.  The upload finished 
very quickly, but then it takes another 5 minutes to write the file to 
disk...Here is what I'm doing:

        // Create a new file upload handler
        ServletFileUpload upload = new ServletFileUpload();
        
        // Parse the request
        FileItemIterator iter = upload.getItemIterator(request);
        while (iter.hasNext()) {
            FileItemStream item = iter.next();
            String name = item.getFieldName();
            InputStream stream = item.openStream();
            if (item.isFormField()) {
                System.out.println("Form field " + name + " with value "
                        + StreamUtil.asString(stream) + " detected.");
            } else {
                System.out.println("File field " + name + " with file name "
                        + item.getName() + " detected.");
                // Process the input stream
                File fNew = new File(dropbox, name);
                InputStream in = new BufferedInputStream(stream);
                OutputStream out = new BufferedOutputStream(new FileOutputStream(fNew));
                byte[] byteBuf = new byte[1024];
                int numBytes = 0;
                int totalBytes = 0;
                while (-1 != (numBytes = in.read(byteBuf))) {
                    out.write(byteBuf, 0, numBytes);
                    totalBytes += numBytes;
                }
            }
        }

Do you see anything I'm doing wrong?  Is there a better buffer size to 
use?  Should I not use a buffered stream to read the input stream?  Am I 
just defeating the purpose of what you added? 

Let me know if you see any where I can make an improvement.
Thanks!
Andrew
Paul J DeCoursey wrote:
> Are you reading as a Stream or are you buffering the file on disk?
>
> http://jakarta.apache.org/commons/fileupload/streaming.html
>
>
> Also, it looks like UnknownSizeException has been deprecated because 
> it doesn't exist in the latest release.
>
> p
>
>
>
>
> Andrew wrote:
>> We are running JBoss on RedHat AS3 on a linux box.  I understand your 
>> comment about the protocol.  I have other requirements that forced me 
>> to do it over https though.
>>
>> Any ideas on the UnknownSizeException though and how I can disable 
>> it?  I'm going to checkout the code today and look for myself, but if 
>> you already know, please enlighten me!
>>
>> Thanks!
>> Andrew
>> Paul J DeCoursey wrote:
>>> What is the OS and server you are running?  I know that IIS has a 
>>> limit, I think it's 2GB, for uploading files.  I'm of the opinion 
>>> that you shouldn't be using a web based fileupload for files that 
>>> large, you should use some more robust file transfer protocol, like 
>>> FTP, SFTP, SCP or USPS.
>>>
>>> Paul
>>>
>>> Andrew wrote:
>>>> I'm running into the same problem.  I send this email to the list 
>>>> on Friday, but never heard back from anyone...maybe it will get 
>>>> looked at here:
>>>>
>>>> I'm trying to upload files larger than 2GB.  It seems I can send a 
>>>> file larger, but FileUpload throws an exception when it is larger 
>>>> than 2GB.  It throws:
>>>> FileUploadBase$UnknownSizeException: the request was rejected 
>>>> because its size is unknown.
>>>> I'm assuming it can't get the content size because it overflowed an 
>>>> int.  Now, I don't care if it knows how big the file is, I just 
>>>> want it to take when ever file is sent to it.  Is there a way to 
>>>> disable this check?  I'm already setting setMaxSize(-1) (the 
>>>> default) but that doesn't seem to matter.
>>>> Let me know if you all have any ideas.
>>>> Thanks!
>>>> Andrew
>>>>
>>>> Arijit Mukherjee wrote:
>>>>> I've been able to upload files almost upto 2GB. I'm stuck at files >
>>>>> 2GB, and I suspect it's because of the native O/S and not a commons
>>>>> fileupload problem.
>>>>>
>>>>> Cheers
>>>>> Arijit
>>>>>  
>>>>>> -----Original Message-----
>>>>>> From: Leena Kulkarni [mailto:leenakulkarni2003@yahoo.com] Sent: 
>>>>>> 03 October 2006 11:53
>>>>>> To: commons-user@jakarta.apache.org
>>>>>> Subject: [fielupload] how much big size file can be handled?
>>>>>>
>>>>>> Hi Friends,
>>>>>>
>>>>>> I have few questions about the commons file upload.
>>>>>>
>>>>>> 1) Is commons file upload suitable for uploading
>>>>>> files>10MB in size?
>>>>>> What we have observed that commons file upload load a file in 
>>>>>> memory. So for files > 10MB the deply in response is too much.
>>>>>> Is this right? Is there something we are missing?
>>>>>> Is there any other good variation of commons file upload if we 
>>>>>> have to use it for uploads upto 30MB?
>>>>>>
>>>>>> 2)When we try uploading a file>40MB, we are getting Items list
as 0.
>>>>>> So there is nothing to iterate on and check if it is a file.
>>>>>>
>>>>>> Code is like the following:
>>>>>> // Create a new file upload handler
>>>>>> ServletFileUpload upload = new
>>>>>> ServletFileUpload(factory);
>>>>>>
>>>>>> // Parse the request
>>>>>> List /* FileItem */ items =
>>>>>> upload.parseRequest(request); // Process the uploaded items
>>>>>> Iterator iter = items.iterator();
>>>>>> while (iter.hasNext()) {
>>>>>>    FileItem item = (FileItem) iter.next();
>>>>>>
>>>>>>    if (item.isFormField()) {
>>>>>>        processFormField(item);
>>>>>>    } else {
>>>>>>        processUploadedFile(item);
>>>>>>    }
>>>>>> }
>>>>>>
>>>>>> We trapped this scenario and added code to show some customized 
>>>>>> error when Items are 0. But then it takes agaes to display the 
>>>>>> error message on screen when from the logs we can make out that 
>>>>>> the control did go till the point of displaying the error message

>>>>>> to screen.
>>>>>>
>>>>>> Do not know why this behaviour for large files [>40MB]. It can
be 
>>>>>> good to know the some threshold at which such error occurs and if

>>>>>> it is a known problem with commons fileupload.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> __________________________________________________
>>>>>> Do You Yahoo!?
>>>>>> Tired of spam?  Yahoo! Mail has the best spam protection around 
>>>>>> http://mail.yahoo.com
>>>>>> ---------------------------------------------------------------------

>>>>>>
>>>>>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>>>>>> For additional commands, e-mail: 
>>>>>> commons-user-help@jakarta.apache.org
>>>>>>
>>>>>>
>>>>>>     
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>>>>> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>>>>>
>>>>>   
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org


Mime
View raw message