incubator-flex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avinash Narayanan <avinasha...@gmail.com>
Subject Re: SFTP file transfer for large files using flex
Date Thu, 19 Apr 2012 09:00:48 GMT
Hi Guys,

Sorry for the late reply but I wanted to have some concrete answers before
posting to the group again. I used the byte array & URLLoader to upload
files in separate post methods. I was also able to divide the file into
byte array chunks + boundary and then upload in multi-part.

Now the problem is if I use HTTP it works perfectly for over 3.5 Gb, when I
use HTTPS it gets stops abruptly somewhere around the 1.5 Gb mark for 10 Mb
chunks and 300 Mb mark for 50 Mb chunks. No clue why (help please)

I also observed that if you're uploading 10Mb chunks it takes a peak memory
of 117 Mb and avg mem of 30 Mb. For 50 Mb chunks it takes a peak memory of
300 Mb and average of 80 Mb. Why so much difference in peak memory? Any
ideas why?

Will be happy to share code in case it will be useful to someone.

Thanks
Avinash Y


2012/3/29 filippo dipisa <filippo@dipisa.net>

> I had many problems using the filereference in terms of perfomance.
> For my experience the best approach is to read the file, splitting it in
> chunks of 100kb (buffersize), send the chunks to the server.
> In this case Air will perform a file stream like,  to the server.
> Using the FileReference it loads the file first and then send it to the
> server using a lot of memory and putting your app in a hight risk of
> crashing.
>
> I added more code to my example to let you understand better
>
> Example:
>                fileStream = new FileStream();
>                fileStream.open( uploadedFile, FileMode.READ);
>                //fileStream.addEventListener(ProgressEvent.PROGRESS,
> onLoadProgress);
>                 var byteArraytoSend:ByteArray = new ByteArray();
>                var currentLength:int = Math.min(BUFFERSIZE,
> fileStream.bytesAvailable);
>                var bytesAvailable:uint = fileStream.bytesAvailable -
> byteArrayStartPosition;
>                var chunk:ByteArray = new ByteArray();
>                 while (fileStream.bytesAvailable > bytesAvailable) {
>                   chunk = new ByteArray();
>                   fileStream.readBytes(chunk, 0, currentLength);
>                   byteArraytoSend.writeBytes(chunk);
>                   fileUploadService.doUpload(byteArraytoSend);
>                 }
>
>               Then you have to do the checksum
>
>
> In this way you can stream chunks of 100kb and upload the files that you
> want.
> This solution is definetly more complex than using the FIleReference, so it
> is up to you.
> Hope it helps
> Filippo
>
> On 28 March 2012 22:05, Avinash Narayanan <avinashapr9@gmail.com> wrote:
>
> > Hi All,
> >
> > Looks like my little problem started quite the discussion!!.
> >
> > Ok, need to give a few updates. I managed to upload 1.07 Gb file (windows
> > image file actually) via fileReference using HTTP. Still no luck with
> HTTP.
> > And the data upload is NOT causing any increase in the usage of RAM!
> (very
> > strange).
> >
> > The reason why I need the air app to upload large files is that we're
> > creating some sort of sales portal using Adobe Air and the client wants
> to
> > be able to upload these really large files which are supposed to make
> sense
> > to the client regarding the engagement. I have no clue what this file is
> > for except that its upwards of 1Gb till 2Gb.
> >
> > Hope this helps!
> >
> > Thanks
> > Avinash Y
> >
> >
> > 2012/3/29 Jarosław Szczepankiewicz <jszczepankiewicz@gmail.com>
> >
> > > Supporting rest without tunneling through post is not possible due to
> > > limitations in flash player and due to plugin api limitations. Also
> full
> > > rest answers using full http response codes are not possible due to the
> > > same limitations of browser plugin api :(
> > > 28-03-2012 22:46 użytkownik "Hugo Matinho" <hugo.mig@gmail.com>
> napisał:
> > >
> > > > I know this might be the wrong place to ask and there is of course
> the
> > > > limitation on how the FP handles the HTTP transport but couldn't we
> do
> > > > something about flex supporting REST natively it's been several years
> > > > already and yes there's blazeDS and GraniteDS and others but most
> > > > JavaScript libraries support it natively most modern browsers
> implement
> > > > it and it wasn't made available on the player because browsers didn't
> > > > had it back then but come on it's 2012 some rest support would be
> nice
> > > >
> > > > Sent from my Windows Phone
> > > > From: Ariel Jakobovits
> > > > Sent: 28-03-2012 21:07
> > > > To: flex-dev@incubator.apache.org
> > > > Subject: Re: SFTP file transfer for large files using flex
> > > > > I might be off, but I don't think an ANE is, or would be, part of
> > Flex.
> > > > It is an extension for AIR.
> > > >
> > > > Flex has components intended only for AIR (such as FileSystemList),
> > > > and offering a component that exists as an ANE/AS combo that can be
> > > > added to a project to upload large files does not seem out of the
> > > > scope of Flex.
> > > >
> > > > Ariel Jakobovits
> > > > Email: arieljake@yahoo.com
> > > > Phone: 650-690-2213
> > > > Fax: 650-641-0031
> > > > Cell: 650-823-8699
> > > >
> > > >
> > > > ________________________________
> > > >  From: JP Bader <jp@zavteq.com>
> > > > To: flex-dev@incubator.apache.org; Ariel Jakobovits <
> > arieljake@yahoo.com
> > > >
> > > > Sent: Wednesday, March 28, 2012 12:04 PM
> > > > Subject: Re: SFTP file transfer for large files using flex
> > > >
> > > > I might be off, but I don't think an ANE is, or would be, part of
> > > > Flex.  It is an extension for AIR, which is still controlled by
> Adobe.
> > > > If we want to see about spawning new threads that could consume a
> > > > massive dump of data files, process, and pass those into a service
> > > > (FTP/SFTP/REST/Telnet/etc), that seems like a bit outside of scope of
> > > > the Flex project, and more in line w/ Tamarin.
> > > >
> > > > We are working on the framework, which currently has limitations to
> > > > the FP and AIR (which are Adobe).  Potentially once we have access to
> > > > Falcon(JS), there might be alternatives, but my question is, what 2GB
> > > > file are you trying to have FP/AIR upload?  There might be some
> tricks
> > > > to chunking out the flow of data using byte arrays, and clearing them
> > > > out as you go along with loading them to whatever repository/location
> > > > you need, but I can't think of too many desktop applications that
> will
> > > > not just freeze up when you load a 2GB (or any major file) into it.
> > > > For video, that's one thing, but beyond a certain size, and normally
> > > > the machine itself will slow down/become unresponsive.
> > > >
> > > > Most enterprises where I have done work, accessing and uploading
> > > > massive amounts of data would be FTP via terminals or clients, not
> > > > through a separate app (especially not AIR).  Maybe I'm missing the
> > > > reason behind wanting FP/AIR to handle that much data, but given the
> > > > limitations of FP/AIR for processing data, I am not sure how you
> would
> > > > approach building that into the framework.  I guess examples of
> > > > BigAssCanvas show that you could stitch together large files (larger
> > > > than FP would allow), but once you get to a certain size, it becomes
> > > > unresponsive.
> > > >
> > > > If there is a way to do it, I apologize for going off tangent, but we
> > > > need to focus on getting bugs fixed and improving the framework
> first,
> > > > and I'm not sure this is an improvement of the current framework so
> > > > much as a nice feature request.
> > > >
> > > > Regards,
> > > >
> > > > On Wed, Mar 28, 2012 at 1:14 PM, Ariel Jakobovits <
> arieljake@yahoo.com
> > >
> > > > wrote:
> > > > > would we consider developing ANEs to incorporate into Flex?
> > > > >
> > > > > Ariel Jakobovits
> > > > > Email: arieljake@yahoo.com
> > > > > Phone: 650-690-2213
> > > > > Fax: 650-641-0031
> > > > > Cell: 650-823-8699
> > > > >
> > > > >
> > > > > ________________________________
> > > > >  From: Marcus Fritze <marcus.fritze@googlemail.com>
> > > > > To: flex-dev@incubator.apache.org; Marcus Fritze <
> > > > marcus.fritze@googlemail.com>
> > > > > Sent: Wednesday, March 28, 2012 10:34 AM
> > > > > Subject: Re: SFTP file transfer for large files using flex
> > > > >
> > > > > Hi,
> > > > >
> > > > > as maybe all know Flash supports only a file size of 100MB.
> > > > >
> > > > > from the docs:
> > > > >
> > > > > FileReference -> upload()
> > > > >
> > > > >> Although Flash Player has no restriction on the size of files
you
> > can
> > > > upload or download, the player officially supports uploads or
> downloads
> > > of
> > > > up to 100 MB.
> > > > >
> > > > > I experienced that an upload / download of a larger file via Flash
> > > > (FileReference) is possible.
> > > > >
> > > > > I agree with Avinash! I think this is a good place to discuss this
> > > topic
> > > > (and not the stack overflow forum). Because this mailing list is the
> > > place
> > > > where we can improve the Flex SDK / Flash.
> > > > >
> > > > > Correct me if I am wrong, but an upload of large files (2 GB or
> more)
> > > is
> > > > currently not possible in Flash. And maybe we can check if it is
> > possible
> > > > to enable a larger file upload / download via Flex / Flash. This
> > feature
> > > is
> > > > very useful for Enterprise RIA's. I have already developed such an
> app
> > > and
> > > > have the same problem.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Marcus
> > > > >
> > > > > Am 28.03.2012 um 16:55 schrieb Avinash Narayanan:
> > > > >
> > > > >> Hi JP,
> > > > >>
> > > > >> Thanks for taking time to reply! Stackoverflow had questions
like
> > this
> > > > but
> > > > >> not satisfactory answers. This was my last ditch effort since
the
> > > other
> > > > >> option is to write my own SFTP plug (which clients wont pay me
to
> do
> > > > :|).
> > > > >>
> > > > >> As for the apps, this upload needs to be 'part' of a desktop
> > > application
> > > > >> written in adobe air so can't use any of these other tools :(
> > > > >>
> > > > >> Thanks
> > > > >> Avinash Y
> > > > >>
> > > > >>
> > > > >> On Wed, Mar 28, 2012 at 6:54 PM, JP Bader <jp@zavteq.com>
wrote:
> > > > >>
> > > > >>> Hi Avinash,
> > > > >>>
> > > > >>> My quick take on this is why you would want to recreate a
tool
> for
> > > > >>> this?  Aren't there plenty of free SFTP apps already?  Why
not
> take
> > > > >>> advantage of one of them? Tortoise, Cute, WinSCP, FileZilla,
etc.
> > > > >>>
> > > > >>> Also, questions like these might be better asked on other
forums
> > like
> > > > >>> stack overflow.  This mailing list is for questions and
> development
> > > > >>> regarding the next release of Flex.
> > > > >>>
> > > > >>> Regards,
> > > > >>>
> > > > >>> JP
> > > > >>>
> > > > >>> On Wed, Mar 28, 2012 at 3:16 AM, Avinash Narayanan
> > > > >>> <avinashapr9@gmail.com> wrote:
> > > > >>>> Hi All,
> > > > >>>>
> > > > >>>> I know this community is meant for future development
of Apache
> > Flex
> > > > but
> > > > >>>> I'm a little lost here. I need to upload ~2Gb file from
my local
> > > > machine
> > > > >>> to
> > > > >>>> a url via an air app using sftp. problem is we don't
have sftp
> lib
> > > > >>>> available.
> > > > >>>>
> > > > >>>> Also, I'm confused as to why in forums it says the flash
player
> > > needs
> > > > to
> > > > >>>> take the entire file into memory before loading it but
I don't
> see
> > > any
> > > > >>>> memory increase in my task manager (using the fileReference
> > class).
> > > > >>>>
> > > > >>>> Another thing I wanted to know is how best to split up
the file
> > into
> > > > >>> chunks
> > > > >>>> for multi-part loading? Any help in any of these areas
will be
> > > > supremely
> > > > >>>> helpful!
> > > > >>>>
> > > > >>>> I've already seen the following links
> > > > >>>>
> > > > >>>> http://www.actionscript.org/forums/showthread.php3?t=181895
> > > > >>>> http://forums.adobe.com/thread/631103
> > > > >>>> http://blog.ansuz.nl/index.php/2011/02/11/flex-ftp-abort/
> > > > >>>> http://maliboo.pl/projects/FlexFTP/
> > > > >>>> http://forums.adobe.com/thread/245294
> > > > >>>>
> > > > >>>
> > > >
> > >
> >
> http://blog.flexexamples.com/2007/09/21/uploading-files-in-flex-using-the-filereference-class/
> > > > >>>>
> > > > >>>> Thanks in Advance.
> > > > >>>>
> > > > >>>>
> > > > >>>> With Warm Regards,
> > > > >>>> Avinash
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> JP Bader
> > > > >>> Principal
> > > > >>> Zavteq, Inc.
> > > > >>> @lordB8r | jp@zavteq.com
> > > > >>> 608.692.2468
> > > > >>>
> > > >
> > > >
> > > >
> > > > --
> > > > JP Bader
> > > > Principal
> > > > Zavteq, Inc.
> > > > @lordB8r | jp@zavteq.com
> > > > 608.692.2468
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message