incubator-flex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avinash Narayanan <avinasha...@gmail.com>
Subject Re: SFTP file transfer for large files using flex
Date Thu, 19 Apr 2012 09:11:13 GMT
Hi Disa,

The problem is since we're using HTTPS, we get 4Kb of authentication as a
separate response + 800bytes headers & 900 bytes response in the POST.
we're now looking at 15 Gb uploads. Seems to be too many transactions?

Thanks
Avinash Y


2012/4/19 filippo dipisa <filippo@dipisa.net>

> usually my buffer is 100K for chunk
> hope it helps
> Filippo
>
> On 19 April 2012 10:00, Avinash Narayanan <avinashapr9@gmail.com> wrote:
>
> > Hi Guys,
> >
> > Sorry for the late reply but I wanted to have some concrete answers
> before
> > posting to the group again. I used the byte array & URLLoader to upload
> > files in separate post methods. I was also able to divide the file into
> > byte array chunks + boundary and then upload in multi-part.
> >
> > Now the problem is if I use HTTP it works perfectly for over 3.5 Gb,
> when I
> > use HTTPS it gets stops abruptly somewhere around the 1.5 Gb mark for 10
> Mb
> > chunks and 300 Mb mark for 50 Mb chunks. No clue why (help please)
> >
> > I also observed that if you're uploading 10Mb chunks it takes a peak
> memory
> > of 117 Mb and avg mem of 30 Mb. For 50 Mb chunks it takes a peak memory
> of
> > 300 Mb and average of 80 Mb. Why so much difference in peak memory? Any
> > ideas why?
> >
> > Will be happy to share code in case it will be useful to someone.
> >
> > Thanks
> > Avinash Y
> >
> >
> > 2012/3/29 filippo dipisa <filippo@dipisa.net>
> >
> > > I had many problems using the filereference in terms of perfomance.
> > > For my experience the best approach is to read the file, splitting it
> in
> > > chunks of 100kb (buffersize), send the chunks to the server.
> > > In this case Air will perform a file stream like,  to the server.
> > > Using the FileReference it loads the file first and then send it to the
> > > server using a lot of memory and putting your app in a hight risk of
> > > crashing.
> > >
> > > I added more code to my example to let you understand better
> > >
> > > Example:
> > >                fileStream = new FileStream();
> > >                fileStream.open( uploadedFile, FileMode.READ);
> > >                //fileStream.addEventListener(ProgressEvent.PROGRESS,
> > > onLoadProgress);
> > >                 var byteArraytoSend:ByteArray = new ByteArray();
> > >                var currentLength:int = Math.min(BUFFERSIZE,
> > > fileStream.bytesAvailable);
> > >                var bytesAvailable:uint = fileStream.bytesAvailable -
> > > byteArrayStartPosition;
> > >                var chunk:ByteArray = new ByteArray();
> > >                 while (fileStream.bytesAvailable > bytesAvailable) {
> > >                   chunk = new ByteArray();
> > >                   fileStream.readBytes(chunk, 0, currentLength);
> > >                   byteArraytoSend.writeBytes(chunk);
> > >                   fileUploadService.doUpload(byteArraytoSend);
> > >                 }
> > >
> > >               Then you have to do the checksum
> > >
> > >
> > > In this way you can stream chunks of 100kb and upload the files that
> you
> > > want.
> > > This solution is definetly more complex than using the FIleReference,
> so
> > it
> > > is up to you.
> > > Hope it helps
> > > Filippo
> > >
> > > On 28 March 2012 22:05, Avinash Narayanan <avinashapr9@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Looks like my little problem started quite the discussion!!.
> > > >
> > > > Ok, need to give a few updates. I managed to upload 1.07 Gb file
> > (windows
> > > > image file actually) via fileReference using HTTP. Still no luck with
> > > HTTP.
> > > > And the data upload is NOT causing any increase in the usage of RAM!
> > > (very
> > > > strange).
> > > >
> > > > The reason why I need the air app to upload large files is that we're
> > > > creating some sort of sales portal using Adobe Air and the client
> wants
> > > to
> > > > be able to upload these really large files which are supposed to make
> > > sense
> > > > to the client regarding the engagement. I have no clue what this file
> > is
> > > > for except that its upwards of 1Gb till 2Gb.
> > > >
> > > > Hope this helps!
> > > >
> > > > Thanks
> > > > Avinash Y
> > > >
> > > >
> > > > 2012/3/29 Jarosław Szczepankiewicz <jszczepankiewicz@gmail.com>
> > > >
> > > > > Supporting rest without tunneling through post is not possible due
> to
> > > > > limitations in flash player and due to plugin api limitations. Also
> > > full
> > > > > rest answers using full http response codes are not possible due
to
> > the
> > > > > same limitations of browser plugin api :(
> > > > > 28-03-2012 22:46 użytkownik "Hugo Matinho" <hugo.mig@gmail.com>
> > > napisał:
> > > > >
> > > > > > I know this might be the wrong place to ask and there is of
> course
> > > the
> > > > > > limitation on how the FP handles the HTTP transport but couldn't
> we
> > > do
> > > > > > something about flex supporting REST natively it's been several
> > years
> > > > > > already and yes there's blazeDS and GraniteDS and others but
most
> > > > > > JavaScript libraries support it natively most modern browsers
> > > implement
> > > > > > it and it wasn't made available on the player because browsers
> > didn't
> > > > > > had it back then but come on it's 2012 some rest support would
be
> > > nice
> > > > > >
> > > > > > Sent from my Windows Phone
> > > > > > From: Ariel Jakobovits
> > > > > > Sent: 28-03-2012 21:07
> > > > > > To: flex-dev@incubator.apache.org
> > > > > > Subject: Re: SFTP file transfer for large files using flex
> > > > > > > I might be off, but I don't think an ANE is, or would be,
part
> of
> > > > Flex.
> > > > > > It is an extension for AIR.
> > > > > >
> > > > > > Flex has components intended only for AIR (such as
> FileSystemList),
> > > > > > and offering a component that exists as an ANE/AS combo that
can
> be
> > > > > > added to a project to upload large files does not seem out of
the
> > > > > > scope of Flex.
> > > > > >
> > > > > > Ariel Jakobovits
> > > > > > Email: arieljake@yahoo.com
> > > > > > Phone: 650-690-2213
> > > > > > Fax: 650-641-0031
> > > > > > Cell: 650-823-8699
> > > > > >
> > > > > >
> > > > > > ________________________________
> > > > > >  From: JP Bader <jp@zavteq.com>
> > > > > > To: flex-dev@incubator.apache.org; Ariel Jakobovits <
> > > > arieljake@yahoo.com
> > > > > >
> > > > > > Sent: Wednesday, March 28, 2012 12:04 PM
> > > > > > Subject: Re: SFTP file transfer for large files using flex
> > > > > >
> > > > > > I might be off, but I don't think an ANE is, or would be, part
of
> > > > > > Flex.  It is an extension for AIR, which is still controlled
by
> > > Adobe.
> > > > > > If we want to see about spawning new threads that could consume
a
> > > > > > massive dump of data files, process, and pass those into a
> service
> > > > > > (FTP/SFTP/REST/Telnet/etc), that seems like a bit outside of
> scope
> > of
> > > > > > the Flex project, and more in line w/ Tamarin.
> > > > > >
> > > > > > We are working on the framework, which currently has limitations
> to
> > > > > > the FP and AIR (which are Adobe).  Potentially once we have
> access
> > to
> > > > > > Falcon(JS), there might be alternatives, but my question is,
what
> > 2GB
> > > > > > file are you trying to have FP/AIR upload?  There might be some
> > > tricks
> > > > > > to chunking out the flow of data using byte arrays, and clearing
> > them
> > > > > > out as you go along with loading them to whatever
> > repository/location
> > > > > > you need, but I can't think of too many desktop applications
that
> > > will
> > > > > > not just freeze up when you load a 2GB (or any major file) into
> it.
> > > > > > For video, that's one thing, but beyond a certain size, and
> > normally
> > > > > > the machine itself will slow down/become unresponsive.
> > > > > >
> > > > > > Most enterprises where I have done work, accessing and uploading
> > > > > > massive amounts of data would be FTP via terminals or clients,
> not
> > > > > > through a separate app (especially not AIR).  Maybe I'm missing
> the
> > > > > > reason behind wanting FP/AIR to handle that much data, but given
> > the
> > > > > > limitations of FP/AIR for processing data, I am not sure how
you
> > > would
> > > > > > approach building that into the framework.  I guess examples
of
> > > > > > BigAssCanvas show that you could stitch together large files
> > (larger
> > > > > > than FP would allow), but once you get to a certain size, it
> > becomes
> > > > > > unresponsive.
> > > > > >
> > > > > > If there is a way to do it, I apologize for going off tangent,
> but
> > we
> > > > > > need to focus on getting bugs fixed and improving the framework
> > > first,
> > > > > > and I'm not sure this is an improvement of the current framework
> so
> > > > > > much as a nice feature request.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > On Wed, Mar 28, 2012 at 1:14 PM, Ariel Jakobovits <
> > > arieljake@yahoo.com
> > > > >
> > > > > > wrote:
> > > > > > > would we consider developing ANEs to incorporate into Flex?
> > > > > > >
> > > > > > > Ariel Jakobovits
> > > > > > > Email: arieljake@yahoo.com
> > > > > > > Phone: 650-690-2213
> > > > > > > Fax: 650-641-0031
> > > > > > > Cell: 650-823-8699
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > >  From: Marcus Fritze <marcus.fritze@googlemail.com>
> > > > > > > To: flex-dev@incubator.apache.org; Marcus Fritze <
> > > > > > marcus.fritze@googlemail.com>
> > > > > > > Sent: Wednesday, March 28, 2012 10:34 AM
> > > > > > > Subject: Re: SFTP file transfer for large files using flex
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > as maybe all know Flash supports only a file size of 100MB.
> > > > > > >
> > > > > > > from the docs:
> > > > > > >
> > > > > > > FileReference -> upload()
> > > > > > >
> > > > > > >> Although Flash Player has no restriction on the size
of files
> > you
> > > > can
> > > > > > upload or download, the player officially supports uploads or
> > > downloads
> > > > > of
> > > > > > up to 100 MB.
> > > > > > >
> > > > > > > I experienced that an upload / download of a larger file
via
> > Flash
> > > > > > (FileReference) is possible.
> > > > > > >
> > > > > > > I agree with Avinash! I think this is a good place to discuss
> > this
> > > > > topic
> > > > > > (and not the stack overflow forum). Because this mailing list
is
> > the
> > > > > place
> > > > > > where we can improve the Flex SDK / Flash.
> > > > > > >
> > > > > > > Correct me if I am wrong, but an upload of large files
(2 GB or
> > > more)
> > > > > is
> > > > > > currently not possible in Flash. And maybe we can check if it
is
> > > > possible
> > > > > > to enable a larger file upload / download via Flex / Flash.
This
> > > > feature
> > > > > is
> > > > > > very useful for Enterprise RIA's. I have already developed such
> an
> > > app
> > > > > and
> > > > > > have the same problem.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Marcus
> > > > > > >
> > > > > > > Am 28.03.2012 um 16:55 schrieb Avinash Narayanan:
> > > > > > >
> > > > > > >> Hi JP,
> > > > > > >>
> > > > > > >> Thanks for taking time to reply! Stackoverflow had
questions
> > like
> > > > this
> > > > > > but
> > > > > > >> not satisfactory answers. This was my last ditch effort
since
> > the
> > > > > other
> > > > > > >> option is to write my own SFTP plug (which clients
wont pay me
> > to
> > > do
> > > > > > :|).
> > > > > > >>
> > > > > > >> As for the apps, this upload needs to be 'part' of
a desktop
> > > > > application
> > > > > > >> written in adobe air so can't use any of these other
tools :(
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >> Avinash Y
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Mar 28, 2012 at 6:54 PM, JP Bader <jp@zavteq.com>
> > wrote:
> > > > > > >>
> > > > > > >>> Hi Avinash,
> > > > > > >>>
> > > > > > >>> My quick take on this is why you would want to
recreate a
> tool
> > > for
> > > > > > >>> this?  Aren't there plenty of free SFTP apps already?
 Why
> not
> > > take
> > > > > > >>> advantage of one of them? Tortoise, Cute, WinSCP,
FileZilla,
> > etc.
> > > > > > >>>
> > > > > > >>> Also, questions like these might be better asked
on other
> > forums
> > > > like
> > > > > > >>> stack overflow.  This mailing list is for questions
and
> > > development
> > > > > > >>> regarding the next release of Flex.
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>>
> > > > > > >>> JP
> > > > > > >>>
> > > > > > >>> On Wed, Mar 28, 2012 at 3:16 AM, Avinash Narayanan
> > > > > > >>> <avinashapr9@gmail.com> wrote:
> > > > > > >>>> Hi All,
> > > > > > >>>>
> > > > > > >>>> I know this community is meant for future development
of
> > Apache
> > > > Flex
> > > > > > but
> > > > > > >>>> I'm a little lost here. I need to upload ~2Gb
file from my
> > local
> > > > > > machine
> > > > > > >>> to
> > > > > > >>>> a url via an air app using sftp. problem is
we don't have
> sftp
> > > lib
> > > > > > >>>> available.
> > > > > > >>>>
> > > > > > >>>> Also, I'm confused as to why in forums it says
the flash
> > player
> > > > > needs
> > > > > > to
> > > > > > >>>> take the entire file into memory before loading
it but I
> don't
> > > see
> > > > > any
> > > > > > >>>> memory increase in my task manager (using the
fileReference
> > > > class).
> > > > > > >>>>
> > > > > > >>>> Another thing I wanted to know is how best
to split up the
> > file
> > > > into
> > > > > > >>> chunks
> > > > > > >>>> for multi-part loading? Any help in any of
these areas will
> be
> > > > > > supremely
> > > > > > >>>> helpful!
> > > > > > >>>>
> > > > > > >>>> I've already seen the following links
> > > > > > >>>>
> > > > > > >>>> http://www.actionscript.org/forums/showthread.php3?t=181895
> > > > > > >>>> http://forums.adobe.com/thread/631103
> > > > > > >>>> http://blog.ansuz.nl/index.php/2011/02/11/flex-ftp-abort/
> > > > > > >>>> http://maliboo.pl/projects/FlexFTP/
> > > > > > >>>> http://forums.adobe.com/thread/245294
> > > > > > >>>>
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://blog.flexexamples.com/2007/09/21/uploading-files-in-flex-using-the-filereference-class/
> > > > > > >>>>
> > > > > > >>>> Thanks in Advance.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> With Warm Regards,
> > > > > > >>>> Avinash
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> JP Bader
> > > > > > >>> Principal
> > > > > > >>> Zavteq, Inc.
> > > > > > >>> @lordB8r | jp@zavteq.com
> > > > > > >>> 608.692.2468
> > > > > > >>>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > JP Bader
> > > > > > Principal
> > > > > > Zavteq, Inc.
> > > > > > @lordB8r | jp@zavteq.com
> > > > > > 608.692.2468
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message