incubator-flex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Om <bigosma...@gmail.com>
Subject Re: SFTP file transfer for large files using flex
Date Thu, 19 Apr 2012 19:02:08 GMT
Avinash,

Did you try the method that Clint has suggested?  That approach looks
solid.  The beauty of AIR is that you can use the NativeProcess API for
such special requirements.  No need to build everything in
Flex/Actionscript.

Thanks,
Om

On Thu, Apr 19, 2012 at 2:11 AM, Avinash Narayanan <avinashapr9@gmail.com>wrote:

> Hi Disa,
>
> The problem is since we're using HTTPS, we get 4Kb of authentication as a
> separate response + 800bytes headers & 900 bytes response in the POST.
> we're now looking at 15 Gb uploads. Seems to be too many transactions?
>
> Thanks
> Avinash Y
>
>
> 2012/4/19 filippo dipisa <filippo@dipisa.net>
>
> > usually my buffer is 100K for chunk
> > hope it helps
> > Filippo
> >
> > On 19 April 2012 10:00, Avinash Narayanan <avinashapr9@gmail.com> wrote:
> >
> > > Hi Guys,
> > >
> > > Sorry for the late reply but I wanted to have some concrete answers
> > before
> > > posting to the group again. I used the byte array & URLLoader to upload
> > > files in separate post methods. I was also able to divide the file into
> > > byte array chunks + boundary and then upload in multi-part.
> > >
> > > Now the problem is if I use HTTP it works perfectly for over 3.5 Gb,
> > when I
> > > use HTTPS it gets stops abruptly somewhere around the 1.5 Gb mark for
> 10
> > Mb
> > > chunks and 300 Mb mark for 50 Mb chunks. No clue why (help please)
> > >
> > > I also observed that if you're uploading 10Mb chunks it takes a peak
> > memory
> > > of 117 Mb and avg mem of 30 Mb. For 50 Mb chunks it takes a peak memory
> > of
> > > 300 Mb and average of 80 Mb. Why so much difference in peak memory? Any
> > > ideas why?
> > >
> > > Will be happy to share code in case it will be useful to someone.
> > >
> > > Thanks
> > > Avinash Y
> > >
> > >
> > > 2012/3/29 filippo dipisa <filippo@dipisa.net>
> > >
> > > > I had many problems using the filereference in terms of perfomance.
> > > > For my experience the best approach is to read the file, splitting it
> > in
> > > > chunks of 100kb (buffersize), send the chunks to the server.
> > > > In this case Air will perform a file stream like,  to the server.
> > > > Using the FileReference it loads the file first and then send it to
> the
> > > > server using a lot of memory and putting your app in a hight risk of
> > > > crashing.
> > > >
> > > > I added more code to my example to let you understand better
> > > >
> > > > Example:
> > > >                fileStream = new FileStream();
> > > >                fileStream.open( uploadedFile, FileMode.READ);
> > > >                //fileStream.addEventListener(ProgressEvent.PROGRESS,
> > > > onLoadProgress);
> > > >                 var byteArraytoSend:ByteArray = new ByteArray();
> > > >                var currentLength:int = Math.min(BUFFERSIZE,
> > > > fileStream.bytesAvailable);
> > > >                var bytesAvailable:uint = fileStream.bytesAvailable -
> > > > byteArrayStartPosition;
> > > >                var chunk:ByteArray = new ByteArray();
> > > >                 while (fileStream.bytesAvailable > bytesAvailable)
{
> > > >                   chunk = new ByteArray();
> > > >                   fileStream.readBytes(chunk, 0, currentLength);
> > > >                   byteArraytoSend.writeBytes(chunk);
> > > >                   fileUploadService.doUpload(byteArraytoSend);
> > > >                 }
> > > >
> > > >               Then you have to do the checksum
> > > >
> > > >
> > > > In this way you can stream chunks of 100kb and upload the files that
> > you
> > > > want.
> > > > This solution is definetly more complex than using the FIleReference,
> > so
> > > it
> > > > is up to you.
> > > > Hope it helps
> > > > Filippo
> > > >
> > > > On 28 March 2012 22:05, Avinash Narayanan <avinashapr9@gmail.com>
> > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Looks like my little problem started quite the discussion!!.
> > > > >
> > > > > Ok, need to give a few updates. I managed to upload 1.07 Gb file
> > > (windows
> > > > > image file actually) via fileReference using HTTP. Still no luck
> with
> > > > HTTP.
> > > > > And the data upload is NOT causing any increase in the usage of
> RAM!
> > > > (very
> > > > > strange).
> > > > >
> > > > > The reason why I need the air app to upload large files is that
> we're
> > > > > creating some sort of sales portal using Adobe Air and the client
> > wants
> > > > to
> > > > > be able to upload these really large files which are supposed to
> make
> > > > sense
> > > > > to the client regarding the engagement. I have no clue what this
> file
> > > is
> > > > > for except that its upwards of 1Gb till 2Gb.
> > > > >
> > > > > Hope this helps!
> > > > >
> > > > > Thanks
> > > > > Avinash Y
> > > > >
> > > > >
> > > > > 2012/3/29 Jarosław Szczepankiewicz <jszczepankiewicz@gmail.com>
> > > > >
> > > > > > Supporting rest without tunneling through post is not possible
> due
> > to
> > > > > > limitations in flash player and due to plugin api limitations.
> Also
> > > > full
> > > > > > rest answers using full http response codes are not possible
due
> to
> > > the
> > > > > > same limitations of browser plugin api :(
> > > > > > 28-03-2012 22:46 użytkownik "Hugo Matinho" <hugo.mig@gmail.com>
> > > > napisał:
> > > > > >
> > > > > > > I know this might be the wrong place to ask and there is
of
> > course
> > > > the
> > > > > > > limitation on how the FP handles the HTTP transport but
> couldn't
> > we
> > > > do
> > > > > > > something about flex supporting REST natively it's been
several
> > > years
> > > > > > > already and yes there's blazeDS and GraniteDS and others
but
> most
> > > > > > > JavaScript libraries support it natively most modern browsers
> > > > implement
> > > > > > > it and it wasn't made available on the player because browsers
> > > didn't
> > > > > > > had it back then but come on it's 2012 some rest support
would
> be
> > > > nice
> > > > > > >
> > > > > > > Sent from my Windows Phone
> > > > > > > From: Ariel Jakobovits
> > > > > > > Sent: 28-03-2012 21:07
> > > > > > > To: flex-dev@incubator.apache.org
> > > > > > > Subject: Re: SFTP file transfer for large files using flex
> > > > > > > > I might be off, but I don't think an ANE is, or would
be,
> part
> > of
> > > > > Flex.
> > > > > > > It is an extension for AIR.
> > > > > > >
> > > > > > > Flex has components intended only for AIR (such as
> > FileSystemList),
> > > > > > > and offering a component that exists as an ANE/AS combo
that
> can
> > be
> > > > > > > added to a project to upload large files does not seem
out of
> the
> > > > > > > scope of Flex.
> > > > > > >
> > > > > > > Ariel Jakobovits
> > > > > > > Email: arieljake@yahoo.com
> > > > > > > Phone: 650-690-2213
> > > > > > > Fax: 650-641-0031
> > > > > > > Cell: 650-823-8699
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > >  From: JP Bader <jp@zavteq.com>
> > > > > > > To: flex-dev@incubator.apache.org; Ariel Jakobovits <
> > > > > arieljake@yahoo.com
> > > > > > >
> > > > > > > Sent: Wednesday, March 28, 2012 12:04 PM
> > > > > > > Subject: Re: SFTP file transfer for large files using flex
> > > > > > >
> > > > > > > I might be off, but I don't think an ANE is, or would be,
part
> of
> > > > > > > Flex.  It is an extension for AIR, which is still controlled
by
> > > > Adobe.
> > > > > > > If we want to see about spawning new threads that could
> consume a
> > > > > > > massive dump of data files, process, and pass those into
a
> > service
> > > > > > > (FTP/SFTP/REST/Telnet/etc), that seems like a bit outside
of
> > scope
> > > of
> > > > > > > the Flex project, and more in line w/ Tamarin.
> > > > > > >
> > > > > > > We are working on the framework, which currently has
> limitations
> > to
> > > > > > > the FP and AIR (which are Adobe).  Potentially once we
have
> > access
> > > to
> > > > > > > Falcon(JS), there might be alternatives, but my question
is,
> what
> > > 2GB
> > > > > > > file are you trying to have FP/AIR upload?  There might
be some
> > > > tricks
> > > > > > > to chunking out the flow of data using byte arrays, and
> clearing
> > > them
> > > > > > > out as you go along with loading them to whatever
> > > repository/location
> > > > > > > you need, but I can't think of too many desktop applications
> that
> > > > will
> > > > > > > not just freeze up when you load a 2GB (or any major file)
into
> > it.
> > > > > > > For video, that's one thing, but beyond a certain size,
and
> > > normally
> > > > > > > the machine itself will slow down/become unresponsive.
> > > > > > >
> > > > > > > Most enterprises where I have done work, accessing and
> uploading
> > > > > > > massive amounts of data would be FTP via terminals or clients,
> > not
> > > > > > > through a separate app (especially not AIR).  Maybe I'm
missing
> > the
> > > > > > > reason behind wanting FP/AIR to handle that much data,
but
> given
> > > the
> > > > > > > limitations of FP/AIR for processing data, I am not sure
how
> you
> > > > would
> > > > > > > approach building that into the framework.  I guess examples
of
> > > > > > > BigAssCanvas show that you could stitch together large
files
> > > (larger
> > > > > > > than FP would allow), but once you get to a certain size,
it
> > > becomes
> > > > > > > unresponsive.
> > > > > > >
> > > > > > > If there is a way to do it, I apologize for going off tangent,
> > but
> > > we
> > > > > > > need to focus on getting bugs fixed and improving the framework
> > > > first,
> > > > > > > and I'm not sure this is an improvement of the current
> framework
> > so
> > > > > > > much as a nice feature request.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > On Wed, Mar 28, 2012 at 1:14 PM, Ariel Jakobovits <
> > > > arieljake@yahoo.com
> > > > > >
> > > > > > > wrote:
> > > > > > > > would we consider developing ANEs to incorporate into
Flex?
> > > > > > > >
> > > > > > > > Ariel Jakobovits
> > > > > > > > Email: arieljake@yahoo.com
> > > > > > > > Phone: 650-690-2213
> > > > > > > > Fax: 650-641-0031
> > > > > > > > Cell: 650-823-8699
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > >  From: Marcus Fritze <marcus.fritze@googlemail.com>
> > > > > > > > To: flex-dev@incubator.apache.org; Marcus Fritze <
> > > > > > > marcus.fritze@googlemail.com>
> > > > > > > > Sent: Wednesday, March 28, 2012 10:34 AM
> > > > > > > > Subject: Re: SFTP file transfer for large files using
flex
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > as maybe all know Flash supports only a file size
of 100MB.
> > > > > > > >
> > > > > > > > from the docs:
> > > > > > > >
> > > > > > > > FileReference -> upload()
> > > > > > > >
> > > > > > > >> Although Flash Player has no restriction on the
size of
> files
> > > you
> > > > > can
> > > > > > > upload or download, the player officially supports uploads
or
> > > > downloads
> > > > > > of
> > > > > > > up to 100 MB.
> > > > > > > >
> > > > > > > > I experienced that an upload / download of a larger
file via
> > > Flash
> > > > > > > (FileReference) is possible.
> > > > > > > >
> > > > > > > > I agree with Avinash! I think this is a good place
to discuss
> > > this
> > > > > > topic
> > > > > > > (and not the stack overflow forum). Because this mailing
list
> is
> > > the
> > > > > > place
> > > > > > > where we can improve the Flex SDK / Flash.
> > > > > > > >
> > > > > > > > Correct me if I am wrong, but an upload of large files
(2 GB
> or
> > > > more)
> > > > > > is
> > > > > > > currently not possible in Flash. And maybe we can check
if it
> is
> > > > > possible
> > > > > > > to enable a larger file upload / download via Flex / Flash.
> This
> > > > > feature
> > > > > > is
> > > > > > > very useful for Enterprise RIA's. I have already developed
such
> > an
> > > > app
> > > > > > and
> > > > > > > have the same problem.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Marcus
> > > > > > > >
> > > > > > > > Am 28.03.2012 um 16:55 schrieb Avinash Narayanan:
> > > > > > > >
> > > > > > > >> Hi JP,
> > > > > > > >>
> > > > > > > >> Thanks for taking time to reply! Stackoverflow
had questions
> > > like
> > > > > this
> > > > > > > but
> > > > > > > >> not satisfactory answers. This was my last ditch
effort
> since
> > > the
> > > > > > other
> > > > > > > >> option is to write my own SFTP plug (which clients
wont pay
> me
> > > to
> > > > do
> > > > > > > :|).
> > > > > > > >>
> > > > > > > >> As for the apps, this upload needs to be 'part'
of a desktop
> > > > > > application
> > > > > > > >> written in adobe air so can't use any of these
other tools
> :(
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >> Avinash Y
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Mar 28, 2012 at 6:54 PM, JP Bader <jp@zavteq.com>
> > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi Avinash,
> > > > > > > >>>
> > > > > > > >>> My quick take on this is why you would want
to recreate a
> > tool
> > > > for
> > > > > > > >>> this?  Aren't there plenty of free SFTP apps
already?  Why
> > not
> > > > take
> > > > > > > >>> advantage of one of them? Tortoise, Cute,
WinSCP,
> FileZilla,
> > > etc.
> > > > > > > >>>
> > > > > > > >>> Also, questions like these might be better
asked on other
> > > forums
> > > > > like
> > > > > > > >>> stack overflow.  This mailing list is for
questions and
> > > > development
> > > > > > > >>> regarding the next release of Flex.
> > > > > > > >>>
> > > > > > > >>> Regards,
> > > > > > > >>>
> > > > > > > >>> JP
> > > > > > > >>>
> > > > > > > >>> On Wed, Mar 28, 2012 at 3:16 AM, Avinash Narayanan
> > > > > > > >>> <avinashapr9@gmail.com> wrote:
> > > > > > > >>>> Hi All,
> > > > > > > >>>>
> > > > > > > >>>> I know this community is meant for future
development of
> > > Apache
> > > > > Flex
> > > > > > > but
> > > > > > > >>>> I'm a little lost here. I need to upload
~2Gb file from my
> > > local
> > > > > > > machine
> > > > > > > >>> to
> > > > > > > >>>> a url via an air app using sftp. problem
is we don't have
> > sftp
> > > > lib
> > > > > > > >>>> available.
> > > > > > > >>>>
> > > > > > > >>>> Also, I'm confused as to why in forums
it says the flash
> > > player
> > > > > > needs
> > > > > > > to
> > > > > > > >>>> take the entire file into memory before
loading it but I
> > don't
> > > > see
> > > > > > any
> > > > > > > >>>> memory increase in my task manager (using
the
> fileReference
> > > > > class).
> > > > > > > >>>>
> > > > > > > >>>> Another thing I wanted to know is how
best to split up the
> > > file
> > > > > into
> > > > > > > >>> chunks
> > > > > > > >>>> for multi-part loading? Any help in any
of these areas
> will
> > be
> > > > > > > supremely
> > > > > > > >>>> helpful!
> > > > > > > >>>>
> > > > > > > >>>> I've already seen the following links
> > > > > > > >>>>
> > > > > > > >>>>
> http://www.actionscript.org/forums/showthread.php3?t=181895
> > > > > > > >>>> http://forums.adobe.com/thread/631103
> > > > > > > >>>> http://blog.ansuz.nl/index.php/2011/02/11/flex-ftp-abort/
> > > > > > > >>>> http://maliboo.pl/projects/FlexFTP/
> > > > > > > >>>> http://forums.adobe.com/thread/245294
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://blog.flexexamples.com/2007/09/21/uploading-files-in-flex-using-the-filereference-class/
> > > > > > > >>>>
> > > > > > > >>>> Thanks in Advance.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> With Warm Regards,
> > > > > > > >>>> Avinash
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> JP Bader
> > > > > > > >>> Principal
> > > > > > > >>> Zavteq, Inc.
> > > > > > > >>> @lordB8r | jp@zavteq.com
> > > > > > > >>> 608.692.2468
> > > > > > > >>>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > JP Bader
> > > > > > > Principal
> > > > > > > Zavteq, Inc.
> > > > > > > @lordB8r | jp@zavteq.com
> > > > > > > 608.692.2468
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message