hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sachin Shetty" <sshe...@egnyte.com>
Subject RE: How to stream a file via PUT to multiple HttpClients
Date Wed, 20 Jul 2011 15:47:23 GMT
Yeah, I already tried httpurlconnnection and got road blocked by the int field input in httpConnection.setFixedLengthStreamingMode()
- max file size can only be 2GB until 7.0.

So still looking for some real options.

Thanks
Sachin


-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org] 
Sent: Wednesday, July 20, 2011 7:42 PM
To: HttpClient User Discussion
Subject: Re: How to stream a file via PUT to multiple HttpClients

On Wed, 2011-07-20 at 09:37 -0400, Jon Moore wrote:
> Hi Sachin,
> 
> In this case, I would suggest running the .execute() calls in separate
> threads, or perhaps using the asynchronous HttpClient:
> http://hc.apache.org/httpcomponents-asyncclient-dev/index.html
> 
> I think those are your only options given the existing APIs for HttpClient.
> 
> Jon
> 

I agree with Jon. Zero copy support in HttpAsyncClient might be the best
option available to you. Otherwise you will have to use
HttpUrlConnection or some other HTTP client.

Oleg


> On Wed, Jul 20, 2011 at 9:20 AM, Sachin Shetty <sshetty@egnyte.com> wrote:
> > But since I would execute client.execute(put1) and client.execute(put2)
> > serially in a single thread, ForkedInputStream will end up buffering almost
> > the entire file for put2 which I want to avoid at any cost.
> >
> > In short, I want to spawn two puts and write to them in blocks one after
> > another.
> >
> > -----Original Message-----
> > From: Jon Moore [mailto:jonm@apache.org]
> > Sent: Wednesday, July 20, 2011 6:42 PM
> > To: HttpClient User Discussion
> > Subject: Re: How to stream a file via PUT to multiple HttpClients
> >
> > Hi Sachin,
> >
> > I think what you want to do is wrap your FileInputStream in another
> > object, let's call it a ForkedInputStream:
> >
> > FileInputStream fis = ...;
> > ForkedInputStream forked = new ForkedInputStream(fis);
> > HttpPut put1 = new HttpPut(...);
> > HttpPut put2 = new HttpPut(...);
> > put1.setEntity(new InputStreamEntity(forked.getLeft()));
> > put2.setEntity(new InputStreamEntity(forked.getRight()));
> >
> > etc.
> >
> > Then your ForkedInputStream takes care of reading from the underlying
> > File (perhaps on demand) and keeping the multiple writers in as much
> > lockstep or with as much buffering as desired.
> >
> > Jon
> >
> > On Wed, Jul 20, 2011 at 6:02 AM, Sachin Shetty <sshetty@egnyte.com> wrote:
> >> Hi,
> >>
> >>
> >>
> >> We have a unique case where we want to put a file via HTTP PUT to more
> > than
> >> one http servers  but we want to read the file only once which means we
> > want
> >> to read a block of data from the file and write  to two entities
> >> simultaneously. Streaming the PUT and reading the file only once without
> >> having to hold it in memory are the two things we cannot avoid.
> >>
> >>
> >>
> >> These things are pretty easy when we have a direct outputstream to write
> > to,
> >> but I am not able to figure out how this can be achieved using the
> > entities.
> >>
> >>
> >>
> >> basically we want an httpclient equivalent of:
> >>
> >>
> >>
> >> OutputStream stream1 = ...
> >>
> >> OutputStream stream2 = ...
> >>
> >>
> >>
> >> while (input.read(byte[1024]) {
> >>
> >>   stream1.write(bytes);
> >>
> >>   stream2.write(bytes);
> >>
> >> }
> >>
> >>
> >>
> >> Please let me know if there is a way out?
> >>
> >>
> >>
> >> Thanks
> >>
> >> Sachin
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message