hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleg Kalnichevski <ol...@apache.org>
Subject Re: Handling (indefinitely) streaming sources
Date Sun, 25 Mar 2012 14:04:57 GMT
On Sat, 2012-03-24 at 17:15 -0700, Josh Gordineer wrote:
> We do count the bytes we read from the inputstream but in our case we don't
> get the HttpResponse back from the execute so we don't have a chance to
> stop the execution while reading.
> 

HttpClient returns a HttpResponse object as soon as it receives an HTTP
message head consisting of a status line and response headers. The
message body is not processed inside the #execute method. If HttpClient
does not return from the method, it means the server has sent no
response at all.

Besides, one can abort the request at any point of its execution by
using HttpUriRequest#abort method.

Hope this helps

Oleg

> Am I missing something obvious about how to access the inputstream from the
> entity?
> 
> Here's a quick sample of our usage in this case:
> 
> import java.io.IOException;
> import java.io.InputStream;
> 
> import org.apache.http.HttpResponse;
> import org.apache.http.client.HttpClient;
> import org.apache.http.client.methods.HttpGet;
> import org.apache.http.impl.client.DefaultHttpClient;
> 
> public class StreamTest {
> 
>   public static void main(String[] args) {
>     HttpClient httpclient = new DefaultHttpClient();
>     HttpGet get = new HttpGet(
>         "http://scfire-dtc-aa04.stream.aol.com:80/stream/1030/7.html");
>     try {
>       HttpResponse response = httpclient.execute(get); // does not return
>       InputStream stream = response.getEntity().getContent();
>       // read stream and count bytes, exit if we go above our max bytes
>     } catch (IOException e) {
>       e.printStackTrace();
>     }
>   }
> }
> 
> 
> 
> 
> On Sat, Mar 24, 2012 at 4:34 AM, Oleg Kalnichevski <olegk@apache.org> wrote:
> 
> > On Fri, 2012-03-23 at 17:05 -0700, Josh Gordineer wrote:
> > > I have a question about how to protect against requests to streamed
> > > sources.  A project background is that we allow execution on our servers
> > of
> > > outbound http requests based on user input.  So in essence we need to
> > > protect against abuse by adding restrictions on response size/time etc.
> >  We
> > > have done this successfully in the past by adding a counter to the
> > > inputstream to make sure data is below an arbitrary max size however, we
> > > have come across a case when a user inputs (either incorrectly or
> > > maliciously) tries to fetch an audio stream which causes our read thread
> > to
> > > continue endlessly reading the content (details of the actual feed and
> > > stacktrace below).  This request hangs getting the HttpResponse from the
> > > httpclient.execute(request) call.  Ideally we could read the response
> > > header however since the client is hanging at the execute line I don't
> > have
> > > the handle to fetch the headers.
> > >
> > > Basically we need identify these requests and kill them prior to causing
> > > our machine to churn reading data.  My first idea is adding a hook in
> > > httpclient to track the size of the request we are reading and kill it
> > > after it gets past a threshold however it wasn't clear to me how to do so
> > > with httpclient (I looked at creating a specialization of
> > > DefaultResponseParser however I didn't see how to instantiate my
> > version).
> > >
> > > Any suggestions are welcome.  Thanks in advance!
> > >
> >
> > Why can't you just read from the input stream, count bytes read, and
> > abort the request in case the count exceeds a certain limit?
> >
> > Oleg
> >
> > > --Josh
> > >
> > > Detailed info:
> > >
> > > $ curl -v http://scfire-dtc-aa04.stream.aol.com:80/stream/1030/7.html
> > > * About to connect() to scfire-dtc-aa04.stream.aol.com port 80 (#0)
> > > *   Trying 205.188.234.4... connected
> > > * Connected to scfire-dtc-aa04.stream.aol.com (205.188.234.4) port 80
> > (#0)
> > > > GET /stream/1030/7.html HTTP/1.1
> > > > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7
> > > OpenSSL/0.9.8r zlib/1.2.3
> > > > Host: scfire-dtc-aa04.stream.aol.com
> > > > Accept: */*
> > > >
> > > ICY 200 OK
> > > icy-notice1: <BR>This stream requires <a href="http://www.winamp.com/
> > > ">Winamp</a><BR>
> > > icy-notice2: Firehose Ultravox/SHOUTcast Relay Server/Linux v2.6.0<BR>
> > > icy-name: RADIOUP.COM - THE HITLIST (formely 108.fm) - #1 FOR ALL HIT
> > MUSIC
> > > icy-genre: Top 40 Pop Rap Hip Hop Top40
> > > icy-url: http://www.radioup.com/
> > > content-type: audio/mpeg
> > > icy-pub: 1
> > > icy-br: 128
> > >
> > > Sstack dump for the read thread (versions httpclient-4.1.3.jar
> > >  httpcore-4.1.4.jar):
> > >
> > > "Instance-thread-1" prio=10 tid=0x89ae5c00 nid=0xe04 runnable
> > [0x03785000]
> > >    java.lang.Thread.State: RUNNABLE
> > > at java.net.SocketInputStream.socketRead0(Native Method)
> > > at java.net.SocketInputStream.read(SocketInputStream.java:129)
> > > at
> > >
> > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149)
> > > at
> > >
> > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:111)
> > > at
> > >
> > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:264)
> > > at
> > >
> > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > > at
> > >
> > org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > > at
> > >
> > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
> > > at
> > >
> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:282)
> > > at
> > >
> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
> > > at
> > >
> > org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:216)
> > > at
> > >
> > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
> > > at
> > >
> > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> > > at
> > >
> > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:647)
> > > at
> > >
> > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
> > > at
> > >
> > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
> > > at
> > >
> > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
> > > at
> > >
> > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
> > > <snip>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Mime
View raw message