hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Gordineer <joshg...@gmail.com>
Subject Re: Handling (indefinitely) streaming sources
Date Sun, 25 Mar 2012 17:50:53 GMT
Aha, since the response isn't a valid HTTP message (ICY 200 OK) the
DefaultResponseParser loops looking for a valid protocol which will never
come.  Setting the maxGarbageLines in my request fixed the issue.

Thanks!

--Josh



On Sun, Mar 25, 2012 at 7:04 AM, Oleg Kalnichevski <olegk@apache.org> wrote:

> On Sat, 2012-03-24 at 17:15 -0700, Josh Gordineer wrote:
> > We do count the bytes we read from the inputstream but in our case we
> don't
> > get the HttpResponse back from the execute so we don't have a chance to
> > stop the execution while reading.
> >
>
> HttpClient returns a HttpResponse object as soon as it receives an HTTP
> message head consisting of a status line and response headers. The
> message body is not processed inside the #execute method. If HttpClient
> does not return from the method, it means the server has sent no
> response at all.
>
> Besides, one can abort the request at any point of its execution by
> using HttpUriRequest#abort method.
>
> Hope this helps
>
> Oleg
>
> > Am I missing something obvious about how to access the inputstream from
> the
> > entity?
> >
> > Here's a quick sample of our usage in this case:
> >
> > import java.io.IOException;
> > import java.io.InputStream;
> >
> > import org.apache.http.HttpResponse;
> > import org.apache.http.client.HttpClient;
> > import org.apache.http.client.methods.HttpGet;
> > import org.apache.http.impl.client.DefaultHttpClient;
> >
> > public class StreamTest {
> >
> >   public static void main(String[] args) {
> >     HttpClient httpclient = new DefaultHttpClient();
> >     HttpGet get = new HttpGet(
> >         "http://scfire-dtc-aa04.stream.aol.com:80/stream/1030/7.html");
> >     try {
> >       HttpResponse response = httpclient.execute(get); // does not return
> >       InputStream stream = response.getEntity().getContent();
> >       // read stream and count bytes, exit if we go above our max bytes
> >     } catch (IOException e) {
> >       e.printStackTrace();
> >     }
> >   }
> > }
> >
> >
> >
> >
> > On Sat, Mar 24, 2012 at 4:34 AM, Oleg Kalnichevski <olegk@apache.org>
> wrote:
> >
> > > On Fri, 2012-03-23 at 17:05 -0700, Josh Gordineer wrote:
> > > > I have a question about how to protect against requests to streamed
> > > > sources.  A project background is that we allow execution on our
> servers
> > > of
> > > > outbound http requests based on user input.  So in essence we need to
> > > > protect against abuse by adding restrictions on response size/time
> etc.
> > >  We
> > > > have done this successfully in the past by adding a counter to the
> > > > inputstream to make sure data is below an arbitrary max size
> however, we
> > > > have come across a case when a user inputs (either incorrectly or
> > > > maliciously) tries to fetch an audio stream which causes our read
> thread
> > > to
> > > > continue endlessly reading the content (details of the actual feed
> and
> > > > stacktrace below).  This request hangs getting the HttpResponse from
> the
> > > > httpclient.execute(request) call.  Ideally we could read the response
> > > > header however since the client is hanging at the execute line I
> don't
> > > have
> > > > the handle to fetch the headers.
> > > >
> > > > Basically we need identify these requests and kill them prior to
> causing
> > > > our machine to churn reading data.  My first idea is adding a hook in
> > > > httpclient to track the size of the request we are reading and kill
> it
> > > > after it gets past a threshold however it wasn't clear to me how to
> do so
> > > > with httpclient (I looked at creating a specialization of
> > > > DefaultResponseParser however I didn't see how to instantiate my
> > > version).
> > > >
> > > > Any suggestions are welcome.  Thanks in advance!
> > > >
> > >
> > > Why can't you just read from the input stream, count bytes read, and
> > > abort the request in case the count exceeds a certain limit?
> > >
> > > Oleg
> > >
> > > > --Josh
> > > >
> > > > Detailed info:
> > > >
> > > > $ curl -v
> http://scfire-dtc-aa04.stream.aol.com:80/stream/1030/7.html
> > > > * About to connect() to scfire-dtc-aa04.stream.aol.com port 80 (#0)
> > > > *   Trying 205.188.234.4... connected
> > > > * Connected to scfire-dtc-aa04.stream.aol.com (205.188.234.4) port
> 80
> > > (#0)
> > > > > GET /stream/1030/7.html HTTP/1.1
> > > > > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7
> > > > OpenSSL/0.9.8r zlib/1.2.3
> > > > > Host: scfire-dtc-aa04.stream.aol.com
> > > > > Accept: */*
> > > > >
> > > > ICY 200 OK
> > > > icy-notice1: <BR>This stream requires <a href="
> http://www.winamp.com/
> > > > ">Winamp</a><BR>
> > > > icy-notice2: Firehose Ultravox/SHOUTcast Relay Server/Linux
> v2.6.0<BR>
> > > > icy-name: RADIOUP.COM - THE HITLIST (formely 108.fm) - #1 FOR ALL
> HIT
> > > MUSIC
> > > > icy-genre: Top 40 Pop Rap Hip Hop Top40
> > > > icy-url: http://www.radioup.com/
> > > > content-type: audio/mpeg
> > > > icy-pub: 1
> > > > icy-br: 128
> > > >
> > > > Sstack dump for the read thread (versions httpclient-4.1.3.jar
> > > >  httpcore-4.1.4.jar):
> > > >
> > > > "Instance-thread-1" prio=10 tid=0x89ae5c00 nid=0xe04 runnable
> > > [0x03785000]
> > > >    java.lang.Thread.State: RUNNABLE
> > > > at java.net.SocketInputStream.socketRead0(Native Method)
> > > > at java.net.SocketInputStream.read(SocketInputStream.java:129)
> > > > at
> > > >
> > >
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149)
> > > > at
> > > >
> > >
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:111)
> > > > at
> > > >
> > >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:264)
> > > > at
> > > >
> > >
> org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > > > at
> > > >
> > >
> org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > > > at
> > > >
> > >
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
> > > > at
> > > >
> > >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:282)
> > > > at
> > > >
> > >
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
> > > > at
> > > >
> > >
> org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:216)
> > > > at
> > > >
> > >
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
> > > > at
> > > >
> > >
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> > > > at
> > > >
> > >
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:647)
> > > > at
> > > >
> > >
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)
> > > > at
> > > >
> > >
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
> > > > at
> > > >
> > >
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
> > > > at
> > > >
> > >
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
> > > > <snip>
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message