Return-Path: Delivered-To: apmail-perl-modperl-archive@www.apache.org Received: (qmail 80935 invoked from network); 26 Aug 2010 16:29:17 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 Aug 2010 16:29:17 -0000 Received: (qmail 27375 invoked by uid 500); 26 Aug 2010 16:29:15 -0000 Delivered-To: apmail-perl-modperl-archive@perl.apache.org Received: (qmail 27311 invoked by uid 500); 26 Aug 2010 16:29:14 -0000 Mailing-List: contact modperl-help@perl.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: List-Id: Delivered-To: mailing list modperl@perl.apache.org Received: (qmail 27304 invoked by uid 99); 26 Aug 2010 16:29:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:29:14 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of modperl.nunk@gmail.com designates 209.85.214.49 as permitted sender) Received: from [209.85.214.49] (HELO mail-bw0-f49.google.com) (209.85.214.49) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:29:09 +0000 Received: by bwz13 with SMTP id 13so1428222bwz.22 for ; Thu, 26 Aug 2010 09:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=JgSNubebViixUZ4dI2AVElQiB3FpWqvsdSvnRbOM3Po=; b=ciSx7pwgRhWyeAx/Sf400fvQXc4rzQJyf2j+jtuLX9QK2UxEMjqCN2Lh12DI6OsUsU Q7roQHzMFO7We2D1+YBnDrtE59l7r6hAJr3tJQn4YvkLDI7qc69YAlLQJSRjPd7eLoCC D9RLwcZtsK9gmYsbtkUkuKwOKNMpY0JzN+7Mo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Z9h4lQyKGKcj8RODMxHkGWglAlPknvgg8NmwFS/8FBF3XoSl2kqD2PvvPM6QHk59kf yxoHO6YYXD+S/8rl5CaQYcArcSmFVog5z6XxUf9zcfbEIXg2gdUqqN9ZNFsmoyuOStdt U6CESqn5ZMjNBulxSojUFd3GrN0yZ08RYd+6k= MIME-Version: 1.0 Received: by 10.204.102.2 with SMTP id e2mr6560888bko.112.1282840127625; Thu, 26 Aug 2010 09:28:47 -0700 (PDT) Received: by 10.204.63.140 with HTTP; Thu, 26 Aug 2010 09:28:47 -0700 (PDT) In-Reply-To: References: <4C74B44B.7080608@ice-sa.com> Date: Thu, 26 Aug 2010 17:28:47 +0100 Message-ID: Subject: Re: mp2] [mod_proxy] [filter] mod_proxy is not playing with my request filter. From: James Lee To: mod_perl list Content-Type: multipart/alternative; boundary=001636c5a2581c6ab1048ebc7eb5 --001636c5a2581c6ab1048ebc7eb5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I'm still having issues attempting to modify the body of an http request - this time using a FilterConnectionHandler. I've had to park this due to deadlines and write it using LWP in a PerlResponseHandler. This is far from ideal as some of the responses are upwards of 5MB, and what I'm trying to do does feel very doable in mod_perl= . What I really want is to be able to do is pass the request off to mod_proxy (which I assume is better setup for streaming responses through Apache ... = I may be wrong) and then through mod_deflate to compress the results. I've attached the code and configuration if anybody has a second and could shed some light over what I'm doing wrong. I'm basically trying to convert a GET request to 'http://hostname/echo' int= o a POST request and attach a payload, however as it hangs mod_proxy becomes largely academic. Again any help would be much appreciated. Thanks, James. PerlModule Example::Echo PerlModule Example::ConnectionFilter PerlInputFilterHandler Example::ConnectionFilter::forward_get_as_post SetHandler modperl PerlResponseHandler Example::Echo package Example::Echo; use strict; use warnings; use Apache2::Const -compile =3D> qw(OK); use Apache2::RequestIO; use Apache2::RequestRec; sub handler { my $r =3D shift; $r->content_type('text/plain'); $r->read(my $buffer, 1024); $r->print("received post data: '".$buffer."'"); return Apache2::Const::OK; } 1; package Example::ConnectionFilter; use strict; use warnings; use base qw/Apache2::Filter/; use Apache2::Connection; use Apache2::Const -compile =3D> qw(OK DECLINED); use Apache2::FilterRec; use Apache2::Log; use Apache2::RequestRec; use Apache2::RequestIO; use APR::Const -compile =3D> ':common'; use APR::Brigade; use APR::Bucket; use APR::BucketType; use APR::Error; sub forward_get_as_post :FilterConnectionHandler { my ($f, $bb, $mode, $block, $readbytes) =3D @_; my $ctx =3D $f->ctx || { 'state' =3D> 'waiting_for_request_line' }; warn "state =3D ".$ctx->{'state'}."\r\n"; # check whether we need to process this request. return Apache2::Const::DECLINED if ($ctx->{'state'} eq 'ignore'); # read into a tmp brigade. my $connection =3D $f->c; my $tmp_bb =3D APR::Brigade->new($connection->pool, $connection->bucket_alloc); my $rv =3D $f->next->get_brigade($tmp_bb, $mode, $block, $readbytes); return $rv unless $rv =3D=3D APR::Const::SUCCESS; while (!$tmp_bb->is_empty) { # pop buckets from this brigade. my $bucket =3D $tmp_bb->first; $bucket->remove(); if ($ctx->{'state'} eq 'waiting_for_request_line') { # assumes request line is first bucket. $bucket->read(my $request_line); my ($method, $uri, $version) =3D ($request_line =3D~ m|^(.*?) (= .*?) HTTP/(.*?)\r\n$|); if (defined ($method) and $method eq "GET" and $uri =3D~ m|^/echo|) { my $new_uri =3D 'POST '.$uri.' HTTP/'.$version."\r\n"; my $new_uri_bucket =3D APR::Bucket->new($connection->bucket_alloc, $new_uri); $bb->insert_tail($new_uri_bucket); my $bucket2 =3D APR::Bucket->new($connection->bucket_alloc, "Content-Type: application/x-www-form-urlencoded\r\n"); $bb->insert_tail($bucket2); my $bucket3 =3D APR::Bucket->new($connection->bucket_alloc, "Content-Length: 9\r\n"); $bb->insert_tail($bucket3); $ctx->{'state'} =3D 'waiting_for_end_of_headers'; } else { $bb->insert_tail($bucket); $ctx->{'state'} =3D 'ignore'; } } elsif ($ctx->{'state'} eq 'waiting_for_end_of_headers') { $bucket->read(my $header); warn "received header ... ".$header."\r\n"; if ($header =3D~ m|^\r\n$|) { warn "detected end_of_headers\r\n"; my $post_data =3D &get_post_data(); ### as soon as I add 'data=3Dtest' to this bucket the reque= st appears to hang. ############################################ my $end_of_headers_bucket =3D APR::Bucket->new($connection->bucket_alloc, "\r\ndata=3Dtest"); $bb->insert_tail($end_of_headers_bucket); $ctx->{'state'} =3D 'finished'; } else { $bb->insert_tail($bucket); } } } # set context. $f->ctx($ctx); return Apache2::Const::OK; } 1; On 25 August 2010 09:41, James Lee wrote: > Hi Andre, thanks for the response. > > I don't actually want to use a PerlResponseHandler, I was just using that > to make sure my filter did what I wanted it to do. > > I actually wanted the request filter to add the POST body expecting then > that mod_proxy would do the rest. I expected mod_proxy to kick in around > PerlTransHandler time but wasn't sure when my request filter got called. > Either way it wasn't working :) > > I may see whether I can get the connection filter to add the content I wa= s > just after a little clarification when filters were called in relation to > mod_proxy. > > Thanks again. > > > > On 25 August 2010 07:12, Andr=E9 Warnier wrote: > >> James Lee wrote: >> >>> Hi mod_perl community, this is my first post so be gentle with me. >>> >>> I'm trying to create something which translates a GET request into a PO= ST >>> (creating the body dynamically) and then hand off the POST to another >>> server. >>> >>> I have created a mod_perl connection filter to change the GET to a POST >>> in >>> the request line. It also adds Content-Length and Content-Type headers >>> and >>> this works fine. >>> >>> I then have a request filter which creates the POST body. This works wh= en >>> the request is handled by a PerlResponseHandler but I'm trying to offlo= ad >>> the request to mod_proxy as the responses can be quite large but I just >>> can't get it to play ball. I see the request filter being called but th= e >>> POST body never arrives at the target server. I suspect this is a timin= g >>> issue, ie: mod_proxy kicking in before my request filter but I'm not >>> certain. >>> >> >> Exactly. By the time your response handler is invoked, the time for >> mod_proxy is long past. >> >> One approach would be : have your PerlResponseHandler send the request >> directly to the back-end server, using LWP, read the response, and retur= n >> this response as the response for your response handler. In other words= , do >> yourself what mod_proxy would do. >> >> >> >>> Can anybody shed some light on this or correct my approach. I was going >>> to >>> add the POST body in the connection filter but it seemed cleaner/easier >>> to >>> do it in a request filter. >>> >> >> In the mod_perl documentation, there is somewhere a schema of the >> different Apache phases, and when the different handlers get invoked. I >> beliebe mod_proxygets invoked somewhere around the Fixup phase, so if yo= u >> want to beat it, you have to be earlier than that. >> >> >> >>> I've looked on the mailing list and found a few things that touch on th= is >>> ( >>> http://tech.groups.yahoo.com/group/modperl/message/54541) but nothing >>> that's >>> close enough to help. >>> >>> Config below ... I've not included perl code as this message is quite >>> long >>> anyway. Please let me know if it would be helpful. >>> Thanks in advance, James. >>> >>> >>> *** >>> httpd.conf extract: >>> >>> >>> PerlInputFilterHandler Sample::RequestTweaker::change_get_to_post >>> >>> >>> PerlInputFilterHandler Sample::RequestTweaker::inject_post_data >>> ProxyPass http://appserver/reports-engine >>> >>> >>> >> > --001636c5a2581c6ab1048ebc7eb5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I'm still having issues attempting to modify the body of an http reques= t - this time using a FilterConnectionHandler.

I've had to park = this due to deadlines and write it using LWP in a PerlResponseHandler. This= is far from ideal as some of the responses are upwards of 5MB, and what I&= #39;m trying to do does feel very doable in mod_perl.

What I really want is to be able to do is pass the request off to mod_p= roxy (which I assume is better setup for streaming responses through Apache= ... I may be wrong) and then through mod_deflate to compress the results.<= br>
I've attached the code and configuration if anybody has a second an= d could shed some light over what I'm doing wrong.

I'm basic= ally trying to convert a GET request to 'http://hostname/echo' into a POST request and attach a payload, = however as it hangs mod_proxy becomes largely academic.

Again any help would be much appreciated.
Thanks, James.


= PerlModule Example::Echo
PerlModule Example::ConnectionFilter

Per= lInputFilterHandler Example::ConnectionFilter::forward_get_as_post

<Location /echo>
=A0=A0=A0 SetHandler modperl
=A0=A0=A0 PerlRes= ponseHandler Example::Echo
</Location>



package Exam= ple::Echo;

use strict;
use warnings;

use Apache2::Const -c= ompile =3D> qw(OK);
use Apache2::RequestIO;
use Apache2::RequestRec;


sub handler<= br>{
=A0=A0=A0 my $r =3D shift;

=A0=A0=A0 $r->content_type(= 9;text/plain');
=A0=A0=A0 $r->read(my $buffer, 1024);
=A0=A0= =A0 $r->print("received post data: '".$buffer."'&= quot;);
=A0=A0=A0
=A0=A0=A0 return Apache2::Const::OK;
}


1;


package Example::ConnectionFilter;

use strict;
use warn= ings;

use base qw/Apache2::Filter/;

use Apache2::Connection;<= br>use Apache2::Const -compile =3D> qw(OK DECLINED);
use Apache2::FilterRec;
use Apache2::Log;
use Apache2::RequestRec;use Apache2::RequestIO;

use APR::Const -compile =3D> ':commo= n';
use APR::Brigade;
use APR::Bucket;
use APR::BucketType; use APR::Error;


sub forward_get_as_post :FilterConnectionHandler=
{
=A0=A0=A0 my ($f, $bb, $mode, $block, $readbytes) =3D @_;
=A0= =A0=A0 my $ctx =3D $f->ctx || { 'state' =3D> 'waiting_for= _request_line' };
=A0=A0=A0
=A0=A0=A0 warn "state =3D ".$ctx->{'state= 9;}."\r\n";

=A0=A0=A0 # check whether we need to process t= his request.
=A0=A0=A0 return Apache2::Const::DECLINED if ($ctx->{= 9;state'} eq 'ignore');
=A0=A0=A0
=A0=A0=A0 # read into a tmp brigade.
=A0=A0=A0 my $connect= ion =3D $f->c;
=A0=A0=A0 my $tmp_bb =3D APR::Brigade->new($connect= ion->pool, $connection->bucket_alloc);
=A0=A0=A0 my $rv =3D $f->= ;next->get_brigade($tmp_bb, $mode, $block, $readbytes);
=A0=A0=A0
=A0=A0=A0 return $rv unless $rv =3D=3D APR::Const::SUCCESS;=A0=A0=A0
=A0=A0=A0 while (!$tmp_bb->is_empty)
=A0=A0=A0 {
= =A0=A0=A0 =A0=A0=A0 # pop buckets from this brigade.
=A0=A0=A0 =A0=A0=A0= my $bucket =3D $tmp_bb->first;
=A0=A0=A0 =A0=A0=A0 $bucket->remov= e();

=A0=A0=A0 =A0=A0=A0 if ($ctx->{'state'} eq 'waiting_for_= request_line')
=A0=A0=A0 =A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 =A0=A0= =A0 # assumes request line is first bucket.
=A0=A0=A0 =A0=A0=A0 =A0=A0= =A0 $bucket->read(my $request_line);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 my= ($method, $uri, $version) =3D ($request_line =3D~ m|^(.*?) (.*?) HTTP/(.*?= )\r\n$|);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (defined= ($method) and $method eq "GET" and $uri =3D~ m|^/echo|)
=A0= =A0=A0 =A0=A0=A0 =A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 my = $new_uri =3D 'POST '.$uri.' HTTP/'.$version."\r\n"= ;;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 my $new_uri_bucket =3D APR::Bucket-= >new($connection->bucket_alloc, $new_uri);
=A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 $bb->insert= _tail($new_uri_bucket);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0= =A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 my $bucket2 =3D APR::Bucket->new($c= onnection->bucket_alloc, "Content-Type: application/x-www-form-urle= ncoded\r\n");
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 $bb->insert_tail($bucket2);
= =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 = =A0=A0=A0 my $bucket3 =3D APR::Bucket->new($connection->bucket_alloc,= "Content-Length: 9\r\n");
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 $bb->insert_tail($bucket3);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 = =A0=A0=A0 $ctx->{'state'} =3D 'waiting_for_end_of_headers= 9;;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 els= e
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 $bb->insert_tail($bucket);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0= =A0=A0 $ctx->{'state'} =3D 'ignore';
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0= =A0=A0 elsif ($ctx->{'state'} eq 'waiting_for_end_of_headers= ')
=A0=A0=A0 =A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 $bucket-&g= t;read(my $header);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 warn "received he= ader ... ".$header."\r\n";

=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if ($header =3D~ m|^\r\n$|)
=A0=A0=A0 = =A0=A0=A0 =A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 warn "= ;detected end_of_headers\r\n";
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0= =A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 my $post_data =3D &get_= post_data();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 = =A0=A0=A0 ### as soon as I add 'data=3Dtest' to this bucket the req= uest appears to hang.
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 ##########= ##################################
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0= =A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 my $end_of_headers_bucket =3D APR::= Bucket->new($connection->bucket_alloc, "\r\ndata=3Dtest");<= br>=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0
=A0=A0=A0 =A0=A0=A0 =A0=A0= =A0 =A0=A0=A0 $bb->insert_tail($end_of_headers_bucket);
=A0=A0=A0 =A0= =A0=A0 =A0=A0=A0 =A0=A0=A0 $ctx->{'state'} =3D 'finished'= ;;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 else
= =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 = $bb->insert_tail($bucket);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0= =A0 =A0=A0=A0 }
=A0=A0=A0 }

=A0=A0=A0 # set context.=A0=A0=A0 =A0=A0=A0 $f->ctx($ctx);

=A0=A0=A0 return Apache2::Const::OK; }


1;







O= n 25 August 2010 09:41, James Lee <modperl.nunk@gmail.com> wrote:
Hi Andre, thanks for the response.

I don't actually want to use = a PerlResponseHandler, I was just using that to make sure my filter did wha= t I wanted it to do.

I actually wanted the request filter to add the= POST body expecting then that mod_proxy would do the rest. I expected mod_= proxy to kick in around PerlTransHandler time but wasn't sure when my r= equest filter got called. Either way it wasn't working :)

I may see whether I can get the connection filter to add the content I = was just after a little clarification when filters were called in relation = to mod_proxy.

Thanks again.



On 25 August 2010 07:12, Andr=E9 Warnier <aw@ice-sa.com> wrote:<= br>
James Lee wrote:
Hi mod_perl community, this is my first post so be gentle with me.

I'm trying to create something which translates a GET request into a PO= ST
(creating the body dynamically) and then hand off the POST to another
server.

I have created a mod_perl connection filter to change the GET to a POST in<= br> the request line. It also adds Content-Length and Content-Type headers and<= br> this works fine.

I then have a request filter which creates the POST body. This works when the request is handled by a PerlResponseHandler but I'm trying to offlo= ad
the request to mod_proxy as the responses can be quite large but I just
can't get it to play ball. I see the request filter being called but th= e
POST body never arrives at the target server. I suspect this is a timing issue, ie: mod_proxy kicking in before my request filter but I'm not certain.

Exactly. By the time your response handler is invoked, the time for mod_pro= xy is long past.

One approach would be : have your PerlResponseHandler send the request dire= ctly to the back-end server, using LWP, read the response, and return this = response as the response for your response handler. =A0In other words, do y= ourself what mod_proxy would do.



Can anybody shed some light on this or correct my approach. I was going to<= br> add the POST body in the connection filter but it seemed cleaner/easier to<= br> do it in a request filter.

In the mod_perl documentation, there is somewhere a schema of the different= Apache phases, and when the different handlers get invoked. =A0I beliebe m= od_proxygets invoked somewhere around the Fixup phase, so if you want to be= at it, you have to be earlier than that.



I've looked on the mailing list and found a few things that touch on th= is (
http://tech.groups.yahoo.com/group/modperl/message/54541) b= ut nothing that's
close enough to help.

Config below ... I've not included perl code as this message is quite l= ong
anyway. Please let me know if it would be helpful.
Thanks in advance, James.


***
httpd.conf extract:


PerlInputFilterHandler Sample::RequestTweaker::change_get_to_post

<Location /reports>
=A0 PerlInputFilterHandler Sample::RequestTweaker::inject_post_data
=A0 ProxyPass http://appserver/reports-engine
</Location>




--001636c5a2581c6ab1048ebc7eb5--