Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Subject: Re: Jetty consumer responds with a 500 error java.io.IOException:
 Response header too large
To: users@camel.apache.org
References: <CAFjL1LyXw_MXSrSLWZPQ5To6MLM-GAXU01df666BA01KpHY+GA@mail.gmail.com>
 <505217981.1571878.1513700823494@mail.yahoo.com>
 <CAFjL1Lw9pvhA0=HTA8ATEB+ZkM7AZaXGzKzR+zgO_PX78pRU6g@mail.gmail.com>
From: Roman Vottner <rovo@gmx.at>
Message-ID: <129f8a1f-3ee4-e561-36b5-bc2b8541a229@gmx.at>
Date: Wed, 20 Dec 2017 15:04:40 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.0
MIME-Version: 1.0
In-Reply-To: <CAFjL1Lw9pvhA0=HTA8ATEB+ZkM7AZaXGzKzR+zgO_PX78pRU6g@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Wed, 20 Dec 2017 14:04:59 -0000

Do you somehow store the received file into camel headers? Camel will 
return most headers to the invoker and usually HTTP headers > 8kb will 
produce that kind of exception.

You might want to introduce a customized HttpHeaderFilterStrategy 
similar to the sample below to avoid returning certain headers.

import org.apache.camel.Exchange;
import org.apache.camel.http.common.HttpHeaderFilterStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.lang.invoke.MethodHandles;
import java.util.Arrays;
import java.util.List;

public class CustomHeaderFilterStrategy extends HttpHeaderFilterStrategy {

   private final static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
   private boolean debugLog = false;

   protected List<String> queryParamsToFilter = Arrays.asList("appKey", "limit", "offset", "filter");
   protected List<String> otherParamsToFilter =
       Arrays.asList("breadcrumbId", "User-Agent", "NewRelicID", "NewRelicTransaction",
                     "Transfer-Encoding", "Authorization", "Accept",
                     "HttpResponseContent", "TLS_PROTOCOL_VERSION", "TLS_CIPHER_SUITE");

   public CustomHeaderFilterStrategy() {

   }

   public CustomHeaderFilterStrategy(boolean debugLog) {
     this.debugLog = debugLog;
   }

   // outgoing headers
   @Override
   public boolean applyFilterToCamelHeaders(String headerName, Object headerValue,
                                            Exchange exchange) {

     // do not copy Camel specific headers from the Camel exchange to the out headers
     if (super.applyFilterToCamelHeaders(headerName, headerValue, exchange)) {
       if (debugLog) {
         LOG.debug("Filtered header {} through HTTP header filter", headerName);
       }
       return true;
     }

     // do not copy SAMPLE_* headers from the Camel exchange to the out headers
     if (headerName.startsWith("SAMPLE_")) {
       if (debugLog) {
         LOG.debug("Filtering header {}", headerName);
       }
       return true;
     }
     if (headerName.startsWith("JMS")) {
       if (debugLog) {
         LOG.debug("Filtering header {}", headerName);
       }
       return true;
     }
     if (queryParamsToFilter.contains(headerName)) {
       if (debugLog) {
         LOG.debug("Filtering header {}", headerName);
       }
       return true;
     }
     if (otherParamsToFilter.contains(headerName)) {
       if (debugLog) {
         LOG.debug("Filtering header: {}", headerName);
       }
       return true;
     }

     if (debugLog) {
       LOG.debug("Outgoing header passed: {} - value: {}", headerName, headerValue);
     }
     return false;
   }

   // incoming headers
   @Override
   public boolean applyFilterToExternalHeaders(String headerName, Object headerValue,
                                               Exchange exchange) {
     // allow all headers to be copied from the request to camel headers
     return false;
   }
}

You can utilize this custom header filter by appending 
?headerFilterStrategy=#refNameOfFilterStrategy to the route specifying 
the Jetty consumer, where "refNameOfFilterStrategy" is the name you 
register the class above with Camel (or Spring).

This thread 
(https://stackoverflow.com/questions/17550471/camel-jetty-component-custom-header-filter-strategy) 
also states that you may customize the DefaultHttpBinding, though I'd 
try the primer solution first.

HTH,

Roman

Am 20.12.2017 um 09:42 schrieb Laurentiu Trica:
> Camel version: 2.16.3
>
> Thanks!
>
> On Tue, Dec 19, 2017 at 6:27 PM, Andrea Cosentino <
> ancosen1985@yahoo.com.invalid> wrote:
>
>> What version of camel are you using?
>>
>> Inviato da Yahoo Mail su Android
>>
>>    Il mar, 19 dic, 2017 alle 16:45, Laurentiu Trica<laurentiu.trica@
>> moredevs.ro> ha scritto:   Hello,
>>
>> I have a problem with the Jetty consumer. I receive a file with an attached
>> file (Multi-Part Form), but if the file is bigger than a few KB, I get the
>> bellow Stack trace. If the file is smaller, everything works fine.
>>
>> The strange part is that for a size of 80KB I still receive the file in the
>> route, but the response to the HTTP client is:
>> HTTP/1.1 500 Server Error
>> Connection: close
>> Server: Jetty(9.2.14.v20151106)
>>
>> If the file's size is, let's say, 2 MB, then the route doesn't get the file
>> anymore and the response to the client is the same.
>>
>> I tried to set the request/response buffers to bigger values, but it didn't
>> help:
>> <from uri="jetty:
>> http://0.0.0.0:9086/Test?responseHeaderSize=32768000&
>> responseBufferSize=32768000&requestBufferSize=32768000&
>> requestHeaderSize=32768000
>> "/>
>>
>> Any ideas? Can you please help?
>>
>> *Stack trace:*
>> 2017-12-19 16:33:02,802 | WARN  | tp466415455-3362 | ServletHandler
>>            | 119 - org.eclipse.jetty.util - 9.2.14.v20151106 | /Test
>> java.io.IOException: Response header too large
>> at
>> org.eclipse.jetty.http.HttpGenerator.generateResponse(
>> HttpGenerator.java:400)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpConnection$SendCallback.
>> process(HttpConnection.java:655)[116:org.eclipse.jetty.
>> server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.util.IteratingCallback.processing(
>> IteratingCallback.java:246)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.util.IteratingCallback.iterate(
>> IteratingCallback.java:208)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)[116:
>> org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpChannel.sendResponse(
>> HttpChannel.java:763)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpChannel.write(HttpChannel.
>> java:801)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:147)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:140)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.
>> java:242)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.apache.camel.util.IOHelper.copy(IOHelper.java:
>> 201)[59:org.apache.camel.camel-core:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.copyStream(
>> DefaultHttpBinding.java:369)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.doWriteDirectResponse(
>> DefaultHttpBinding.java:433)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.doWriteResponse(
>> DefaultHttpBinding.java:332)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.writeResponse(
>> DefaultHttpBinding.java:264)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.component.jetty.CamelContinuationServlet.service(
>> CamelContinuationServlet.java:227)[155:org.apache.camel.
>> camel-jetty-common:2.16.3]
>> at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)[54:
>> javax.servlet-api:3.1.0]
>> at
>> org.eclipse.jetty.servlet.ServletHolder.handle(
>> ServletHolder.java:812)[117:org.eclipse.jetty.servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1669)[117:org.eclipse.jetty.
>> servlet:9.2.14.v20151106]
>> at
>> org.apache.camel.component.jetty.CamelFilterWrapper.
>> doFilter(CamelFilterWrapper.java:45)[155:org.apache.camel.
>> camel-jetty-common:2.16.3]
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1652)[117:org.eclipse.jetty.
>> servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(
>> ServletHandler.java:585)[117:org.eclipse.jetty.servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1127)[116:org.eclipse.jetty.
>> server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(
>> ServletHandler.java:515)[117:org.eclipse.jetty.servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1061)[116:org.eclipse.jetty.
>> server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:141)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:97)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.Server.handleAsync(Server.
>> java:549)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318)[116:org.
>> eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:257)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(
>> AbstractConnection.java:544)[108:org.eclipse.jetty.io:9
>> .2.14.v20151106]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:635)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at java.lang.Thread.run(Thread.java:748)[:1.7.0_151]
>> Caused by: java.nio.BufferOverflowException
>> at java.nio.Buffer.nextPutIndex(Buffer.java:513)[:1.7.0_151]
>> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:163)[:1.7.0_151]
>> at
>> org.eclipse.jetty.http.HttpGenerator.putSanitisedValue(
>> HttpGenerator.java:1057)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:1079)[107:
>> org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.http.HttpGenerator.generateHeaders(
>> HttpGenerator.java:703)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.http.HttpGenerator.generateResponse(
>> HttpGenerator.java:385)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> ... 33 more
>> 2017-12-19 16:33:02,804 | WARN  | tp466415455-3362 | HttpChannel
>>            | 119 - org.eclipse.jetty.util - 9.2.14.v20151106 |
>> /LogsolToBBATest
>> java.io.IOException: Response header too large
>> at
>> org.eclipse.jetty.http.HttpGenerator.generateResponse(
>> HttpGenerator.java:400)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpConnection$SendCallback.
>> process(HttpConnection.java:655)[116:org.eclipse.jetty.
>> server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.util.IteratingCallback.processing(
>> IteratingCallback.java:246)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.util.IteratingCallback.iterate(
>> IteratingCallback.java:208)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)[116:
>> org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpChannel.sendResponse(
>> HttpChannel.java:763)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpChannel.write(HttpChannel.
>> java:801)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:147)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:140)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.
>> java:242)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.apache.camel.util.IOHelper.copy(IOHelper.java:
>> 201)[59:org.apache.camel.camel-core:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.copyStream(
>> DefaultHttpBinding.java:369)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.doWriteDirectResponse(
>> DefaultHttpBinding.java:433)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.doWriteResponse(
>> DefaultHttpBinding.java:332)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.http.common.DefaultHttpBinding.writeResponse(
>> DefaultHttpBinding.java:264)[143:org.apache.camel.camel-
>> http-common:2.16.3]
>> at
>> org.apache.camel.component.jetty.CamelContinuationServlet.service(
>> CamelContinuationServlet.java:227)[155:org.apache.camel.
>> camel-jetty-common:2.16.3]
>> at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)[54:
>> javax.servlet-api:3.1.0]
>> at
>> org.eclipse.jetty.servlet.ServletHolder.handle(
>> ServletHolder.java:812)[117:org.eclipse.jetty.servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1669)[117:org.eclipse.jetty.
>> servlet:9.2.14.v20151106]
>> at
>> org.apache.camel.component.jetty.CamelFilterWrapper.
>> doFilter(CamelFilterWrapper.java:45)[155:org.apache.camel.
>> camel-jetty-common:2.16.3]
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
>> doFilter(ServletHandler.java:1652)[117:org.eclipse.jetty.
>> servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(
>> ServletHandler.java:585)[117:org.eclipse.jetty.servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1127)[116:org.eclipse.jetty.
>> server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(
>> ServletHandler.java:515)[117:org.eclipse.jetty.servlet:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1061)[116:org.eclipse.jetty.
>> server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:141)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:97)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.Server.handleAsync(Server.
>> java:549)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318)[116:org.
>> eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:257)[116:org.eclipse.jetty.server:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(
>> AbstractConnection.java:544)[108:org.eclipse.jetty.io:9
>> .2.14.v20151106]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:635)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:555)[119:org.eclipse.jetty.util:9.2.14.v20151106]
>> at java.lang.Thread.run(Thread.java:748)[:1.7.0_151]
>> Caused by: java.nio.BufferOverflowException
>> at java.nio.Buffer.nextPutIndex(Buffer.java:513)[:1.7.0_151]
>> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:163)[:1.7.0_151]
>> at
>> org.eclipse.jetty.http.HttpGenerator.putSanitisedValue(
>> HttpGenerator.java:1057)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.http.HttpGenerator.putTo(HttpGenerator.java:1079)[107:
>> org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.http.HttpGenerator.generateHeaders(
>> HttpGenerator.java:703)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> at
>> org.eclipse.jetty.http.HttpGenerator.generateResponse(
>> HttpGenerator.java:385)[107:org.eclipse.jetty.http:9.2.14.v20151106]
>> ... 33 more
>> 2017-12-19 16:33:02,806 | WARN  | tp466415455-3362 | HttpChannel
>>            | 119 - org.eclipse.jetty.util - 9.2.14.v20151106 | Could not
>> send response error 500: java.io.IOException: Response header too large
>>
>>
>> Regards,
>> Laurentiu
>>
>>