camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oliver Geisser <oliver.geis...@gmail.com>
Subject Re: Splitter and combining messages / exchanges
Date Sat, 01 Dec 2012 16:31:11 GMT
Hi Raul,

thanks a lot. I think this will work best.
It solves the problem where to store the first line in the best way
(property)
and it also supports streaming mode.

I was misguided in my assumption that I can not access the message
before the splitter because of the size. But of course I can use plain old
Java to read just the first line.

Thanks again.

Oliver


2012/12/1 Raul Kripalani <raul@evosent.com>

> I would suggest the following pipeline:
>
> - Before the splitter, read the first line and store it in a property
> - Run the splitter, and for every element concatenate the above property
> with the incoming line
> - Discard the first Exchange, by using a filter EIP to only allow messages
> whose header CamelSplitIndex is greater than 1 or 2 (can't remember if the
> index starts at 0 or 1).
>
> Does this approach help? Just to clarify, you wouldn't be using the
> aggregator EIP at all.
>
> Regards,
> Raúl.
>
> Sent from a mobile device
> On 30 Nov 2012 22:33, "Oliver Geisser" <oliver.geisser@gmail.com> wrote:
>
> > Hi Claus,
> >
> > thanks for your help. I've checked the Composed Message Processor EIP.
> >
> > Maybe I'm missing something but as far as I can see the EIP will not work
> > for my use case.
> >
> > Let's recap my problem with the input file
> >
> > Hello
> > A
> > B
> > C
> > ...
> >
> > Because it's a very large file it will be splitted in streaming mode.
> > Therfor
> > every line becomes a separate Message.
> >
> > Using the Aggregator the "Hello" Message will be stored in the
> > AggregationRepository.
> > The the second message ("A") will be aggregated with "Hello" to "Hello
> A".
> > And now the important point: the "completion predicate" is now true and
> the
> > meesage is released from the Aggregator, e.g.
> > AggregateProcessor.onCompletion will
> > be called and the "Hello" Exchange is removed from the
> > AggregationRepository.
> >
> > The Exchange needs to be released (completed) at this point because it
> is a
> > big file and
> > we need to operate in streaming mode.
> >
> > Now the third message ("B") comes in. Because the "Hello" Exchange was
> > removed
> > we can not aggregate the Exchanges to "Hello B" anymore.
> >
> > So the question remains: what is the best way to solve the given problem
> > (see below) with Camel?
> >
> > One idea I had was to mimic the Aggregate Repository but without removing
> > the "Hello" Exchange
> > after combining it with the "A", "B", "C", etc. messages.
> >
> > But is this really the best solution for my problem?
> >
> > What I need is access to a storage location which is scoped to the
> > "Splitter input Exchange"
> > during processing the "splitted" Exchanges. For example: if it would be
> > possible to access
> > the "input" Exchange during processing the "splitted" Exchanges I would
> set
> > a property
> > on the "input" Exchange during processing the first line and reading it
> > afterwards during
> > processing the remaining lines.
> >
> > Is there any other storage location besides an Exchange property which is
> > scoped to the
> > Exchange and is accessible from the "splitted" Exchanges?
> >
> > Greetings
> > Oliver
> >
> >
> >
> > 2012/11/30 Claus Ibsen <claus.ibsen@gmail.com>
> >
> > > Hi
> > >
> > > See this EIP
> > > http://camel.apache.org/composed-message-processor.html
> > >
> > > On Thu, Nov 29, 2012 at 11:27 PM, Oliver Geisser
> > > <oliver.geisser@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > I am using a splitter to handle a big txt file (streaming mode). I
> need
> > > to
> > > > combine the first line
> > > > with all the other lines and store each result into a database.
> > > >
> > > > What is a good Camel solution for this?
> > > >
> > > > Example input file:
> > > >
> > > > Hello
> > > > A
> > > > B
> > > > C
> > > > ...
> > > >
> > > >
> > > > This needs to be combined into:
> > > >
> > > > Hello A
> > > > Hello B
> > > > Hello C
> > > > ...
> > > >
> > > > Every result is stored into a database.
> > > > It's important to use streaming because of the size of the input
> file.
> > > >
> > > >
> > > > My first idea was to use a Splitter and then store the first line
> into
> > an
> > > > Exchange property.
> > > > But this does not work because the Splitter creates new Exchanges for
> > > every
> > > > line.
> > > >
> > > > So how do I combine the first line with all the other lines?
> > > >
> > > > Thanks
> > > > Oliver
> > > >
> > > > --
> > > > og
> > >
> > >
> > >
> > > --
> > > Claus Ibsen
> > > -----------------
> > > Red Hat, Inc.
> > > FuseSource is now part of Red Hat
> > > Email: cibsen@redhat.com
> > > Web: http://fusesource.com
> > > Twitter: davsclaus
> > > Blog: http://davsclaus.com
> > > Author of Camel in Action: http://www.manning.com/ibsen
> > >
> >
> >
> >
> > --
> > og
> >
>



-- 
og

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message