Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (athena.apache.org: domain of claus.ibsen@gmail.com
 designates 209.85.220.223 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=WsZKR4Z3VjWldZNkdR1tlQCV4YilkLqZR/VonawUyZy/7DCBD/TrzPVoFhLTgyFLVR
         oCCr7Yx+1z6C9eB5fCp2aFAPaY7bn4ZbUiAbmYSCrIibXRXFvUD9e57NIaPAdtbAqOYs
         90ZHVVq+YRefg1ad3E11svF6F3PPZ5gtGDXFI=
MIME-Version: 1.0
In-Reply-To: <25890696.post@talk.nabble.com>
References: <25826380.post@talk.nabble.com> <25828828.post@talk.nabble.com>
	<25890696.post@talk.nabble.com>
From: Claus Ibsen <claus.ibsen@gmail.com>
Date: Wed, 14 Oct 2009 15:07:41 +0200
Message-ID: <5380c69c0910140607w3fb011e7xca420a20ac2ad504@mail.gmail.com>
Subject: Re: handling large files
To: users@camel.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi

How do you want to split the file?
Is there a special character that denotes a new "record"

Using java.util.Scanner is great as it can do streaming. And also what
Camel can do if you for example want to split by new line etc.


On Wed, Oct 14, 2009 at 3:01 PM, mcarson <mcarson@amsa.com> wrote:
>
> I have to solve this exact same problem! =A0I am also trying to:
> =A0 -- read an extremely large file from a file system
> =A0 -- doing a split on that file
> =A0 -- send each split record to a JMS queue
>
> However I am relatively new to Camel.. I do not understand how the stream
> component can help specify how much data to read before the split occurs.=
..
> and when I tried to stream the file I got exceptions in my split because
> when streaming the document it is no longer well formed XML?
>
> Can someone help explain in more detail how using the stream component
> solves this problem?
>
>
> This is the camel context I was using successfully to perform these tasks
> for SMALL files:
>
> <?xml version=3D"1.0" encoding=3D"UTF-8"?>
> <beans xmlns=3D"http://www.springframework.org/schema/beans"
> =A0 =A0 =A0 xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"
> =A0 =A0 =A0 xsi:schemaLocation=3D"
> =A0 =A0 =A0 http://www.springframework.org/schema/beans
> =A0 =A0 =A0 http://www.springframework.org/schema/beans/spring-beans-2.5.=
xsd
> =A0 =A0 =A0 http://camel.apache.org/schema/spring
> =A0 =A0 =A0 http://camel.apache.org/schema/spring/camel-spring.xsd">
>
> =A0<bean id=3D"jms" class=3D"org.apache.camel.component.jms.JmsComponent"=
>
> =A0 =A0<property name=3D"connectionFactory">
> =A0 =A0 =A0 =A0<bean class=3D"org.apache.activemq.ActiveMQConnectionFacto=
ry">
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<property name=3D"brokerURL" value=3D"tcp:=
//localhost:61616"/>
> =A0 =A0 =A0 =A0</bean>
> =A0 =A0</property>
> =A0</bean>
>
> =A0<camelContext xmlns=3D"http://camel.apache.org/schema/spring"
> =A0 =A0 =A0 =A0xmlns:s=3D"http://schemas.xmlsoap.org/soap/envelope/">
> =A0 =A0<package>camel.myprototype</package>
>
> =A0 =A0<route>
> =A0 =A0 =A0<from uri=3D"file:src/inbox?delete=3Dtrue"/>
> =A0 =A0 =A0<split>
> =A0 =A0 =A0 =A0<xpath>/s:Envelope/s:Body/person</xpath>
> =A0 =A0 =A0 =A0<to uri=3D"jms:queue:Q_People"/>
> =A0 =A0 =A0</split>
> =A0 =A0</route>
> =A0</camelContext>
> --
> View this message in context: http://www.nabble.com/handling-large-files-=
tp25826380p25890696.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--=20
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus