camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From davscl...@apache.org
Subject [camel] branch master updated: CAMEL-13108: camel-core - Move dataset into camel-dataset component.
Date Sun, 07 Apr 2019 07:39:53 GMT
This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/camel.git


The following commit(s) were added to refs/heads/master by this push:
     new 2ec6ff9  CAMEL-13108: camel-core - Move dataset into camel-dataset component.
2ec6ff9 is described below

commit 2ec6ff90eb58e33ae2ac31de030cf24328fa8b8e
Author: Claus Ibsen <claus.ibsen@gmail.com>
AuthorDate: Sun Apr 7 09:39:40 2019 +0200

    CAMEL-13108: camel-core - Move dataset into camel-dataset component.
---
 docs/components/modules/ROOT/nav.adoc              |   2 +
 .../modules/ROOT/pages/dataset-component.adoc      | 248 +++++++++++++++++++++
 .../modules/ROOT/pages/dataset-test-component.adoc | 152 +++++++++++++
 .../camel-spring-boot-dependencies/pom.xml         |  10 +
 4 files changed, 412 insertions(+)

diff --git a/docs/components/modules/ROOT/nav.adoc b/docs/components/modules/ROOT/nav.adoc
index c91fd7c..5fe4287 100644
--- a/docs/components/modules/ROOT/nav.adoc
+++ b/docs/components/modules/ROOT/nav.adoc
@@ -74,6 +74,8 @@
 * xref:cxf-component.adoc[CXF Component]
 * xref:cxfrs-component.adoc[CXF-RS Component]
 * xref:dataformat-component.adoc[Data Format Component]
+* xref:dataset-component.adoc[Dataset Component]
+* xref:dataset-test-component.adoc[DataSet Test Component]
 * xref:digitalocean-component.adoc[DigitalOcean Component]
 * xref:direct-component.adoc[Direct Component]
 * xref:direct-vm-component.adoc[Direct VM Component]
diff --git a/docs/components/modules/ROOT/pages/dataset-component.adoc b/docs/components/modules/ROOT/pages/dataset-component.adoc
new file mode 100644
index 0000000..4d12d6f
--- /dev/null
+++ b/docs/components/modules/ROOT/pages/dataset-component.adoc
@@ -0,0 +1,248 @@
+[[dataset-component]]
+== Dataset Component
+
+*Available as of Camel version 1.3*
+
+Testing of distributed and asynchronous processing is
+notoriously difficult. The <<mock-component,Mock>>, <<mock-component,Test>>
+and <<dataset-component,DataSet>> endpoints work great with the
+Camel Testing Framework to simplify your unit and
+integration testing using
+link:enterprise-integration-patterns.html[Enterprise Integration
+Patterns] and Camel's large range of Components
+together with the powerful Bean Integration.
+
+The DataSet component provides a mechanism to easily perform load & soak
+testing of your system. It works by allowing you to create
+http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/component/dataset/DataSet.html[DataSet
+instances] both as a source of messages and as a way to assert that the
+data set is received.
+
+Camel will use the <<log-component,throughput logger>> when sending
+dataset's.
+
+=== URI format
+
+[source]
+----
+dataset:name[?options]
+----
+
+Where *name* is used to find the
+http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/component/dataset/DataSet.html[DataSet
+instance] in the Registry
+
+Camel ships with a support implementation of
+`org.apache.camel.component.dataset.DataSet`, the
+`org.apache.camel.component.dataset.DataSetSupport` class, that can be
+used as a base for implementing your own DataSet. Camel also ships with
+some implementations that can be used for testing:
+ `org.apache.camel.component.dataset.SimpleDataSet`, `org.apache.camel.component.dataset.ListDataSet`
+and `org.apache.camel.component.dataset.FileDataSet`, all of which
+extend `DataSetSupport`.
+
+=== Options
+
+
+// component options: START
+The Dataset component has no options.
+// component options: END
+
+
+
+// endpoint options: START
+The Dataset endpoint is configured using URI syntax:
+
+----
+dataset:name
+----
+
+with the following path and query parameters:
+
+==== Path Parameters (1 parameters):
+
+
+[width="100%",cols="2,5,^1,2",options="header"]
+|===
+| Name | Description | Default | Type
+| *name* | *Required* Name of DataSet to lookup in the registry |  | DataSet
+|===
+
+
+==== Query Parameters (16 parameters):
+
+
+[width="100%",cols="2,5,^1,2",options="header"]
+|===
+| Name | Description | Default | Type
+| *dataSetIndex* (common) | Controls the behaviour of the CamelDataSetIndex header. For Consumers:
- off = the header will not be set - strict/lenient = the header will be set For Producers:
- off = the header value will not be verified, and will not be set if it is not present =
strict = the header value must be present and will be verified = lenient = the header value
will be verified if it is present, and will be set if it is not present | lenient | String
+| *initialDelay* (consumer) | Time period in millis to wait before starting sending messages.
| 1000 | long
+| *minRate* (consumer) | Wait until the DataSet contains at least this number of messages
| 0 | int
+| *preloadSize* (consumer) | Sets how many messages should be preloaded (sent) before the
route completes its initialization | 0 | long
+| *produceDelay* (consumer) | Allows a delay to be specified which causes a delay when a
message is sent by the consumer (to simulate slow processing) | 3 | long
+| *consumeDelay* (producer) | Allows a delay to be specified which causes a delay when a
message is consumed by the producer (to simulate slow processing) | 0 | long
+| *assertPeriod* (producer) | Sets a grace period after which the mock endpoint will re-assert
to ensure the preliminary assertion is still valid. This is used for example to assert that
exactly a number of messages arrives. For example if expectedMessageCount(int) was set to
5, then the assertion is satisfied when 5 or more message arrives. To ensure that exactly
5 messages arrives, then you would need to wait a little period to ensure no further message
arrives. This is what you can us [...]
+| *expectedCount* (producer) | Specifies the expected number of message exchanges that should
be received by this endpoint. Beware: If you want to expect that 0 messages, then take extra
care, as 0 matches when the tests starts, so you need to set a assert period time to let the
test run for a while to make sure there are still no messages arrived; for that use setAssertPeriod(long).
An alternative is to use NotifyBuilder, and use the notifier to know when Camel is done routing
some mess [...]
+| *reportGroup* (producer) | A number that is used to turn on throughput logging based on
groups of the size. |  | int
+| *resultMinimumWaitTime* (producer) | Sets the minimum expected amount of time (in millis)
the assertIsSatisfied() will wait on a latch until it is satisfied | 0 | long
+| *resultWaitTime* (producer) | Sets the maximum amount of time (in millis) the assertIsSatisfied()
will wait on a latch until it is satisfied | 0 | long
+| *retainFirst* (producer) | Specifies to only retain the first n'th number of received Exchanges.
This is used when testing with big data, to reduce memory consumption by not storing copies
of every Exchange this mock endpoint receives. Important: When using this limitation, then
the getReceivedCounter() will still return the actual number of received Exchanges. For example
if we have received 5000 Exchanges, and have configured to only retain the first 10 Exchanges,
then the getReceive [...]
+| *retainLast* (producer) | Specifies to only retain the last n'th number of received Exchanges.
This is used when testing with big data, to reduce memory consumption by not storing copies
of every Exchange this mock endpoint receives. Important: When using this limitation, then
the getReceivedCounter() will still return the actual number of received Exchanges. For example
if we have received 5000 Exchanges, and have configured to only retain the last 20 Exchanges,
then the getReceivedCo [...]
+| *sleepForEmptyTest* (producer) | Allows a sleep to be specified to wait to check that this
endpoint really is empty when expectedMessageCount(int) is called with zero | 0 | long
+| *copyOnExchange* (producer) | Sets whether to make a deep copy of the incoming Exchange
when received at this mock endpoint. Is by default true. | true | boolean
+| *synchronous* (advanced) | Sets whether synchronous processing should be strictly used,
or Camel is allowed to use asynchronous processing (if supported). | false | boolean
+|===
+// endpoint options: END
+// spring-boot-auto-configure options: START
+=== Spring Boot Auto-Configuration
+
+When using Spring Boot make sure to use the following Maven dependency to have support for
auto configuration:
+
+[source,xml]
+----
+<dependency>
+  <groupId>org.apache.camel</groupId>
+  <artifactId>camel-dataset-starter</artifactId>
+  <version>x.x.x</version>
+  <!-- use the same version as your Camel core version -->
+</dependency>
+----
+
+
+The component supports 2 options, which are listed below.
+
+
+
+[width="100%",cols="2,5,^1,2",options="header"]
+|===
+| Name | Description | Default | Type
+| *camel.component.dataset.enabled* | Whether to enable auto configuration of the dataset
component. This is enabled by default. |  | Boolean
+| *camel.component.dataset.resolve-property-placeholders* | Whether the component should
resolve property placeholders on itself when starting. Only properties which are of String
type can use property placeholders. | true | Boolean
+|===
+// spring-boot-auto-configure options: END
+
+You can append query options to the URI in the following format,
+`?option=value&option=value&...`
+
+=== Configuring DataSet
+
+Camel will lookup in the Registry for a bean
+implementing the DataSet interface. So you can register your own DataSet
+as:
+
+[source,xml]
+----
+<bean id="myDataSet" class="com.mycompany.MyDataSet">
+  <property name="size" value="100"/>
+</bean>
+----
+
+=== Example
+
+For example, to test that a set of messages are sent to a queue and then
+consumed from the queue without losing any messages:
+
+[source,java]
+----
+// send the dataset to a queue
+from("dataset:foo").to("activemq:SomeQueue");
+
+// now lets test that the messages are consumed correctly
+from("activemq:SomeQueue").to("dataset:foo");
+----
+
+The above would look in the Registry to find the
+*foo* DataSet instance which is used to create the messages.
+
+Then you create a DataSet implementation, such as using the
+`SimpleDataSet` as described below, configuring things like how big the
+data set is and what the messages look like etc.  
+ 
+
+=== DataSetSupport (abstract class)
+
+The DataSetSupport abstract class is a nice starting point for new
+DataSets, and provides some useful features to derived classes.
+
+==== Properties on DataSetSupport
+
+[width="100%",cols="10%,10%,10%,70%",options="header",]
+|===
+|Property |Type |Default |Description
+
+|`defaultHeaders` |`Map<String,Object>` |`null` |Specifies the default message body.
For SimpleDataSet it is a constant
+payload; though if you want to create custom payloads per message,
+create your own derivation of `DataSetSupport`.
+
+|`outputTransformer` |`org.apache.camel.Processor` |null |
+
+|`size` |`long` |`10` |Specifies how many messages to send/consume.
+
+|`reportCount` |`long` |`-1` |Specifies the number of messages to be received before reporting
+progress. Useful for showing progress of a large load test. If < 0, then
+`size` / 5, if is 0 then `size`, else set to `reportCount` value.
+|===
+
+=== SimpleDataSet
+
+The `SimpleDataSet` extends `DataSetSupport`, and adds a default body.
+
+==== Additional Properties on SimpleDataSet
+
+[width="100%",cols="10%,10%,10%,70%",options="header",]
+|===
+|Property |Type |Default |Description
+
+|`defaultBody` |`Object` |`<hello>world!</hello>` |Specifies the default message
body. By default, the `SimpleDataSet`
+produces the same constant payload for each exchange. If you want to
+customize the payload for each exchange, create a Camel `Processor` and
+configure the `SimpleDataSet` to use it by setting the
+`outputTransformer` property.
+|===
+
+=== ListDataSet
+
+*Available since Camel 2.17*
+
+The List`DataSet` extends `DataSetSupport`, and adds a list of default
+bodies.
+
+==== Additional Properties on ListDataSet
+
+[width="100%",cols="10%,10%,10%,70%",options="header",]
+|===
+|Property |Type |Default |Description
+
+|`defaultBodies` |`List<Object>` |`empty LinkedList<Object>` |Specifies the default
message body. By default, the `ListDataSet`
+selects a constant payload from the list of `defaultBodies` using the
+`CamelDataSetIndex`. If you want to customize the payload, create a
+Camel `Processor` and configure the `ListDataSet` to use it by setting
+the `outputTransformer` property.
+
+|`size` |`long` |the size of the defaultBodies list |Specifies how many messages to send/consume.
This value can be different
+from the size of the `defaultBodies` list. If the value is less than the
+size of the `defaultBodies` list, some of the list elements will not be
+used. If the value is greater than the size of the `defaultBodies` list,
+the payload for the exchange will be selected using the modulus of the
+`CamelDataSetIndex` and the size of the `defaultBodies` list (i.e.
+`CamelDataSetIndex % defaultBodies.size()` )
+|===
+
+=== FileDataSet
+
+*Available since Camel 2.17*
+
+The `FileDataSet` extends `ListDataSet`, and adds support for loading
+the bodies from a file.
+
+==== Additional Properties on FileDataSet
+
+[width="100%",cols="10%,10%,10%,70%",options="header",]
+|===
+|Property |Type |Default |Description
+
+|`sourceFile` |`File` |null |Specifies the source file for payloads
+
+|`delimiter` |`String` |\z |Specifies the delimiter pattern used by a `java.util.Scanner`
to split
+the file into multiple payloads.
+|===
diff --git a/docs/components/modules/ROOT/pages/dataset-test-component.adoc b/docs/components/modules/ROOT/pages/dataset-test-component.adoc
new file mode 100644
index 0000000..d4bf677
--- /dev/null
+++ b/docs/components/modules/ROOT/pages/dataset-test-component.adoc
@@ -0,0 +1,152 @@
+[[dataset-test-component]]
+== DataSet Test Component
+*Available as of Camel version 1.3*
+*Available as of Camel version 1.3*
+
+
+
+Testing of distributed and asynchronous processing is
+notoriously difficult. The <<mock-component,Mock>>, <<mock-component,Test>>
+and <<dataset-component,DataSet>> endpoints work great with the
+Camel Testing Framework to simplify your unit and
+integration testing using
+link:enterprise-integration-patterns.html[Enterprise Integration
+Patterns] and Camel's large range of Components
+together with the powerful Bean Integration.
+
+The *dataset-test* component extends the <<mock-component,Mock>> component to
+support pulling messages from another endpoint on startup to set the
+expected message bodies on the underlying <<mock-component,Mock>> endpoint.
+That is, you use the test endpoint in a route and messages arriving on
+it will be implicitly compared to some expected messages extracted from
+some other location.
+
+So you can use, for example, an expected set of message bodies as files.
+This will then set up a properly configured <<mock-component,Mock>>
+endpoint, which is only valid if the received messages match the number
+of expected messages and their message payloads are equal.
+
+Maven users will need to add the following dependency to their `pom.xml`
+for this component when using *Camel 2.8* or older:
+
+[source,xml]
+----
+<dependency>
+    <groupId>org.apache.camel</groupId>
+    <artifactId>camel-spring</artifactId>
+    <version>x.x.x</version>
+    <!-- use the same version as your Camel core version -->
+</dependency>
+----
+
+From Camel 2.9 onwards the <<test-component,Test>> component is provided
+directly in the camel-core.
+
+=== URI format
+
+[source]
+----
+dataset-test:expectedMessagesEndpointUri
+----
+
+Where *expectedMessagesEndpointUri* refers to some other
+Component URI that the expected message bodies are
+pulled from before starting the test.
+
+=== URI Options
+
+// component options: START
+The DataSet Test component has no options.
+// component options: END
+
+
+// endpoint options: START
+The DataSet Test endpoint is configured using URI syntax:
+
+----
+dataset-test:name
+----
+
+with the following path and query parameters:
+
+==== Path Parameters (1 parameters):
+
+
+[width="100%",cols="2,5,^1,2",options="header"]
+|===
+| Name | Description | Default | Type
+| *name* | *Required* Name of endpoint to lookup in the registry to use for polling messages
used for testing |  | String
+|===
+
+
+==== Query Parameters (14 parameters):
+
+
+[width="100%",cols="2,5,^1,2",options="header"]
+|===
+| Name | Description | Default | Type
+| *anyOrder* (producer) | Whether the expected messages should arrive in the same order or
can be in any order. | false | boolean
+| *delimiter* (producer) | The split delimiter to use when split is enabled. By default the
delimiter is new line based. The delimiter can be a regular expression. |  | String
+| *split* (producer) | If enabled the messages loaded from the test endpoint will be split
using new line delimiters so each line is an expected message. For example to use a file endpoint
to load a file where each line is an expected message. | false | boolean
+| *timeout* (producer) | The timeout to use when polling for message bodies from the URI
| 2000 | long
+| *assertPeriod* (producer) | Sets a grace period after which the mock endpoint will re-assert
to ensure the preliminary assertion is still valid. This is used for example to assert that
exactly a number of messages arrives. For example if expectedMessageCount(int) was set to
5, then the assertion is satisfied when 5 or more message arrives. To ensure that exactly
5 messages arrives, then you would need to wait a little period to ensure no further message
arrives. This is what you can us [...]
+| *expectedCount* (producer) | Specifies the expected number of message exchanges that should
be received by this endpoint. Beware: If you want to expect that 0 messages, then take extra
care, as 0 matches when the tests starts, so you need to set a assert period time to let the
test run for a while to make sure there are still no messages arrived; for that use setAssertPeriod(long).
An alternative is to use NotifyBuilder, and use the notifier to know when Camel is done routing
some mess [...]
+| *reportGroup* (producer) | A number that is used to turn on throughput logging based on
groups of the size. |  | int
+| *resultMinimumWaitTime* (producer) | Sets the minimum expected amount of time (in millis)
the assertIsSatisfied() will wait on a latch until it is satisfied | 0 | long
+| *resultWaitTime* (producer) | Sets the maximum amount of time (in millis) the assertIsSatisfied()
will wait on a latch until it is satisfied | 0 | long
+| *retainFirst* (producer) | Specifies to only retain the first n'th number of received Exchanges.
This is used when testing with big data, to reduce memory consumption by not storing copies
of every Exchange this mock endpoint receives. Important: When using this limitation, then
the getReceivedCounter() will still return the actual number of received Exchanges. For example
if we have received 5000 Exchanges, and have configured to only retain the first 10 Exchanges,
then the getReceive [...]
+| *retainLast* (producer) | Specifies to only retain the last n'th number of received Exchanges.
This is used when testing with big data, to reduce memory consumption by not storing copies
of every Exchange this mock endpoint receives. Important: When using this limitation, then
the getReceivedCounter() will still return the actual number of received Exchanges. For example
if we have received 5000 Exchanges, and have configured to only retain the last 20 Exchanges,
then the getReceivedCo [...]
+| *sleepForEmptyTest* (producer) | Allows a sleep to be specified to wait to check that this
endpoint really is empty when expectedMessageCount(int) is called with zero | 0 | long
+| *copyOnExchange* (producer) | Sets whether to make a deep copy of the incoming Exchange
when received at this mock endpoint. Is by default true. | true | boolean
+| *synchronous* (advanced) | Sets whether synchronous processing should be strictly used,
or Camel is allowed to use asynchronous processing (if supported). | false | boolean
+|===
+// endpoint options: END
+// spring-boot-auto-configure options: START
+=== Spring Boot Auto-Configuration
+
+When using Spring Boot make sure to use the following Maven dependency to have support for
auto configuration:
+
+[source,xml]
+----
+<dependency>
+  <groupId>org.apache.camel</groupId>
+  <artifactId>camel-dataset-starter</artifactId>
+  <version>x.x.x</version>
+  <!-- use the same version as your Camel core version -->
+</dependency>
+----
+
+
+The component supports 2 options, which are listed below.
+
+
+
+[width="100%",cols="2,5,^1,2",options="header"]
+|===
+| Name | Description | Default | Type
+| *camel.component.dataset-test.enabled* | Whether to enable auto configuration of the dataset-test
component. This is enabled by default. |  | Boolean
+| *camel.component.dataset-test.resolve-property-placeholders* | Whether the component should
resolve property placeholders on itself when starting. Only properties which are of String
type can use property placeholders. | true | Boolean
+|===
+// spring-boot-auto-configure options: END
+
+
+=== Example
+
+For example, you could write a test case as follows:
+
+[source,java]
+----
+from("seda:someEndpoint").
+  to("dataset-test:file://data/expectedOutput?noop=true");
+----
+
+If your test then invokes the
+http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/component/mock/MockEndpoint.html#assertIsSatisfied(org.apache.camel.CamelContext)[MockEndpoint.assertIsSatisfied(camelContext)
+method], your test case will perform the necessary assertions.
+
+To see how you can set other expectations on the test endpoint, see the
+<<mock-component,Mock>> component.
+
+=== See Also
+
+* Spring Testing
\ No newline at end of file
diff --git a/platforms/spring-boot/spring-boot-dm/camel-spring-boot-dependencies/pom.xml b/platforms/spring-boot/spring-boot-dm/camel-spring-boot-dependencies/pom.xml
index 2a6ab61..5a22a52 100644
--- a/platforms/spring-boot/spring-boot-dm/camel-spring-boot-dependencies/pom.xml
+++ b/platforms/spring-boot/spring-boot-dm/camel-spring-boot-dependencies/pom.xml
@@ -949,6 +949,16 @@
       </dependency>
       <dependency>
         <groupId>org.apache.camel</groupId>
+        <artifactId>camel-dataset</artifactId>
+        <version>${project.version}</version>
+      </dependency>
+      <dependency>
+        <groupId>org.apache.camel</groupId>
+        <artifactId>camel-dataset-starter</artifactId>
+        <version>${project.version}</version>
+      </dependency>
+      <dependency>
+        <groupId>org.apache.camel</groupId>
         <artifactId>camel-digitalocean</artifactId>
         <version>${project.version}</version>
       </dependency>


Mime
View raw message