camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Preben Asmussen (Confluence)" <conflue...@apache.org>
Subject [CONF] Apache Camel > Splunk
Date Thu, 17 Oct 2013 19:12:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/en/2176/1/1/_/styles/combined.css?spaceKey=CAMEL&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/CAMEL/Splunk">Splunk</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~preben">Preben
Asmussen</a>
    </h4>
        <br/>
                         <h4>Changes (5)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >*Available as of Camel 2.13* <br>
<br></td></tr>
            <tr><td class="diff-changed-lines" >The Splunk component provides
access to <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[Splunk|http://splunk.com]</span>
<span class="diff-added-words"style="background-color: #dfd;">[Splunk|http://docs.splunk.com/Documentation/Splunk/latest]</span>
using the Splunk provided [client|https://github.com/splunk/splunk-sdk-java] api, and it enables
you to publish and search for events in Splunk. <br></td></tr>
            <tr><td class="diff-unchanged" > <br>Maven users will need to
add the following dependency to their pom.xml for this component: <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br>|| Endpoint || Description
                                                                                         
                                   <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">|
stream    | Streams data to a named index or the default if not specified.  <br></td></tr>
            <tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">|
stream    | streaming mode.</span> When using stream mode be aware of that Splunk has
some internal buffer (about 1MB or so) before events gets to the index. <br></td></tr>
            <tr><td class="diff-unchanged" >If you need realtime, better use submit
or tcp mode.  | <br></td></tr>
            <tr><td class="diff-changed-lines" >| submit    | submit mode. <span
class="diff-added-words"style="background-color: #dfd;">Uses Splunk rest api to publish
events to a named index or the default if not specified.</span>     <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">
</span> | <br></td></tr>
            <tr><td class="diff-changed-lines" >| tcp       | tcp mode. <span
class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">Requires</span>
<span class="diff-added-words"style="background-color: #dfd;">Streams data to</span>
a <span class="diff-added-words"style="background-color: #dfd;">tcp port, and requires
a</span> open receiver port in Splunk.| <br></td></tr>
            <tr><td class="diff-unchanged" > <br>When publishing events
the message body should contain a SplunkEvent.  <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h2><a name="Splunk-SplunkComponent"></a>Splunk Component</h2>

<p><b>Available as of Camel 2.13</b></p>

<p>The Splunk component provides access to <a href="http://docs.splunk.com/Documentation/Splunk/latest"
class="external-link" rel="nofollow">Splunk</a> using the Splunk provided <a href="https://github.com/splunk/splunk-sdk-java"
class="external-link" rel="nofollow">client</a> api, and it enables you to publish
and search for events in Splunk.</p>

<p>Maven users will need to add the following dependency to their pom.xml for this component:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
	&lt;dependency&gt;
    	&lt;groupId&gt;org.apache.camel&lt;/groupId&gt;
    	&lt;artifactId&gt;camel-splunk&lt;/artifactId&gt;
    	&lt;version&gt;${camel-version}&lt;/version&gt;
	&lt;/dependency&gt;
</pre>
</div></div>

<h3><a name="Splunk-URIformat"></a>URI format </h3>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
  splunk://[endpoint]?[options]
</pre>
</div></div>

<h3><a name="Splunk-ProducerEndpoints%3A"></a>Producer Endpoints: </h3>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Endpoint </th>
<th class='confluenceTh'> Description</th>
</tr>
<tr>
<td class='confluenceTd'> stream    </td>
<td class='confluenceTd'> Streams data to a named index or the default if not specified.
<br/>
When using stream mode be aware of that Splunk has some internal buffer (about 1MB or so)
before events gets to the index. <br/>
If you need realtime, better use submit or tcp mode.  </td>
</tr>
<tr>
<td class='confluenceTd'> submit    </td>
<td class='confluenceTd'> submit mode. Uses Splunk rest api to publish events to a named
index or the default if not specified.     </td>
</tr>
<tr>
<td class='confluenceTd'> tcp       </td>
<td class='confluenceTd'> tcp mode. Streams data to a tcp port, and requires a open
receiver port in Splunk.</td>
</tr>
</tbody></table>
</div>


<p>When publishing events the message body should contain a SplunkEvent. </p>

<p><b>Example</b></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
      from("direct:start").convertBodyTo(SplunkEvent.class)
          .to("splunk://submit?username=user&amp;password=123&amp;index=myindex&amp;sourceType=someSourceType&amp;source=mySource")...
</pre>
</div></div>
<p>In this example a converter is required to convert to a SplunkEvent class. </p>

<h3><a name="Splunk-ConsumerEndpoints%3A"></a>Consumer Endpoints: </h3>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Endpoint </th>
<th class='confluenceTh'> Description</th>
</tr>
<tr>
<td class='confluenceTd'>normal     </td>
<td class='confluenceTd'> Performs normal search and requires a search query in the
search option.</td>
</tr>
<tr>
<td class='confluenceTd'>savedsearch</td>
<td class='confluenceTd'> Performs search based on a search query saved in splunk and
requires the name of the query in the savedSearch option.</td>
</tr>
</tbody></table>
</div>


<p><b>Example</b></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
      from("splunk://normal?delay=5s&amp;username=user&amp;password=123&amp;initEarliestTime=-10s&amp;search=search
index=myindex sourcetype=someSourcetype")
          .to("direct:search-result");
</pre>
</div></div>

<p>camel-splunk creates a route exchange per search result with a SplunkEvent in the
body. </p>

<h3><a name="Splunk-URIOptions"></a>URI Options</h3>
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Name </th>
<th class='confluenceTh'> Default Value </th>
<th class='confluenceTh'> Context </th>
<th class='confluenceTh'> Description </th>
</tr>
<tr>
<td class='confluenceTd'>host </td>
<td class='confluenceTd'> localhost </td>
<td class='confluenceTd'> Both </td>
<td class='confluenceTd'> Splunk host.</td>
</tr>
<tr>
<td class='confluenceTd'>port </td>
<td class='confluenceTd'> 8089 </td>
<td class='confluenceTd'> Both </td>
<td class='confluenceTd'> Splunk port</td>
</tr>
<tr>
<td class='confluenceTd'> username </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Both </td>
<td class='confluenceTd'> Username for Splunk</td>
</tr>
<tr>
<td class='confluenceTd'> password </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Both </td>
<td class='confluenceTd'> Password for Splunk</td>
</tr>
<tr>
<td class='confluenceTd'> connectionTimeout </td>
<td class='confluenceTd'> 5000 </td>
<td class='confluenceTd'> Both </td>
<td class='confluenceTd'> Timeout in MS when connecting to Splunk server</td>
</tr>
<tr>
<td class='confluenceTd'> useSunHttpsHandler </td>
<td class='confluenceTd'> false </td>
<td class='confluenceTd'> Both </td>
<td class='confluenceTd'> Use sun.net.www.protocol.https.Handler Https hanlder to establish
the Splunk Connection. <br/>
Can be useful when running in application servers to avoid app. server https handling.</td>
</tr>
<tr>
<td class='confluenceTd'> index </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Producer </td>
<td class='confluenceTd'> Splunk index to write to</td>
</tr>
<tr>
<td class='confluenceTd'> sourceType </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Producer </td>
<td class='confluenceTd'> Splunk sourcetype arguement</td>
</tr>
<tr>
<td class='confluenceTd'> source </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Producer </td>
<td class='confluenceTd'> Splunk source arguement</td>
</tr>
<tr>
<td class='confluenceTd'> tcpReceiverPort </td>
<td class='confluenceTd'> 0 </td>
<td class='confluenceTd'> Producer </td>
<td class='confluenceTd'> Splunk tcp receiver port when using tcp producer endpoint.</td>
</tr>
<tr>
<td class='confluenceTd'> initEarliestTime </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Consumer </td>
<td class='confluenceTd'> Initial start offset of the first search. Required</td>
</tr>
<tr>
<td class='confluenceTd'> earliestTime </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Consumer </td>
<td class='confluenceTd'> Earliest time of the search time window.</td>
</tr>
<tr>
<td class='confluenceTd'> latestTime </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Consumer </td>
<td class='confluenceTd'> Latest time of the search time window.</td>
</tr>
<tr>
<td class='confluenceTd'> count </td>
<td class='confluenceTd'> 0 </td>
<td class='confluenceTd'> Consumer </td>
<td class='confluenceTd'> A number that indicates the maximum number of entities to
return. <br/>
Note this is not the same as maxMessagesPerPoll which currently is unsupported</td>
</tr>
<tr>
<td class='confluenceTd'> search </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Consumer </td>
<td class='confluenceTd'> The Splunk query to run</td>
</tr>
<tr>
<td class='confluenceTd'> savedSearch </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> Consumer </td>
<td class='confluenceTd'> The name of the query saved in Splunk to run</td>
</tr>
</tbody></table>
</div>


<h3><a name="Splunk-Messagebody"></a>Message body</h3>
<p>Splunk operates on data in key/value pairs. The SplunkEvent class is a placeholder
for such data, and should be in the message body <br/>
for the producer. Likewise it will be returned in the body per search result for the consumer.
</p>

<h3><a name="Splunk-UseCases"></a>Use Cases</h3>
<p>Search Twitter for tweets with music and publish events to Splunk</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
      from("twitter://search?type=polling&amp;keywords=music&amp;delay=10&amp;consumerKey=abc&amp;consumerSecret=def&amp;accessToken=hij&amp;accessTokenSecret=xxx")
          .convertBodyTo(SplunkEvent.class)
          .to("splunk://submit?username=foo&amp;password=bar&amp;index=camel-tweets&amp;sourceType=twitter&amp;source=music-tweets");
</pre>
</div></div>

<p>To convert a Tweet to a SplunkEvent you could use a converter like</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
@Converter
public class Tweet2SplunkEvent {
    @Converter
    public static SplunkEvent convertTweet(Status status) {
        SplunkEvent data = new SplunkEvent("twitter-message", null);
        //data.addPair("source", status.getSource());
        data.addPair("from_user", status.getUser().getScreenName());
        data.addPair("in_reply_to", status.getInReplyToScreenName());
        data.addPair(SplunkEvent.COMMON_START_TIME, status.getCreatedAt());
        data.addPair(SplunkEvent.COMMON_EVENT_ID, status.getId());
        data.addPair("text", status.getText());
        data.addPair("retweet_count", status.getRetweetCount());
        if (status.getPlace() != null) {
            data.addPair("place_country", status.getPlace().getCountry());
            data.addPair("place_name", status.getPlace().getName());
            data.addPair("place_street", status.getPlace().getStreetAddress());
        }
        if (status.getGeoLocation() != null) {
            data.addPair("geo_latitude", status.getGeoLocation().getLatitude());
            data.addPair("geo_longitude", status.getGeoLocation().getLongitude());
        }
        return data;
    }
}
</pre>
</div></div>

<p>Search Splunk for tweets</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
      from("splunk://normal?username=foo&amp;password=bar&amp;initEarliestTime=-2m&amp;search=search
index=camel-tweets sourcetype=twitter")
          .log("${body}");
</pre>
</div></div>

<h3><a name="Splunk-Othercomments"></a>Other comments</h3>
<p>Splunk comes with a variety of options for leveraging machine generated data with
prebuilt apps for analyzing and displaying this. <br/>
For example the jmx app. could be used to publish jmx attributes, eg. route and jvm metrics
to Splunk, and displaying this on a dashboard.</p>

    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;" class="grey">
                        <a href="https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=CAMEL">Stop
watching space</a>
            <span style="padding: 0px 5px;">|</span>
                <a href="https://cwiki.apache.org/confluence/users/editmyemailsettings.action">Change
email notification preferences</a>
</div>
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/Splunk">View Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=34836117&revisedVersion=13&originalVersion=12">View
Changes</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message