Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0AC4B200D14 for ; Tue, 3 Oct 2017 15:30:41 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 09876160BD5; Tue, 3 Oct 2017 13:30:41 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A8178160BDC for ; Tue, 3 Oct 2017 15:30:38 +0200 (CEST) Received: (qmail 29430 invoked by uid 500); 3 Oct 2017 13:30:37 -0000 Mailing-List: contact commits-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list commits@nifi.apache.org Received: (qmail 29421 invoked by uid 99); 3 Oct 2017 13:30:37 -0000 Received: from Unknown (HELO svn01-us-west.apache.org) (209.188.14.144) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Oct 2017 13:30:37 +0000 Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id 527723A05D3 for ; Tue, 3 Oct 2017 13:30:34 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1811008 [7/43] - in /nifi/site/trunk/docs: ./ nifi-docs/ nifi-docs/components/ nifi-docs/components/org.apache.nifi/ nifi-docs/components/org.apache.nifi/nifi-ambari-nar/ nifi-docs/components/org.apache.nifi/nifi-ambari-nar/1.4.0/ nifi-doc... Date: Tue, 03 Oct 2017 13:30:27 -0000 To: commits@nifi.apache.org From: jstorck@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20171003133034.527723A05D3@svn01-us-west.apache.org> archived-at: Tue, 03 Oct 2017 13:30:41 -0000 Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailAttachments/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailAttachments/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailAttachments/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailAttachments/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ExtractEmailAttachments

ExtractEmailAttachments

Description:

Extract attachments from a mime formatted email file, splitting them into individual flowfiles.

Tags:

split, email

Properties:

This component has no required or optional properties.

Relationships:

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
failureFlowfiles that could not be parsed
originalThe original file
attachmentsEach individual attachment will be routed to the attachments relationship
NameDescription
filename The filename of the attachment
email.attachment.parent.filename The filename of the parent FlowFile
email.attachment.parent.uuidThe UUID of the original FlowFile.
mime.typeThe mime type of the attachment.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ExtractEmailHeaders

ExtractEmailHeaders

Description:

Using the flowfile content as source of data, extract header from an RFC compliant email file adding the relevant attributes to the flowfile. This processor does not perform extensive RFC validation but still requires a bare minimum compliance with RFC 2822

Tags:

split, email

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

Name< /th>Default ValueAllowable ValuesDescription
Additional Header Listx-mailerCOLON separated list of additional headers to be extracted from the flowfile content.NOTE the header key is case insensitive and will be matched as lower-case. Values will respect email contents.
Email Address Parsingtrue
  • Strict Address Parsing Strict email address format will be enforced. FlowFiles will be transfered to the failure relationship if the email address is invalid.
  • Non-Strict Address Parsing Accept emails, 
 even if the address is poorly formed and doesn't strictly comply with RFC Validation.
If "strict", strict address format parsing rules are applied to mailbox and mailbox list fields, such as "to" and "from" headers, and FlowFiles with poorly formed addresses will be routed to the failure relationship, similar to messages that fail RFC compliant format validation. If "non-strict", the processor will extract the contents of mailbox list headers as comma-separated values without attempting to parse each value as well-formed Internet mailbox addresses. This is optional and defaults to Strict Address Parsing

Relationships:

NameDescription
successExtraction was successful
failureFlowfiles that could not be parsed as a RF C-2822 compliant message

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
email.headers.bcc.*Each individual BCC recipient (if available)
email.headers.cc.*Each individual CC recipient (if available)
email.headers.from.*Each individual mailbox contained in the From of the Email (array as per RFC-2822)
email.headers.message-idThe value of the Message-ID header (if available)
email.headers.received_dateThe Received-Date of the message (if available)
email.headers.sent_dateDate the message was sent
email.headers.subjectSubject of the message (if available)
email.headers.to.*Each individual TO recipient (if available)
email.attachment_countNumber of attachm ents of the message

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ExtractTNEFAttachments/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ExtractTNEFAttachments

ExtractTNEFAttachments

Description:

Extract attachments from a mime formatted email file, splitting them into individual flowfiles.

Tags:

split, email

Properties:

This component has no required or optional properties.

Relationships:

NameDescription
failureEach individual flowfile that could not be parsed will be routed to the failure relationship
originalEach original flowfile (i.e. before extraction) will be routed to the original relati onship
attachmentsEach individual attachment will be routed to the attachments relationship

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
filename The filename of the attachment
email.tnef.attachment.parent.filename The filename of the parent FlowFile
email.tnef.attachment.parent.uuidThe UUID of the original FlowFile.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-email-nar/1.4.0/org.apache.nifi.processors.email.ListenSMTP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListenSMTP

ListenSMTP

Description:

This processor implements a lightweight SMTP server to an arbitrary port, allowing nifi to listen for incoming email. Note this server does not perform any email validation. If direct exposure to the internet is sought, it may be a better idea to use the combination of NiFi and an industrial scale MTA (e.g. Postfix). Threading for this processor is managed by the underlying smtp server used so the processor need not support more than one thread.

Tags:

listen, email, smtp

Properties:

In the list below, the names of required properti es appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
Listening PortThe TCP port the ListenSMTP processor will bind to.NOTE that on Unix derivative operating systems this port must be higher than 1024 unless NiFi is running as with root user permissions.
Maximum number of SMTP connection1The maximum number of simultaneous SMTP connections.
SMTP connection timeout60 secondsThe maximum time to wait for an action of SMTP client.
SMTP Maximum Message Size20 MBThe maximum number of bytes the server will accept.
SSL Context ServiceController Service API:
RestrictedSSLContextService
Implementation: StandardRestrictedSSLContextService
The Controller Service to use in order to obtain an SSL Context. If this property is set, messages will be received over a secure connection.
Client Auth
  • NONE
  • REQUIRED
The client authentication policy to use for the SSL Context. Only used if an SSL Context Service is provided.
SMTP hostnameThe hostname to be embedded into the banner displayed when an SMTP client connects to the processor TCP port .

Relationships:

NameDescription
successAll new messages will be routed as FlowFiles to this relationship

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
smtp.heloThe value used during HELO
smtp.certificates.*.serialThe serial numbers for each of the certificates used by an TLS peer
smtp.certificates.*.principalThe principal for each of the certificates used by an TLS peer
smtp.srcThe source IP and port of the SMTP connection
smtp.fromThe value used during MAIL FROM (i.e. envelope)
smtp.recipient.*The values used during RCPT TO (i.e. envelope)
mime.typeMime type of the message

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.GeoEnrichIP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +GeoEnrichIP

GeoEnrichIP

Description:

Looks up geolocation information for an IP address and adds the geo information to FlowFile attributes. The geo data is provided as a MaxMind database. The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the the attributes added by enrichment will take the form X.geo.<fieldName>

Tags:

geo, enrich, ip, maxmind

Properties:

In the list below, the names of required properties appear in bold. Any other propert ies (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
MaxMind Database FilePath to Maxmind IP Enrichment Database File
IP Address AttributeThe name of an attribute whose value is a dotted decimal IP address for which enrichment should occur
Supports Expression Language: true

Relationships:

NameDescription
not foundWhere to route flow files after unsucces sfully enriching attributes because no data was found
foundWhere to route flow files after successfully enriching attributes with data provided by database

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
X.geo.lookup.microsThe number of microseconds that the geo lookup took
X.geo.cityThe city identified for the IP address
X.geo.accuracyThe accuracy radius if provided by the database (in Kilometers)
X.geo.latitudeThe latitude identified for this IP address
X.geo.longitudeThe longitude identified for this IP address
X.geo.subdivision.NEach subdivision that is identified for this IP address is added with a one-up number appended to the attribute name, starting with 0
X.geo.subdivision.isoco de.NThe ISO code for the subdivision that is identified by X.geo.subdivision.N
X.geo.countryThe country identified for this IP address
X.geo.country.isocodeThe ISO Code for the country identified
X.geo.postalcodeThe postal code for the country identified

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.ISPEnrichIP/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ISPEnrichIP

ISPEnrichIP

Description:

Looks up ISP information for an IP address and adds the information to FlowFile attributes. The ISP data is provided as a MaxMind ISP database (Note that this is NOT the same as the GeoLite database utilizedby some geo enrichment tools). The attribute that contains the IP address to lookup is provided by the 'IP Address Attribute' property. If the name of the attribute provided is 'X', then the the attributes added by enrichment will take the form X.isp.<fieldName>

Tags:

ISP, enrich, ip, maxmind

Properties:

In the list belo w, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
MaxMind Database FilePath to Maxmind IP Enrichment Database File
IP Address AttributeThe name of an attribute whose value is a dotted decimal IP address for which enrichment should occur
Supports Expression Language: true

Relationships:

NameD escription
not foundWhere to route flow files after unsuccessfully enriching attributes because no data was found
foundWhere to route flow files after successfully enriching attributes with data provided by database

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
X.isp.lookup.microsThe number of microseconds that the geo lookup took
X.isp.asnThe Autonomous System Number (ASN) identified for the IP address
X.isp.asn.organizationThe Organization Associated with the ASN identified
X.isp.nameThe name of the ISP associated with the IP address provided
X.isp.organizationThe Organization associated with the IP address provided

State management:

This component does not store state.

R estricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryDNS/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,2 @@ +QueryDNS

QueryDNS

Description:

A powerful DNS query processor primary designed to enrich DataFlows with DNS based APIs (e.g. RBLs, ShadowServer's ASN lookup) but that can be also used to perform regular DNS lookups.

Tags:

dns, enrich, ip

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescriptionLookup valueThe value that should be used to populate the query
Supports Expression Language: trueResults ParserNone
  • Split Use a delimiter character or RegEx  to split the results into attributes
  • RegEx Use a regular expression to split the results into attributes
  • None Do not split 
 results
The method used to slice the results into attribute groupsParser RegExChoice between a splitter and regex matcher used to parse the results of the query into attribute groups. +NOTE: This is a multiline regular expression, therefore, the DFM should decide how to handle trailing new line characters.DNS Query Retries1The number of attempts before giving up and moving onDNS Query Timeout1500 msThe amount of time to wait until considering a query as failedDNS ServersA comma separated list of DNS servers to be used. (Defaults to system wide if none is used)DNS Query TypeTXTThe DNS query type to be used by the processor (e.g. TXT, A)

Relationships :

NameDescription
not foundWhere to route flow files if data enrichment query rendered no results
foundWhere to route flow files after successfully enriching attributes with data

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
enrich.dns.record*.group*The captured fields of the DNS query response for each of the records received

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-enrich-nar/1.4.0/org.apache.nifi.processors.enrich.QueryWhois/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,2 @@ +QueryWhois

QueryWhois

Description:

A powerful whois query processor primary designed to enrich DataFlows with whois based APIs (e.g. ShadowServer's ASN lookup) but that can be also used to perform regular whois lookups.

Tags:

whois, enrich, ip

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Lookup valueThe value that should be used to populate the query
Supports Expression Language: true
Whois Query TypeThe Whois query type to be used by the processor (if used)
Whois ServerThe Whois server to be used
Whois Server Port43The TCP port of the remote Whois server
Whois Query Timeout1500 m sThe amount of time to wait until considering a query as failed
Batch Size25The number of incoming FlowFiles to process in a single execution of this processor.
Bulk ProtocolNone
  • Begin/End The evaluated input of each flowfile is enclosed within begin and end tags. Each row contains a delimited set of fields
  • None Queries are made without any particular dialect
The protocol used to perform the bulk query.
Results ParserNone
  • Split Use a delimiter character or RegEx  to split the results into attributes
  • RegEx Use a regular expression to split the results into attributes
  • None Do not split results
The method used to slice the results into attribute groups
Parser RegExChoice between a splitte r and regex matcher used to parse the results of the query into attribute groups. +NOTE: This is a multiline regular expression, therefore, the DFM should decide how to handle trailing new line characters.
Key lookup group (multiline / batch)When performing a batched lookup, the following RegEx numbered capture group or Column number will be used to match the whois server response with the lookup field

Relationships:

NameDescription
not foundWhere to route flow files if data enrichment query rendered no results
foundWhere to route flow files after successfully enriching attributes with data

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
enrich.dns.record*.group*The captured fields of the Whois query response for each of the records received

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/additionalDetails.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,91 @@ + + + + + + ParseEvtx + + + + + + +

Description:

+

This processor is used to parse Windows event logs in the binary evtx format. The input flow files' content should be evtx files. The processor has 4 outputs: +

    +
  • The original unmodified FlowFile
  • +
  • The XML resulting from parsing at the configured granularity
  • +
  • Failed parsing with partial output
  • +
  • Malformed chunk in binary form
  • +
+

+

Output XML Example:

+

+

+<?xml version="1.0"?>
+<Events>
+  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
+    <System>
+      <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" Ev
+entSourceName="Service Control Manager"/>
+      <EventID Qualifiers="16384">7036</EventID>
+      <Version>0</Version>
+      <Level>4</Level>
+      <Task>0</Task>
+      <Opcode>0</Opcode>
+      <Keywords>0x8080000000000000</Keywords>
+      <TimeCreated SystemTime="2016-01-08 16:49:47.518"/>
+      <EventRecordID>780</EventRecordID>
+      <Correlation ActivityID="" RelatedActivityID=""/>
+      <Execution ProcessID="480" ThreadID="596"/>
+      <Channel>System</Channel>
+      <Computer>win7-pro-vm</Computer>
+      <Security UserID=""/>
+    </System>
+    <EventData>
+      <Data Name="param1">Workstation</Data>
+      <Data Name="param2">running</Data>
+      <Binary>TABhAG4AbQBhAG4AVwBvAHIAawBzAHQAYQB0AGkAbwBuAC8ANAAAAA==</Binary>
+    </EventData>
+  </Event>
+  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
+    <System>
+      <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager"/>
+      <EventID Qualifiers="16384">7036</EventID>
+      <Version>0</Version>
+      <Level>4</Level>
+      <Task>0</Task>
+      <Opcode>0</Opcode>
+      <Keywords>0x8080000000000000</Keywords>
+      <TimeCreated SystemTime="2016-01-08 16:49:47.535"/>
+      <EventRecordID>781</EventRecordID>
+      <Correlation ActivityID="" RelatedActivityID=""/>
+      <Execution ProcessID="480" ThreadID="576"/>
+      <Channel>System</Channel>
+      <Computer>win7-pro-vm</Computer>
+      <Security UserID=""/>
+    </System>
+    <EventData>
+      <Data Name="param1">Cryptographic Services</Data>
+      <Data Name="param2">running</Data>
+      <Binary>QwByAHkAcAB0AFMAdgBjAC8ANAAAAA==</Binary>
+    </EventData>
+  </Event>
+</Events>
+             
+

+ + Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-evtx-nar/1.4.0/org.apache.nifi.processors.evtx.ParseEvtx/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ParseEvtx

ParseEvtx

Description:

Parses the contents of a Windows Event Log file (evtx) and writes the resulting XML to the FlowFile

Additional Details...

Tags:

logs, windows, event, evtx, message, file

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
GranularityChunk
  • Record
  • Chunk
  • File
Output flow file for each Record, Chunk, or File encountered in the event log

Relationships:

NameDescription
successAny FlowFile that was successfully converted from evtx to XML
failureAny FlowFile that encountered an exception during conversion will be transferred to this relationship with as much parsing as possible done
originalThe unmodified input FlowFile will be transferred to this relationship
bad chunkAny bad chunks of records will be transferred to this relationship in their original binary form

Reads Attributes:

NameDescription
filename The filename of the evtx file

Writes Attributes:

NameDescription
filenameThe output filename
mime.typeThe output filetype (application/xml for success and failure relationships, original value for bad chunk and original relationships)

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,155 @@ + + + + + + ExecuteFlumeSink + + + + +

Data Model

+

+ This processor executes an Apache Flume sink. FlowFiles are wrapped in Flume's + Event interface. The content of the FlowFile becomes the body of the Event and + the attributes of the FlowFile become Event headers. The following special + headers are also set: +

+ + + + + + + + + + + + + + + + + + + + + + + + + +
Flume Event HeaderFlowFile Attribute
nifi.entry.dateFlowFile#getEntryDate()
nifi.idFlowFile#getId()
nifi.last.queue.dateFlowFile#getLastQueueDate()
nifi.lineage.start.dateFlowFile#getLineageStartDate()
nifi.sizeFlowFile#getSize()
+

Warning

+

+ In NiFi, the contents of a FlowFile are accessed via a stream, but in Flume it is + stored in a byte array. This means the full content will be loaded into memory when + a FlowFile is processed by the ExecuteFlumeSink processor. You should consider the + typical size of the FlowFiles you'll process and the batch size, if any, your sink + is configured with when setting NiFi's heap size. +

+

Configuration Details

+

+ This processor is designed to execute arbitrary Flume sinks. Most of the details + of configuring the sink is deferred to Flume's built-in configuration system. + For details on the available settings for each sink type, refer to the Flume + User Guide. + Configuring the Flume sink is a four step process: +

+
    +
  1. Set the Sink Type property to a valid Flume sink type.
  2. +
  3. + Set the Agent Name property to the name of the agent in your + Flume configuration. This is the prefix of the properties in the Flume + configuration file. Example: tier1 +
  4. +
  5. + Set the Sink Name property to the name of the sink in your Flume + configuration. If Agent Name is tier1, then the Sink Name + is the value of the tier1.sinks property. Example: sink-1 +
  6. +
  7. + Copy and paste the configuration for the sink from your Flume configuration + file into the Flume Configuration property. Assuming you're using + the same Agent Name and Sink Name as in the examples above, this will be all + of the properties that start with tier1.sinks.sink-1. + Do not copy the tier1.sinks.sink-1.type or + tier1.sinks.sink-1.channel properties. +
  8. +
+

Usage Example

+

+ Assuming you had the following existing Flume configuration file: +

+
+a1.sources = r1
+a1.sinks = k1
+a1.channels = c1
+
+a1.sources.r1.type = netcat
+a1.sources.r1.bind = localhost
+a1.sources.r1.port = 44444
+
+a1.sinks.k1.type = hdfs
+a1.sinks.k1.channel = c1
+a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+a1.sinks.k1.hdfs.filePrefix = events-
+a1.sinks.k1.hdfs.round = true
+a1.sinks.k1.hdfs.roundValue = 10
+a1.sinks.k1.hdfs.roundUnit = minute
+
+a1.channels.c1.type = memory
+a1.channels.c1.capacity = 1000
+a1.channels.c1.transactionCapacity = 100
+
+a1.sources.r1.channels = c1
+a1.sinks.k1.channel = c1
+

+ Then you'd configure the ExecuteFlumeSink as follows: +

+ + + + + + + + + + + + + + + + + + + + + +
PropertyValue
Sink Typehdfs
Agent Namea1
Sink Namek1
Flume Configuration + + a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+ a1.sinks.k1.hdfs.filePrefix = events-
+ a1.sinks.k1.hdfs.round = true
+ a1.sinks.k1.hdfs.roundValue = 10
+ a1.sinks.k1.hdfs.roundUnit = minute +
+
+ + \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSink/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ExecuteFlumeSink

ExecuteFlumeSink

Description:

Execute a Flume sink. Each input FlowFile is converted into a Flume Event for processing by the sink.

Additional Details...

Tags:

flume, hadoop, put, sink, restricted

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
Sink TypeThe component type name for the sink. For some sinks, this is a short, symbolic name (e.g. hdfs). For others, it's the fully-qualified name of the Sink class. See the Flume User Guide for details.
Agent Nametier1The name of the agent used in the Flume sink configuration
Sink Namesink-1The name of the sink used in the Flume sink configuration
Flume ConfigurationThe Flume configuration for the sink copied from the flume.properties file

Relationships:

NameDescription
success
failure

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

Provides operator the ability to execute arbitrary Flume configurations assuming all permissions that NiFi has.

Input requirement:

This component requires an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1,114 @@ + + + + + + ExecuteFlumeSource + + + + +

Data Model

+

+ This processor executes an Apache Flume source. Each Flume Event is turned into + a FlowFile. The content of the FlowFile is set to the body of the Event and + the Event headers become FlowFile attributes. +

+

Configuration Details

+

+ This processor is designed to execute arbitrary Flume sources. Most of the details + of configuring the source is deferred to Flume's built-in configuration system. + For details on the available settings for each source type, refer to the Flume + User Guide. + Configuring the Flume source is a four step process: +

+
    +
  1. Set the Source Type property to a valid Flume source type.
  2. +
  3. + Set the Agent Name property to the name of the agent in your + Flume configuration. This is the prefix of the properties in the Flume + configuration file. Example: tier1 +
  4. +
  5. + Set the Source Name property to the name of the source in your Flume + configuration. If Agent Name is tier1, then the Source Name + is the value of the tier1.sources property. Example: src-1 +
  6. +
  7. + Copy and paste the configuration for the source from your Flume configuration + file into the Flume Configuration property. Assuming you're using + the same Agent Name and Source Name as in the examples above, this will be all + of the properties that start with tier1.sources.src-1. + Do not copy the tier1.sources.src-1.type or + tier1.sources.src-1.channel properties. +
  8. +
+

Usage Example

+

+ Assuming you had the following existing Flume configuration file: +

+
+a1.sources = r1
+a1.sinks = k1
+a1.channels = c1
+
+a1.sources.r1.type = multiport_syslogtcp
+a1.sources.r1.channels = c1
+a1.sources.r1.host = 0.0.0.0
+a1.sources.r1.ports = 10001 10002 10003
+a1.sources.r1.portHeader = port
+
+a1.sinks.k1.type = logger
+
+a1.channels.c1.type = memory
+a1.channels.c1.capacity = 1000
+a1.channels.c1.transactionCapacity = 100
+
+a1.sources.r1.channels = c1
+a1.sinks.k1.channel = c1
+

+ Then you'd configure the ExecuteFlumeSource as follows: +

+ + + + + + + + + + + + + + + + + + + + + +
PropertyValue
Source Typemultiport_syslogtcp
Agent Namea1
Source Namer1
Flume Configuration + + a1.sources.r1.host = 0.0.0.0
+ a1.sources.r1.ports = 10001 10002 10003
+ a1.sources.r1.portHeader = port +
+
+ + \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-flume-nar/1.4.0/org.apache.nifi.processors.flume.ExecuteFlumeSource/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ExecuteFlumeSource

ExecuteFlumeSource

Description:

Execute a Flume source. Each Flume Event is sent to the success relationship as a FlowFile

Additional Details...

Tags:

flume, hadoop, get, source, restricted

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
Source TypeThe component type name for the source. For some sources, this is a short, symbolic name (e.g. spooldir). For others, it's the fully-qualified name of the Source class. See the Flume User Guide for details.
Agent Nametier1The name of the agent used in the Flume source configuration
Source Namesrc-1The name of the source used in the Flume source configuration
Flume ConfigurationThe Flume configuration for the source copied from the flume.properties file

Rela tionships:

NameDescription
success

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

Provides operator the ability to execute arbitrary Flume configurations assuming all permissions that NiFi has.

Input requirement:

This component does not allow an incoming relationship. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.credentials.service.GCPCredentialsControllerService/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +GCPCredentialsControllerService

GCPCredentialsControllerService

Description:

Defines credentials for Google Cloud Platform processors. Uses Application Default credentials without configuration. Application Default credentials support environmental variable (GOOGLE_APPLICATION_CREDENTIALS) pointing to a credential file, the config generated by `gcloud auth application-default login`, AppEngine/Compute Engine service accounts, etc.

Tags:

gcp, credentials, provider

Properties:

In the list below, the names of required properties appear in bold. Any oth er properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
Use Application Default Credentialsfalse
  • true
  • false
If true, uses Google Application Default Credentials, which checks the GOOGLE_APPLICATION_CREDENTIALS environment variable for a filepath to a service account JSON key, the config generated by the gcloud sdk, the A pp Engine service account, and the Compute Engine service account.
Use Compute Engine Credentialsfalse
  • true
  • false
If true, uses Google Compute Engine Credentials of the Compute Engine VM Instance which NiFi is running on.
Service Account JSON FilePath to a file containing a Service Account key file in JSON format.
Service Account JSONThe raw JSON containing a Service Account keyfile.
Sensitive Property: true
Supports Expression Language: true

State management:

This component does not store state.

Restricted:

This component is not restricted. \ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.DeleteGCSObject/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +DeleteGCSObject

DeleteGCSObject

Description:

Deletes objects from a Google Cloud Bucket. If attempting to delete a file that does not exist, FlowFile is routed to success.

Tags:

google cloud, gcs, google, storage, delete

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
GCP Credentials Provider ServiceController Service API:
GCPCredentialsService
Implementation: GCPCredentialsControllerService
The Controller Service used to obtain Google Cloud Platform credentials.
Project IDGoogle Cloud Project ID
Number of retries6How many retry attempts should be made before routing to the failure relationship.
Buck et${gcs.bucket}Bucket of the object.
Supports Expression Language: true
Key${filename}Name of the object.
Supports Expression Language: true
GenerationThe generation of the object to be deleted. If null, will use latest version of the object.
Supports Expression Language: true

Relationships:

NameDescription
successFlowFiles are routed to this relationship after a successful Google Cloud Storage operation.
failureFlowFiles are routed to this relationship if the Google Cloud Storage operation fails.

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

See Also:

PutGCSObject, FetchGCSObject, ListGCSBucket

\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.FetchGCSObject/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +FetchGCSObject

FetchGCSObject

Description:

Fetches a file from a Google Cloud Bucket. Designed to be used in tandem with ListGCSBucket.

Tags:

google cloud, google, storage, gcs, fetch

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
GCP Credentials Provider ServiceController Service API:
GCPCredentialsService
Implementation: GCPCredentialsControllerService
The Controller Service used to obtain Google Cloud Platform credentials.
Project IDGoogle Cloud Project ID
Number of retries6How many retry attempts should be made before routing to the failure relationship.
Bucket${gcs.bucket}Bucket of the object.
Supports Expression Language: true
Key${filename}Name of the object.
Supports Expression Language: true
Object GenerationThe generation of the Object to download. If null, will download latest generation.
Supports Expression Language: true
Server Side Encryption KeyAn AES256 Key (encoded in base64) which the object has been encrypted in.
Sensitive Property: true
Supports Expression Language: true

Relationships:

NameDescription
successFlowFiles are routed to this relationship after a successful Google Cloud Storage operation.
failureFlowFiles are routed to this relationship if the Google Cloud Storage operation fails.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
filenameThe name of the file, parsed if possible from the Content-Disposition response header
gcs.bucketBucket of the object.
gcs.keyName of the object.
gcs.sizeSize of the objec t.
gcs.cache.controlData cache control of the object.
gcs.component.countThe number of components which make up the object.
gcs.content.dispositionThe data content disposition of the object.
gcs.content.encodingThe content encoding of the object.
gcs.content.languageThe content language of the object.
mime.typeThe MIME/Content-Type of the object
gcs.crc32cThe CRC32C checksum of object's data, encoded in base64 in big-endian order.
gcs.create.timeThe creation time of the object (milliseconds)
gcs.update.timeThe last modification time of the object (milliseconds)
gcs.encryption.algorithmThe algorithm used to encrypt the object.
gcs.encryption.sha256The SHA256 hash of the key used to encrypt the object
gcs.etagThe HTTP 1.1 Entity tag for the object.
gcs.generated.idThe service-generated for the object
gcs.generationThe data generation of the object.
gcs.md5The MD5 hash of the object's data encoded in base64.
gcs.media.linkThe media download link to the object.
gcs.metagenerationThe metageneration of the object.
gcs.ownerThe owner (uploader) of the object.
gcs.owner.typeThe ACL entity type of the uploader of the object.
gcs.uriThe URI of the object as a string.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

See Also:

ListGCSBucket, PutGCSObject, DeleteGCSObject

\ No newline at end of file Added: nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html?rev=1811008&view=auto ============================================================================== --- nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html (added) +++ nifi/site/trunk/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.4.0/org.apache.nifi.processors.gcp.storage.ListGCSBucket/index.html Tue Oct 3 13:30:16 2017 @@ -0,0 +1 @@ +ListGCSBucket

ListGCSBucket

Description:

Retrieves a listing of objects from an GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.

Tags:

google cloud, google, storage, gcs, list

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values.

NameDefault ValueAllowable ValuesDescription
GCP Credentials Provider ServiceController Service API:
GCPCredentialsService
Implementation: GCPCredentialsControllerService
The Controller Service used to obtain Google Cloud Platform credentials.
Project IDGoogle Cloud Project ID
Number of retries6How many retry attempts should be made before routing to the failure relationship.
BucketBucket of the object.
PrefixThe prefix used to filter the object list. In most cases, it should end with a forward slash ('/').
Use Generationsfalse
  • true
  • false
Specifies whether to use GCS Generations, if applicable. If false, only the latest version of each object will be returned.

Relationships:

NameDescription
successFlowFiles are routed to this relationship after a successful Google Cloud Storage ope ration.

Reads Attributes:

None specified.

Writes Attributes:

< /tr>
NameDescription
filenameThe name of the file
gcs.bucketBucket of the object.
gcs.keyName of the object.
gcs.sizeSize of the object.
gcs.cache.controlData cache control of the object.
gcs.component.countThe number of components which make up the object.
gcs.content.dispositionThe data content disposition of the object.
gcs.content.encodingThe content encoding of the object.
gcs.content.languageThe content language of the object.
mime.typeThe MIME/Content-Type of the object
gcs.crc32cThe CRC32C checksum of object's data, encoded in base64 in big-endian order.
gcs.create.timeThe creation time of the object (milliseconds)
gcs.update.timeThe last modification time of the object (milliseconds)
gcs.encryption.algorithmThe algorithm used to encrypt the object.
gcs.encryption.sha256The SHA256 hash of the key used to encrypt the object
gcs.etagThe HTTP 1.1 Entity tag for the object.
gcs.generated.idThe service-generated for the object
gcs.generationThe data generation of the object.
gcs.md5The MD5 hash of the object's data encoded in base64.
gcs.media.linkThe media download link to the object.
gcs.metagenerationThe metageneration of the object.
gcs.ownerThe owner (uploader) of the object.
gcs.owner.typeThe ACL entity type of the uploader of the object.
gcs.uriThe URI of the object as a string.

State management:

ScopeDescription
CLUSTERAfter performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

See Also:

PutGCSObject, DeleteGCSObject, FetchGCSObject

\ No newline at end of file