flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonghao Zou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3021) flume Elasticsearch 5.0 not support
Date Mon, 20 Mar 2017 14:30:42 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932726#comment-15932726
] 

Yonghao Zou commented on FLUME-3021:
------------------------------------

Hi, [~ytc301]
I tested with following configuration:

--------------------------------------------------------------------------------------------------------------------
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# source interceptors

a1.sources.r1.interceptors.i0.regex = ([^ ]*)\\s+-\\s+(.+?|-)\\s+\\[(.*)\\]\\s+\"([^ ]*)\\s+([^
]*)\\s+([^ |\"]*)\"\\s+(-|[0-9]*)\\s+(-|[0-9]*)\\s+\"(.+?|-)\"\\s+\"(.+?|-)\"\\s+\"(.+?|-)\"\\s+\"(.+?|-)\"\\s+(.+?|-)\\s+(.+?|-)\\s+(.+?|-)\\s+\"(.+?|-)\"\\s+(.+?|-)\\s+(.*)
a1.sources.r1.interceptors=i0 i3
a1.sources.r1.interceptors.i0.type=regex_extractor
a1.sources.r1.interceptors.i0.serializers = s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14
s15 s16 s17 s18
a1.sources.r1.interceptors.i0.serializers.s1.name = remote_addr
a1.sources.r1.interceptors.i0.serializers.s2.name = user
a1.sources.r1.interceptors.i0.serializers.s3.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
a1.sources.r1.interceptors.i0.serializers.s3.name = timestamp
a1.sources.r1.interceptors.i0.serializers.s3.pattern= yyyy-MM-dd'T'HH:mm:ssZ
a1.sources.r1.interceptors.i0.serializers.s4.name = request_methond
a1.sources.r1.interceptors.i0.serializers.s5.name = request_url
a1.sources.r1.interceptors.i0.serializers.s6.name = request_http
a1.sources.r1.interceptors.i0.serializers.s7.name = http_status
a1.sources.r1.interceptors.i0.serializers.s8.name = body_bytes_sent
a1.sources.r1.interceptors.i0.serializers.s9.name = referer
a1.sources.r1.interceptors.i0.serializers.s10.name = user_Agent
a1.sources.r1.interceptors.i0.serializers.s11.name = http_x_forwarded_for
a1.sources.r1.interceptors.i0.serializers.s12.name = gzip_ratio
a1.sources.r1.interceptors.i0.serializers.s13.name = request_time
a1.sources.r1.interceptors.i0.serializers.s14.name = bytes_sent
a1.sources.r1.interceptors.i0.serializers.s15.name = request_length
a1.sources.r1.interceptors.i0.serializers.s16.name = upstream_addr
a1.sources.r1.interceptors.i0.serializers.s17.name = upstream_status
a1.sources.r1.interceptors.i0.serializers.s18.name = upstream_response_time
a1.sources.r1.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder
a1.sources.r1.interceptors.i3.hostHeader = host


# Describe the sink
# a1.sinks.k1.type = logger

a1.sinks.k1.type = elasticsearch
a1.sinks.k1.hostNames = 127.0.0.1:9300
a1.sinks.k1.indexName = foo_index
a1.sinks.k1.indexType = bar_type
a1.sinks.k1.clusterName = elasticsearch
a1.sinks.k1.batchSize = 500
a1.sinks.k1.ttl = 5d
a1.sinks.k1.serializer.outputCharset = UTF-8
a1.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer

# a1.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

--------------------------------------------------------------------------------------------------------------------


Then I telnet to localhost:44444 and input your data which is

--------------------------------------------------------------------------------------------------------------------
119.12.11.22 - - [2016-11-07T11:52:45+08:00] "GET /skin//frontend/default/images/transparAent.gif
HTTP/1.1" 304 0 "http://www.baidu.com/ti/account/amsders" "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36" "125.77.22.1" "-"
0.000 223 979 "192.168.2.101:8011" 304 0.000
--------------------------------------------------------------------------------------------------------------------

and it inserted into es successfully.

It seems your error's reason maybe some connection problems? I see ex like NodeNotConnectedException
happend.

And you can email to me directly with Chinese, I'll be happy to help you with.

> flume Elasticsearch 5.0 not support
> -----------------------------------
>
>                 Key: FLUME-3021
>                 URL: https://issues.apache.org/jira/browse/FLUME-3021
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.7.0
>            Reporter: tycho_yang
>            Assignee: Yonghao Zou
>             Fix For: 1.8.0
>
>         Attachments: FLUME-3021-0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message