flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balázs Donát Bessenyei <bes...@cloudera.com>
Subject Re: Review Request 51182: FLUME-2954: make raw data appearing in log messages explicit
Date Wed, 24 Aug 2016 10:17:35 GMT


> On Aug. 24, 2016, 10:17 a.m., Balázs Donát Bessenyei wrote:
> >

Otherwise, LGTM


- Balázs Donát


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51182/#review146634
-----------------------------------------------------------


On Aug. 24, 2016, 8:46 a.m., Attila Simon wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51182/
> -----------------------------------------------------------
> 
> (Updated Aug. 24, 2016, 8:46 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2954
>     https://issues.apache.org/jira/browse/FLUME-2954
> 
> 
> Repository: flume-git
> 
> 
> Description
> -------
> 
> --------------------------------------------------------------------------------
> flume-ng-channel                              ---
>   flume-jdbc-channel                          ---
>     JdbcChannelProviderImpl#98                <- fail properties <REMOVED>
>     JdbcChannelProviderImpl#261 #431          <- fail properties: jdbc url might include
password <KEPT><FOLLOWUP IN JIRA>
>   flume-kafka-channel                         ---
>     KafkaChannel#230 #253                     <- fail properties <REMOVED>
> --------------------------------------------------------------------------------
> flume-ng-configuration                        ---
>   FlumeConfiguration#315 #372                 <- fail properties <DRIVE BY PROPERTY>
> --------------------------------------------------------------------------------
> flume-ng-core                                 ---
>   SyslogAvroEventSerializer#150               <- fail data: SyslogEvent.message gets
logged <DRIVE BY PROPERTY>
>   GangliaServer#224 #245                      <- safe data: only flume component metrics
data <KEPT>
>   LoggerSink#95                               <- fail data: on purpose <KEPT>
>   AvroSource#347                              <- fail data: log whole message <DRIVE
BY PROPERTY>
>   MultiportSyslogTCPSource#360                <- fail data: log whole message <DRIVE
BY PROPERTY>
>   BLOBHandler#70                              <- fail data: logs http request headers
<DRIVE BY PROPERTY>
> -------------------------------------------------------------------q-------------
> flume-ng-embedded-agent                       ---
>   EmbeddedAgent#155                           <- fail properties: printing all config
<DRIVE BY PROPERTY>
> --------------------------------------------------------------------------------
> flume-ng-sinks                                ---
>   flume-hive-sink                             ---
>     HiveEndPoint has an URI field.            <- fail properties <KEPT><FOLLOWUP
IN JIRA>
>         It may contain private data
>         (URI string may contain password) as it is
>         excessively logged within this module.
>         Appears in HiveSink#298 #342 #400 #403 #428,
>         HiveWriter#210 #319 #330 #337 #353 #365 #368 #407...)
>         HiveEndPoint is also attached to exception logs as well
>   flume-ng-hbase-sink                         ---
>     AsyncHBaseSink#641                        <- safe data: error details gets logged
in case of failure <KEPT>
>   flume-ng-kafka-sink                         ---
>     KafkaSink#179                             <- fail data: log whole message <REMOVED>
>     KafkaSink#304                             <- fail properties <REMOVED>
>   flume-ng-morphline-solr-sink                ---
>     BlobHandler#98 #113                       <- fail data: log http request headers
<DRIVE BY PROPERTY>
>     MorphlineSink#139                         <- fail data: logs event <DRIVE BY
PROPERTY>
> --------------------------------------------------------------------------------
> flume-ng-sources                              ---
>   flume-kafka-source                          ---
>     KafkaSource#247                           <- fail data: log whole <DRIVE BY
PROPERTY>
>   flume-twitter-source                        ---
>     TwitterSource#110-113                     <- fail properties <REMOVED>
> --------------------------------------------------------------------------------
> 
> 
> Diffs
> -----
> 
>   conf/flume-env.ps1.template 8bf535a 
>   conf/flume-env.sh.template c8b660f 
>   flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
845b794 
>   flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
90e3288 
>   flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
9b3a434 
>   flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 8b9b956 
>   flume-ng-core/src/main/java/org/apache/flume/source/MultiportSyslogTCPSource.java b9f2438

>   flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java e24d4c6 
>   flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java
05af3b1 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst 5e677c6 
>   flume-ng-embedded-agent/src/main/java/org/apache/flume/agent/embedded/EmbeddedAgent.java
ad3e138 
>   flume-ng-sdk/src/main/java/org/apache/flume/util/LogRawDataUtil.java PRE-CREATION 
>   flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
9453546 
>   flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java
ca7614a 
>   flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/MorphlineSink.java
f7a73f3 
>   flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
90e4715 
>   flume-ng-sources/flume-twitter-source/src/main/java/org/apache/flume/source/twitter/TwitterSource.java
f5c8328 
> 
> Diff: https://reviews.apache.org/r/51182/diff/
> 
> 
> Testing
> -------
> 
> compiles, site builds, all unit test passes, distribution target handles the system properties
as expected:
> bin/flume-ng agent --conf conf --conf-file ../../../../../flume-conf/flume-log.conf --name
a1 -Dflume.root.logger=DEBUG,console -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true
(with and without the extra properties)
> 
> 
> Thanks,
> 
> Attila Simon
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message