eagle-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MyGmail <amriv...@gmail.com>
Subject Re: Instructions on installing Eagle 0.5 on HDP 2.5
Date Sat, 18 Mar 2017 08:24:55 GMT
Hi Jayesh,

I'm sorry, but I can't change storm version. HDP2.5 includes 1.0.1.

Any workaround?

Regards,
Andrey

> 18 марта 2017 г., в 5:53, Jayesh Senjaliya <jaysen@apache.org> написал(а):
> 
> Hi Markovich,
> 
> Eagle 0.5 is well supported with storm 0.9.3  , can you please try using that ?
> 
> or do you have to use it with storm 1.x ?
> 
> - Jayesh
> 
> 
>> On Fri, Mar 17, 2017 at 6:50 AM, Markovich <amrivkin@gmail.com> wrote:
>> Hello eagle users and dev,
>> 
>> I'm stuck on installing eagle on HDP2.5 cluster with jdk 1.8.0_101.
>> 
>> Here is my service versions and my cluster is secured using kerberos + ranger. 
>> 
>> HDFS	2.7.3
>> Hive	1.2.1000
>> Storm	1.0.1
>> Kafka	0.10.0
>> Kerberos	1.10.3-10
>> 
>> Here is what I've done already:
>> 
>> 1) Downloaded latest eagle from github (version 0.5 snapshot).
>> 2) Builded it using mvn clean package -DskipTests 
>> [INFO] BUILD SUCCESS
>> [INFO] ------------------------------------------------------------------------
>> [INFO] Total time: 16:11 min
>> [INFO] Finished at: 2017-03-17T16:00:45+03:00
>> [INFO] Final Memory: 183M/1755M
>> [INFO] ------------------------------------------------------------------------
>> 
>> 3) Moved tarball to /usr/hdp/current/ and extracted to eagle
>> 4) Changed conf/eagle.conf for my cluster:
>> zkQuorum
>> zookeeperZnodeParent = "/hbase-secure"
>> metadata -> jdbc -> user, pass and host
>> nimbusHost
>> 5) Launched bin/eagle-env.sh and bin/eagle-server start
>> 
>> Eagle started on 9090 port. 
>> 6) In Web UI entered SiteId
>> 7) Selected install on HDFS Audit Log Monitor and chenged General settings (Kafka
hosts) and Advanced fs.defaultFS. Execution Mode - cluster
>> 8) Created hdfs_audit_log_{SITE_ID}, hdfs_audit_log_enriched_{SITE_ID} 
>> 9) Launched Logstash to write to hdfs_audit_log_{SITE_ID}
>> 10) Checked, logs are pushed in kafka
>> 11) Appliaction is initialized
>> 12) Tried to start this appliaction:
>> 
>> INFO  [2017-03-17 13:28:45,926] org.apache.eagle.dataproc.impl.storm.kafka.KafkaSpoutProvider:
Use topic : hdfs_audit_log, zkConnection : localhost:2181 , fetchSize : 1048576
>> WARN  [2017-03-17 13:28:46,216] org.apache.eagle.app.messaging.KafkaStreamProvider:
Using default shared sink topic dataSinkConfig.topic: hdfs_audit_event
>> INFO  [2017-03-17 13:28:46,331] org.apache.eagle.app.environment.impl.StormExecutionRuntime:
Starting HDFS_AUDIT_LOG_MONITOR_APP_MYSITE
>>  (org.apache.eagle.security.auditlog.HdfsAuditLogApplication), mode: CLUSTER
>> INFO  [2017-03-17 13:28:46,332] org.apache.eagle.app.environment.impl.StormExecutionRuntime:
Overriding application.storm.nimbusHost = demo5
>> INFO  [2017-03-17 13:28:46,332] org.apache.eagle.app.environment.impl.StormExecutionRuntime:
Overriding application.storm.nimbusThriftPort = 6627
>> INFO  [2017-03-17 13:28:46,332] org.apache.eagle.app.environment.impl.StormExecutionRuntime:
Submitting as cluster mode ...
>> 
>> Nothing in Storm UI. Nothing in logs.
>> Tried to change KafkaSpoutProvider to normal with SITE_ID. -> Nothing changed.

>> 
>> I think I need to enter some kerberos related configs for storm...
>> 
>> 13) Ok, changed  Execution Mode - Local. 
>> 14) Started. Status in UI changed to starting. In logs a lot of activity, but also
errors:
>> 
>> WARN  [2017-03-17 13:33:40,952] storm.kafka.KafkaUtils: there are more tasks than
partitions (tasks: 2; partitions: 1), some tasks will be idle
>> INFO  [2017-03-17 13:33:40,952] storm.kafka.KafkaUtils: Task [1/2] assigned [Partition{host=null:-1,
partition=0}]
>> INFO  [2017-03-17 13:33:40,952] storm.kafka.ZkCoordinator: Task [1/2] Deleted partition
managers: []
>> INFO  [2017-03-17 13:33:40,952] storm.kafka.ZkCoordinator: Task [1/2] New partition
managers: [Partition{host=null:-1, partition=0}]
>> INFO  [2017-03-17 13:33:40,990] storm.kafka.PartitionManager: Read partition information
from: /consumers/hdfs_audit_log_tuskpro/eagleConsumer/partition_0  --> null
>> ERROR [2017-03-17 13:33:41,047] backtype.storm.util: Async loop died!
>> ! java.lang.NullPointerException: null
>> ! at org.apache.kafka.common.utils.Utils.formatAddress(Utils.java:312) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer$$anonfun$disconnect$1.apply(SimpleConsumer.scala:49)
~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer$$anonfun$disconnect$1.apply(SimpleConsumer.scala:49)
~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.utils.Logging$class.debug(Logging.scala:52) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer.debug(SimpleConsumer.scala:30) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer.disconnect(SimpleConsumer.scala:49) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:82) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68)
~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:74) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:64) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at storm.kafka.PartitionManager.<init>(PartitionManager.java:89) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! ... 6 common frames omitted
>> ! Causing: java.lang.RuntimeException: java.lang.NullPointerException
>> ! at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:103) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) ~[eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565)
~[storm-core-0.9.3.jar:0.9.3]
>> ! at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) ~[storm-core-0.9.3.jar:0.9.3]
>> ! at clojure.lang.AFn.run(AFn.java:24) [eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>> 
>> ....
>> 
>> ERROR [2017-03-17 13:33:41,079] backtype.storm.util: Halting process: ("Worker died")
>> ! java.lang.RuntimeException: ("Worker died")
>> ! at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.3.jar:0.9.3]
>> ! at clojure.lang.RestFn.invoke(RestFn.java:423) [eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at backtype.storm.daemon.worker$fn__3808$fn__3809.invoke(worker.clj:452) [storm-core-0.9.3.jar:0.9.3]
>> ! at backtype.storm.daemon.executor$mk_executor_data$fn__3274$fn__3275.invoke(executor.clj:240)
[storm-core-0.9.3.jar:0.9.3]
>> ! at backtype.storm.util$async_loop$fn__464.invoke(util.clj:473) [storm-core-0.9.3.jar:0.9.3]
>> ! at clojure.lang.AFn.run(AFn.java:24) [eagle-topology-0.5.0-SNAPSHOT-assembly.jar:na]
>> ! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>> 
>> ERROR [2017-03-17 13:34:10,013] org.apache.eagle.security.enrich.DataEnrichJob: Fail
to load sensitivity data
>> ! java.net.ConnectException: Connection refused
>> ! at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_101]
>> ! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[na:1.8.0_101]
>> ! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[na:1.8.0_101]
>> ! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_101]
>> ! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_101]
>> ! at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_101]
>> ! at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_101]
>> ! at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.8.0_101]
>> ! at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.8.0_101]
>> ! at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[na:1.8.0_101]
>> ! at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[na:1.8.0_101]
>> ! at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[na:1.8.0_101]
>> ! at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1169)
~[na:1.8.0_101]
>> ! at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105)
~[na:1.8.0_101]
>> ! at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999)
~[na:1.8.0_101]
>> ! at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:933)
~[na:1.8.0_101]
>> ! at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513)
~[na:1.8.0_101]
>> ! at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
~[na:1.8.0_101]
>> ! at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_101]
>> ! at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
~[jersey-client-1.19.1.jar:1.19.1]
>> ! at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
~[jersey-client-1.19.1.jar:1.19.1]
>> ! ... 12 common frames omitted
>> ! Causing: com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException:
Connection refused
>> ! at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
~[jersey-client-1.19.1.jar:1.19.1]
>> ! at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
~[jersey-client-1.19.1.jar:1.19.1]
>> ! at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.1.jar:1.19.1]
>> 
>> Ui wasn't accessable, but server was still running. 
>> It was failing again and again.
>> 
>> 13) Restarted server
>> 14) Nothing was saved. Again fresh install. UI asks for site_id.
>> 15) Checked Hbase and MySQL Server - all is empty.
>> 
>> 
>> So can someone please help me to get started with Eagle on my cluster?
>> 
>> 
>> Also here is Logstash info:
>> logstash-5.2.2
>> 
>> 
>>   output {                                                                      
                         
>>                                                                                 
                          
>>       if [type] == "hdp-nn-audit" {                                             
                         
>>         kafka   {                                                               
                         
>>                 codec => plain {format => "%{message}"}                   
                                
>>                 bootstrap_servers => "demo4:6667"                            
                     
>>                 topic_id => "hdfs_audit_log_tuskpro"                         
                             
>>                 security_protocol => "SASL_PLAINTEXT"                        
                             
>>                 sasl_kerberos_service_name => "kafka"                        
                             
>>                 jaas_path => "/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"
               
>>                 kerberos_config => "/etc/krb5.conf"                          
                            
>>                 client_id => "hdp-nn-audit"                                  
                            
>>                 message_key => "%{user}"                                     
                             
>>                 }                                                               
                         
>>                                   }                                             
                         
>>          }       
>> 
>> 
>> Also I don't know why, but logstash is not using kafka_client_jaas. 
>> There is info about sasl_kerberos_service_name , but without this property logstash
is not working. 
>> 
>> ·
>> KafkaClient {                                                                   
                                                                                        
>>    com.sun.security.auth.module.Krb5LoginModule required                        
                                                                                        
>>    useTicketCache=true                                                          
                                                                                       
>>    renewTicket=true                                                             
                                                                                       
>>    serviceName="kafka";                                                         
                                                                                       
>> };                                                                              
                                                                                        
>> Client {                                                                        
                                                                                        
>>    com.sun.security.auth.module.Krb5LoginModule required                        
                                                                                        
>>    useTicketCache=true                                                          
                                                                                       
>>    renewTicket=true                                                             
                                                                                       
>>    serviceName="zookeeper";                                                     
                                                                                        
>> };                                                                              
                                                                                        
>>                                     
>> 
>> Regards,
>> Andrey                                                                          
                                                          
>>               
>> 
> 

Mime
View raw message