metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anil Donthireddy <Anil.Donthire...@sstech.us>
Subject RE: Unable to see alerts in metron alert UI with solr
Date Thu, 23 Aug 2018 14:01:42 GMT
I see the below error in the logs. The url that is tried http://<StormInstance>:8983/solr/demo_unix_shard1_replica1<http://%3cStormInstance%3e:8983/solr/demo_unix_shard1_replica1>
seems doesn’t work. When I tried from browser http://<StormInstance>:8983/solr/#/demo_unix_shard1_replica1<http://%3cStormInstance%3e:8983/solr/#/demo_unix_shard1_replica1>
worked.

I see the collections list it tries to get from solr is fine. Issue seems to be the url tried
to access the shard should be changed.

***************Logs***********
18/08/23 09:55:17 ERROR impl.CloudSolrClient: Request to collection demo_asa,demo_windowsnxlog,demo_bluecoat,demo_unix,demo_sysmon,bro,yaf,demo_msexchange,demo_wgdhcp,demo_wgtraffic,snort,metaalert
failed due to (503) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at http://<StormInstance>:8983/solr/demo_unix_shard1_replica1: no
servers hosting shard: snort_shard1, retry? 0
18/08/23 09:55:17 ERROR dao.SolrSearchDao: No live SolrServers available to handle this request:[http://<StormInstance>:8983/solr/demo_unix_shard1_replica1]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle
this request:[http://<StormInstance>:8983/solr/demo_unix_shard1_replica1]
                at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
*****************************


From: Ryan Merriman [mailto:merrimanr@gmail.com]
Sent: Thursday, August 23, 2018 7:12 PM
To: user@metron.apache.org
Subject: Re: Unable to see alerts in metron alert UI with solr

You can pass credentials in curl with the -u flag:  curl -u user:password -X GET --header
'Accept: application/json' 'http://{rest host}:8082/api/v1/sensor/indexing/config/list/indices/solr'.
 I prefer to use Swagger.

We need to dig into the 500 response.  An error message should be included in the response
or you can try running it separately in Swagger.  I would also check the log at /var/log/metron/metron-rest.log.



On Thu, Aug 23, 2018 at 6:35 AM Anil Donthireddy <Anil.Donthireddy@sstech.us<mailto:Anil.Donthireddy@sstech.us>>
wrote:
@Ryan: I tried the api call from swagger UI. I can see the list of indices is being returned
fine as present in Solr.

From: Anil Donthireddy [mailto:Anil.Donthireddy@sstech.us<mailto:Anil.Donthireddy@sstech.us>]
Sent: Thursday, August 23, 2018 5:03 PM
To: user@metron.apache.org<mailto:user@metron.apache.org>
Subject: RE: Unable to see alerts in metron alert UI with solr

Hi,

Thanks a lot for the inputs.

@Stefan: We don’t have ranger setup for access permissions.

@Ryan: Below are the issues I observed from your suggestion
                For the search request in network tab, I see the response is “500 Internal
Server Error”
                In the request payload I see all the params are emtpy as below.
                                {indices: [], facetFields: [], query: "*", from: 0, size:
25}
       When I try to run the curl command, I am getting unauthorized error as below. How can
I pass credentials for authentication along with the request or how can I authenticate via
curl command and store the jessionid as cookie and pass through your curl command?
              {"timestamp":"2018-08-23 11:13:22","status":401,"error":"Unauthorized","message":"Unauthorized","path":"/api/v1/sensor/indexing/config/list/indices/solr"}


Thanks,
Anil.

From: Ryan Merriman [mailto:merrimanr@gmail.com]
Sent: Thursday, August 23, 2018 3:41 AM
To: user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Re: Unable to see alerts in metron alert UI with solr

The first thing I would do is figure out which indices are included in the Alerts UI.  If
you open the network tab in your browser there should be a "search" request sent to REST.
 There is an "indices" property that will have a list of indices include in the search.  If
it's empty ("[]") then this call should tell you what the default indices are:

curl -X GET --header 'Accept: application/json' 'http://{rest host}:8082/api/v1/sensor/indexing/config/list/indices/solr'

Verify your collection is in the list and we can go from there.

On Wed, Aug 22, 2018 at 4:47 PM Stefan Kupstaitis-Dunkler <stefan.dun@gmail.com<mailto:stefan.dun@gmail.com>>
wrote:
Hey Anil,

do you use any kind of policies to prevent access by the metron user? Do you use Ranger to
manage access policies?

The user running the Metron REST service (normally "metron") needs to have access to all collections.

Check the (Ranger) audit log to see if access to the collections is denied/allowed.

If you use Ranger + Solr with Metron you might run into a current Solr plugin bug. I've run
into that a few weeks ago and provided a workaround here:

https://community.hortonworks.com/articles/203326/searching-in-multiple-collections-in-one-query-wit.html<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcommunity.hortonworks.com%2farticles%2f203326%2fsearching-in-multiple-collections-in-one-query-wit.html&c=E,1,9dD6y1u3ennoQvuZCZ45l9VNmxiTrI6YNYlhGiC0_1XXX0pMj7qHTj6yBi6drFMrosDMlFfiJthaQqZTUjr6MVXrJwuRPuh_eNO6-s9QM6lnTnhpPaw,&typo=1>

Best,
Stefan


On Wed, Aug 22, 2018, 18:40 Anil Donthireddy <Anil.Donthireddy@sstech.us<mailto:Anil.Donthireddy@sstech.us>>
wrote:
Yes, necessary two fields are set properly as below

"source.type.field" : "source.type",
  "threat.triage.score.field" : "threat.triage.score"


From: Anand Subramanian [mailto:asubramanian@hortonworks.com<mailto:asubramanian@hortonworks.com>]
Sent: Wednesday, August 22, 2018 10:07 PM
To: user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Re: Unable to see alerts in metron alert UI with solr

Hey Anil,

You might also want to check if the “source.type.field” is set to “source.type” (and
NOT source:type) in global config.

See:
https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/README.md#global-configuration-properties

HTH,
Anand

From: Anil Donthireddy <Anil.Donthireddy@sstech.us<mailto:Anil.Donthireddy@sstech.us>>
Reply-To: "user@metron.apache.org<mailto:user@metron.apache.org>" <user@metron.apache.org<mailto:user@metron.apache.org>>
Date: Wednesday, August 22, 2018 at 10:04 PM
To: "user@metron.apache.org<mailto:user@metron.apache.org>" <user@metron.apache.org<mailto:user@metron.apache.org>>
Subject: RE: Unable to see alerts in metron alert UI with solr

Hi Stefan,

I can see the events being written to solr collection. There are no issues upto storing data
to Solr. There are events which have is_alert=true. But in the metron alerts UI I dint see
any events.

Thanks,
Anil.

From: Stefan Kupstaitis-Dunkler [mailto:stefan.dun@gmail.com]
Sent: Wednesday, August 22, 2018 9:52 PM
To: user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Re: Unable to see alerts in metron alert UI with solr

Hi Anil,

The alerts UI just queries your Solr collections and displays them

Things you could do:
•         You could check the collection the event is supposed to be directly
•         If it’s not there you could check the Solr “error” collection.
•         Use the Kafka console consumer and
o   check the parser topic (you define the name in the management ui) if events are being
written to it
o   check the “enrichments” topic if events are being written to it
o   check the “indexing” topic if events are being written to it.

Hope that helps as a start.

Best,
Stefan




From: Anil Donthireddy <Anil.Donthireddy@sstech.us<mailto:Anil.Donthireddy@sstech.us>>
Reply-To: "user@metron.apache.org<mailto:user@metron.apache.org>" <user@metron.apache.org<mailto:user@metron.apache.org>>
Date: Wednesday, 22. August 2018 at 18:07
To: "user@metron.apache.org<mailto:user@metron.apache.org>" <user@metron.apache.org<mailto:user@metron.apache.org>>
Subject: Unable to see alerts in metron alert UI with solr

Hi,

I have created a collection in solr for my source as documented in metron-slor readme in git
to enable solr and writing data to solr. I see the data is being loaded to Solr for my new
source and is_alert is set to tru. But still I am not seeing any data in metron alerts UI.

I checked the metron alerts UI logs, but could not find any error logs. I am blocked to proceed
further to display alerts in the metron alerts ui. May I get any suggestions to debug the
issue.

Thanks,
Anil.
--
Stefan Kupstaitis-Dunkler
https://datahovel.com/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fdatahovel.com%2f&c=E,1,rgB6RnyVWsWRLyz4Pfv-ZJ0kCNc2qsDK9f0qPgKhSLPGI8cUtFb3JzjKtCJ0C5P5533BhkBS_k9oAum3K-N7e8fYNQHryxJrRnCjR5uHcw,,&typo=1>
https://www.meetup.com/Hadoop-User-Group-Vienna/
https://twitter.com/StefanDunkler
Mime
View raw message