metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: No data in HDFS at /apps/metron/indexing/indexed after complete deployment of Full Development VM
Date Fri, 09 Feb 2018 15:45:47 GMT
Related:  The documentation
<https://github.com/apache/metron/tree/master/metron-platform/metron-indexing#indexing-architecture>
should probably be updated to account for METRON-1302.

On Fri, Feb 9, 2018 at 8:18 AM R K Sharma <rksun78@gmail.com> wrote:

> Thanks Michael... I can see similar information for my setup as well. What
> I could notice is that MySQL service is not running on my machine Which I
> think could be an issue for kibana. What's your view ?
>
> Regards
> RK Sharma
>
> On Thu, Feb 8, 2018 at 9:11 PM, Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> We now have 2 topologies for indexing - random access and batch. Double
>> check that both are currently running - our full dev environment is pretty
>> full with resources currently.
>> random_access_indexing
>> batch_indexing
>>
>> random_access_indexing is responsible for getting data into
>> Elasticsearch. You can also check ES has indexes by going into Ambari ->
>> Elasticsearch -> Quick Links -> Elasticsearch Indexes. You should see
>> something like the following:
>>
>> health status index                     uuid                   pri rep docs.count
docs.deleted store.size pri.store.size
>> yellow open   .kibana                   qbpdYf_RTMa_Rd2dB9q7oA   1   1         44
           0      120kb          120kb
>> yellow open   bro_index_2018.02.06.22   -FiQxEGEQtSec0sC4oGAFA   5   1       7990
           0     12.8mb         12.8mb
>> yellow open   bro_index_2018.02.06.23   AS4DHjrBQNyFrzDOxpGFeQ   5   1       8100
           0     12.7mb         12.7mb
>> yellow open   snort_index_2018.02.06.20 Sxg-JGI3SAeXdg-V11BNkg   5   1       7530
           0     11.9mb         11.9mb
>> yellow open   bro_index_2018.02.06.18   U1RTmFnpTCCDAicwWxc7Mg   5   1       4640
           0        8mb            8mb
>>
>> ...
>>
>>
>>
>>
>> On Thu, Feb 8, 2018 at 3:19 AM, R K Sharma <rksun78@gmail.com> wrote:
>>
>>> Thanks Ryan...I see some data fro Snort & Bro sensors. Another  problem
>>> which I have is that there is no information from Kibana dashboard.Do I
>>> need to deploy some additional component to populate kibana ?
>>>
>>> Regards
>>> RK Sharma
>>>
>>> On Wed, Feb 7, 2018 at 3:38 PM, Ryan Merriman <merrimanr@gmail.com>
>>> wrote:
>>>
>>>> I think you need to go one level deeper, those are directories.  Here
>>>> is what I see in my dev environment:
>>>>
>>>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed
>>>> Found 2 items
>>>> drwxrwxr-x   - storm hadoop          0 2018-02-07 01:20
>>>> /apps/metron/indexing/indexed/bro
>>>> drwxrwxr-x   - storm hadoop          0 2018-02-07 01:20
>>>> /apps/metron/indexing/indexed/snort
>>>>
>>>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed/bro
>>>> Found 1 items
>>>> -rw-r--r--   1 storm hadoop   12842043 2018-02-07 01:20
>>>> /apps/metron/indexing/indexed/bro/enrichment-hdfsIndexingBolt-3-0-1517966421778.json
>>>>
>>>> On Wed, Feb 7, 2018 at 3:58 AM, R K Sharma <rksun78@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>       I have deployed Full Development VM on Virtual Box and all
>>>>> services including metron, kafka, storm etc. are started. However, when
I
>>>>> check if there is some data written into HDFS (
>>>>> /apps/metron/indexing/indexed/yaf|bro|snort ) for any data sources, I
>>>>> don't see any data. Hereby below is output.
>>>>>
>>>>>
>>>>>     [vagrant@node1 bin]$ hdfs dfs -ls /apps/metron/indexing/indexed/
>>>>> Found 3 items
>>>>> drwxrwxr-x   - storm hadoop          0 2018-02-06 13:03
>>>>> /apps/metron/indexing/in
>>>>>                                                                  dexed/bro
>>>>> drwxrwxr-x   - storm hadoop          0 2018-01-31 13:35
>>>>> /apps/metron/indexing/in
>>>>>                                                                  dexed/error
>>>>> drwxrwxr-x   - storm hadoop          0 2018-02-07 04:53
>>>>> /apps/metron/indexing/in
>>>>>                                                                  dexed/snort
>>>>>
>>>>> On other-hand, I can see sensors (Snort & Bro) started on
>>>>> http://node1:4200 and is having some throughput, although very low.
>>>>> Hereby below is sensor status.
>>>>>
>>>>>
>>>>> GrokWebSphere Stopped - -
>>>>> jsonMap JSONMap Stopped - -
>>>>> squid Grok Stopped - -
>>>>> snort Snort Running 3.862s 1.89kb/s
>>>>> asa Asa Stopped - -
>>>>> bro Bro Running 4.25s 1.94kb/s
>>>>> yaf Grok Running 0s 0kb/s
>>>>> Can anybody guide me what should I check to ensure sensors produce
>>>>> data and HDFS should be populated with this data ?
>>>>>
>>>>> Thanks & Regards
>>>>> RK Sharma
>>>>>
>>>>>
>>>>
>>>
>>
> --

Jon

Mime
View raw message