From user-return-1939-archive-asf-public=cust-asf.ponee.io@metron.apache.org Fri Feb 9 16:46:11 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id AA94A180654 for ; Fri, 9 Feb 2018 16:46:11 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9A226160C4C; Fri, 9 Feb 2018 15:46:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 44F7C160C2E for ; Fri, 9 Feb 2018 16:46:10 +0100 (CET) Received: (qmail 82986 invoked by uid 500); 9 Feb 2018 15:46:09 -0000 Mailing-List: contact user-help@metron.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@metron.apache.org Delivered-To: mailing list user@metron.apache.org Received: (qmail 82976 invoked by uid 99); 9 Feb 2018 15:46:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Feb 2018 15:46:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id CF1F0C031C for ; Fri, 9 Feb 2018 15:46:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.88 X-Spam-Level: * X-Spam-Status: No, score=1.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id qOjEjnUwqjrj for ; Fri, 9 Feb 2018 15:46:06 +0000 (UTC) Received: from mail-qt0-f175.google.com (mail-qt0-f175.google.com [209.85.216.175]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5CE845F341 for ; Fri, 9 Feb 2018 15:46:06 +0000 (UTC) Received: by mail-qt0-f175.google.com with SMTP id i8so11038343qtj.9 for ; Fri, 09 Feb 2018 07:46:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=2DmImAu0tJN9hlF00AF/5fNvN70vnFuvN3MufNPn18k=; b=NeR1MmDIneqH7Ta3jB2ESF0r9P7pLI1j7nMtc7oznxOHN4KaVuzwfwKoc4S2f7n9UQ rbjMQinOrzBTrAt7Ln50lzkgi/PIteApJ3cEzALZMuJVywQ8/y6CaPingdQN9GE2dyzF m8pkY4eXYxRtQsCjyOS3s4l8ju273tfaN28k49Ox7ptPKm9GcqdiVkulzC/kZs+yj3Sp EbfFZULmi9bJduMZZ7ZpCuov/NAXm2Gqyde+6xwws4iCFoslLsLkQJUXhDCCPUH5w/5P MNWT8ZVrdatO5PoeHRcuYYAC2r7VlEVNCil3jtyD0AvXnujPbS8e8Er4qcJDAXS83pAi O9rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=2DmImAu0tJN9hlF00AF/5fNvN70vnFuvN3MufNPn18k=; b=Ht9rmh4WhGvlJGDba+EKdkKgmsiqVXXssNOy+dj2GA1zlztDAVvJUr3ppItRbP5S+G dyX0G95PmhipGaHK0FRxUSm/PMkMyo9oKlmpNLGDXH2g30sXzu3CtCJfeliFBKV+Z893 1F5Xo0r8UqhYFyKdPOvnMZBfx1JaKWHGCT/P7No5Ireh3EDTHKYlZLejxOVucLHvWkw1 iyhjNxMP3bEeO3UTbdQbaHirnxv0kkT7qPCICS9nBa8BOJ60zNpHyxN4SWRH03sIfoBP ZOcoT0g3LaKj5j7AqzXSOF5lNE0GZb5n3vY+o0wnKzVNXoIY1ZBMxJV75a9XOrOlvZC9 Yn2g== X-Gm-Message-State: APf1xPCfcVOgWOyCpcU9HgDQy1QWQe7bBiNWROit8XkTejKl8c1V6D+A b331CSuP57fba16rr4C34EdTEjsjjOm6HUIVigIHtA== X-Google-Smtp-Source: AH8x227M2+jqxdXGYbcqLRYfgUWa1rd4YjYxEknD/zfZ9/PdKsu+EAjeo5hkR12sDYbu8c/pEbyO+VSPoscdnGZEZXg= X-Received: by 10.237.48.106 with SMTP id 97mr5056950qte.48.1518191159507; Fri, 09 Feb 2018 07:45:59 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: "Zeolla@GMail.com" Date: Fri, 09 Feb 2018 15:45:47 +0000 Message-ID: Subject: Re: No data in HDFS at /apps/metron/indexing/indexed after complete deployment of Full Development VM To: user@metron.apache.org Content-Type: multipart/alternative; boundary="94eb2c0cabe8c382c40564c96c02" --94eb2c0cabe8c382c40564c96c02 Content-Type: text/plain; charset="UTF-8" Related: The documentation should probably be updated to account for METRON-1302. On Fri, Feb 9, 2018 at 8:18 AM R K Sharma wrote: > Thanks Michael... I can see similar information for my setup as well. What > I could notice is that MySQL service is not running on my machine Which I > think could be an issue for kibana. What's your view ? > > Regards > RK Sharma > > On Thu, Feb 8, 2018 at 9:11 PM, Michael Miklavcic < > michael.miklavcic@gmail.com> wrote: > >> We now have 2 topologies for indexing - random access and batch. Double >> check that both are currently running - our full dev environment is pretty >> full with resources currently. >> random_access_indexing >> batch_indexing >> >> random_access_indexing is responsible for getting data into >> Elasticsearch. You can also check ES has indexes by going into Ambari -> >> Elasticsearch -> Quick Links -> Elasticsearch Indexes. You should see >> something like the following: >> >> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size >> yellow open .kibana qbpdYf_RTMa_Rd2dB9q7oA 1 1 44 0 120kb 120kb >> yellow open bro_index_2018.02.06.22 -FiQxEGEQtSec0sC4oGAFA 5 1 7990 0 12.8mb 12.8mb >> yellow open bro_index_2018.02.06.23 AS4DHjrBQNyFrzDOxpGFeQ 5 1 8100 0 12.7mb 12.7mb >> yellow open snort_index_2018.02.06.20 Sxg-JGI3SAeXdg-V11BNkg 5 1 7530 0 11.9mb 11.9mb >> yellow open bro_index_2018.02.06.18 U1RTmFnpTCCDAicwWxc7Mg 5 1 4640 0 8mb 8mb >> >> ... >> >> >> >> >> On Thu, Feb 8, 2018 at 3:19 AM, R K Sharma wrote: >> >>> Thanks Ryan...I see some data fro Snort & Bro sensors. Another problem >>> which I have is that there is no information from Kibana dashboard.Do I >>> need to deploy some additional component to populate kibana ? >>> >>> Regards >>> RK Sharma >>> >>> On Wed, Feb 7, 2018 at 3:38 PM, Ryan Merriman >>> wrote: >>> >>>> I think you need to go one level deeper, those are directories. Here >>>> is what I see in my dev environment: >>>> >>>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed >>>> Found 2 items >>>> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20 >>>> /apps/metron/indexing/indexed/bro >>>> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20 >>>> /apps/metron/indexing/indexed/snort >>>> >>>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed/bro >>>> Found 1 items >>>> -rw-r--r-- 1 storm hadoop 12842043 2018-02-07 01:20 >>>> /apps/metron/indexing/indexed/bro/enrichment-hdfsIndexingBolt-3-0-1517966421778.json >>>> >>>> On Wed, Feb 7, 2018 at 3:58 AM, R K Sharma wrote: >>>> >>>>> Hi, >>>>> I have deployed Full Development VM on Virtual Box and all >>>>> services including metron, kafka, storm etc. are started. However, when I >>>>> check if there is some data written into HDFS ( >>>>> /apps/metron/indexing/indexed/yaf|bro|snort ) for any data sources, I >>>>> don't see any data. Hereby below is output. >>>>> >>>>> >>>>> [vagrant@node1 bin]$ hdfs dfs -ls /apps/metron/indexing/indexed/ >>>>> Found 3 items >>>>> drwxrwxr-x - storm hadoop 0 2018-02-06 13:03 >>>>> /apps/metron/indexing/in >>>>> dexed/bro >>>>> drwxrwxr-x - storm hadoop 0 2018-01-31 13:35 >>>>> /apps/metron/indexing/in >>>>> dexed/error >>>>> drwxrwxr-x - storm hadoop 0 2018-02-07 04:53 >>>>> /apps/metron/indexing/in >>>>> dexed/snort >>>>> >>>>> On other-hand, I can see sensors (Snort & Bro) started on >>>>> http://node1:4200 and is having some throughput, although very low. >>>>> Hereby below is sensor status. >>>>> >>>>> >>>>> GrokWebSphere Stopped - - >>>>> jsonMap JSONMap Stopped - - >>>>> squid Grok Stopped - - >>>>> snort Snort Running 3.862s 1.89kb/s >>>>> asa Asa Stopped - - >>>>> bro Bro Running 4.25s 1.94kb/s >>>>> yaf Grok Running 0s 0kb/s >>>>> Can anybody guide me what should I check to ensure sensors produce >>>>> data and HDFS should be populated with this data ? >>>>> >>>>> Thanks & Regards >>>>> RK Sharma >>>>> >>>>> >>>> >>> >> > -- Jon --94eb2c0cabe8c382c40564c96c02 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Related:=C2=A0 The doc= umentation should probably be updated to account for METRON-1302.
=
On Fri, Feb 9, 2018 at 8:18= AM R K Sharma <rksun78@gmail.com> wrote:

On Thu, Feb 8, 2018 at 9:11 PM, Michael Miklavcic <michael.miklavcic@gmail.com> wrote:
We now have 2 topologies for indexing - r= andom access and batch. Double check that both are currently running - our = full dev environment is pretty full with resources currently.
random_ac= cess_indexing
batch_indexing

random_= access_indexing is responsible for getting data into Elasticsearch. You can= also check ES has indexes by going into Ambari -> Elasticsearch -> Q= uick Links -> Elasticsearch Indexes. You should see something like the f= ollowing:
health status index                 =
    uuid                   pri rep docs.count docs.deleted store.size pri.s=
tore.size
yellow open   .kibana                   qbpdYf_RTMa_Rd2dB9q7oA   1   1     =
    44            0      120kb          120kb
yellow open   bro_index_2018.02.06.22   -FiQxEGEQtSec0sC4oGAFA   5   1     =
  7990            0     12.8mb         12.8mb
yellow open   bro_index_2018.02.06.23   AS4DHjrBQNyFrzDOxpGFeQ   5   1     =
  8100            0     12.7mb         12.7mb
yellow open   snort_index_2018.02.06.20 Sxg-JGI3SAeXdg-V11BNkg   5   1     =
  7530            0     11.9mb         11.9mb
yellow open   bro_index_2018.02.06.18   U1RTmFnpTCCDAicwWxc7Mg   5   1     =
  4640            0        8mb            8mb
...
=


=
On Thu, Feb 8, 2018 at 3:19 AM, R K Sharma <= span dir=3D"ltr"><rksun78@gmail.com> wrote:
=
Thanks Ryan...I see some data fro Snort & Br= o sensors. Another=C2=A0 problem which I have is that there is no informati= on from Kibana dashboard.Do I need to deploy some additional component to p= opulate kibana ?

Regards
RK Sharma

On Wed, Feb 7, 2018 at 3:38 PM, Ryan Merr= iman <merrimanr@gmail.com> wrote:
I think you need to go one level deeper, those a= re directories.=C2=A0 Here is what I see in my dev environment:

[root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed
Found 2 items=
drwxrwxr-x =C2=A0 - storm hadoop =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 20= 18-02-07 01:20 /apps/metron/indexing/indexed/bro
drwxrwxr-x =C2=A0 - sto= rm hadoop =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 2018-02-07 01:20 /apps/metron= /indexing/indexed/snort

[root@node1 ~]# hdfs dfs -= ls /apps/metron/indexing/indexed/bro
Found 1 items
-rw-r--r-- = =C2=A0 1 storm hadoop =C2=A0 12842043 2018-02-07 01:20 /apps/metron/indexin= g/indexed/bro/enrichment-hdfsIndexingBolt-3-0-1517966421778.json

On Wed, Feb 7, 2018 at 3:58 AM, R K Sharma <rksun78@gmail.com> wrote:
Hi,
=C2= =A0 =C2=A0 =C2=A0 I have deployed Full Development VM on Virtual Box and al= l services including metron, kafka, storm etc. are started. However, when I check if there is some data written into HDFS ( =C2=A0/apps/metron/indexing/indexed/= yaf|bro|snort=C2=A0) for any data sources, I don't see any data.= Hereby below is output.


=C2= =A0 =C2=A0 [vagrant@node1 bin]$ hdfs dfs -ls /apps/metron/indexing/indexed/=
Found 3 items
drwxrwxr-x=C2=A0 =C2=A0- storm hadoop=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2018-02-06 13:03 = /apps/metron/indexing/in=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0de= xed/bro
drwxrwxr-x=C2=A0 =C2=A0- storm hadoop=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2018-01-31 13:35 = /apps/metron/indexing/in=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0de= xed/error
drwxrwxr-x=C2=A0 =C2=A0- storm hadoop=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 2018-02-07 04:53 = /apps/metron/indexing/in=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0de= xed/snort

On other-hand, I can see sensors (= Snort & Bro) started on http://node1:4200 and is having some throughput, although very low. He= reby below is sensor status.


<= tbody><= /tr>
GrokWebSphere Stop= ped - - =20 =20 =20 =20 =
jsonMap JSONMap Stop= ped - - =20 =20 =20 =20
squid Grok Stop= ped - - =20 =20 =20 =20
snort Snort Running 3.862s 1.89kb/s =20 =20 =20
asa Asa Stop= ped - - =20 =20 =20 =20
bro Bro Running 4.25s 1.94kb/s =20 =20 =20
yaf Grok Running 0s 0kb/s =20 =20 =20

Can anybody guide me what should I check to ensure = sensors produce data and HDFS should be populated with this data ?

<= /div>
Thanks & Regards
RK Sharma





--

Jon

--94eb2c0cabe8c382c40564c96c02--