spot-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikash Kumar <vikash.ku...@oneconvergence.com>
Subject Re: sample data for spot ingest
Date Fri, 28 Jul 2017 18:56:56 GMT
Before. Do I need to pass those files after ?

Regards,
Vikash

On Sat, Jul 29, 2017 at 12:25 AM, Barona, Ricardo <ricardo.barona@intel.com>
wrote:

> Did you move nfcapd file to /home/cloudera-scm/spot-data/flow after
> ingest started? Or before?
>
>
>
> *From: *Vikash Kumar <vikash.kumar@oneconvergence.com>
> *Reply-To: *"user@spot.incubator.apache.org" <
> user@spot.incubator.apache.org>
> *Date: *Friday, July 28, 2017 at 1:40 PM
>
> *To: *"user@spot.incubator.apache.org" <user@spot.incubator.apache.org>
> *Subject: *Re: sample data for spot ingest
>
>
>
> I have ingest running. Below is the process stdout log:
>
> 17/07/28 17:34:11 INFO zookeeper.ZooKeeper: Client environment:user.name
> =cloudera-scm
> 17/07/28 17:34:11 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/home/cloudera-scm
> 17/07/28 17:34:11 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/home/cloudera-scm/incubator-spot/spot-ingest
> 17/07/28 17:34:11 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=192.168.200.3:2181 sessionTimeout=30000
> watcher=org.I0Itec.zkclient.ZkClient@43d3774
> 17/07/28 17:34:11 INFO zkclient.ZkClient: Waiting for keeper state
> SyncConnected
> 17/07/28 17:34:11 INFO zookeeper.ClientCnxn: Opening socket connection to
> server compute-3/192.168.200.3:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 17/07/28 17:34:11 INFO zookeeper.ClientCnxn: Socket connection
> established, initiating session, client: /192.168.200.3:42474, server:
> compute-3/192.168.200.3:2181
> 17/07/28 17:34:11 INFO zookeeper.ClientCnxn: Session establishment
> complete on server compute-3/192.168.200.3:2181, sessionid =
> 0x15d89f36c6200aa, negotiated timeout = 30000
> 17/07/28 17:34:11 INFO zkclient.ZkClient: zookeeper state changed
> (SyncConnected)
> WARNING: Due to limitations in metric names, topics with a period ('.') or
> underscore ('_') could collide. To avoid issues it is best to use either,
> but not both.
> Error while executing topic command : Topic 'SPOT-INGEST-flow-10_34_08'
> already exists.
> 17/07/28 17:34:11 ERROR admin.TopicCommand$: org.apache.kafka.common.errors.TopicExistsException:
> Topic 'SPOT-INGEST-flow-10_34_08' already exists.
>
> 17/07/28 17:34:11 INFO zkclient.ZkEventThread: Terminate ZkClient event
> thread.
> 17/07/28 17:34:11 INFO zookeeper.ZooKeeper: Session: 0x15d89f36c6200aa
> closed
> 17/07/28 17:34:11 INFO zookeeper.ClientCnxn: EventThread shut down
> 2017-07-28 17:34:12,123 - SPOT.INGEST - INFO - Starting
> SPOT-INGEST-flow-10_34_08 ingest instance
> 2017-07-28 17:34:12,145 - SPOT.INGEST.WATCHER - INFO - Creating File
> watcher
> 2017-07-28 17:34:12,145 - SPOT.INGEST.WATCHER - INFO - Supported Files:
> [u'nfcapd.']
> 2017-07-28 17:34:12,156 - SPOT.INGEST.FLOW - INFO - Starting FLOW ingest
> 2017-07-28 17:34:12,156 - SPOT.INGEST.WATCHER - INFO - Watching:
> /home/cloudera-scm/spot-data/flow
>
>
> I have data in '/home/cloudera-scm/spot-data/flow' , but there is no data
> in hive.
>
>
> Regards,
>
> Vikash
>
> *"Without requirements or design, programming is the art of adding bugs to
> an empty text file."- Louis Srygley*
>
>
>
> On Fri, Jul 28, 2017 at 11:59 PM, Barona, Ricardo <
> ricardo.barona@intel.com> wrote:
>
> Ok, in order to ingest nfcapd.201601281600 you need to set up spot-ingest.
> Once that’s running you can place the file so it gets processed and will
> automatically be saved into Hive table.
>
> Here is some additional documentation other than the github README.md
> files: http://spot.incubator.apache.org/doc/
>
>
>
> *From: *Vikash Kumar <vikash.kumar@oneconvergence.com>
> *Reply-To: *"user@spot.incubator.apache.org" <
> user@spot.incubator.apache.org>
> *Date: *Friday, July 28, 2017 at 1:23 PM
> *To: *"user@spot.incubator.apache.org" <user@spot.incubator.apache.org>
> *Subject: *Re: sample data for spot ingest
>
>
>
> I have data in following format:
>
>
> dns - .pcap
>
> flow - nfcapd.201601281600 (eg.)
>
> log - .log
>
>
> Regards,
>
> Vikash
>
>
>
> On Fri, Jul 28, 2017 at 11:49 PM, Barona, Ricardo <
> ricardo.barona@intel.com> wrote:
>
> Hi Vikash,
>
>
>
> What format is your data?
>
>
>
> *From: *Vikash Kumar <vikash.kumar@oneconvergence.com>
> *Reply-To: *"user@spot.incubator.apache.org" <
> user@spot.incubator.apache.org>
> *Date: *Friday, July 28, 2017 at 1:09 PM
> *To: *"user@spot.incubator.apache.org" <user@spot.incubator.apache.org>
> *Subject: *sample data for spot ingest
>
>
>
> Hi All,
>
>
> I have raw data for spot , but those data are not in .csv format. Is there
> any sample data available to store in table for ingest. Or what is the
> suggested tool to convert ?
>
> Are there any instructions available to upload same ?
>
>
> Regards,
>
> Vikash
>
>
>
>
>

Mime
View raw message