asterixdb-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Glenn Marsh <bryan.ma...@email.ucr.edu>
Subject Data Feeds in AsterixDB
Date Fri, 30 Oct 2015 18:16:15 GMT
Hi guys,

I know that there has already been a discussion/questions reagarding data
feeds and I have looked all messages pertaining as far back as May.

In one of the posts it suggests looking at
https://asterix-jenkins.ics.uci.edu/job/asterix-test-full/site/asterix-doc/feeds/tutorial.html
but
I did not find that helpful.

Let me step back and explain what I am trying to do. I want t create a
dataset and be able to insert records into this dataset through the use of
feeds.

I have tried implementing feeds with Asterix in the following two ways and
both have been unsuccessful.

The first method I tried was creating a socket adapter and try to send
packets to the feed and be able to insert records into the dataset. I based
this off of examples from here
https://github.com/apache/incubator-asterixdb/tree/master/asterix-app/src/test/resources/runtimets/queries/feeds/feeds_06


Here are the statements that I used with in the WEBUI

DDL:

drop dataverse feeds if exists;
create dataverse feeds;
use dataverse feeds;

create type Log_Type as closed {
LogId :string,
updateString:string
}

create dataset AstrixDB_Transaction_Log(Log_Type)
primary key LogId;
create index updateLogIndex on AstrixDB_Transaction_Log(LogId);

create feed socket_feed
using socket_adapter
(("sockets"="127.0.0.1:9009
"),("addressType"="IP"),("type-name"="Log_Type"),("format"="adm"));

UPDATE 1(connect feed):
use dataverse feeds;

set wait-for-completion-feed "true";

connect feed socket_feed to dataset AstrixDB_Transaction_Log;

NOTE: when I execute this command I do not get any response back saying it
was successful.

UPDATE 2 (disconnect feed):

use dataverse feeds;
disconnect feed socket_feed from dataset AstrixDB_Transaction_Log;


Query:

use dataverse feeds;

for $x in dataset AstrixDB_Transaction_Log
return $x

When I run the DDL and UPDATE I get know error, but when I try and send a
packet to 127.0.0.1:9009 the packet gets stuck, I set a timeout value to 4
seconds, but then it times out. Below I have attached some python code that
sends one adm records from the file load.txt.
I tried attaching the files, but google would not let me so please get the
files from this dropbox link
https://www.dropbox.com/s/mw2j6xsqed9a0cg/AsterixDB_FeedTest.tgz?dl=0

I thought maybe the port wasn't open, so I opened the port 9009 to allow
for traffic but that did not help either.

To run:


   1. Untar: AsterixDB_FeedTest.tar.gz
   2. cd into AsterixDB_FeedTest
   3. In AsterixDB WEBUI enter in the DDL and UPDATE statements above
   4. In the terminal enter : python feedInjector.py



The second method I tried was to insert values from a file into the feed. I
took this example from
https://github.com/apache/incubator-asterixdb/tree/master/asterix-app/src/test/resources/runtimets/queries/feeds/feeds_11

Here are the statements that I used in the WEBUI

DDL:

drop dataverse feeds if exists;
create dataverse feeds;
use dataverse feeds;

create type Log_Type as open {
LogId:string,
updateString:string
}

create dataset AstrixDB_Transaction_Log(Log_Type)
primary key LogId;
create index updateLogIndex on AstrixDB_Transaction_Log(LogId);

create feed log_feed
using file_feed
(("fs"="localfs"),("path"="127.0.0.1
:///Users/Bryan/Desktop/Masters_Project_Files/Data/DataFeedInjector/load.asm"),("format"="adm"),("type-name"="Log_Type"),("tuple-interval"="10"));

UPDATE :

use dataverse feeds;

set wait-for-completion-feed "true";

connect feed feeds.log_feed to dataset AstrixDB_Transaction_Log;

Query:

use dataverse feeds;

for $x in dataset AstrixDB_Transaction_Log
return $x

Bellow I have attached the load.adm file that I used for the loading.

When I execute the DDL statement into AsterixDB WEBUI,  it is successful,
but when I try to execute the Update statement I get the following error:

Failure in feed [AsterixException]

I am not sure what I am doing wrong.


One thing I did see happening is that when the feed does not work, 8/10
times the WEBUI becomes unresponsive yet the logs still say that the
instance is still ACTIVE.

Any help that you guys could give me would be helpful, I have been
stressing out for the last two weeks trying to figure this out. If possible
getting the first method to work would be ideal.


Thank you
Sincerely, Bryan Marsh

Mime
View raw message