streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Blackmon [W2O Digital]" <sblack...@w2odigital.com>
Subject Re: Streams Subscriptions
Date Fri, 01 Feb 2013 00:00:19 GMT
First post from me (btw I am Steve, stoked about this project and meeting
everyone eventually.)

Sorry if I missed the point of the thread, but I think this is related and
might be educational for some in the group.

I like the way DataSift's API lets you establish streams - you POST a
definition, it returns a hash, and thereafter their service follows the
instructions you gave it as new messages meet the filter you defined.  In
addition, once a stream exists, then you can set up listeners on that
specific hash via web sockets with the hash.

For example, here is how you instruct DataSift to push new messages
meeting your criteria to a WebHooks end-point.

curl -X POST 'https://api.datasift.com/push/create' \
-d 'name=connectorhttp' \
-d 'hash=dce320ce31a8919784e6e85aecbd040e' \
-d 'output_type=http' \
-d 'output_params.method=post' \
-d 'output_params.url=http.example.com:8888' \
-d 'output_params.use_gzip' \
-d 'output_params.delivery_frequency=60' \
-d 'output_params.max_size=10485760' \
-d 'output_params.verify_ssl=false' \
-d 'output_params.auth.type=none' \
-d 'output_params.auth.username=YourHTTPServerUsername' \
-d 'output_params.auth.password=YourHTTPServerPassword' \
-H 'Auth: datasift-user:your-datasift-api-key


Now new messages get pushed to me every 60 seconds, and I can get the feed
in real-time like this:

var websocketsUser = 'datasift-user';
var websocketsHost = 'websocket.datasift.com';
var streamHash = 'dce320ce31a8919784e6e85aecbd040e';
var apiKey = 'your-datasift-api-key';


var ws = new 
WebSocket('ws://'+websocketsHost+'/'+streamHash+'?username='+websocketsUser
+'&api_key='+apiKey);
 
ws.onopen = function(evt) {
    // connection event
	$("#stream").append('open: '+evt.data+'<br/>');
}
 
ws.onmessage = function(evt) {
    // parse received message
	$("#stream").append('message: '+evt.data+'<br/>');
}
 
ws.onclose = function(evt) {
    // parse event
	$("#stream").append('close: '+evt.data+'<br/>');
}
 
// No event object is passed to the event callback, so no useful debugging
can be done
ws.onerror = function() {
    // Some error occurred
	$("#stream").append('error: '+evt.data+'<br/>');
}


At W2OGroup we have built utility libraries for receiving and processing
Json object streams from data sift in Storm/Kafka that I'm interested in
extending to work with Streams, and can probably commit to the project if
the community would find them useful.


Steve Blackmon
Director, Data Sciences

101 W. 6th Street
Austin, Texas 78701
cell 512.965.0451 | work 512.402.6366
twitter @steveblackmon







On 1/31/13 5:45 PM, "Craig McClanahan" <craigmcc@gmail.com> wrote:

>We'll probably want some way to do the equivalent of ">", ">=", "<", "<=",
>and "!=" in addition to the implicit "equal" that I assume you mean in
>this
>example.
>
>Craig
>
>On Thu, Jan 31, 2013 at 3:39 PM, Jason Letourneau
><jletourneau80@gmail.com>wrote:
>
>> I really like this - this is somewhat what I was getting at with the
>> JSON object i.e. POST:
>> {
>> "subscriptions":
>> [{"activityField":"value"},
>> {"activityField":"value",
>>  "anotherActivityField":"value" }
>> ]
>> }
>>
>> On Thu, Jan 31, 2013 at 4:32 PM, Craig McClanahan <craigmcc@gmail.com>
>> wrote:
>> > On Thu, Jan 31, 2013 at 12:00 PM, Jason Letourneau
>> > <jletourneau80@gmail.com>wrote:
>> >
>> >> I am curious on the group's thinking about subscriptions to activity
>> >> streams.  As I am stubbing out the end-to-end heartbeat on my
>>proposed
>> >> architecture, I've just been working with URL sources as the
>> >> subscription mode.  Obviously this is a way over-simplification.
>> >>
>> >> I know for shindig the social graph can be used, but we don't
>> >> necessarily have that.  Considering the mechanism for establishing a
>> >> new subscription stream (defined as aggregated individual activities
>> >> pulled from a varying array of sources) is POSTing to the Activity
>> >> Streams server to establish the channel (currently just a
>> >> subscriptions=url1,url2,url3 is the over simplified mechanism)...what
>> >> would people see as a reasonable way to establish subscriptions?
>>List
>> >> of userIds? Subjects?  How should these be represented?  I was
>> >> thinking of a JSON object, but any one have other thoughts?
>> >>
>> >> Jason
>> >>
>> >
>> > One idea would be take some inspiration from how JIRA lets you (in
>> effect)
>> > create a WHERE clause that looks at any fields (in all the activities
>> > flowing through the server) that you want.
>> >
>> > Example filter criteria
>> > * provider.id = 'xxx' // Filter on a particular provider
>> > * verb = 'yyy'
>> > * object.type = 'blogpost'
>> > and you'd want to accept more than one value (effectively creating OR
>>or
>> IN
>> > type clauses).
>> >
>> > For completeness, I'd want to be able to specify more than one filter
>> > expression in the same subscription.
>> >
>> > Craig
>>


Mime
View raw message