streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Young <>
Subject Re: Granary & SocketHub
Date Thu, 20 Oct 2016 19:14:08 GMT
Then it sounds like we're decidedly on the same page...and now to the work! ;)

Also (for later):

It's more or less what you're after, I think: descriptions of endpoints and HTTP-based actions
to be taken on them. There are certainly others like this out there also, but most are more
limited and more focused on documenting or describing one's own API vs. doing that same thing
for any API anywhere + Linked Data. ^_^

Thanks, Steve.



From: sblackmon <>
Sent: Thursday, October 20, 2016 3:06:21 PM
To:; Benjamin Young
Cc: Matt Franklin
Subject: Re: Granary & SocketHub

On October 20, 2016 at 1:33:33 PM, Benjamin Young (<>)

Great points, Steven.

What's always attracted me to Apache Streams is it's descriptiveness (via JSON Schemas documents)
vs. prescriptive-ness. Granary's approach is (currently? ;) ) more prescriptive:


...which is mostly (though not all) a collection of .json and .conf files with a handful of
.java files needed (afaict) for last-mile integration with one's tool.

The future I dream about is one where I can pick my tool for my idiosyncratic language, operating
system, license reasons, but they'll all work off shared, descriptive "knowledge" documents.

I think we (the active streams developers) agree that data descriptor and translation rules
should ideally be
a) human-readable
b) machine-readable
c) programming language-agnostic
d) internationalized
e) compliant with web standards, or community standards where suitable web standards don’t

With regard to operating system flexibility, java byte-code and docker containers are as universal
as any other run-time platform, with the possible exception of javascript.

Apache License 2.0 is about as permissive as they come, so hopefully no concerns there?

Otherwise, we're all pulling separately, and end up with snowflake systems to process snowflake
APIs. However, I also know it's unlikely everyone will come "under one roof" to work on things.
My hope, though, is that the output of this group (and Granary and Sockethub and...) will
be re-usable by as wide an audience as possible--hence the value of description over prescription
(at least in my book ;) ).

A major driver behind using json schemas and hocon snippets so widely throughout the project,
and the reason we push all of them onto the website with each snapshot and each release, is
to facilitate re-use and re-mixing of those artifacts in other projects using URIs.

Ideally it should be possible for some other project to build a system with black-box behavior
identical to Apache Streams in another language with much less code by piggy-backing on these
public resources.  This could even extend to the mechanics of collection, if we specified
how to submit and parse HTTP requests with text resources rather than using java sdks.  We’ve
got a long way to go to reach this objective, but philosophically I’m all for maximizing
the re-usability of this codebase and supporting complementary efforts.

Granted, if I'm barking up the wrong tree (again), I'm happy to wander off...

Please don’t!  Stay and help!

Is anything in the above sane? ;)

I think so!  P.S. I’m hopeful that bringing this code-base into compliance with AS 2.0 and
JSON-LD will open up the world of RDF tools and specs and allow more of what’s currently
expressible only with source code to be expressed with W3C standards compliant resource files
published on the web going forward.




From: sblackmon <>
Sent: Thursday, October 20, 2016 1:26:38 PM
Cc: Matt Franklin; Benjamin Young
Subject: Re: Granary & SocketHub

On October 18, 2016 at 6:09:49 PM, Matt Franklin (<>)
On Tue, Oct 18, 2016 at 10:39 AM Benjamin Young <>

> (resending from the correct account…likely the other got spammed…)
> Granary is a project with similar ideas and intents as Apache Streams
> (which also needs AS2 support ;) ):

Ryan from Granary is on the list I think.  Hey Ryan!  Cool stuff, too bad it’s python :)

> In fact Apache Streams gets a mention in their “Related Work” section:
> Also mentioned in the Granary related work section is SocketHub:

Cool stuff, too bad it’s LGPL :)

> It’s aims are similar, but it’s reaching way beyond Web-based social APIs
> and “back” to including things like IRC, Email, etc.

Non-SNS data sources are important for sure. I’ve posted some work on my personal github
using the streams framework to parse MBOX files -
- and to collect quantified self data -
IRC is interesting as well.

> What’s significant about both these projects (and others they link to) are
> the stories they’re telling developers—which we can crib from as we think
> about the Streams “pitch.” They also have relatively minimal setup
> docs—which Streams is also heading toward (go Steve!).

Agreed this is key

The existence of other open-source projects with similar themes suggests we’re onto an important
problem.  We should pay attention to these projects and what is working for them WRT user
growth, community growth, tech media coverage, etc...

> Again, my key objective is to understand the Apache Streams vision along
> side projects like these and within the wider space of consolidating social
> data. What market does it serve? Is it “personal” (as these projects seem
> to be)? Or commercial? Or developer-only (library/framework for wiring up
> your own idiosyncratic stuff)?

I think the overall objective of streams remains very similar to what it
started as: A way to easily and flexibly ingest multiple different sources
of 'activity' data in a normalized ActivityStreams format. For me
personally, my interest is in ingesting this data at scale and with as
little internally-maintained code as possible.

While most of the development so far has been geared toward enabling back-end / commercial-scale
data collection and management, I think the future should be more about enabling individuals
and businesses to transcend data silos using computing resources and code entirely under their
own control. This might mean supporting regular users with a full-featured SaaS application
in addition to continued work on data interoperability.

> Thanks for reading, pondering, and helping me help. :)
> Cheers!
> Benjamin
> --

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message