Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
From: Benjamin Young <byoung@bigbluehat.com>
To: sblackmon <sblackmon@apache.org>, "dev@streams.incubator.apache.org"
	<dev@streams.incubator.apache.org>
CC: Matt Franklin <m.ben.franklin@gmail.com>
Subject: Re: Granary & SocketHub
Thread-Topic: Granary & SocketHub
Thread-Index: AQHSKU1krrRQMxxt7kO2c+heMbg3EKCxm6oBgAAHMSs=
Date: Thu, 20 Oct 2016 18:32:32 +0000
Message-ID: <CY1PR0601MB14882FB21A908A2493FE6713B2D50@CY1PR0601MB1488.namprd06.prod.outlook.com>
References: <CY1PR0601MB148850A0302A8E8931E310B5B2D30@CY1PR0601MB1488.namprd06.prod.outlook.com>
 <CAENpzYUmv5qs9fOQZN-i+a9HRcqGke7isJCjRMPmWhyxTqM85g@mail.gmail.com>,<etPan.5808fe4e.107d5fb1.824@apache.org>
In-Reply-To: <etPan.5808fe4e.107d5fb1.824@apache.org>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: bigbluehat.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_CY1PR0601MB14882FB21A908A2493FE6713B2D50CY1PR0601MB1488_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Oct 2016 18:32:32.0281
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 98d05955-de73-4052-a2d4-cbaabc15b584
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0601MB1486
archived-at: Thu, 20 Oct 2016 18:37:59 -0000

--_000_CY1PR0601MB14882FB21A908A2493FE6713B2D50CY1PR0601MB1488_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Great points, Steven.


What's always attracted me to Apache Streams is it's descriptiveness (via J=
SON Schemas documents) vs. prescriptive-ness. Granary's approach is (curren=
tly? ;) ) more prescriptive:

https://github.com/snarfed/granary/blob/master/granary/twitter.py

vs.

https://github.com/apache/streams/tree/STREAMS-26/streams-contrib/streams-p=
rovider-twitter

...which is mostly (though not all) a collection of .json and .conf files w=
ith a handful of .java files needed (afaict) for last-mile integration with=
 one's tool.


The future I dream about is one where I can pick my tool for my idiosyncrat=
ic language, operating system, license reasons, but they'll all work off sh=
ared, descriptive "knowledge" documents.


Otherwise, we're all pulling separately, and end up with snowflake systems =
to process snowflake APIs. However, I also know it's unlikely everyone will=
 come "under one roof" to work on things. My hope, though, is that the outp=
ut of this group (and Granary and Sockethub and...) will be re-usable by as=
 wide an audience as possible--hence the value of description over prescrip=
tion (at least in my book ;) ).


Granted, if I'm barking up the wrong tree (again), I'm happy to wander off.=
..


Is anything in the above sane? ;)


Cheers!

Benjamin

--

http://bigbluehat.com/

http://linkedin.com/in/benjaminyoung

________________________________
From: sblackmon <sblackmon@apache.org>
Sent: Thursday, October 20, 2016 1:26:38 PM
To: dev@streams.incubator.apache.org
Cc: Matt Franklin; Benjamin Young
Subject: Re: Granary & SocketHub

On October 18, 2016 at 6:09:49 PM, Matt Franklin (m.ben.franklin@gmail.com<=
mailto:m.ben.franklin@gmail.com>) wrote:
On Tue, Oct 18, 2016 at 10:39 AM Benjamin Young <byoung@bigbluehat.com>
wrote:

> (resending from the correct account...likely the other got spammed...)
>
> Granary is a project with similar ideas and intents as Apache Streams
> (which also needs AS2 support ;) ):
> https://github.com/snarfed/granary
>

Ryan from Granary is on the list I think.  Hey Ryan!  Cool stuff, too bad i=
t's python :)

> In fact Apache Streams gets a mention in their "Related Work" section:
> https://github.com/snarfed/granary#related-work
>
> Also mentioned in the Granary related work section is SocketHub:
> https://github.com/sockethub/sockethub
>

Cool stuff, too bad it's LGPL :)

> It's aims are similar, but it's reaching way beyond Web-based social APIs
> and "back" to including things like IRC, Email, etc.


Non-SNS data sources are important for sure. I've posted some work on my pe=
rsonal github using the streams framework to parse MBOX files - https://git=
hub.com/steveblackmon/streams-apache - and to collect quantified self data =
- https://github.com/steveblackmon/humanapi-streams
IRC is interesting as well.


> What's significant about both these projects (and others they link to) ar=
e
> the stories they're telling developers-which we can crib from as we think
> about the Streams "pitch." They also have relatively minimal setup
> docs-which Streams is also heading toward (go Steve!).
>

Agreed this is key


The existence of other open-source projects with similar themes suggests we=
're onto an important problem.  We should pay attention to these projects a=
nd what is working for them WRT user growth, community growth, tech media c=
overage, etc...


>
> Again, my key objective is to understand the Apache Streams vision along
> side projects like these and within the wider space of consolidating soci=
al
> data. What market does it serve? Is it "personal" (as these projects seem
> to be)? Or commercial? Or developer-only (library/framework for wiring up
> your own idiosyncratic stuff)?
>

I think the overall objective of streams remains very similar to what it
started as: A way to easily and flexibly ingest multiple different sources
of 'activity' data in a normalized ActivityStreams format. For me
personally, my interest is in ingesting this data at scale and with as
little internally-maintained code as possible.

While most of the development so far has been geared toward enabling back-e=
nd / commercial-scale data collection and management, I think the future sh=
ould be more about enabling individuals and businesses to transcend data si=
los using computing resources and code entirely under their own control. Th=
is might mean supporting regular users with a full-featured SaaS applicatio=
n in addition to continued work on data interoperability.


>
> Thanks for reading, pondering, and helping me help. :)
>
> Cheers!
> Benjamin
>
> --
> http://bigbluehat.com/
> http://linkedin.com/in/benjaminyoung
>
>

--_000_CY1PR0601MB14882FB21A908A2493FE6713B2D50CY1PR0601MB1488_--