incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Trivial Update of "SensSoftProposal" by LewisJohnMcgibbney
Date Mon, 23 May 2016 18:01:08 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "SensSoftProposal" page has been changed by LewisJohnMcgibbney:
https://wiki.apache.org/incubator/SensSoftProposal?action=diff&rev1=1&rev2=2

     * STOUT supports between and within-subjects experimental designs, with random assignment
to experimental conditions. This allows for testing across different versions of applications.
     * STOUT can also use Django output (e.g., task complete) to automate other processes,
such as automated polling applications serving 3rd party form data APIs (e.g.,SurveyMonkey),
and python or R scripts to provide automated post-processing on task or survey data.
     * STOUT provides adopters a comprehensive dashboard view of data collected and post-processed
through its extensions; in addition to user enrollment, task completion, and experiment progress
metrics, STOUT allows adopters to visualize distributions of scores collected from task and
survey data.
+ 
  Each component is available through its own repository to support organic growth for each
component, as well as growth of the whole platform’s capabilities.
  
  == Background and Rationale ==
- A number of factors make this a good time for an Apache project focused on machine translation
(MT): the quality of MT output (for many language pairs); the average computing resources
available on computers, relative to the needs of MT systems; and the availability of a number
of high-quality toolkits, together with a large base of researchers working on them.
+ Any tool that people use to accomplish a task can be instrumented; once instrumented, those
tools can be used to report how they were used to perform that task. Software tools are ubiquitous
interfaces for people to interact with data and other technology that can be instrumented
for such a purpose. Tools are different than web pages or simple displays, however; they are
not simply archives for information. Rather, they are ways of interfacing with and manipulating
data and other technology. There are numerous consumer solutions for understanding how people
move through web pages and displays (e.g., Google Analytics, Adobe Omniture). There are far
fewer options for understanding how software tools are used. This requires understanding how
users integrate a tool’s functionality into usage strategies to perform tasks, how users
sequence the functionality provided them, and deeper knowledge of how users understand the
features of software as a cohesive tool. The Software as a Sensor™ Project is designed to
address this gap, providing the public an agile, cost-efficient solution for improving software
tool design, implementation, and usability.
  
- Over the past decade, machine translation (MT; the automatic translation of one human language
to another) has become a reality. The research into statistical approaches to translation
that began in the early nineties, together with the availability of large amounts of training
data, and better computing infrastructure, have all come together to produce translations
results that are “good enough” for a large set of language pairs and use cases. Free services
like [[https://www.bing.com/translator|Bing Translator]] and  [[https://translate.google.com|Google
Translate]] have made these services available to the average person through direct interfaces
and through tools like browser plugins, and sites across the world with higher translation
needs use them to translate their pages through automatically.
+ == Software as a Sensor™ Project Overview ==
+ Figure 1. User ALE Elastic Back End Schema, with Transfer Protocols.
  
- MT does not require the infrastructure of large corporations in order to produce feasible
output. Machine translation can be resource-intensive, but need not be prohibitively so. Disk
and memory usage are mostly a matter of model size, which for most language pairs is a few
gigabytes at most, at which size models can provide coverage on the order of tens or even
hundreds of thousands of words in the input and output languages. The computational complexity
of the algorithms used to search for translations of new sentences are typically linear in
the number of words in the input sentence, making it possible to run a translation engine
on a personal computer.
+ Funded through the DARPA XDATA program and other sources, the Software as a Sensor™ Project
provides an open source (ALv2.0) solution for instrumenting software tools developed for the
web so that when users interact with it, their behavior is captured. User behavior, or user
activities, are captured and time-stamped through a simple application program interface (API)
called User Analytic Logging Engine (User ALE). User ALE’s key differentiator is the schema
that it uses to collect information about user activities; it provides sufficient context
to understand activities within the software tool’s overall functionality. User ALE captures
each user initiated action, or event (e.g., hover, click, etc.), as a nested action within
a specific element (e.g., map object, drop down item, etc.), which are in turn nested within
element groups (e.g., map, drop down list) (see Figure 1). This information schema provides
sufficient context to understand and disambiguate user events from one another. In turn, this
enables myriad analysis possibilities at different levels of tool design and more utility
to end-user than commercial services currently offer. 
+ Once instrumented with User ALE, software tools become human signal sensors in their own
right. Most importantly, the data that User ALE collects is owned outright by adopters and
can be made available to other processes through scalable Elastic infrastructure and easy-to-manage
Restful APIs. 
+ Distill is the analytic framework of the Software as a Sensor™ Project, providing (at
release) segmentation and graph analysis metrics describing users’ interactions with the
application to adopters. The segmentation features allow adopters to focus their analyses
of user activity data based on desired data attributes (e.g., certain interactions, elements,
etc.), as well as attributes describing the software tool users, if that data was also collected.
Distill’s usage and usability metrics are derived from a representation of users’ sequential
interactions with the application as a directed graph. This provides an extensible framework
for providing insight as to how users integrate the functional components of the application
to accomplish tasks.
  
- The research community has produced many different open source translation projects for
a range of programming languages and under a variety of licenses. These projects include the
core “decoder”, which takes a model and uses it to translate new sentences between the
language pair the model was defined for. They also typically include a large set of tools
that enable new models to be built from large sets of example translations (“parallel data”)
and monolingual texts. These toolkits are usually built to support the agendas of the (largely)
academic researchers that build them: the repeated cycle of building new models, tuning model
parameters against development data, and evaluating them against held-out test data, using
standard metrics for testing the quality of MT output.
+ Figure 2. Software as a Sensor™ System Architecture with all components.
  
- Together, these three factors—the quality of machine translation output, the feasibility
of translating on standard computers, and the availability of tools to build models—make
it reasonable for the end users to use MT as a black-box service, and to run it on their personal
machine.
+ The Test Application Portal (TAP) provides a single point of interface for adopters of the
Software as a Sensor™ project. Through the Portal, adopters can register their applications,
providing version data and permissions to others for accessing data. The Portal ensures that
all components of the Software as a Sensor™ Project have the same information. The Portal
also hosts a number of python D3 visualization libraries, providing adopters with a customizable
“dashboard” with which to analyze and view user activity data, calling analytic processes
from Distill.
+ Finally, the Subject Tracking and Online User Testing (STOUT) application, provides support
for HCI/UX researchers that want to collect data from users in systematic ways or within experimental
designs. STOUT supports user registration, anonymization, user tracking, tasking (see Figure
3), and data integration from a variety of services. STOUT allows adopters to perform human
subject review board compliant research studies, and both between- and within-subjects designs.
Adopters can add tasks, surveys and questionnaires through 3rd party services (e.g., SurveyMonkey).
STOUT tracks users’ progress by passing a unique user IDs to other services, allowing researchers
to trace progress by passing a unique user IDs to other services, allowing researchers to
trace form data and User ALE logs to specific users and task sets (see Figure 4).
  
- These factors make it a good time for an organization with the status of the Apache Foundation
to host a machine translation project.
+ Figure 3. STOUT assigns participants subjects to experimental conditions and ensures the
correct task sequence. STOUT’s Django back end provides data on task completion, this can
be used to drive other automation, including unlocking different task sequences and/or achievements.
+ 
+ Figure 4. STOUT User Tracking. Anonymized User IDs (hashes) are concatenated with unique
Task IDs. This “Session ID” is appended to URLs (see Highlighted region), custom variable
fields, and User ALE, to provide and integrated user testing data collection service.
+ 
+ STOUT also provides for data polling from third party services (e.g., SurveyMonkey) and
integration with python or R scripts for statistical processing of data collected through
STOUT. D3 visualization libraries embedded in STOUT allow adopters to view distributions of
quantitative data collected from form data (see Figure 5).
+ 
+ Figure 5. STOUT Visualization. STOUT gives experimenters direct and continuous access to
automatically processed research data.
+ 
+ == Insights from User Activity Logs ==
+ 
+ The Software as a Sensor™ Project provides data collection and analytic services for user
activities collected during interaction with software tools. However, the Software as a Sensor™
Project emerged from years of research focused on the development of novel, reliable methods
for measuring individuals’ cognitive state in a variety of contexts. Traditional approaches
to assessment in a laboratory setting include surveys, questionnaires, and physiology (Poore
et al., 2016). Research performed as part of the Software as a Sensor™ project has shown
that the same kind of insights derived from these standard measurement approaches can also
be derived from users’ behavior. Additionally, we have explored insights that can only be
gained by analyzing raw behavior collected through software interactions (Mariano et al.,
2015). The signal processing and algorithmic approaches resulting from this research have
been integrated into the Distill analytics stack. This means that adopters will not be left
to discern for themselves how to draw insights from the data they gather about their software
tools, although they will have the freedom to explore their own methods as well.
+ Insights from user activities provided by Distill’s analytics framework fall under two
categories, broadly classified as functional workflow and usage statistics:  
+ Functional workflow insights tell adopters how user activities are connected, providing
them with representations of how users integrate the application’s features together in
time. These insights are informative for understanding the step-by-step process by which users
interact with certain facets of a tool. For example, questions like “how are my users, constructing
plots?” are addressable through workflow analysis. Workflows provide granular understanding
of process level mechanics and can be modeled probabilistically through a directed graph representation
of the data, and by identification of meaningful sub-sequences of user activities actually
observed in the population. Metrics derived provide insight about the structure and temporal
features of these mechanics, and can help highlight efficiency problems within workflows.
For example, workflow analysis could help identify recursive, repetitive behaviors, and might
be used to define what “floundering” looks like for that particular tool. 
+ Functional workflow analysis can also support analyses with more breadth. Questions like,
“how are my users integrating my tools’ features into a cohesive whole? Are they relying
on the tool as a whole or just using very specific parts of it?” Adopters will be able to
explore how users think about software as cohesive tools and examine if users are relying
on certain features as central navigation or analytic features. This allows for insights into
whether tools are designed well enough for users to understand that they need to rely on multiple
features together. 
+ Through segmentation, adopters can select the subset of the data -software element, action,
user demographics, geographic location, etc.- they want to analyze. This will allow them to
compare, for example, specific user populations against one another in terms of how they integrate
software functionality. Importantly, the graph-based analytics approach provides a flexible
representation of the time series data that can capture and quantify canonical usage patterns,
enabling direct comparisons between users based on attributes of interest. Other modeling
approaches have been utilized to explore similar insights and may be integrated at a later
date (Mariano, et al., 2015).
+ Usage statistics derive metrics from simple frequentist approaches to understanding, coarsely,
how much users are actually using applications. This is different from simple “traffic”
metrics, however, which assess how many users are navigating to a page or tool. Rather usage
data provides insight on how much raw effort (e.g., number of activities) is being expended
while users are interacting with the application. This provides deeper insight into discriminating
“visitors” from “users” of software tools. Moreover, given the information schema
User ALE provides, adopters will be able to delve into usage metrics related to specific facets
of their application.
+ Given these insights, different sets of adopters—software developers, HCI/UX researchers,
and project managers—may utilize The Software as a Sensor™ Project for a variety different
use cases, which may include:
+  * Testing to see if users are interacting with software tools in expected or unexpected
ways.
+  * Understanding how much users are using different facets of different features in service
of planning future developments.
+  * Gaining additional context for translating user/customer comments into actionable software
fixes.
+  * Understanding which features users have trouble integrating to guide decisions on how
to allocate resources to further documentation.
+  * Understanding the impact that new developments have on usability from version to version.

+  * Market research on how users make use of competitors’ applications to guide decisions
on how to build discriminating software tools.
+  * General research on Human Computer Interaction in service of refining UX and design principles.
+  * Psychological science research using software as data collection platforms for cognitive
tasks.
+ 
+ == Differentiators ==
+ 
+ The Software as a Sensor™ Project is ultimately designed to address the wide gaps between
current best practices in software user testing and trends toward agile software development
practices. Like much of the applied psychological sciences, user testing methods generally
borrow heavily from basic research methods. These methods are designed to make data collection
systematic and remove extraneous influences on test conditions. However, this usually means
removing what we test from dynamic, noisy—real-life—environments. The Software as a Sensor™
Project is designed to allow for the same kind of systematic data collection that we expect
in the laboratory, but in real-life software environments, by making software environments
data collection platforms. In doing so, we aim to not only collect data from more realistic
environments, and use-cases, but also to integrate the test enterprise into agile software
development process. 
+ Our vision for The Software as a Sensor™ Project is that it provides software developers,
HCI/UX researchers, and project managers a mechanism for continuous, iterative usability testing
for software tools in a way that supports the flow (and schedule) of modern software development
practices—Iterative, Waterfall, Spiral, and Agile. This is enabled by a few discriminating
facets:
+ 
+ Figure 6. Version to Version Testing for Agile, Iterative Software Development Methods.
The Software as a Sensor™ Project enables new methods for collecting large amounts of data
on software tools, deriving insights rapidly to inject into subsequent iterations
+ 
+  * Insights enabling software tool usability assessment and improvement can be inferred
directly from interactions with the tool in “real-world” environments. This is a sea-change
in thinking compared to canonical laboratory approaches that seek to artificially isolate
extraneous influences on the user and the software. The Software as a Sensor™ Project enables
large scale, remote, opportunities for data collection with minimal investment and no expensive
lab equipment (or laboratory training). This allows adopters to see how users will interact
with their technology in their places of work, at home, etc. 
+ 
+  * Insights are traceable to the software itself. Traditionally laboratory measures—questionnaires,
interviews, and physiology—collect data that is convenient for making inferences about psychological
states. However, it is notoriously difficult to translate this data into actionable “get-well”
strategies in technology development. User ALE’s information schema is specifically designed
to dissect user interaction within the terminology of application design, providing a familiar
nomenclature for software developers to interpret findings with. 
+ 
+  * Granular data collection enables advanced modeling and analytics. User ALE’s information
schema dissects user interaction by giving context to activity within the functional architecture
of software tools. Treating each time-series of user activity as a set of events nested within
functional components provides sufficient information for a variety of modeling approaches
that can be used to understand user states (e.g., engagement and cognitive load), user workflows
(e.g., sub-sequences), and users’ mental models of how software tool features can be integrated
(in time) to perform tasks. In contrast, commercial services such as Google Analytics and
Adobe Analytics (Omniture) provide very sparse options for describing events. They generally
advocate for using “boiler plate” event sets that are more suited to capturing count data
for interactions with specific content (e.g., videos, music, banners) and workflows through
“marketplace” like pages. User ALE provides content agnostic approaches for capturing
user activities by letting adopters label them in domain specific ways that give them context.
This provides a means by which identical user activities (e.g. click, select, etc.) can be
disambiguated from each other based on which functional sub-component of the tool they have
been assigned to.
+ 
+  * Adopter-generated content, analytics and data ownership. The Software as a Sensor™
Project is a set of open-source products built from other open-source products. This project
will allow adopters to generate their own content easily, using open source analytics and
visualization capabilities. By design, we also allow adopters to collect and manage their
own data with support from widely used open source data architectures (e.g., Elastic). This
means that adopters will not have to pay for additional content that they can develop themselves
to make use of the service, and do not have to expose their data to third party commercial
services. This is useful for highly proprietary software tools that are designed to make use
of sensitive data, or are themselves sensitive.
  
  == Current Status ==
- Joshua was originally ported from David Chiang’s Python implementation of Hiero by Zhifei
Li, while he was a Ph.D. student at Johns Hopkins University. The current version is maintained
by Matt Post at Johns Hopkins’ Human Language Technology Center of Excellence. Joshua has
made many releases with a list of over 20 source code tags. The last release of Joshua was
6.0.5 on November 5th, 2015.
+ 
+ All components of the Software as a Sensor™ Project were originally designed and developed
by Draper as part of DARPA’s XDATA project, although User ALE is being used on other funded
R&D projects, including DARPA RSPACE, AFRL project, and Draper internally funded projects.

+ Currently, only User ALE is publically available, however, the Portal, Distill, and STOUT
will be publically available in the May/June 2016 time-frame. The last major release of User
ALE was May, 2015. All components are currently maintained in separate repositories through
GitHub (github.com/draperlaboratory).
+ Currently, only software tools developed with Javascript are supported. However, we are
currently working on pythonQT implementations for User ALE that will support many desktop
applications.
  
  == Meritocracy ==
  The current developers are familiar with meritocratic open source development at Apache.
Apache was chosen specifically because we want to encourage this style of development for
the project.
  
  == Community ==
- Joshua is used widely across the world. Perhaps its biggest (known) research / industrial
user is the Amazon research group in Berlin. Another user is the US Army Research Lab. No
formal census has been undertaken, but posts to the Joshua technical support mailing list,
along with the occasional contributions, suggest small research and academic communities spread
across the world, many of them in India.
- 
+ The Software as a Sensor™ Project is new and our community is not yet established. However,
community building and publicity is a major thrust. Our technology is generating interest
within industry, particularly in the HCI/UX community, both Aptima and Charles River Analytics,
for example are interested in being adopters. We have also begun publicizing the project to
software development companies and universities, recently hosting a public focus group for
Boston, MA area companies. 
+ We are also developing communities of interested within the DoD and Intelligence community.
The NGA Xperience Lab has expressed interest in becoming a transition partner as has the Navy’s
HCIL group. We are also aggressively pursuing adopters at AFRL’s Human Performance Wing,
Analyst Test Bed. 
- During incubation, we will explicitly seek to increase our usage across the board, including
academic research, industry, and other end users interested in statistical machine translation.
+ During incubation, we will explicitly seek to increase our adoption, including academic
research, industry, and other end users interested in usability research.
  
  == Core Developers ==
- The current set of core developers is fairly small, having fallen with the graduation from
Johns Hopkins of some core student participants. However, Joshua is used fairly widely, as
mentioned above, and there remains a commitment from the principal researcher at Johns Hopkins
to continue to use and develop it. Joshua has seen a number of new community members become
interested recently due to a potential for its projected use in a number of ongoing DARPA
projects such as XDATA and Memex.
+ The current set of core developers is relatively small, but includes Draper full-time staff.
Community management will very likely be distributed across a few full-time staff that have
been with the project for at least 2 years. Core personnel can be found on our website: http://www.draper.com/softwareasasensor

  
  == Alignment ==
- Joshua is currently Copyright (c) 2015, Johns Hopkins University All rights reserved and
licensed under BSD 2-clause license. It would of course be the intention to relicense this
code under AL2.0 which would permit expanded and increased use of the software within Apache
projects. There is currently an ongoing effort within the Apache Tika community to utilize
Joshua within Tika’s Translate API, see [[https://issues.apache.org/jira/browse/TIKA-1343|TIKA-1343]].
+ The Software as a Sensor™ Project is currently Copyright (c) 2015, 2016 The Charles Stark
Draper Laboratory, Inc. All rights reserved and licensed under Apache v2.0. 
  
  == Known Risks ==
  
  === Orphaned products ===
- At the moment, regular contributions are made by a single contributor, the lead maintainer.
He (Matt Post) plans to continue development for the next few years, but it is still a single
point of failure, since the graduate students who worked on the project have moved on to jobs,
mostly in industry. However, our goal is to help that process by growing the community in
Apache, and at least in growing the community with users and participants from NASA JPL.
+ There are currently no orphaned products. Each component of The Software as a Sensor™
Project has roughly 1-2 dedicated staff, and there is substantial collaboration between projects.
  
  === Inexperience with Open Source ===
+ Draper has a number of open source software projects available through www.github.com/draperlaboratory.
- The team both at Johns Hopkins and NASA JPL have experience with many OSS software projects
at Apache and elsewhere. We understand "how it works" here at the foundation.
- 
  
  == Relationships with Other Apache Products ==
- Joshua includes dependences on Hadoop, and also is included as a plugin in Apache Tika.
We are also interested in coordinating with other projects including Spark, and other projects
needing MT services for language translation.
+ Software as a Sensor™ Project does not currently have any dependences on Apache Products.
We are also interested in coordinating with other projects including Usergrid, and others
involving data processing at large scales, time-series analysis and ETL processes.
  
  == Developers ==
- Joshua only has one regular developer who is employed by Johns Hopkins University. NASA
JPL (Mattmann and McGibbney) have been contributing lately including a Brew formula and other
contributions to the project through the DARPA XDATA and Memex programs.
+ The Software as a Sensor™ Project is primarily funded through contract work. There are
currently no “dedicated” developers, however, the same core team does work will continue
work on the project across different contracts that support different features. We do intend
to maintain a core set of key personnel engaged in community development and maintenance—in
the future this may mean dedicated developers funded internally to support the project, however,
the project is tied to business development strategy to maintain funding into various facets
of the project.
  
  == Documentation ==
  Documentation and publications related to Joshua can be found at joshua-decoder.org. The
source for the Joshua documentation is currently hosted on Github at https://github.com/joshua-decoder/joshua-decoder.github.com

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message