Batch Predictions

Overview
Process predictions for many queries using efficient parallelization through Spark. Useful for mass auditing of predictions and for generating predictions to push into other systems.
Batch predict reads and writes multi-object JSON files similar to the batch import format. JSON objects are separated by newlines and cannot themselves contain unencoded newlines.
Compatibility
pio batchpredict loads the engine and processes queries exactly like pio deploy. There is only one additional requirement for engines to utilize batch predict:
All algorithm classes used in the engine must be serializable. This is already true for PredictionIO's base algorithm classes, but may be broken by including non-serializable fields in their constructor. Using the @transient annotatio n may help in these cases.
This requirement is due to processing the input queries as a Spark RDD which enables high-performance parallelization, even on a single machine.
Usage
pio batchpredict
Command to process bulk predictions. Takes the same options as pio deploy plus:
--input <value>
Path to file containing queries; a multi-object JSON file with one query object per line. Accepts any valid Hadoop file URL.
Default: batchpredict-input.json
--output <value>
Path to file to receive results; a multi-objec t JSON file with one object per line, the prediction + original query. Accepts any valid Hadoop file URL. Actual output will be written as Hadoop partition files in a directory with the output name.
Default: batchpredict-output.json
--query-partitions <value>
Configure the concurrency of predictions by setting the number of partitions used internally for the RDD of queries. This will directly effect the number of resulting part-* output files. While setting to 1 may seem appealing to get a single output file, this will remove parallelization for the batch process, reducing performance and possibly exhausting memory.
Default: number created by Spark context's textFile (probably the number of cores available on the local machine)
--engine-instance-id <value>
Identifier for the trained instance to use for batch predict.
Default: the latest trained instance.
Example
Input
A multi-object JSON file of queries as they would be sent to the engine's HTTP Queries API.
Read via SparkContext's textFile and so may be a single file or any supported Hadoop format.
File: batchpredict-input.json
1 +2 +3 +4 +5
{"user":"1"} +{"user":"2"} +{"user":"3"} +{"user":"4"} +{"user":"5"} +

Execute
1 +2 +3
pio batchpredict \ + --input batchpredict-input.json \ + --output batchpredict-output.json +

This command will run to completion, aborting if any errors are encountered.
Output
A multi-object JSON file of predictions + original queries. The predictions are JSON objects as they would be returned from the engine's HTTP Queries API.
Results are written via Spark RDD's saveAsTextFile so each partition will be written to its own part-* file. See post-processing results.
File 1: batchpredict-output.json/part-00000
1 +2 +3
{"query":{"user":"1"},"prediction":{"itemScores":[{"item":"1","score":33},{"item":"2","score":32}]}} +{"query":{"user":"3"},"prediction":{"itemScores":[{"item":"2","score":16},{"item":"3","score":12}]}} +{"query":{"user":"4"},"prediction":{"itemScores":[{"item":"3","score":19},{"item":"1","score":18}]}} +

File 2: batchpredict-output.json/part-00001
1 +2
{"query":{"user":"2"},"prediction":{"itemScores":[{"item":"5","score":55},{"item":"3","score":28}]}} +{"query":{"user":"5"},"prediction":{"itemScores":[{"item":"1","score":24},{"item":"4","score":14}]}} +

Post-processing Results
After the process exits successfully, the parts may be concatenated into a single output file using a command like:
1
cat batchpredict-output.json/part-* > batchpredict-output-all.json +

Thank you for your interest in contributing to Apache PredictionIO (incubating). Our mission is to enable developers to build scalable machine learning applications easily. Here is how you can help with the project development. If you have any question regarding development at anytime, please free to subscribe and post to the Development Mailing List.
Areas in Need of Help
We accept contributions of all kinds at any time. We are compiling this list to show features that are highly sought after by the community.

Tests and CI
Engine template, tutorials, and samples

Client SDKs

Building engines in Java (updating the Java controller API)

Code clean up and refactoring

Code and data pipeline optimization

Developer experience (UX) improvement

How to Report an Issue
If you wish to report an issue you found, you can do so on Apache PredictionIO (incubating) JIRA.
How to Help Resolve Existing Issues
In general, bug fixes should be done the same way as new features, but critical bug fixes will follow a different path.
How to Add / Propose a New Feature
Before adding new features into JIRA, please check that the feature does not currently exist in JIRA.

To propose a new feature, simply subscribe and post your proposal to Apache PredictionIO (incubating) Development Mailing List.

Discuss with the community and the core development team on what needs to be done, and lay down concrete plans on deliverables.

Once solid plans are made, start creating tickets in the issue tracker.

Work side by side with other developers using Apache PredictionIO (incubating) Development Mailing List as primary mode of communication. You never know if someone else has a better idea. ;)

Adding ticket to JIRA

Add a descriptive Summary and a detailed description

Set Issue Type to Bug, Improvement, New Feature, Test or Wish

Set Priority to Blocker, Critical, Major, Minor or Trivial

Fill out Affects Version with the version of PredictionIO you are currently using

Fill out Environment if needed for description of your bug / feature

Please leave other fields blank

Triaging JIRA
Tickets will be triaged by PredictionIO committers.

Target Version: Either a particular version or Future if to be done later

Once a fix has been committed, the Fix Version will filled in with the appropriate release

Component: Each ticket will be annotated with one or more of the following Components

Core: affects the main code branch / will be part of a release

Documentation: affects the documents / will be pushed to livedoc branch

Templates: affects one of the separate github repositories for a template

How to Issue a Pull Request
When you have finished your code, you can create a pull request against the develop branch.

The title must contain a tag associating with an existing JIRA ticket. You must create a ticket so that the infrastructure can correctly track issues across Apache JIRA and GitHub. If your ticket is PIO-789, your title must look something like [PIO-789] Some short description.

Please also, in your commit message summary, include the JIRA ticket number similar to above.

Make sure the title and description are clear and concise. For more details on writing a good commit message, check out this guide.

If the change is visual, make sure to include a scree nshot or GIF.

Make sure it is being opened into the right branch.

Make sure it has been rebased on top of that branch.

When it is close to a release, and if there are major development ongoing, a release branch will be forked from the develop branch to stabilize the code for binary release. Please refer to the git flow methodology page for more information.
Getting Started
Apache PredictionIO (incubating) relies heavily on the git flow methodology. Please make sure you read and understand it before you start your development. By default, cloning Apache PredictionIO (incubating) will put you in the develop branch, which in most cases is where all the latest development go to.
For core development, please follow the Scala Style Guide.
Create a Fork of the Apache PredictionIO (incubating) Repository

Start by creating a GitHub account if you do not already have one.

Go to Apache PredictionIO (incubating)’s GitHub mirror and fork it to your own account.

Clone your fork to your local machine.

If you need additional help, please refer to https://help.github.com/articles/fork-a-repo/.
Building Apache PredictionIO (incubating) from Source
After the previous section, you should have a copy of Apache PredictionIO (incubating) in your local machine ready to be built.

Make sure you are on the devel op branch. You can double check by git status or simply git checkout develop.

At the root of the repository, do ./make-distribution.sh to build PredictionIO.

Setting Up the Environment
Apache PredictionIO (incubating) relies on 3rd party software to perform its tasks. To set them up, simply follow this documentation.
Start Hacking
You should have a Apache PredictionIO (incubating) development environment by now. Happy hacking!
Anatomy of Apache PredictionIO (incubating) Code Tree
The following describes each directory’s purpose.
bin
Shell scripts and any relevant components to go into the binary distribution. Utility shell scripts can also be included here.
conf
Configuration files that are used by both a source tree and binary distribution.
core
Core Apache PredictionIO (incubating) code that provides the DASE controller API, core data structures, and workflow creation and management code.
data
Apache PredictionIO (incubating) Event Server, and backend-agnostic storage layer for event store and metadata store.
docs
Source code for http://predictionio.incubator.apache.org site, and any other documentation support files.
examples
Complete code examples showing Apache PredictionIO (incubating)'s application.
sbt
Embedded SBT (Simple Build Tool) launcher.
storage
Storage implementations.
tools
Tools for running Apache PredictionIO (incubating). Contains primarily the CLI (command-line interface) and its supporting code, and the experimental evaluation dashboard.

Batch Predictions

PredictionIO Docs

Batch Predictions

On this page

Batch Predictions

Overview

Compatibility

Usage

pio batchpredict

--input <value>

--output <value>

--query-partitions <value>

--engine-instance-id <value>

Example

Input

Execute

Output

Post-processing Results

Command-line Interface

PredictionIO Docs

Command Line

On this pag e

Command Line

Overview

General Commands

Event Server Commands

Engine Commands

Contribute Code

PredictionIO Docs

Contribute Code

On this page

Contribute Code

Areas in Need of Help

How to Report an Issue

How to Help Resolve Existing Issues

How to Add / Propose a New Feature

Adding ticket to JIRA

Triaging JIRA

How to Issue a Pull Request

Getting Started

Create a Fork of the Apache PredictionIO (incubating) Repository

Building Apache PredictionIO (incubating) from Source

Setting Up the Environment

Start Hacking

Anatomy of Apache PredictionIO (incubating) Code Tree

bin

conf

core

data

docs

examples

sbt

storage

tools

`pio batchpredict`

`--input <value>`

`--output <value>`

`--query-partitions <value>`

`--engine-instance-id <value>`