singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangwei (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SINGA-20) Web interface for launching training jobs
Date Fri, 19 Jun 2015 13:34:00 GMT

     [ https://issues.apache.org/jira/browse/SINGA-20?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

wangwei updated SINGA-20:
-------------------------
    Description: 
Job Launching. Currently users have to set the model and cluster configuration by writing
the google protocol buffer files (e.g., model.conf and cluster.conf). However it is not easy
for beginning users as it requires full understanding of all configuration fields. To make
it easier, a web interface can be provided, which assist users to configure the cluster and
model by providing hints and selections. For example, if a user wants to train a MLP model,
then we only need to show him/her the fields necessary for layers of MLP model. 

To implement this feature, we need to do 
1. write a daemon program to bridge SINGA system and the web interface. It receives start
request (i.e., start training) from web browsers and runs the script (i.e., /bin/singa-run)
to launch the training job. This program would later be implemented to handle other requests
like performance report which is to visualize the training performance in real-time in the
browsers.
2. decide the interface/format of communication message between the daemon program and the
web browser. To launch a training job, two configuration files are necessary, i.e., cluster
configuration and model configuration. Hence the start request should provide the corresponding
configurations. Currently the configuration fields are defined using google protocol buffer
format (e.g., model.proto and cluster.proto). However, the web pages are written in javascript
which is more familiar with json format. Current solution is to write the configuration definition
in json format and convert it into protocol buffer format using a simple script. The json
format is like
[
{
"type": "message",
"key": "ModelProto",
"value": 
   [
     { "id": 2,
    "qualifier": "required",
     "type": "string",
      "key": "name",
     "comment":"model name"} ,
    
     { "id": 3,
     "qualifier": "repeated",
     "type": "LayerProto",
      "key": "layer",
      "selection":  "LayerType",
     "comment":"layers" },
      {}...
   ]
},
{
"type": "message",
"key": "LayerProto",
"value": 
   [
     { "id": 0,
     "qualifier": "optional",
     "type": "int",
     "key": "locationid",
     "default": 0,
     "comment":"location of the layer"},
     {}...
   ]
},
{
"type":"enum",
"key": "LayerType",
value:
   [
    {"id": 2,
     "key":"kInnerProduct" },
     ...
   ]
},
...
]
If the qualifier is "optional", then a default field must be provided. The value of "selection"
is the key of a enum object, which indicates the value of that variable should be selected
from the numeration list.

The web browser parses this json format file to determine the content of the web page, i.e.,
which configuration fields should be hidden or shown. The webpage should update automatically
upon user's input. For example, if the user adds an inner product layer, then the configuration
fields of that layer should be expanded. Once the user submit the job, the web browser should
send all configurations to the daemon. The communication message is similar to the above snippet
except that a value field is added for each key field. If the qualifier for one key is "repeated",
then an array of values are added.

The daemon program then converts the json code to the google protocol buffer messages like
those in model.conf and cluster.conf.





  was:
Job Launching. Currently users have to set the model and cluster configuration by writing
the google protocol buffer files (e.g., model.conf and cluster.conf). However it is not easy
for beginning users as it requires full understanding of all configuration fields. To make
it easier, a web interface can be provided, which assist users to configure the cluster and
model by providing hints and selections. For example, if a user wants to train a MLP model,
then we only need to show him/her the fields necessary for layers of MLP model. 

To implement this feature, we need to do 
1. write a daemon program to bridge SINGA system and the web interface. It receives start
request (i.e., start training) from web browsers and runs the script (i.e., /bin/singa-run)
to launch the training job. This program would later be implemented to handle other requests
like performance report which is to visualize the training performance in real-time in the
browsers.
2. decide the interface/format of communication message between the daemon program and the
web browser. To launch a training job, two configuration files are necessary, i.e., cluster
configuration and model configuration. Hence the start request should provide the corresponding
configurations. Currently the configuration fields are defined using google protocol buffer
format (e.g., model.proto and cluster.proto). However, the web pages are written in javascript
which is more familiar with json format. Current solution is to write the configuration definition
in json format and convert it into protocol buffer format using a simple script. The json
format is like
{
"type": "message",
"key": "ModelProto",
"value": 
    {
     ["qualifier": "required",
     "type": "string",
      "key": "name",
     "comment":"model name" ],
      ["qualifier": "repeated",
     "type": "LayerProto",
      "key": "layer",
     "comment":"layers" ],
     []...
     }
}
{
"type": "message",
"key": "LayerProto",
"value": 
    {
     ["qualifier": "optional",
     "type": "int",
      "key": "locationid",
      "default": 0,
      "comment":"location of the layer"],
     []...
     }
}
...
If the qualifier is "optional", then a default field must be provided. 

The web browser parses this json format file to determine the content of the web page, i.e.,
which configuration fields should be hidden or shown. The webpage should update automatically
upon user's input. For example, if the user adds an inner product layer, then the configuration
fields of that layer should be expanded. Once the user submit the job, the web browser should
send all configurations to the daemon. The communication message is similar to the above snippet
except that a value field is added for each key field. If the qualifier for one key is "repeated",
then an array of values are added.

The daemon program then converts the json code to the google protocol buffer messages like
those in model.conf and cluster.conf.






> Web interface for launching training jobs
> -----------------------------------------
>
>                 Key: SINGA-20
>                 URL: https://issues.apache.org/jira/browse/SINGA-20
>             Project: Singa
>          Issue Type: New Feature
>            Reporter: wangwei
>            Assignee: Jinyang Gao
>
> Job Launching. Currently users have to set the model and cluster configuration by writing
the google protocol buffer files (e.g., model.conf and cluster.conf). However it is not easy
for beginning users as it requires full understanding of all configuration fields. To make
it easier, a web interface can be provided, which assist users to configure the cluster and
model by providing hints and selections. For example, if a user wants to train a MLP model,
then we only need to show him/her the fields necessary for layers of MLP model. 
> To implement this feature, we need to do 
> 1. write a daemon program to bridge SINGA system and the web interface. It receives start
request (i.e., start training) from web browsers and runs the script (i.e., /bin/singa-run)
to launch the training job. This program would later be implemented to handle other requests
like performance report which is to visualize the training performance in real-time in the
browsers.
> 2. decide the interface/format of communication message between the daemon program and
the web browser. To launch a training job, two configuration files are necessary, i.e., cluster
configuration and model configuration. Hence the start request should provide the corresponding
configurations. Currently the configuration fields are defined using google protocol buffer
format (e.g., model.proto and cluster.proto). However, the web pages are written in javascript
which is more familiar with json format. Current solution is to write the configuration definition
in json format and convert it into protocol buffer format using a simple script. The json
format is like
> [
> {
> "type": "message",
> "key": "ModelProto",
> "value": 
>    [
>      { "id": 2,
>     "qualifier": "required",
>      "type": "string",
>       "key": "name",
>      "comment":"model name"} ,
>     
>      { "id": 3,
>      "qualifier": "repeated",
>      "type": "LayerProto",
>       "key": "layer",
>       "selection":  "LayerType",
>      "comment":"layers" },
>       {}...
>    ]
> },
> {
> "type": "message",
> "key": "LayerProto",
> "value": 
>    [
>      { "id": 0,
>      "qualifier": "optional",
>      "type": "int",
>      "key": "locationid",
>      "default": 0,
>      "comment":"location of the layer"},
>      {}...
>    ]
> },
> {
> "type":"enum",
> "key": "LayerType",
> value:
>    [
>     {"id": 2,
>      "key":"kInnerProduct" },
>      ...
>    ]
> },
> ...
> ]
> If the qualifier is "optional", then a default field must be provided. The value of "selection"
is the key of a enum object, which indicates the value of that variable should be selected
from the numeration list.
> The web browser parses this json format file to determine the content of the web page,
i.e., which configuration fields should be hidden or shown. The webpage should update automatically
upon user's input. For example, if the user adds an inner product layer, then the configuration
fields of that layer should be expanded. Once the user submit the job, the web browser should
send all configurations to the daemon. The communication message is similar to the above snippet
except that a value field is added for each key field. If the qualifier for one key is "repeated",
then an array of values are added.
> The daemon program then converts the json code to the google protocol buffer messages
like those in model.conf and cluster.conf.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message