Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <816238670.1231891019859.JavaMail.jira@brutus>
Date: Tue, 13 Jan 2009 15:56:59 -0800 (PST)
From: "Jerome Boulon (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Issue Comment Edited: (HADOOP-5018) Chukwa should support
 pipelined writers
In-Reply-To: <2124989561.1231812659831.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663552#action_12663552 ] 

jboulon edited comment on HADOOP-5018 at 1/13/09 3:56 PM:
----------------------------------------------------------------

My point is hat the pipelineWriter should be an implementation of the ChukwaWriter interface and that's really the only thing that the collector should be aware of.
So to be able to do what you want:

1) The collector should instantiate one writer implementation based on his configuration
2) The writer should be able to get the collector configuration from somewhere (current design) or should have an init method with a Configuration parameter
3) The contract from the collector point of view is the same call one method on the writer class and the result is success if there's no exception

the delta with your implementation is:

- Remove code from     if (conf.get("chukwaCollector.pipeline") != null) ..
- Replace by something like:

writerClassName = conf.get("chukwaCollector.writer","org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter").
Class myWriter = conf.getClassByName(writerClassName);
Writer st = myWriter.newInstance()
st.init();

- Remove all writer initialization from CollectorStub.java
- and move all the code to create the pipeline to the init method inside a PipelineWriter class, instead of ServletCollector.java

That way the writer interface is still simple, the collector class stay very simple and this does not prevent anybody from having a specific writer implementation.
So at the end you have:

public class PipelineWriter implements ChukwaWriter
{

public void init() throws WriterException
{
+    if (conf.get("chukwaCollector.pipeline") != null) {
+      String pipeline = conf.get("chukwaCollector.pipeline");
+      try {
+        String[] classes = pipeline.split(",");
+        ArrayList<PipelineStageWriter> stages = new ArrayList<PipelineStageWriter>();
[...]
}

public void add(List<Chunk> chunks) throws WriterException
{
// call all PipelineStageWriter in sequence
}

}


      was (Author: jboulon):
    My point is hat the pipelineWriter should be an implementation of the ChukwaWriter interface and that's really the only thing that the collector should be aware of.
So to be able to do what you want:

1) The collector should instantiate one writer implementation based on his configuration
2) The writer should be able to get the collector configuration from somewhere (current design) or should have an init method with a Configuration parameter
3) The contract from the collector point of view is the same call one method on the writer class and the result is success if there's no exception

the delta with your implementation is:

- Remove code from     if (conf.get("chukwaCollector.pipeline") != null) ..
- Replace by something like:

writerClassName = conf.get("chukwaCollector.writer","org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter").
Class myWriter = conf.getClassByName(writerClassName);
Writer st = myWriter.newInstance()
st.init();

- Remove all writer initialization from CollectorStub.java
- and move all the code to create the pipeline to the init method inside a PipelineWriter class, instead of ServletCollector.java

That way the writer interface is still simple, the collector class stay very simple and this does not prevent anybody from having a specific writer implementation.
So at the end you have:

public class PipelineWriter implements ChukwaWriter
{

public void init() throws WriterException
{
+    if (conf.get("chukwaCollector.pipeline") != null) {
+      String pipeline = conf.get("chukwaCollector.pipeline");
+      try {
+        String[] classes = pipeline.split(",");
+        ArrayList<PipelineStageWriter> stages = new ArrayList<PipelineStageWriter>();
[...]
}

public void add(List<Chunk> chunks) throws WriterException
{
// call all PipelineStageWriter in sequence
}


> Chukwa should support pipelined writers
> ---------------------------------------
>
>                 Key: HADOOP-5018
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5018
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: contrib/chukwa
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>         Attachments: pipeline.patch
>
>
> We ought to support chaining together writers; this will radically increase flexibility and make it practical to add new features without major surgery by putting them in pass-through or filter classes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.