hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Combine all Writables into a new package
Date Mon, 21 Oct 2013 22:18:29 GMT
+1 for hama-commons. Please use HAMA-699.

In my opinion, io and math package should be moved to commons module.
Currently SpMV example has already another own math writable classes.
See http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/util/

This matter should have been brought to our attention long ago. There
was an unnecessary conflicts, but creating commons module is right
way. Additionally if we move VertexInputReader to commons module, we
can simplify the code complexity of input partitioning (in core
module) and vertex job runner (in graph module).


On Tue, Oct 22, 2013 at 1:54 AM, Martin Illecker <millecker@apache.org> wrote:
> +1 for hama-io or hama-commons
>
> But we should discuss the parts that can be combined in a common module.
>
> I would suggest the following:
>
> *hama-commons module*
>
> -- io --
>
> CombineFileInputFormat.java
>
> CombineFileSplit.java
>
> FileInputFormat.java
>
> FileOutputFormat.java
>
> FileSplit.java
>
> HashPartitioner.java
>
> InputFormat.java
>
> InputSplit.java
>
> LineRecordReader.java
>
> Messagable.java
>
> MonitorManager.java
>
> NullInputFormat.java
>
> NullOutputFormat.java
>
> OutputCollector.java
>
> OutputFormat.java
>
> Partitioner.java
>
> PartitioningRunner.java
>
> RecordReader.java
>
> RecordWriter.java
>
> SequenceFileInputFormat.java
>
> SequenceFileOutputFormat.java
>
> SequenceFileRecordReader.java
>
> SequenceFileRecordWriter.java
>
> TrackedRecordReader.java
>
> TextInputFormat.java
>
> TextOutputFormat.java
>
>
> -- io.writable --
>
> MatrixWritable.java
>
> TextArrayWritable.java
>
> VectorWritable.java
>
>
> -- math --
>
> CrossEntropy.java
>
> DenseDoubleMatrix.java
>
> DenseDoubleVector.java
>
> DoubleDoubleFunction.java
>
> DoubleDoubleVectorFunction.java
>
> DoubleFunction.java
>
> DoubleMatrix.java
>
> DoubleVector.java
>
> DoubleVectorFunction.java
>
> Function.java
>
> FunctionFactory.java
>
> IdentityFunction.java
>
> Sigmoid.java
>
> SquaredError.java
>
> Tanh.java
>
> Tuple.java
>
>
> *hama-core module submitted by Suraj*
>
>
> -- bsp --
>
> BSP.java
>
> BSPInterface.java
>
> BSPJob.java
>
> BSPJobClient.java
>
> BSPJobContext.java
>
> BSPJobID.java
>
> BSPPeer.java
>
> Counters.java
>
> Combiner.java
>
> ID.java
>
> LocalBSPRunner.java
>
>
> -- bps.master --
>
> BSPMaster.java
>
>
> -- bsp.peer.ipc --
>
>
> -- bsp.peer.ipc.message --
>
> BSPMessageBundle.java
>
>
> -- bsp.peer --
>
> BSPPeerImpl.java
>
>
> -- bsp.task --
>
> BSPTask.java
>
> BSPTaskRunner.java
>
> Task.java
>
> TaskAttemptContext.java
>
> TaskAttemptID.java
>
> TaskCompletionEvent.java
>
> TaskID.java
>
> TaskInProgress.java
>
> TaskLog.java
>
> TaskLogAppender.java
>
> TaskLogServlet.java
>
> TaskRunner.java
>
> TaskStatus.java
>
>
> -- bsp.ipc --
>
> Directive.java
>
> DirectiveException.java
>
> DirectiveHandler.java
>
> DispatchTasksDirective.java
>
>
> -- bsp.ipc.action --
>
> CommitTaskAction.java
>
> GroomServerAction.java
>
> JobChangeEvent.java
>
> KillJobAction.java
>
> KillTaskAction.java
>
> LaunchTaskAction.java
>
> RecoverTaskAction.java
>
> ReinitGroomAction.java
>
> ReportGroomStatusDirective.java
>
> UpdatePeerAction.java
>
> ClusterStatus.java
>
>
> -- bsp.groom --
>
> GroomServer.java
>
> GroomServerManager.java
>
> GroomServerStatus.java
>
>
> -- bsp.job --
>
> JobInProgress.java
>
> JobInProgressListener.java
>
> JobProfile.java
>
> JobStatus.java
>
> RunningJob.java
>
>
> -- bsp.scheduler --
>
> Schedulable.java
>
> SimpleTaskScheduler.java
>
> TaskScheduler.java
>
> FCFSQueue.java
>
> Queue.java
>
> QueueManager.java
>
>
> -- bsp.superstep --
>
> Superstep.java
>
> SuperstepBSP.java
>
>
> Thanks!
>
>
> 2013/10/21 Chia-Hung Lin <clin4j@googlemail.com>
>
>> +1 for hama-io or hama-commons
>>
>>
>> On 21 October 2013 21:35, Tommaso Teofili <tommaso.teofili@gmail.com>
>> wrote:
>> > what about creating a module for that (Writables and InputFormats for
>> now)
>> > hama-io / hama-commons that can be used by both (containing math stuff as
>> > well) ?
>> >
>> > Tommaso
>> >
>> >
>> > 2013/10/21 Martin Illecker <millecker@apache.org>
>> >
>> >> VectorWritable and MatrixWritable have both some dependencies
>> >> to org.apache.hama.ml.math.
>> >> (DenseDoubleVector, DoubleVector and DenseDoubleMatrix, DoubleMatrix)
>> >>
>> >> If we move VectorWritable and MatrixWritable to core (e.g.,
>> >> org.apache.hama.io.writable), we have to move org.apache.hama.ml.math as
>> >> well.
>> >> I think that's not possible because of other classes in hama-ml
>> depending
>> >> on ml.math.
>> >>
>> >> Temporary I will have to copy VectorWritable to the core to use it in a
>> >> test case.
>> >>
>> >>
>> >> 2013/10/21 Tommaso Teofili <tommaso.teofili@gmail.com>
>> >>
>> >> > 2013/10/21 Martin Illecker <millecker@apache.org>
>> >> >
>> >> > > Hello,
>> >> > >
>> >> > > regarding to my Hama Pipes test case [1], I want to use
>> VectorWritable
>> >> > > inside the hama-core module.
>> >> > > Therefore I would need a dependency to hama-ml but this will cause
a
>> >> > cyclic
>> >> > > dependency.
>> >> > >
>> >> > > So is it possible to move both writables, VectorWritable and
>> >> > > MatrixWritable, from org.apache.hama.ml.writable into a new package?
>> >> > > e.g., org.apache.hama.io.writable based on [2]
>> >> > >
>> >> >
>> >> > I think this really makes sense.
>> >> >
>> >> >
>> >> > >
>> >> > > Regarding to [3] we can also move TextArrayWritable
>> >> > > from org.apache.hama.bsp into this new package.
>> >> > >
>> >> > > Do you think we can move the writables of
>> org.apache.hama.ml.writable
>> >> to
>> >> > > the core module?
>> >> > >
>> >> >
>> >> > +1
>> >> >
>> >> >
>> >> > > And can we do the package refactoring [2] of org.apache.hama.bsp
>> >> > submitted
>> >> > > by Suraj?
>> >> > >
>> >> >
>> >> > +1 here too.
>> >> >
>> >> > Tommaso
>> >> >
>> >> >
>> >> > >
>> >> > > Thanks!
>> >> > >
>> >> > > Martin
>> >> > >
>> >> > > [1] https://issues.apache.org/jira/browse/HAMA-808
>> >> > > [2]
>> >> > https://issues.apache.org/jira/secure/attachment/12609417/bsplist.txt
>> >> > > [3] https://issues.apache.org/jira/browse/HAMA-727
>> >> > >
>> >> >
>> >>
>>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message