Return-Path: X-Original-To: apmail-hama-dev-archive@www.apache.org Delivered-To: apmail-hama-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7F143108A8 for ; Mon, 21 Oct 2013 22:18:42 +0000 (UTC) Received: (qmail 37950 invoked by uid 500); 21 Oct 2013 22:18:36 -0000 Delivered-To: apmail-hama-dev-archive@hama.apache.org Received: (qmail 37898 invoked by uid 500); 21 Oct 2013 22:18:34 -0000 Mailing-List: contact dev-help@hama.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hama.apache.org Delivered-To: mailing list dev@hama.apache.org Received: (qmail 37873 invoked by uid 99); 21 Oct 2013 22:18:30 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Oct 2013 22:18:30 +0000 Received: from localhost (HELO mail-ie0-f172.google.com) (127.0.0.1) (smtp-auth username edwardyoon, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Oct 2013 22:18:30 +0000 Received: by mail-ie0-f172.google.com with SMTP id tp5so284503ieb.17 for ; Mon, 21 Oct 2013 15:18:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=n7SP+dg8CuFJwk8LxAefK7HMgI9KkQy2aC6MC77+hOY=; b=bRvNiAauZjRHrPrZ1vi8FKi493Aj6dnF1SSwrhZk/2bnVbRTtHDb82BaPRDF+WFKAa CH83+Kt/XqLqJpdin6aZjQLz/6ZSZtr9smdohCU0EKljqnKfKiOD5Q6nei7aoTjRBcqt d5m/B4BfJRZIvMOFyN/P1THHdt4xyRPfGp0Xre+Yg8XCzsyi8eOigt0LSPNHVezOM77Y xprAKCDDAUtf2IgB4nfot/O2nwytp4AetTxriZDA2nh3M8Akre9iHXaJAeAFoaEZsR1p SZzhnOak3qmiP+kR/+MSW63oHPVTUTrgAPibs0wFjmkzVYTYJASa7mijCiAZyANgPan6 yHPA== X-Gm-Message-State: ALoCoQkN0kkNxsHqQk316sv7fxlhodsW6l5ZMR9BPvekXtcMQq+fKMoyGuY6+vNt1scEGZXN2FAv MIME-Version: 1.0 X-Received: by 10.50.109.132 with SMTP id hs4mr11113293igb.34.1382393909471; Mon, 21 Oct 2013 15:18:29 -0700 (PDT) Received: by 10.64.249.74 with HTTP; Mon, 21 Oct 2013 15:18:29 -0700 (PDT) In-Reply-To: References: Date: Tue, 22 Oct 2013 07:18:29 +0900 Message-ID: Subject: Re: Combine all Writables into a new package From: "Edward J. Yoon" To: "dev@hama.apache.org" Content-Type: text/plain; charset=UTF-8 +1 for hama-commons. Please use HAMA-699. In my opinion, io and math package should be moved to commons module. Currently SpMV example has already another own math writable classes. See http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/util/ This matter should have been brought to our attention long ago. There was an unnecessary conflicts, but creating commons module is right way. Additionally if we move VertexInputReader to commons module, we can simplify the code complexity of input partitioning (in core module) and vertex job runner (in graph module). On Tue, Oct 22, 2013 at 1:54 AM, Martin Illecker wrote: > +1 for hama-io or hama-commons > > But we should discuss the parts that can be combined in a common module. > > I would suggest the following: > > *hama-commons module* > > -- io -- > > CombineFileInputFormat.java > > CombineFileSplit.java > > FileInputFormat.java > > FileOutputFormat.java > > FileSplit.java > > HashPartitioner.java > > InputFormat.java > > InputSplit.java > > LineRecordReader.java > > Messagable.java > > MonitorManager.java > > NullInputFormat.java > > NullOutputFormat.java > > OutputCollector.java > > OutputFormat.java > > Partitioner.java > > PartitioningRunner.java > > RecordReader.java > > RecordWriter.java > > SequenceFileInputFormat.java > > SequenceFileOutputFormat.java > > SequenceFileRecordReader.java > > SequenceFileRecordWriter.java > > TrackedRecordReader.java > > TextInputFormat.java > > TextOutputFormat.java > > > -- io.writable -- > > MatrixWritable.java > > TextArrayWritable.java > > VectorWritable.java > > > -- math -- > > CrossEntropy.java > > DenseDoubleMatrix.java > > DenseDoubleVector.java > > DoubleDoubleFunction.java > > DoubleDoubleVectorFunction.java > > DoubleFunction.java > > DoubleMatrix.java > > DoubleVector.java > > DoubleVectorFunction.java > > Function.java > > FunctionFactory.java > > IdentityFunction.java > > Sigmoid.java > > SquaredError.java > > Tanh.java > > Tuple.java > > > *hama-core module submitted by Suraj* > > > -- bsp -- > > BSP.java > > BSPInterface.java > > BSPJob.java > > BSPJobClient.java > > BSPJobContext.java > > BSPJobID.java > > BSPPeer.java > > Counters.java > > Combiner.java > > ID.java > > LocalBSPRunner.java > > > -- bps.master -- > > BSPMaster.java > > > -- bsp.peer.ipc -- > > > -- bsp.peer.ipc.message -- > > BSPMessageBundle.java > > > -- bsp.peer -- > > BSPPeerImpl.java > > > -- bsp.task -- > > BSPTask.java > > BSPTaskRunner.java > > Task.java > > TaskAttemptContext.java > > TaskAttemptID.java > > TaskCompletionEvent.java > > TaskID.java > > TaskInProgress.java > > TaskLog.java > > TaskLogAppender.java > > TaskLogServlet.java > > TaskRunner.java > > TaskStatus.java > > > -- bsp.ipc -- > > Directive.java > > DirectiveException.java > > DirectiveHandler.java > > DispatchTasksDirective.java > > > -- bsp.ipc.action -- > > CommitTaskAction.java > > GroomServerAction.java > > JobChangeEvent.java > > KillJobAction.java > > KillTaskAction.java > > LaunchTaskAction.java > > RecoverTaskAction.java > > ReinitGroomAction.java > > ReportGroomStatusDirective.java > > UpdatePeerAction.java > > ClusterStatus.java > > > -- bsp.groom -- > > GroomServer.java > > GroomServerManager.java > > GroomServerStatus.java > > > -- bsp.job -- > > JobInProgress.java > > JobInProgressListener.java > > JobProfile.java > > JobStatus.java > > RunningJob.java > > > -- bsp.scheduler -- > > Schedulable.java > > SimpleTaskScheduler.java > > TaskScheduler.java > > FCFSQueue.java > > Queue.java > > QueueManager.java > > > -- bsp.superstep -- > > Superstep.java > > SuperstepBSP.java > > > Thanks! > > > 2013/10/21 Chia-Hung Lin > >> +1 for hama-io or hama-commons >> >> >> On 21 October 2013 21:35, Tommaso Teofili >> wrote: >> > what about creating a module for that (Writables and InputFormats for >> now) >> > hama-io / hama-commons that can be used by both (containing math stuff as >> > well) ? >> > >> > Tommaso >> > >> > >> > 2013/10/21 Martin Illecker >> > >> >> VectorWritable and MatrixWritable have both some dependencies >> >> to org.apache.hama.ml.math. >> >> (DenseDoubleVector, DoubleVector and DenseDoubleMatrix, DoubleMatrix) >> >> >> >> If we move VectorWritable and MatrixWritable to core (e.g., >> >> org.apache.hama.io.writable), we have to move org.apache.hama.ml.math as >> >> well. >> >> I think that's not possible because of other classes in hama-ml >> depending >> >> on ml.math. >> >> >> >> Temporary I will have to copy VectorWritable to the core to use it in a >> >> test case. >> >> >> >> >> >> 2013/10/21 Tommaso Teofili >> >> >> >> > 2013/10/21 Martin Illecker >> >> > >> >> > > Hello, >> >> > > >> >> > > regarding to my Hama Pipes test case [1], I want to use >> VectorWritable >> >> > > inside the hama-core module. >> >> > > Therefore I would need a dependency to hama-ml but this will cause a >> >> > cyclic >> >> > > dependency. >> >> > > >> >> > > So is it possible to move both writables, VectorWritable and >> >> > > MatrixWritable, from org.apache.hama.ml.writable into a new package? >> >> > > e.g., org.apache.hama.io.writable based on [2] >> >> > > >> >> > >> >> > I think this really makes sense. >> >> > >> >> > >> >> > > >> >> > > Regarding to [3] we can also move TextArrayWritable >> >> > > from org.apache.hama.bsp into this new package. >> >> > > >> >> > > Do you think we can move the writables of >> org.apache.hama.ml.writable >> >> to >> >> > > the core module? >> >> > > >> >> > >> >> > +1 >> >> > >> >> > >> >> > > And can we do the package refactoring [2] of org.apache.hama.bsp >> >> > submitted >> >> > > by Suraj? >> >> > > >> >> > >> >> > +1 here too. >> >> > >> >> > Tommaso >> >> > >> >> > >> >> > > >> >> > > Thanks! >> >> > > >> >> > > Martin >> >> > > >> >> > > [1] https://issues.apache.org/jira/browse/HAMA-808 >> >> > > [2] >> >> > https://issues.apache.org/jira/secure/attachment/12609417/bsplist.txt >> >> > > [3] https://issues.apache.org/jira/browse/HAMA-727 >> >> > > >> >> > >> >> >> -- Best Regards, Edward J. Yoon @eddieyoon