giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doron Cohen <>
Subject what is the recommended way to develop Giraph code with Eclipse or other IDEs
Date Mon, 22 Oct 2012 09:22:19 GMT
Hi, I am new to Giraph.

Looking at GIRAPH-42 I used Eclipse (on Windows 7) to modify the code.

The process I went through was quite cumbersome, so I wonder if there is a
more efficient way...

Here is what I did.
Checked out giraph-trunk and ran this command:
   mvn -Dhadoop=non_secure clean eclipse:eclipse

This created:
- the giraph/target dir with munged code suitable for unsecured env
- Eclipse definition files .project and .classpath for two sub-projects:
giraph and giraph-formats-contrib.

I concentrated in the giraph sub project which has the sent messages
counters code (and all of giraph core I assume).

It turns out that the Eclipse project has - as source folders - the target
source, result of munging (pre-processing).
(Never used munge before, just read about it now trying to figure out what
was going on):
- target/munged/test
- target/munged/main

The Eclipse project compiles with no errors.

Tests are running ok from command line, like this: mvn -Dhadoop=non_secure
(Giraph Formats build fails but ignoring this now)
The output summary is:
   [INFO] Apache Giraph Parent .............................. SUCCESS
   [INFO] Apache Giraph ..................................... SUCCESS
   [INFO] Apache Giraph Formats ............................. FAILURE
(many exception stack traces printed, but ignoring this for now).

Running the tests from Eclipse fails: 16 out of 120 tests are failing, most
or all of them with this exception:
java.lang.IllegalArgumentException: Can not create a Path from an empty
    at org.apache.hadoop.fs.Path.checkPathArg(
    at org.apache.hadoop.fs.Path.<init>(
    at org.apache.hadoop.mapreduce.Job.submit(
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(

This makes it hard to fix tests, or run them through a debugger.
Also, each execution of the mvn command line wipes out the the target dir
and the Eclipse project must be refreshed.
It will also undo any code changes that were done within Eclipse, as the
sources under target are rewritten.

Last, the source files under target are not in SVN, so it is not possible
to use svn stat/diff etc. to see what has changed or to create a patch.
To workaround this, I opened the source files (as resources) in Eclipse and
edited, then ran the mvn target and refreshed Eclipse.
This is quite inconvenient and might lead to errors.

This is also mentioned in,
which is also where I learned about the eclipse:eclipse target.

How do others develop Giraph code then?
Using Eclipse or other IDEs perhaps? (how?)
Plain VI or EMACS editor with command line mvn commands?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message