incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (Commented) (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-100) Data input sampling and testing improvements
Date Thu, 01 Dec 2011 22:08:41 GMT


Jakob Homan commented on GIRAPH-100:

bq. Which exceptions are you referring to?
{noformat}+                    throw new IllegalStateException(
+                        "prepareSuperstep: Impossible that this worker " +
+                        service.getWorkerInfo() + " was sent " +
+                        entry.getValue().size() + " message(s) with " +
+                        "vertex id " + entry.getKey() +
+                        " when it does not own this partition.  It should " +
+                        "have gone to partition owner " +
+                        service.getVertexPartitionOwner(entry.getKey()) +
+                        ".  The partition owners are " +
+                        service.getPartitionOwners());{noformat}
{noformat}+                            throw new IllegalStateException(
+                                "prepareSuperstep: Impossible to not remove " +
+                                vertex);{noformat}
{noformat}+                throw new IllegalStateException(
+                    "coordinateSuperstep: Worker failed during input split " +
+                    "(currently not supported)");{noformat}
{noformat}+                throw new IllegalStateException(
+                    "barrierOnWorkerList: KeeperException - " +
+                    "Couldn't get " + workerInfoHealthyPath, e);{noformat}
{noformat}+            throw new IllegalStateException(
+                "loadVertices: KeeperException on " +
+                inputSplitFinishedPath, e);{noformat}
etc. These are all specific types of exceptions being wrapped in IllegalStateException.  We'll
likely want to catch and handle them later in an effort to be more robust. It'd be better
if these existed as their own types, so we don't have to try to tease out the details later.
bq. Can we do that in another issue? I agree that it isn't a good example, but it's still
a good test since it guarantees partition movement between workers.
I have trouble putting something that we agree is a bad example into the example directory.
The issue is that it's not actually a unit test, since it involves Hadoop.  That makes it
an integration test.  The best answer is to have integration tests in their own directory
(and either bundled with the main jar or a separate integration test directory).  Since this
verifies important behavior, the basic test itself should run without Hadoop, via mocking,
and the ability to run it as an integration test under a real Hadoop maintained.
> Data input sampling and testing improvements
> --------------------------------------------
>                 Key: GIRAPH-100
>                 URL:
>             Project: Giraph
>          Issue Type: New Feature
>          Components: graph
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>         Attachments: GIRAPH-100.patch
> It would be really nice to help debug an application by limiting the input data (% of
input splits, max vertices per input split).  Also, it would be nice for the workers to provide
a little more debugging info on how far along they are with processing the input data.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message