Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A3AE710F65 for ; Tue, 7 May 2013 18:43:21 +0000 (UTC) Received: (qmail 43535 invoked by uid 500); 7 May 2013 18:43:21 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 43488 invoked by uid 500); 7 May 2013 18:43:21 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 43431 invoked by uid 99); 7 May 2013 18:43:21 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 May 2013 18:43:21 +0000 Received: from localhost (HELO [172.19.44.57]) (127.0.0.1) (smtp-auth username aching, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 May 2013 18:43:20 +0000 Message-ID: <51894B47.9070300@apache.org> Date: Tue, 07 May 2013 11:43:19 -0700 From: Avery Ching User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130216 Thunderbird/17.0.3 MIME-Version: 1.0 To: user@giraph.apache.org CC: Kiru Pakkirisamy Subject: Re: TestJsonBase64Format failure on 1.0.0 References: <1367952034.23233.YahooMailClassic@web181602.mail.ne1.yahoo.com> In-Reply-To: <1367952034.23233.YahooMailClassic@web181602.mail.ne1.yahoo.com> Content-Type: multipart/alternative; boundary="------------040505030408010908010907" This is a multi-part message in MIME format. --------------040505030408010908010907 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Here's an easy way to calculate the number of map slots required. workers + 1. So for instance if you only have 2 map slots, you can only run at most one worker. If you're running on your laptop, feel free to change the number of map slots to have you need. Hope that helps, Avery On 5/7/13 11:40 AM, Kiru Pakkirisamy wrote: > Avery et al, > Please let me know if I should use Giraph only on a multi-node cluster > (what is the min # of nodes ?) > > Regards, > - kiru > > Kiru Pakkirisamy | webcloudtech.wordpress.com > > --- On *Mon, 5/6/13, Kiru Pakkirisamy //* > wrote: > > > From: Kiru Pakkirisamy > Subject: TestJsonBase64Format failure on 1.0.0 > To: user@giraph.apache.org > Date: Monday, May 6, 2013, 9:40 AM > > I got over my compilation issues (thanks - @Avery, @Roman). > Now, I am trying to run the test and one pearticular test is failing. > I want to get to the bottom of this, because I am unable to run > the PageRank example. Maybe, it is because I have only on > tasktracker (?) (Apache pseudo-cluster on my ubuntu laptop). > > Regards, > - kiru > > 2013-05-06 09:12:32,197 INFO org.apache.hadoop.mapred.JobTracker: > Adding task (MAP) 'attempt_201305052325_0013_m_000003_0' to tip > task_201305052325_0013_m_000003, for tracker > 'tracker_kiru-N53SV:localhost/127.0.0.1:42265' > 2013-05-06 09:12:35,198 INFO > org.apache.hadoop.mapred.TaskInProgress: Error from > attempt_201305052325_0013_m_000002_0: > java.lang.IllegalStateException: run: Caught an unrecoverable > exception waitFor: ExecutionException occurred while waiting for > org.apache.giraph.utils.ProgressableUtils$FutureWaitable@482d59a3 > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.lang.IllegalStateException: waitFor: > ExecutionException occurred while waiting for > org.apache.giraph.utils.ProgressableUtils$FutureWaitable@482d59a3 > at > org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151) > at > org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:111) > at > org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:73) > at > org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:192) > at > org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:276) > at > org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:323) > at > org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:506) > at > org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:230) > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92) > ... 7 more > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: call: IOException > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232) > at java.util.concurrent.FutureTask.get(FutureTask.java:91) > at > org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:271) > at > org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143) > ... 15 more > Caused by: java.lang.IllegalStateException: call: IOException > at > org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:172) > at > org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:58) > at > org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.FileNotFoundException: > /tmp/_giraphTests/testContinue/_logs (Is a directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:120) > at > org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.(RawLocalFileSystem.java:71) > at > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.(RawLocalFileSystem.java:107) > at > org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:177) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:126) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67) > at > org.apache.giraph.io.formats.TextVertexInputFormat$TextVertexReader.initialize(TextVertexInputFormat.java:96) > at > org.apache.giraph.io.formats.JsonBase64VertexInputFormat$JsonBase64VertexReader.initialize(JsonBase64VertexInputFormat.java:71) > at > org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:120) > at > org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:220) > at > org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:161) > ... 7 more > > 2013-05-06 09:22:44,485 INFO > org.apache.hadoop.mapred.TaskInProgress: Error from > attempt_201305052325_0013_m_000002_0: Task > attempt_201305052325_0013_m_000002_0 failed to report status for > 602 seconds. Killing! > 2013-05-06 09:22:44,485 INFO > org.apache.hadoop.mapred.TaskInProgress: TaskInProgress > task_201305052325_0013_m_000002 has failed 1 times. > 2013-05-06 09:22:44,485 INFO > org.apache.hadoop.mapred.JobInProgress: Aborting job > job_201305052325_0013 > 2013-05-06 09:22:44,485 INFO > org.apache.hadoop.mapred.JobInProgress: Killing job > 'job_201305052325_0013' > > Regards, > - kiru > > Kiru Pakkirisamy | webcloudtech.wordpress.com > > --- On *Mon, 5/6/13, Kiru Pakkirisamy > //*wrote: > > > From: Kiru Pakkirisamy > Subject: Re: Compiling 1.0.0 distribution > To: user@giraph.apache.org, "Avery Ching" > Date: Monday, May 6, 2013, 12:02 AM > > Yes, I am trying to run on my Ubuntu laptop. Let me look at > the log files. Thanks for the help. Much appreciated. > > Regards, > - kiru > > Kiru Pakkirisamy | webcloudtech.wordpress.com > > --- On *Sun, 5/5/13, Avery Ching //* wrote: > > > From: Avery Ching > Subject: Re: Compiling 1.0.0 distribution > To: user@giraph.apache.org > Cc: "Kiru Pakkirisamy" > Date: Sunday, May 5, 2013, 11:51 PM > > My guess is that you don't have enough workers to run the > job and the master kills the job (i.e. are you running on > a single machine setup?). You can try to run first with > one worker (this will take 2 map slots - one for the > master and one for the worker). You can also look at the > logs from map task 0 to see more clearly what the error was. > > Avery > > On 5/5/13 11:16 PM, Kiru Pakkirisamy wrote: >> Yup, I did a mvn3 install and then a mvn3 compile to get >> around that already. >> Right now, I am trying to run the PageRank, even after a >> few runs I have not had one successful run . The maps >> progress decreases in percentage (second time around) !! >> I have never seen this before (?) >> >> Regards, >> - kiru >> >> Kiru Pakkirisamy | webcloudtech.wordpress.com >> >> --- On *Sun, 5/5/13, Roman Shaposhnik //* >> wrote: >> >> >> From: Roman Shaposhnik >> Subject: Re: Compiling 1.0.0 distribution >> To: user@giraph.apache.org >> Date: Sunday, May 5, 2013, 10:50 PM >> >> To pile on top of that -- you can also run mvn -pl >> module-name from the top >> level to short-circuit the build to that module (and >> yet still honor the dependencies). >> >> Thanks, >> Roman. >> >> On Sun, May 5, 2013 at 10:44 PM, Avery Ching >> wrote: >> >> The easiest way is to compile from the base >> directory, which will build everything. >> >> You can build individual directories, but you >> have to install the core jars first (i.e. go to >> giraph-core and do 'mvn clean install'). Then you >> can build the directory of your choice. >> >> Hope that helps, >> >> Avery >> >> On 5/5/13 11:11 AM, Kiru Pakkirisamy wrote: >>> Hi, >>> I am unable to compile giraph-examples because >>> it is not able to reach the core jar files on >>> the repo. Why doesn't it pick it up from the >>> root build dir ? >>> >>> Regards, >>> - kiru >>> >>> Kiru Pakkirisamy | webcloudtech.wordpress.com >>> >>> >> >> > --------------040505030408010908010907 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Here's an easy way to calculate the number of map slots required.  workers + 1.  So for instance if you only have 2 map slots, you can only run at most one worker.  If you're running on your laptop, feel free to change the number of map slots to have you need.

Hope that helps,

Avery

On 5/7/13 11:40 AM, Kiru Pakkirisamy wrote:
Avery et al,
Please let me know if I should use Giraph only on a multi-node cluster (what is the min # of nodes ?)

Regards,
- kiru

Kiru Pakkirisamy | webcloudtech.wordpress.com

--- On Mon, 5/6/13, Kiru Pakkirisamy <kirupakkirisamy@yahoo.com> wrote:

From: Kiru Pakkirisamy <kirupakkirisamy@yahoo.com>
Subject: TestJsonBase64Format failure on 1.0.0
To: user@giraph.apache.org
Date: Monday, May 6, 2013, 9:40 AM

I got over my compilation issues (thanks - @Avery, @Roman).
Now, I am trying to run the test and one pearticular test is failing.
I want to get to the bottom of this, because I am unable to run the PageRank example. Maybe, it is because I have only on tasktracker (?) (Apache pseudo-cluster on my ubuntu laptop).

Regards,
- kiru

2013-05-06 09:12:32,197 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201305052325_0013_m_000003_0' to tip task_201305052325_0013_m_000003, for tracker 'tracker_kiru-N53SV:localhost/127.0.0.1:42265'
2013-05-06 09:12:35,198 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305052325_0013_m_000002_0: java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@482d59a3
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@482d59a3
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:111)
at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:73)
at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:192)
at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:276)
at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:323)
at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:506)
at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:230)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: call: IOException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:271)
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
... 15 more
Caused by: java.lang.IllegalStateException: call: IOException
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:172)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:58)
at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /tmp/_giraphTests/testContinue/_logs (Is a directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.<init>(RawLocalFileSystem.java:71)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:107)
at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:177)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:126)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
at org.apache.giraph.io.formats.TextVertexInputFormat$TextVertexReader.initialize(TextVertexInputFormat.java:96)
at org.apache.giraph.io.formats.JsonBase64VertexInputFormat$JsonBase64VertexReader.initialize(JsonBase64VertexInputFormat.java:71)
at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:120)
at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:220)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:161)
... 7 more

2013-05-06 09:22:44,485 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305052325_0013_m_000002_0: Task attempt_201305052325_0013_m_000002_0 failed to report status for 602 seconds. Killing!
2013-05-06 09:22:44,485 INFO org.apache.hadoop.mapred.TaskInProgress: TaskInProgress task_201305052325_0013_m_000002 has failed 1 times.
2013-05-06 09:22:44,485 INFO org.apache.hadoop.mapred.JobInProgress: Aborting job job_201305052325_0013
2013-05-06 09:22:44,485 INFO org.apache.hadoop.mapred.JobInProgress: Killing job 'job_201305052325_0013'

Regards,
- kiru

Kiru Pakkirisamy | webcloudtech.wordpress.com

--- On Mon, 5/6/13, Kiru Pakkirisamy <kirupakkirisamy@yahoo.com> wrote:

From: Kiru Pakkirisamy <kirupakkirisamy@yahoo.com>
Subject: Re: Compiling 1.0.0 distribution
To: user@giraph.apache.org, "Avery Ching" <aching@apache.org>
Date: Monday, May 6, 2013, 12:02 AM

Yes, I am trying to run on my Ubuntu laptop. Let me look at the log files. Thanks for the help. Much appreciated.

Regards,
- kiru

Kiru Pakkirisamy | webcloudtech.wordpress.com

--- On Sun, 5/5/13, Avery Ching <aching@apache.org> wrote:

From: Avery Ching <aching@apache.org>
Subject: Re: Compiling 1.0.0 distribution
To: user@giraph.apache.org
Cc: "Kiru Pakkirisamy" <kirupakkirisamy@yahoo.com>
Date: Sunday, May 5, 2013, 11:51 PM

My guess is that you don't have enough workers to run the job and the master kills the job (i.e. are you running on a single machine setup?).  You can try to run first with one worker (this will take 2 map slots - one for the master and one for the worker).  You can also look at the logs from map task 0 to see more clearly what the error was.

Avery

On 5/5/13 11:16 PM, Kiru Pakkirisamy wrote:
Yup, I did a mvn3 install and then a mvn3 compile to get around that already.
Right now, I am trying to run the PageRank, even after a few runs I have not had one successful run . The maps progress decreases in percentage (second time around) !! I have never seen this before (?)

Regards,
- kiru

Kiru Pakkirisamy | webcloudtech.wordpress.com

--- On Sun, 5/5/13, Roman Shaposhnik <rvs@apache.org> wrote:

From: Roman Shaposhnik <rvs@apache.org>
Subject: Re: Compiling 1.0.0 distribution
To: user@giraph.apache.org
Date: Sunday, May 5, 2013, 10:50 PM

To pile on top of that -- you can also run mvn -pl module-name from the top
level to short-circuit the build to that module (and yet still honor the dependencies).

Thanks,
Roman.

On Sun, May 5, 2013 at 10:44 PM, Avery Ching <aching@apache.org> wrote:
The easiest way is to compile from the base directory, which will build everything.

You can build individual directories, but you have to install the core jars first (i.e. go to giraph-core and do 'mvn clean install').  Then you can build the directory of your choice.

Hope that helps,

Avery

On 5/5/13 11:11 AM, Kiru Pakkirisamy wrote:
Hi,
I am unable to compile giraph-examples because it is not able to reach the core jar files on the repo. Why doesn't it pick it up from the root build dir ?

Regards,
- kiru

Kiru Pakkirisamy | webcloudtech.wordpress.com




--------------040505030408010908010907--