giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liannet Reyes <lianne...@gmail.com>
Subject Re: why this messages?
Date Tue, 01 Apr 2014 16:27:37 GMT
Hi Nishant,

Have you look at the jobtracker logs? (localhost:50030/jobtracker.jsp)
It is likely you will find the cause of the failure in the jobtasks logs.

Be sure you the tiny_graph file has no empty lines in the end by mistake,
 that may cause that error.

Also, I've once run into the same "Loading data ... min free memory on
worker" when I try to use more workers than the
mapred.tasktracker.map.tasks.maximum.
I guest this was the normal behaviour as is the user responsibility to
guarantee that the number of workers is less than
mapred.tasktracker.map.tasks.maximum
- 1 (master). Am I right?
However, this is not your case as you are setting w=1

Regards,
Liannet



2014-04-01 13:57 GMT+02:00 nishant gandhi <nishantgandhi99@gmail.com>:

> My code:
> import java.io.IOException;
> import java.util.Iterator;
>
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.DoubleWritable;
> import org.apache.hadoop.io.FloatWritable;
> import org.apache.giraph.edge.Edge;
> import org.apache.giraph.graph.Vertex;
> import org.apache.giraph.graph.BasicComputation;
>
> public class InDegree extends
> BasicComputation<LongWritable,DoubleWritable,FloatWritable,DoubleWritable> {
>
>     @Override
>     public void compute(
>             Vertex<LongWritable, DoubleWritable, FloatWritable> v,
>             Iterable<DoubleWritable> msg) throws IOException {
>         // TODO Auto-generated method stub
>
>
>         if(getSuperstep()==0)
>         {
>             Iterable< Edge<LongWritable,FloatWritable> > edge =
> v.getEdges();
>
>             for(Edge<LongWritable,FloatWritable> i: edge)
>             {
>                 sendMessage(i.getTargetVertexId(),new DoubleWritable(1));
>             }
>         }
>         else
>         {
>             long sum=0;
>             for (Iterator<DoubleWritable> iterator = msg.iterator();
> iterator.hasNext();)
>             {
>                 sum++;
>             }
>             v.setValue(new DoubleWritable(sum));
>             v.voteToHalt();
>         }
>
>     }
>
> }
>
>
> How i am running it:
>
> hadoop jar
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner InDegree  -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexOutputFormat
> -op /output/InDegree -w 1
>
> I am using that same classic example tiny_graph file
>
>
> On Tue, Apr 1, 2014 at 3:17 PM, ghufran malik <ghufran1malik@gmail.com>wrote:
>
>> Hi,
>>
>>
>> 14/03/31 15:48:01 INFO job.JobProgressTracker: Data from 1 workers -
>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>> average 50.18MB
>>
>> 14/03/31 15:48:06 INFO job.JobProgressTracker: Data from 1 workers -
>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>> average 50.18MB
>> 14/03/31 15:48:11 INFO job.JobProgressTracker: Data from 1 workers -
>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>> average 50.18MB
>>
>> I may be wrong but, I have received this output before, and it had
>> something to do with the format of my text file. Is your Input Format class
>> splitting the line by the separator pattern [\t ] ? if so are you
>> separating the values in your .txt file by a space or by a tab space?
>>
>> Ghufran
>>
>>
>>
>> On Tue, Apr 1, 2014 at 6:02 AM, Agrta Rawat <agrta.rawat@gmail.com>wrote:
>>
>>> Perhaps you have not specified EdgeInputFormat and EdgeOutFormat in your
>>> jar run command. And it is just a message not exception as you can see that
>>> your task runs.
>>>
>>> Regards,
>>> Agrta Rawat
>>>
>>>
>>> On Mon, Mar 31, 2014 at 10:09 PM, nishant gandhi <
>>> nishantgandhi99@gmail.com> wrote:
>>>
>>>> Why this kind of error comes? What could be wrong? Is it related with
>>>> hadoop configuration or giraph code?
>>>>
>>>>
>>>> 14/03/31 15:47:29 INFO utils.ConfigurationUtils: No edge input format
>>>> specified.  Ensure your InputFormat does not require one.
>>>> 14/03/31 15:47:29 INFO utils.ConfigurationUtils: No edge output format
>>>> specified . Ensure your OutputFormat does not require one.
>>>> 14/03/31 15:47:30 INFO job.GiraphJob: run: Since checkpointing is
>>>> disabled (defa ult), do not allow any task retries (setting
>>>> mapred.map.max.attempts = 0, old va lue = 4)
>>>> 14/03/31 15:47:31 INFO job.GiraphJob: run: Tracking URL:
>>>> http://localhost:50030/ jobdetails.jsp?jobid=job_201403310811_0012
>>>> 14/03/31 15:47:56 INFO
>>>> job.HaltApplicationUtils$DefaultHaltInstructionsWriter: w
>>>> riteHaltInstructions: To halt after next superstep execute:
>>>> 'bin/halt-applicatio n --zkServer localhost:22181 --zkNode
>>>> /_hadoopBsp/job_201403310811_0012/_haltCom putation'
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:zookeeper.version =3.4.5-1392090, built on 09/30/2012 17:52 GMT
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client environment:
>>>> host.name=localho st
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.version=1.7. 0_21
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.vendor=Oracl e Corporation
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.home=/usr/li b/jvm/java-7-openjdk-amd64/jre
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.class.path=/
>>>> usr/local/hadoop/bin/../conf:/usr/lib/jvm/java-7-openjdk-amd64/lib/tools.jar:/us
>>>> r/local/hadoop/bin/..:/usr/local/hadoop/bin/../hadoop-core-0.20.203.0.jar:/usr/l
>>>> ocal/hadoop/bin/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop/bin/../lib/aspectjt
>>>> ools-1.6.5.jar:/usr/local/hadoop/bin/../lib/commons-beanutils-1.7.0.jar:/usr/loc
>>>> al/hadoop/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/bin/../l
>>>> ib/commons-cli-1.2.jar:/usr/local/hadoop/bin/../lib/commons-codec-1.4.jar:/usr/l
>>>> ocal/hadoop/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop/bin/../li
>>>> b/commons-configuration-1.6.jar:/usr/local/hadoop/bin/../lib/commons-daemon-1.0.
>>>> 1.jar:/usr/local/hadoop/bin/../lib/commons-digester-1.8.jar:/usr/local/hadoop/bi
>>>> n/../lib/commons-el-1.0.jar:/usr/local/hadoop/bin/../lib/commons-httpclient-3.0.
>>>> 1.jar:/usr/local/hadoop/bin/../lib/commons-lang-2.4.jar:/usr/local/hadoop/bin/..
>>>> /lib/commons-logging-1.1.1.jar:/usr/local/hadoop/bin/../lib/commons-logging-api-
>>>> 1.0.4.jar:/usr/local/hadoop/bin/../lib/commons-math-2.1.jar:/usr/local/hadoop/bi
>>>> n/../lib/commons-net-1.4.1.jar:/usr/local/hadoop/bin/../lib/core-3.1.1.jar:/usr/
>>>> local/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/bin/../lib/jackson
>>>> -core-asl-1.0.1.jar:/usr/local/hadoop/bin/../lib/jackson-mapper-asl-1.0.1.jar:/u
>>>> sr/local/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop/bin/../l
>>>> ib/jasper-runtime-5.5.12.jar:/usr/local/hadoop/bin/../lib/jets3t-0.6.1.jar:/usr/
>>>> local/hadoop/bin/../lib/jetty-6.1.26.jar:/usr/local/hadoop/bin/../lib/jetty-util
>>>> -6.1.26.jar:/usr/local/hadoop/bin/../lib/jsch-0.1.42.jar:/usr/local/hadoop/bin/.
>>>> ./lib/junit-4.5.jar:/usr/local/hadoop/bin/../lib/kfs-0.2.2.jar:/usr/local/hadoop
>>>> /bin/../lib/log4j-1.2.15.jar:/usr/local/hadoop/bin/../lib/mockito-all-1.8.5.jar:
>>>> /usr/local/hadoop/bin/../lib/oro-2.0.8.jar:/usr/local/hadoop/bin/../lib/servlet-
>>>> api-2.5-20081211.jar:/usr/local/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/usr/local
>>>> /hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop/bin/../lib/xmlenc-0
>>>> .52.jar:/usr/local/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/bin/.
>>>> ./lib/jsp-2.1/jsp-api-2.1.jar
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.library.path
>>>> =/usr/local/hadoop/bin/../lib/native/Linux-amd64-64
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.io.tmpdir=/t mp
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:java.compiler=<NA >
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client environment:os.name
>>>> =Linux
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:os.arch=amd64
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:os.version=3.8.0- 23-generic
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client environment:
>>>> user.name=hduser
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:user.home=/home/h duser
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Client
>>>> environment:user.dir=/home/hd user
>>>> 14/03/31 15:47:56 INFO zookeeper.ZooKeeper: Initiating client
>>>> connection, connec tString=localhost:22181 sessionTimeout=60000
>>>> watcher=org.apache.giraph.job.JobPr ogressTracker@599a2875
>>>> 14/03/31 15:47:56 INFO mapred.JobClient: Running job:
>>>> job_201403310811_0012
>>>> 14/03/31 15:47:56 INFO zookeeper.ClientCnxn: Opening socket connection
>>>> to server  localhost/127.0.0.1:22181. Will not attempt to authenticate
>>>> using SASL (unknown  error)
>>>> 14/03/31 15:47:56 INFO zookeeper.ClientCnxn: Socket connection
>>>> established to lo calhost/127.0.0.1:22181, initiating session
>>>> 14/03/31 15:47:56 INFO zookeeper.ClientCnxn: Session establishment
>>>> complete on s erver localhost/127.0.0.1:22181, sessionid =
>>>> 0x14518d346810002, negotiated timeo ut = 600000
>>>> 14/03/31 15:47:56 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>>>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>>>> average 50.18MB
>>>> 14/03/31 15:47:57 INFO mapred.JobClient:  map 50% reduce 0%
>>>> 14/03/31 15:48:00 INFO mapred.JobClient:  map 100% reduce 0%
>>>> 14/03/31 15:48:01 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>>>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>>>> average 50.18MB
>>>> 14/03/31 15:48:06 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>>>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>>>> average 50.18MB
>>>> 14/03/31 15:48:11 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Loading dat a: 0 vertices loaded, 0 vertex input splits loaded; 0 edges
>>>> loaded, 0 edge input  splits loaded; min free memory on worker 1 - 50.18MB,
>>>> average 50.18MB
>>>> 14/03/31 15:48:16 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.47MB, average 70.47MB
>>>> 14/03/31 15:48:21 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.47MB, average 70.47MB
>>>> 14/03/31 15:48:26 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.47MB, average 70.47MB
>>>> 14/03/31 15:48:31 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.47MB, average 70.47MB
>>>> 14/03/31 15:48:36 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.29MB, average 70.29MB
>>>> 14/03/31 15:48:41 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.29MB, average 70.29MB
>>>> 14/03/31 15:48:46 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.29MB, average 70.29MB
>>>> 14/03/31 15:48:51 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 70.29MB, average 70.29MB
>>>> 14/03/31 15:48:56 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.44MB, average 69.44MB
>>>> 14/03/31 15:49:01 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.44MB, average 69.44MB
>>>> 14/03/31 15:49:06 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.44MB, average 69.44MB
>>>> 14/03/31 15:49:11 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.22MB, average 69.22MB
>>>> 14/03/31 15:49:16 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.22MB, average 69.22MB
>>>> 14/03/31 15:49:21 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.22MB, average 69.22MB
>>>> 14/03/31 15:49:26 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.22MB, average 69.22MB
>>>> 14/03/31 15:49:31 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.22MB, average 69.22MB
>>>> 14/03/31 15:49:36 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.22MB, average 69.22MB
>>>> 14/03/31 15:49:41 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.21MB, average 69.21MB
>>>> 14/03/31 15:49:46 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 69.21MB, average 69.21MB
>>>> 14/03/31 15:49:51 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 68.86MB, average 68.86MB
>>>> 14/03/31 15:49:56 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 68.86MB, average 68.86MB
>>>> 14/03/31 15:50:01 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 68.86MB, average 68.86MB
>>>> 14/03/31 15:50:06 INFO job.JobProgressTracker: Data from 1 workers -
>>>> Compute sup erstep 1: 0 out of 5 vertices computed; 0 out of 1 partitions
>>>> computed; min free  memory on worker 1 - 68.86MB, average 68.86MB
>>>> ^Z
>>>>
>>>>
>>>
>>
>

Mime
View raw message