hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: [CIS-CMMI-3] IllegalArgumentException: Row length 41221 is > 32767
Date Thu, 21 Jan 2016 17:09:20 GMT
Maximum size of row in HBase is 32767 and your application is trying to get
rows which exceed this limit.

You probably better ask your question in gora/nutch user group.

-Vlad

On Thu, Jan 21, 2016 at 5:39 AM, Kshitij Shukla <kshitij.s@cisinlabs.com>
wrote:

> Hello everyone,
>
> Software stack is *nutch-branch-2.3.1, gora-hbase 0.6.1 Hadoop 2.5.2,
> hbase-0.98.8-hadoop2
>
> * I have added a set of seeds to crawl using this command
> *
> ./bin/crawl /largeSeeds 1 http://localhost:8983/solr/ddcd 4*
>
> For first iteration all of the commands(*inject, **generate, **fetch,
> **parse, **update-table, **Indexer & delete duplicates.*) got executed
> successfully.
> For second iteration, *"CrawlDB update" *command got failed (please see
> error log for reference), because of failure of this command the whole
> process gets terminated.
>
>
> ****************************************************LOG
> START************************************************************************************************
> 16/01/20 02:45:19 INFO parse.ParserJob: ParserJob: finished at 2016-01-20
> 02:45:19, time elapsed: 00:06:57
> CrawlDB update for 1
> /usr/share/searchEngine/nutch-branch-2.3.1/runtime/deploy/bin/nutch
> updatedb -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D
> mapred.reduce.tasks.speculative.execution=false -D
> mapred.map.tasks.speculative.execution=false -D
> mapred.compress.map.output=true 1453230757-13191 -crawlId 1
> 16/01/20 02:45:27 INFO crawl.DbUpdaterJob: DbUpdaterJob: starting at
> 2016-01-20 02:45:27
> 16/01/20 02:45:27 INFO crawl.DbUpdaterJob: DbUpdaterJob: batchId:
> 1453230757-13191
> 16/01/20 02:45:27 INFO plugin.PluginRepository: Plugins: looking in:
> /tmp/hadoop-root/hadoop-unjar5654418190157422003/classes/plugins
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Plugin Auto-activation
> mode: [true]
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Registered Plugins:
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     HTTP Framework
> (lib-http)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Html Parse Plug-in
> (parse-html)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     MetaTags
> (parse-metatags)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     the nutch core
> extension points (nutch-extensionpoints)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Basic Indexing Filter
> (index-basic)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     XML Libraries (lib-xml)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Anchor Indexing Filter
> (index-anchor)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Basic URL Normalizer
> (urlnormalizer-basic)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Language
> Identification Parser/Filter (language-identifier)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Metadata Indexing
> Filter (index-metadata)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     CyberNeko HTML Parser
> (lib-nekohtml)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Subcollection indexing
> and query filter (subcollection)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: SOLRIndexWriter
> (indexer-solr)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Rel-Tag microformat
> Parser/Indexer/Querier (microformats-reltag)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Http / Https Protocol
> Plug-in (protocol-httpclient)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     JavaScript Parser
> (parse-js)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Tika Parser Plug-in
> (parse-tika)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Top Level Domain
> Plugin (tld)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Regex URL Filter
> Framework (lib-regex-filter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Regex URL Normalizer
> (urlnormalizer-regex)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Link Analysis Scoring
> Plug-in (scoring-link)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     OPIC Scoring Plug-in
> (scoring-opic)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     More Indexing Filter
> (index-more)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Http Protocol Plug-in
> (protocol-http)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Creative Commons
> Plugins (creativecommons)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Registered
> Extension-Points:
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Parse Filter
> (org.apache.nutch.parse.ParseFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch Index Cleaning
> Filter (org.apache.nutch.indexer.IndexCleaningFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch Content Parser
> (org.apache.nutch.parse.Parser)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch URL Filter (
> org.apache.nutch.net.URLFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch Scoring
> (org.apache.nutch.scoring.ScoringFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch URL Normalizer (
> org.apache.nutch.net.URLNormalizer)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch Protocol
> (org.apache.nutch.protocol.Protocol)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch Index Writer
> (org.apache.nutch.indexer.IndexWriter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository:     Nutch Indexing Filter
> (org.apache.nutch.indexer.IndexingFilter)
> 16/01/20 02:45:29 INFO Configuration.deprecation:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
> 16/01/20 02:45:29 INFO Configuration.deprecation:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> 16/01/20 02:45:29 INFO Configuration.deprecation:
> mapred.compress.map.output is deprecated. Instead, use
> mapreduce.map.output.compress
> 16/01/20 02:45:29 INFO Configuration.deprecation: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces
> 16/01/20 02:45:29 INFO zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x60a2630a connecting to ZooKeeper
> ensemble=localhost:2181
> 16/01/20 02:45:29 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 16/01/20 02:45:29 INFO zookeeper.ZooKeeper: Client environment:host.name
> =cism479
> 16/01/20 02:45:29 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.8.0_65
> 16/01/20 02:45:29 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 16/01/20 02:45:29 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/jdk1.8.0_65/jre
> 16/01/20 02:45:35 INFO zookeeper.ClientCnxn: EventThread shut down
> 16/01/20 02:45:35 INFO mapreduce.JobSubmitter: number of splits:2
> 16/01/20 02:45:36 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1453210838763_0011
> 16/01/20 02:45:36 INFO impl.YarnClientImpl: Submitted application
> application_1453210838763_0011
> 16/01/20 02:45:36 INFO mapreduce.Job: The url to track the job:
> http://cism479:8088/proxy/application_1453210838763_0011/
> 16/01/20 02:45:36 INFO mapreduce.Job: Running job: job_1453210838763_0011
> 16/01/20 02:45:48 INFO mapreduce.Job: Job job_1453210838763_0011 running
> in uber mode : false
> 16/01/20 02:45:48 INFO mapreduce.Job:  map 0% reduce 0%
> 16/01/20 02:47:31 INFO mapreduce.Job:  map 33% reduce 0%
> 16/01/20 02:47:47 INFO mapreduce.Job:  map 50% reduce 0%
> 16/01/20 02:48:08 INFO mapreduce.Job:  map 83% reduce 0%
> 16/01/20 02:48:16 INFO mapreduce.Job:  map 100% reduce 0%
> 16/01/20 02:48:31 INFO mapreduce.Job:  map 100% reduce 31%
> 16/01/20 02:48:34 INFO mapreduce.Job:  map 100% reduce 33%
> 16/01/20 02:50:30 INFO mapreduce.Job:  map 100% reduce 34%
> 16/01/20 03:01:18 INFO mapreduce.Job:  map 100% reduce 35%
> 16/01/20 03:11:58 INFO mapreduce.Job:  map 100% reduce 36%
> 16/01/20 03:22:50 INFO mapreduce.Job:  map 100% reduce 37%
> 16/01/20 03:24:22 INFO mapreduce.Job:  map 100% reduce 50%
> 16/01/20 03:24:35 INFO mapreduce.Job:  map 100% reduce 82%
> 16/01/20 03:24:38 INFO mapreduce.Job:  map 100% reduce 83%
> 16/01/20 03:26:33 INFO mapreduce.Job:  map 100% reduce 84%
> 16/01/20 03:37:35 INFO mapreduce.Job:  map 100% reduce 85%
> 16/01/20 03:39:38 INFO mapreduce.Job: Task Id :
> attempt_1453210838763_0011_r_000001_0, Status : FAILED
> *Error: java.lang.IllegalArgumentException: Row length 41221 is > 32767*
>     at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:506)
>     at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:487)
>     at org.apache.hadoop.hbase.client.Get.<init>(Get.java:89)
>     at org.apache.gora.hbase.store.HBaseStore.get(HBaseStore.java:208)
>     at org.apache.gora.hbase.store.HBaseStore.get(HBaseStore.java:79)
>     at org.apache.gora.store.impl.DataStoreBase.get(DataStoreBase.java:156)
>     at org.apache.gora.store.impl.DataStoreBase.get(DataStoreBase.java:56)
>     at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:114)
>     at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:42)
>     at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
>     at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
> 16/01/20 03:39:39 INFO mapreduce.Job:  map 100% reduce 50%
> 16/01/20 03:39:52 INFO mapreduce.Job:  map 100% reduce 82%
> 16/01/20 03:39:55 INFO mapreduce.Job:  map 100% reduce 83%
> 16/01/20 03:41:56 INFO mapreduce.Job:  map 100% reduce 84%
> 16/01/20 03:53:39 INFO mapreduce.Job:  map 100% reduce 85%
> 16/01/20 03:55:49 INFO mapreduce.Job: Task Id :
> attempt_1453210838763_0011_r_000001_1, Status : FAILED
> *Error: java.lang.IllegalArgumentException: Row length 41221 is > 32767*
>     at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:506)
>     at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:487)
>     at org.apache.hadoop.hbase.client.Get.<init>(Get.java:89)
>     at org.apache.gora.hbase.store.HBaseStore.get(HBaseStore.java:208)
>     at org.apache.gora.hbase.store.HBaseStore.get(HBaseStore.java:79)
>     at org.apache.gora.store.impl.DataStoreBase.get(DataStoreBase.java:156)
>     at org.apache.gora.store.impl.DataStoreBase.get(DataStoreBase.java:56)
>     at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:114)
>     at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:42)
>     at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
>     at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
> 16/01/20 03:55:50 INFO mapreduce.Job:  map 100% reduce 50%
> 16/01/20 03:56:01 INFO mapreduce.Job:  map 100% reduce 83%
> 16/01/20 03:58:02 INFO mapreduce.Job:  map 100% reduce 84%
> 16/01/20 04:10:09 INFO mapreduce.Job:  map 100% reduce 85%
> 16/01/20 04:12:33 INFO mapreduce.Job: Task Id :
> attempt_1453210838763_0011_r_000001_2, Status : FAILED
> *Error: java.lang.IllegalArgumentException: Row length 41221 is > 32767*
>     at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:506)
>     at org.apache.hadoop.hbase.client.Mutation.checkRow(Mutation.java:487)
>     at org.apache.hadoop.hbase.client.Get.<init>(Get.java:89)
>     at org.apache.gora.hbase.store.HBaseStore.get(HBaseStore.java:208)
>     at org.apache.gora.hbase.store.HBaseStore.get(HBaseStore.java:79)
>     at org.apache.gora.store.impl.DataStoreBase.get(DataStoreBase.java:156)
>     at org.apache.gora.store.impl.DataStoreBase.get(DataStoreBase.java:56)
>     at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:114)
>     at
> org.apache.nutch.crawl.DbUpdateReducer.reduce(DbUpdateReducer.java:42)
>     at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
>     at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
>     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>
> 16/01/20 04:12:34 INFO mapreduce.Job:  map 100% reduce 50%
> 16/01/20 04:12:45 INFO mapreduce.Job:  map 100% reduce 82%
> 16/01/20 04:12:48 INFO mapreduce.Job:  map 100% reduce 83%
> 16/01/20 04:14:46 INFO mapreduce.Job:  map 100% reduce 84%
> 16/01/20 04:26:53 INFO mapreduce.Job:  map 100% reduce 85%
> 16/01/20 04:29:09 INFO mapreduce.Job:  map 100% reduce 100%
> 16/01/20 04:29:10 INFO mapreduce.Job: Job job_1453210838763_0011 failed
> with state FAILED due to: Task failed task_1453210838763_0011_r_000001
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
> 16/01/20 04:29:11 INFO mapreduce.Job: Counters: 50
>     File System Counters
>         FILE: Number of bytes read=38378343
>         FILE: Number of bytes written=115957636
>         FILE: Number of read operations=0
>         FILE: Number of large read operations=0
>         FILE: Number of write operations=0
>         HDFS: Number of bytes read=2382
>         HDFS: Number of bytes written=0
>         HDFS: Number of read operations=2
>         HDFS: Number of large read operations=0
>         HDFS: Number of write operations=0
>     Job Counters
>         Failed reduce tasks=4
>         Launched map tasks=2
>         Launched reduce tasks=5
>         Data-local map tasks=2
>         Total time spent by all maps in occupied slots (ms)=789909
>         Total time spent by all reduces in occupied slots (ms)=30215090
>         Total time spent by all map tasks (ms)=263303
>         Total time spent by all reduce tasks (ms)=6043018
>         Total vcore-seconds taken by all map tasks=263303
>         Total vcore-seconds taken by all reduce tasks=6043018
>         Total megabyte-seconds taken by all map tasks=808866816
>         Total megabyte-seconds taken by all reduce tasks=30940252160
>     Map-Reduce Framework
>         Map input records=49929
>         Map output records=1777904
>         Map output bytes=382773368
>         Map output materialized bytes=77228942
>         Input split bytes=2382
>         Combine input records=0
>         Combine output records=0
>         Reduce input groups=754170
>         Reduce shuffle bytes=38318183
>         Reduce input records=881156
>         Reduce output records=754170
>         Spilled Records=2659060
>         Shuffled Maps =2
>         Failed Shuffles=0
>         Merged Map outputs=2
>         GC time elapsed (ms)=17993
>         CPU time spent (ms)=819690
>         Physical memory (bytes) snapshot=4080136192
>         Virtual memory (bytes) snapshot=15234293760
>         Total committed heap usage (bytes)=4149739520
>     Shuffle Errors
>         BAD_ID=0
>         CONNECTION=0
>         IO_ERROR=0
>         WRONG_LENGTH=0
>         WRONG_MAP=0
>         WRONG_REDUCE=0
>     File Input Format Counters
>         Bytes Read=0
>     File Output Format Counters
>         Bytes Written=0
> Exception in thread "main" java.lang.RuntimeException: job failed:
> name=[1]update-table, jobid=job_1453210838763_0011
>     at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
>     at org.apache.nutch.crawl.DbUpdaterJob.run(DbUpdaterJob.java:111)
>     at
> org.apache.nutch.crawl.DbUpdaterJob.updateTable(DbUpdaterJob.java:140)
>     at org.apache.nutch.crawl.DbUpdaterJob.run(DbUpdaterJob.java:174)
>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>     at org.apache.nutch.crawl.DbUpdaterJob.main(DbUpdaterJob.java:178)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:497)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Error running:
> /usr/share/searchEngine/nutch-branch-2.3.1/runtime/deploy/bin/nutch
> updatedb -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D
> mapred.reduce.tasks.speculative.execution=false -D
> mapred.map.tasks.speculative.execution=false -D
> mapred.compress.map.output=true 1453230757-13191 -crawlId 1
> Failed with exit value 1.
> ****************************************************LOG
> END************************************************************************************************
>
> Please advise.
>
> --
>
> ------------------------------
>
> *Cyber Infrastructure (P) Limited, [CIS] **(CMMI Level 3 Certified)*
>
> Central India's largest Technology company.
>
> *Ensuring the success of our clients and partners through our highly
> optimized Technology solutions.*
>
> www.cisin.com | +Cisin <https://plus.google.com/+Cisin/> | Linkedin <
> https://www.linkedin.com/company/cyber-infrastructure-private-limited> |
> Offices: *Indore, India.* *Singapore. Silicon Valley, USA*.
>
> DISCLAIMER:  INFORMATION PRIVACY is important for us, If you are not the
> intended recipient, you should delete this message and are notified that
> any disclosure, copying or distribution of this message, or taking any
> action based on it, is strictly prohibited by Law.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message