pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4314) BigData_5 hang on some machine
Date Sat, 08 Nov 2014 02:23:33 GMT

     [ https://issues.apache.org/jira/browse/PIG-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-4314:
----------------------------
    Attachment: PIG-4314-1.patch

> BigData_5 hang on some machine
> ------------------------------
>
>                 Key: PIG-4314
>                 URL: https://issues.apache.org/jira/browse/PIG-4314
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.14.0
>
>         Attachments: PIG-4314-1.patch
>
>
> Hang for both tez and mr. Here is the log:
> {code}
> 2014-11-07 19:49:45,799 INFO [TezChild] relationalOperators.POPartialAgg: Aggregated
10000 raw tuples. Processed tuples before aggregation = 0, after aggregation = 676
> 2014-11-07 19:49:45,799 INFO [TezChild] relationalOperators.POPartialAgg: After reduction,
processed map: 676; raw map: 0
> 2014-11-07 19:49:45,800 INFO [TezChild] relationalOperators.POPartialAgg: Observed reduction
factor: from 10000 to 676 => 14.
> 2014-11-07 19:49:45,992 INFO [TezChild] relationalOperators.POPartialAgg: Getting mem
limits; considering 1 POPArtialAgg objects. with memory percentage 0.2
> 2014-11-07 19:49:46,290 INFO [TezChild] relationalOperators.POPartialAgg: Estimated total
tuples to buffer, based on 10000 tuples that took up 2364864 bytes: 1052574
> 2014-11-07 19:49:46,290 INFO [TezChild] relationalOperators.POPartialAgg: Setting thresholds.
Primary: 977390. Secondary: 75184
> 2014-11-07 19:49:50,641 INFO [Service Thread] util.SpillableMemoryManager: first memory
handler call - Collection threshold init = 216530944(211456K) used = 724849000(707860K) committed
= 927989760(906240K) max = 1146093568(1119232K)
> 2014-11-07 19:49:50,644 INFO [Service Thread] relationalOperators.POPartialAgg: Spill
triggered by SpillableMemoryManager
> 2014-11-07 19:49:50,993 INFO [TezChild] relationalOperators.POPartialAgg: Aggregated
346545 raw tuples. Processed tuples before aggregation = 676, after aggregation = 1352
> 2014-11-07 19:49:50,998 INFO [TezChild] relationalOperators.POPartialAgg: Aggregated
1352 processed tuples to 676 tuples
> 2014-11-07 19:49:50,998 INFO [TezChild] relationalOperators.POPartialAgg: Avoided emitting
records during spill memory call.
> 2014-11-07 19:49:51,046 INFO [Service Thread] relationalOperators.POPartialAgg: Finished
spill for SpillableMemoryManager call
> 2014-11-07 19:49:51,851 INFO [Service Thread] util.SpillableMemoryManager: Spilled an
estimate of 81943920 bytes from 1 objects. init = 216530944(211456K) used = 724849000(707860K)
committed = 927989760(906240K) max = 1146093568(1119232K)
> 2014-11-07 19:49:59,644 INFO [Service Thread] relationalOperators.POPartialAgg: Spill
triggered by SpillableMemoryManager
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message