pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Szita (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
Date Thu, 20 Dec 2018 13:57:00 GMT

    [ https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725861#comment-16725861
] 

Adam Szita edited comment on PIG-5371 at 12/20/18 1:56 PM:
-----------------------------------------------------------

Hi [~abstractdog],

Yeah sorry that's a typo, indeed -Dtestcase should be used.

The test doesn't hang on my side, it finishes successfully in 9 minutes.
 Do you see which test method completes on your side and which one doesn't?

In the past when I faced with the hanging issue was due to my Mac's HDD had over 90% utilisation
which some HDFS code in MiniCluster did not like


was (Author: szita):
Hi [~abstractdog],

Yeah sorry that's a typo, indeed -Dtestcase should be used.

The doesn't hang on my side, it finishes successfully in 9 minutes.
Do you see which test method completes on your side and which one doesn't?

In the past when I faced with the hanging issue was due to my Mac's HDD had over 90% utilisation
which some HDFS code in MiniCluster did not like

> Hdfs bytes written assertions fail in TestPigRunner
> ---------------------------------------------------
>
>                 Key: PIG-5371
>                 URL: https://issues.apache.org/jira/browse/PIG-5371
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Laszlo Bodor
>            Assignee: Laszlo Bodor
>            Priority: Major
>         Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached  [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns
the byte count not only for the result of pig store operator, but it includes the size of
the jar files as well. The problem is this could change very easily, so in my opinion the
best would be to remove these assertions from TestPigRunner as this is just causing intermittent
and/or persistent failures.
> The test class is for basic testing of PigRunner, and this is achieved well enough without
the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO  org.apache.hadoop.hdfs.StateChange
- BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943
for /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930, 127.0.0.1:54943]] INFO  org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace
- src: /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1959727442_1,
offset: 0, srvID: 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
duration(ns): 57162859
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message