pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laszlo Bodor (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
Date Fri, 23 Nov 2018 13:54:00 GMT

     [ https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Laszlo Bodor updated PIG-5371:
------------------------------
    Description: 
Attached  [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns the byte
count not only for the result of pig store operator, but it includes the jar files as well.
The problem is this could change very easily, so in my opinion the best would be to remove
these assertions from TestPigRunner as this is just causing intermittent and/or persistent
failures.
The test class is for basic testing of PigRunner, and this is achieved well enough without
the asserts.

{code}
2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO  org.apache.hadoop.hdfs.StateChange
- BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943
for /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
...
2018-11-23 10:14:52,735 [PacketResponder: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930, 127.0.0.1:54943]] INFO  org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace
- src: /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1959727442_1,
offset: 0, srvID: 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
duration(ns): 57162859
{code}

  was:
Attached  [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns the byte
count not only for the result of pig store operator, but it includes the jar files as well.

{code}
2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO  org.apache.hadoop.hdfs.StateChange
- BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943
for /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
...
2018-11-23 10:14:52,735 [PacketResponder: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930, 127.0.0.1:54943]] INFO  org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace
- src: /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1959727442_1,
offset: 0, srvID: 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
duration(ns): 57162859
{code}


> Hdfs bytes written assertions fail in TestPigRunner
> ---------------------------------------------------
>
>                 Key: PIG-5371
>                 URL: https://issues.apache.org/jira/browse/PIG-5371
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Laszlo Bodor
>            Assignee: Laszlo Bodor
>            Priority: Major
>         Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached  [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns
the byte count not only for the result of pig store operator, but it includes the jar files
as well. The problem is this could change very easily, so in my opinion the best would be
to remove these assertions from TestPigRunner as this is just causing intermittent and/or
persistent failures.
> The test class is for basic testing of PigRunner, and this is achieved well enough without
the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO  org.apache.hadoop.hdfs.StateChange
- BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943
for /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930, 127.0.0.1:54943]] INFO  org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace
- src: /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1959727442_1,
offset: 0, srvID: 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid: BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
duration(ns): 57162859
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message