hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George Huang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11912) Add a snapshot unit test with randomized file IO operations
Date Thu, 17 Aug 2017 17:43:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130850#comment-16130850
] 

George Huang edited comment on HDFS-11912 at 8/17/17 5:42 PM:
--------------------------------------------------------------

Test was having a 10 min timeout. However, it took more than 10 mins to create 5000 test files
in HDFS:

2017-08-17 03:12:54,076 [Thread-126] INFO  common.Storage (Storage.java:tryLock(847)) - Lock
on /testptch/hadoop/hadoop-hdf
...[truncated 9653305 chars]...
wed=true	ugi=jenkins (auth:SIMPLE)	ip=/127.0.0.1	cmd=create	src=/WITNESSDIR/1720/1719/1718/1717/1716/1715/1714/1713/file1720
dst=null	perm=jenkins:supergroup:rw-r--r--	proto=rpc
2017-08-17 03:22:52,582 [IPC Server handler 6 on 40751] INFO  hdfs.StateChange (FSDirWriteFileOp.java:logAllocatedBlock(787))
- BLOCK* allocate blk_1073745266_4442, replicas=127.0.0.1:43960 for /WITNESSDIR/1720/1719/1718/1717/1716/1715/1714/1713/file1720
2017-08-17 03:22:52,583 [DataXceiver for client DFSClient_NONMAPREDUCE_1460336802_1 at /127.0.0.1:53834
[Receiving block BP-233349655-172.17.0.2-1502939568997:blk_1073745266_4442]] INFO  datanode.DataNode
(DataXceiver.java:writeBlock(742)) - Receiving BP-233349655-172.17.0.2-1502939568997:blk_1073745266_4442
src: /127.0.0.1:53834 dest: /127.0.0.1:43960
2017-08-17 03:22:52,586 [PacketResponder: BP-233349655-172.17.0.2-1502939568997:blk_1073745266_4442,
type=LAST_IN_PIPELINE] INFO  DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1523))
- src: /127.0.0.1:53834, dest: /127.0.0.1:43960, bytes: 69, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1460336802_1,
offset: 0, srvID: ae9a2151-cb92-461b-8d73-cd9641184228, blockid: BP-233349655-172.17.0.2-1502939568997:blk_1073745266_4442,
duration(ns): 1413179
2017-08-17 03:22:52,586 [PacketResponder: BP-233349655-172.17.0.2-1502939568997:blk_1073745266_4442,
type=LAST_IN_PIPELINE] INFO  datanode.DataNode (BlockReceiver.java:run(1496)) - PacketResponder:
BP-233349655-172.17.0.2-1502939568997:blk_1073745266_4442, type=LAST_IN_PIPELINE terminating
2017-08-17 03:22:52,590 [IPC Server handler 4 on 40751] INFO  hdfs.StateChange (FSNamesystem.java:completeFile(2755))
- DIR* completeFile: /WITNESSDIR/1720/1719/1718/1717/1716/1715/1714/1713/file1720 is closed
by DFSClient_NONMAPREDUCE_1460336802_1
2017-08-17 03:22:52,600 [IPC Server handler 5 on 40751] INFO  FSNamesystem.audit (FSNamesystem.java:logAuditMessage(7512))
- allowed=true	ugi=jenkins (auth:SIMPLE)	ip=/127.0.0.1	cmd=getfileinfo	src=/WITNESSDIR/1720/1719/1718/1717/1716/1715/1714/1713/file1720
dst=null	perm=null	proto=rpc
2017-08-17 03:22:52,601 [Thread-173] INFO  snapshot.TestRandomOpsWithSnapshots (TestRandomOpsWithSnapshots.java:createFiles(634))
- createFiles, file:


was (Author: ghuangups):
Test was having a 10 min timeout. However, setup related operations took almost 10 mins and
hence left no time for test to finish. Test starts at around 03:12, but it reaches to actual
test at around 03:22:  :(

2017-08-17 03:12:47,798 [main] INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:<init>(469))
- starting cluster: numNameNodes=1, numDataNodes=3
Formatting using clusterid: testClusterID
:
:
2017-08-17 03:12:54,076 [Thread-126] INFO  common.Storage (Storage.java:tryLock(847)) - Lock
on /testptch/hadoop/hadoop-hdf
...[truncated 9653305 chars]...
wed=true	ugi=jenkins (auth:SIMPLE)	ip=/127.0.0.1	cmd=create	src=/WITNESSDIR/1720/1719/1718/1717/1716/1715/1714/1713/file1720
dst=null	perm=jenkins:supergroup:rw-r--r--	proto=rpc
2017-08-17 03:22:52,582 [IPC Server handler 6 on 40751] INFO  hdfs.StateChange 
:
:



> Add a snapshot unit test with randomized file IO operations
> -----------------------------------------------------------
>
>                 Key: HDFS-11912
>                 URL: https://issues.apache.org/jira/browse/HDFS-11912
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: hdfs
>            Reporter: George Huang
>            Assignee: George Huang
>            Priority: Minor
>              Labels: TestGap
>         Attachments: HDFS-11912.001.patch, HDFS-11912.002.patch, HDFS-11912.003.patch,
HDFS-11912.004.patch
>
>
> Adding a snapshot unit test with randomized file IO operations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message