hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-451) PutKey failed due to error "Rejecting write chunk request. Chunk overwrite without explicit request"
Date Fri, 14 Sep 2018 22:53:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615481#comment-16615481
] 

Tsz Wo Nicholas Sze commented on HDDS-451:
------------------------------------------

> ...  commit chunk call is happening twice It affirms that there two raft log entries
for the same client request.

The retry cache timeout is 60s.  The RaftClient may keep retrying even after 60s so that the
retry is considered as a new request in the server side.
{code}
2018-09-13 10:29:58,081 INFO org.apache.ratis.conf.ConfUtils: raft.server.retrycache.expirytime
= 60 s (default)
{code}
Increase it should also help.  The client retry policy and the retry cache timeout should
match.

> PutKey failed due to error "Rejecting write chunk request. Chunk overwrite without explicit
request"
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-451
>                 URL: https://issues.apache.org/jira/browse/HDDS-451
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Client
>    Affects Versions: 0.2.1
>            Reporter: Nilotpal Nandi
>            Assignee: Shashikant Banerjee
>            Priority: Blocker
>         Attachments: all-node-ozone-logs-1536841590.tar.gz
>
>
> steps taken :
> ------------------
>  # Ran Put Key command to write 50GB data. Put Key client operation failed after 17 mins.
> error seen  ozone.log :
> ------------------------------------
>  
> {code}
> 2018-09-13 12:11:53,734 [ForkJoinPool.commonPool-worker-20] DEBUG (ChunkManagerImpl.java:85)
- writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1
chunk stage:COMMIT_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1
tmp chunk file
> 2018-09-13 12:11:56,576 [pool-3-thread-60] DEBUG (ChunkManagerImpl.java:85) - writing
chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
chunk stage:WRITE_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
tmp chunk file
> 2018-09-13 12:11:56,739 [ForkJoinPool.commonPool-worker-20] DEBUG (ChunkManagerImpl.java:85)
- writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
chunk stage:COMMIT_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
tmp chunk file
> 2018-09-13 12:12:21,410 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:148)
- Executing cycle Number : 206
> 2018-09-13 12:12:51,411 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:148)
- Executing cycle Number : 207
> 2018-09-13 12:12:53,525 [BlockDeletingService#1] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79)
- Stop looking for next container, there is no pending deletion block contained in remaining
containers.
> 2018-09-13 12:12:55,048 [Datanode ReportManager Thread - 1] DEBUG (ContainerSet.java:191)
- Starting container report iteration.
> 2018-09-13 12:13:02,626 [pool-3-thread-1] ERROR (ChunkUtils.java:244) - Rejecting write
chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
offset=0, len=16777216}
> 2018-09-13 12:13:03,035 [pool-3-thread-1] INFO (ContainerUtils.java:149) - Operation:
WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting write chunk
request. OverWrite flag required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED
> 2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] ERROR (ChunkUtils.java:244)
- Rejecting write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
offset=0, len=16777216}
> 2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] INFO (ContainerUtils.java:149)
- Operation: WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting
write chunk request. OverWrite flag required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED
>  
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message