hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-403) infoVolume, infoBucket, infoKey shows wrong "createdOn" / "modifiedOn" metadata for volume, bucket, key
Date Fri, 07 Sep 2018 23:45:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607787#comment-16607787
] 

Xiaoyu Yao commented on HDDS-403:
---------------------------------

{quote}The issue is that while creating the entities we use UTC but when retrieving entities
we try to parse it into GMT and hence the garbage values.
{quote}
We persisted the timestamp in the format of milliseconds since 1970-01-01T00:00:00Z using
Time.now(). This will be the same no matter UTC or GMT. 

When interpret the persisted point in time values in human readable format, there will be
subtle difference: At second level, they are the same. UTC is more precise at sub-second level
where it uses a dedicated integer to saves the nano parts along with a long for the second
part. (More details can be found in JLS or documents of java.time.Instant class from JDK)

 

 

> infoVolume, infoBucket, infoKey shows wrong "createdOn" / "modifiedOn" metadata for volume,
bucket, key
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-403
>                 URL: https://issues.apache.org/jira/browse/HDDS-403
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Manager
>    Affects Versions: 0.2.1
>            Reporter: Nilotpal Nandi
>            Assignee: Dinesh Chitlangia
>            Priority: Blocker
>              Labels: newbie
>             Fix For: 0.2.1
>
>         Attachments: HDDS-403.001.patch, HDDS-403.002.patch, HDDS-403.003.patch
>
>
> Example for Key shown below. Similar observations can be made for Bucket and Volume.
>  # ran putKey command for a file
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# ./ozone oz -putKey /test-vol1/test-bucket1/file1
-file /etc/passwd -v
> 2018-09-05 10:25:11,498 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> Volume Name : test-vol1
> Bucket Name : test-bucket1
> Key Name : file1
> File Hash : 8164cc3d5b05c44b73a6277661aa4645
> 2018-09-05 10:25:12,377 INFO conf.ConfUtils: raft.rpc.type = GRPC (default)
> 2018-09-05 10:25:12,390 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:12,402 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default)
> 2018-09-05 10:25:12,407 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max
= 100 (default)
> 2018-09-05 10:25:12,407 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3
(default)
> 2018-09-05 10:25:12,518 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:12,518 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:12,866 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default)
> 2018-09-05 10:25:13,644 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:13,644 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:13,645 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default)
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# ./ozone oz -getKey /test-vol1/test-bucket1/file1
-file getkey3
> 2018-09-05 10:25:22,020 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> 2018-09-05 10:25:22,778 INFO conf.ConfUtils: raft.rpc.type = GRPC (default)
> 2018-09-05 10:25:22,790 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:22,800 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default)
> 2018-09-05 10:25:22,804 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max
= 100 (default)
> 2018-09-05 10:25:22,805 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3
(default)
> 2018-09-05 10:25:22,890 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:22,890 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:23,250 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default)
> 2018-09-05 10:25:24,066 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:24,067 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:24,067 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default){noformat}
> 2. Ran infoKey on that key
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# ./ozone oz -infoKey /test-vol1/test-bucket1/file1
-v
> 2018-09-05 10:54:42,053 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> Volume Name : test-vol1
> Bucket Name : test-bucket1
> Key Name : file1
> {
>  "version" : 0,
>  "md5hash" : null,
>  "createdOn" : "Sat, 14 Dec +114522267 00:51:17 GMT",
>  "modifiedOn" : "Fri, 09 Jun +50648 04:30:12 GMT",
>  "size" : 4659,
>  "keyName" : "file1",
>  "keyLocations" : [ {
>  "containerID" : 16,
>  "localID" : 1536143112267,
>  "length" : 4659,
>  "offset" : 0
>  } ]
> }{noformat}
> "createdOn" and "modifiedOn" metadata are incorrect.
> Here is the current date:
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# date
> Wed Sep 5 10:54:52 UTC 2018{noformat}
> Also , the "md5hash" for the key is showing as null.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message