hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dinesh Chitlangia (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDDS-403) infoVolume, infoBucket, infoKey shows wrong "createdOn" / "modifiedOn" metadata for volume, bucket, key
Date Sat, 08 Sep 2018 00:54:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607831#comment-16607831
] 

Dinesh Chitlangia edited comment on HDDS-403 at 9/8/18 12:53 AM:
-----------------------------------------------------------------

Manual validation samples:

Volume:
{noformat}
hadoop@1987b5de4203:~$ ./bin/ozone oz -createVolume /xmen -user xiaoyu
2018-09-08 00:27:36 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2018-09-08 00:27:37 WARN ShellBasedUnixGroupsMapping:210 - unable to return groups for user
xiaoyu
PartialGroupNameException The user name 'xiaoyu' is not found. id: ‘xiaoyu’: no such user
id: ‘xiaoyu’: no such user

at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
at ....... (truncated stack trace)
2018-09-08 00:27:37 INFO RpcClient:206 - Creating Volume: xmen, with xiaoyu as owner and quota
set to 1152921504606846976 bytes.
hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /xmen
2018-09-08 00:27:47 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
{
"owner" : {
"name" : "xiaoyu"
},
"quota" : {
"unit" : "TB",
"size" : 1048576
},
"volumeName" : "xmen",
"createdOn" : "Sat, 08 Sep 2018 00:27:37 GMT",
"createdBy" : "xiaoyu"
}{noformat}
Bucket:
{noformat}
hadoop@1987b5de4203:~$ ./bin/ozone oz -createBucket /xmen/charles
2018-09-08 00:28:09 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2018-09-08 00:28:10 INFO RpcClient:335 - Creating Bucket: xmen/charles, with Versioning false
and Storage Type set to DISK
hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /xmen/charles
2018-09-08 00:28:19 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
{
"volumeName" : "xmen",
"bucketName" : "charles",
"createdOn" : "Sat, 08 Sep 2018 00:28:10 GMT",
"acls" : [ {
"type" : "USER",
"name" : "hadoop",
"rights" : "READ_WRITE"
}, {
"type" : "GROUP",
"name" : "users",
"rights" : "READ_WRITE"
} ],
"versioning" : "DISABLED",
"storageType" : "DISK"
}{noformat}
Key:
{noformat}
hadoop@1987b5de4203:~$ ./bin/ozone oz -putKey /xmen/charles/xav -file k1.orc
2018-09-08 00:28:52 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /xmen/charles/xav
2018-09-08 00:29:02 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
{
"version" : 0,
"md5hash" : null,
"createdOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"modifiedOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"size" : 0,
"keyName" : "xav",
"keyLocations" : [ ]
}
hadoop@1987b5de4203:~$ ./bin/ozone oz -listKey /xmen/charles
2018-09-08 00:29:13 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
[ {
"version" : 0,
"md5hash" : null,
"createdOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"modifiedOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"size" : 0,
"keyName" : "xav"
} ]
hadoop@1987b5de4203:~$



{noformat}
 


was (Author: dineshchitlangia):
Manual validation samples:

Volume:
{noformat}
hadoop@1987b5de4203:~$ ./bin/ozone oz -createVolume /xmen -user xiaoyu
2018-09-08 00:27:36 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2018-09-08 00:27:37 WARN ShellBasedUnixGroupsMapping:210 - unable to return groups for user
xiaoyu
PartialGroupNameException The user name 'xiaoyu' is not found. id: ‘xiaoyu’: no such user
id: ‘xiaoyu’: no such user

at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294)
at ....... (truncated stack trace)
2018-09-08 00:27:37 INFO RpcClient:206 - Creating Volume: xmen, with xiaoyu as owner and quota
set to 1152921504606846976 bytes.
hadoop@1987b5de4203:~$ ./bin/ozone oz -infoVolume /xmen
2018-09-08 00:27:47 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
{
"owner" : {
"name" : "xiaoyu"
},
"quota" : {
"unit" : "TB",
"size" : 1048576
},
"volumeName" : "xmen",
"createdOn" : "Sat, 08 Sep 2018 00:27:37 GMT",
"createdBy" : "xiaoyu"
}{noformat}
Bucket:
{noformat}
hadoop@1987b5de4203:~$ ./bin/ozone oz -createBucket /xmen/charles
2018-09-08 00:28:09 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2018-09-08 00:28:10 INFO RpcClient:335 - Creating Bucket: xmen/charles, with Versioning false
and Storage Type set to DISK
hadoop@1987b5de4203:~$ ./bin/ozone oz -infoBucket /xmen/charles
2018-09-08 00:28:19 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
{
"volumeName" : "xmen",
"bucketName" : "charles",
"createdOn" : "Sat, 08 Sep 2018 00:28:10 GMT",
"acls" : [ {
"type" : "USER",
"name" : "hadoop",
"rights" : "READ_WRITE"
}, {
"type" : "GROUP",
"name" : "users",
"rights" : "READ_WRITE"
} ],
"versioning" : "DISABLED",
"storageType" : "DISK"
}{noformat}
Key:
{noformat}
hadoop@1987b5de4203:~$ ./bin/ozone oz -putKey /xmen/charles/xav -file k1.orc
2018-09-08 00:28:52 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
hadoop@1987b5de4203:~$ ./bin/ozone oz -infoKey /xmen/charles/xav
2018-09-08 00:29:02 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
{
"version" : 0,
"md5hash" : null,
"createdOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"modifiedOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"size" : 0,
"keyName" : "xav",
"keyLocations" : [ ]
}
hadoop@1987b5de4203:~$ ./bin/ozone oz -listKey /xmen/charles
2018-09-08 00:29:13 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
[ {
"version" : 0,
"md5hash" : null,
"createdOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"modifiedOn" : "Sat, 08 Sep 2018 00:28:52 GMT",
"size" : 0,
"keyName" : "xav"
} ]
hadoop@1987b5de4203:~${noformat}

> infoVolume, infoBucket, infoKey shows wrong "createdOn" / "modifiedOn" metadata for volume,
bucket, key
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-403
>                 URL: https://issues.apache.org/jira/browse/HDDS-403
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Manager
>    Affects Versions: 0.2.1
>            Reporter: Nilotpal Nandi
>            Assignee: Dinesh Chitlangia
>            Priority: Blocker
>              Labels: newbie
>             Fix For: 0.2.1
>
>         Attachments: HDDS-403.001.patch, HDDS-403.002.patch, HDDS-403.003.patch, HDDS-403.004.patch
>
>
> Example for Key shown below. Similar observations can be made for Bucket and Volume.
>  # ran putKey command for a file
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# ./ozone oz -putKey /test-vol1/test-bucket1/file1
-file /etc/passwd -v
> 2018-09-05 10:25:11,498 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> Volume Name : test-vol1
> Bucket Name : test-bucket1
> Key Name : file1
> File Hash : 8164cc3d5b05c44b73a6277661aa4645
> 2018-09-05 10:25:12,377 INFO conf.ConfUtils: raft.rpc.type = GRPC (default)
> 2018-09-05 10:25:12,390 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:12,402 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default)
> 2018-09-05 10:25:12,407 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max
= 100 (default)
> 2018-09-05 10:25:12,407 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3
(default)
> 2018-09-05 10:25:12,518 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:12,518 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:12,866 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default)
> 2018-09-05 10:25:13,644 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:13,644 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:13,645 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default)
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# ./ozone oz -getKey /test-vol1/test-bucket1/file1
-file getkey3
> 2018-09-05 10:25:22,020 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> 2018-09-05 10:25:22,778 INFO conf.ConfUtils: raft.rpc.type = GRPC (default)
> 2018-09-05 10:25:22,790 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:22,800 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default)
> 2018-09-05 10:25:22,804 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max
= 100 (default)
> 2018-09-05 10:25:22,805 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3
(default)
> 2018-09-05 10:25:22,890 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:22,890 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:23,250 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default)
> 2018-09-05 10:25:24,066 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576)
(default)
> 2018-09-05 10:25:24,067 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom)
> 2018-09-05 10:25:24,067 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms
(default){noformat}
> 2. Ran infoKey on that key
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# ./ozone oz -infoKey /test-vol1/test-bucket1/file1
-v
> 2018-09-05 10:54:42,053 WARN util.NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
> Volume Name : test-vol1
> Bucket Name : test-bucket1
> Key Name : file1
> {
>  "version" : 0,
>  "md5hash" : null,
>  "createdOn" : "Sat, 14 Dec +114522267 00:51:17 GMT",
>  "modifiedOn" : "Fri, 09 Jun +50648 04:30:12 GMT",
>  "size" : 4659,
>  "keyName" : "file1",
>  "keyLocations" : [ {
>  "containerID" : 16,
>  "localID" : 1536143112267,
>  "length" : 4659,
>  "offset" : 0
>  } ]
> }{noformat}
> "createdOn" and "modifiedOn" metadata are incorrect.
> Here is the current date:
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000003 bin]# date
> Wed Sep 5 10:54:52 UTC 2018{noformat}
> Also , the "md5hash" for the key is showing as null.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message