Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: 
 <CACmJb3yJSFXfZ3L9084ucpLs-bccCD6XaF2o=TT6_synUz_FUA@mail.gmail.com>
References: 
 <CAH2nEUgsSeoJTJkD4T8z=D18ECnRgQ3Qvz861gU5+NPSsNgT=A@mail.gmail.com>
 <CACmJb3zHG8ghn+bSgcZ+=jV3tqvxkbPfSPz3guy_c_8gqe1GVQ@mail.gmail.com>
 <CAH2nEUjHTxY5mZQF7nizp_fv1=ju+jEu9QnKr3zgPQiyoz4WSQ@mail.gmail.com>
 <CACmJb3yJSFXfZ3L9084ucpLs-bccCD6XaF2o=TT6_synUz_FUA@mail.gmail.com>
From: Samuel <samuelrivas@gmail.com>
Date: Fri, 11 Dec 2015 15:43:23 +0100
Message-ID: 
 <CAH2nEUg=jVrjVa50GpYebshBKTvABMVHxfhkNmQA_EzW==damA@mail.gmail.com>
Subject: Re: Dependency on absolute path /bin/ls in Shell.java
To: Namikaze Minato <lloydsensei@gmail.com>
Cc: user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

I am not using hadoop-util directly, it is Spark code what uses it
(i,e. not directly under my control).

Regarding ls, for my particular use case it is fine if you use "ls"
instead of "/bin/ls".

However, I do agree that using ls to fetch file permissions is
incorrect, so a better solution (in terms of code quality) would be
not to use ls at all.


On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com> wrote:
> So what you ultimately need is a piece of java code listing the rwx
> permissions for user, group and others that is not using ls
> internally, is that correct?
> If "RawLocalFileSystem" is not HDFS, do you really need to use
> hadoop-util for that?
> Can you tell us more about your use case?
>
> Regards,
> LLoyd
>
>
>
>
> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>>> Using ls to figure out permissions is a bad design anyway, so I would
>>> not be surprised if this hardcode was reported as a bug.
>>
>> Of course, I have no idea why it was implemented like this. I assume
>> it was written at some point in time where Java didn't provide the
>> needed APIS (?)
>>
>> Implementing the permission check without relying in ls at all is also
>> a solution for the problem I have :)
>>
>>> LLoyd
>>>
>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I am experiencing some crashes when using spark over local files (mainly for
>>> > testing). Some operations fail with
>>> >
>>> > java.lang.RuntimeException: Error while running command to get file
>>> > permissions : java.io.IOException: Cannot run program "/bin/ls": error=2, No
>>> > such file or directory
>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>> >         at
>>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489)
>>> >         at
>>> > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292)
>>> >
>>> > etcetera...
>>> >
>>> > Which seems to be related to Shell.java in org.apache.hadoop-util, that uses
>>> > ls -ld to figure out file permissions (that is in
>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead of just
>>> > calling ls, Shell .java calls /bin/ls, which is usually available, but in
>>> > certain circumstances might not. Regardless of the reasons not to have ls in
>>> > /bin, hardcoding the directory bans users from using the standard mechanisms
>>> > to decide which binaries to run in their systems (in this case, $PATH), so I
>>> > wonder if there is a particular reason why that path has been hardcoded to
>>> > an absolute path instead to something resolvable using$PATH.
>>> >
>>> > Or in other words, is this a bug or a feature?
>>> >
>>> > Best
>>> >
>>> > --
>>> > Samuel
>>
>>
>>
>>
>> --
>> Samuel


-- 
Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org