spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-12100) bug in spark/python/pyspark/rdd.py portable_hash()
Date Sat, 08 Oct 2016 11:28:20 GMT

     [ https://issues.apache.org/jira/browse/SPARK-12100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen resolved SPARK-12100.
-------------------------------
    Resolution: Duplicate

> bug in spark/python/pyspark/rdd.py portable_hash()
> --------------------------------------------------
>
>                 Key: SPARK-12100
>                 URL: https://issues.apache.org/jira/browse/SPARK-12100
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 1.5.1
>            Reporter: Andrew Davidson
>            Priority: Minor
>              Labels: hashing, pyspark
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> I am using spark-1.5.1-bin-hadoop2.6. I used spark-1.5.1-bin-hadoop2.6/ec2/spark-ec2
to create a cluster and configured spark-env to use python3. I get and exception 'Randomness
of hash of string should be disabled via PYTHONHASHSEED’. Is there any reason rdd.py should
not just set PYTHONHASHSEED ?
> Should I file a bug?
> Kind regards
> Andy
> details
> http://spark.apache.org/docs/latest/api/python/pyspark.html?highlight=subtract#pyspark.RDD.subtract
> Example from documentation does not work out of the box
> Subtract(other, numPartitions=None)
> Return each value in self that is not contained in other.
> >>> x = sc.parallelize([("a", 1), ("b", 4), ("b", 5), ("a", 3)])
> >>> y = sc.parallelize([("a", 3), ("c", None)])
> >>> sorted(x.subtract(y).collect())
> [('a', 1), ('b', 4), ('b', 5)]
> It raises 
>     if sys.version >= '3.3' and 'PYTHONHASHSEED' not in os.environ:
>         raise Exception("Randomness of hash of string should be disabled via PYTHONHASHSEED")
> The following script fixes the problem 
> Sudo printf "\n# set PYTHONHASHSEED so python3 will not generate Exception'Randomness
of hash of string should be disabled via PYTHONHASHSEED'\nexport PYTHONHASHSEED=123\n" >>
/root/spark/conf/spark-env.sh
> sudo pssh -i -h /root/spark-ec2/slaves cp /root/spark/conf/spark-env.sh  /root/spark/conf/spark-env.sh-`date
"+%Y-%m-%d:%H:%M"`
> Sudo for i in `cat slaves` ; do scp spark-env.sh root@$i:/root/spark/conf/spark-env.sh;
done
> This is how I am starting spark
> export PYSPARK_PYTHON=python3.4
> export PYSPARK_DRIVER_PYTHON=python3.4
> export IPYTHON_OPTS="notebook --no-browser --port=7000 --log-level=WARN"  
> $SPARK_ROOT/bin/pyspark \
>     --master $MASTER_URL \
>     --total-executor-cores $numCores \
>     --driver-memory 2G \
>     --executor-memory 2G \
>     $extraPkgs \
>     $*
> see email thread "possible bug spark/python/pyspark/rdd.py portable_hash()' on user@spark
for more info



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message