spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [jira] [Created] (SPARK-23495) Creating a json file using a dataframe creates an issue
Date Fri, 23 Feb 2018 14:00:00 GMT
AIT OUFKIR created SPARK-23495:

             Summary: Creating a json file using a dataframe creates an issue
                 Key: SPARK-23495
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.1.0
            Reporter: AIT OUFKIR
             Fix For: 2.1.0

Issue happen when trying to create json file using a dataframe (see code below)

catis = ["CAT1","CAT2"]
constis = ["CONST1","CONST2","CONST3"]
datis = ["DAT1","DATE2","DATE3"]
dictis = \{'A':1, 'B':2}
dummis = ['dum1','dumm2','dumm3']
fifis = \{'fifi1':1, 'fifi2':2, 'fifi3':3}
khikhis = ['khikhi1','khikhi12','khikhi3','khikhi4']

metadata_dump = dict(cati=catis, consti=constis, dati=datis, dicti=dictis, khikhi=khikhis,
dummi=dummis, fifi=fifis)
md = sqlContext.createDataFrame([metadata_dump]).collect()
metadata = sqlContext.createDataFrame(md,['cati', 'consti', 'dati', 'dicti','khikhi', 'dummi',

metadata_path = "/mypath"

This gives the following Results :


Which is wrong


When I try switching the fifis dict and not putting it at the end of the dict metadata_dump
then I get the correct results :



This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message