如何从数据框字典创建数据框?

Posted

技术标签:

【中文标题】如何从数据框字典创建数据框?【英文标题】:How to create a dataframe from a dictionary of dataframes? 【发布时间】:2019-05-27 05:59:40 【问题描述】:

问题是我想从递归嵌套的 JSON 文件创建一个 Dataframe,我想出了自己的方法。如果您有其他更简单的方法,请告诉我

我在一个 json 文件中记录了 24 小时的航班数据,并想使用它创建一个数据框来形成一天的数据框。我可以通过每小时创建一个对象然后使用 append 合并所有对象来做到这一点,但这似乎是一个乏味的过程,即使是 24 小时,如果我有 1000 条记录以这种方式合并的问题怎么办。因此我想出了这个代码 - 我尝试遍历 JSON 文件以获取每小时数据并一次创建两个数据帧,使用 concat 合并它们。我稍后将这两个数据帧存储在字典中,其中的键值由变量 z 指示。当我打印它时,输出一次包含 2 条记录并打印 12 次,因此使其成为 24 小时内的所有数据,但是当我将数据存储在 csv 文件中时,仅存储与最后两个小时相对应的数据.

    import pandas as pd
    z = 0
    result1 = 
    for i in range(0,24):
        if (i%2)==0:
            dataF = data["data"]["weather"][0]["hourly"][i]
            dataF2 = data["data"]["weather"][0]["hourly"][i+1]
            dataFrame = pd.DataFrame(dataF,index=[i+1])
            dataFrame2 = pd.DataFrame(dataF2,index=[i+2])
            frame = [dataFrame,dataFrame2]
            result = pd.concat(frame)
            #result.append(result)
            #print(result1)
            result1[z] = result
            z += 1

    print(result1)`

打印结果:

0:   windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC windspeedMiles  \
1            11         46             NW          8         8              7   
2            13         42             NW          6         7              8   

  DewPointF HeatIndexF cloudcover HeatIndexC ...   \
1        47         49        100         10 ...    
2        44         46        100          8 ...    

                         weatherDesc visibility weatherCode tempC tempF  \
1  'value': 'Patchy rain possible'         10         176    10    49   
2  'value': 'Patchy rain possible'         10         176     8    46   

  WindChillF WindChillC winddirDegree humidity time  
1         46          8           318       91    0  
2         42          6           317       92  100  

[2 rows x 25 columns], 1:   windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC windspeedMiles  \
3            14         38             NW          3         5              9   
4            16         33             NW          1         3             10   

  DewPointF HeatIndexF cloudcover HeatIndexC ...             weatherDesc  \
3        41         43        100          6 ...   'value': 'Overcast'   
4        38         40        100          4 ...   'value': 'Overcast'   

  visibility weatherCode tempC tempF WindChillF WindChillC winddirDegree  \
3         10         122     6    43         38          3           315   
4         10         122     4    40         33          1           314   

  humidity time  
3       92  200  
4       93  300  

[2 rows x 25 columns], 2:   windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC windspeedMiles  \
5            17         33             NW          1         3             10   
6            17         33             NW          0         3             11   

  DewPointF HeatIndexF cloudcover HeatIndexC ...             weatherDesc  \
5        38         40        100          4 ...   'value': 'Overcast'   
6        38         39        100          4 ...   'value': 'Overcast'   

  visibility weatherCode tempC tempF WindChillF WindChillC winddirDegree  \
5         10         122     4    40         33          1           314   
6         10         122     4    40         33          0           315   

  humidity time  
5       93  400  
6       93  500  

[2 rows x 25 columns], 3:   windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC windspeedMiles  \
7            18         32             NW          0         3             11   
8            18         33             NW          1         3             11   

  DewPointF HeatIndexF cloudcover HeatIndexC ...             weatherDesc  \
7        38         39        100          4 ...   'value': 'Overcast'   
8        38         40        100          4 ...   'value': 'Overcast'   

  visibility weatherCode tempC tempF WindChillF WindChillC winddirDegree  \
7         10         122     4    40         32          0           315   
8         10         122     5    41         33          1           317   

  humidity time  
7       93  600  
8       93  700  

[2 rows x 25 columns], 4:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
9             17         34             NW          1         4   
10            17         35             NW          2         4   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC ...   \
9              11        38         40        100          5 ...    
10             11        39         41        100          5 ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
9   'value': 'Overcast'         10         122     5    42         34   
10  'value': 'Overcast'         10         122     6    42         35   

   WindChillC winddirDegree humidity time  
9           1           318       92  800  
10          2           319       92  900  

[2 rows x 25 columns], 5:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
11            17         36             NW          2         4   
12            17         37             NW          3         4   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
11             10        38         42        100          6  ...    
12             10        38         43        100          6  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
11  'value': 'Overcast'         10         122     7    44         36   
12  'value': 'Overcast'         10         122     7    45         37   

   WindChillC winddirDegree humidity  time  
11          2           318       88  1000  
12          3           317       84  1100  

[2 rows x 25 columns], 6:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
13            17         38             NW          3         3   
14            17         38             NW          4         3   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
13             10        38         44        100          7  ...    
14             10        38         44        100          7  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
13  'value': 'Overcast'         10         122     8    47         38   
14  'value': 'Overcast'         10         122     9    47         38   

   WindChillC winddirDegree humidity  time  
13          3           317       81  1200  
14          4           317       79  1300  

[2 rows x 25 columns], 7:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
15            17         39             NW          4         3   
16            17         39             NW          4         3   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
15             10        38         44        100          7  ...    
16             11        37         45        100          7  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
15  'value': 'Overcast'         10         122     9    48         39   
16  'value': 'Overcast'         10         122    10    49         39   

   WindChillC winddirDegree humidity  time  
15          4           317       76  1400  
16          4           317       74  1500  

[2 rows x 25 columns], 8:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
17            17         39             NW          4         3   
18            17         39             NW          4         2   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
17             11        37         44        100          7  ...    
18             11        36         44        100          7  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
17  'value': 'Overcast'         10         122     9    48         39   
18  'value': 'Overcast'         10         122     8    46         39   

   WindChillC winddirDegree humidity  time  
17          4           318       74  1600  
18          4           318       73  1700  

[2 rows x 25 columns], 9:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
19            17         38             NW          3         2   
20            17         37             NW          3         2   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
19             11        36         44        100          7  ...    
20             11        35         43        100          6  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
19  'value': 'Overcast'         10         122     7    45         38   
20  'value': 'Overcast'         10         122     7    44         37   

   WindChillC winddirDegree humidity  time  
19          3           319       73  1800  
20          3           318       72  1900  

[2 rows x 25 columns], 10:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
21            17         37             NW          3         1   
22            16         36             NW          2         1   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
21             10        34         43        100          6  ...    
22             10        33         42        100          6  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
21  'value': 'Overcast'         10         122     6    43         37   
22  'value': 'Overcast'         10         122     6    42         36   

   WindChillC winddirDegree humidity  time  
21          3           317       71  2000  
22          2           316       71  2100  

[2 rows x 25 columns], 11:    windspeedKmph FeelsLikeF winddir16Point FeelsLikeC DewPointC  \
23            16         35             NW          2         1   
24            16         35             NW          2         1   

   windspeedMiles DewPointF HeatIndexF cloudcover HeatIndexC  ...   \
23             10        33         41         95          5  ...    
24             10        33         41         91          5  ...    

              weatherDesc visibility weatherCode tempC tempF WindChillF  \
23  'value': 'Overcast'         10         122     5    41         35   
24  'value': 'Overcast'         10         122     5    41         35   

   WindChillC winddirDegree humidity  time  
23          2           317       72  2200  
24          2           318       74  2300  

[2 rows x 25 columns]

【问题讨论】:

Pravin,您需要提供原始数据样本才能重现,尽管我在答案部分添加了一个借用的示例。 你可以检查一下 pandas read_json 函数是否可以帮助你解决这个pandas.pydata.org/pandas-docs/stable/generated/… 【参考方案1】:

只是为了你的例子:

如果您将整个 json 作为字典(或列表)加载,例如使用 json.load,你可以使用 json_normalize:

>>> import pandas as pd
>>> import json
>>> from pandas.io.json import json_normalize

数据:

>>> d = "response": "body": "contact": "email": "mr@abc.com", "mobile_number": "0123456789", "personal": "last_name": "Muster", "gender": "m", "first_name": "Max", "dob": "1985-12-23", "family_status": "single", "title": "Dr.", "customer": "verified": "true", "customer_id": "1234567", "token": "dsfgf", "version": "1.1"
>>> df = pd.io.json.json_normalize(d)
>>> df
  response.body.contact.email response.body.contact.mobile_number response.body.customer.customer_id       ...        response.body.personal.title response.token response.version
0                  mr@abc.com                          0123456789                            1234567       ...                                 Dr.          dsfgf              1.1

[1 rows x 12 columns]
>>> df.columns = df.columns.map(lambda x: x.split(".")[-1])
>>> df
        email mobile_number customer_id verified         dob family_status first_name gender last_name title  token version
0  mr@abc.com    0123456789     1234567     true  1985-12-23        single        Max      m    Muster   Dr.  dsfgf     1.1

看here

另一个post

【讨论】:

以上是关于如何从数据框字典创建数据框?的主要内容,如果未能解决你的问题,请参考以下文章

如何从嵌套字典创建多索引数据框?

pandas:从字典创建数据框

Python:如何从熊猫数据框创建字典? [复制]

如何从大字典中列出的每个类别的所有可能的值组合创建数据框

如何从Twitter Search API创建pandas数据框?

Scala和Spark,rdd从字典创建数据框