将Dask包的Pandas DataFrame转换为单个Dask DataFrame

Question

问题摘要

短版

我如何从Dask包的Pandas DataFrame转到单个Dask DataFrame？

Long Version

我有许多dask.dataframe的各种read函数（例如dd.read_csv或dd.read_parquet）无法读取的文件。我确实有自己的函数，可以将它们读取为Pandas DataFrames（函数一次只能在一个文件上使用，类似于pd.read_csv）。我想将所有这些单个Pandas DataFrames放在一个大的Dask DataFrame中。

最小工作示例

[以下是一些示例CSV数据（我的数据实际上不是CSV格式，但是为了便于示例，在此使用它）。要创建一个最低限度的工作示例，可以将其另存为CSV并制作一些副本，然后使用下面的代码

"gender","race/ethnicity","parental level of education","lunch","test preparation course","math score","reading score","writing score"
"female","group B","bachelor's degree","standard","none","72","72","74"
"female","group C","some college","standard","completed","69","90","88"
"female","group B","master's degree","standard","none","90","95","93"
"male","group A","associate's degree","free/reduced","none","47","57","44"
"male","group C","some college","standard","none","76","78","75"

from glob import glob
import pandas as pd
import dask.bag as db

files = glob('/path/to/your/csvs/*.csv')
bag = db.from_sequence(files).map(pd.read_csv)

到目前为止我尝试过的事情

import pandas as pd
import dask.bag as db
import dask.dataframe as dd

# Create a Dask bag of pandas dataframes
bag = db.from_sequence(list_of_files).map(my_reader_function)

df = bag.map(lambda x: x.to_records()).to_dataframe() # this doesn't work
df = bag.map(lambda x: x.to_dict(orient = <any option>)).to_dataframe() # neither does this

# This gets me really close. It's a bag of Dask DataFrames. 
# But I can't figure out how to concatenate them together
df = bag.map(dd.from_pandas, npartitions = 1)

df = dd.from_delayed(bag) # returns an error