Data Cleaning 4
Posted 阿难的机器学习计划
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Data Cleaning 4相关的知识,希望对你有一定的参考价值。
1. Read the data:
1.1 If the data is not in .csv file. We have to search for the special read method
all_survey = pandas.read_csv("schools/survey_all.txt", delimiter="\t", encoding=‘windows-1252‘) # read http://kunststube.net/encoding/ for the introduction of encoding.
1.2 Read a big set of data, So we are using for loop to read through the data.
for f in data_files:
file = pd.read_csv("schools/{0}".format(f)) #When it related to a variable in the " ",we can not directly use the variable name in the string.
f = f.replace(".csv","")
data[f] = file
1.3 Combine some dataframe into one by using concat() function.
survey = pd.concat([all_survey,d75_survey],axis = 0) #
2. Cleaning up the data:
In the combined dataframe, it is inavoidable to have lots of ‘NaN‘ inside. So we need to deal with these "NaN"
2.1 We need to figure out which column are relevant. And extract them from the original Dataframe.
2.2 Some of the column name may have different column name but shows the same content. We need to change them into one.
2.3 To unify the string, we can add,minus, change, numeric the column names
3. Filting the data:
3.1 We can use findall() and split function to extract certain string we need from the whole string.
def extract_lat(data):
lat_lon = re.findall("\(.+,.+\)",data)
lat = re.split(",",lat_lon[0])
final_lat = lat[0].replace("(","")
return final_lat
data["hs_directory"]["lat"] = data["hs_directory"]["Location 1"].apply(extract_lat) #loop through each row of the DataFrame in certain column to call the function.
3.2 Find the relevant dataset from each column. And store them into another Dataframe.
4.
以上是关于Data Cleaning 4的主要内容,如果未能解决你的问题,请参考以下文章