利用python 数据分析入门,详细教程,教小白快速入门

Posted wangshicheng

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了利用python 数据分析入门,详细教程,教小白快速入门相关的知识,希望对你有一定的参考价值。

 

  这是一篇的数据的分析的典型案列,本人也是经历一次从无到有的过程,倍感珍惜,所以将其详细的记录下来,用来帮助后来者快速入门,,希望你能看到最后!

  需求:对obo文件进行解析,输出为json字典格式

  数据的格式如下:

  

  

  我们设定 一个trem or  typedef为一条标签,一行为一条记录或者是键值对,以此为标准!

  下面我们来对数据进行分析

  数据集中一共包含两种标签[trem] and [typedef]两种标签,每个标签下边有多个键值对,和唯一的标识符id,每行记录以“/n”结尾,且每条标签下下有多个相同的键值对,for examble: is_a,synonym...

 

  算法设计

  1. 数据集中含有【trem】和【typedef】两种标签,因此,我们将数据分成两个数据集分别来进行处理。

  2.循环遍历数据集,将键值对的键去除,并且对每一个键进行计数,并且进行去重操作

   (我刚开始的想法是根据id的数量于其他的键的数量进行比较,找出每个标签下存在重复的键值对,进而确定每个标签下存在重复的键值对 :is_a,有点想多了,呵呵~)

  3.由于发现每条标签下的记录的顺序都是一定的,id永远排在前面,用字典的形式存储是顺序是乱的,看上去很不舒服,所以我们相办法将他存在list里面,最大限度的还原了原有数据。

  4. 处理相同键的键值对,字典中不允许存在一键多值的情况,我们将他存到一个list里面,也就相当于大list里面套小list

  5.对数据集进行遍历,

    (1)将取出来的键值对的键值存储起来

    (2)以“【”作为我们的结束,将键值对的值存储到相对应的键下面,也就是一条标签

    (3)将我们所取得值存储到汇总在一起,并且对声明的字典和list进行初始化,方便进行下一次的循环

    (4)进行到这里,我们处理仅仅只是处理完了一个标签,还需要一个总的list,将所有的标签都存储进去

    (这里的算法还是不完善的,我希望看到这篇博客的人可以提出宝贵的建议)

代码设计以及踩过的坑:

1.打印出所有的键

附引用代码:

复制代码
 
\'\'\'
打印出所有的键
\'\'\'
with open(\'go.obo\',\'r\',encoding="utf-8") as f: #打开文件

for line in f.readlines(): #对数据进行每一行的循环
list = [] ## 空列表
lable = line.split(":")[0] #读取列表名,
print(lable)
list.append(lable) ## 使用 append() 向list中添加元素
# print(list)

#print(lable)

# lst2 = list(set(lst1))
# print(lst2)
print(list)
复制代码
 

 

2.但是在做上一步的时候,出现了一个问题,那就是没有区分局部变量和全局变量,问题发现的思路,先观察list输出的值,发现只有最后一个值,这时候就要考虑值是否被覆盖,找到问题,于是把list升级为全局变量

附引用代码:

复制代码

with open(\'go.obo\',\'r\',encoding="utf-8") as f: #打开文件
# dict = {}
list = [] ## 空列表

for line in f.readlines(): #对数据进行每一行的循环
total = []
lable = line.split(":")[0] #读取列表名,正确来说读取完列表名之后,还要进行去重的处理
# print(lable)
# list.append(lable) ## 使用 append() 向list中添加元素
# print(list) 这种操作list中每次都只有一个变量
list.append(lable)



#print(lable)
# lst2 = list(set(lst1))
# print(lst2)

# print(list)
dict = {}
for key in list:
dict[key] = dict.get(key, 0) + 1
print(dict)
 
复制代码

 

 

 

3.我们将统计的结果输出在txt中,这个时候问题出现了,输出的键值对中只有键没有值,这就搞笑了,接着往下走

附引用代码:

复制代码
\'\'\'
将dict在txt中输出
\'\'\'
with open(\'go.obo\', \'r\', encoding="utf-8") as f: # 打开文件
# dict = {}
list = [] ## 空列表

for line in f.readlines(): # 对数据进行每一行的循环
total = []
lable = line.split(":")[0] # 读取列表名,正确来说读取完列表名之后,还要进行去重的处理
# print(lable)
# list.append(lable) ## 使用 append() 向list中添加元素
# print(list) 这种操作list中每次都只有一个变量
list.append(lable)

# print(lable)
print("################################################")
# lst2 = list(set(lst1))
# print(lst2)

# print(list)
dict = {}
for key in list:
dict[key] = dict.get(key, 0) + 1
print(dict)

fileObject = open(\'sampleList.txt\', \'w\')

for ip in dict:
fileObject.write(ip)
fileObject.write(\'\\n\')

fileObject.close()
 
复制代码

 

4.由于我平时处理的json文件比较多,主要面向mongo,所以我试着将其转化为json格式,发现问题解决了,这里还是很神奇的,但是不明确问题出在什么地方。

附引用代码:

复制代码
import json
with open(\'go.obo\', \'r\', encoding="utf-8") as f: # 打开文件
# dict = {}
list = [] ## 空列表

for line in f.readlines(): # 对数据进行每一行的循环
total = []
lable = line.split(":")[0] # 读取列表名,正确来说读取完列表名之后,还要进行去重的处理
# print(lable)
# list.append(lable) ## 使用 append() 向list中添加元素
# print(list) 这种操作list中每次都只有一个变量
list.append(lable)

# print(lable)
print("################################################")
# lst2 = list(set(lst1))
# print(lst2)

# print(list)
dict = {}
for key in list:
dict[key] = dict.get(key, 0) + 1
print(dict)

fileObject = open(\'sampleList.txt\', \'w\')

# for ip in dict:
# fileObject.write(ip)
# fileObject.write(\'\\n\')
#
# fileObject.close()

jsObj = json.dumps(dict)

fileObject = open(\'jsonFile.json\', \'w\')
fileObject.write(jsObj)
fileObject.close()
复制代码

 

5.接下来我先实现简单的测试,抽取部分数据,抽取三个标签,然后再取标签里的两个值

附引用代码:

复制代码
with open(\'nitian\',\'r\',encoding="utf-8") as f:         #打开文件
# dic={} #新建的字典
total = [] #列表
newdic = [] #列表


#在这里进行第一次初始化
#这里的每一个字段都要写两个
id = {} #
id_number = ""#含有一行的为“”\\ 含有一行的为字符串
is_a = {}
is_a_list = []#含有多行的为[] 含有多行的为list


for line in f.readlines(): #对数据进行每一行的循环
lable = line.split(":")[0] #读取列表名,正确来说读取完列表名之后,还要进行去重的处理
#print(lable)
#开始判断
if lable == "id": #冒号前的内容 开始判断冒号之前的内容
id_number = line[3:] #id 两个字母+
# 一个冒号
elif lable == "is_a":
is_a_list.append(line[5:].split(\'\\n\'))

elif line[0] == "[":
#把数据存入newdic[]中
id["id"] = id_number
newdic.append(id)

is_a["is_a"] = is_a_list
newdic.append(is_a)

#把newdic存入总的里面去
total.append(newdic)
#初始化所有新的标签
id = {} # 含有一个的为“”\\
id_number = ""
is_a = {}
is_a_list = []

#初始化小的newdic
newdic = []

total.append(newdic)

print(total)
复制代码

 

6.做到这里我们发现问题出了很多,也就是算法设计出现了问题

数据的开头出现了一系列的空的{id :“ ”} {name:“”} {},{}.....,多了一行初始化,回头检查算法,找到问题:我们用的“[”来判断一个标签的结束

修改方式(1)将符号“[”作为我们判断的开始

    (2)修改数据,将数据中的开头的[term]去掉,加在数据集的结尾

 

7.数据的后面出现了总是出现一些没有意义的“ ”,我们发现是我们没有对每个键值对后面的标签进行处理,所以我们引入了strip()函数,但是strip()函数只能作用于字符串,当你想要作用于list时,要先把list里面的东西拿出来,进而进行操作。

8.键值对的键def 与关键字冲突,我们的解决简单粗暴,直接将其转化为大写

9.完整的代码如下:

附引用代码:

复制代码
 
import json


class GeneOntology(object):

def __init__(self, path):
self.path = path
self.total = []

# Use a dictionary to remove extra values to Simplified procedure
# def rebuild_list(self,record_name):
# records = {id,is_a}
#
# list = rebuile_list(\'HEADER\'\')
# records.get(record_name)


# Define a function to read and store data
def read_storage_data(self):

id = {} #Use a dictionary to store each keyword
id_number = "" # Store the value of each row as a string

is_obsolete = {}
is_obsolete_number = ""

is_class_level = {}
is_class_level_number = ""

transitive_over = {}
transitive_over_number = ""

# There is a place where the keyword “def” conflicts, so I want to change the name here.
DEF = {}
DEF_number = ""

property_value = {}
property_value_number = ""

namespace = {}
namespace_number = ""

comment = {}
comment_number = ""

intersection_of = {}
intersection_of_number = ""

xref = {}
xref_number = ""

name = {}
name_number = ""

disjoint_from = {}
disjoint_from_number = ""

replaced_by = {}
replaced_by_number = ""

relationship = {}
relationship_number = ""

alt_id = {}
alt_id_number = ""

holds_over_chain = {}
holds_over_chain_number = ""

subset = {}
subset_number = ""

expand_assertion_to = {}
expand_assertion_to_number = ""

is_transitive = {}
is_transitive_number = ""

is_metadata_tag = {}
is_metadata_tag_number = ""

inverse_of = {}
inverse_of_number = ""

created_by = {}
created_by_number = ""

creation_date = {}
creation_date_number = ""

consider = {}
consider_number = ""

is_a = {}
is_a_list = [] # A field name may have multiple values, so it is stored in the form of a “list”.

synonym = {}
synonym_list = []

newdic = []
f = open(self.path, \'r\', encoding="utf-8")
for line in f.readlines():
lable = line.split(":")[0] # Read the list ‘name’, starting from the position of \'0\', ending with ":", reading all field names

# View the name of the list that was read

# print(lable)

# Start to judge

if lable == "id": # Judge the label for storage
id_number = line[3:].strip() # Remove the label and colon, occupy 3 positions, and strip() is used to remove the trailing spaces.

elif lable == "is_obsolete":
is_obsolete_number = line[12:].strip()

elif lable == "is_class_level":
is_class_level_number = line[15:].strip()

elif lable == "transitive_over":
transitive_over_number = line[16:]

elif lable == "def":
DEF_number = line[5:].strip()

elif lable == "property_value":
property_value_number = line[15:].strip()

elif lable == "namespace":
namespace_number = line[10:].strip()

elif lable == "comment":
comment_number = line[8:].strip()

elif lable == "intersection_of":
intersection_of_number = line[16:].strip()

elif lable == "xref":
xref_number = line[5:].strip()

elif lable == "name":
name_number = line[5:].strip()

elif lable == "disjoint_from":
disjoint_from_number = line[14:].strip()

elif lable == "replaced_by":
replaced_by_number = line[12:].strip()

elif lable == "relationship":
relationship_number = line[13:].strip()

elif lable == "alt_id":
alt_id_number = line[7:].strip()

elif lable == "holds_over_chain":
holds_over_chain_number = line[17:].strip()

elif lable == "subset":
subset_number = line[7:].strip()

elif lable == "expand_assertion_to":
expand_assertion_to_number = line[20:].strip()

elif lable == "is_transitive":
is_transitive_number = line[14:].strip()

elif lable == "is_metadata_tag":
is_metadata_tag_number = line[16:].strip()

elif lable == "inverse_of":
inverse_of_number = line[11:].strip()

elif lable == "created_by":
created_by_number = line[11:].strip()

elif lable == "creation_date":
creation_date_number = line[14:].strip()

elif lable == "consider":
consider_number = line[9:].strip()


elif lable == "is_a":
is_a_list.append(line[5:].strip().split(\'\\n\'))

elif lable == "synonym":
synonym_list.append(line[8:].strip().split(\'\\n\'))




# Put "[" as the end of the store.
# If you want to "[" as the beginning of your storage, you will have to change the storage format of the data.

elif line[0] == "[":

# Assign values ​​and store the data in newdic[]

id["id"] = id_number
newdic.append(id)

is_obsolete["is_obsolete"] = is_obsolete_number
newdic.append(is_obsolete)

is_class_level["is_class_level"] = is_class_level_number
newdic.append(is_class_level)

transitive_over["transitive_over"] = transitive_over_number
newdic.append(transitive_over)

DEF["def"] = DEF_number
newdic.append(DEF)

property_value["property_value"] = property_value_number
newdic.append(property_value)

namespace["namespace"] = namespace_number
newdic.append(namespace)

comment["comment"] = comment_number
newdic.append(comment)

intersection_of["intersection_of"] = intersection_of_number
newdic.append(intersection_of)

xref["xref"] = xref_number
newdic.append(xref)

name["name"] = name_number
newdic.append(name)

disjoint_from["disjoint_from"] = disjoint_from_number
newdic.append(disjoint_from)

replaced_by["replaced_by"] = replaced_by_number
newdic.append(replaced_by)

relationship["relationship"] = relationship_number
newdic.append(relationship)

alt_id["alt_id"] = alt_id_number
newdic.append(alt_id)

holds_over_chain["holds_over_chain"] = holds_over_chain_number
newdic.append(holds_over_chain)

subset["subset"] = subset_number
newdic.append(subset)

expand_assertion_to["expand_assertion_to"] = expand_assertion_to_number
newdic.append(expand_assertion_to)

is_transitive["is_transitive"] = is_transitive_number
newdic.append(is_transitive)

is_metadata_tag["is_metadata_tag"] = is_metadata_tag_number
newdic.append(is_metadata_tag)

inverse_of["inverse_of"] = inverse_of_number
newdic.append(inverse_of)

created_by["created_by"] = created_by_number
newdic.append(created_by)

creation_date["creation_date"] = creation_date_number
newdic.append(creation_date)

consider["consider"] = consider_number
newdic.append(consider)

is_a["is_a"] = is_a_list
newdic.append(is_a)

synonym["synonym"] = synonym_list
newdic.append(synonym)

# Save newdic in the total data set
self.total.append(newdic)

# Initialize all new tags
id = {}
id_number = ""

is_obsolete = {}
is_obsolete_number = ""

is_class_level = {}
is_class_level_number = ""

transitive_over = {}
transitive_over_number = ""

DEF = {}
DEF_number = ""

property_value = {}
property_value_number = ""

namespace = {}
namespace_number = ""

comment = {}
comment_number = ""

intersection_of = {}
intersection_of_number = ""

xref = {}
xref_number = ""

name = {}
name_number = ""

disjoint_from = {}
disjoint_from_number = ""

replaced_by = {}
replaced_by_number = ""

relationship = {}
relationship_number = ""

alt_id = {}
alt_id_number = ""

holds_over_chain = {}
holds_over_chain_number = ""

subset = {}
subset_number = ""

expand_assertion_to = {}
expand_assertion_to_number = ""

is_transitive = {}
is_transitive_number = ""

is_metadata_tag = {}
is_metadata_tag_number = ""

inverse_of = {}
inverse_of_number = ""

created_by = {}
created_by_number = ""

creation_date = {}
creation_date_number = ""

is_a = {}
is_a_list = []

synonym = {}
synonym_list = []

# Initialize newdic
newdic = []

# total.append(newdic)
# self.total.append(newdic) #You append an empty newdic, so there is an empty one behind []


if __name__ == "__main__":
class1 = GeneOntology(\'go (1).obo\')
class1.read_storage_data()
print(class1.total)

jsObj = json.dumps(class1.total)
fileObject = open(\'jsonFile8.json\', \'w\')
fileObject.write(jsObj)
fileObject.close()
复制代码
 
 
10.总结,做到这一块,也发现了自己的不足,不能依赖于代码,重要的自己思考问题的过程,将自己的逻辑表达清楚,然后哪里出问题,在查找相应的解决方法!

以上是关于利用python 数据分析入门,详细教程,教小白快速入门的主要内容,如果未能解决你的问题,请参考以下文章

小白都能看得懂的教程 一本教你如何在前端实现富文本编辑器

Python 词云 中/英小白简单入门教程

Python安装教程入门必看

pycharm安装2021最新详细教程小白入门

资深黑客教小白如何攻破一个网站!超级详细的教学教程!太牛逼了

小白都能看懂的实战教程 手把手教你Python Web全栈开发 (DAY 2)