python 使用yield进行数据的流式处理

Posted 2021-01-15 将者，智、信、仁、勇、严也。

demo：从文件中取包含字符“a”的5行数据做一次批处理！！！

# coding: utf-8

import time

def cat(f):
    for line in f:
        yield line

def grep(lines,searchtext):
  for line in lines:
    if searchtext in line:
      yield line

def process(segment_lines):
  print (segment_lines)

flog = cat(open(‘/tmp/tflearn_logs/vae/events.out.tfevents.1540467860.lanzihuazaideMacBook-Pro.local‘))
pylines = grep(flog, ‘a‘)
segment_lines = []
for line in pylines:
  if len(segment_lines) < 5:
      segment_lines.append(line)
  else:
      print ("*"*88)
      process(segment_lines)
      segment_lines = []
if segment_lines:
    print ("*"*88)
    process(segment_lines)