Python log parser

Question

I have a large log with several commands (ending with ;) and their outputs (till END) like the following:

<blabla;

foo
...
...

END

<xyz;

...
...

END

--and so on

The requirement is to have separate files with command names like

blabla
xyz

and in each file should be their respective outputs.

So far I have:

def generateDicts(log_fh):
currentDict = {}
for line in log_fh:
    if line.endswith(";"):
       if line.endswith("END"):
          yield currentDict
       currentDict = {""}
   else:
      currentDict["text"] += line
yield currentDict

with open("logfile.txt") as f:
print list(generateDicts(f))

Please help.

Answer 1

Your post says you need to write to files, but your example doesn't do any file I/O. Here is a program that opens, closes, and writes to files.

import fileinput

output = None
for line in fileinput.input():
    line2 = line.strip()
    if line2.startswith('<'):
        output = open(line2[1:].split(';')[0], 'w')
    elif line2 == 'END':
        output.close()
        output = None
    elif output:
        output.write(line)

Answer 2

You can use re module

import re
with open('test','r') as f,open('output','w') as f1:
    f1.write("\n".join(re.findall(r'\<(\w+)\;',f.read())))

Output:

blabla
xyz

However, if file size is too large, you can consider reading line by line from the file rather than read it as a whole.