从日志文件 python 创建 csv 标题

我的日志文件每行都包含一些信息，如下所示

Info1:NewOrder|key:123 |Info3:10|Info5:abc
Info3:10|Info1:OldOrder| key:456| Info6:xyz
Info1:NewOrder|key:007

我想将其更改为如下所示的 csv（如果我提供 key、Info1、Info3 作为所需标题）

key,Info1.Info3
123,NewOrder,10
456,OldOrder,10
007,NewOrder,

之前我使用 awk 来获取字段值，但日志记录可以更改一行中打印的信息和键的顺序。所以我不能确定 Info3 总是在某个特定的列中。每次记录更改时，都需要更改脚本。

然后我打算在 pandas 数据框中加载 csv。所以 python 解决方案会更好。这更多的是从日志文件生成 csv 的数据清理任务。

这是我阅读答案后使用的

import csv
import sys
with open(sys.argv[1], 'r') as myLogfile:
        log=myLogfile.read().replace('\n', '')

requested_columns = ["OrderID", "TimeStamp", "ErrorCode"]

def wrangle(string, requested_columns):
        data = [dict([element.strip().split(":") for element in row.split("|")]) for row in string.split("\n")]
        body = [[row.get(column) for column in requested_columns] for row in data]
        return [requested_columns] + body

outpath = sys.argv[2]
open(outpath, "w", newline = "") with open(outpath, 'wb')
        writer = csv.writer(file)
        writer.writerows(wrangle(log, requested_columns))

示例日志文件=https://ideone.com/cny805 https://ideone.com/cny805

您可以使用 csv 阅读器|分隔符让您开始，然后使用分割:为您提供每行字典，如下所示：

import csv

with open('input.csv', 'rb') as f_input, open('output.csv', 'wb') as f_output:
    csv_output = csv.writer(f_output)
    cols = ["OrderID", "TimeStamp", "ErrorCode"]
    csv_output.writerow(cols)

    for row in csv.reader(f_input, delimiter='|'):
        # Remove any entries that do not have a colon
        row = [c for c in row if c.find(':') != -1]
        # Convert remaining columns into a dictionary
        entries = {c.split(':')[0].strip() : c.split(':')[1].strip() for c in row}
        csv_output.writerow([entries.get(c, "") for c in cols])

给你一个输出文件：

OrderID,TimeStamp,ErrorCode
3000000,1488948188555841641,
3000000,1488948188556444675,0

要将数据直接读入 Pandas 数据帧：

import pandas as pd
import csv

cols = ["OrderID", "TimeStamp", "ErrorCode"]
data = []

with open('input.csv', 'rb') as f_input:
    csv_output = csv.writer(f_output)

    for row in csv.reader(f_input, delimiter='|'):
        # Remove any entries that do not have a colon
        row = [c for c in row if c.find(':') != -1]
        # Convert remaining columns into a dictionary
        entries = {c.split(':')[0].strip() : c.split(':')[1].strip() for c in row}
        data.append([entries.get(c, "") for c in cols])

df = pd.DataFrame(data, columns=cols)
print df

给你：

   OrderID            TimeStamp ErrorCode
0  3000000  1488948188555841641          
1  3000000  1488948188556444675         0