Is there a log file joining utility...?

[joecar]

Is there currently a way to join multiple log files...?

Idea/Suggestion: Scan tool could have a File->"Save Appending to File" option.

[TAQuickness]

I think it's been talked about before, but I'm not aware of such a utility yet.

[Blacky]

Yes, its been requested before. No there is no easy way to do it - yet.

The format of the log files is published (well if not published, then at least it is available from EFILive - its not secret). So anyone who feels like writing a utility to merge log files, feel free.

I've looked at doing it a few times and it is not a trivial process. some things that cause problems:

PID selections.
Order of PID data in frames, even if PIDs selection is the same.
Notes.
Log file comments.
Meta data, such as DTC's, LTFT matrix.
Customer/Vehicle info.

All that data can differ significantly between log files and would need to be "merged" somehow.

If restrictions were put in place so that only the PID data from one log file was appended to an existing log file, that would make the process more managable.

Basically you'd need to read both files into memory, create a new PID selection list (from both sets of PIDs) and then write out a new file with the complete set of PIDs in each frame. Obviously if some PIDs were in one file and not the other then their values would be "undefined" for half of the merged file. (Note: There is a limit of 96 channels per frame).

Regards
Paul

[joecar]

Could I please get a copy of the log file spec...

I don't guarrantee I'll come up with anything, but I'll start in my spare time (oxymoron) to see if I can join log files containing the same pids (ignoring all the other details for now).

[TAQuickness]

Me too please.

[Blacky]

The first part of the log file is plain text and can be viewed using Notepad.
However, do not try to edit the file with Notepad - it can't cope with the non-printable characters. You need to use a hex editor like Hex Workshop from BreakPoint Software.

The first part consists of sections in the format:
[SECTION NAME],COUNT
followed by COUNT items in the form:
NAME=VALUE

The [PID] section shows the PID selection and each PID's offset into the frame.
Note: CALC PIDs are not stored in the data frame.

[PID],22
CALC.CYLAIR=0
CALC.INJDC1=0
CALC.INJDC2=0
CALC.POWER_RW=0
GM.AFR=0
GM.HO2S11=2
GM.HO2S21=3
GM.IBPW1=4
GM.IBPW2=6
GM.KR=8
GM.MAFFREQ=9
SAE.ECT=11
SAE.FUELSYS=12
SAE.IAT=14
SAE.LONGFT1=15
SAE.LONGFT2=16
SAE.MAF=17
SAE.MAP=19
SAE.RPM=20
SAE.SPARKADV=22
SAE.TP=23
SAE.VSS=24

The data starts after the [End_Of_Meta_Data] section. The section name is followed by
1. A comma
2. A count of items in the section (for [End_Of_Meta_Data] the count is always 0)
3. 0x0D Carriage return
4. 0x0A Line Feed
Next byte is byte 0 of first data frame.
Data frame extends for CHANNELCOUNT bytes.
Each data frame is followed by a 4 byte NOTES field and a 4 byte TIMESTAMP field.
The TIMESTAMP field is number of milliseconds since 12:00am and is stored LSB first.
There are FRAMECOUNT frames in the file.

[Highlander]

No utility?

[Blacky]

No utility?

No.

If you have two files with exactly the same PID selection then you could theoretically join them using a hex editor - but it would not be easy.

Regards
Paul

[swingtan]

How interested are people in this?

I had a play in PERL last night and I have a quick and dirty script that will split a log file into it's 3 main components...

• Header
• Metadata
• Log data

and then put the file back together. The final file can be open in EFILive Scan tool with no errors. It shouldn't be too hard to add some more code to allow the same thing to be done on multiple log files...

With exactly the same PID set up

if there is enough interest, I'll then look at being able to do the same thing with differing PID set ups. It'll probably take the form of setting any missing data to "0" or just not allowing any PIDs that don't exist in all files.

As Blacky has said, there will be no fix for the time stamps, but testing so far indicates that it doesn't effect the log data, apart from the time indicator in Scantool. Likewise, flags in the file are probably not going to work, so I'll either have to strip them, or re number them. The only reason I can think of wanting to join log files, is to take a few shorter log files and stitch them together to allow a single BEN map to be set up.

Anyway.... I'd better do some work.

Simon.

[redhardsupra]

Simon, if you need any regex, hit me up, i'm a regex ninja