Is there a log file joining utility...?

[joecar]

Is there currently a way to join multiple log files...?

Idea/Suggestion: Scan tool could have a File->"Save Appending to File" option.

:cheers:

[TAQuickness]

I think it's been talked about before, but I'm not aware of such a utility yet.

[Blacky]

Yes, its been requested before. No there is no easy way to do it - yet.

The format of the log files is published (well if not published, then at least it is available from EFILive - its not secret). So anyone who feels like writing a utility to merge log files, feel free.

I've looked at doing it a few times and it is not a trivial process. some things that cause problems:

PID selections.
Order of PID data in frames, even if PIDs selection is the same.
Notes.
Log file comments.
Meta data, such as DTC's, LTFT matrix.
Customer/Vehicle info.

All that data can differ significantly between log files and would need to be "merged" somehow.

If restrictions were put in place so that only the PID data from one log file was appended to an existing log file, that would make the process more managable.

Basically you'd need to read both files into memory, create a new PID selection list (from both sets of PIDs) and then write out a new file with the complete set of PIDs in each frame. Obviously if some PIDs were in one file and not the other then their values would be "undefined" for half of the merged file. (Note: There is a limit of 96 channels per frame).

Regards
Paul

[joecar]

Could I please get a copy of the log file spec...

I don't guarrantee I'll come up with anything, but I'll start in my spare time (oxymoron) to see if I can join log files containing the same pids (ignoring all the other details for now).

[TAQuickness]

Me too please.

[Blacky]

The first part of the log file is plain text and can be viewed using Notepad.
However, do not try to edit the file with Notepad - it can't cope with the non-printable characters. You need to use a hex editor like Hex Workshop from BreakPoint Software.

The first part consists of sections in the format:
[SECTION NAME],COUNT<count><count>
followed by <count> <count> COUNT items in the form:
NAME=VALUE

The [PID] section shows the PID selection and each PID's offset into the frame.
Note: CALC PIDs are not stored in the data frame.

[PID],22
CALC.CYLAIR=0
CALC.INJDC1=0
CALC.INJDC2=0
CALC.POWER_RW=0
GM.AFR=0
GM.HO2S11=2
GM.HO2S21=3
GM.IBPW1=4
GM.IBPW2=6
GM.KR=8
GM.MAFFREQ=9
SAE.ECT=11
SAE.FUELSYS=12
SAE.IAT=14
SAE.LONGFT1=15
SAE.LONGFT2=16
SAE.MAF=17
SAE.MAP=19
SAE.RPM=20
SAE.SPARKADV=22
SAE.TP=23
SAE.VSS=24

The data starts after the [End_Of_Meta_Data] section. The section name is followed by
1. A comma
2. A count of items in the section (for [End_Of_Meta_Data] the count is always 0)
3. 0x0D Carriage return
4. 0x0A Line Feed
Next byte is byte 0 of first data frame.
Data frame extends for CHANNELCOUNT bytes.
Each data frame is followed by a 4 byte NOTES field and a 4 byte TIMESTAMP field.
The TIMESTAMP field is number of milliseconds since 12:00am and is stored LSB first.
There are FRAMECOUNT frames in the file.

</count></count></count></count>

[Highlander]

No utility?

[Blacky]

No utility?
No.

If you have two files with exactly the same PID selection then you could theoretically join them using a hex editor - but it would not be easy.

Regards
Paul

[swingtan]

How interested are people in this?

I had a play in PERL last night and I have a quick and dirty script that will split a log file into it's 3 main components...


Header
Metadata
Log data


and then put the file back together. The final file can be open in EFILive Scan tool with no errors. It shouldn't be too hard to add some more code to allow the same thing to be done on multiple log files...

With exactly the same PID set up

if there is enough interest, I'll then look at being able to do the same thing with differing PID set ups. It'll probably take the form of setting any missing data to "0" or just not allowing any PIDs that don't exist in all files.

As Blacky has said, there will be no fix for the time stamps, but testing so far indicates that it doesn't effect the log data, apart from the time indicator in Scantool. Likewise, flags in the file are probably not going to work, so I'll either have to strip them, or re number them. The only reason I can think of wanting to join log files, is to take a few shorter log files and stitch them together to allow a single BEN map to be set up.

Anyway.... I'd better do some work.

Simon.

[redhardsupra]

Simon, if you need any regex, hit me up, i'm a regex ninja ;)

[mr.prick]

How about being able to compare logs like tunes?

[swingtan]

Simon, if you need any regex, hit me up, i'm a regex ninja ;)

Not a lot of regex is needed, but if I get stuck I'll ask. I've done a ton of shell scripting and some perl before so I have some background.......


How about being able to compare logs like tunes?

I guess that could be done, but it's a whole new world of pain. I get the feeling that the EFILive log files save data in raw hex values and then use the ScanTool to convert them to the desired units for display. This means that the data in the log file means very little on it's own. It would be easy to do a "diff" on the files, but you would get a lot of mess due to time stamps and notes. It would also be a "direct" binary compare so wouldn't be too helpful. What would you be trying to achieve from a "compare" ?

Simon

[mr.prick]

The I guess same thing you achieve when comparing tunes. :hihi:
Honestly I would rather be able to manually select the cells I want to hide.

[swingtan]

Hmmm.... So you could compare say, 2 1/4 mile runs and see the differences? I guess you could do a "frame by frame" difference and output a "delta log file" that would show the difference between the 2 runs, but it would show the difference between 2 "frames" and not the whole log. Interesting thought though.

Simon

[Blacky]

Hmmm.... So you could compare say, 2 1/4 mile runs and see the differences? I guess you could do a "frame by frame" difference and output a "delta log file" that would show the difference between the 2 runs, but it would show the difference between 2 "frames" and not the whole log. Interesting thought though.

Simon

When I've been considering log file comparisons in the past, I've always thought there would have to be one PID (user selected) that could be used as the "join". I.e. calculate comparison data in both log files where the RPM was the same (or within an error range). You'd need to throw out (or average) some frames to keep the two log files synchronized on the selected join PID.

The default join PID would probably be time, other good candidates would be TPS, MAP and MAF.

Just a thought...
Paul

[Blacky]

Honestly I would rather be able to manually select the cells I want to hide.


Its still in the V8 wish list...
Paul

[Highlander]

When I've been considering log file comparisons in the past, I've always thought there would have to be one PID (user selected) that could be used as the "join". I.e. calculate comparison data in both log files where the RPM was the same (or within an error range). You'd need to throw out (or average) some frames to keep the two log files synchronized on the selected join PID.

The default join PID would probably be time, other good candidates would be TPS, MAP and MAF.

Just a thought...
Paul

That would be awesome.. I have always wanted to log 2 instances of a different set of pids and join them together to see a big picture.

[swingtan]

OK, I have something working. It's not pretty, but it does work and will take any number of log files and join them into a single log. I need to re do the help section though so it gives some instructions if needed, but it's pretty basic. Currently you just pass it the source file ( which is where the metadata is taken from ) each additional file, ( whose log data is to be added to the source file ) and finally the "output" file is passed in.

Here is a a join of 4 log files. To show that it really is 4 separate files, I've joined the start of each log that shows the ECT warming up. You can also follow the time stamps. Finally, I have noticed that notes in the log files work to some degree. I'm not sure how well they work but they are at least still visible in the joined log.

Simon.

[joecar]

Cool... good job...:cheers:

[mr.prick]

My update display frequency is 2, but this log plays faster like it is set at 1.
I set it to 4 and it played back at the speed of 2.
Is this a side effect or do logs have the display frequency stored in them?

[swingtan]

Hmmmm I didn't actually check that. I'm looking at it now and is seems to be running at normal speed for me.

I'm now looking at compiling the PERL code so it'll run without the need to install PERL ( though the *nix uses will already have that ).

Simon.

[vxchev8]

Nice work Simon.

[swingtan]

OK, version 0.1 is available for Beta testing......

Remember: This is an "all care, no responsibility" production....

I've tested it on both E38 and LS1 log files and it works OK. It should be fine for the other controllers as long as...


The 1KB header is std.
The metadata section starts and ends the same way.
There are no additional data sections in the log file ( apart from the actual log data).


Basics:


This utility ONLY works from the command line. If you are not comfortable using the command prompt / DOS window / ETC. then this is not for you. While the utility is only meant to work on EFILive log files, it will happily write to any file named. It is possible to overwrite files that you didn't actually want over written. PEBCAK is the responsibility of the user....


Set up:


simply unzip the file and save it to where ever you want. You can save it to C:\Windows or C:\Windows\system32 and it will be in your default path for the command prompt. If you put it somewhere else, you may need to add that directory to the path variable or use the full path when running the tool. I'd stick it in C:\Windows, unless there is already a file there called "join.exe"


Usage:


Start a command prompt window.


click on [start] - [run]
type in "cmd" ( no quotes )
then click on [OK]


Then change directory to where you have stored your log files. The STD. directory is under your "My Documents" folder.


cd "My Documents"\EFILive\V7.5\"Logged Data"


Next identify the 3 required parameters for the tool...


Source File: This file is used for the PID description and it's data will appear first in the final log.
Additional Files: the names of each file that you want to add to the source file. They are added in series so you can set the order you want.
Output File: This is what the final joined log file will be called.
WARNING: This tool will overwrite any file named as the output file if it exists. It will not ask for confirmation.


You can run the tool in it's basic form....



join -s [source file] -a [add file-1] ... -a [add file-n] -o [output file]


It will take a couple of seconds to set up the environment, and then output some details on the screen as to what is going on. A sample screen will look like this...


> join -s BBL00119.efi -a BBL00120.efi -a BBL00121.efi -o test.efi

Source file: BBL00119.efi
Source size: 51637
Source Meta size: 613

Next File: BBL00120.efi
Next size: 51637
Next meta: 613
Next File: BBL00121.efi
Next size: 52137
Next meta: 613

Completed join to output file test.efi


There are 3 other command line switches.....

-h : The "help" option, which gives some tips on the tool and how to use it.
-v : Verbose mode, which just prints the metadata data as found in the source file.
-d : Debug mode, which does everything EXCEPT write to the output file. This mode is useful when combined with "-v" to list the metadata if you want to check things like the PID selections.



So, if you feel up to it, try it out and see if it is of any use.

Simon.

[Blacky]

My update display frequency is 2, but this log plays faster like it is set at 1.
I set it to 4 and it played back at the speed of 2.
Is this a side effect or do logs have the display frequency stored in them?

The playback speed is based on the timestamps in the frames in the log file. It is anchored off of the first frame. I guess if the timestamp of the frames in the appended file are later then the actual file, then EFILive will insert a configurable delay (Properties->Playback->Frame times->Max pause) whenever it encounters two frames with a gap larger than the configured time delay between them.

If two frames have a gap that is negative, i.e. the second frame has an earlier timestamp then the first frame then I don't know what will happen. I never coded for that possibility. (I figured backward time travel had not been invented yet so I would be safe with that assumption).

I think what would happen it the gap between the two frames would blow out to a few years (around 2^31 seconds or about 68 years) and EFILive would just make the delay between the frames whatever Max pause is set to.

Regards
Paul

[mr.prick]

Like I said my settings for playback are a little slower than real time,
I opened this file and it played back in real time regardless of my settings.
I found this odd.

Looking at the log again I see that 30 channels are selected too,
making this all the more spooky.

[swingtan]

E38....... It's amazing how much data you can log

[vxchev8]

:cucumber:

:thankyou2: Nice work Simon - It allowed me to join two logs without any problems.

I put the join.exe file into the windows folder then simply followed your instructions.

:cheers:

[swingtan]

Hi All,
I'm part way through the next version that will hopefully work on specific PID's. It should be able to pull a set list of PID's from multiple files and join them together.

Here's a quick screen dump....



swingtan > logtool.pl -s small.efi -f

Source file: small.efi
Source size: 4031
Source Meta size: 1957

Channel Count: 42
Frame size: 50
Data length: 1050
Frame Data:

49 a3 2b 3b 00 3c 03 00 00 01 8e 85 2f 85 00 00 00 03 82 0c 86 05 68 80 80 0c 39 65 d4 00 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 c7 9a 93 03
49 a6 2b 3b 00 3c 06 00 00 01 8e 85 2f 85 00 00 00 03 8a 0c 92 05 6b 80 80 0c 3e 65 d4 01 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 e6 9a 93 03
49 a6 2b 3b 00 3c 06 00 00 01 9b 85 2f 8c 00 00 00 03 86 0c 97 05 6a 80 80 0c 68 65 be 03 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 05 9b 93 03
49 a2 2b 3b 00 3c 09 00 00 01 95 85 2f 8c 00 00 00 03 93 0c 9f 05 6a 80 80 0c 73 66 00 05 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 25 9b 93 03
49 a1 2b 3b 00 3c 07 00 00 01 95 85 2f 8c 00 00 00 03 93 0c 9f 05 6d 80 80 0c 85 65 d4 06 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 44 9b 93 03
49 a1 2b 3b 00 3c 06 00 00 01 8e 85 2f 8c 00 00 00 03 8d 0c 8c 05 6e 80 80 0c 96 66 00 00 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 63 9b 93 03
49 9d 2b 3c 00 3c 06 00 00 01 8e 85 2f 8c 00 00 00 03 8a 0c 92 05 70 80 80 0c 85 65 d4 02 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 82 9b 93 03
49 9f 2b 3c 00 3c 06 00 00 01 88 85 2f 8c 00 00 00 03 7f 0c 82 05 71 80 80 0c 84 65 be 03 00 01 03 7d 05 20 00 00 06 a5 09 c4 00 00 00 00 a1 9b 93 03
49 9f 2b 3c 00 3c 04 00 00 01 88 85 2f 8c 00 00 00 03 7a 0c 80 05 6e 80 80 0c a0 66 16 05 00 01 03 7e 05 22 00 00 06 a5 09 c4 00 00 00 00 c0 9b 93 03
49 9a 2a 3c 00 3c 04 00 00 01 88 85 2f 8c 00 00 00 03 81 0c 85 05 6a 80 80 0c b3 66 00 07 00 01 03 7e 05 22 00 00 06 a5 09 c4 00 00 00 00 df 9b 93 03
49 99 2a 3d 00 3c 04 00 00 01 88 85 2f 92 00 00 00 03 7e 0c 80 05 66 80 80 0c ba 66 2c 00 00 01 03 7e 05 22 00 00 06 a5 09 c4 00 00 00 00 fe 9b 93 03
49 98 2b 3d 00 3c 04 00 00 01 88 85 2f 92 00 00 00 03 7e 0c 77 05 65 80 80 0c a2 66 16 02 00 01 03 7f 05 23 00 00 06 a5 09 c4 01 00 00 00 1d 9c 93 03
49 9b 2b 3d 00 3c 03 00 00 01 8e 85 2f 92 00 00 00 03 79 0c 79 05 65 80 80 0c 7c 66 00 04 00 01 03 80 05 25 00 00 06 a5 09 c4 00 00 00 00 3c 9c 93 03
49 a0 2b 3b 00 3c 03 00 00 01 88 85 2f 92 00 00 00 03 81 0c 85 05 65 80 80 0c 78 66 16 05 00 01 03 81 05 26 00 00 06 a5 09 c4 00 00 00 00 5c 9c 93 03
49 a0 2b 3b 00 3c 03 00 00 01 88 85 2f 92 00 00 00 03 7a 0c 77 05 63 80 80 0c 63 66 00 07 00 01 03 81 05 26 00 00 06 a5 09 c4 00 00 00 00 7b 9c 93 03
49 a3 2a 3b 00 3c 03 00 00 01 88 85 2f 92 00 00 00 03 7f 0c 81 05 65 80 80 0c 56 66 2c 01 00 01 03 81 05 26 00 00 06 a5 09 c4 00 00 00 00 9a 9c 93 03
49 a3 2a 3b 00 3c 03 00 00 01 8e 85 2f 92 00 00 00 03 81 0c 85 05 65 80 80 0c 8d 66 00 02 00 01 03 82 05 28 00 00 06 a5 09 c4 00 00 00 00 b9 9c 93 03
49 a2 2b 3b 00 3c 03 00 00 01 88 85 2f 92 00 00 00 03 80 0c 7d 05 61 80 80 0c 9d 66 43 04 00 01 03 84 05 2b 00 00 06 a5 09 c4 00 00 00 00 d8 9c 93 03
49 9c 2a 3b 00 3c 06 00 00 01 88 85 2f 98 00 00 00 03 7c 0c 7d 05 62 80 80 0c ae 66 2c 06 00 01 03 84 05 2b 00 00 06 a5 09 c4 00 00 00 00 f7 9c 93 03
49 9a 2b 3d 00 3c 04 00 00 01 88 85 2f 98 00 00 00 03 7c 0c 7d 05 5e 80 80 0c 95 66 16 07 00 01 03 86 05 2d 00 00 06 a5 09 c4 00 00 00 00 16 9d 93 03
49 9d 2b 3d 00 3c 04 00 00 01 88 85 2f 98 00 00 00 03 7b 0c 80 05 61 80 80 0c 87 66 2c 01 00 01 03 86 05 2d 00 00 06 a5 09 c4 00 00 00 00 36 9d 93 03

[joecar]

Simon, good job mate...:cheers:

[vxchev8]

...