Data Analysis Software - Motec, Gems, AEM

General Q&A specific to RaceCapture/Pro. For app related questions post in the RaceCapture App forum. also see the <a href="http://www.autosportlabs.net/RaceCapture">RaceCapture Information Page</a>

Moderators: JeffC, rdoherty, stieg, brentp

redparchel
Posts: 22
Joined: Fri Nov 01, 2013 9:48 pm
Location: Bay Area, CA

Post by redparchel »

Big thanks to jpf11 and neoraptor for the conversion script, a small tweak for me and it worked great! (now i just need more data make use of the tools better)
brentp wrote:Thinking we should make it a 'save as' in Race Analyzer.
+1

chono
Posts: 1
Joined: Sat Nov 30, 2013 6:38 pm

Post by chono »

Minor Tweak:
There are more than one column (in my log) with "Time" in the name.

So I changed:
if ( $array[0][$i] =~ /Time/ ) {

to:
if ( $array[0][$i] =~ /\bTime\b/ ) {

jpf11
Posts: 57
Joined: Wed Oct 23, 2013 9:42 pm
Location: United States

Post by jpf11 »

Chono, thanks for that. I had read your post a while ago and didn't really think about what you were saying, and today ran into the exact problem having SplitTime and LapTime columns in my latest data. Totally screwed up my script's execution. I'll have to add your fix and update in Git.

L Cubed
Posts: 24
Joined: Wed Aug 06, 2014 9:11 pm

Post by L Cubed »

So I am trying to use the AEM/GEMS data analysis software for its added functionality. I installed Perl and have successfully run the rccsv2gems program (said the command prompt line). The notes in the rccsv2gems program mention a .srt file type which neither program recognizes. If i output a .stf file (GEMS base data log file) then I try to open the file and a warning which says "failed to load" (screenshot attached). Has anyone run into this problem before? Anything I might be doing wrong in converting the file with the perl program? I did notice that there is an extra line created in the data; line 495 "print OUTFILE "\r\n";". Changing this to either "\r" or "\n" does not fix the file loading problem.
Attachments
Screenshot after trying to load converted file.
Screenshot after trying to load converted file.
Failed to Load.JPG (76.23 KiB) Viewed 221201 times

L Cubed
Posts: 24
Joined: Wed Aug 06, 2014 9:11 pm

Post by L Cubed »

So with some fiddling I figured it out. I am adding this description since I got lost trying to do this for a while.

First use the rccsv2gems Perl program to convert the .log file from RCP to a .csv file. This program re-arranges the data so that imports better. Using the DLOG99 software from the GEMS UK site, select File then Import and select the newly created .csv file. This will prompt you to save it as a certain file type, save as whatever name you like with the .stf (data file) type. This new data log file of .stf type can be opened by either GEMS Data Analysis or AEMdata. I prefer AEMdata because it is free and has the math channel funcitonality which the free version of GEMS Data Analysis does not have.

To add a little more information, I used Strawberry Perl for Windows. Once the Perl software is loaded you will need to add the Text::CSV functionality (check out this link http://www.cpan.org/modules/INSTALL.html). Then you can use the command line to execute the rccsv2gems.pl file and convert the files. The command line will look like "C:\File Location\> perl rccsv2gems.pl RC_123input.log RC_123output.csv --disable-gps-cleanup" if you don't need the GPS cleanup.

Hope this helps some others get into data analysis software that can help them get driving faster!

brentp
Site Admin
Posts: 6274
Joined: Wed Jan 24, 2007 6:36 am

Post by brentp »

This is great, thank you for the update!

We will be building in the GEMS export feature into the V2 software so it'll be easier still.
Brent Picasso
CEO and Founder, Autosport Labs
Facebook | Twitter

Billy C
Posts: 31
Joined: Thu Jan 03, 2013 2:11 am

Post by Billy C »

brentp wrote: We will be building in the GEMS export feature into the V2 software so it'll be easier still.
I can't wait for this. :D :D :D
It will be an awesome feature!!!

I just can't seem to get the perl script/DLog99/GEMS to cooperate. I'm not much of a programmer yet.
-Billy C

brentp
Site Admin
Posts: 6274
Joined: Wed Jan 24, 2007 6:36 am

Post by brentp »

Excellent. Watch for updates to the V2 software, it's actively being worked on now!
Brent Picasso
CEO and Founder, Autosport Labs
Facebook | Twitter

toga94m
Posts: 127
Joined: Wed Jun 05, 2013 1:57 am
Location: Upstate NY

Post by toga94m »

chono wrote:Minor Tweak:
There are more than one column (in my log) with "Time" in the name.

So I changed:
if ( $array[0][$i] =~ /Time/ ) {

to:
if ( $array[0][$i] =~ /\bTime\b/ ) {
This needs to get to github still, at autosportlabs/RCP2GEMS (don't know if the original author's repo is current). I had more trouble getting Text::CSV added to perl on my box than I did getting the perl script to run... learning curve. One more thing to add to the .sig I guess...
------------
Learning Race Capture Pro... on someone else's car
Learning Python/Kivy on my own PC

Copper280z
Posts: 30
Joined: Sun Mar 15, 2015 3:03 am
Location: United States

Post by Copper280z »

Do I have my RCP settings wrong or is there no "Time" column in the current datalogs? All I have are "Interval" and "Utc". I get an error saying "Time column is not present in input csv". Neither "Interval" or "Utc" are directly equivalent to what the script expects for "Time" either, I've tried sticking either heading into line 139. The result is zeros in column 1.

Also, I think on line 333, "GpsSats" needs to be changed to "GPSSats", otherwise it tells me to run with the GPS cleanup disable flag.

Edit: Fixed it. Found this thread http://www.autosportlabs.org/viewtopic.php?t=4334, incorporated the modification mentioned there and my fix for the GPSSats column. I also had to remove \n from line 499. Tested it and it seems to work. I'm not sure if the code for going past midnight is needed anymore or if it's going to cause any issues, I'd need to look into how it works a little more. I added a few (hopefully accurate) comments too.

Here is the current code that seems to work for me:

Code: Select all

#!/usr/bin/perl

##  Author&#58;		John Freund  [email protected]
##  Script Name&#58;	rccsv2gems.pl
##  Function&#58;		Takes Race Capture Pro &#40;v1.13&#41; csv logs and converts them to a
##			GEMS Dlog99 compatible format, so you can then save logs in
##			Dlog99 into .srt files which can be opened by GEMS Data Analysis
##			or AEM Data Analysis.
##  Notes&#58;		Thanks to neoraptor on the autosportlabs.org forums for
##			providing the format!
##  Version&#58;		1.6
##
##  Changes&#58;		1.6 - Added time interpolation.  Found that Dlog99 was dropping data
##				when data points had the same time values, which they would
##				as this script was setting empty time values to the last known
##				good time value.  To prevent the dropping I added a function
##				that interpolates time for data points between logged time
##				and sets the interpolated time values in the output.	
##			1.5 - Removed second tag_junk_lines function call from main as it 	
##				was redundant.  Added options for new parameters mingps and
##				disable-gps-cleanup.  "--mingps=X" sets the minimum
##				number of gps satellites required for gps data to be valid, 
##				defaults to 4.  "--disable-gps-cleanup" will disable cleaning up
##				by the data &#40;defaults to false&#41; and switch pre-cleanup to setting
##				all gps values prior to the first good set to the same as the
##				first good set.  Added lots of output to the
##				various data processing functions to indicate their status.
##				Enabled the gps lat/long conversion from degrees to radians
##				to make AEM/GEMS DA setup more seamless since they expect 
##				radians by default.
##			1.4 - Fixed an issue with how we fill in the blanks.  We were filling
##				in the blanks with last known good values, including "good" 
##				values from lines tagged for removal.  Changed to ignore those
##				lines so values from them don't corrupt the rest of the data. 	
##			1.3 - Added comments.  Changed junk character handling from setting
##				values to prior "good" values to instead just delete the
##				offending lines.  Also changed ctrl M stripping to be all 
##				elements as random ^M were screwing up other text matches. 
##				Moved the ^M stripping to earlier in the pipeline too.	
##			1.2 - Found that AEM/GEMS DA had issues with automatically
##				figuring out the track layout with how I was cleaning
##				up gps data.  For any missing data in the beginning of the
##				file I was entering "0.000" up until the data starts showing
##				up &#40;after which any missing data is filled with the last 
##				known data&#41;.  This is a problem if all your gps data isn't 
##				around 0.000,0.000 as your data goes from that coordinate
##				to whatever the first real lat/long you have in your data. 
##				When AEM/GEMS DA reads the initial 0, it uses that to figure
##				out along with your other gps data how to draw the track 
##				and basically your line starts at 0,0 and goes immediately to
##				your first coordinate and continues on, and the track scales
##				WAY out &#40;unless you happened to be logging data around 0,0&#41;.
##				My fix was basically to wipe any data before the first real
##				GPS coordinates.
##			1.1 - Added code to zero_start_time function that handles time
##				rollover at midnight.  Not a complete catch as to make
##				things simpler I assume the log will end before 10am 
##				the next day.  Also fixes time conversion issues for
##				times earlier than 10am as the values did not have
##				leading zeroes &#40;i.e. 00 or 01-09 for hour&#41; by and thus
##				the function I was using for time conversion to epoch
##				time was failing.  I just pad some zeroes in front 
##				right before the conversion, and then I add 86400 secs
##				if we're rolled past midnight.
##				Also got rid of the fill_in_the_blanks function by 
##				rolling it into the zero_start_time function.
##			1.0	initial release
##
##  Usage&#58;		rccsv2gems.pl INPUTFILENAME OUTPUTFILENAME --minsats=X --disable-gps-cleanup
##			&#40;optional&#41;  --minsats=X where X is the minimum number of gps satellites required 
##				for valid data &#40;sets lat/long to null if GpsSats value is less than min&#41;
##				Default is 4.
##			&#40;optional&#41;  --disable-gps-cleanup disables the cleanup per the GpsSats value.
##					Useful when you don't care or don't have the GpsSats column logged.

use strict;
use Text&#58;&#58;CSV;
use Scalar&#58;&#58;Util 'looks_like_number';
use Time&#58;&#58;Piece;
use POSIX qw&#40;ceil floor&#41;;

my $script_name="rccsv2gems.pl";
my $version="1.0";

my $file = shift&#40;@ARGV&#41;; #takes parameter of RaceCapture Pro csv log to modify for GEMS
my $output = shift&#40;@ARGV&#41;;
my $arg3 = shift&#40;@ARGV&#41;;
my $arg4 = shift&#40;@ARGV&#41;;

my $minimum_sats=4;  #default minimum number of gps satellites required
my $disable_gps_cleanup="false"; #gps cleanup is enabled by default


## read in parameters
if &#40;!defined $file&#41; &#123;
	die "Error - No input file specified.  Terminating.\n";
&#125;

if &#40;!defined $output&#41; &#123;
	die "Error - No output file specified.  Terminating\n";
&#125;

if &#40; $arg3 &#41; &#123;
	if &#40; $arg3 =~ /\-\-minsats\=/ &#41; &#123;
		$minimum_sats = &#40;split&#40; /\=/ ,$arg3 &#41;&#41;&#91;1&#93;;
	&#125;
	if &#40; $arg3 =~ /\-\-disable\-gps\-cleanup/ &#41; &#123;
		$disable_gps_cleanup="true";		
	&#125;
&#125;

if &#40; $arg4 &#41; &#123;
        if &#40; $arg4 =~ /\-\-minsats\=/ &#41; &#123;
                $minimum_sats= &#40;split&#40; /\=/ ,$arg4 &#41;&#41;&#91;1&#93;;
        &#125;
        if &#40; $arg4 =~ /\-\-disable\-gps\-cleanup/ &#41; &#123;
                $disable_gps_cleanup="true";
        &#125;
&#125;

## Functions
sub modify_headers &#123;
	my @array=@_;
	print "Initiating Header Cleanup - Cleans up column headers.\n";
	for&#40;my $i = 0; $i <= $#&#123;$array&#91;0&#93;&#125; ; $i++&#41;&#123;
		my @element = split &#40;/\|/, $array&#91;0&#93;&#91;$i&#93;&#41;;
		$array&#91;0&#93;&#91;$i&#93;=$element&#91;0&#93;;	
		$array&#91;0&#93;&#91;$i&#93; =~ s/\"//g; 
	&#125;
	print "Header Cleanup Completed Successfully!\n";
	print "\n";
&#125;

sub move_time_to_first_column &#123;
	my @array=@_;
	my $timecolumn="null";
	print "Initiating Moving Time Column to Front - Moves the Time column to the first column.\n";
	for&#40;my $i = 0; $i <= $#&#123;$array&#91;0&#93;&#125; ; $i++&#41;&#123;
		if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Utc/ &#41; &#123;		#identifies the column that contains time, was originally "Time", but I don't see that in my logs
			$timecolumn = $i;
		&#125;
	&#125;
	if &#40;$timecolumn == "null"&#41;&#123;
		die "Error - Time column is not present in input csv.  Terminating\n";
	&#125;
	for&#40;my $i = 0; $i <= $#array; $i++&#41;&#123;
		my $swap = $array&#91;$i&#93;&#91;0&#93;;
		$array&#91;$i&#93;&#91;0&#93; = $array&#91;$i&#93;&#91;$timecolumn&#93;;
		$array&#91;$i&#93;&#91;$timecolumn&#93; = $swap;
	&#125;
	print "Time Column to Front Completed Successfully!\n";
	print "\n";
&#125;

sub convert_time &#123;
	
	my $time=$_&#91;0&#93;;
	my $newtime = $time / 1000;  # ver 2 firmware timestamp is UTC milliseconds already - tcm 21Jan15 
	return $newtime;
&#125;

sub zero_start_time &#123;  #resets the time columns value to start at 0 by subtracting first value, also covers for going over midnight somewhat
	my @array=@_;
	my $base_time="null";
	my $pastmidnight="0";
	my $last_time="0";
	my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;
	print "Initiating Time Adjustment - Sets all times relative to the start time and converts times to seconds.\n";
	for&#40;my $i = 1; $i <= $#array ; $i++&#41;&#123;
		if &#40;$array&#91;$i&#93;&#91;$remove_column&#93; ne "remove"&#41;&#123;	#ignores rows tagged for removal
		if &#40;&#40; looks_like_number&#40;$array&#91;$i&#93;&#91;0&#93;&#41; != 0 &#41; && &#40; $last_time > 230000 &#41; && &#40; $array&#91;$i&#93;&#91;0&#93; < 100000 &#41;&#41;&#123; #if it's a number, time from previous iteration is greater than 230000ms and current time is less than 100000
			$pastmidnight="1";
		&#125;
		if &#40;&#40; looks_like_number&#40;$array&#91;$i&#93;&#91;0&#93;&#41; != 0 &#41; && &#40;$base_time == "null" &#41;&#41; &#123; #if it's a number, and this is the first time entry
			$last_time=$array&#91;$i&#93;&#91;0&#93;;
			$base_time=convert_time&#40;$array&#91;$i&#93;&#91;0&#93;&#41;;
			$array&#91;$i&#93;&#91;0&#93;="0.000";
			
		&#125;
		else &#123;
			if &#40; looks_like_number&#40;$array&#91;$i&#93;&#91;0&#93;&#41; != 0 &#41; &#123;
				if &#40; $array&#91;$i&#93;&#91;0&#93; < 100000&#41; &#123;
					if &#40; $array&#91;$i&#93;&#91;0&#93; < 10 &#41; &#123;
							$array&#91;$i&#93;&#91;0&#93;="00000$array&#91;$i&#93;&#91;0&#93;";
					&#125; elsif &#40; $array&#91;$i&#93;&#91;0&#93; < 100 &#41; &#123;
                                                        $array&#91;$i&#93;&#91;0&#93;="0000$array&#91;$i&#93;&#91;0&#93;";
					&#125; elsif &#40; $array&#91;$i&#93;&#91;0&#93; < 1000 &#41; &#123;
                                                        $array&#91;$i&#93;&#91;0&#93;="000$array&#91;$i&#93;&#91;0&#93;";
                                        &#125; elsif &#40; $array&#91;$i&#93;&#91;0&#93; < 10000 &#41; &#123;
                                                        $array&#91;$i&#93;&#91;0&#93;="00$array&#91;$i&#93;&#91;0&#93;";
                                        &#125; else &#123; $array&#91;$i&#93;&#91;0&#93; = "0$array&#91;$i&#93;&#91;0&#93;";
					&#125;
					
				&#125;
				my $converted_time = convert_time&#40;$array&#91;$i&#93;&#91;0&#93;&#41;;
				if &#40; $pastmidnight == "1" &#41; &#123;
					$converted_time+=86400;
				&#125;	
				my $diff=&#40;$converted_time - $base_time&#41;;
				$array&#91;$i&#93;&#91;0&#93;=sprintf"%.3f",$diff;
			&#125;
		&#125;
		&#125;
	&#125;
	if &#40;$base_time == "null"&#41;&#123;
		die "Error - No time values were present.  Terminating.\n";
	&#125;
	print "Time Adjustment Completed Successfully!\n";
	print "\n";
&#125;

sub strip_ctrlm &#123;
	my @array=@_;
	for&#40;my $i = 0; $i <= $#array ; $i++&#41;&#123;
		for&#40;my $j = 0; $j <= $#&#123;$array&#91;0&#93;&#125; ; $j++&#41;&#123;
			$array&#91;$i&#93;&#91;$j&#93; =~ s/\r//g;	
		&#125;
	&#125;
&#125;


sub fill_in_the_blanks &#123; # fills in missing data with last known good data.  Ignores lines tagged for removal.
	my @array = @_;
	my @line = @&#123; $array&#91;1&#93; &#125;;
	my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;
	print "Initiating Filling in the Blanks - Filling in empty data values with last known good value to make AEM/GEMS DA happy.\n";
	for&#40;my $i = 1; $i <= $#array ; $i++&#41;&#123;
		if &#40; $array&#91;$i&#93;&#91;$remove_column&#93; ne "remove" &#41; &#123;
			for&#40;my $j = 0; $j <= $#&#123;$array&#91;0&#93;&#125; ; $j++&#41;&#123;
				if &#40; looks_like_number&#40;$array&#91;$i&#93;&#91;$j&#93;&#41; == "0" &#41; &#123;
					if &#40; looks_like_number&#40;$line&#91;$j&#93;&#41; == "0" &#41; &#123;
						$line&#91;$j&#93;="0.000";
					&#125;	
					$array&#91;$i&#93;&#91;$j&#93;="$line&#91;$j&#93;";
				&#125;
				else &#123;
					$line&#91;$j&#93;=$array&#91;$i&#93;&#91;$j&#93;;
				&#125;
			&#125;
		&#125;
	&#125;
	print "Filling in the Blanks Completed Successfully!\n";
	print "\n";

&#125;

sub convert_gpsdegrees_to_radians &#123;
	my @array = @_;
        my $lat_col="null";
	my $long_col="null";
	my $radian_conv="0.01745329251";
	print "Initializing GPS Degrees to Radians - Converts Lat/Long values from Degrees to Radians &#40;which is default for AEM/GEMS DA&#41;.\n";
        for&#40;my $i = 0; $i <= $#&#123;$array&#91;0&#93;&#125; ; $i++&#41;&#123;
                if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Longitude/ &#41; &#123;
                        $long_col = $i;
                &#125;
		if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Latitude/ &#41; &#123;
			$lat_col = $i;
		&#125;
        &#125;
        if &#40;$long_col == "null"&#41;&#123;
                die "Error - Longitude column is not present in input csv.  Terminating.\n";
        &#125;
	if &#40;$lat_col == "null"&#41;&#123;
		die "Error - Latitude column is not present in input csv.  Terminating\n";
	&#125;
	for&#40;my $i = 1; $i <= $#array ; $i++&#41;&#123;
		$array&#91;$i&#93;&#91;$long_col&#93; = sprintf"%.13f",&#40;$array&#91;$i&#93;&#91;$long_col&#93; * $radian_conv&#41;;
		$array&#91;$i&#93;&#91;$lat_col&#93; = sprintf"%.13f",&#40;$array&#91;$i&#93;&#91;$lat_col&#93; * $radian_conv&#41;;
	&#125;
	print "GPS Degrees to Radians Completed Successfully!\n";
	print "\n";
&#125;

sub tag_pre_gps_data &#123; #tags data before the gps data shows up.  no gps data screws up track setup in GEMS DA. must be run before fill_in_the_blanks
        my @array = @_;
        my $lat_col="null";
        my $long_col="null";
	my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;
	print "Initiating Pre-GPS Cleanup - removing all data before the first GPS entries.\n";
	for&#40;my $i = 0; $i <= $#&#123;$array&#91;0&#93;&#125; ; $i++&#41;&#123;
		if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Longitude/ &#41; &#123;
                        $long_col = $i;
                &#125;
                if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Latitude/ &#41; &#123;
                        $lat_col = $i;
                &#125;
        &#125;
	my $i=1;
	while &#40;&#40;looks_like_number&#40;$array&#91;$i&#93;&#91;$long_col&#93;&#41; == 0&#41; || &#40;looks_like_number&#40;$array&#91;$i&#93;&#91;$lat_col&#93;&#41; == 0&#41; || &#40;&#40;$array&#91;$i&#93;&#91;$long_col&#93; == 0 &#41; && &#40;$array&#91;$i&#93;&#91;$lat_col&#93; == 0&#41;&#41;&#41; &#123;
		print "line $i, latitude is \"$array&#91;$i&#93;&#91;$lat_col&#93;\", longitude is \"$array&#91;$i&#93;&#91;$long_col&#93;\", erroneous values, tagging line for removal\n";
		$array&#91;$i&#93;&#91;$remove_column&#93;="remove";
		$i++;
	&#125;
	print "Found first line of valid gps data in line $i, latitude is \"$array&#91;$i&#93;&#91;$lat_col&#93;\", longitude is \"$array&#91;$i&#93;&#91;$long_col&#93;\".\n";
	print "Pre-GPS Cleanup Completed Successfully!\n";
	print "\n";
&#125;

sub tag_junk_lines &#123; #looks for non-numerical characters in the data and tags them for removal, needed for successful import into Dlog99
	my @array = @_;
	my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;
	my $remove_line;
	print "Initiating Junk Line Removal Tagging - looking for lines with non-numerical data and tagging for removal during file output.\n";
	for&#40;my $i = 1; $i <= $#array ; $i++&#41;&#123;
		$remove_line="false";
		for&#40;my $j = 0; $j <= $#&#123;$array&#91;0&#93;&#125; ; $j++&#41;&#123;	
			if &#40;&#40; looks_like_number&#40;$array&#91;$i&#93;&#91;$j&#93;&#41; == 0 &#41; && &#40; $array&#91;$i&#93;&#91;$j&#93; ne "" &#41;&#41; &#123;
				print "line $i, element $j, value is $array&#91;$i&#93;&#91;$j&#93;, has invalid characters, removing\n";
				$remove_line="true";	
			&#125;
		&#125;
		if &#40; $remove_line eq "true" &#41; &#123;
			print "Tagging line $i for remove!\n";
			$array&#91;$i&#93;&#91;$remove_column&#93;="remove";
		&#125;
	&#125;
	print "Junk Line Removal Tagging Completed Successfully!\n";
	print "\n";

&#125;

sub clean_gps_for_sats &#123; # processes data removing any lines where GPS Sats are less than 4. Uses the global minimum_sats variable. Ignores lines tagged for removal.
	my @array = @_;
	my $gpssats_column;
	my $lat_column;
	my $long_column;
	my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;
	print "Initiating GPS Cleanup - Removing lat/long when GpsSats is less than minimum of $minimum_sats&#58;\n";
	for&#40;my $i = 0; $i <= $#&#123;$array&#91;0&#93;&#125; ; $i++&#41;&#123;
		if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /GPSSats/ &#41; &#123;
			$gpssats_column="$i";
		&#125;
		if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Latitude/ &#41; &#123;
                        $lat_column="$i";
                &#125;
		if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Longitude/ &#41; &#123;
                        $long_column="$i";
                &#125;
			
	&#125;
	if &#40;!defined $gpssats_column&#41; &#123;
		die "Error - GPS Processing enabled but no GpsSats column present in data.  Try --disable-gps-cleanup option.  Terminating.\n";
	&#125;
	for&#40;my $i = 1; $i <= $#array ; $i++&#41;&#123;
		if &#40;$array&#91;$i&#93;&#91;$remove_column&#93; ne "remove"&#41; &#123;
			if &#40;&#40;defined $array&#91;$i&#93;&#91;$gpssats_column&#93;&#41; && &#40;$array&#91;$i&#93;&#91;$gpssats_column&#93; < $minimum_sats&#41; && &#40;looks_like_number&#40;$array&#91;$i&#93;&#91;$gpssats_column&#93;&#41; > 0&#41;&#41; &#123;
				print "line $i, GpsSats is $array&#91;$i&#93;&#91;$gpssats_column&#93; which is less than the minimum sats of $minimum_sats, setting lat and long to null\n";
				$array&#91;$i&#93;&#91;$lat_column&#93;="";
				$array&#91;$i&#93;&#91;$long_column&#93;="";
			&#125;
		&#125;
	&#125;
	print "GPS Cleanup Completed Successfully!\n";
	print "\n";
&#125;

sub init_pre_gps_data &#123; # finds the first numerical and non-zero lat/long values and then sets all prior lat/long to match.  Basically cleans up gps data until real gps data shows.  Needed because if you set pre-gps lat/long to zero you get weird rounding when going through Dlog99->AEM DA resulting in loss of gps resolution
        my @array = @_;
        my $lat_col="null";
        my $long_col="null";
        my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;
        print "Initiating Pre-GPS initialization - Finds the first real gps lat/long values then sets all prior lat/long to match.\n";
        for&#40;my $i = 0; $i <= $#&#123;$array&#91;0&#93;&#125; ; $i++&#41;&#123;
                if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Longitude/ &#41; &#123;
                        $long_col = $i;
                &#125;
                if &#40; $array&#91;0&#93;&#91;$i&#93; =~ /Latitude/ &#41; &#123;
                        $lat_col = $i;
                &#125;
        &#125;
        my $i=1;
        while &#40;&#40;&#40;&#40;looks_like_number&#40;$array&#91;$i&#93;&#91;$long_col&#93;&#41; == 0&#41; || &#40;looks_like_number&#40;$array&#91;$i&#93;&#91;$lat_col&#93;&#41; == 0&#41; || &#40;&#40;$array&#91;$i&#93;&#91;$long_col&#93; == 0 &#41; && &#40;$array&#91;$i&#93;&#91;$lat_col&#93; == 0&#41;&#41; || &#40;$array&#91;$i&#93;&#91;$remove_column&#93; eq "remove"&#41;&#41;&#41; && &#40;$i <= $#array&#41;&#41; &#123;
                $i++;
        &#125;
	if &#40;$i > $#array&#41; &#123;
		print "Did not find any valid gps data!\n";
	&#125;
	else &#123;
	        print "Found first line of valid gps data in line $i, latitude is \"$array&#91;$i&#93;&#91;$lat_col&#93;\", longitude is \"$array&#91;$i&#93;&#91;$long_col&#93;\".\n";
		print "Setting all lat/long data prior to line $i to \"$array&#91;$i&#93;&#91;$lat_col&#93;,$array&#91;$i&#93;&#91;$long_col&#93;\".\n";
		for&#40;my $x = 1; $x <= $i ; $x++&#41;&#123;
			$array&#91;$x&#93;&#91;$lat_col&#93;=$array&#91;$i&#93;&#91;$lat_col&#93;;
			$array&#91;$x&#93;&#91;$long_col&#93;=$array&#91;$i&#93;&#91;$long_col&#93;;
		&#125;
	&#125;
        print "Pre-GPS Cleanup Completed Successfully!\n";
        print "\n";	
&#125;

sub interpolate_time &#123; #interpolates time data for lines between time changes so Dlog99 doesn't drop data when time values are the same
	my @array=@_;
	print "Initiating Time Interpolation - spaces out time measurements so Dlog99 doesn't drop data points for lines with the same time\n";
	my $remove_column=&#40;$#&#123;$array&#91;0&#93;&#125; + 1&#41;;	
	for&#40;my $i = 1; $i <= $#array ; $i++&#41;&#123;
		if &#40;&#40;$array&#91;$i&#93;&#91;$remove_column&#93; ne "remove"&#41; || &#40;&#40;$i+1&#41; > $#array&#41;&#41; &#123; # ignore lines tagged for removal and stop if we're at the end
			my $current_time=$array&#91;$i&#93;&#91;0&#93;;
			my $next_time_line=$i;
			my $num_lines=0;
			while &#40;&#40;&#40;$current_time == $array&#91;$next_time_line&#93;&#91;0&#93;&#41; || &#40;$array&#91;$next_time_line&#93;&#91;$remove_column&#93; eq "remove"&#41;&#41; && &#40;$next_time_line < $#array&#41;&#41; &#123; 
				$next_time_line++;
				if &#40;$array&#91;$next_time_line&#93;&#91;0&#93; ne "remove"&#41; &#123;
					$num_lines++;		
				&#125;
			&#125;
				
			if &#40;$num_lines > 1&#41; &#123;
				my $diff= &#40;&#40;$array&#91;$next_time_line&#93;&#91;0&#93; - $array&#91;$i&#93;&#91;0&#93;&#41; / $num_lines&#41;;
				my $j;
				if &#40;$next_time_line == $#array&#41; &#123;
					for&#40;$j = &#40;$i+1&#41;; $j <= $next_time_line; $j++&#41; &#123;
                                                $array&#91;$j&#93;&#91;0&#93; = sprintf"%.3f",&#40;$array&#91;$j - 1&#93;&#91;0&#93; + $diff&#41;;
					&#125;
				&#125;
				else &#123;
					for&#40;$j = &#40;$i+1&#41;; $j < $next_time_line; $j++&#41; &#123;
						$array&#91;$j&#93;&#91;0&#93; = sprintf"%.3f",&#40;$array&#91;$j - 1&#93;&#91;0&#93; + $diff&#41;;	
					&#125;
				&#125;
				$i=&#40;$next_time_line - 1&#41;; # step forward in the data to the next time
			&#125;
		&#125;
	&#125;
	print "Time Interpolation Completed Successfully!\n";
	print "\n";
&#125;
	

sub main &#123;
	my @data;
	my $csv = Text&#58;&#58;CSV->new;
	my $input="$file";
	my $output="$output";
	my $num_lines_removed=0;

	print "$script_name - Race Capture Pro CSV to GEMS DA conversion utility v$version\n\n";
	print "\n";
	print "Filtering data for a minimum of $minimum_sats GPS satellites.\n";
	if &#40;$disable_gps_cleanup eq "true"&#41; &#123;
		print "Disabling clean-up of GPS &#40;if your data has bad gps data it's up to you to figure out how to get your track to draw properly in AEM/GEMS Data Analysis&#41;.\n";
	&#125;
	print "\n";
	print "Initiating conversion...\n";
	

	open&#40;INFILE,$input&#41; || die "Can't open file $input";
	open&#40;OUTFILE,">$output"&#41; || die "FATAL - Can't open file $output";

	my $i=0;
	while &#40;<INFILE>&#41; &#123;
		chomp;
		if &#40;$i == 0&#41; &#123;
			push @data, &#91;split /,/&#93;;
		&#125;
		else &#123;  #Now to deal with the data I want to keep
			if&#40;$csv->parse&#40;$_&#41;&#41; &#123;       #checks to see if data exists in $_ and parses it if it does
				$i++;
				my @fields=$csv->fields;  # puts the values from each field in an array
				push @data, &#91; @fields &#93;;
				my $elements=@fields;     #gets the number of elements in the array
			&#125;
		&#125;
	&#125;

	strip_ctrlm&#40;@data&#41;;
        modify_headers&#40;@data&#41;;
        move_time_to_first_column&#40;@data&#41;;
	tag_junk_lines&#40;@data&#41;;
	if &#40;$disable_gps_cleanup ne "true"&#41; &#123;
		clean_gps_for_sats&#40;@data&#41;;
		tag_pre_gps_data&#40;@data&#41;;
	&#125;
	else &#123;
		init_pre_gps_data&#40;@data&#41;;	
	&#125;
        zero_start_time&#40;@data&#41;;
	fill_in_the_blanks&#40;@data&#41;;
	interpolate_time&#40;@data&#41;;
	convert_gpsdegrees_to_radians&#40;@data&#41;;

	my $remove_column=&#40;$#&#123;$data&#91;0&#93;&#125; + 1&#41;;
        for&#40;my $i = 0; $i <= $#data; $i++&#41;&#123;
		if &#40; $data&#91;$i&#93;&#91;$remove_column&#93; ne "remove" &#41;&#123;  #only print lines not tagged with "remove" in last column
			for&#40;my $j = 0; $j <= $#&#123;$data&#91;0&#93;&#125; ; $j++&#41;&#123;
				if &#40;$#&#123;$data&#91;0&#93;&#125; == $j&#41;&#123;
					print OUTFILE "$data&#91;$i&#93;&#91;$j&#93;";
				&#125;
				else &#123;
					print OUTFILE "$data&#91;$i&#93;&#91;$j&#93;,";
				&#125;
			&#125;
		
			print OUTFILE "\r";
		&#125;
        &#125;

	close INFILE;
	close OUTFILE;

	print "\n";
	print "Success!  Input file $file has been converted to $output.\n"; 
&#125;

main;

JMcDonough
Posts: 70
Joined: Tue Mar 10, 2015 3:31 pm
Location: Detroit, MI

Post by JMcDonough »

First off, thanks for all those who put time and effort into making this script.

I copied and pasted Copper280z's script into a new file called rccsv2gems_copper280z.pl then ran the following script at the windows command line:

Code: Select all

perl rccsv2gems_copper280z.pl RC_0.csv NewFile1
As someone else mentioned above, I had to install Text::CSV first using the following command:

Code: Select all

cpan Text&#58;&#58;CSV
(this was after running cpan App::cpanminus during installation, more info here: http://www.cpan.org/modules/INSTALL.html)

The script gets hung up when trying to open the input file:

Code: Select all

open&#40;INFILE,$input&#41; || die "Can't open file $input"; 
I'm running this from the windows command prompt after changing the directory to where the script and data files reside (same folder).

I tried the file with and without .csv at the end. No change.

Any suggestions? This is my first time using Perl, so perhaps I'm missing something completely obvious.

TIA!
Josh

brentp
Site Admin
Posts: 6274
Joined: Wed Jan 24, 2007 6:36 am

Post by brentp »

Hi all,

Thanks for looking into the current script. Yes, the column names have changed for the V2 firmware and now the UTC time in milliseconds are now present on every log sample.

Could someone share their log file so I can help look at this? Also thinking of a python version of the script so we can ultimately get it integrated with the app.
Brent Picasso
CEO and Founder, Autosport Labs
Facebook | Twitter

Copper280z
Posts: 30
Joined: Sun Mar 15, 2015 3:03 am
Location: United States

Post by Copper280z »

Josh,

The untouched racecapture logs are named RC_x.LOG, not .CSV, this is important. Try cding to your directory and just typing RC_0.log, notepad should open the (gibberish looking) log file, if it doesn't then make sure you have the name and extension correct. Once you've got it opening like that, try running the script again with the filename you're sure is right.

I'm not sure if I did something different (also very new to perl), but I didn't have to install text::CSV and I execute the script without the 'perl' in the beginning.

My rcp is currently riding shotgun, so the data isn't the greatest, but I could probably send you something that's just got some street driving later in the week.

JMcDonough
Posts: 70
Joined: Tue Mar 10, 2015 3:31 pm
Location: Detroit, MI

Post by JMcDonough »

Copper280z wrote:Josh,

The untouched racecapture logs are named RC_x.LOG, not .CSV, this is important. Try cding to your directory and just typing RC_0.log, notepad should open the (gibberish looking) log file, if it doesn't then make sure you have the name and extension correct. Once you've got it opening like that, try running the script again with the filename you're sure is right.

I'm not sure if I did something different (also very new to perl), but I didn't have to install text::CSV and I execute the script without the 'perl' in the beginning.

My rcp is currently riding shotgun, so the data isn't the greatest, but I could probably send you something that's just got some street driving later in the week.

Typing RC_0.log instead of .csv did the trick. I still have to use perl at the beginning of the command for whatever reason, but it works! Thanks for the help.

For anyone else going through this, to open the output of the perl scrip in Dlog99 you have to do a File->Import to read the output of the perl script.

Happy to finally be looking at the data.
Josh

benk
Posts: 1
Joined: Mon Mar 30, 2015 11:44 pm

Post by benk »

has anyone had luck using the maths channel in AEM data after importing with DLog?

Post Reply