Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

@vg071 regarding your samples, what kind of activities do you film?

Sightseer
Posts: 8
Re: Extracting the metadata in a useful format

Wingsuit skydives primarily 

Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format
Oh that'd be great. I'll get in touch privately
Tourist
Posts: 6
Re: Extracting the metadata in a useful format

Using the batch file I can see the gyro info is available in a GoPro Hero5 file (which I need to get pitch and roll info as gauges in Dashware). However, Dashware doesn´t seem to extract/recognize this from a GoPro file format. Will there be an update in Dashware?

 

Any link to an EASY to understand tutorail on how to create a new GoPro data profile (cause I get lost in reading the ones I find).

 

thanks

Tourist
Posts: 4
Betreff: Extracting the metadata in a useful format

Thank you for Your work, but as i have nearly zero experience with computer sciences, I'd like to ask how I should exactly use those downloadable parser scripts... you say it works fine by "just dropping the video files on the GPMD2CSV.bat " (htis thing does nothing when I drop something on it)...

Also I used Racerender to export a cvs file (like many advised me because there is htis function there) but when I open those files, I can see only the coodrinates and the altitude and a speed column, but I'd like to have the acceleration (the accelerometer column)... does anyone know where I can get that from? (I have a mac so I cannot use Dashware)...

 

Thanks you all!

 

Nomad
Posts: 1,110
Betreff: Extracting the metadata in a useful format

@bluecreek74714 Are you on Windows? I will have something ready for other platforms in a few days.

Nomad
Posts: 1,110
Betreff: Extracting the metadata in a useful format
[ Edited ]

Alright, with the new tool you can extract a simplified GPX file for Virb Edit.

 

https://tailorandwayne.com/gopro-telemetry-extractor/

 

It's not the optimal solution yet, but it's what I can do with the limited time I have.

Let me know how that goes.

Tourist
Posts: 4
Betreff: Extracting the metadata in a useful format
[ Edited ]

I'm having a few issues with the online tool.  Sometimes I get error processing file when I try to save it and need to reanalyze.  Sometimes I also can download the .gpx file but its blank.  I did manage to get a couple videos with GPX data but even then, its a much smaller filesize compared to when using GPMD2SCV

 

I have linked my 2 files below.  GPMD2CSV.GPX (This is the GPX I get using GPMD2SCV (564KB), which of course does not work with Verb edit) and VirbEdit.GPX (This is the GPX I get using the Online Beta tool and selecting VIRBedit (36kb)), for the same video.  Maybe this can help troubleshoot, or perhaps its coded correctly and the online tool is causing the issue? (I had to try several times for some video to even get that 36KB .gpx file.)  Is there a way to use GPMD2CSV offline tool and Save VIRBedit compatible .gpx files?


Thanks for all your effort with this.  Virb Edit is clearly the easiest to use software with gague editing out there (And its free).  If we can use our GoPro footage in verb Edit it will put great shame to GoPro and their inability to create native software that supports GPS gagues (Quik doesnt work with HVEC) and also finally give us an easy way to make videos with gagues. (Its shocking to me that GPS has been a part of GoPros for 4 years now and they still dont have a native solution to thi)

 

VirbEdit.GPX

GPMD2CSV.GPX 

Nomad
Posts: 1,110
Betreff: Extracting the metadata in a useful format

@loyaldusk1743 The smaller file size is expected. Virb does not handle milliseconds properly, so what I have done to create GPX files compatible with Virb easily is to simplify the data to just one sample per second. Do those files work decently in Virb? There are better approaches that would probably allow us to keep both accuracy and speed, but I have not had the time to develop them yet.

Regarding the errors you get when processing the files, would you be able to send me a sample file that fails, privately? I have been fixing errors with large files recently, it could be that. So maybe try again or try to send me something I can test (or screenshot the error and the exact time you got it). Thanks

Nomad
Posts: 107
Betreff: Extracting the metadata in a useful format

@kajuna Could I ask for your help? I'm trying to leverage some of the code from https://github.com/JuanIrache/gopro-utils. I downloaded the code and tried to build some of the modules but get "can't find package" errors. The code seems to be dependent on other code. For example, telemetry.go has an import statement for "github.com/paulmach/go.geo". I went there on github but it says "Geometry/geography library in Go, DEPRECATED, use -> https://github.com/paulmach/orb". I downloaded that but the directory links to the modules don't seem to match the import statements in the gopro modules. Could you please tell me where to get all the necessary dependent modules and how to install them so they can be found by the gopro-utils modules? Thanks in advance.

Nomad
Posts: 1,110
Betreff: Extracting the metadata in a useful format

@dzeek Sure, the initial post of this thread covers how I built my setup. I am not a golang developer and had almost no idea what I was doing, so it's very chaotic and probably not the best approach. Also, many thigs have changed since then. My current setup is:

%USERPROFILE%\go\src\github.com\

 

There I have a folder for each of the github accounts I use dependencies from: juanirache, stilldavid, paulmach and so on. Within each of those folders are the repos I depend on, and my own repo.

%USERPROFILE%\go\src\github.com\paulmach\go.geo

I currently use the VSCode text editor for other stuff and some of its plugins manages the imports the repo needs automatically. Not sure if that'd be easier to setup than doing it manually from my instructions. I didn't know Github either, back then, so it's pretty embarrassing to read my post now. But I guess that might make it understandable to noobs like I was.

I still use whatever dependencies are listed in my repo, hadn't noticed that one was deprecated.

Nomad
Posts: 107
Betreff: Extracting the metadata in a useful format

@kajuna Thanks. I forgot about your initial post since I read it so long ago! I'm a newbie at this also. I'll do my setup like that and go from there. Thanks!

Sightseer
Posts: 4
Betreff: Extracting the metadata in a useful format

Have the 7 or 360 improved the GPS or IRS sensors from the 5 at all?

 

~400 Hz 3-axis gyro readings
~200 Hz 3-axis accelerometer readings
~18 Hz GPS position (lat/lon/alt/spd)
1 Hz GPS timestamps
1 Hz GPS accuracy (cm) and fix (2d/3d)
1 Hz temperature of camera

 

Also anyway to find angular and acceleration rate and bias stability specs at all?

Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format
Sightseer
Posts: 4
Re: Extracting the metadata in a useful format

Great! So the 5 actually has a better accelerometer than the 6 and 7 while the fusion has the best by far and a magnetometer to boot. 

Sightseer
Posts: 8
Re: Extracting the metadata in a useful format

Love the new site - you did so much work! It's impressive.

 

Question for you: just tried importing a clip in and here's the negative numbers data I'm seeing when I click on 'view map':

https://tailorandwayne.com/gopro-telemetry-extractor/#

 

What is the reasoning for the Go Pro 7 Black showing negative data? And is there a way to change this?

 

(I'm about to import this into Virb Edit)

 

 

negative numbers.PNG
Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

Thanks, glad you like it :).

The negative height occurs because the raw height data is not based on sea level. Negative vertical speed is normal, it just means you were going down (quite fast apparently, although the peak speeds often are exaggerated by small inaccuracies).

I would like to adjust the heights to sea level at some point, but for now my priority for the little time I can spare would be refactoring the tools in JavaScript. I'm not a Golang developer, so every new feature is painful to add and my implementations are not ideal.

 

So if anyone here has experience working with binary data in JavaScript, their help is very much welcome. Once we solve this, new features should be much easier to implement.

Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format
[ Edited ]

Here's my progress in Javascript.

- Extracting the data from the mp4: https://github.com/JuanIrache/gpmf-extract

- Parsing the exctracted data: https://github.com/JuanIrache/gopro-telemetry

This will give us access to many more data types. Things like face detection, highlights, image uniformity and many more.

Sightseer
Posts: 2
Re: Extracting the metadata in a useful format
Hi kajuna! Great work with the gopro meta extract tools!
I've tried your gpmf-exrtact one.
I have very big GoPro Hero 6 Black files, around 4GB each.

What's the best way to pass these to the gpmf-extract tool? It seems to expect a Buffer, but there's no way NodeJS can load such a big file into memory (i.e. there's no way you can load it at once via fs.readFile or readFileSync).
When I try, I get the typical NodeJS error
"RangeError: "size" argument must not be larger than 2147483647"

I also tried the web app, and it works, so I suspect the FileReader approach (for browser) does not suffer from the same issue as NodeJS?
Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format
Yes. The browser approach will work, but it is slow. FileReader is not available natively in Node, but maybe you can try the NPM modules that claim to reproduce it. It should also be possible to analyse the file in chunks like the browser does, but I would not know how to approach that just now.

Please let us know if you make any progress. I might have to adapt the extractor to solve this.

By the way, if you use the output of gpmf-extract in the gopro-telemetry package, keep in mind that it is in a very very early stage
Sightseer
Posts: 2
Re: Extracting the metadata in a useful format
Thanks for the super quick response!

Yes, I might try the browser way for now.

Basically I've been having troubles with one of my GoPro videos: it seems to have some sort of bad/corrupted data in the gpmd metadata.
I generally use RaceRender3 to extract the data automatically and it has worked fine with other files. Just this one seems problematic.

That's why I searched for alternatives and stumbled into this thread.
Interestingly, your web app "GoPro Telemetry Extractor" also gives me an error with this particular file (it works perfectly with others).

So I thought that the data was corrupted, even though GoPro Quik is able to read it and show the gauges correctly... so it can't be 100% bad. Perhaps it contains and edge case that your script or RaceRender3 extraction script can't handle.

I've also tried the original gopro-utils and GPMD2CSV, same result.

If of any interest I can send on the extracted `.bin`
Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

The web app uses the same tools as GPMD2CSV in the background (for now). You might have better luck debugging with the new JavaScript tools, but they are under construction, so you might have to play with the code: https://github.com/JuanIrache/gopro-telemetry

Sightseer
Posts: 4
Re: Extracting the metadata in a useful format

What a great thread! 

 

Has anyone found a good applicatoin for visualizing 3D movement including rolls, flips, spins etc. Say for snowboarding, Redbull air racing or aerobatics?

Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

@jw86 I don't think there's anything that does that out of the box yet. It's not easy as GPS altitude is not very accurate, and the gyroscope is prone to drift (and calculating absolute positions is really complicated). I wanted to give it a try a while ago, but there's just too many things I want to try.

Sightseer
Posts: 4
Re: Extracting the metadata in a useful format
Ha! Indeed, it would be really cool wouldn't it, especially if you can use the accelerometer and magnetometer to dampen and interpolate the GPS data :)
Nomad
Posts: 1,110
Betreff: Extracting the metadata in a useful format

@loyaldusk1743 Would you mind sharing that VirbEdit.GPX file with me again? I can implement the format now, but the link has expired.

 

Thanks!

Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format
[ Edited ]

Hey guys,

 

Would anyone that uses Virb Edit for their gauges mind testing my new extraction format? I have switched the tools that run the site in the background and I believe the data should work better in Virb now: https://tailorandwayne.com/gopro-telemetry-extractor/

Also, you can now download .mgjson files and import them directly into After Effects, if you'd like to design your own gauges.

More sweet updates coming soon :)

Explorer
Posts: 13,046
Re: Extracting the metadata in a useful format

@kajuna 

You have done some amazing work!  Have you reached out to GoPro with this?

Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

Hi @danielr15 , Not by any official channel. What do you suggest?

Explorer
Posts: 13,046
Re: Extracting the metadata in a useful format
I think you should try the following:
https://gopro.com/en/us/connect (Fill out the forms under Partner With Us>Software and also Contact Public Relations

Also https://gopro.com/en/us/feedback under Product and Web

I sent feedback on your behalf, but you should do the same.

The site you created is really well done. I'd love to see more support documentation or perhaps you could do a walk through video of the different functions and how to utilize and interpret the data.
Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

Thanks @danielr15 ! Will give it a try.

I wish I had more time to write documentation, tutorials... It's in my to-do list, like publishing an offline premium version of the app with extra features.

Hiker
Posts: 47
Re: Extracting the metadata in a useful format

Hi

Recently I have many problems regarding GPS data in videos taken with GP7 Black. It has been already replaced by GoPro but still the same problems: camera freezing, some videos are lost and GPS data is corrupted. The GP7 can lose the GPS lock in the open space free of obstacles. Attached the screen shot from trassy.pl/syncloc 

What I actually do because of that is I record the GPS track with 2 smartphones(with Strava and Endomondo) - if I do not have GPS data from GP7, which is the most comfortable, I can synchronize with GPX from mobile application. And, somehow smartphones do not lose the GPS lock (Samsung and iPhone).

 

cheers

Eligiusz

Zrzut ekranu 2019-07-28 o 20.18.49-min.png
Explorer
Posts: 13,046
Re: Extracting the metadata in a useful format
Because smartphones also use cell towers, which the camera cannot. Download UAV forecast and turn off Galileo and GLONASS. You will be able to check to see how many satellites are available, estimated lock, and kp index (for atmospheric interference). You want 9+ satellites for accuracy and a kp less than 4.
Hiker
Posts: 47
Re: Extracting the metadata in a useful format

I do not agree that this is the cause of GPS issues for GoPro 7 Black (in that open space you could not have better visibility of satellites but GP7 could not get a lock for an hour. For the rest of that 3 hour shooting the GPS was horible as well. The other day with more cloudy conditions I followed the same route with the same GP7 and the collected GPS data was OK - so it can capture proper data without cell towers assistance when it works).

But I do not want to continue and spam the discussion in that thread. If anybody is interested I can open a new thread.

cheers

Eligiusz

Explorer
Posts: 13,046
Re: Extracting the metadata in a useful format
But how many satellites were overhead and what was the kp index when you couldn't get good GPS. I've been flying drones for years and know for a fact that open field and clear sky only tells you a fraction of what you need to know for estimating good GPS signal. Phones will always get better GPS because they are assisted by cell towers.
Hiker
Posts: 31
Re: Extracting the metadata in a useful format
Hello

[Hero6 black]

I have used the free version for one of my files

https://tailorandwayne.com/gopro-telemetry-extractor/

After waiting a while , I'have choosen the CSV file

My trip was in Luxembourg (between France Belgium and Germany)

And there are 5 columns Of interest
1 is the time (on row per second)
Perfect for position (for speed I hope for 10 to 18 Hz)

One column is home longitude
One is home latitude

One is GPS longitude
One is GPS latitude.

I have imported the file I Google maps, and the coordinate are in New Zéland ( a beautiful country but ..)

Does anyone have idea about what is going on ?

The coordinates looks strange ...

Regards for your ansmers
Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format
Hey @pd45pd45 , I'm a bit confused. Where are you seeing those columns? I don't think any GoPro cameras record home coordinates. You can send me the video file privately for analysis. Or the LVR file if you keep it.
Hiker
Posts: 31
Re: Extracting the metadata in a useful format
Thank you kajuna, ii will do it , probably next week end ! It is very kind !
I see the column in the resulting CSV file !
Hiker
Posts: 31
Re: Extracting the metadata in a useful format
Hello Kajuna !

I owe thousands of apologizes to you !
Hiker
Posts: 31
Re: Extracting the metadata in a useful format
After double check , my video files was WITHOUT GPS data (something went wrong)

Thus your extractor must have extract default data ...

Anyway ! I have checked first (by Quick tm) the presence of GPS data on a different video file
And it works perfectly !

My car was on the road drawed on the map !

For your information my final ambition is to use Dashware for adding beautiful gauges on my car trips .
And using extra data provided by a blue tooth On Board Diagnostic device which send data to an Android app (mine is In Car Doc Pro) that will provide rpm, wheel speed, and phone GPS data.

As you can guess, the main issue will be the synchronisation of the CSv file.

(The video extracted file will be of course the reference!)

Thank you again for your work !
Nomad
Posts: 1,110
Re: Extracting the metadata in a useful format

@anubisher6blck Yes, what you were seeing was sample data in the GPS viewer. However, if you want to export a CSV, you'll get better results directly from the Telemetry Extractor instead of the GPS Viewer (the viewer supports other formats like DJI drone logs, so the information you get is not that GoPro-specific).

Hope you create great trip videos. Feel free to share them in the "Post your videos" section if you want :)