INACCURATE MOTION DATA (due to wrong aspect ratio)

[kanyisignvrse]

Hello @AmmarkoV . The code from this colab notebook you shared for this repo https://colab.research.google.com/github/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/mocapnet4.ipynb#scrollTo=05pHqR5D57jG is not outputting the expected motion. I tried to run it with my own video but the BVH file generated has inaccurate motion. What could be the issue?

Here is the video:

ASL.webm

Here is how the motion looks like:

Screen.Recording.2025-07-29.144059.mp4

[AmmarkoV]

Dear kanyisignvrse

This is the output of the method on the video you provided me with :
http://ammar.gr/mocapnet/mnet4/kanyisignvrse.mp4

As mentioned in the other thread
The main problem is the aspect ratio, your video has 608x1080 dimensions
The MocapNET neural network has been trained on 1920x1080 dimensions :)
It might sound odd but this is very important for the 2D point distributions !

I "padded" your video using the following ffmpeg command:

ffmpeg -i input.webm -vf "scale=w=iw:h=ih:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2" -c:a copy output.mp4 

And then run it through the network using the following command:

python3 mediapipeHolisticWebcamMocapNET.py --from output.mp4 --all --save

When rendering the BVH file with blender it looks like this:
http://ammar.gr/mocapnet/mnet4/kanyisignvrse_blender.mkv

You see the body bending slighlty forward, indicating that the intrinsics calibration
of the camera recording the subject is also slightly off compared to the default

by supplying the --calib parameter and giving something like https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/dataset/logitech_c920_720p.calib it might behave better,
The default GoPRO that the network is trained on and works best is https://github.com/FORTH-ModelBasedTracker/MocapNET/blob/mnet4/dataset/gopro1080p.calib

You can also get slightly better results if you tweak the hcd settings ( --ik 0.001 99 99 ) ,
this is with vanilla default settings.

Last but not least, the legs are not visible ( but the BVH has legs )
so ideally a video with no occlusions would also help even more.!

Looking forward to your response!

[kanyisignvrse]

Hello @AmmarkoV , thankyou for your quick response. I tried your suggestions on the video i.e padding to 1920 x 1080 and passed the --calib argument the gopro calibration parameters but maintained the default HCD settings. The bvh output has vastly improved but the motion of the fingers seem random.
Here is how it looks on Blender:

Screen.Recording.2025-07-30.105023.mp4

[AmmarkoV]

Hello, trying a few different focal lengths I suppose
python3 mediapipeHolisticWebcamMocapNET.py --from output.mp4 --scale 0.9 --focalLength 500 500 --all --ik 0.001 200 200
is sort of an approximation of the output you can get.
However in any case the RGB video is very close to the camera (the NN is trained for 1.5m - 3.5m) so the video maybe a little out of the training spec and as a result the output is not so good!

[kanyisignvrse]

H

Hello, trying a few different focal lengths I suppose python3 mediapipeHolisticWebcamMocapNET.py --from output.mp4 --scale 0.9 --focalLength 500 500 --all --ik 0.001 200 200 is sort of an approximation of the output you can get. However in any case the RGB video is very close to the camera (the NN is trained for 1.5m - 3.5m) so the video maybe a little out of the training spec and as a result the output is not so good!

Hello @AmmarkoV I've tried a few different focal lengths and it is improving. Let me try test it out with videos that are within the training spec.