React Native MediaPipe Pose

React Native module for real-time pose detection using Google's MediaPipe BlazePose

Production-ready pose detection with GPU acceleration and automatic hardware optimization

Installation • Quick Start • API Reference • Performance • Examples

🚀 Features

🎯 Real-time Pose Detection

MediaPipe BlazePose - Google's production-grade pose detection model
33 Pose Landmarks - Full body keypoint detection with 3D coordinates
GPU Acceleration - Automatic hardware acceleration with Metal framework
High Accuracy - 0.5 confidence threshold for precise detection
Live Streaming - Optimized for real-time video processing

⚡ Performance Optimization

Automatic GPU/CPU Selection - Dynamic delegate switching for maximum accuracy
Data Streaming Control - Optional pose data transmission (disabled by default for max performance)
Throttling System - Configurable data throttling to reduce bridge overhead
FPS Optimization - Smart FPS reporting with change threshold detection
Memory Efficient - Minimal memory footprint with efficient processing pipeline

🔧 Advanced Configuration

Detailed Logging Control - Production-ready logging system (disabled by default)
Performance Monitoring - Real-time processing time and GPU status tracking
Device Capability Detection - Automatic hardware tier classification
Auto FPS Adjustment - Dynamic frame rate optimization based on device performance
Error Handling - Comprehensive error reporting and recovery mechanisms

📱 Camera Management

Dual Camera Support - Front/back camera switching
Permission Handling - Automatic iOS camera permission management
Custom Styling - Fully customizable camera view
Orientation Support - Proper handling of device orientation changes

🚧 Platform Support

iOS - Fully implemented with MediaPipe BlazePose
Android - In development (currently under active development)
Web - Basic camera support (pose detection coming soon)

📦 Installation

Prerequisites

React Native >= 0.72
Expo SDK >= 53
iOS >= 12.0 (for iOS development)
Xcode >= 14 (for iOS development)

Install Package

npm install react-native-mediapipe-pose
# or
yarn add react-native-mediapipe-pose

iOS Setup

⚠️ CRITICAL: Manual Configuration Required

Your app will crash without these steps. The package cannot automatically configure these settings for security reasons.

1. Camera Permissions (REQUIRED)

Add camera permission to your app's ios/YourAppName/Info.plist:

<key>NSCameraUsageDescription</key>
<string>This app needs camera access for pose detection</string>

Alternative descriptions you can use:

<!-- For fitness apps -->
<string>Camera access is required for real-time pose tracking and exercise analysis</string>

<!-- For AR/entertainment apps -->
<string>Camera access enables pose detection for interactive experiences</string>

<!-- Generic -->
<string>This app uses the camera to detect and analyze human poses in real-time</string>

2. Minimum iOS Version (REQUIRED)

Ensure your app targets iOS 13.0+ (MediaPipe requirement):

In ios/YourApp.xcodeproj:

Select your project in Xcode
Go to Build Settings → Deployment Target
Set iOS Deployment Target to 13.0 or higher

In package.json:

{
  "engines": {
    "ios": ">=13.0"
  }
}

3. iOS Dependencies (AUTO-INSTALLED)

iOS dependencies are automatically installed when you build your app. However, if you encounter issues, you can manually run:

cd ios
pod install
cd ..

Note: Modern React Native (0.72+) and Expo automatically handle pod installation during the build process.

4. Performance Optimization (RECOMMENDED)

For better camera performance, add to ios/YourAppName/Info.plist:

<key>CADisableMinimumFrameDurationOnPhone</key>
<true/>

5. Orientation Support (OPTIONAL)

If you want to support different orientations, configure in ios/YourAppName/Info.plist:

<!-- For iPhone - Portrait only (recommended for pose detection) -->
<key>UISupportedInterfaceOrientations</key>
<array>
  <string>UIInterfaceOrientationPortrait</string>
</array>

<!-- For iPad - All orientations -->
<key>UISupportedInterfaceOrientations~ipad</key>
<array>
  <string>UIInterfaceOrientationPortrait</string>
  <string>UIInterfaceOrientationPortraitUpsideDown</string>
  <string>UIInterfaceOrientationLandscapeLeft</string>
  <string>UIInterfaceOrientationLandscapeRight</string>
</array>

6. MediaPipe Model (AUTO-INCLUDED)

The MediaPipe pose detection model (pose_landmarker_full.task) is automatically included with the package. No manual download required.

7. Complete Info.plist Example

Your ios/YourAppName/Info.plist should include at minimum:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <!-- Your existing keys... -->

  <!-- REQUIRED: Camera permission -->
  <key>NSCameraUsageDescription</key>
  <string>This app needs camera access for pose detection</string>

  <!-- REQUIRED: Minimum iOS version -->
  <key>LSMinimumSystemVersion</key>
  <string>13.0</string>

  <!-- RECOMMENDED: Performance optimization -->
  <key>CADisableMinimumFrameDurationOnPhone</key>
  <true/>

  <!-- RECOMMENDED: Portrait orientation for best pose detection -->
  <key>UISupportedInterfaceOrientations</key>
  <array>
    <string>UIInterfaceOrientationPortrait</string>
  </array>

  <!-- Your other existing keys... -->
</dict>
</plist>

8. Verification Steps

After configuration, verify your setup:

Build and run your app on a physical iOS device
Check camera permission - app should prompt for camera access
Test pose detection - camera should display with pose overlay
Check console logs - look for MediaPipe initialization messages

9. Common Issues & Solutions

App crashes immediately:

✅ Check that NSCameraUsageDescription is added to Info.plist
✅ Ensure iOS deployment target is 13.0+
✅ Try cleaning build: npx expo run:ios --clear
✅ If issues persist, manually run cd ios && pod install

Camera permission denied:

✅ Add proper NSCameraUsageDescription
✅ Manually enable camera permission in iOS Settings → Privacy → Camera

Pose detection not working:

✅ Test on physical device (camera not available in simulator)
✅ Ensure good lighting conditions
✅ Check that enablePoseDetection={true}

Performance issues:

✅ Add CADisableMinimumFrameDurationOnPhone to Info.plist
✅ Set enablePoseDataStreaming={false} for production
✅ Use autoAdjustFPS={true} for device optimization

Android Setup

Note: Android support is currently under development and not yet available.

Android implementation is in active development. Follow our progress:

MediaPipe Android integration
Camera2 API implementation
GPU acceleration support
Performance optimization

Expected Android support in upcoming releases.

🚀 Quick Start

Basic Implementation

import React, { useState } from 'react';
import { View, StyleSheet, Alert } from 'react-native';
import ReactNativeMediapipePose, {
  ReactNativeMediapipePoseView,
  PoseDetectionResult,
  CameraType,
  DeviceCapability,
  FrameProcessingInfo,
} from 'react-native-mediapipe-pose';

export default function App() {
  const [cameraType, setCameraType] = useState<CameraType>('front');
  const [isPoseDetectionEnabled, setIsPoseDetectionEnabled] = useState(false);
  const [fps, setFps] = useState<number>(0);

  // Request camera permissions
  const requestPermissions = async () => {
    try {
      const granted = await ReactNativeMediapipePose.requestCameraPermissions();
      if (!granted) {
        Alert.alert(
          'Camera Permission Required',
          'This app requires camera access to detect poses.'
        );
      }
    } catch (error) {
      Alert.alert('Error', 'Failed to request camera permissions');
    }
  };

  const handlePoseDetected = (event: { nativeEvent: PoseDetectionResult }) => {
    const { landmarks, processingTime, confidence } = event.nativeEvent;
    console.log(
      `Detected ${landmarks.length} landmarks in ${processingTime}ms`
    );
  };

  const handleFrameProcessed = (event: {
    nativeEvent: FrameProcessingInfo;
  }) => {
    setFps(Math.round(event.nativeEvent.fps));
  };

  const handleDeviceCapability = (event: { nativeEvent: DeviceCapability }) => {
    const { deviceTier, recommendedFPS } = event.nativeEvent;
    console.log(`Device: ${deviceTier}, Recommended FPS: ${recommendedFPS}`);
  };

  const handleGPUStatus = (event: { nativeEvent: any }) => {
    const { isUsingGPU, processingUnit } = event.nativeEvent;
    console.log(
      `GPU Acceleration: ${isUsingGPU ? 'Enabled' : 'Disabled'} (${processingUnit})`
    );
  };

  const switchCamera = () => {
    // Enhanced camera switching with full refresh to prevent coordinate system issues
    const wasEnabled = isPoseDetectionEnabled;
    setIsPoseDetectionEnabled(false); // Temporarily disable

    // Switch camera type
    setCameraType((current) => (current === 'back' ? 'front' : 'back'));

    // Re-enable pose detection after brief delay for proper initialization
    setTimeout(() => {
      if (wasEnabled) {
        setIsPoseDetectionEnabled(true);
      }
    }, 100);
  };

  React.useEffect(() => {
    requestPermissions();
  }, []);

  return (
    <View style={styles.container}>
      <ReactNativeMediapipePoseView
        key={`camera-${cameraType}`} // Force re-render on camera switch
        style={styles.camera}
        cameraType={cameraType}
        enablePoseDetection={isPoseDetectionEnabled}
        enablePoseDataStreaming={true} // Enable to receive pose data
        targetFPS={30}
        autoAdjustFPS={true}
        onCameraReady={(event) =>
          console.log('Camera ready:', event.nativeEvent.ready)
        }
        onError={(event) =>
          Alert.alert('Camera Error', event.nativeEvent.error)
        }
        onPoseDetected={handlePoseDetected}
        onFrameProcessed={handleFrameProcessed}
        onDeviceCapability={handleDeviceCapability}
        onGPUStatus={handleGPUStatus}
      />

      {/* Simple UI overlay showing FPS */}
      <View style={styles.overlay}>
        <Text style={styles.fpsText}>FPS: {fps}</Text>
        <TouchableOpacity style={styles.button} onPress={switchCamera}>
          <Text style={styles.buttonText}>Switch Camera</Text>
        </TouchableOpacity>
        <TouchableOpacity
          style={[styles.button, isPoseDetectionEnabled && styles.buttonActive]}
          onPress={() => setIsPoseDetectionEnabled(!isPoseDetectionEnabled)}
        >
          <Text style={styles.buttonText}>
            {isPoseDetectionEnabled ? 'Stop Pose' : 'Start Pose'}
          </Text>
        </TouchableOpacity>
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
  camera: {
    flex: 1,
  },
  overlay: {
    position: 'absolute',
    top: 50,
    left: 20,
    right: 20,
    flexDirection: 'row',
    justifyContent: 'space-between',
    alignItems: 'center',
  },
  fpsText: {
    color: 'white',
    fontSize: 16,
    fontWeight: 'bold',
  },
  button: {
    backgroundColor: 'rgba(255, 255, 255, 0.2)',
    padding: 10,
    borderRadius: 8,
  },
  buttonActive: {
    backgroundColor: 'rgba(76, 175, 80, 0.8)',
  },
  buttonText: {
    color: 'white',
    fontSize: 14,
    fontWeight: '600',
  },
});

Performance Optimized Setup

import React, { useState, useEffect } from 'react';
import { View, Text, TouchableOpacity, StyleSheet } from 'react-native';
import ReactNativeMediapipePose, {
  ReactNativeMediapipePoseView,
  DeviceCapability,
  FrameProcessingInfo,
} from 'react-native-mediapipe-pose';

export default function OptimizedPoseDetection() {
  const [deviceCapability, setDeviceCapability] =
    useState<DeviceCapability | null>(null);
  const [fps, setFps] = useState<number>(0);
  const [targetFPS, setTargetFPS] = useState<number>(30);
  const [autoAdjustFPS, setAutoAdjustFPS] = useState<boolean>(true);
  const [enablePoseDataStreaming, setEnablePoseDataStreaming] =
    useState<boolean>(false);
  const [showPerformanceControls, setShowPerformanceControls] =
    useState<boolean>(false);

  const handleDeviceCapability = (event: { nativeEvent: DeviceCapability }) => {
    const capability = event.nativeEvent;
    setDeviceCapability(capability);
    setTargetFPS(capability.recommendedFPS); // Use device-recommended FPS
  };

  const handleFrameProcessed = (event: {
    nativeEvent: FrameProcessingInfo;
  }) => {
    const { fps: currentFps, autoAdjusted, newTargetFPS } = event.nativeEvent;
    setFps(Math.round(currentFps));

    if (autoAdjusted && newTargetFPS) {
      setTargetFPS(newTargetFPS);
      console.log(`Auto-adjusted FPS to ${newTargetFPS}`);
    }
  };

  const handleGPUStatus = (event: { nativeEvent: any }) => {
    const { isUsingGPU, delegate, deviceTier } = event.nativeEvent;
    console.log(
      `GPU: ${isUsingGPU}, Delegate: ${delegate}, Device: ${deviceTier}`
    );
  };

  return (
    <View style={styles.container}>
      <ReactNativeMediapipePoseView
        key={`camera-optimized-${Date.now()}`} // Ensure clean re-renders
        style={styles.camera}
        enablePoseDetection={true}
        // Performance optimizations (recommended for production)
        enablePoseDataStreaming={enablePoseDataStreaming} // Toggle data streaming
        enableDetailedLogs={false} // Disable for production
        poseDataThrottleMs={100} // Throttle data updates
        fpsChangeThreshold={2.0} // Only report significant FPS changes
        fpsReportThrottleMs={500} // Throttle FPS reports
        // Auto performance adjustment
        autoAdjustFPS={autoAdjustFPS} // Enable automatic FPS optimization
        targetFPS={targetFPS} // Dynamic target based on device
        onFrameProcessed={handleFrameProcessed}
        onDeviceCapability={handleDeviceCapability}
        onGPUStatus={handleGPUStatus}
        onPoseDetected={(event) => {
          if (enablePoseDataStreaming) {
            const { landmarks, processingTime } = event.nativeEvent;
            console.log(
              `Pose: ${landmarks.length} landmarks, ${processingTime}ms`
            );
          }
        }}
      />

      {/* Performance Controls Overlay */}
      <View style={styles.performanceOverlay}>
        <Text style={styles.performanceText}>
          FPS: {fps} | Target: {targetFPS} | Device:{' '}
          {deviceCapability?.deviceTier?.toUpperCase()}
        </Text>

        <TouchableOpacity
          style={styles.toggleButton}
          onPress={() => setShowPerformanceControls(!showPerformanceControls)}
        >
          <Text style={styles.toggleButtonText}>Performance Controls</Text>
        </TouchableOpacity>

        {showPerformanceControls && (
          <View style={styles.controlsPanel}>
            <TouchableOpacity
              style={[
                styles.controlButton,
                enablePoseDataStreaming && styles.controlButtonActive,
              ]}
              onPress={() =>
                setEnablePoseDataStreaming(!enablePoseDataStreaming)
              }
            >
              <Text style={styles.controlButtonText}>
                Data Streaming: {enablePoseDataStreaming ? 'ON' : 'OFF'}
              </Text>
            </TouchableOpacity>

            <TouchableOpacity
              style={[
                styles.controlButton,
                autoAdjustFPS && styles.controlButtonActive,
              ]}
              onPress={() => setAutoAdjustFPS(!autoAdjustFPS)}
            >
              <Text style={styles.controlButtonText}>
                Auto FPS: {autoAdjustFPS ? 'ON' : 'OFF'}
              </Text>
            </TouchableOpacity>

            <View style={styles.fpsButtons}>
              {[15, 30, 60].map((fpsValue) => (
                <TouchableOpacity
                  key={fpsValue}
                  style={[
                    styles.fpsButton,
                    targetFPS === fpsValue && styles.fpsButtonActive,
                  ]}
                  onPress={() => setTargetFPS(fpsValue)}
                >
                  <Text style={styles.fpsButtonText}>{fpsValue}</Text>
                </TouchableOpacity>
              ))}
            </View>
          </View>
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
  camera: {
    flex: 1,
  },
  performanceOverlay: {
    position: 'absolute',
    top: 50,
    left: 20,
    right: 20,
    backgroundColor: 'rgba(0, 0, 0, 0.7)',
    borderRadius: 10,
    padding: 15,
  },
  performanceText: {
    color: 'white',
    fontSize: 14,
    fontWeight: '600',
    marginBottom: 10,
  },
  toggleButton: {
    backgroundColor: '#007AFF',
    padding: 8,
    borderRadius: 6,
    alignItems: 'center',
  },
  toggleButtonText: {
    color: 'white',
    fontSize: 12,
    fontWeight: '600',
  },
  controlsPanel: {
    marginTop: 10,
  },
  controlButton: {
    backgroundColor: 'rgba(255, 255, 255, 0.2)',
    padding: 8,
    borderRadius: 6,
    marginBottom: 8,
    alignItems: 'center',
  },
  controlButtonActive: {
    backgroundColor: '#4CAF50',
  },
  controlButtonText: {
    color: 'white',
    fontSize: 12,
    fontWeight: '500',
  },
  fpsButtons: {
    flexDirection: 'row',
    justifyContent: 'space-around',
    marginTop: 8,
  },
  fpsButton: {
    backgroundColor: 'rgba(255, 255, 255, 0.2)',
    padding: 8,
    borderRadius: 6,
    minWidth: 40,
    alignItems: 'center',
  },
  fpsButtonActive: {
    backgroundColor: '#007AFF',
  },
  fpsButtonText: {
    color: 'white',
    fontSize: 12,
    fontWeight: '600',
  },
});

📚 API Reference