mediapipeunityplugin

## MediaPipe Unity Plugin This is a Unity (>= 2022.3) Native Plugin to use MediaPipe (0.10.22). The goal is to port the MediaPipe C++ API one-by-one to C# so it can be called from Unity. This approach may sacrifice performance when calling multiple APIs in a loop, but offers flexibility. ### Features - Write MediaPipe code in C#. - Run MediaPipe's official solutions on Unity. - Run custom `Calculator` and `CalculatorGraph` on Unity (may require C++ for certain input/output types). ### Hello World Example ```cs using Mediapipe; using UnityEngine; public sealed class HelloWorld : MonoBehaviour { private const string _ConfigText = @" input_stream: ""in"" output_stream: ""out"" node { calculator: ""PassThroughCalculator"" input_stream: ""in"" output_stream: ""out1"" } node { calculator: ""PassThroughCalculator"" input_stream: ""out1"" output_stream: ""out"" } "; private void Start() { using var graph = new CalculatorGraph(_ConfigText); using var poller = graph.AddOutputStreamPoller<string>("out"); graph.StartRun(); for (var i = 0; i < 10; i++) { graph.AddPacketToInputStream("in", Packet.CreateStringAt("Hello World!", i)); } graph.CloseInputStream("in"); var packet = new Packet<string>(); while (poller.Next(packet)) { Debug.Log(packet.Get()); } graph.WaitUntilDone(); } } ``` For more detailed usage, see the API Overview or the tutorials. ### Installation Download the pre-built package from the releases page: | file | contents | | --- | --- | | MediaPipeUnityPlugin-all.zip | All source code with required libraries. | | MediaPipeUnityPlugin-all-stripped.zip | Same as above but symbols stripped. | | com.github.homuler.mediapipe-*.tgz | Tarball package for Unity Package Manager. | | MediaPipeUnityPlugin.*.unitypackage | Unity package file. | For mobile sample scenes, prefer `MediaPipeUnityPlugin-all.zip` or `-stripped.zip` as they include required models and setup. ### Building the Plugin Yourself In most cases, you don't need to build the plugin yourself. Only if the pre-built package doesn't work, follow the build guide. The repository doesn't include required libraries or models, so cloning requires building. ### Building a Package - **Unity package**: Open the project, click `Tools > Export Unitypackage`. - **Tarball**: Install `npm`, then run `npm pack` in `Packages/com.github.homuler.mediapipe`. ### Supported Platforms GPU mode is not supported on macOS and Windows. Supported platforms include Linux (x86_64), macOS (x86_64/ARM64), Windows (x86_64), Android, iOS, and WebGL (with limitations). ### Supported Solutions The plugin implements C# APIs for several MediaPipe Tasks, including: - Object detection - Image segmentation - Hand landmark detection - Gesture recognition - Face detection - Face landmark detection - Pose landmark detection - Audio classification Legacy Solutions via MediaPipe Framework are also available but support has ended. ### Usage Import the plugin into your project using `.unitypackage` or `.tar.gz`. #### Android Note For Android builds, ensure `libstdc++_shared.so` is included in your APK, otherwise `DllNotFoundException` will be thrown. Place it in `Assets/Plugins/Android` or add Gradle code to include it at build time. #### Sample App Check sample scenes under `Assets/MediaPipeUnity/Samples/Scenes`. In the Inspector, select the proper Inference Mode (CPU or GPU) and Asset Loader Type (Local for Editor, StreamingAssets for devices). ### Technical Limitations - The plugin uses native libraries; bugs may crash UnityEditor or the app. On Linux/macOS, `SIGABRT` is handled to avoid crashes, but on Windows it's not possible. - For GPU inference, OpenGL Core API is not supported; use Vulkan on PC standalone builds. ### License MIT, with some files under other licenses (MediaPipe Apache 2.0, emscripten MIT, FontAwesome). See Third Party Notices for details.