Project import generated by Copybara.
GitOrigin-RevId: f72a0f86c2c2acdb1920973c718a9e26ed3ec4b6
This commit is contained in:
MediaPipe Team
@@ -1,21 +0,0 @@
|
||||
# Minimal makefile for Sphinx documentation
|
||||
#
|
||||
|
||||
# You can set these variables from the command line, and also
|
||||
# from the environment for the first two.
|
||||
SPHINXOPTS ?=
|
||||
SPHINXBUILD ?= sphinx-build
|
||||
SOURCEDIR = .
|
||||
BUILDDIR = _build
|
||||
|
||||
# Put it first so that "make" without argument is like "make help".
|
||||
help:
|
||||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
|
||||
.PHONY: help Makefile
|
||||
|
||||
# Catch-all target: route all unknown targets to Sphinx using the new
|
||||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
||||
%: Makefile
|
||||
rm -rf ./_build
|
||||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
|
||||
@@ -1,2 +1,3 @@
|
||||
This directory contains the source markdown files presented on
|
||||
the [MediaPipe Read-the-Docs](https://mediapipe.readthedocs.io) documentation site.
|
||||
This directory contains legacy markdown docs referenced in external sites and blog posts, and the docs have messages to redirect users to the corresponding up-to-date docs in other locations.
|
||||
|
||||
Source files of the update-to-date docs are in `docs` directly under root.
|
||||
|
||||
@@ -1,132 +0,0 @@
|
||||
## MediaPipe Android Archive Library
|
||||
|
||||
***Experimental Only***
|
||||
|
||||
The MediaPipe Android Archive (AAR) library is a convenient way to use MediaPipe
|
||||
with Android Studio and Gradle. MediaPipe doesn't publish a general AAR that can
|
||||
be used by all projects. Instead, developers need to add a mediapipe_aar()
|
||||
target to generate a custom AAR file for their own projects. This is necessary
|
||||
in order to include specific resources such as MediaPipe calculators needed for
|
||||
each project.
|
||||
|
||||
### Steps to build a MediaPipe AAR
|
||||
|
||||
1. Create a mediapipe_aar() target.
|
||||
|
||||
In the MediaPipe directory, create a new mediapipe_aar() target in a BUILD
|
||||
file. You need to figure out what calculators are used in the graph and
|
||||
provide the calculator dependencies to the mediapipe_aar(). For example, to
|
||||
build an AAR for [face detection gpu](./face_detection_mobile_gpu.md), you
|
||||
can put the following code into
|
||||
mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/BUILD.
|
||||
|
||||
```
|
||||
load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")
|
||||
|
||||
mediapipe_aar(
|
||||
name = "mp_face_detection_aar",
|
||||
calculators = ["//mediapipe/graphs/face_detection:mobile_calculators"],
|
||||
)
|
||||
```
|
||||
|
||||
2. Run the Bazel build command to generate the AAR.
|
||||
|
||||
```bash
|
||||
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \
|
||||
//path/to/the/aar/build/file:aar_name
|
||||
```
|
||||
|
||||
For the face detection AAR target we made in the step 1, run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \
|
||||
//mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar
|
||||
|
||||
# It should print:
|
||||
# Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar up-to-date:
|
||||
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
|
||||
```
|
||||
|
||||
3. (Optional) Save the AAR to your preferred location.
|
||||
|
||||
```bash
|
||||
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
|
||||
/absolute/path/to/your/preferred/location
|
||||
```
|
||||
|
||||
### Steps to use a MediaPipe AAR in Android Studio with Gradle
|
||||
|
||||
1. Start Android Studio and go to your project.
|
||||
|
||||
2. Copy the AAR into app/libs.
|
||||
|
||||
```bash
|
||||
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
|
||||
/path/to/your/app/libs/
|
||||
```
|
||||
|
||||
|
||||
|
||||
3. Make app/src/main/assets and copy assets (graph, model, and etc) into
|
||||
app/src/main/assets.
|
||||
|
||||
Build the MediaPipe binary graph and copy the assets into
|
||||
app/src/main/assets, e.g., for the face detection graph, you need to build
|
||||
and copy
|
||||
[the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41),
|
||||
[the tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite),
|
||||
and
|
||||
[the label map](https://github.com/google/mediapipe/blob/master/mediapipe/models/face_detection_front_labelmap.txt).
|
||||
|
||||
```bash
|
||||
bazel build -c opt mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu:binary_graph
|
||||
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.binarypb /path/to/your/app/src/main/assets/
|
||||
cp mediapipe/models/face_detection_front.tflite /path/to/your/app/src/main/assets/
|
||||
cp mediapipe/models/face_detection_front_labelmap.txt /path/to/your/app/src/main/assets/
|
||||
```
|
||||
|
||||
|
||||
|
||||
4. Make app/src/main/jniLibs and copy OpenCV JNI libraries into
|
||||
app/src/main/jniLibs.
|
||||
|
||||
MediaPipe depends on OpenCV, you will need to copy the precompiled OpenCV so
|
||||
files into app/src/main/jniLibs. You can download the official OpenCV
|
||||
Android SDK from
|
||||
[here](https://github.com/opencv/opencv/releases/download/3.4.3/opencv-3.4.3-android-sdk.zip)
|
||||
and run:
|
||||
|
||||
```bash
|
||||
cp -R ~/Downloads/OpenCV-android-sdk/sdk/native/libs/arm* /path/to/your/app/src/main/jniLibs/
|
||||
```
|
||||
|
||||
|
||||
|
||||
5. Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR.
|
||||
|
||||
```
|
||||
dependencies {
|
||||
implementation fileTree(dir: 'libs', include: ['*.jar', '*.aar'])
|
||||
implementation 'androidx.appcompat:appcompat:1.0.2'
|
||||
implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
|
||||
testImplementation 'junit:junit:4.12'
|
||||
androidTestImplementation 'androidx.test.ext:junit:1.1.0'
|
||||
androidTestImplementation 'androidx.test.espresso:espresso-core:3.1.1'
|
||||
// MediaPipe deps
|
||||
implementation 'com.google.flogger:flogger:0.3.1'
|
||||
implementation 'com.google.flogger:flogger-system-backend:0.3.1'
|
||||
implementation 'com.google.code.findbugs:jsr305:3.0.2'
|
||||
implementation 'com.google.guava:guava:27.0.1-android'
|
||||
implementation 'com.google.guava:guava:27.0.1-android'
|
||||
implementation 'com.google.protobuf:protobuf-java:3.11.4''
|
||||
// CameraX core library
|
||||
def camerax_version = "1.0.0-alpha06"
|
||||
implementation "androidx.camera:camera-core:$camerax_version"
|
||||
implementation "androidx.camera:camera-camera2:$camerax_version"
|
||||
}
|
||||
```
|
||||
|
||||
6. Follow our Android app examples to use MediaPipe in Android Studio for your
|
||||
use case. If you are looking for an example, a face detection
|
||||
example can be found
|
||||
[here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and a multi-hand tracking example can be found [here](https://github.com/jiuqiant/mediapipe_multi_hands_tracking_aar_example).
|
||||
+2
-344
@@ -1,344 +1,2 @@
|
||||
# Saliency-Aware Video Cropping using AutoFlip
|
||||
|
||||
## Introduction
|
||||
|
||||
AutoFlip is an automatic video cropping pipeline built on top of MediaPipe. This
|
||||
example focuses on demonstrating how to use AutoFlip to convert an input video
|
||||
to arbitrary aspect ratios.
|
||||
|
||||
For overall context on AutoFlip, please read this
|
||||
[Google AI Blog](https://mediapipe.page.link/autoflip).
|
||||
|
||||
|
||||
|
||||
## Building
|
||||
|
||||
Run the following command to build the AutoFlip pipeline:
|
||||
|
||||
Note: AutoFlip currently only works with OpenCV 3 . Please verify your OpenCV
|
||||
version beforehand.
|
||||
|
||||
```bash
|
||||
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/autoflip:run_autoflip
|
||||
```
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/autoflip/run_autoflip \
|
||||
--calculator_graph_config_file=mediapipe/examples/desktop/autoflip/autoflip_graph.pbtxt \
|
||||
--input_side_packets=input_video_path=/absolute/path/to/the/local/video/file,output_video_path=/absolute/path/to/save/the/output/video/file,aspect_ratio=1:1
|
||||
```
|
||||
|
||||
Use the `aspect_ratio` flag to provide the output aspect ratio. The format
|
||||
should be `width:height`, where the `width` and `height` are two positive
|
||||
integers. AutoFlip supports both landscape-to-portrait and portrait-to-landscape
|
||||
conversions. The pipeline internally compares the target aspect ratio against
|
||||
the original one, and determines the correct conversion automatically.
|
||||
|
||||
We have put a couple test videos under this
|
||||
[Google Drive folder](https://drive.google.com/corp/drive/u/0/folders/1KK9LV--Ey0UEVpxssVLhVl7dypgJSQgk).
|
||||
You could download the videos into your local file system, then modify the
|
||||
command above accordingly to run AutoFlip against the videos.
|
||||
|
||||
## MediaPipe Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev).
|
||||
|
||||
```bash
|
||||
# Autoflip graph that only renders the final cropped video. For use with
|
||||
# end user applications.
|
||||
max_queue_size: -1
|
||||
|
||||
# VIDEO_PREP: Decodes an input video file into images and a video header.
|
||||
node {
|
||||
calculator: "OpenCvVideoDecoderCalculator"
|
||||
input_side_packet: "INPUT_FILE_PATH:input_video_path"
|
||||
output_stream: "VIDEO:video_raw"
|
||||
output_stream: "VIDEO_PRESTREAM:video_header"
|
||||
output_side_packet: "SAVED_AUDIO_PATH:audio_path"
|
||||
}
|
||||
|
||||
# VIDEO_PREP: Scale the input video before feature extraction.
|
||||
node {
|
||||
calculator: "ScaleImageCalculator"
|
||||
input_stream: "FRAMES:video_raw"
|
||||
input_stream: "VIDEO_HEADER:video_header"
|
||||
output_stream: "FRAMES:video_frames_scaled"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ScaleImageCalculatorOptions]: {
|
||||
preserve_aspect_ratio: true
|
||||
output_format: SRGB
|
||||
target_width: 480
|
||||
algorithm: DEFAULT_WITHOUT_UPSCALE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# VIDEO_PREP: Create a low frame rate stream for feature extraction.
|
||||
node {
|
||||
calculator: "PacketThinnerCalculator"
|
||||
input_stream: "video_frames_scaled"
|
||||
output_stream: "video_frames_scaled_downsampled"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.PacketThinnerCalculatorOptions]: {
|
||||
thinner_type: ASYNC
|
||||
period: 200000
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# DETECTION: find borders around the video and major background color.
|
||||
node {
|
||||
calculator: "BorderDetectionCalculator"
|
||||
input_stream: "VIDEO:video_raw"
|
||||
output_stream: "DETECTED_BORDERS:borders"
|
||||
}
|
||||
|
||||
# DETECTION: find shot/scene boundaries on the full frame rate stream.
|
||||
node {
|
||||
calculator: "ShotBoundaryCalculator"
|
||||
input_stream: "VIDEO:video_frames_scaled"
|
||||
output_stream: "IS_SHOT_CHANGE:shot_change"
|
||||
options {
|
||||
[type.googleapis.com/mediapipe.autoflip.ShotBoundaryCalculatorOptions] {
|
||||
min_shot_span: 0.2
|
||||
min_motion: 0.3
|
||||
window_size: 15
|
||||
min_shot_measure: 10
|
||||
min_motion_with_shot_measure: 0.05
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# DETECTION: find faces on the down sampled stream
|
||||
node {
|
||||
calculator: "AutoFlipFaceDetectionSubgraph"
|
||||
input_stream: "VIDEO:video_frames_scaled_downsampled"
|
||||
output_stream: "DETECTIONS:face_detections"
|
||||
}
|
||||
node {
|
||||
calculator: "FaceToRegionCalculator"
|
||||
input_stream: "VIDEO:video_frames_scaled_downsampled"
|
||||
input_stream: "FACES:face_detections"
|
||||
output_stream: "REGIONS:face_regions"
|
||||
}
|
||||
|
||||
# DETECTION: find objects on the down sampled stream
|
||||
node {
|
||||
calculator: "AutoFlipObjectDetectionSubgraph"
|
||||
input_stream: "VIDEO:video_frames_scaled_downsampled"
|
||||
output_stream: "DETECTIONS:object_detections"
|
||||
}
|
||||
node {
|
||||
calculator: "LocalizationToRegionCalculator"
|
||||
input_stream: "DETECTIONS:object_detections"
|
||||
output_stream: "REGIONS:object_regions"
|
||||
options {
|
||||
[type.googleapis.com/mediapipe.autoflip.LocalizationToRegionCalculatorOptions] {
|
||||
output_all_signals: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# SIGNAL FUSION: Combine detections (with weights) on each frame
|
||||
node {
|
||||
calculator: "SignalFusingCalculator"
|
||||
input_stream: "shot_change"
|
||||
input_stream: "face_regions"
|
||||
input_stream: "object_regions"
|
||||
output_stream: "salient_regions"
|
||||
options {
|
||||
[type.googleapis.com/mediapipe.autoflip.SignalFusingCalculatorOptions] {
|
||||
signal_settings {
|
||||
type { standard: FACE_CORE_LANDMARKS }
|
||||
min_score: 0.85
|
||||
max_score: 0.9
|
||||
is_required: false
|
||||
}
|
||||
signal_settings {
|
||||
type { standard: FACE_ALL_LANDMARKS }
|
||||
min_score: 0.8
|
||||
max_score: 0.85
|
||||
is_required: false
|
||||
}
|
||||
signal_settings {
|
||||
type { standard: FACE_FULL }
|
||||
min_score: 0.8
|
||||
max_score: 0.85
|
||||
is_required: false
|
||||
}
|
||||
signal_settings {
|
||||
type: { standard: HUMAN }
|
||||
min_score: 0.75
|
||||
max_score: 0.8
|
||||
is_required: false
|
||||
}
|
||||
signal_settings {
|
||||
type: { standard: PET }
|
||||
min_score: 0.7
|
||||
max_score: 0.75
|
||||
is_required: false
|
||||
}
|
||||
signal_settings {
|
||||
type: { standard: CAR }
|
||||
min_score: 0.7
|
||||
max_score: 0.75
|
||||
is_required: false
|
||||
}
|
||||
signal_settings {
|
||||
type: { standard: OBJECT }
|
||||
min_score: 0.1
|
||||
max_score: 0.2
|
||||
is_required: false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# CROPPING: make decisions about how to crop each frame.
|
||||
node {
|
||||
calculator: "SceneCroppingCalculator"
|
||||
input_side_packet: "EXTERNAL_ASPECT_RATIO:aspect_ratio"
|
||||
input_stream: "VIDEO_FRAMES:video_raw"
|
||||
input_stream: "KEY_FRAMES:video_frames_scaled_downsampled"
|
||||
input_stream: "DETECTION_FEATURES:salient_regions"
|
||||
input_stream: "STATIC_FEATURES:borders"
|
||||
input_stream: "SHOT_BOUNDARIES:shot_change"
|
||||
output_stream: "CROPPED_FRAMES:cropped_frames"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.autoflip.SceneCroppingCalculatorOptions]: {
|
||||
max_scene_size: 600
|
||||
key_frame_crop_options: {
|
||||
score_aggregation_type: CONSTANT
|
||||
}
|
||||
scene_camera_motion_analyzer_options: {
|
||||
motion_stabilization_threshold_percent: 0.5
|
||||
salient_point_bound: 0.499
|
||||
}
|
||||
padding_parameters: {
|
||||
blur_cv_size: 200
|
||||
overlay_opacity: 0.6
|
||||
}
|
||||
target_size_type: MAXIMIZE_TARGET_DIMENSION
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ENCODING(required): encode the video stream for the final cropped output.
|
||||
node {
|
||||
calculator: "VideoPreStreamCalculator"
|
||||
# Fetch frame format and dimension from input frames.
|
||||
input_stream: "FRAME:cropped_frames"
|
||||
# Copying frame rate and duration from original video.
|
||||
input_stream: "VIDEO_PRESTREAM:video_header"
|
||||
output_stream: "output_frames_video_header"
|
||||
}
|
||||
|
||||
node {
|
||||
calculator: "OpenCvVideoEncoderCalculator"
|
||||
input_stream: "VIDEO:cropped_frames"
|
||||
input_stream: "VIDEO_PRESTREAM:output_frames_video_header"
|
||||
input_side_packet: "OUTPUT_FILE_PATH:output_video_path"
|
||||
input_side_packet: "AUDIO_FILE_PATH:audio_path"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: {
|
||||
codec: "avc1"
|
||||
video_format: "mp4"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Parameters
|
||||
|
||||
### Required vs. Best-Effort Saliency Features
|
||||
|
||||
AutoFlip allows users to implement and specify custom features to be used in the
|
||||
camera trajectory computation. If the user would like to detect and preserve
|
||||
scenes of lions in a wildlife protection video, for example, they could
|
||||
implement and add a feature detection calculator for lions into the pipeline.
|
||||
Refer to `AutoFlipFaceDetectionSubgraph` and `FaceToRegionCalculator`, or
|
||||
`AutoFlipObjectDetectionSubgraph` and `LocalizationToRegionCalculator` for
|
||||
examples of how to create new feature detection calculators.
|
||||
|
||||
After adding different feature signals into the graph, use the
|
||||
`SignalFusingCalculator` node to specify types and weights for different feature
|
||||
signals. For example, in the graph above, we specified a `face_region` and an
|
||||
`object_region` input streams, to represent face signals and agnostic object
|
||||
signals, respectively.
|
||||
|
||||
The larger the weight, the more important the features will be considered when
|
||||
AutoFlip computes the camera trajectory. Use the `is_required` flag to mark a
|
||||
feature as a hard constraint, in which case the computed camera trajectory will
|
||||
try best to cover these feature types in the cropped videos. If for some reason
|
||||
the required features cannot be all covered (for example, when they are too
|
||||
spread out in the video), AutoFlip will apply a padding effect to cover as much
|
||||
salient content as possible. See an illustration below.
|
||||
|
||||
|
||||
|
||||
### Stable vs Tracking Camera Motion
|
||||
|
||||
AutoFlip makes a decision on each scene whether to have the cropped viewpoint
|
||||
follow an object or if the crop should remain stable (centered on detected
|
||||
objects). The parameter `motion_stabilization_threshold_percent` value is used
|
||||
to make the decision to track action or keep the camera stable. If, over the
|
||||
duration of the scene, all detected focus objects remain within this ratio of
|
||||
the frame (e.g. 0.5 = 50% or 1920 * .5 = 960 pixels on 1080p video) then the
|
||||
camera is held steady. Otherwise the camera tracks activity within the frame.
|
||||
|
||||
### Snap To Center
|
||||
|
||||
For some scenes the camera viewpoint will remain stable at the center of
|
||||
activity (see `motion_stabilization_threshold_percent` setting). In this case,
|
||||
if the determined best stable viewpoint is within
|
||||
`snap_center_max_distance_percent` of the frame's center the camera will be
|
||||
shifted to be locked to the center of the frame. This setting is useful for
|
||||
videos where the camera operator did a good job already centering content or if
|
||||
titles and logos are expected to appear in the center of the frame. It may be
|
||||
less useful on raw content where objects are not already well positioned on
|
||||
screen.
|
||||
|
||||
### Visualization to Facilitate Debugging
|
||||
|
||||
`SceneCroppingCalculator` provides two extra output streams
|
||||
`KEY_FRAME_CROP_REGION_VIZ_FRAMES` and `SALIENT_POINT_FRAME_VIZ_FRAMES` to
|
||||
visualize the cropping window as well as salient points detected on each frame.
|
||||
You could modify the `SceneCroppingCalculator` node like below to enable these
|
||||
two output streams.
|
||||
|
||||
```bash
|
||||
node {
|
||||
calculator: "SceneCroppingCalculator"
|
||||
input_side_packet: "EXTERNAL_ASPECT_RATIO:aspect_ratio"
|
||||
input_stream: "VIDEO_FRAMES:video_raw"
|
||||
input_stream: "KEY_FRAMES:video_frames_scaled_downsampled"
|
||||
input_stream: "DETECTION_FEATURES:salient_regions"
|
||||
input_stream: "STATIC_FEATURES:borders"
|
||||
input_stream: "SHOT_BOUNDARIES:shot_change"
|
||||
output_stream: "CROPPED_FRAMES:cropped_frames"
|
||||
output_stream: "KEY_FRAME_CROP_REGION_VIZ_FRAMES:key_frame_crop_viz_frames"
|
||||
output_stream: "SALIENT_POINT_FRAME_VIZ_FRAMES:salient_point_viz_frames"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.autoflip.SceneCroppingCalculatorOptions]: {
|
||||
max_scene_size: 600
|
||||
key_frame_crop_options: {
|
||||
score_aggregation_type: CONSTANT
|
||||
}
|
||||
scene_camera_motion_analyzer_options: {
|
||||
motion_stabilization_threshold_percent: 0.5
|
||||
salient_point_bound: 0.499
|
||||
}
|
||||
padding_parameters: {
|
||||
blur_cv_size: 200
|
||||
overlay_opacity: 0.6
|
||||
}
|
||||
target_size_type: MAXIMIZE_TARGET_DIMENSION
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Content moved to
|
||||
[AutoFlip: Saliency-aware Video Cropping](https://google.github.io/mediapipe/solutions/autoflip)
|
||||
|
||||
@@ -1,327 +0,0 @@
|
||||
# Building MediaPipe Examples
|
||||
|
||||
* [Android](#android)
|
||||
* [iOS](#ios)
|
||||
* [Desktop](#desktop)
|
||||
|
||||
## Android
|
||||
|
||||
### Prerequisite
|
||||
|
||||
* Java Runtime.
|
||||
* Android SDK release 28.0.3 and above.
|
||||
* Android NDK r18b and above.
|
||||
|
||||
MediaPipe recommends setting up Android SDK and NDK via Android Studio (and see
|
||||
below for Android Studio setup). However, if you prefer using MediaPipe without
|
||||
Android Studio, please run
|
||||
[`setup_android_sdk_and_ndk.sh`](https://github.com/google/mediapipe/tree/master/setup_android_sdk_and_ndk.sh)
|
||||
to download and setup Android SDK and NDK before building any Android example
|
||||
apps.
|
||||
|
||||
If Android SDK and NDK are already installed (e.g., by Android Studio), set
|
||||
$ANDROID_HOME and $ANDROID_NDK_HOME to point to the installed SDK and NDK.
|
||||
|
||||
```bash
|
||||
export ANDROID_HOME=<path to the Android SDK>
|
||||
export ANDROID_NDK_HOME=<path to the Android NDK>
|
||||
```
|
||||
|
||||
In order to use MediaPipe on earlier Android versions, MediaPipe needs to switch
|
||||
to a lower Android API level. You can achieve this by specifying `api_level =
|
||||
<api level integer>` in android_ndk_repository() and/or android_sdk_repository()
|
||||
in the [`WORKSPACE`](https://github.com/google/mediapipe/tree/master/WORKSPACE) file.
|
||||
|
||||
Please verify all the necessary packages are installed.
|
||||
|
||||
* Android SDK Platform API Level 28 or 29
|
||||
* Android SDK Build-Tools 28 or 29
|
||||
* Android SDK Platform-Tools 28 or 29
|
||||
* Android SDK Tools 26.1.1
|
||||
* Android NDK 17c or above
|
||||
|
||||
### Option 1: Build with Bazel in Command Line
|
||||
|
||||
1. To build an Android example app, for instance, for MediaPipe Hand, run:
|
||||
|
||||
Note: To reduce the binary size, consider appending `--linkopt="-s"` to the
|
||||
command below to strip symbols.
|
||||
|
||||
~~~
|
||||
```bash
|
||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu
|
||||
```
|
||||
~~~
|
||||
|
||||
1. Install it on a device with:
|
||||
|
||||
```bash
|
||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
|
||||
```
|
||||
|
||||
### Option 2: Build with Bazel in Android Studio
|
||||
|
||||
The MediaPipe project can be imported into Android Studio using the Bazel
|
||||
plugins. This allows the MediaPipe examples to be built and modified in Android
|
||||
Studio.
|
||||
|
||||
To incorporate MediaPipe into an existing Android Studio project, see these
|
||||
[instructions](./android_archive_library.md) that use Android Archive (AAR) and
|
||||
Gradle.
|
||||
|
||||
The steps below use Android Studio 3.5 to build and install a MediaPipe example
|
||||
app:
|
||||
|
||||
1. Install and launch Android Studio 3.5.
|
||||
|
||||
2. Select `Configure` | `SDK Manager` | `SDK Platforms`.
|
||||
|
||||
* Verify that Android SDK Platform API Level 28 or 29 is installed.
|
||||
* Take note of the Android SDK Location, e.g.,
|
||||
`/usr/local/home/Android/Sdk`.
|
||||
|
||||
3. Select `Configure` | `SDK Manager` | `SDK Tools`.
|
||||
|
||||
* Verify that Android SDK Build-Tools 28 or 29 is installed.
|
||||
* Verify that Android SDK Platform-Tools 28 or 29 is installed.
|
||||
* Verify that Android SDK Tools 26.1.1 is installed.
|
||||
* Verify that Android NDK 17c or above is installed.
|
||||
* Take note of the Android NDK Location, e.g.,
|
||||
`/usr/local/home/Android/Sdk/ndk-bundle` or
|
||||
`/usr/local/home/Android/Sdk/ndk/20.0.5594570`.
|
||||
|
||||
4. Set environment variables `$ANDROID_HOME` and `$ANDROID_NDK_HOME` to point
|
||||
to the installed SDK and NDK.
|
||||
|
||||
```bash
|
||||
export ANDROID_HOME=/usr/local/home/Android/Sdk
|
||||
|
||||
# If the NDK libraries are installed by a previous version of Android Studio, do
|
||||
export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk-bundle
|
||||
# If the NDK libraries are installed by Android Studio 3.5, do
|
||||
export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk/<version number>
|
||||
```
|
||||
|
||||
5. Select `Configure` | `Plugins` install `Bazel`.
|
||||
|
||||
6. On Linux, select `File` | `Settings`| `Bazel settings`. On macos, select
|
||||
`Android Studio` | `Preferences` | `Bazel settings`. Then, modify `Bazel
|
||||
binary location` to be the same as the output of `$ which bazel`.
|
||||
|
||||
7. Select `Import Bazel Project`.
|
||||
|
||||
* Select `Workspace`: `/path/to/mediapipe` and select `Next`.
|
||||
* Select `Generate from BUILD file`: `/path/to/mediapipe/BUILD` and select
|
||||
`Next`.
|
||||
* Modify `Project View` to be the following and select `Finish`.
|
||||
|
||||
```
|
||||
directories:
|
||||
# read project settings, e.g., .bazelrc
|
||||
.
|
||||
-mediapipe/objc
|
||||
-mediapipe/examples/ios
|
||||
|
||||
targets:
|
||||
//mediapipe/examples/android/...:all
|
||||
//mediapipe/java/...:all
|
||||
|
||||
android_sdk_platform: android-29
|
||||
|
||||
sync_flags:
|
||||
--host_crosstool_top=@bazel_tools//tools/cpp:toolchain
|
||||
```
|
||||
|
||||
8. Select `Bazel` | `Sync` | `Sync project with Build files`.
|
||||
|
||||
Note: Even after doing step 4, if you still see the error: `"no such package
|
||||
'@androidsdk//': Either the path attribute of android_sdk_repository or the
|
||||
ANDROID_HOME environment variable must be set."`, please modify the
|
||||
[`WORKSPACE`](https://github.com/google/mediapipe/tree/master/WORKSPACE) file to point to your
|
||||
SDK and NDK library locations, as below:
|
||||
|
||||
```
|
||||
android_sdk_repository(
|
||||
name = "androidsdk",
|
||||
path = "/path/to/android/sdk"
|
||||
)
|
||||
|
||||
android_ndk_repository(
|
||||
name = "androidndk",
|
||||
path = "/path/to/android/ndk"
|
||||
)
|
||||
```
|
||||
|
||||
9. Connect an Android device to the workstation.
|
||||
|
||||
10. Select `Run...` | `Edit Configurations...`.
|
||||
|
||||
* Select `Templates` | `Bazel Command`.
|
||||
* Enter Target Expression:
|
||||
`//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu`
|
||||
* Enter Bazel command: `mobile-install`.
|
||||
* Enter Bazel flags: `-c opt --config=android_arm64`.
|
||||
* Press the `[+]` button to add the new configuration.
|
||||
* Select `Run` to run the example app on the connected Android device.
|
||||
|
||||
## iOS
|
||||
|
||||
### Prerequisite
|
||||
|
||||
1. Install [Xcode](https://developer.apple.com/xcode/) and the Command Line
|
||||
Tools.
|
||||
|
||||
Follow Apple's instructions to obtain the required development certificates
|
||||
and provisioning profiles for your iOS device. Install the Command Line
|
||||
Tools by
|
||||
|
||||
```bash
|
||||
xcode-select --install
|
||||
```
|
||||
|
||||
2. Install [Bazel](https://bazel.build/).
|
||||
|
||||
We recommend using [Homebrew](https://brew.sh/) to get the latest version.
|
||||
|
||||
3. Set Python 3.7 as the default Python version and install the Python "six"
|
||||
library.
|
||||
|
||||
To make Mediapipe work with TensorFlow, please set Python 3.7 as the default
|
||||
Python version and install the Python "six" library.
|
||||
|
||||
```bash
|
||||
pip3 install --user six
|
||||
```
|
||||
|
||||
4. Clone the MediaPipe repository.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/google/mediapipe.git
|
||||
```
|
||||
|
||||
5. Symlink or copy your provisioning profile to
|
||||
`mediapipe/mediapipe/provisioning_profile.mobileprovision`.
|
||||
|
||||
```bash
|
||||
cd mediapipe
|
||||
ln -s ~/Downloads/MyProvisioningProfile.mobileprovision mediapipe/provisioning_profile.mobileprovision
|
||||
```
|
||||
|
||||
Tip: You can use this command to see the provisioning profiles you have
|
||||
previously downloaded using Xcode: `open
|
||||
~/Library/MobileDevice/"Provisioning Profiles"`. If there are none, generate
|
||||
and download a profile on
|
||||
[Apple's developer site](https://developer.apple.com/account/resources/).
|
||||
|
||||
### Option 1: Build with Bazel in Command Line
|
||||
|
||||
1. Modify the `bundle_id` field of the app's `ios_application` target to use
|
||||
your own identifier. For instance, for
|
||||
[MediaPipe Hand](./hand_tracking_mobile_gpu.md), the `bundle_id` is in the
|
||||
`HandTrackingGpuApp` target in the
|
||||
[BUILD](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu/BUILD)
|
||||
file.
|
||||
|
||||
2. Again using [MediaPipe Hand](./hand_tracking_mobile_gpu.md) for example,
|
||||
run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
|
||||
```
|
||||
|
||||
You may see a permission request from `codesign` in order to sign the app.
|
||||
|
||||
3. In Xcode, open the `Devices and Simulators` window (command-shift-2).
|
||||
|
||||
4. Make sure your device is connected. You will see a list of installed apps.
|
||||
Press the "+" button under the list, and select the `.ipa` file built by
|
||||
Bazel.
|
||||
|
||||
5. You can now run the app on your device.
|
||||
|
||||
### Option 2: Build in Xcode
|
||||
|
||||
Note: This workflow requires a separate tool in addition to Bazel. If it fails
|
||||
to work for some reason, please resort to the command-line build instructions in
|
||||
the previous section.
|
||||
|
||||
1. We will use a tool called [Tulsi](https://tulsi.bazel.build/) for generating
|
||||
Xcode projects from Bazel build configurations.
|
||||
|
||||
```bash
|
||||
# cd out of the mediapipe directory, then:
|
||||
git clone https://github.com/bazelbuild/tulsi.git
|
||||
cd tulsi
|
||||
# remove Xcode version from Tulsi's .bazelrc (see http://github.com/bazelbuild/tulsi#building-and-installing):
|
||||
sed -i .orig '/xcode_version/d' .bazelrc
|
||||
# build and run Tulsi:
|
||||
sh build_and_run.sh
|
||||
```
|
||||
|
||||
This will install `Tulsi.app` inside the `Applications` directory in your
|
||||
home directory.
|
||||
|
||||
2. Open `mediapipe/Mediapipe.tulsiproj` using the Tulsi app.
|
||||
|
||||
Important: If Tulsi displays an error saying "Bazel could not be found",
|
||||
press the "Bazel..." button in the Packages tab and select the `bazel`
|
||||
executable in your homebrew `/bin/` directory.
|
||||
|
||||
3. Select the MediaPipe config in the Configs tab, then press the Generate
|
||||
button below. You will be asked for a location to save the Xcode project.
|
||||
Once the project is generated, it will be opened in Xcode.
|
||||
|
||||
4. You can now select any of the MediaPipe demos in the target menu, and build
|
||||
and run them as normal.
|
||||
|
||||
Note: When you ask Xcode to run an app, by default it will use the Debug
|
||||
configuration. Some of our demos are computationally heavy; you may want to
|
||||
use the Release configuration for better performance.
|
||||
|
||||
Tip: To switch build configuration in Xcode, click on the target menu,
|
||||
choose "Edit Scheme...", select the Run action, and switch the Build
|
||||
Configuration from Debug to Release. Note that this is set independently for
|
||||
each target.
|
||||
|
||||
## Desktop
|
||||
|
||||
### Option 1: Running on CPU
|
||||
|
||||
1. To build, for example, [MediaPipe Hand](./hand_tracking_mobile_gpu.md), run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
|
||||
```
|
||||
|
||||
This will open up your webcam as long as it is connected and on. Any errors
|
||||
is likely due to your webcam being not accessible.
|
||||
|
||||
2. To run the application:
|
||||
|
||||
```bash
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
|
||||
```
|
||||
|
||||
### Option 2: Running on GPU
|
||||
|
||||
Note: This currently works only on Linux, and please first follow
|
||||
[OpenGL ES Setup on Linux Desktop](./gpu.md#opengl-es-setup-on-linux-desktop).
|
||||
|
||||
1. To build, for example, [MediaPipe Hand](./hand_tracking_mobile_gpu.md), run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
|
||||
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
|
||||
```
|
||||
|
||||
This will open up your webcam as long as it is connected and on. Any errors
|
||||
is likely due to your webcam being not accessible, or GPU drivers not setup
|
||||
properly.
|
||||
|
||||
2. To run the application:
|
||||
|
||||
```bash
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
|
||||
```
|
||||
@@ -1,164 +0,0 @@
|
||||
## Building MediaPipe Calculators
|
||||
|
||||
- [Example calculator](#example-calculator)
|
||||
|
||||
|
||||
### Example calculator
|
||||
|
||||
This section discusses the implementation of `PacketClonerCalculator`, which
|
||||
does a relatively simple job, and is used in many calculator graphs.
|
||||
`PacketClonerCalculator` simply produces a copy of its most recent input
|
||||
packets on demand.
|
||||
|
||||
`PacketClonerCalculator` is useful when the timestamps of arriving data packets
|
||||
are not aligned perfectly. Suppose we have a room with a microphone, light
|
||||
sensor and a video camera that is collecting sensory data. Each of the sensors
|
||||
operates independently and collects data intermittently. Suppose that the output
|
||||
of each sensor is:
|
||||
|
||||
* microphone = loudness in decibels of sound in the room (Integer)
|
||||
* light sensor = brightness of room (Integer)
|
||||
* video camera = RGB image frame of room (ImageFrame)
|
||||
|
||||
Our simple perception pipeline is designed to process sensory data from these 3
|
||||
sensors such that at any time when we have image frame data from the camera that
|
||||
is synchronized with the last collected microphone loudness data and light
|
||||
sensor brightness data. To do this with MediaPipe, our perception pipeline has 3
|
||||
input streams:
|
||||
|
||||
* room_mic_signal - Each packet of data in this input stream is integer data
|
||||
representing how loud audio is in a room with timestamp.
|
||||
* room_lightening_sensor - Each packet of data in this input stream is integer
|
||||
data representing how bright is the room illuminated with timestamp.
|
||||
* room_video_tick_signal - Each packet of data in this input stream is
|
||||
imageframe of video data representing video collected from camera in the
|
||||
room with timestamp.
|
||||
|
||||
Below is the implementation of the `PacketClonerCalculator`. You can see
|
||||
the `GetContract()`, `Open()`, and `Process()` methods as well as the instance
|
||||
variable `current_` which holds the most recent input packets.
|
||||
|
||||
```c++
|
||||
// This takes packets from N+1 streams, A_1, A_2, ..., A_N, B.
|
||||
// For every packet that appears in B, outputs the most recent packet from each
|
||||
// of the A_i on a separate stream.
|
||||
|
||||
#include <vector>
|
||||
|
||||
#include "absl/strings/str_cat.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
// For every packet received on the last stream, output the latest packet
|
||||
// obtained on all other streams. Therefore, if the last stream outputs at a
|
||||
// higher rate than the others, this effectively clones the packets from the
|
||||
// other streams to match the last.
|
||||
//
|
||||
// Example config:
|
||||
// node {
|
||||
// calculator: "PacketClonerCalculator"
|
||||
// input_stream: "first_base_signal"
|
||||
// input_stream: "second_base_signal"
|
||||
// input_stream: "tick_signal"
|
||||
// output_stream: "cloned_first_base_signal"
|
||||
// output_stream: "cloned_second_base_signal"
|
||||
// }
|
||||
//
|
||||
class PacketClonerCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
const int tick_signal_index = cc->Inputs().NumEntries() - 1;
|
||||
// cc->Inputs().NumEntries() returns the number of input streams
|
||||
// for the PacketClonerCalculator
|
||||
for (int i = 0; i < tick_signal_index; ++i) {
|
||||
cc->Inputs().Index(i).SetAny();
|
||||
// cc->Inputs().Index(i) returns the input stream pointer by index
|
||||
cc->Outputs().Index(i).SetSameAs(&cc->Inputs().Index(i));
|
||||
}
|
||||
cc->Inputs().Index(tick_signal_index).SetAny();
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) final {
|
||||
tick_signal_index_ = cc->Inputs().NumEntries() - 1;
|
||||
current_.resize(tick_signal_index_);
|
||||
// Pass along the header for each stream if present.
|
||||
for (int i = 0; i < tick_signal_index_; ++i) {
|
||||
if (!cc->Inputs().Index(i).Header().IsEmpty()) {
|
||||
cc->Outputs().Index(i).SetHeader(cc->Inputs().Index(i).Header());
|
||||
// Sets the output stream of index i header to be the same as
|
||||
// the header for the input stream of index i
|
||||
}
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) final {
|
||||
// Store input signals.
|
||||
for (int i = 0; i < tick_signal_index_; ++i) {
|
||||
if (!cc->Inputs().Index(i).Value().IsEmpty()) {
|
||||
current_[i] = cc->Inputs().Index(i).Value();
|
||||
}
|
||||
}
|
||||
|
||||
// Output if the tick signal is non-empty.
|
||||
if (!cc->Inputs().Index(tick_signal_index_).Value().IsEmpty()) {
|
||||
for (int i = 0; i < tick_signal_index_; ++i) {
|
||||
if (!current_[i].IsEmpty()) {
|
||||
cc->Outputs().Index(i).AddPacket(
|
||||
current_[i].At(cc->InputTimestamp()));
|
||||
// Add a packet to output stream of index i a packet from inputstream i
|
||||
// with timestamp common to all present inputs
|
||||
//
|
||||
} else {
|
||||
cc->Outputs().Index(i).SetNextTimestampBound(
|
||||
cc->InputTimestamp().NextAllowedInStream());
|
||||
// if current_[i], 1 packet buffer for input stream i is empty, we will set
|
||||
// next allowed timestamp for input stream i to be current timestamp + 1
|
||||
}
|
||||
}
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
private:
|
||||
std::vector<Packet> current_;
|
||||
int tick_signal_index_;
|
||||
};
|
||||
|
||||
REGISTER_CALCULATOR(PacketClonerCalculator);
|
||||
} // namespace mediapipe
|
||||
```
|
||||
|
||||
Typically, a calculator has only a .cc file. No .h is required, because
|
||||
mediapipe uses registration to make calculators known to it. After you have
|
||||
defined your calculator class, register it with a macro invocation
|
||||
REGISTER_CALCULATOR(calculator_class_name).
|
||||
|
||||
Below is a trivial MediaPipe graph that has 3 input streams, 1 node
|
||||
(PacketClonerCalculator) and 3 output streams.
|
||||
|
||||
```proto
|
||||
input_stream: "room_mic_signal"
|
||||
input_stream: "room_lighting_sensor"
|
||||
input_stream: "room_video_tick_signal"
|
||||
|
||||
node {
|
||||
calculator: "PacketClonerCalculator"
|
||||
input_stream: "room_mic_signal"
|
||||
input_stream: "room_lighting_sensor"
|
||||
input_stream: "room_video_tick_signal"
|
||||
output_stream: "cloned_room_mic_signal"
|
||||
output_stream: "cloned_lighting_sensor"
|
||||
}
|
||||
```
|
||||
|
||||
The diagram below shows how the `PacketClonerCalculator` defines its output
|
||||
packets based on its series of input packets.
|
||||
|
||||
|  |
|
||||
|:--:|
|
||||
| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets is determined by the sequene of input packets and their timestamps. The timestamps are shows along the right side of the diagram.* |
|
||||
|
||||
|
||||
@@ -1,92 +0,0 @@
|
||||
# MediaPipe Concepts
|
||||
|
||||
## The basics
|
||||
|
||||
### Packet
|
||||
|
||||
The basic data flow unit. A packet consists of a numeric timestamp and a shared pointer to an **immutable** payload. The payload can be of any C++ type, and the payload's type is also referred to as the type of the packet. Packets are value classes and can be copied cheaply. Each copy shares ownership of the payload, with reference-counting semantics. Each copy has its own timestamp. [Details](packets.md).
|
||||
|
||||
### Graph
|
||||
|
||||
MediaPipe processing takes place inside a graph, which defines packet flow paths
|
||||
between **nodes**. A graph can have any number of inputs and outputs, and data
|
||||
flow can branch and merge. Generally data flows forward, but
|
||||
[backward loops](cycles.md) are possible.
|
||||
|
||||
### Nodes
|
||||
|
||||
Nodes produce and/or consume packets, and they are where the bulk of the graph’s
|
||||
work takes place. They are also known as “calculators”, for historical reasons.
|
||||
Each node’s interface defines a number of input and output **ports**, identified by
|
||||
a tag and/or an index.
|
||||
|
||||
### Streams
|
||||
|
||||
A stream is a connection between two nodes that carries a sequence of packets,
|
||||
whose timestamps must be monotonically increasing.
|
||||
|
||||
### Side packets
|
||||
|
||||
A side packet connection between nodes carries a single packet (with unspecified
|
||||
timestamp). It can be used to provide some data that will remain constant,
|
||||
whereas a stream represents a flow of data that changes over time.
|
||||
|
||||
### Packet Ports
|
||||
|
||||
A port has an associated type; packets transiting through the port must be of
|
||||
that type. An output stream port can be connected to any number of
|
||||
input stream ports of the same type; each consumer receives a separate copy of
|
||||
the output packets, and has its own queue, so it can consume them at its own
|
||||
pace. Similarly, a side packet output port can be connected to as many side
|
||||
packet input ports as desired.
|
||||
|
||||
A port can be required, meaning that a connection must be made for the graph to
|
||||
be valid, or optional, meaning it may remain unconnected.
|
||||
|
||||
Note: even if a stream connection is required, the stream may not carry a packet for all timestamps.
|
||||
|
||||
## Input and output
|
||||
|
||||
Data flow can originate from **source nodes**, which have no input streams and
|
||||
produce packets spontaneously (e.g. by reading from a file); or from **graph input streams**, which let an application feed packets into a graph.
|
||||
|
||||
Similarly, there are **sink nodes** that receive data and write it to various
|
||||
destinations (e.g. a file, a memory buffer, etc.), and an application can also
|
||||
receive output from the graph using **callbacks**.
|
||||
|
||||
## Runtime behavior
|
||||
|
||||
### Graph lifetime
|
||||
|
||||
Once a graph has been initialized, it can be **started** to begin processing
|
||||
data, and can process a stream of packets until each stream is closed or the
|
||||
graph is **canceled**. Then the graph can be destroyed or **started** again.
|
||||
|
||||
### Node lifetime
|
||||
|
||||
There are three main lifetime methods the framework will call on a node:
|
||||
|
||||
- Open: called once, before the other methods. When it is called, all input
|
||||
side packets required by the node will be available.
|
||||
- Process: called multiple times, when a new set of inputs is available,
|
||||
according to the node’s input policy.
|
||||
- Close: called once, at the end.
|
||||
|
||||
In addition, each calculator can define constructor and destructor, which are
|
||||
useful for creating and deallocating resources that are independent of the
|
||||
processed data.
|
||||
|
||||
### Input policies
|
||||
|
||||
The default input policy is deterministic collation of packets by timestamp. A node receives
|
||||
all inputs for the same timestamp at the same time, in an invocation of its
|
||||
Process method; and successive input sets are received in their timestamp order. This can
|
||||
require delaying the processing of some packets until a packet with the same
|
||||
timestamp is received on all input streams, or until it can be guaranteed that a
|
||||
packet with that timestamp will not be arriving on the streams that have not
|
||||
received it.
|
||||
|
||||
Other policies are also available, implemented using a separate kind of
|
||||
component known as an InputStreamHandler.
|
||||
|
||||
See [scheduling](scheduling_sync.md) for more details.
|
||||
@@ -1,57 +0,0 @@
|
||||
"""Configuration file for the Sphinx documentation builder.
|
||||
|
||||
This file only contains a selection of the most common options.
|
||||
For a full list see the documentation:
|
||||
http://www.sphinx-doc.org/en/master/config
|
||||
-- Path setup --------------------------------------------------------------
|
||||
If extensions (or modules to document with autodoc) are in another directory,
|
||||
add these directories to sys.path here.
|
||||
If the directory is relative to the documentation root,
|
||||
use os.path.abspath to make it absolute, like shown here.
|
||||
|
||||
"""
|
||||
import sphinx_rtd_theme
|
||||
|
||||
|
||||
# -- Project information -----------------------------------------------------
|
||||
|
||||
project = 'MediaPipe'
|
||||
author = 'Google LLC'
|
||||
|
||||
# The full version, including alpha/beta/rc tags
|
||||
release = 'v0.5'
|
||||
|
||||
|
||||
# -- General configuration ---------------------------------------------------
|
||||
|
||||
# Add any Sphinx extension module names here, as strings. They can be
|
||||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
extensions = [
|
||||
'recommonmark'
|
||||
]
|
||||
|
||||
master_doc = 'index'
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
|
||||
# List of patterns, relative to source directory, that match files and
|
||||
# directories to ignore when looking for source files.
|
||||
# This pattern also affects html_static_path and html_extra_path.
|
||||
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
|
||||
|
||||
|
||||
# -- Options for HTML output -------------------------------------------------
|
||||
|
||||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||
# a list of builtin themes.
|
||||
#
|
||||
html_theme = 'sphinx_rtd_theme'
|
||||
|
||||
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
|
||||
|
||||
# Add any paths that contain custom static files (such as style sheets) here,
|
||||
# relative to this directory. They are copied after the builtin static files,
|
||||
# so a file named "default.css" will overwrite the builtin "default.css".
|
||||
html_static_path = ['_static']
|
||||
@@ -1,128 +0,0 @@
|
||||
# Cycles in MediaPipe Graphs
|
||||
|
||||
<!-- TODO: add discussion of PreviousLoopbackCalculator -->
|
||||
|
||||
[TOC]
|
||||
|
||||
By default, MediaPipe requires calculator graphs to be acyclic and treats cycles
|
||||
in a graph as errors. If a graph is intended to have cycles, the cycles need to
|
||||
be annotated in the graph config. This page describes how to do that.
|
||||
|
||||
NOTE: The current approach is experimental and subject to change. We welcome
|
||||
your feedback.
|
||||
|
||||
Please use the `CalculatorGraphTest.Cycle` unit test in
|
||||
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown
|
||||
below is the cyclic graph in the test. The `sum` output of the adder is the sum
|
||||
of the integers generated by the integer source calculator.
|
||||
|
||||
|
||||
|
||||
This simple graph illustrates all the issues in supporting cyclic graphs.
|
||||
|
||||
## Back Edge Annotation
|
||||
|
||||
We require that an edge in each cycle be annotated as a back edge. This allows
|
||||
MediaPipe’s topological sort to work, after removing all the back edges.
|
||||
|
||||
There are usually multiple ways to select the back edges. Which edges are marked
|
||||
as back edges affects which nodes are considered as upstream and which nodes are
|
||||
considered as downstream, which in turn affects the priorities MediaPipe assigns
|
||||
to the nodes.
|
||||
|
||||
For example, the `CalculatorGraphTest.Cycle` test marks the `old_sum` edge as a
|
||||
back edge, so the Delay node is considered as a downstream node of the adder
|
||||
node and is given a higher priority. Alternatively, we could mark the `sum`
|
||||
input to the delay node as the back edge, in which case the delay node would be
|
||||
considered as an upstream node of the adder node and is given a lower priority.
|
||||
|
||||
## Initial Packet
|
||||
|
||||
For the adder calculator to be runnable when the first integer from the integer
|
||||
source arrives, we need an initial packet, with value 0 and with the same
|
||||
timestamp, on the `old_sum` input stream to the adder. This initial packet
|
||||
should be output by the delay calculator in the `Open()` method.
|
||||
|
||||
## Delay in a Loop
|
||||
|
||||
Each loop should incur a delay to align the previous `sum` output with the next
|
||||
integer input. This is also done by the delay node. So the delay node needs to
|
||||
know the following about the timestamps of the integer source calculator:
|
||||
|
||||
* The timestamp of the first output.
|
||||
|
||||
* The timestamp delta between successive outputs.
|
||||
|
||||
We plan to add an alternative scheduling policy that only cares about packet
|
||||
ordering and ignores packet timestamps, which will eliminate this inconvenience.
|
||||
|
||||
## Early Termination of a Calculator When One Input Stream is Done
|
||||
|
||||
By default, MediaPipe calls the `Close()` method of a non-source calculator when
|
||||
all of its input streams are done. In the example graph, we want to stop the
|
||||
adder node as soon as the integer source is done. This is accomplished by
|
||||
configuring the adder node with an alternative input stream handler,
|
||||
`EarlyCloseInputStreamHandler`.
|
||||
|
||||
## Relevant Source Code
|
||||
|
||||
### Delay Calculator
|
||||
|
||||
Note the code in `Open()` that outputs the initial packet and the code in
|
||||
`Process()` that adds a (unit) delay to input packets. As noted above, this
|
||||
delay node assumes that its output stream is used alongside an input stream with
|
||||
packet timestamps 0, 1, 2, 3, ...
|
||||
|
||||
```c++
|
||||
class UnitDelayCalculator : public Calculator {
|
||||
public:
|
||||
static ::util::Status FillExpectations(
|
||||
const CalculatorOptions& extendable_options, PacketTypeSet* inputs,
|
||||
PacketTypeSet* outputs, PacketTypeSet* input_side_packets) {
|
||||
inputs->Index(0)->Set<int>("An integer.");
|
||||
outputs->Index(0)->Set<int>("The input delayed by one time unit.");
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::util::Status Open() final {
|
||||
Output()->Add(new int(0), Timestamp(0));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::util::Status Process() final {
|
||||
const Packet& packet = Input()->Value();
|
||||
Output()->AddPacket(packet.At(packet.Timestamp().NextAllowedInStream()));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Graph Config
|
||||
|
||||
Note the `back_edge` annotation and the alternative `input_stream_handler`.
|
||||
|
||||
```proto
|
||||
node {
|
||||
calculator: 'GlobalCountSourceCalculator'
|
||||
input_side_packet: 'global_counter'
|
||||
output_stream: 'integers'
|
||||
}
|
||||
node {
|
||||
calculator: 'IntAdderCalculator'
|
||||
input_stream: 'integers'
|
||||
input_stream: 'old_sum'
|
||||
input_stream_info: {
|
||||
tag_index: ':1' # 'old_sum'
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: 'sum'
|
||||
input_stream_handler {
|
||||
input_stream_handler: 'EarlyCloseInputStreamHandler'
|
||||
}
|
||||
}
|
||||
node {
|
||||
calculator: 'UnitDelayCalculator'
|
||||
input_stream: 'sum'
|
||||
output_stream: 'old_sum'
|
||||
}
|
||||
```
|
||||
Binary file not shown.
@@ -1,238 +0,0 @@
|
||||
# Examples
|
||||
|
||||
Below are code samples on how to run MediaPipe on both mobile and desktop. We
|
||||
currently support MediaPipe APIs on mobile for Android only but will add support
|
||||
for Objective-C shortly.
|
||||
|
||||
## Mobile
|
||||
|
||||
### Hello World! on Android
|
||||
|
||||
[Hello World! on Android](./hello_world_android.md) should be the first mobile
|
||||
Android example users go through in detail. It teaches the following:
|
||||
|
||||
* Introduction of a simple MediaPipe graph running on mobile GPUs for
|
||||
[Sobel edge detection](https://en.wikipedia.org/wiki/Sobel_operator).
|
||||
* Building a simple baseline Android application that displays "Hello World!".
|
||||
* Adding camera preview support into the baseline application using the
|
||||
Android [CameraX] API.
|
||||
* Incorporating the Sobel edge detection graph to process the live camera
|
||||
preview and display the processed video in real-time.
|
||||
|
||||
### Hello World! on iOS
|
||||
|
||||
[Hello World! on iOS](./hello_world_ios.md) is the iOS version of Sobel edge
|
||||
detection example.
|
||||
|
||||
### Object Detection with GPU
|
||||
|
||||
[Object Detection with GPU](./object_detection_mobile_gpu.md) illustrates how to
|
||||
use MediaPipe with a TFLite model for object detection in a GPU-accelerated
|
||||
pipeline.
|
||||
|
||||
* [Android](./object_detection_mobile_gpu.md)
|
||||
* [iOS](./object_detection_mobile_gpu.md)
|
||||
|
||||
### Object Detection with CPU
|
||||
|
||||
[Object Detection with CPU](./object_detection_mobile_cpu.md) illustrates using
|
||||
the same TFLite model in a CPU-based pipeline. This example highlights how
|
||||
graphs can be easily adapted to run on CPU v.s. GPU.
|
||||
|
||||
### Object Detection and Tracking with GPU
|
||||
|
||||
[Object Detection and Tracking with GPU](./object_tracking_mobile_gpu.md) illustrates how to
|
||||
use MediaPipe for object detection and tracking.
|
||||
|
||||
### Objectron: 3D Object Detection and Tracking with GPU
|
||||
|
||||
[MediaPipe Objectron is 3D Object Detection with GPU](./objectron_mobile_gpu.md)
|
||||
illustrates mobile real-time 3D object detection and tracking pipeline for every
|
||||
day objects like shoes and chairs
|
||||
|
||||
* [Android](./objectron_mobile_gpu.md)
|
||||
|
||||
### Face Detection with GPU
|
||||
|
||||
[Face Detection with GPU](./face_detection_mobile_gpu.md) illustrates how to use
|
||||
MediaPipe with a TFLite model for face detection in a GPU-accelerated pipeline.
|
||||
The selfie face detection TFLite model is based on
|
||||
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface),
|
||||
and model details are described in the
|
||||
[model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
|
||||
|
||||
* [Android](./face_detection_mobile_gpu.md)
|
||||
* [iOS](./face_detection_mobile_gpu.md)
|
||||
|
||||
### Face Detection with CPU
|
||||
|
||||
[Face Detection with CPU](./face_detection_mobile_cpu.md) illustrates using the
|
||||
same TFLite model in a CPU-based pipeline. This example highlights how graphs
|
||||
can be easily adapted to run on CPU v.s. GPU.
|
||||
|
||||
* [Android](./face_detection_mobile_cpu.md)
|
||||
* [iOS](./face_detection_mobile_cpu.md)
|
||||
|
||||
### Face Mesh with GPU
|
||||
|
||||
[Face Mesh with GPU](./face_mesh_mobile_gpu.md) illustrates how to run the
|
||||
MediaPipe Face Mesh pipeline to perform 3D face landmark estimation in real-time
|
||||
on mobile devices, utilizing GPU acceleration. The pipeline is based on
|
||||
["Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs"](https://arxiv.org/abs/1907.06724),
|
||||
and details of the underlying ML models are described in the
|
||||
[model card](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view).
|
||||
|
||||
* [Android](./face_mesh_mobile_gpu.md)
|
||||
* [iOS](./face_mesh_mobile_gpu.md)
|
||||
|
||||
### Hand Detection with GPU
|
||||
|
||||
[Hand Detection with GPU](./hand_detection_mobile_gpu.md) illustrates how to use
|
||||
MediaPipe with a TFLite model for hand detection in a GPU-accelerated pipeline.
|
||||
|
||||
* [Android](./hand_detection_mobile_gpu.md)
|
||||
* [iOS](./hand_detection_mobile_gpu.md)
|
||||
|
||||
### Hand Tracking with GPU
|
||||
|
||||
[Hand Tracking with GPU](./hand_tracking_mobile_gpu.md) illustrates how to use
|
||||
MediaPipe with TFLite models for hand tracking in a GPU-accelerated pipeline.
|
||||
|
||||
* [Android](./hand_tracking_mobile_gpu.md)
|
||||
* [iOS](./hand_tracking_mobile_gpu.md)
|
||||
|
||||
### Multi-Hand Tracking with GPU
|
||||
|
||||
[Multi-Hand Tracking with GPU](./multi_hand_tracking_mobile_gpu.md) illustrates
|
||||
how to use MediaPipe with TFLite models for multi-hand tracking in a
|
||||
GPU-accelerated pipeline.
|
||||
|
||||
* [Android](./multi_hand_tracking_mobile_gpu.md)
|
||||
* [iOS](./multi_hand_tracking_mobile_gpu.md)
|
||||
|
||||
### Hair Segmentation with GPU
|
||||
|
||||
[Hair Segmentation on GPU](./hair_segmentation_mobile_gpu.md) illustrates how to
|
||||
use MediaPipe with a TFLite model for hair segmentation in a GPU-accelerated
|
||||
pipeline. The selfie hair segmentation TFLite model is based on
|
||||
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation),
|
||||
and model details are described in the
|
||||
[model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
|
||||
|
||||
* [Android](./hair_segmentation_mobile_gpu.md)
|
||||
|
||||
### Template Matching using KNIFT with CPU
|
||||
|
||||
[Template Matching using KNIFT on Mobile](./template_matching_mobile_cpu.md)
|
||||
shows how to use MediaPipe with TFLite model for template matching using Knift
|
||||
on mobile using CPU.
|
||||
|
||||
* [Android](./template_matching_mobile_cpu.md)
|
||||
|
||||
## Desktop
|
||||
|
||||
### Hello World for C++
|
||||
|
||||
[Hello World for C++](./hello_world_desktop.md) shows how to run a simple graph
|
||||
using the MediaPipe C++ APIs.
|
||||
|
||||
### Feature Extraction and Model Inference for YouTube-8M Challenge
|
||||
|
||||
[Feature Extraction and Model Inference for YouTube-8M Challenge](./youtube_8m.md)
|
||||
shows how to use MediaPipe to prepare training data for the YouTube-8M Challenge
|
||||
and do the model inference with the baseline model.
|
||||
|
||||
### Preparing Data Sets with MediaSequence
|
||||
|
||||
[Preparing Data Sets with MediaSequence](./media_sequence.md) shows how to use
|
||||
MediaPipe for media processing to prepare video data sets for training a
|
||||
TensorFlow model.
|
||||
|
||||
### AutoFlip - Automatic video cropping
|
||||
|
||||
[AutoFlip](./autoflip.md) shows how to use MediaPipe to build an automatic video
|
||||
cropping pipeline that can convert an input video to arbitrary aspect ratios.
|
||||
|
||||
### Object Detection on Desktop
|
||||
|
||||
[Object Detection on Desktop](./object_detection_desktop.md) shows how to run
|
||||
object detection models (TensorFlow and TFLite) using the MediaPipe C++ APIs.
|
||||
|
||||
[Sobel edge detection]:https://en.wikipedia.org/wiki/Sobel_operator
|
||||
[CameraX]:https://developer.android.com/training/camerax
|
||||
|
||||
### Face Detection on Desktop with Webcam
|
||||
|
||||
[Face Detection on Desktop with Webcam](./face_detection_desktop.md) shows how
|
||||
to use MediaPipe with a TFLite model for face detection on desktop using CPU or
|
||||
GPU with live video from a webcam.
|
||||
|
||||
* [Desktop GPU](./face_detection_desktop.md)
|
||||
* [Desktop CPU](./face_detection_desktop.md)
|
||||
|
||||
### Face Mesh on Desktop with Webcam
|
||||
|
||||
[Face Mesh on Desktop with Webcam](./face_mesh_desktop.md) shows how to run the
|
||||
MediaPipe Face Mesh pipeline to perform 3D face landmark estimation in real-time
|
||||
on desktop with webcam input.
|
||||
|
||||
* [Desktop GPU](./face_mesh_desktop.md)
|
||||
* [Desktop CPU](./face_mesh_desktop.md)
|
||||
|
||||
### Hand Tracking on Desktop with Webcam
|
||||
|
||||
[Hand Tracking on Desktop with Webcam](./hand_tracking_desktop.md) shows how to
|
||||
use MediaPipe with TFLite models for hand tracking on desktop using CPU or GPU
|
||||
with live video from a webcam.
|
||||
|
||||
* [Desktop GPU](./hand_tracking_desktop.md)
|
||||
* [Desktop CPU](./hand_tracking_desktop.md)
|
||||
|
||||
### Multi-Hand Tracking on Desktop with Webcam
|
||||
|
||||
[Multi-Hand Tracking on Desktop with Webcam](./multi_hand_tracking_desktop.md)
|
||||
shows how to use MediaPipe with TFLite models for multi-hand tracking on desktop
|
||||
using CPU or GPU with live video from a webcam.
|
||||
|
||||
* [Desktop GPU](./multi_hand_tracking_desktop.md)
|
||||
* [Desktop CPU](./multi_hand_tracking_desktop.md)
|
||||
|
||||
### Hair Segmentation on Desktop with Webcam
|
||||
|
||||
[Hair Segmentation on Desktop with Webcam](./hair_segmentation_desktop.md) shows
|
||||
how to use MediaPipe with a TFLite model for hair segmentation on desktop using
|
||||
GPU with live video from a webcam.
|
||||
|
||||
* [Desktop GPU](./hair_segmentation_desktop.md)
|
||||
|
||||
## Google Coral (ML acceleration with Google EdgeTPU)
|
||||
|
||||
Below are code samples on how to run MediaPipe on Google Coral Dev Board.
|
||||
|
||||
### Object Detection on Coral
|
||||
|
||||
[Object Detection on Coral with Webcam](./object_detection_coral_devboard.md)
|
||||
shows how to run quantized object detection TFlite model accelerated with
|
||||
EdgeTPU on
|
||||
[Google Coral Dev Board](https://coral.withgoogle.com/products/dev-board).
|
||||
|
||||
### Face Detection on Coral
|
||||
|
||||
[Face Detection on Coral with Webcam](./face_detection_coral_devboard.md) shows
|
||||
how to use quantized face detection TFlite model accelerated with EdgeTPU on
|
||||
[Google Coral Dev Board](https://coral.withgoogle.com/products/dev-board).
|
||||
|
||||
|
||||
## Web Browser
|
||||
|
||||
Below are samples that can directly be run in your web browser.
|
||||
See more details in [MediaPipe on the Web](./web.md) and
|
||||
[Google Developer blog post](https://mediapipe.page.link/webdevblog)
|
||||
|
||||
### [Face Detection In Browser](https://viz.mediapipe.dev/demo/face_detection)
|
||||
|
||||
### [Hand Detection In Browser](https://viz.mediapipe.dev/demo/hand_detection)
|
||||
|
||||
### [Hand Tracking In Browser](https://viz.mediapipe.dev/demo/hand_tracking)
|
||||
|
||||
### [Hair Segmentation In Browser](https://viz.mediapipe.dev/demo/hair_segmentation)
|
||||
@@ -1,23 +0,0 @@
|
||||
## Face Detection on Coral with Webcam
|
||||
|
||||
MediaPipe is able to run cross platform across device types like desktop, mobile
|
||||
and edge devices. Here is an example of running MediaPipe
|
||||
[face detection pipeline](./face_detection_desktop.md) on edge device like the
|
||||
[Coral Dev Board](https://coral.ai/products/dev-board).
|
||||
|
||||
This MediaPipe Coral face
|
||||
detection pipeline is running [coral specific quantized version](https://github.com/google/mediapipe/blob/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
|
||||
of the [MediaPipe face detection TFLite model](https://github.com/google/mediapipe/blob/master/mediapipe/models/face_detection_front.tflite)
|
||||
accelerated on Edge TPU.
|
||||
|
||||
### Cross compilation of MediaPipe Coral binaries in Docker
|
||||
|
||||
We recommend building the MediaPipe binaries not on the edge device due to
|
||||
limited compute resulting in long build times. Instead, we will build MediaPipe
|
||||
binaries using Docker containers on a more powerful host machine.
|
||||
|
||||
For step by
|
||||
step details of cross compiling and running MediaPipe binaries on the Coral Dev
|
||||
Board, please refer to [README.md in MediaPipe Coral example folder](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral).
|
||||
|
||||
|
||||
@@ -1,264 +1,2 @@
|
||||
## Face Detection on Desktop
|
||||
|
||||
This is an example of using MediaPipe to run face detection models (TensorFlow
|
||||
Lite) and render bounding boxes on the detected faces. To know more about the
|
||||
face detection models, please refer to the model [`README file`]. Moreover, if
|
||||
you are interested in running the same TensorfFlow Lite model on Android/iOS,
|
||||
please see the
|
||||
[Face Detection on GPU on Android/iOS](face_detection_mobile_gpu.md) and
|
||||
[Face Detection on CPU on Android/iOS](face_detection_mobile_cpu.md) examples.
|
||||
|
||||
We show the face detection demos with TensorFlow Lite model using the Webcam:
|
||||
|
||||
- [TensorFlow Lite Face Detection Demo with Webcam (CPU)](#tensorflow-lite-face-detection-demo-with-webcam-cpu)
|
||||
|
||||
- [TensorFlow Lite Face Detection Demo with Webcam (GPU)](#tensorflow-lite-face-detection-demo-with-webcam-gpu)
|
||||
|
||||
Note: If MediaPipe depends on OpenCV 2, please see the
|
||||
[known issues with OpenCV 2](./object_detection_desktop.md#known-issues-with-opencv-2)
|
||||
section.
|
||||
|
||||
### TensorFlow Lite Face Detection Demo with Webcam (CPU)
|
||||
|
||||
To build and run the TensorFlow Lite example on desktop (CPU) with Webcam, run:
|
||||
|
||||
```bash
|
||||
# Video from webcam running on desktop CPU
|
||||
$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
|
||||
mediapipe/examples/desktop/face_detection:face_detection_cpu
|
||||
|
||||
# It should print:
|
||||
# Target //mediapipe/examples/desktop/face_detection:face_detection_cpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/face_detection/face_detection_cpu
|
||||
# INFO: Elapsed time: 36.417s, Critical Path: 23.22s
|
||||
# INFO: 711 processes: 710 linux-sandbox, 1 local.
|
||||
# INFO: Build completed successfully, 734 total actions
|
||||
|
||||
# This will open up your webcam as long as it is connected and on
|
||||
# Any errors is likely due to your webcam being not accessible
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/face_detection/face_detection_cpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/face_detection/face_detection_desktop_live.pbtxt
|
||||
```
|
||||
|
||||
### TensorFlow Lite Face Detection Demo with Webcam (GPU)
|
||||
|
||||
Note: This currently works only on Linux, and please first follow
|
||||
[OpenGL ES Setup on Linux Desktop](./gpu.md#opengl-es-setup-on-linux-desktop).
|
||||
|
||||
To build and run the TensorFlow Lite example on desktop (GPU) with Webcam, run:
|
||||
|
||||
```bash
|
||||
# Video from webcam running on desktop GPU
|
||||
# This works only for Linux currently
|
||||
$ bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
|
||||
mediapipe/examples/desktop/face_detection:face_detection_gpu
|
||||
|
||||
# It should print:
|
||||
# Target //mediapipe/examples/desktop/face_detection:face_detection_gpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/face_detection/face_detection_gpu
|
||||
# INFO: Elapsed time: 36.417s, Critical Path: 23.22s
|
||||
# INFO: 711 processes: 710 linux-sandbox, 1 local.
|
||||
# INFO: Build completed successfully, 734 total actions
|
||||
|
||||
# This will open up your webcam as long as it is connected and on
|
||||
# Any errors is likely due to your webcam being not accessible,
|
||||
# or GPU drivers not setup properly.
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/face_detection/face_detection_gpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt
|
||||
```
|
||||
|
||||
#### Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev).
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on CPU & GPU.
|
||||
# Used in the examples in
|
||||
# mediapipe/examples/desktop/face_detection:face_detection_cpu.
|
||||
|
||||
# Images on CPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transforms the input image on CPU to a 128x128 image. To scale the input
|
||||
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
|
||||
# resulting in potential letterboxing in the transformed image.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
output_stream: "IMAGE:transformed_input_video_cpu"
|
||||
output_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 128
|
||||
output_height: 128
|
||||
scale_mode: FIT
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts the transformed input image on CPU into an image tensor stored as a
|
||||
# TfLiteTensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE:transformed_input_video_cpu"
|
||||
output_stream: "TENSORS:image_tensor"
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
|
||||
# vector of tensors representing, for instance, detection boxes/keypoints and
|
||||
# scores.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS:image_tensor"
|
||||
output_stream: "TENSORS:detection_tensors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/models/face_detection_front.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a vector of SSD anchors based on
|
||||
# the specification in the options.
|
||||
node {
|
||||
calculator: "SsdAnchorsCalculator"
|
||||
output_side_packet: "anchors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
|
||||
num_layers: 4
|
||||
min_scale: 0.1484375
|
||||
max_scale: 0.75
|
||||
input_size_height: 128
|
||||
input_size_width: 128
|
||||
anchor_offset_x: 0.5
|
||||
anchor_offset_y: 0.5
|
||||
strides: 8
|
||||
strides: 16
|
||||
strides: 16
|
||||
strides: 16
|
||||
aspect_ratios: 1.0
|
||||
fixed_anchor_size: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
|
||||
# the SSD anchors and the specification in the options, into a vector of
|
||||
# detections. Each detection describes a detected object.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToDetectionsCalculator"
|
||||
input_stream: "TENSORS:detection_tensors"
|
||||
input_side_packet: "ANCHORS:anchors"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
|
||||
num_classes: 1
|
||||
num_boxes: 896
|
||||
num_coords: 16
|
||||
box_coord_offset: 0
|
||||
keypoint_coord_offset: 4
|
||||
num_keypoints: 6
|
||||
num_values_per_keypoint: 2
|
||||
sigmoid_score: true
|
||||
score_clipping_thresh: 100.0
|
||||
reverse_output_order: true
|
||||
x_scale: 128.0
|
||||
y_scale: 128.0
|
||||
h_scale: 128.0
|
||||
w_scale: 128.0
|
||||
min_score_thresh: 0.75
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Performs non-max suppression to remove excessive detections.
|
||||
node {
|
||||
calculator: "NonMaxSuppressionCalculator"
|
||||
input_stream: "detections"
|
||||
output_stream: "filtered_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
|
||||
min_suppression_threshold: 0.3
|
||||
overlap_type: INTERSECTION_OVER_UNION
|
||||
algorithm: WEIGHTED
|
||||
return_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Maps detection label IDs to the corresponding label text ("Face"). The label
|
||||
# map is provided in the label_map_path option.
|
||||
node {
|
||||
calculator: "DetectionLabelIdToTextCalculator"
|
||||
input_stream: "filtered_detections"
|
||||
output_stream: "labeled_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
|
||||
label_map_path: "mediapipe/models/face_detection_front_labelmap.txt"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
|
||||
# letterboxed image (after image transformation with the FIT scale mode) to the
|
||||
# corresponding locations on the same image with the letterbox removed (the
|
||||
# input image to the graph before image transformation).
|
||||
node {
|
||||
calculator: "DetectionLetterboxRemovalCalculator"
|
||||
input_stream: "DETECTIONS:labeled_detections"
|
||||
input_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
output_stream: "DETECTIONS:output_detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:output_detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE:output_video"
|
||||
}
|
||||
```
|
||||
|
||||
[`README file`]:https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md
|
||||
Content moved to
|
||||
[MediapPipe Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
||||
|
||||
@@ -1,246 +0,0 @@
|
||||
# Face Detection (CPU)
|
||||
|
||||
This doc focuses on the
|
||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_cpu.pbtxt)
|
||||
that performs face detection with TensorFlow Lite on CPU.
|
||||
|
||||
|
||||
|
||||
## Android
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu)
|
||||
|
||||
To build and install the app:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu
|
||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu/facedetectioncpu.apk
|
||||
```
|
||||
|
||||
## iOS
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectioncpu).
|
||||
|
||||
See the general [instructions](./building_examples.md#ios) for building iOS
|
||||
examples and generating an Xcode project. This will be the FaceDetectionCpuApp
|
||||
target.
|
||||
|
||||
To build on the command line:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectioncpu:FaceDetectionCpuApp
|
||||
```
|
||||
|
||||
## Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_cpu.pbtxt)
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on CPU.
|
||||
# Used in the examples in
|
||||
# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectioncpu and
|
||||
# mediapipie/examples/ios/facedetectioncpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transfers the input image from GPU to CPU memory for the purpose of
|
||||
# demonstrating a CPU-based pipeline. Note that the input image on GPU has the
|
||||
# origin defined at the bottom-left corner (OpenGL convention). As a result,
|
||||
# the transferred image on CPU also shares the same representation.
|
||||
node: {
|
||||
calculator: "GpuBufferToImageFrameCalculator"
|
||||
input_stream: "throttled_input_video"
|
||||
output_stream: "input_video_cpu"
|
||||
}
|
||||
|
||||
# Transforms the input image on CPU to a 128x128 image. To scale the input
|
||||
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
|
||||
# resulting in potential letterboxing in the transformed image.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE:input_video_cpu"
|
||||
output_stream: "IMAGE:transformed_input_video_cpu"
|
||||
output_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 128
|
||||
output_height: 128
|
||||
scale_mode: FIT
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts the transformed input image on CPU into an image tensor stored as a
|
||||
# TfLiteTensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE:transformed_input_video_cpu"
|
||||
output_stream: "TENSORS:image_tensor"
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
|
||||
# vector of tensors representing, for instance, detection boxes/keypoints and
|
||||
# scores.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS:image_tensor"
|
||||
output_stream: "TENSORS:detection_tensors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/models/face_detection_front.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a vector of SSD anchors based on
|
||||
# the specification in the options.
|
||||
node {
|
||||
calculator: "SsdAnchorsCalculator"
|
||||
output_side_packet: "anchors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
|
||||
num_layers: 4
|
||||
min_scale: 0.1484375
|
||||
max_scale: 0.75
|
||||
input_size_height: 128
|
||||
input_size_width: 128
|
||||
anchor_offset_x: 0.5
|
||||
anchor_offset_y: 0.5
|
||||
strides: 8
|
||||
strides: 16
|
||||
strides: 16
|
||||
strides: 16
|
||||
aspect_ratios: 1.0
|
||||
fixed_anchor_size: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
|
||||
# the SSD anchors and the specification in the options, into a vector of
|
||||
# detections. Each detection describes a detected object.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToDetectionsCalculator"
|
||||
input_stream: "TENSORS:detection_tensors"
|
||||
input_side_packet: "ANCHORS:anchors"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
|
||||
num_classes: 1
|
||||
num_boxes: 896
|
||||
num_coords: 16
|
||||
box_coord_offset: 0
|
||||
keypoint_coord_offset: 4
|
||||
num_keypoints: 6
|
||||
num_values_per_keypoint: 2
|
||||
sigmoid_score: true
|
||||
score_clipping_thresh: 100.0
|
||||
reverse_output_order: true
|
||||
x_scale: 128.0
|
||||
y_scale: 128.0
|
||||
h_scale: 128.0
|
||||
w_scale: 128.0
|
||||
min_score_thresh: 0.75
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Performs non-max suppression to remove excessive detections.
|
||||
node {
|
||||
calculator: "NonMaxSuppressionCalculator"
|
||||
input_stream: "detections"
|
||||
output_stream: "filtered_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
|
||||
min_suppression_threshold: 0.3
|
||||
overlap_type: INTERSECTION_OVER_UNION
|
||||
algorithm: WEIGHTED
|
||||
return_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Maps detection label IDs to the corresponding label text ("Face"). The label
|
||||
# map is provided in the label_map_path option.
|
||||
node {
|
||||
calculator: "DetectionLabelIdToTextCalculator"
|
||||
input_stream: "filtered_detections"
|
||||
output_stream: "labeled_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
|
||||
label_map_path: "mediapipe/models/face_detection_front_labelmap.txt"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
|
||||
# letterboxed image (after image transformation with the FIT scale mode) to the
|
||||
# corresponding locations on the same image with the letterbox removed (the
|
||||
# input image to the graph before image transformation).
|
||||
node {
|
||||
calculator: "DetectionLetterboxRemovalCalculator"
|
||||
input_stream: "DETECTIONS:labeled_detections"
|
||||
input_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
output_stream: "DETECTIONS:output_detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:output_detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE:input_video_cpu"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE:output_video_cpu"
|
||||
}
|
||||
|
||||
# Transfers the annotated image from CPU back to GPU memory, to be sent out of
|
||||
# the graph.
|
||||
node: {
|
||||
calculator: "ImageFrameToGpuBufferCalculator"
|
||||
input_stream: "output_video_cpu"
|
||||
output_stream: "output_video"
|
||||
}
|
||||
```
|
||||
@@ -1,228 +1,2 @@
|
||||
# Face Detection (GPU)
|
||||
|
||||
This doc focuses on the
|
||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
|
||||
that performs face detection with TensorFlow Lite on GPU.
|
||||
|
||||
|
||||
|
||||
## Android
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu)
|
||||
|
||||
To build and install the app:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu
|
||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.apk
|
||||
```
|
||||
|
||||
## iOS
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu).
|
||||
|
||||
See the general [instructions](./building_examples.md#ios) for building iOS
|
||||
examples and generating an Xcode project. This will be the FaceDetectionGpuApp
|
||||
target.
|
||||
|
||||
To build on the command line:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp
|
||||
```
|
||||
|
||||
## Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
|
||||
# Used in the examples in
|
||||
# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectiongpu and
|
||||
# mediapipie/examples/ios/facedetectiongpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transforms the input image on GPU to a 128x128 image. To scale the input
|
||||
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
|
||||
# resulting in potential letterboxing in the transformed image.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
output_stream: "IMAGE_GPU:transformed_input_video"
|
||||
output_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 128
|
||||
output_height: 128
|
||||
scale_mode: FIT
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts the transformed input image on GPU into an image tensor stored as a
|
||||
# TfLiteTensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE_GPU:transformed_input_video"
|
||||
output_stream: "TENSORS_GPU:image_tensor"
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
|
||||
# vector of tensors representing, for instance, detection boxes/keypoints and
|
||||
# scores.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS_GPU:image_tensor"
|
||||
output_stream: "TENSORS:detection_tensors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "face_detection_front.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a vector of SSD anchors based on
|
||||
# the specification in the options.
|
||||
node {
|
||||
calculator: "SsdAnchorsCalculator"
|
||||
output_side_packet: "anchors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
|
||||
num_layers: 4
|
||||
min_scale: 0.1484375
|
||||
max_scale: 0.75
|
||||
input_size_height: 128
|
||||
input_size_width: 128
|
||||
anchor_offset_x: 0.5
|
||||
anchor_offset_y: 0.5
|
||||
strides: 8
|
||||
strides: 16
|
||||
strides: 16
|
||||
strides: 16
|
||||
aspect_ratios: 1.0
|
||||
fixed_anchor_size: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
|
||||
# the SSD anchors and the specification in the options, into a vector of
|
||||
# detections. Each detection describes a detected object.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToDetectionsCalculator"
|
||||
input_stream: "TENSORS:detection_tensors"
|
||||
input_side_packet: "ANCHORS:anchors"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
|
||||
num_classes: 1
|
||||
num_boxes: 896
|
||||
num_coords: 16
|
||||
box_coord_offset: 0
|
||||
keypoint_coord_offset: 4
|
||||
num_keypoints: 6
|
||||
num_values_per_keypoint: 2
|
||||
sigmoid_score: true
|
||||
score_clipping_thresh: 100.0
|
||||
reverse_output_order: true
|
||||
x_scale: 128.0
|
||||
y_scale: 128.0
|
||||
h_scale: 128.0
|
||||
w_scale: 128.0
|
||||
min_score_thresh: 0.75
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Performs non-max suppression to remove excessive detections.
|
||||
node {
|
||||
calculator: "NonMaxSuppressionCalculator"
|
||||
input_stream: "detections"
|
||||
output_stream: "filtered_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
|
||||
min_suppression_threshold: 0.3
|
||||
overlap_type: INTERSECTION_OVER_UNION
|
||||
algorithm: WEIGHTED
|
||||
return_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Maps detection label IDs to the corresponding label text ("Face"). The label
|
||||
# map is provided in the label_map_path option.
|
||||
node {
|
||||
calculator: "DetectionLabelIdToTextCalculator"
|
||||
input_stream: "filtered_detections"
|
||||
output_stream: "labeled_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
|
||||
label_map_path: "face_detection_front_labelmap.txt"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
|
||||
# letterboxed image (after image transformation with the FIT scale mode) to the
|
||||
# corresponding locations on the same image with the letterbox removed (the
|
||||
# input image to the graph before image transformation).
|
||||
node {
|
||||
calculator: "DetectionLetterboxRemovalCalculator"
|
||||
input_stream: "DETECTIONS:labeled_detections"
|
||||
input_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
output_stream: "DETECTIONS:output_detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:output_detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 10.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE_GPU:output_video"
|
||||
}
|
||||
```
|
||||
Content moved to
|
||||
[MediapPipe Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
## Face Mesh on Desktop with Webcam
|
||||
|
||||
This doc focuses on running the **MediaPipe Face Mesh** pipeline to perform 3D
|
||||
face landmark estimation in real-time on desktop with webcam input. The pipeline
|
||||
internally incorporates TensorFlow Lite models. To know more about the models,
|
||||
please refer to the model
|
||||
[README file](https://github.com/google/mediapipe/tree/master/mediapipe/models/README.md#face-mesh).
|
||||
Moreover, if you are interested in running the same pipeline on Android/iOS,
|
||||
please see [Face Mesh on Android/iOS](face_mesh_mobile_gpu.md).
|
||||
|
||||
- [Face Mesh on Desktop with Webcam (CPU)](#face-mesh-on-desktop-with-webcam-cpu)
|
||||
|
||||
- [Face Mesh on Desktop with Webcam (GPU)](#face-mesh-on-desktop-with-webcam-gpu)
|
||||
|
||||
Note: If MediaPipe depends on OpenCV 2, please see the [known issues with OpenCV 2](#known-issues-with-opencv-2) section.
|
||||
|
||||
### Face Mesh on Desktop with Webcam (CPU)
|
||||
|
||||
To build and run Face Mesh on desktop with webcam (CPU), run:
|
||||
|
||||
```bash
|
||||
$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
|
||||
mediapipe/examples/desktop/face_mesh:face_mesh_cpu
|
||||
|
||||
# It should print:
|
||||
# Target //mediapipe/examples/desktop/face_mesh:face_mesh_cpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_cpu
|
||||
|
||||
# This will open up your webcam as long as it is connected. Errors are likely
|
||||
# due to your webcam being not accessible.
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_cpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt
|
||||
```
|
||||
|
||||
### Face Mesh on Desktop with Webcam (GPU)
|
||||
|
||||
Note: This currently works only on Linux, and please first follow
|
||||
[OpenGL ES Setup on Linux Desktop](./gpu.md#opengl-es-setup-on-linux-desktop).
|
||||
|
||||
To build and run Face Mesh on desktop with webcam (GPU), run:
|
||||
|
||||
```bash
|
||||
# This works only for Linux currently
|
||||
$ bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
|
||||
mediapipe/examples/desktop/face_mesh:face_mesh_gpu
|
||||
|
||||
# It should print:
|
||||
# Target //mediapipe/examples/desktop/face_mesh:face_mesh_gpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_gpu
|
||||
|
||||
# This will open up your webcam as long as it is connected. Errors are likely
|
||||
# due to your webcam being not accessible, or GPU drivers not setup properly.
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_gpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/face_mesh/face_mesh_desktop_live_gpu.pbtxt
|
||||
```
|
||||
@@ -1,90 +0,0 @@
|
||||
# Face Mesh (GPU)
|
||||
|
||||
This example focuses on running the **MediaPipe Face Mesh** pipeline on mobile
|
||||
devices to perform 3D face landmark estimation in real-time, utilizing GPU
|
||||
acceleration. The pipeline internally incorporates TensorFlow Lite models. To
|
||||
know more about the models, please refer to the model
|
||||
[README file](https://github.com/google/mediapipe/tree/master/mediapipe/models/README.md#face-mesh).
|
||||
The pipeline is related to the
|
||||
[face detection example](./face_detection_mobile_gpu.md) as it internally
|
||||
utilizes face detection and performs landmark estimation only within the
|
||||
detected region.
|
||||
|
||||
|
||||
|
||||
**MediaPipe Face Mesh** generates 468 3D face landmarks in real-time on mobile
|
||||
devices. In the visualization above, the red dots represent the landmarks, and
|
||||
the green lines connecting landmarks illustrate the contours around the eyes,
|
||||
eyebrows, lips and the entire face.
|
||||
|
||||
## Android
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu)
|
||||
|
||||
A prebuilt arm64 APK can be
|
||||
[downloaded here](https://drive.google.com/open?id=1pUmd7CXCL_onYMbsZo5p91cH0oNnR4gi).
|
||||
|
||||
To build the app yourself, run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu
|
||||
```
|
||||
|
||||
Once the app is built, install it on Android device with:
|
||||
|
||||
```bash
|
||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/facemeshgpu.apk
|
||||
```
|
||||
|
||||
## iOS
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facemeshgpu).
|
||||
|
||||
See the general [instructions](./building_examples.md#ios) for building iOS
|
||||
examples and generating an Xcode project. This will be the FaceMeshGpuApp
|
||||
target.
|
||||
|
||||
To build on the command line:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp
|
||||
```
|
||||
|
||||
## Graph
|
||||
|
||||
The face mesh [main graph](#main-graph) utilizes a
|
||||
[face landmark subgraph](#face-landmark-subgraph) from the
|
||||
[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark),
|
||||
and renders using a dedicated [face renderer subgraph](#face-renderer-subgraph).
|
||||
|
||||
The subgraphs show up in the main graph visualization as nodes colored in
|
||||
purple, and the subgraph itself can also be visualized just like a regular
|
||||
graph. For more information on how to visualize a graph that includes subgraphs,
|
||||
see the Visualizing Subgraphs section in the
|
||||
[visualizer documentation](./visualizer.md).
|
||||
|
||||
### Main Graph
|
||||
|
||||
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt)
|
||||
|
||||
### Face Landmark Subgraph
|
||||
|
||||
The
|
||||
[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark)
|
||||
contains several subgraphs that can be used to detect and track face landmarks.
|
||||
In particular, in this example the
|
||||
[FaceLandmarkFrontGPU](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
|
||||
subgraph, suitable for images from front-facing cameras (i.e., selfie images)
|
||||
and utilizing GPU acceleration, is selected.
|
||||
|
||||
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
|
||||
|
||||
### Face Renderer Subgraph
|
||||
|
||||
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/subgraphs/face_renderer_gpu.pbtxt)
|
||||
@@ -1,334 +0,0 @@
|
||||
## Framework Concepts
|
||||
|
||||
- [CalculatorBase](#calculatorbase)
|
||||
- [Life of a Calculator](#life-of-a-calculator)
|
||||
- [Identifying inputs and outputs](#identifying-inputs-and-outputs)
|
||||
- [Processing](#processing)
|
||||
- [GraphConfig](#graphconfig)
|
||||
- [Subgraph](#subgraph)
|
||||
|
||||
Each calculator is a node of a graph. We describe how to create a new
|
||||
calculator, how to initialize a calculator, how to perform its calculations,
|
||||
input and output streams, timestamps, and options. Each node in the graph is
|
||||
implemented as a `Calculator`. The bulk of graph execution happens inside its
|
||||
calculators. A calculator may receive zero or more input streams and/or side
|
||||
packets and produces zero or more output streams and/or side packets.
|
||||
|
||||
### CalculatorBase
|
||||
|
||||
A calculator is created by defining a new sub-class of the
|
||||
[`CalculatorBase`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.cc)
|
||||
class, implementing a number of methods, and registering the new sub-class with
|
||||
Mediapipe. At a minimum, a new calculator must implement the below four methods
|
||||
|
||||
* `GetContract()`
|
||||
* Calculator authors can specify the expected types of inputs and outputs of a calculator in GetContract(). When a graph is initialized, the framework calls a static method to verify if the packet types of the connected inputs and outputs match the information in this specification.
|
||||
* `Open()`
|
||||
* After a graph starts, the framework calls `Open()`. The input side packets are available to the calculator at this point. `Open()` interprets the node configuration operations (see Section [GraphConfig](#graphconfig)) and prepares the calculator's per-graph-run state. This function may also write packets to calculator outputs. An error during `Open()` can terminate the graph run.
|
||||
* `Process()`
|
||||
* For a calculator with inputs, the framework calls `Process()` repeatedly whenever at least one input stream has a packet available. The framework by default guarantees that all inputs have the same timestamp (see [Framework Architecture](scheduling_sync.md) for more information). Multiple `Process()` calls can be invoked simultaneously when parallel execution is enabled. If an error occurs during `Process()`, the framework calls `Close()` and the graph run terminates.
|
||||
* `Close()`
|
||||
* After all calls to `Process()` finish or when all input streams close, the framework calls `Close()`. This function is always called if `Open()` was called and succeeded and even if the graph run terminated because of an error. No inputs are available via any input streams during `Close()`, but it still has access to input side packets and therefore may write outputs. After `Close()` returns, the calculator should be considered a dead node. The calculator object is destroyed as soon as the graph finishes running.
|
||||
|
||||
The following are code snippets from
|
||||
[CalculatorBase.h](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h).
|
||||
|
||||
```c++
|
||||
class CalculatorBase {
|
||||
public:
|
||||
...
|
||||
|
||||
// The subclasses of CalculatorBase must implement GetContract.
|
||||
// ...
|
||||
static ::MediaPipe::Status GetContract(CalculatorContract* cc);
|
||||
|
||||
// Open is called before any Process() calls, on a freshly constructed
|
||||
// calculator. Subclasses may override this method to perform necessary
|
||||
// setup, and possibly output Packets and/or set output streams' headers.
|
||||
// ...
|
||||
virtual ::MediaPipe::Status Open(CalculatorContext* cc) {
|
||||
return ::MediaPipe::OkStatus();
|
||||
}
|
||||
|
||||
// Processes the incoming inputs. May call the methods on cc to access
|
||||
// inputs and produce outputs.
|
||||
// ...
|
||||
virtual ::MediaPipe::Status Process(CalculatorContext* cc) = 0;
|
||||
|
||||
// Is called if Open() was called and succeeded. Is called either
|
||||
// immediately after processing is complete or after a graph run has ended
|
||||
// (if an error occurred in the graph). ...
|
||||
virtual ::MediaPipe::Status Close(CalculatorContext* cc) {
|
||||
return ::MediaPipe::OkStatus();
|
||||
}
|
||||
|
||||
...
|
||||
};
|
||||
```
|
||||
### Life of a calculator
|
||||
|
||||
During initialization of a MediaPipe graph, the framework calls a
|
||||
`GetContract()` static method to determine what kinds of packets are expected.
|
||||
|
||||
The framework constructs and destroys the entire calculator for each graph run (e.g. once per video or once per image). Expensive or large objects that remain constant across graph runs should be supplied as input side packets so the calculations are not repeated on subsequent runs.
|
||||
|
||||
After initialization, for each run of the graph, the following sequence occurs:
|
||||
|
||||
* `Open()`
|
||||
* `Process()` (repeatedly)
|
||||
* `Close()`
|
||||
|
||||
The framework calls `Open()` to initialize the calculator. `Open()` should interpret any options and set up the calculator's per-graph-run state. `Open()` may obtain input side packets and write packets to calculator outputs. If appropriate, it should call `SetOffset()` to reduce potential packet buffering of input streams.
|
||||
|
||||
If an error occurs during `Open()` or `Process()` (as indicated by one of them returning a non-`Ok ` status), the graph run is terminated with no further calls to the calculator's methods, and the calculator is destroyed.
|
||||
|
||||
For a calculator with inputs, the framework calls `Process()` whenever at least one input has a packet available. The framework guarantees that inputs all have the same timestamp, that timestamps increase with each call to `Process()` and that all packets are delivered. As a consequence, some inputs may not have any packets when `Process()` is called. An input whose packet is missing appears to produce an empty packet (with no timestamp).
|
||||
|
||||
The framework calls `Close()` after all calls to `Process()`. All inputs will have been exhausted, but `Close()` has access to input side packets and may write outputs. After Close returns, the calculator is destroyed.
|
||||
|
||||
Calculators with no inputs are referred to as sources. A source calculator continues to have `Process()` called as long as it returns an `Ok` status. A source calculator indicates that it is exhausted by returning a stop status (i.e. MediaPipe::tool::StatusStop).
|
||||
|
||||
### Identifying inputs and outputs
|
||||
|
||||
The public interface to a calculator consists of a set of input streams and
|
||||
output streams. In a CalculatorGraphConfiguration, the outputs from some
|
||||
calculators are connected to the inputs of other calculators using named
|
||||
streams. Stream names are normally lowercase, while input and output tags are
|
||||
normally UPPERCASE. In the example below, the output with tag name `VIDEO` is
|
||||
connected to the input with tag name `VIDEO_IN` using the stream named
|
||||
`video_stream`.
|
||||
|
||||
```proto
|
||||
# Graph describing calculator SomeAudioVideoCalculator
|
||||
node {
|
||||
calculator: "SomeAudioVideoCalculator"
|
||||
input_stream: "INPUT:combined_input"
|
||||
output_stream: "VIDEO:video_stream"
|
||||
}
|
||||
node {
|
||||
calculator: "SomeVideoCalculator"
|
||||
input_stream: "VIDEO_IN:video_stream"
|
||||
output_stream: "VIDEO_OUT:processed_video"
|
||||
}
|
||||
```
|
||||
|
||||
Input and output streams can be identified by index number, by tag name, or by a
|
||||
combination of tag name and index number. You can see some examples of input and
|
||||
output identifiers in the example below. `SomeAudioVideoCalculator` identifies
|
||||
its video output by tag and its audio outputs by the combination of tag and
|
||||
index. The input with tag `VIDEO` is connected to the stream named
|
||||
`video_stream`. The outputs with tag `AUDIO` and indices `0` and `1` are
|
||||
connected to the streams named `audio_left` and `audio_right`.
|
||||
`SomeAudioCalculator` identifies its audio inputs by index only (no tag needed).
|
||||
|
||||
```proto
|
||||
# Graph describing calculator SomeAudioVideoCalculator
|
||||
node {
|
||||
calculator: "SomeAudioVideoCalculator"
|
||||
input_stream: "combined_input"
|
||||
output_stream: "VIDEO:video_stream"
|
||||
output_stream: "AUDIO:0:audio_left"
|
||||
output_stream: "AUDIO:1:audio_right"
|
||||
}
|
||||
|
||||
node {
|
||||
calculator: "SomeAudioCalculator"
|
||||
input_stream: "audio_left"
|
||||
input_stream: "audio_right"
|
||||
output_stream: "audio_energy"
|
||||
}
|
||||
```
|
||||
|
||||
In the calculator implementation, inputs and outputs are also identified by tag
|
||||
name and index number. In the function below input are output are identified:
|
||||
|
||||
* By index number: The combined input stream is identified simply by index
|
||||
`0`.
|
||||
* By tag name: The video output stream is identified by tag name "VIDEO".
|
||||
* By tag name and index number: The output audio streams are identified by the
|
||||
combination of the tag name `AUDIO` and the index numbers `0` and `1`.
|
||||
|
||||
```c++
|
||||
// c++ Code snippet describing the SomeAudioVideoCalculator GetContract() method
|
||||
class SomeAudioVideoCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
cc->Inputs().Index(0).SetAny();
|
||||
// SetAny() is used to specify that whatever the type of the
|
||||
// stream is, it's acceptable. This does not mean that any
|
||||
// packet is acceptable. Packets in the stream still have a
|
||||
// particular type. SetAny() has the same effect as explicitly
|
||||
// setting the type to be the stream's type.
|
||||
cc->Outputs().Tag("VIDEO").Set<ImageFrame>();
|
||||
cc->Outputs().Get("AUDIO", 0).Set<Matrix>;
|
||||
cc->Outputs().Get("AUDIO", 1).Set<Matrix>;
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
```
|
||||
|
||||
### Processing
|
||||
|
||||
`Process()` called on a non-source node must return `::mediapipe::OkStatus()` to
|
||||
indicate that all went well, or any other status code to signal an error
|
||||
|
||||
If a non-source calculator returns `tool::StatusStop()`, then this signals the
|
||||
graph is being cancelled early. In this case, all source calculators and graph
|
||||
input streams will be closed (and remaining Packets will propagate through the
|
||||
graph).
|
||||
|
||||
A source node in a graph will continue to have `Process()` called on it as long
|
||||
as it returns `::mediapipe::OkStatus(`). To indicate that there is no more data
|
||||
to be generated return `tool::StatusStop()`. Any other status indicates an error
|
||||
has occurred.
|
||||
|
||||
`Close()` returns `::mediapipe::OkStatus()` to indicate success. Any other
|
||||
status indicates a failure.
|
||||
|
||||
Here is the basic `Process()` function. It uses the `Input()` method (which can
|
||||
be used only if the calculator has a single input) to request its input data. It
|
||||
then uses `std::unique_ptr` to allocate the memory needed for the output packet,
|
||||
and does the calculations. When done it releases the pointer when adding it to
|
||||
the output stream.
|
||||
|
||||
```c++
|
||||
::util::Status MyCalculator::Process() {
|
||||
const Matrix& input = Input()->Get<Matrix>();
|
||||
std::unique_ptr<Matrix> output(new Matrix(input.rows(), input.cols()));
|
||||
// do your magic here....
|
||||
// output->row(n) = ...
|
||||
Output()->Add(output.release(), InputTimestamp());
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
```
|
||||
|
||||
### GraphConfig
|
||||
|
||||
A `GraphConfig` is a specification that describes the topology and functionality
|
||||
of a MediaPipe graph. In the specification, a node in the graph represents an
|
||||
instance of a particular calculator. All the necessary configurations of the
|
||||
node, such its type, inputs and outputs must be described in the specification.
|
||||
Description of the node can also include several optional fields, such as
|
||||
node-specific options, input policy and executor, discussed in
|
||||
[Framework Architecture](scheduling_sync.md).
|
||||
|
||||
`GraphConfig` has several other fields to configure the global graph-level
|
||||
settings, eg, graph executor configs, number of threads, and maximum queue size
|
||||
of input streams. Several graph-level settings are useful for tuning the
|
||||
performance of the graph on different platforms (eg, desktop v.s. mobile). For
|
||||
instance, on mobile, attaching a heavy model-inference calculator to a separate
|
||||
executor can improve the performance of a real-time application since this
|
||||
enables thread locality.
|
||||
|
||||
Below is a trivial `GraphConfig` example where we have series of passthrough
|
||||
calculators :
|
||||
|
||||
```proto
|
||||
# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
|
||||
# passthrough calculators.
|
||||
input_stream: "in"
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "in"
|
||||
output_stream: "out1"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out1"
|
||||
output_stream: "out2"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out2"
|
||||
output_stream: "out3"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out3"
|
||||
output_stream: "out4"
|
||||
}
|
||||
```
|
||||
|
||||
### Subgraph
|
||||
|
||||
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
|
||||
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
|
||||
public interface of a subgraph consists of a set of input and output streams
|
||||
similar to a calculator's public interface. The subgraph can then be
|
||||
included in an `CalculatorGraphConfig` as if it were a calculator. When a
|
||||
MediaPipe graph is loaded from a `CalculatorGraphConfig`, each subgraph node is
|
||||
replaced by the corresponding graph of calculators. As a result, the semantics
|
||||
and performance of the subgraph is identical to the corresponding graph of
|
||||
calculators.
|
||||
|
||||
Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`.
|
||||
|
||||
1. Defining the subgraph.
|
||||
|
||||
```proto
|
||||
# This subgraph is defined in two_pass_through_subgraph.pbtxt
|
||||
# and is registered as "TwoPassThroughSubgraph"
|
||||
|
||||
type: "TwoPassThroughSubgraph"
|
||||
input_stream: "out1"
|
||||
output_stream: "out3"
|
||||
|
||||
node {
|
||||
calculator: "PassThroughculator"
|
||||
input_stream: "out1"
|
||||
output_stream: "out2"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughculator"
|
||||
input_stream: "out2"
|
||||
output_stream: "out3"
|
||||
}
|
||||
```
|
||||
|
||||
The public interface to the subgraph consists of:
|
||||
|
||||
* Graph input streams
|
||||
* Graph output streams
|
||||
* Graph input side packets
|
||||
* Graph output side packets
|
||||
|
||||
2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`. The
|
||||
parameter `register_as` defines the component name for the new subgraph.
|
||||
|
||||
```proto
|
||||
# Small section of BUILD file for registering the "TwoPassThroughSubgraph"
|
||||
# subgraph for use by main graph main_pass_throughcals.pbtxt
|
||||
|
||||
mediapipe_simple_subgraph(
|
||||
name = "twopassthrough_subgraph",
|
||||
graph = "twopassthrough_subgraph.pbtxt",
|
||||
register_as = "TwoPassThroughSubgraph",
|
||||
deps = [
|
||||
"//mediapipe/calculators/core:pass_through_calculator",
|
||||
"//mediapipe/framework:calculator_graph",
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
3. Use the subgraph in the main graph.
|
||||
|
||||
```proto
|
||||
# This main graph is defined in main_pass_throughcals.pbtxt
|
||||
# using subgraph called "TwoPassThroughSubgraph"
|
||||
|
||||
input_stream: "in"
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "in"
|
||||
output_stream: "out1"
|
||||
}
|
||||
node {
|
||||
calculator: "TwoPassThroughSubgraph"
|
||||
input_stream: "out1"
|
||||
output_stream: "out3"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out3"
|
||||
output_stream: "out4"
|
||||
}
|
||||
```
|
||||
@@ -1,315 +0,0 @@
|
||||
## Running on GPUs
|
||||
|
||||
- [Overview](#overview)
|
||||
- [OpenGL ES Support](#opengl-es-support)
|
||||
- [Disable OpenGL ES Support](#disable-opengl-es-support)
|
||||
- [OpenGL ES Setup on Linux Desktop](#opengl-es-setup-on-linux-desktop)
|
||||
- [TensorFlow CUDA Support and Setup on Linux Desktop](#tensorflow-cuda-support-and-setup-on-linux-desktop)
|
||||
- [Life of a GPU Calculator](#life-of-a-gpu-calculator)
|
||||
- [GpuBuffer to ImageFrame Converters](#gpubuffer-to-imageframe-converters)
|
||||
|
||||
### Overview
|
||||
|
||||
MediaPipe supports calculator nodes for GPU compute and rendering, and allows combining multiple GPU nodes, as well as mixing them with CPU based calculator nodes. There exist several GPU APIs on mobile platforms (eg, OpenGL ES, Metal and Vulkan). MediaPipe does not attempt to offer a single cross-API GPU abstraction. Individual nodes can be written using different APIs, allowing them to take advantage of platform specific features when needed.
|
||||
|
||||
GPU support is essential for good performance on mobile platforms, especially for real-time video. MediaPipe enables developers to write GPU compatible calculators that support the use of GPU for:
|
||||
|
||||
* On-device real-time processing, not just batch processing
|
||||
* Video rendering and effects, not just analysis
|
||||
|
||||
Below are the design principles for GPU support in MediaPipe
|
||||
|
||||
* GPU-based calculators should be able to occur anywhere in the graph, and not necessarily be used for on-screen rendering.
|
||||
* Transfer of frame data from one GPU-based calculator to another should be fast, and not incur expensive copy operations.
|
||||
* Transfer of frame data between CPU and GPU should be as efficient as the platform allows.
|
||||
* Because different platforms may require different techniques for best performance, the API should allow flexibility in the way things are implemented behind the scenes.
|
||||
* A calculator should be allowed maximum flexibility in using the GPU for all or part of its operation, combining it with the CPU if necessary.
|
||||
|
||||
### OpenGL ES Support
|
||||
|
||||
MediaPipe supports OpenGL ES up to version 3.2 on Android/Linux and up to ES 3.0
|
||||
on iOS. In addition, MediaPipe also supports Metal on iOS.
|
||||
|
||||
OpenGL ES 3.1 or greater is required (on Android/Linux systems) for running
|
||||
machine learning inference calculators and graphs.
|
||||
|
||||
MediaPipe allows graphs to run OpenGL in multiple GL contexts. For example, this
|
||||
can be very useful in graphs that combine a slower GPU inference path (eg, at 10
|
||||
FPS) with a faster GPU rendering path (eg, at 30 FPS): since one GL context
|
||||
corresponds to one sequential command queue, using the same context for both
|
||||
tasks would reduce the rendering frame rate.
|
||||
|
||||
One challenge MediaPipe's use of multiple contexts solves is the ability to
|
||||
communicate across them. An example scenario is one with an input video that is
|
||||
sent to both the rendering and inferences paths, and rendering needs to have
|
||||
access to the latest output from inference.
|
||||
|
||||
An OpenGL context cannot be accessed by multiple threads at the same time.
|
||||
Furthermore, switching the active GL context on the same thread can be slow on
|
||||
some Android devices. Therefore, our approach is to have one dedicated thread
|
||||
per context. Each thread issues GL commands, building up a serial command queue
|
||||
on its context, which is then executed by the GPU asynchronously.
|
||||
|
||||
### Disable OpenGL ES Support
|
||||
|
||||
By default, building MediaPipe (with no special bazel flags) attempts to compile
|
||||
and link against OpenGL ES (and for iOS also Metal) libraries.
|
||||
|
||||
On platforms where OpenGL ES is not available (see also
|
||||
[OpenGL ES Setup on Linux Desktop](#opengl-es-setup-on-linux-desktop)), you
|
||||
should disable OpenGL ES support with:
|
||||
|
||||
```
|
||||
$ bazel build --define MEDIAPIPE_DISABLE_GPU=1 <my-target>
|
||||
```
|
||||
|
||||
Note: On Android and iOS, OpenGL ES is required by MediaPipe framework and the
|
||||
support should never be disabled.
|
||||
|
||||
### OpenGL ES Setup on Linux Desktop
|
||||
|
||||
On Linux desktop with video cards that support OpenGL ES 3.1+, MediaPipe can run
|
||||
GPU compute and rendering and perform TFLite inference on GPU.
|
||||
|
||||
To check if your Linux desktop GPU can run MediaPipe with OpenGL ES:
|
||||
|
||||
```bash
|
||||
$ sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
|
||||
$ sudo apt-get install mesa-utils
|
||||
$ glxinfo | grep -i opengl
|
||||
```
|
||||
|
||||
For example, it may print:
|
||||
|
||||
```bash
|
||||
$ glxinfo | grep -i opengl
|
||||
...
|
||||
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 430.50
|
||||
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
|
||||
OpenGL ES profile extensions:
|
||||
```
|
||||
|
||||
*Notice the ES 3.20 text above.*
|
||||
|
||||
You need to see ES 3.1 or greater printed in order to perform TFLite inference
|
||||
on GPU in MediaPipe. With this setup, build with:
|
||||
|
||||
```
|
||||
$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 <my-target>
|
||||
```
|
||||
|
||||
If only ES 3.0 or below is supported, you can still build MediaPipe targets that
|
||||
don't require TFLite inference on GPU with:
|
||||
|
||||
```
|
||||
$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 --copt -DMEDIAPIPE_DISABLE_GL_COMPUTE <my-target>
|
||||
```
|
||||
|
||||
Note: MEDIAPIPE_DISABLE_GL_COMPUTE is already defined automatically on all Apple
|
||||
systems (Apple doesn't support OpenGL ES 3.1+).
|
||||
|
||||
### TensorFlow CUDA Support and Setup on Linux Desktop
|
||||
|
||||
MediaPipe framework doesn't require CUDA for GPU compute and rendering. However,
|
||||
MediaPipe can work with TensorFlow to perform GPU inference on video cards that
|
||||
support CUDA.
|
||||
|
||||
To enable TensorFlow GPU inference with MediaPipe, the first step is to follow
|
||||
the
|
||||
[TensorFlow GPU documentation](https://www.tensorflow.org/install/gpu#software_requirements)
|
||||
to install the required NVIDIA software on your Linux desktop.
|
||||
|
||||
After installation, update `$PATH` and `$LD_LIBRARY_PATH` and run `ldconfig`
|
||||
with:
|
||||
|
||||
```
|
||||
$ export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
|
||||
$ export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64,/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
|
||||
$ sudo ldconfig
|
||||
```
|
||||
|
||||
It's recommended to verify the installation of CUPTI, CUDA, CuDNN, and NVCC:
|
||||
|
||||
```
|
||||
$ ls /usr/local/cuda/extras/CUPTI
|
||||
/lib64
|
||||
libcupti.so libcupti.so.10.1.208 libnvperf_host.so libnvperf_target.so
|
||||
libcupti.so.10.1 libcupti_static.a libnvperf_host_static.a
|
||||
|
||||
$ ls /usr/local/cuda-10.1
|
||||
LICENSE bin extras lib64 libnvvp nvml samples src tools
|
||||
README doc include libnsight nsightee_plugins nvvm share targets version.txt
|
||||
|
||||
$ nvcc -V
|
||||
nvcc: NVIDIA (R) Cuda compiler driver
|
||||
Copyright (c) 2005-2019 NVIDIA Corporation
|
||||
Built on Sun_Jul_28_19:07:16_PDT_2019
|
||||
Cuda compilation tools, release 10.1, V10.1.243
|
||||
|
||||
$ ls /usr/lib/x86_64-linux-gnu/ | grep libcudnn.so
|
||||
libcudnn.so
|
||||
libcudnn.so.7
|
||||
libcudnn.so.7.6.4
|
||||
```
|
||||
|
||||
Setting `$TF_CUDA_PATHS` is the way to declare where the CUDA library is. Note
|
||||
that the following code snippet also adds `/usr/lib/x86_64-linux-gnu` and
|
||||
`/usr/include` into `$TF_CUDA_PATHS` for cudablas and libcudnn.
|
||||
|
||||
```
|
||||
$ export TF_CUDA_PATHS=/usr/local/cuda-10.1,/usr/lib/x86_64-linux-gnu,/usr/include
|
||||
```
|
||||
|
||||
To make MediaPipe get TensorFlow's CUDA settings, find TensorFlow's
|
||||
[.bazelrc](https://github.com/tensorflow/tensorflow/blob/master/.bazelrc) and
|
||||
copy the `build:using_cuda` and `build:cuda` section into MediaPipe's .bazelrc
|
||||
file. For example, as of April 23, 2020, TensorFlow's CUDA setting is the
|
||||
following:
|
||||
|
||||
```
|
||||
# This config refers to building with CUDA available. It does not necessarily
|
||||
# mean that we build CUDA op kernels.
|
||||
build:using_cuda --define=using_cuda=true
|
||||
build:using_cuda --action_env TF_NEED_CUDA=1
|
||||
build:using_cuda --crosstool_top=@local_config_cuda//crosstool:toolchain
|
||||
|
||||
# This config refers to building CUDA op kernels with nvcc.
|
||||
build:cuda --config=using_cuda
|
||||
build:cuda --define=using_cuda_nvcc=true
|
||||
```
|
||||
|
||||
Finally, build MediaPipe with TensorFlow GPU with two more flags `--config=cuda`
|
||||
and `--spawn_strategy=local`. For example:
|
||||
|
||||
```
|
||||
$ bazel build -c opt --config=cuda --spawn_strategy=local \
|
||||
--define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS \
|
||||
mediapipe/examples/desktop/object_detection:object_detection_tensorflow
|
||||
```
|
||||
|
||||
While the binary is running, it prints out the GPU device info:
|
||||
|
||||
```
|
||||
I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
|
||||
I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5 coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
|
||||
I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
|
||||
```
|
||||
|
||||
You can monitor the GPU usage to verify whether the GPU is used for model
|
||||
inference.
|
||||
|
||||
```
|
||||
$ nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1
|
||||
|
||||
0 %
|
||||
0 %
|
||||
4 %
|
||||
5 %
|
||||
83 %
|
||||
21 %
|
||||
22 %
|
||||
27 %
|
||||
29 %
|
||||
100 %
|
||||
0 %
|
||||
0%
|
||||
```
|
||||
|
||||
### Life of a GPU Calculator
|
||||
|
||||
This section presents the basic structure of the Process method of a GPU
|
||||
calculator derived from base class GlSimpleCalculator. The GPU calculator
|
||||
`LuminanceCalculator` is shown as an example. The method
|
||||
`LuminanceCalculator::GlRender` is called from `GlSimpleCalculator::Process`.
|
||||
|
||||
```c++
|
||||
// Converts RGB images into luminance images, still stored in RGB format.
|
||||
// See GlSimpleCalculator for inputs, outputs and input side packets.
|
||||
class LuminanceCalculator : public GlSimpleCalculator {
|
||||
public:
|
||||
::mediapipe::Status GlSetup() override;
|
||||
::mediapipe::Status GlRender(const GlTexture& src,
|
||||
const GlTexture& dst) override;
|
||||
::mediapipe::Status GlTeardown() override;
|
||||
|
||||
private:
|
||||
GLuint program_ = 0;
|
||||
GLint frame_;
|
||||
};
|
||||
REGISTER_CALCULATOR(LuminanceCalculator);
|
||||
|
||||
::mediapipe::Status LuminanceCalculator::GlRender(const GlTexture& src,
|
||||
const GlTexture& dst) {
|
||||
static const GLfloat square_vertices[] = {
|
||||
-1.0f, -1.0f, // bottom left
|
||||
1.0f, -1.0f, // bottom right
|
||||
-1.0f, 1.0f, // top left
|
||||
1.0f, 1.0f, // top right
|
||||
};
|
||||
static const GLfloat texture_vertices[] = {
|
||||
0.0f, 0.0f, // bottom left
|
||||
1.0f, 0.0f, // bottom right
|
||||
0.0f, 1.0f, // top left
|
||||
1.0f, 1.0f, // top right
|
||||
};
|
||||
|
||||
// program
|
||||
glUseProgram(program_);
|
||||
glUniform1i(frame_, 1);
|
||||
|
||||
// vertex storage
|
||||
GLuint vbo[2];
|
||||
glGenBuffers(2, vbo);
|
||||
GLuint vao;
|
||||
glGenVertexArrays(1, &vao);
|
||||
glBindVertexArray(vao);
|
||||
|
||||
// vbo 0
|
||||
glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
|
||||
glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
|
||||
GL_STATIC_DRAW);
|
||||
glEnableVertexAttribArray(ATTRIB_VERTEX);
|
||||
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
|
||||
|
||||
// vbo 1
|
||||
glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
|
||||
glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
|
||||
GL_STATIC_DRAW);
|
||||
glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
|
||||
glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
|
||||
|
||||
// draw
|
||||
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
|
||||
|
||||
// cleanup
|
||||
glDisableVertexAttribArray(ATTRIB_VERTEX);
|
||||
glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
|
||||
glBindBuffer(GL_ARRAY_BUFFER, 0);
|
||||
glBindVertexArray(0);
|
||||
glDeleteVertexArrays(1, &vao);
|
||||
glDeleteBuffers(2, vbo);
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
```
|
||||
|
||||
The design principles mentioned above have resulted in the following design
|
||||
choices for MediaPipe GPU support:
|
||||
|
||||
* We have a GPU data type, called `GpuBuffer`, for representing image data, optimized for GPU usage. The exact contents of this data type are opaque and platform-specific.
|
||||
* A low-level API based on composition, where any calculator that wants to make use of the GPU creates and owns an instance of the `GlCalculatorHelper` class. This class offers a platform-agnostic API for managing the OpenGL context, setting up textures for inputs and outputs, etc.
|
||||
* A high-level API based on subclassing, where simple calculators implementing image filters subclass from `GlSimpleCalculator` and only need to override a couple of virtual methods with their specific OpenGL code, while the superclass takes care of all the plumbing.
|
||||
* Data that needs to be shared between all GPU-based calculators is provided as a external input that is implemented as a graph service and is managed by the `GlCalculatorHelper` class.
|
||||
* The combination of calculator-specific helpers and a shared graph service allows us great flexibility in managing the GPU resource: we can have a separate context per calculator, share a single context, share a lock or other synchronization primitives, etc. -- and all of this is managed by the helper and hidden from the individual calculators.
|
||||
|
||||
### GpuBuffer to ImageFrame Converters
|
||||
|
||||
We provide two calculators called `GpuBufferToImageFrameCalculator` and `ImageFrameToGpuBufferCalculator`. These calculators convert between `ImageFrame` and `GpuBuffer`, allowing the construction of graphs that combine GPU and CPU calculators. They are supported on both iOS and Android
|
||||
|
||||
When possible, these calculators use platform-specific functionality to share data between the CPU and the GPU without copying.
|
||||
|
||||
The below diagram shows the data flow in a mobile application that captures video from the camera, runs it through a MediaPipe graph, and renders the output on the screen in real time. The dashed line indicates which parts are inside the MediaPipe graph proper. This application runs a Canny edge-detection filter on the CPU using OpenCV, and overlays it on top of the original video using the GPU.
|
||||
|
||||
|  |
|
||||
|:--:|
|
||||
| *Video frames from the camera are fed into the graph as `GpuBuffer` packets. The input stream is accessed by two calculators in parallel. `GpuBufferToImageFrameCalculator` converts the buffer into an `ImageFrame`, which is then sent through a grayscale converter and a canny filter (both based on OpenCV and running on the CPU), whose output is then converted into a `GpuBuffer` again. A multi-input GPU calculator, GlOverlayCalculator, takes as input both the original `GpuBuffer` and the one coming out of the edge detector, and overlays them using a shader. The output is then sent back to the application using a callback calculator, and the application renders the image to the screen using OpenGL.* |
|
||||
@@ -1,207 +0,0 @@
|
||||
## Hair Segmentation on Desktop
|
||||
|
||||
This is an example of using MediaPipe to run hair segmentation models
|
||||
(TensorFlow Lite) and render a color to the detected hair. To know more about
|
||||
the hair segmentation models, please refer to the model [`README file`].
|
||||
Moreover, if you are interested in running the same TensorfFlow Lite model on
|
||||
Android/iOS, please see the
|
||||
[Hair Segmentation on GPU on Android/iOS](hair_segmentation_mobile_gpu.md) and
|
||||
|
||||
We show the hair segmentation demos with TensorFlow Lite model using the Webcam:
|
||||
|
||||
- [TensorFlow Lite Hair Segmentation Demo with Webcam (GPU)](#tensorflow-lite-hair-segmentation-demo-with-webcam-gpu)
|
||||
|
||||
Note: If MediaPipe depends on OpenCV 2, please see the
|
||||
[known issues with OpenCV 2](./object_detection_desktop.md#known-issues-with-opencv-2)
|
||||
section.
|
||||
|
||||
### TensorFlow Lite Hair Segmentation Demo with Webcam (GPU)
|
||||
|
||||
Note: This currently works only on Linux, and please first follow
|
||||
[OpenGL ES Setup on Linux Desktop](./gpu.md#opengl-es-setup-on-linux-desktop).
|
||||
|
||||
To build and run the TensorFlow Lite example on desktop (GPU) with Webcam, run:
|
||||
|
||||
```bash
|
||||
# Video from webcam running on desktop GPU
|
||||
# This works only for Linux currently
|
||||
$ bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
|
||||
mediapipe/examples/desktop/hair_segmentation:hair_segmentation_gpu
|
||||
|
||||
# It should print:
|
||||
#INFO: Found 1 target...
|
||||
#Target //mediapipe/examples/desktop/hair_segmentation:hair_segmentation_gpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/hair_segmentation/hair_segmentation_gpu
|
||||
#INFO: Build completed successfully, 12210 total actions
|
||||
|
||||
# This will open up your webcam as long as it is connected and on
|
||||
# Any errors is likely due to your webcam being not accessible,
|
||||
# or GPU drivers not setup properly.
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hair_segmentation/hair_segmentation_gpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt
|
||||
```
|
||||
|
||||
#### Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev).
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
|
||||
# Used in the example in
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/hairsegmentationgpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToSegmentationCalculator downstream in the graph to finish
|
||||
# generating the corresponding hair mask before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToSegmentationCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:hair_mask"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transforms the input image on GPU to a 512x512 image. To scale the image, by
|
||||
# default it uses the STRETCH scale mode that maps the entire input image to the
|
||||
# entire transformed image. As a result, image aspect ratio may be changed and
|
||||
# objects in the image may be deformed (stretched or squeezed), but the hair
|
||||
# segmentation model used in this graph is agnostic to that deformation.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
output_stream: "IMAGE_GPU:transformed_input_video"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 512
|
||||
output_height: 512
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Caches a mask fed back from the previous round of hair segmentation, and upon
|
||||
# the arrival of the next input image sends out the cached mask with the
|
||||
# timestamp replaced by that of the input image, essentially generating a packet
|
||||
# that carries the previous mask. Note that upon the arrival of the very first
|
||||
# input image, an empty packet is sent out to jump start the feedback loop.
|
||||
node {
|
||||
calculator: "PreviousLoopbackCalculator"
|
||||
input_stream: "MAIN:throttled_input_video"
|
||||
input_stream: "LOOP:hair_mask"
|
||||
input_stream_info: {
|
||||
tag_index: "LOOP"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "PREV_LOOP:previous_hair_mask"
|
||||
}
|
||||
|
||||
# Embeds the hair mask generated from the previous round of hair segmentation
|
||||
# as the alpha channel of the current input image.
|
||||
node {
|
||||
calculator: "SetAlphaCalculator"
|
||||
input_stream: "IMAGE_GPU:transformed_input_video"
|
||||
input_stream: "ALPHA_GPU:previous_hair_mask"
|
||||
output_stream: "IMAGE_GPU:mask_embedded_input_video"
|
||||
}
|
||||
|
||||
# Converts the transformed input image on GPU into an image tensor stored in
|
||||
# tflite::gpu::GlBuffer. The zero_center option is set to false to normalize the
|
||||
# pixel values to [0.f, 1.f] as opposed to [-1.f, 1.f]. With the
|
||||
# max_num_channels option set to 4, all 4 RGBA channels are contained in the
|
||||
# image tensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE_GPU:mask_embedded_input_video"
|
||||
output_stream: "TENSORS_GPU:image_tensor"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
|
||||
zero_center: false
|
||||
max_num_channels: 4
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a TensorFlow Lite op resolver that
|
||||
# supports custom ops needed by the model used in this graph.
|
||||
node {
|
||||
calculator: "TfLiteCustomOpResolverCalculator"
|
||||
output_side_packet: "op_resolver"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
|
||||
use_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
|
||||
# tensor representing the hair segmentation, which has the same width and height
|
||||
# as the input image tensor.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS_GPU:image_tensor"
|
||||
output_stream: "TENSORS_GPU:segmentation_tensor"
|
||||
input_side_packet: "CUSTOM_OP_RESOLVER:op_resolver"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/models/hair_segmentation.tflite"
|
||||
use_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the segmentation tensor generated by the TensorFlow Lite model into a
|
||||
# mask of values in [0.f, 1.f], stored in the R channel of a GPU buffer. It also
|
||||
# takes the mask generated previously as another input to improve the temporal
|
||||
# consistency.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToSegmentationCalculator"
|
||||
input_stream: "TENSORS_GPU:segmentation_tensor"
|
||||
input_stream: "PREV_MASK_GPU:previous_hair_mask"
|
||||
output_stream: "MASK_GPU:hair_mask"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToSegmentationCalculatorOptions] {
|
||||
tensor_width: 512
|
||||
tensor_height: 512
|
||||
tensor_channels: 2
|
||||
combine_with_previous_ratio: 0.9
|
||||
output_layer_index: 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Colors the hair segmentation with the color specified in the option.
|
||||
node {
|
||||
calculator: "RecolorCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
input_stream: "MASK_GPU:hair_mask"
|
||||
output_stream: "IMAGE_GPU:output_video"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.RecolorCalculatorOptions] {
|
||||
color { r: 0 g: 0 b: 255 }
|
||||
mask_channel: RED
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
[`README file`]:https://github.com/google/mediapipe/tree/master/mediapipe/README.md
|
||||
@@ -1,182 +1,2 @@
|
||||
# Hair Segmentation (GPU)
|
||||
|
||||
This doc focuses on the
|
||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt)
|
||||
that performs hair segmentation with TensorFlow Lite on GPU.
|
||||
|
||||
|
||||
|
||||
## Android
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu)
|
||||
|
||||
To build and install the app:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu
|
||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu/hairsegmentationgpu.apk
|
||||
```
|
||||
|
||||
## Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt)
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
|
||||
# Used in the example in
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/hairsegmentationgpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToSegmentationCalculator downstream in the graph to finish
|
||||
# generating the corresponding hair mask before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToSegmentationCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:hair_mask"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transforms the input image on GPU to a 512x512 image. To scale the image, by
|
||||
# default it uses the STRETCH scale mode that maps the entire input image to the
|
||||
# entire transformed image. As a result, image aspect ratio may be changed and
|
||||
# objects in the image may be deformed (stretched or squeezed), but the hair
|
||||
# segmentation model used in this graph is agnostic to that deformation.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
output_stream: "IMAGE_GPU:transformed_input_video"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 512
|
||||
output_height: 512
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Caches a mask fed back from the previous round of hair segmentation, and upon
|
||||
# the arrival of the next input image sends out the cached mask with the
|
||||
# timestamp replaced by that of the input image, essentially generating a packet
|
||||
# that carries the previous mask. Note that upon the arrival of the very first
|
||||
# input image, an empty packet is sent out to jump start the feedback loop.
|
||||
node {
|
||||
calculator: "PreviousLoopbackCalculator"
|
||||
input_stream: "MAIN:throttled_input_video"
|
||||
input_stream: "LOOP:hair_mask"
|
||||
input_stream_info: {
|
||||
tag_index: "LOOP"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "PREV_LOOP:previous_hair_mask"
|
||||
}
|
||||
|
||||
# Embeds the hair mask generated from the previous round of hair segmentation
|
||||
# as the alpha channel of the current input image.
|
||||
node {
|
||||
calculator: "SetAlphaCalculator"
|
||||
input_stream: "IMAGE_GPU:transformed_input_video"
|
||||
input_stream: "ALPHA_GPU:previous_hair_mask"
|
||||
output_stream: "IMAGE_GPU:mask_embedded_input_video"
|
||||
}
|
||||
|
||||
# Converts the transformed input image on GPU into an image tensor stored in
|
||||
# tflite::gpu::GlBuffer. The zero_center option is set to false to normalize the
|
||||
# pixel values to [0.f, 1.f] as opposed to [-1.f, 1.f]. With the
|
||||
# max_num_channels option set to 4, all 4 RGBA channels are contained in the
|
||||
# image tensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE_GPU:mask_embedded_input_video"
|
||||
output_stream: "TENSORS_GPU:image_tensor"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
|
||||
zero_center: false
|
||||
max_num_channels: 4
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a TensorFlow Lite op resolver that
|
||||
# supports custom ops needed by the model used in this graph.
|
||||
node {
|
||||
calculator: "TfLiteCustomOpResolverCalculator"
|
||||
output_side_packet: "op_resolver"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
|
||||
use_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
|
||||
# tensor representing the hair segmentation, which has the same width and height
|
||||
# as the input image tensor.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS_GPU:image_tensor"
|
||||
output_stream: "TENSORS_GPU:segmentation_tensor"
|
||||
input_side_packet: "CUSTOM_OP_RESOLVER:op_resolver"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "hair_segmentation.tflite"
|
||||
use_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the segmentation tensor generated by the TensorFlow Lite model into a
|
||||
# mask of values in [0.f, 1.f], stored in the R channel of a GPU buffer. It also
|
||||
# takes the mask generated previously as another input to improve the temporal
|
||||
# consistency.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToSegmentationCalculator"
|
||||
input_stream: "TENSORS_GPU:segmentation_tensor"
|
||||
input_stream: "PREV_MASK_GPU:previous_hair_mask"
|
||||
output_stream: "MASK_GPU:hair_mask"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToSegmentationCalculatorOptions] {
|
||||
tensor_width: 512
|
||||
tensor_height: 512
|
||||
tensor_channels: 2
|
||||
combine_with_previous_ratio: 0.9
|
||||
output_layer_index: 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Colors the hair segmentation with the color specified in the option.
|
||||
node {
|
||||
calculator: "RecolorCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
input_stream: "MASK_GPU:hair_mask"
|
||||
output_stream: "IMAGE_GPU:output_video"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.RecolorCalculatorOptions] {
|
||||
color { r: 0 g: 0 b: 255 }
|
||||
mask_channel: RED
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Content moved to
|
||||
[MediapPipe Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)
|
||||
|
||||
@@ -1,350 +0,0 @@
|
||||
# Hand Detection (GPU)
|
||||
|
||||
This doc focuses on the
|
||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
|
||||
that performs hand detection with TensorFlow Lite on GPU. It is related to the
|
||||
[hand tracking example](./hand_tracking_mobile_gpu.md).
|
||||
|
||||
For overall context on hand detection and hand tracking, please read this
|
||||
[Google AI Blog post](https://mediapipe.page.link/handgoogleaiblog).
|
||||
|
||||
|
||||
|
||||
In the visualization above, green boxes represent the results of palm detection,
|
||||
and the red box represents the extended hand rectangle designed to cover the
|
||||
entire hand. The palm detection ML model (see also
|
||||
[model card](https://mediapipe.page.link/handmc)) supports detection of multiple
|
||||
palms, and this example selects only the one with the highest detection
|
||||
confidence score to generate the hand rectangle, to be further utilized in the
|
||||
[hand tracking example](./hand_tracking_mobile_gpu.md).
|
||||
|
||||
## Android
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu)
|
||||
|
||||
An arm64 APK can be
|
||||
[downloaded here](https://drive.google.com/open?id=1qUlTtH7Ydg-wl_H6VVL8vueu2UCTu37E).
|
||||
|
||||
To build the app yourself:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu
|
||||
```
|
||||
|
||||
Once the app is built, install it on Android device with:
|
||||
|
||||
```bash
|
||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu/handdetectiongpu.apk
|
||||
```
|
||||
|
||||
## iOS
|
||||
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu).
|
||||
|
||||
See the general [instructions](./building_examples.md#ios) for building iOS
|
||||
examples and generating an Xcode project. This will be the HandDetectionGpuApp
|
||||
target.
|
||||
|
||||
To build on the command line:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp
|
||||
```
|
||||
|
||||
## Graph
|
||||
|
||||
The hand detection [main graph](#main-graph) internally utilizes a
|
||||
[hand detection subgraph](#hand-detection-subgraph). The subgraph shows up in
|
||||
the main graph visualization as the `HandDetection` node colored in purple, and
|
||||
the subgraph itself can also be visualized just like a regular graph. For more
|
||||
information on how to visualize a graph that includes subgraphs, see the
|
||||
Visualizing Subgraphs section in the
|
||||
[visualizer documentation](./visualizer.md).
|
||||
|
||||
### Main Graph
|
||||
|
||||
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs hand detection with TensorFlow Lite on GPU.
|
||||
# Used in the examples in
|
||||
# mediapipie/examples/android/src/java/com/mediapipe/apps/handdetectiongpu and
|
||||
# mediapipie/examples/ios/handdetectiongpu.
|
||||
|
||||
# Images coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for HandDetectionSubgraph
|
||||
# downstream in the graph to finish its tasks before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images in HandDetectionSubgraph to 1. This prevents the nodes in
|
||||
# HandDetectionSubgraph from queuing up incoming images and data excessively,
|
||||
# which leads to increased latency and memory usage, unwanted in real-time
|
||||
# mobile applications. It also eliminates unnecessarily computation, e.g., the
|
||||
# output produced by a node in the subgraph may get dropped downstream if the
|
||||
# subsequent nodes are still busy processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:hand_rect_from_palm_detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Subgraph that detections hands (see hand_detection_gpu.pbtxt).
|
||||
node {
|
||||
calculator: "HandDetectionSubgraph"
|
||||
input_stream: "throttled_input_video"
|
||||
output_stream: "DETECTIONS:palm_detections"
|
||||
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
|
||||
}
|
||||
|
||||
# Converts detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:palm_detections"
|
||||
output_stream: "RENDER_DATA:detection_render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 0 g: 255 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts normalized rects to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "RectToRenderDataCalculator"
|
||||
input_stream: "NORM_RECT:hand_rect_from_palm_detections"
|
||||
output_stream: "RENDER_DATA:rect_render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
|
||||
filled: false
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
thickness: 4.0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
input_stream: "detection_render_data"
|
||||
input_stream: "rect_render_data"
|
||||
output_stream: "IMAGE_GPU:output_video"
|
||||
}
|
||||
```
|
||||
|
||||
### Hand Detection Subgraph
|
||||
|
||||
|
||||
|
||||
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/hand_detection_gpu.pbtxt)
|
||||
|
||||
```bash
|
||||
# MediaPipe hand detection subgraph.
|
||||
|
||||
type: "HandDetectionSubgraph"
|
||||
|
||||
input_stream: "input_video"
|
||||
output_stream: "DETECTIONS:palm_detections"
|
||||
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
|
||||
|
||||
# Transforms the input image on GPU to a 256x256 image. To scale the input
|
||||
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
|
||||
# resulting in potential letterboxing in the transformed image.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE_GPU:input_video"
|
||||
output_stream: "IMAGE_GPU:transformed_input_video"
|
||||
output_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 256
|
||||
output_height: 256
|
||||
scale_mode: FIT
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a TensorFlow Lite op resolver that
|
||||
# supports custom ops needed by the model used in this graph.
|
||||
node {
|
||||
calculator: "TfLiteCustomOpResolverCalculator"
|
||||
output_side_packet: "opresolver"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
|
||||
use_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts the transformed input image on GPU into an image tensor stored as a
|
||||
# TfLiteTensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE_GPU:transformed_input_video"
|
||||
output_stream: "TENSORS_GPU:image_tensor"
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
|
||||
# vector of tensors representing, for instance, detection boxes/keypoints and
|
||||
# scores.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS_GPU:image_tensor"
|
||||
output_stream: "TENSORS:detection_tensors"
|
||||
input_side_packet: "CUSTOM_OP_RESOLVER:opresolver"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "palm_detection.tflite"
|
||||
use_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a vector of SSD anchors based on
|
||||
# the specification in the options.
|
||||
node {
|
||||
calculator: "SsdAnchorsCalculator"
|
||||
output_side_packet: "anchors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
|
||||
num_layers: 5
|
||||
min_scale: 0.1171875
|
||||
max_scale: 0.75
|
||||
input_size_height: 256
|
||||
input_size_width: 256
|
||||
anchor_offset_x: 0.5
|
||||
anchor_offset_y: 0.5
|
||||
strides: 8
|
||||
strides: 16
|
||||
strides: 32
|
||||
strides: 32
|
||||
strides: 32
|
||||
aspect_ratios: 1.0
|
||||
fixed_anchor_size: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
|
||||
# the SSD anchors and the specification in the options, into a vector of
|
||||
# detections. Each detection describes a detected object.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToDetectionsCalculator"
|
||||
input_stream: "TENSORS:detection_tensors"
|
||||
input_side_packet: "ANCHORS:anchors"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
|
||||
num_classes: 1
|
||||
num_boxes: 2944
|
||||
num_coords: 18
|
||||
box_coord_offset: 0
|
||||
keypoint_coord_offset: 4
|
||||
num_keypoints: 7
|
||||
num_values_per_keypoint: 2
|
||||
sigmoid_score: true
|
||||
score_clipping_thresh: 100.0
|
||||
reverse_output_order: true
|
||||
|
||||
x_scale: 256.0
|
||||
y_scale: 256.0
|
||||
h_scale: 256.0
|
||||
w_scale: 256.0
|
||||
min_score_thresh: 0.7
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Performs non-max suppression to remove excessive detections.
|
||||
node {
|
||||
calculator: "NonMaxSuppressionCalculator"
|
||||
input_stream: "detections"
|
||||
output_stream: "filtered_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
|
||||
min_suppression_threshold: 0.3
|
||||
overlap_type: INTERSECTION_OVER_UNION
|
||||
algorithm: WEIGHTED
|
||||
return_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Maps detection label IDs to the corresponding label text ("Palm"). The label
|
||||
# map is provided in the label_map_path option.
|
||||
node {
|
||||
calculator: "DetectionLabelIdToTextCalculator"
|
||||
input_stream: "filtered_detections"
|
||||
output_stream: "labeled_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
|
||||
label_map_path: "palm_detection_labelmap.txt"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
|
||||
# letterboxed image (after image transformation with the FIT scale mode) to the
|
||||
# corresponding locations on the same image with the letterbox removed (the
|
||||
# input image to the graph before image transformation).
|
||||
node {
|
||||
calculator: "DetectionLetterboxRemovalCalculator"
|
||||
input_stream: "DETECTIONS:labeled_detections"
|
||||
input_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
output_stream: "DETECTIONS:palm_detections"
|
||||
}
|
||||
|
||||
# Extracts image size from the input images.
|
||||
node {
|
||||
calculator: "ImagePropertiesCalculator"
|
||||
input_stream: "IMAGE_GPU:input_video"
|
||||
output_stream: "SIZE:image_size"
|
||||
}
|
||||
|
||||
# Converts results of palm detection into a rectangle (normalized by image size)
|
||||
# that encloses the palm and is rotated such that the line connecting center of
|
||||
# the wrist and MCP of the middle finger is aligned with the Y-axis of the
|
||||
# rectangle.
|
||||
node {
|
||||
calculator: "DetectionsToRectsCalculator"
|
||||
input_stream: "DETECTIONS:palm_detections"
|
||||
input_stream: "IMAGE_SIZE:image_size"
|
||||
output_stream: "NORM_RECT:palm_rect"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRectsCalculatorOptions] {
|
||||
rotation_vector_start_keypoint_index: 0 # Center of wrist.
|
||||
rotation_vector_end_keypoint_index: 2 # MCP of middle finger.
|
||||
rotation_vector_target_angle_degrees: 90
|
||||
output_zero_rect_for_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Expands and shifts the rectangle that contains the palm so that it's likely
|
||||
# to cover the entire hand.
|
||||
node {
|
||||
calculator: "RectTransformationCalculator"
|
||||
input_stream: "NORM_RECT:palm_rect"
|
||||
input_stream: "IMAGE_SIZE:image_size"
|
||||
output_stream: "hand_rect_from_palm_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.RectTransformationCalculatorOptions] {
|
||||
scale_x: 2.6
|
||||
scale_y: 2.6
|
||||
shift_y: -0.5
|
||||
square_long: true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -1,181 +1 @@
|
||||
## Hand Tracking on Desktop
|
||||
|
||||
This is an example of using MediaPipe to run hand tracking models (TensorFlow
|
||||
Lite) and render bounding boxes on the detected hand (one hand only). To know
|
||||
more about the hand tracking models, please refer to the model [`README file`].
|
||||
Moreover, if you are interested in running the same TensorfFlow Lite model on
|
||||
Android/iOS, please see the
|
||||
[Hand Tracking on GPU on Android/iOS](hand_tracking_mobile_gpu.md) and
|
||||
|
||||
We show the hand tracking demos with TensorFlow Lite model using the Webcam:
|
||||
|
||||
- [TensorFlow Lite Hand Tracking Demo with Webcam (CPU)](#tensorflow-lite-hand-tracking-demo-with-webcam-cpu)
|
||||
|
||||
- [TensorFlow Lite Hand Tracking Demo with Webcam (GPU)](#tensorflow-lite-hand-tracking-demo-with-webcam-gpu)
|
||||
|
||||
Note: If MediaPipe depends on OpenCV 2, please see the
|
||||
[known issues with OpenCV 2](./object_detection_desktop.md#known-issues-with-opencv-2)
|
||||
section.
|
||||
|
||||
### TensorFlow Lite Hand Tracking Demo with Webcam (CPU)
|
||||
|
||||
To build and run the TensorFlow Lite example on desktop (CPU) with Webcam, run:
|
||||
|
||||
```bash
|
||||
# Video from webcam running on desktop CPU
|
||||
$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
|
||||
mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
|
||||
|
||||
# It should print:
|
||||
#Target //mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu
|
||||
#INFO: Build completed successfully, 12517 total actions
|
||||
|
||||
# This will open up your webcam as long as it is connected and on
|
||||
# Any errors is likely due to your webcam being not accessible
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
|
||||
```
|
||||
|
||||
### TensorFlow Lite Hand Tracking Demo with Webcam (GPU)
|
||||
|
||||
Note: This currently works only on Linux, and please first follow
|
||||
[OpenGL ES Setup on Linux Desktop](./gpu.md#opengl-es-setup-on-linux-desktop).
|
||||
|
||||
To build and run the TensorFlow Lite example on desktop (GPU) with Webcam, run:
|
||||
|
||||
```bash
|
||||
# Video from webcam running on desktop GPU
|
||||
# This works only for Linux currently
|
||||
$ bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
|
||||
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
|
||||
|
||||
# It should print:
|
||||
# Target //mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu up-to-date:
|
||||
# bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu
|
||||
#INFO: Build completed successfully, 22455 total actions
|
||||
|
||||
# This will open up your webcam as long as it is connected and on
|
||||
# Any errors is likely due to your webcam being not accessible,
|
||||
# or GPU drivers not setup properly.
|
||||
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
|
||||
```
|
||||
|
||||
#### Graph
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev).
|
||||
|
||||
```bash
|
||||
# MediaPipe graph that performs hand tracking on desktop with TensorFlow Lite
|
||||
# on CPU & GPU.
|
||||
# Used in the example in
|
||||
# mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu.
|
||||
|
||||
# Images coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Caches a hand-presence decision fed back from HandLandmarkSubgraph, and upon
|
||||
# the arrival of the next input image sends out the cached decision with the
|
||||
# timestamp replaced by that of the input image, essentially generating a packet
|
||||
# that carries the previous hand-presence decision. Note that upon the arrival
|
||||
# of the very first input image, an empty packet is sent out to jump start the
|
||||
# feedback loop.
|
||||
node {
|
||||
calculator: "PreviousLoopbackCalculator"
|
||||
input_stream: "MAIN:input_video"
|
||||
input_stream: "LOOP:hand_presence"
|
||||
input_stream_info: {
|
||||
tag_index: "LOOP"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "PREV_LOOP:prev_hand_presence"
|
||||
}
|
||||
|
||||
# Drops the incoming image if HandLandmarkSubgraph was able to identify hand
|
||||
# presence in the previous image. Otherwise, passes the incoming image through
|
||||
# to trigger a new round of hand detection in HandDetectionSubgraph.
|
||||
node {
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "DISALLOW:prev_hand_presence"
|
||||
output_stream: "hand_detection_input_video"
|
||||
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.GateCalculatorOptions] {
|
||||
empty_packets_as_allow: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Subgraph that detections hands (see hand_detection_cpu.pbtxt).
|
||||
node {
|
||||
calculator: "HandDetectionSubgraph"
|
||||
input_stream: "hand_detection_input_video"
|
||||
output_stream: "DETECTIONS:palm_detections"
|
||||
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
|
||||
}
|
||||
|
||||
# Subgraph that localizes hand landmarks (see hand_landmark_cpu.pbtxt).
|
||||
node {
|
||||
calculator: "HandLandmarkSubgraph"
|
||||
input_stream: "IMAGE:input_video"
|
||||
input_stream: "NORM_RECT:hand_rect"
|
||||
output_stream: "LANDMARKS:hand_landmarks"
|
||||
output_stream: "NORM_RECT:hand_rect_from_landmarks"
|
||||
output_stream: "PRESENCE:hand_presence"
|
||||
output_stream: "HANDEDNESS:handedness"
|
||||
}
|
||||
|
||||
# Caches a hand rectangle fed back from HandLandmarkSubgraph, and upon the
|
||||
# arrival of the next input image sends out the cached rectangle with the
|
||||
# timestamp replaced by that of the input image, essentially generating a packet
|
||||
# that carries the previous hand rectangle. Note that upon the arrival of the
|
||||
# very first input image, an empty packet is sent out to jump start the
|
||||
# feedback loop.
|
||||
node {
|
||||
calculator: "PreviousLoopbackCalculator"
|
||||
input_stream: "MAIN:input_video"
|
||||
input_stream: "LOOP:hand_rect_from_landmarks"
|
||||
input_stream_info: {
|
||||
tag_index: "LOOP"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "PREV_LOOP:prev_hand_rect_from_landmarks"
|
||||
}
|
||||
|
||||
# Merges a stream of hand rectangles generated by HandDetectionSubgraph and that
|
||||
# generated by HandLandmarkSubgraph into a single output stream by selecting
|
||||
# between one of the two streams. The former is selected if the incoming packet
|
||||
# is not empty, i.e., hand detection is performed on the current image by
|
||||
# HandDetectionSubgraph (because HandLandmarkSubgraph could not identify hand
|
||||
# presence in the previous image). Otherwise, the latter is selected, which is
|
||||
# never empty because HandLandmarkSubgraphs processes all images (that went
|
||||
# through FlowLimiterCaculator).
|
||||
node {
|
||||
calculator: "MergeCalculator"
|
||||
input_stream: "hand_rect_from_palm_detections"
|
||||
input_stream: "prev_hand_rect_from_landmarks"
|
||||
output_stream: "hand_rect"
|
||||
}
|
||||
|
||||
# Subgraph that renders annotations and overlays them on top of the input
|
||||
# images (see renderer_cpu.pbtxt).
|
||||
node {
|
||||
calculator: "RendererSubgraph"
|
||||
input_stream: "IMAGE:input_video"
|
||||
input_stream: "LANDMARKS:hand_landmarks"
|
||||
input_stream: "NORM_RECT:hand_rect"
|
||||
input_stream: "DETECTIONS:palm_detections"
|
||||
input_stream: "HANDEDNESS:handedness"
|
||||
output_stream: "IMAGE:output_video"
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
[`README file`]:https://github.com/google/mediapipe/tree/master/mediapipe/README.md
|
||||
Content moved to [MediapPipe Hand](https://google.github.io/mediapipe/solutions/hand)
|
||||
|
||||
@@ -1,154 +1 @@
|
||||
# MediaPipe Hand
|
||||
|
||||
## Overview
|
||||
|
||||
The ability to perceive the shape and motion of hands can be a vital component
|
||||
in improving the user experience across a variety of technological domains and
|
||||
platforms. For example, it can form the basis for sign language understanding
|
||||
and hand gesture control, and can also enable the overlay of digital content and
|
||||
information on top of the physical world in augmented reality. While coming
|
||||
naturally to people, robust real-time hand perception is a decidedly challenging
|
||||
computer vision task, as hands often occlude themselves or each other (e.g.
|
||||
finger/palm occlusions and hand shakes) and lack high contrast patterns.
|
||||
|
||||
MediaPipe Hand is a high-fidelity hand and finger tracking solution. It employs
|
||||
machine learning (ML) to infer 21 3D landmarks of a hand from just a single
|
||||
frame. Whereas current state-of-the-art approaches rely primarily on powerful
|
||||
desktop environments for inference, our method achieves real-time performance on
|
||||
a mobile phone, and even scales to multiple hands. We hope that providing this
|
||||
hand perception functionality to the wider research and development community
|
||||
will result in an emergence of creative use cases, stimulating new applications
|
||||
and new research avenues.
|
||||
|
||||
|
||||
|
||||
*Fig 1. Tracked 3D hand landmarks are represented by dots in different shades,
|
||||
with the brighter ones denoting landmarks closer to the camera.*
|
||||
|
||||
## ML Pipeline
|
||||
|
||||
MediaPipe Hand utilizes an ML pipeline consisting of multiple models working
|
||||
together: A palm detection model that operates on the full image and returns an
|
||||
oriented hand bounding box. A hand landmark model that operates on the cropped
|
||||
image region defined by the palm detector and returns high-fidelity 3D hand
|
||||
keypoints. This architecture is similar to that employed by our recently
|
||||
released [MediaPipe Face Mesh](./face_mesh_mobile_gpu.md) solution.
|
||||
|
||||
Providing the accurately cropped hand image to the hand landmark model
|
||||
drastically reduces the need for data augmentation (e.g. rotations, translation
|
||||
and scale) and instead allows the network to dedicate most of its capacity
|
||||
towards coordinate prediction accuracy. In addition, in our pipeline the crops
|
||||
can also be generated based on the hand landmarks identified in the previous
|
||||
frame, and only when the landmark model could no longer identify hand presence
|
||||
is palm detection invoked to relocalize the hand.
|
||||
|
||||
The pipeline is implemented as a MediaPipe
|
||||
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt),
|
||||
which internally utilizes a
|
||||
[palm/hand detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/hand_detection_gpu.pbtxt),
|
||||
a
|
||||
[hand landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/hand_landmark_gpu.pbtxt)
|
||||
and a
|
||||
[renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/renderer_gpu.pbtxt).
|
||||
For more information on how to visualize a graph and its associated subgraphs,
|
||||
please see the [visualizer documentation](./visualizer.md).
|
||||
|
||||
## Models
|
||||
|
||||
### Palm Detection Model
|
||||
|
||||
To detect initial hand locations, we designed a
|
||||
[single-shot detector](https://arxiv.org/abs/1512.02325) model optimized for
|
||||
mobile real-time uses in a manner similar to the face detection model in
|
||||
[MediaPipe Face Mesh](./face_mesh_mobile_gpu.md). Detecting hands is a decidedly
|
||||
complex task: our model has to work across a variety of hand sizes with a large
|
||||
scale span (~20x) relative to the image frame and be able to detect occluded and
|
||||
self-occluded hands. Whereas faces have high contrast patterns, e.g., in the eye
|
||||
and mouth region, the lack of such features in hands makes it comparatively
|
||||
difficult to detect them reliably from their visual features alone. Instead,
|
||||
providing additional context, like arm, body, or person features, aids accurate
|
||||
hand localization.
|
||||
|
||||
Our method addresses the above challenges using different strategies. First, we
|
||||
train a palm detector instead of a hand detector, since estimating bounding
|
||||
boxes of rigid objects like palms and fists is significantly simpler than
|
||||
detecting hands with articulated fingers. In addition, as palms are smaller
|
||||
objects, the non-maximum suppression algorithm works well even for two-hand
|
||||
self-occlusion cases, like handshakes. Moreover, palms can be modelled using
|
||||
square bounding boxes (anchors in ML terminology) ignoring other aspect ratios,
|
||||
and therefore reducing the number of anchors by a factor of 3-5. Second, an
|
||||
encoder-decoder feature extractor is used for bigger scene context awareness
|
||||
even for small objects (similar to the RetinaNet approach). Lastly, we minimize
|
||||
the focal loss during training to support a large amount of anchors resulting
|
||||
from the high scale variance.
|
||||
|
||||
With the above techniques, we achieve an average precision of 95.7% in palm
|
||||
detection. Using a regular cross entropy loss and no decoder gives a baseline of
|
||||
just 86.22%.
|
||||
|
||||
### Hand Landmark Model
|
||||
|
||||
After the palm detection over the whole image our subsequent hand landmark model
|
||||
performs precise keypoint localization of 21 3D hand-knuckle coordinates inside
|
||||
the detected hand regions via regression, that is direct coordinate prediction.
|
||||
The model learns a consistent internal hand pose representation and is robust
|
||||
even to partially visible hands and self-occlusions.
|
||||
|
||||
To obtain ground truth data, we have manually annotated ~30K real-world images
|
||||
with 21 3D coordinates, as shown below (we take Z-value from image depth map, if
|
||||
it exists per corresponding coordinate). To better cover the possible hand poses
|
||||
and provide additional supervision on the nature of hand geometry, we also
|
||||
render a high-quality synthetic hand model over various backgrounds and map it
|
||||
to the corresponding 3D coordinates.
|
||||
|
||||
|
||||
|
||||
*Fig 2. Top: Aligned hand crops passed to the tracking network with ground truth
|
||||
annotation. Bottom: Rendered synthetic hand images with ground truth
|
||||
annotation.*
|
||||
|
||||
## Example Apps
|
||||
|
||||
Please see the [general instructions](./building_examples.md) for how to build
|
||||
MediaPipe examples for different platforms.
|
||||
|
||||
#### Main Example
|
||||
|
||||
* Android:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu),
|
||||
[Prebuilt ARM64 APK](https://drive.google.com/open?id=1uCjS0y0O0dTDItsMh8x2cf4-l3uHW1vE)
|
||||
* iOS:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu)
|
||||
* Desktop:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hand_tracking)
|
||||
|
||||
#### With Multi-hand Support
|
||||
|
||||
* Android:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/multihandtrackinggpu),
|
||||
[Prebuilt ARM64 APK](https://drive.google.com/open?id=1Wk6V9EVaz1ks_MInPqqVGvvJD01SGXDc)
|
||||
* iOS:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/multihandtrackinggpu)
|
||||
* Desktop:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/multi_hand_tracking)
|
||||
|
||||
#### Palm/Hand Detection Only (no landmarks)
|
||||
|
||||
* Android:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectionggpu),
|
||||
[Prebuilt ARM64 APK](https://drive.google.com/open?id=1qUlTtH7Ydg-wl_H6VVL8vueu2UCTu37E)
|
||||
* iOS:
|
||||
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu)
|
||||
|
||||
## Resources
|
||||
|
||||
* [Google AI Blog: On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html)
|
||||
* [TensorFlow Blog: Face and hand tracking in the browser with MediaPipe and
|
||||
TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html)
|
||||
* Palm detection model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/palm_detection.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/handdetector/1)
|
||||
* Hand landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/handskeleton/1)
|
||||
* [Model card](https://mediapipe.page.link/handmc)
|
||||
Content moved to [MediapPipe Hand](https://google.github.io/mediapipe/solutions/hand)
|
||||
|
||||
@@ -1,766 +0,0 @@
|
||||
# Hello World! in MediaPipe on Android
|
||||
|
||||
## Introduction
|
||||
|
||||
This codelab uses MediaPipe on an Android device.
|
||||
|
||||
### What you will learn
|
||||
|
||||
How to develop an Android application that uses MediaPipe and run a MediaPipe
|
||||
graph on Android.
|
||||
|
||||
### What you will build
|
||||
|
||||
A simple camera app for real-time Sobel edge detection applied to a live video
|
||||
stream on an Android device.
|
||||
|
||||
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install MediaPipe on your system, see [MediaPipe installation guide] for
|
||||
details.
|
||||
2. Install Android Development SDK and Android NDK. See how to do so also in
|
||||
[MediaPipe installation guide].
|
||||
3. Enable [developer options] on your Android device.
|
||||
4. Setup [Bazel] on your system to build and deploy the Android app.
|
||||
|
||||
## Graph for edge detection
|
||||
|
||||
We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
|
||||
|
||||
```
|
||||
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
|
||||
# Used in the examples
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/basic.
|
||||
# mediapipe/examples/ios/edgedetectiongpu.
|
||||
|
||||
# Images coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Converts RGB images into luminance images, still stored in RGB format.
|
||||
node: {
|
||||
calculator: "LuminanceCalculator"
|
||||
input_stream: "input_video"
|
||||
output_stream: "luma_video"
|
||||
}
|
||||
|
||||
# Applies the Sobel filter to luminance images sotred in RGB format.
|
||||
node: {
|
||||
calculator: "SobelEdgesCalculator"
|
||||
input_stream: "luma_video"
|
||||
output_stream: "output_video"
|
||||
}
|
||||
```
|
||||
|
||||
A visualization of the graph is shown below:
|
||||
|
||||
|
||||
|
||||
This graph has a single input stream named `input_video` for all incoming frames
|
||||
that will be provided by your device's camera.
|
||||
|
||||
The first node in the graph, `LuminanceCalculator`, takes a single packet (image
|
||||
frame) and applies a change in luminance using an OpenGL shader. The resulting
|
||||
image frame is sent to the `luma_video` output stream.
|
||||
|
||||
The second node, `SobelEdgesCalculator` applies edge detection to incoming
|
||||
packets in the `luma_video` stream and outputs results in `output_video` output
|
||||
stream.
|
||||
|
||||
Our Android application will display the output image frames of the
|
||||
`output_video` stream.
|
||||
|
||||
## Initial minimal application setup
|
||||
|
||||
We first start with an simple Android application that displays "Hello World!"
|
||||
on the screen. You may skip this step if you are familiar with building Android
|
||||
applications using `bazel`.
|
||||
|
||||
Create a new directory where you will create your Android application. For
|
||||
example, the complete code of this tutorial can be found at
|
||||
`mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic`. We
|
||||
will refer to this path as `$APPLICATION_PATH` throughout the codelab.
|
||||
|
||||
Note that in the path to the application:
|
||||
|
||||
* The application is named `helloworld`.
|
||||
* The `$PACKAGE_PATH` of the application is
|
||||
`com.google.mediapipe.apps.basic`. This is used in code snippets in this
|
||||
tutorial, so please remember to use your own `$PACKAGE_PATH` when you
|
||||
copy/use the code snippets.
|
||||
|
||||
Add a file `activity_main.xml` to `$APPLICATION_PATH/res/layout`. This displays
|
||||
a [`TextView`] on the full screen of the application with the string `Hello
|
||||
World!`:
|
||||
|
||||
```
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
xmlns:app="http://schemas.android.com/apk/res-auto"
|
||||
xmlns:tools="http://schemas.android.com/tools"
|
||||
android:layout_width="match_parent"
|
||||
android:layout_height="match_parent">
|
||||
|
||||
<TextView
|
||||
android:layout_width="wrap_content"
|
||||
android:layout_height="wrap_content"
|
||||
android:text="Hello World!"
|
||||
app:layout_constraintBottom_toBottomOf="parent"
|
||||
app:layout_constraintLeft_toLeftOf="parent"
|
||||
app:layout_constraintRight_toRightOf="parent"
|
||||
app:layout_constraintTop_toTopOf="parent" />
|
||||
|
||||
</android.support.constraint.ConstraintLayout>
|
||||
```
|
||||
|
||||
Add a simple `MainActivity.java` to `$APPLICATION_PATH` which loads the content
|
||||
of the `activity_main.xml` layout as shown below:
|
||||
|
||||
```
|
||||
package com.google.mediapipe.apps.basic;
|
||||
|
||||
import android.os.Bundle;
|
||||
import androidx.appcompat.app.AppCompatActivity;
|
||||
|
||||
/** Bare-bones main activity. */
|
||||
public class MainActivity extends AppCompatActivity {
|
||||
|
||||
@Override
|
||||
protected void onCreate(Bundle savedInstanceState) {
|
||||
super.onCreate(savedInstanceState);
|
||||
setContentView(R.layout.activity_main);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Add a manifest file, `AndroidManifest.xml` to `$APPLICATION_PATH`, which
|
||||
launches `MainActivity` on application start:
|
||||
|
||||
```
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
|
||||
package="com.google.mediapipe.apps.basic">
|
||||
|
||||
<uses-sdk
|
||||
android:minSdkVersion="19"
|
||||
android:targetSdkVersion="19" />
|
||||
|
||||
<application
|
||||
android:allowBackup="true"
|
||||
android:label="${appName}"
|
||||
android:supportsRtl="true"
|
||||
android:theme="@style/AppTheme">
|
||||
<activity
|
||||
android:name="${mainActivity}"
|
||||
android:exported="true"
|
||||
android:screenOrientation="portrait">
|
||||
<intent-filter>
|
||||
<action android:name="android.intent.action.MAIN" />
|
||||
<category android:name="android.intent.category.LAUNCHER" />
|
||||
</intent-filter>
|
||||
</activity>
|
||||
</application>
|
||||
|
||||
</manifest>
|
||||
```
|
||||
|
||||
In our application we are using a `Theme.AppCompat` theme in the app, so we need
|
||||
appropriate theme references. Add `colors.xml` to
|
||||
`$APPLICATION_PATH/res/values/`:
|
||||
|
||||
```
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<resources>
|
||||
<color name="colorPrimary">#008577</color>
|
||||
<color name="colorPrimaryDark">#00574B</color>
|
||||
<color name="colorAccent">#D81B60</color>
|
||||
</resources>
|
||||
```
|
||||
|
||||
Add `styles.xml` to `$APPLICATION_PATH/res/values/`:
|
||||
|
||||
```
|
||||
<resources>
|
||||
|
||||
<!-- Base application theme. -->
|
||||
<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
|
||||
<!-- Customize your theme here. -->
|
||||
<item name="colorPrimary">@color/colorPrimary</item>
|
||||
<item name="colorPrimaryDark">@color/colorPrimaryDark</item>
|
||||
<item name="colorAccent">@color/colorAccent</item>
|
||||
</style>
|
||||
|
||||
</resources>
|
||||
```
|
||||
|
||||
To build the application, add a `BUILD` file to `$APPLICATION_PATH`, and
|
||||
`${appName}` and `${mainActivity}` in the manifest will be replaced by strings
|
||||
specified in `BUILD` as shown below.
|
||||
|
||||
```
|
||||
android_library(
|
||||
name = "basic_lib",
|
||||
srcs = glob(["*.java"]),
|
||||
manifest = "AndroidManifest.xml",
|
||||
resource_files = glob(["res/**"]),
|
||||
deps = [
|
||||
"//third_party:android_constraint_layout",
|
||||
"//third_party:androidx_appcompat",
|
||||
],
|
||||
)
|
||||
|
||||
android_binary(
|
||||
name = "helloworld",
|
||||
manifest = "AndroidManifest.xml",
|
||||
manifest_values = {
|
||||
"applicationId": "com.google.mediapipe.apps.basic",
|
||||
"appName": "Hello World",
|
||||
"mainActivity": ".MainActivity",
|
||||
},
|
||||
multidex = "native",
|
||||
deps = [
|
||||
":basic_lib",
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
The `android_library` rule adds dependencies for `MainActivity`, resource files
|
||||
and `AndroidManifest.xml`.
|
||||
|
||||
The `android_binary` rule, uses the `basic_lib` Android library generated to
|
||||
build a binary APK for installation on your Android device.
|
||||
|
||||
To build the app, use the following command:
|
||||
|
||||
```
|
||||
bazel build -c opt --config=android_arm64 $APPLICATION_PATH:helloworld
|
||||
```
|
||||
|
||||
Install the generated APK file using `adb install`. For example:
|
||||
|
||||
```
|
||||
adb install bazel-bin/$APPLICATION_PATH/helloworld.apk
|
||||
```
|
||||
|
||||
Open the application on your device. It should display a screen with the text
|
||||
`Hello World!`.
|
||||
|
||||
|
||||
|
||||
## Using the camera via `CameraX`
|
||||
|
||||
### Camera Permissions
|
||||
|
||||
To use the camera in our application, we need to request the user to provide
|
||||
access to the camera. To request camera permissions, add the following to
|
||||
`AndroidManifest.xml`:
|
||||
|
||||
```
|
||||
<!-- For using the camera -->
|
||||
<uses-permission android:name="android.permission.CAMERA" />
|
||||
<uses-feature android:name="android.hardware.camera" />
|
||||
```
|
||||
|
||||
Change the minimum SDK version to `21` and target SDK version to `27` in the
|
||||
same file:
|
||||
|
||||
```
|
||||
<uses-sdk
|
||||
android:minSdkVersion="21"
|
||||
android:targetSdkVersion="27" />
|
||||
```
|
||||
|
||||
This ensures that the user is prompted to request camera permission and enables
|
||||
us to use the [CameraX] library for camera access.
|
||||
|
||||
To request camera permissions, we can use a utility provided by MediaPipe
|
||||
components, namely [`PermissionHelper`]. To use it, add a dependency
|
||||
`"//mediapipe/java/com/google/mediapipe/components:android_components"` in the
|
||||
`mediapipe_lib` rule in `BUILD`.
|
||||
|
||||
To use the `PermissionHelper` in `MainActivity`, add the following line to the
|
||||
`onCreate` function:
|
||||
|
||||
```
|
||||
PermissionHelper.checkAndRequestCameraPermissions(this);
|
||||
```
|
||||
|
||||
This prompts the user with a dialog on the screen to request for permissions to
|
||||
use the camera in this application.
|
||||
|
||||
Add the following code to handle the user response:
|
||||
|
||||
```
|
||||
@Override
|
||||
public void onRequestPermissionsResult(
|
||||
int requestCode, String[] permissions, int[] grantResults) {
|
||||
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
|
||||
PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void onResume() {
|
||||
super.onResume();
|
||||
if (PermissionHelper.cameraPermissionsGranted(this)) {
|
||||
startCamera();
|
||||
}
|
||||
}
|
||||
|
||||
public void startCamera() {}
|
||||
```
|
||||
|
||||
We will leave the `startCamera()` method empty for now. When the user responds
|
||||
to the prompt, the `MainActivity` will resume and `onResume()` will be called.
|
||||
The code will confirm that permissions for using the camera have been granted,
|
||||
and then will start the camera.
|
||||
|
||||
Rebuild and install the application. You should now see a prompt requesting
|
||||
access to the camera for the application.
|
||||
|
||||
Note: If the there is no dialog prompt, uninstall and reinstall the application.
|
||||
This may also happen if you haven't changed the `minSdkVersion` and
|
||||
`targetSdkVersion` in the `AndroidManifest.xml` file.
|
||||
|
||||
### Camera Access
|
||||
|
||||
With camera permissions available, we can start and fetch frames from the
|
||||
camera.
|
||||
|
||||
To view the frames from the camera we will use a [`SurfaceView`]. Each frame
|
||||
from the camera will be stored in a [`SurfaceTexture`] object. To use these, we
|
||||
first need to change the layout of our application.
|
||||
|
||||
Remove the entire [`TextView`] code block from
|
||||
`$APPLICATION_PATH/res/layout/activity_main.xml` and add the following code
|
||||
instead:
|
||||
|
||||
```
|
||||
<FrameLayout
|
||||
android:id="@+id/preview_display_layout"
|
||||
android:layout_width="fill_parent"
|
||||
android:layout_height="fill_parent"
|
||||
android:layout_weight="1">
|
||||
<TextView
|
||||
android:id="@+id/no_camera_access_view"
|
||||
android:layout_height="fill_parent"
|
||||
android:layout_width="fill_parent"
|
||||
android:gravity="center"
|
||||
android:text="@string/no_camera_access" />
|
||||
</FrameLayout>
|
||||
```
|
||||
|
||||
This code block has a new [`FrameLayout`] named `preview_display_layout` and a
|
||||
[`TextView`] nested inside it, named `no_camera_access_preview`. When camera
|
||||
access permissions are not granted, our application will display the
|
||||
[`TextView`] with a string message, stored in the variable `no_camera_access`.
|
||||
Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file:
|
||||
|
||||
```
|
||||
<string name="no_camera_access" translatable="false">Please grant camera permissions.</string>
|
||||
```
|
||||
|
||||
When the user doesn't grant camera permission, the screen will now look like
|
||||
this:
|
||||
|
||||
|
||||
|
||||
Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
|
||||
`MainActivity`:
|
||||
|
||||
```
|
||||
private SurfaceTexture previewFrameTexture;
|
||||
private SurfaceView previewDisplayView;
|
||||
```
|
||||
|
||||
In the `onCreate(Bundle)` function, add the following two lines _before_
|
||||
requesting camera permissions:
|
||||
|
||||
```
|
||||
previewDisplayView = new SurfaceView(this);
|
||||
setupPreviewDisplayView();
|
||||
```
|
||||
|
||||
And now add the code defining `setupPreviewDisplayView()`:
|
||||
|
||||
```
|
||||
private void setupPreviewDisplayView() {
|
||||
previewDisplayView.setVisibility(View.GONE);
|
||||
ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
|
||||
viewGroup.addView(previewDisplayView);
|
||||
}
|
||||
```
|
||||
|
||||
We define a new [`SurfaceView`] object and add it to the
|
||||
`preview_display_layout` [`FrameLayout`] object so that we can use it to display
|
||||
the camera frames using a [`SurfaceTexture`] object named `previewFrameTexture`.
|
||||
|
||||
To use `previewFrameTexture` for getting camera frames, we will use [CameraX].
|
||||
MediaPipe provides a utility named [`CameraXPreviewHelper`] to use [CameraX].
|
||||
This class updates a listener when camera is started via
|
||||
`onCameraStarted(@Nullable SurfaceTexture)`.
|
||||
|
||||
To use this utility, modify the `BUILD` file to add a dependency on
|
||||
`"//mediapipe/java/com/google/mediapipe/components:android_camerax_helper"`.
|
||||
|
||||
Now import [`CameraXPreviewHelper`] and add the following line to
|
||||
`MainActivity`:
|
||||
|
||||
```
|
||||
private CameraXPreviewHelper cameraHelper;
|
||||
```
|
||||
|
||||
Now, we can add our implementation to `startCamera()`:
|
||||
|
||||
```
|
||||
public void startCamera() {
|
||||
cameraHelper = new CameraXPreviewHelper();
|
||||
cameraHelper.setOnCameraStartedListener(
|
||||
surfaceTexture -> {
|
||||
previewFrameTexture = surfaceTexture;
|
||||
// Make the display view visible to start showing the preview.
|
||||
previewDisplayView.setVisibility(View.VISIBLE);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
This creates a new [`CameraXPreviewHelper`] object and adds an anonymous
|
||||
listener on the object. When `cameraHelper` signals that the camera has started
|
||||
and a `surfaceTexture` to grab frames is available, we save that
|
||||
`surfaceTexture` as `previewFrameTexture`, and make the `previewDisplayView`
|
||||
visible so that we can start seeing frames from the `previewFrameTexture`.
|
||||
|
||||
However, before starting the camera, we need to decide which camera we want to
|
||||
use. [`CameraXPreviewHelper`] inherits from [`CameraHelper`] which provides two
|
||||
options, `FRONT` and `BACK`. We can pass in the decision from the `BUILD` file
|
||||
as metadata such that no code change is required to build a another version of
|
||||
the app using a different camera.
|
||||
|
||||
Assuming we want to use `BACK` camera to perform edge detection on a live scene
|
||||
that we view from the camera, add the metadata into `AndroidManifest.xml`:
|
||||
|
||||
```
|
||||
...
|
||||
<meta-data android:name="cameraFacingFront" android:value="${cameraFacingFront}"/>
|
||||
</application>
|
||||
</manifest>
|
||||
```
|
||||
|
||||
and specify the selection in `BUILD` in the `helloworld` android binary rule
|
||||
with a new entry in `manifest_values`:
|
||||
|
||||
```
|
||||
manifest_values = {
|
||||
"applicationId": "com.google.mediapipe.apps.basic",
|
||||
"appName": "Hello World",
|
||||
"mainActivity": ".MainActivity",
|
||||
"cameraFacingFront": "False",
|
||||
},
|
||||
```
|
||||
|
||||
Now, in `MainActivity` to retrieve the metadata specified in `manifest_values`,
|
||||
add an [`ApplicationInfo`] object:
|
||||
|
||||
```
|
||||
private ApplicationInfo applicationInfo;
|
||||
```
|
||||
|
||||
In the `onCreate()` function, add:
|
||||
|
||||
```
|
||||
try {
|
||||
applicationInfo =
|
||||
getPackageManager().getApplicationInfo(getPackageName(), PackageManager.GET_META_DATA);
|
||||
} catch (NameNotFoundException e) {
|
||||
Log.e(TAG, "Cannot find application info: " + e);
|
||||
}
|
||||
```
|
||||
|
||||
Now add the following line at the end of the `startCamera()` function:
|
||||
|
||||
```
|
||||
CameraHelper.CameraFacing cameraFacing =
|
||||
applicationInfo.metaData.getBoolean("cameraFacingFront", false)
|
||||
? CameraHelper.CameraFacing.FRONT
|
||||
: CameraHelper.CameraFacing.BACK;
|
||||
cameraHelper.startCamera(this, cameraFacing, /*surfaceTexture=*/ null);
|
||||
```
|
||||
|
||||
At this point, the application should build successfully. However, when you run
|
||||
the application on your device, you will see a black screen (even though camera
|
||||
permissions have been granted). This is because even though we save the
|
||||
`surfaceTexture` variable provided by the [`CameraXPreviewHelper`], the
|
||||
`previewSurfaceView` doesn't use its output and display it on screen yet.
|
||||
|
||||
Since we want to use the frames in a MediaPipe graph, we will not add code to
|
||||
view the camera output directly in this tutorial. Instead, we skip ahead to how
|
||||
we can send camera frames for processing to a MediaPipe graph and display the
|
||||
output of the graph on the screen.
|
||||
|
||||
## `ExternalTextureConverter` setup
|
||||
|
||||
A [`SurfaceTexture`] captures image frames from a stream as an OpenGL ES
|
||||
texture. To use a MediaPipe graph, frames captured from the camera should be
|
||||
stored in a regular Open GL texture object. MediaPipe provides a class,
|
||||
[`ExternalTextureConverter`] to convert the image stored in a [`SurfaceTexture`]
|
||||
object to a regular OpenGL texture object.
|
||||
|
||||
To use [`ExternalTextureConverter`], we also need an `EGLContext`, which is
|
||||
created and managed by an [`EglManager`] object. Add a dependency to the `BUILD`
|
||||
file to use [`EglManager`], `"//mediapipe/java/com/google/mediapipe/glutil"`.
|
||||
|
||||
In `MainActivity`, add the following declarations:
|
||||
|
||||
```
|
||||
private EglManager eglManager;
|
||||
private ExternalTextureConverter converter;
|
||||
```
|
||||
|
||||
In the `onCreate(Bundle)` function, add a statement to initialize the
|
||||
`eglManager` object before requesting camera permissions:
|
||||
|
||||
```
|
||||
eglManager = new EglManager(null);
|
||||
```
|
||||
|
||||
Recall that we defined the `onResume()` function in `MainActivity` to confirm
|
||||
camera permissions have been granted and call `startCamera()`. Before this
|
||||
check, add the following line in `onResume()` to initialize the `converter`
|
||||
object:
|
||||
|
||||
```
|
||||
converter = new ExternalTextureConverter(eglManager.getContext());
|
||||
```
|
||||
|
||||
This `converter` now uses the `GLContext` managed by `eglManager`.
|
||||
|
||||
We also need to override the `onPause()` function in the `MainActivity` so that
|
||||
if the application goes into a paused state, we close the `converter` properly:
|
||||
|
||||
```
|
||||
@Override
|
||||
protected void onPause() {
|
||||
super.onPause();
|
||||
converter.close();
|
||||
}
|
||||
```
|
||||
|
||||
To pipe the output of `previewFrameTexture` to the `converter`, add the
|
||||
following block of code to `setupPreviewDisplayView()`:
|
||||
|
||||
```
|
||||
previewDisplayView
|
||||
.getHolder()
|
||||
.addCallback(
|
||||
new SurfaceHolder.Callback() {
|
||||
@Override
|
||||
public void surfaceCreated(SurfaceHolder holder) {}
|
||||
|
||||
@Override
|
||||
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
|
||||
// (Re-)Compute the ideal size of the camera-preview display (the area that the
|
||||
// camera-preview frames get rendered onto, potentially with scaling and rotation)
|
||||
// based on the size of the SurfaceView that contains the display.
|
||||
Size viewSize = new Size(width, height);
|
||||
Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
|
||||
|
||||
// Connect the converter to the camera-preview frames as its input (via
|
||||
// previewFrameTexture), and configure the output width and height as the computed
|
||||
// display size.
|
||||
converter.setSurfaceTextureAndAttachToGLContext(
|
||||
previewFrameTexture, displaySize.getWidth(), displaySize.getHeight());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void surfaceDestroyed(SurfaceHolder holder) {}
|
||||
});
|
||||
```
|
||||
|
||||
In this code block, we add a custom [`SurfaceHolder.Callback`] to
|
||||
`previewDisplayView` and implement the `surfaceChanged(SurfaceHolder holder, int
|
||||
format, int width, int height)` function to compute an appropriate display size
|
||||
of the camera frames on the device screen and to tie the `previewFrameTexture`
|
||||
object and send frames of the computed `displaySize` to the `converter`.
|
||||
|
||||
We are now ready to use camera frames in a MediaPipe graph.
|
||||
|
||||
## Using a MediaPipe graph in Android
|
||||
|
||||
### Add relevant dependencies
|
||||
|
||||
To use a MediaPipe graph, we need to add dependencies to the MediaPipe framework
|
||||
on Android. We will first add a build rule to build a `cc_binary` using JNI code
|
||||
of the MediaPipe framework and then build a `cc_library` rule to use this binary
|
||||
in our application. Add the following code block to your `BUILD` file:
|
||||
|
||||
```
|
||||
cc_binary(
|
||||
name = "libmediapipe_jni.so",
|
||||
linkshared = 1,
|
||||
linkstatic = 1,
|
||||
deps = [
|
||||
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "mediapipe_jni_lib",
|
||||
srcs = [":libmediapipe_jni.so"],
|
||||
alwayslink = 1,
|
||||
)
|
||||
```
|
||||
|
||||
Add the dependency `":mediapipe_jni_lib"` to the `mediapipe_lib` build rule in
|
||||
the `BUILD` file.
|
||||
|
||||
Next, we need to add dependencies specific to the MediaPipe graph we want to use
|
||||
in the application.
|
||||
|
||||
First, add dependencies to all calculator code in the `libmediapipe_jni.so`
|
||||
build rule:
|
||||
|
||||
```
|
||||
"//mediapipe/graphs/edge_detection:mobile_calculators",
|
||||
```
|
||||
|
||||
MediaPipe graphs are `.pbtxt` files, but to use them in the application, we need
|
||||
to use the `mediapipe_binary_graph` build rule to generate a `.binarypb` file.
|
||||
|
||||
In the `helloworld` android binary build rule, add the `mediapipe_binary_graph`
|
||||
target specific to the graph as an asset:
|
||||
|
||||
```
|
||||
assets = [
|
||||
"//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph",
|
||||
],
|
||||
assets_dir = "",
|
||||
```
|
||||
|
||||
In the `assets` build rule, you can also add other assets such as TensorFlowLite
|
||||
models used in your graph.
|
||||
|
||||
In addition, add additional `manifest_values` for properties specific to the
|
||||
graph, to be later retrieved in `MainActivity`:
|
||||
|
||||
```
|
||||
manifest_values = {
|
||||
"applicationId": "com.google.mediapipe.apps.basic",
|
||||
"appName": "Hello World",
|
||||
"mainActivity": ".MainActivity",
|
||||
"cameraFacingFront": "False",
|
||||
"binaryGraphName": "mobile_gpu.binarypb",
|
||||
"inputVideoStreamName": "input_video",
|
||||
"outputVideoStreamName": "output_video",
|
||||
},
|
||||
```
|
||||
|
||||
Note that `binaryGraphName` indicates the filename of the binary graph,
|
||||
determined by the `output_name` field in the `mediapipe_binary_graph` target.
|
||||
`inputVideoStreamName` and `outputVideoStreamName` are the input and output
|
||||
video stream name specified in the graph respectively.
|
||||
|
||||
Now, the `MainActivity` needs to load the MediaPipe framework. Also, the
|
||||
framework uses OpenCV, so `MainActvity` should also load `OpenCV`. Use the
|
||||
following code in `MainActivity` (inside the class, but not inside any function)
|
||||
to load both dependencies:
|
||||
|
||||
```
|
||||
static {
|
||||
// Load all native libraries needed by the app.
|
||||
System.loadLibrary("mediapipe_jni");
|
||||
System.loadLibrary("opencv_java3");
|
||||
}
|
||||
```
|
||||
|
||||
### Use the graph in `MainActivity`
|
||||
|
||||
First, we need to load the asset which contains the `.binarypb` compiled from
|
||||
the `.pbtxt` file of the graph. To do this, we can use a MediaPipe utility,
|
||||
[`AndroidAssetUtil`].
|
||||
|
||||
Initialize the asset manager in `onCreate(Bundle)` before initializing
|
||||
`eglManager`:
|
||||
|
||||
```
|
||||
// Initialize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
|
||||
// binary graphs.
|
||||
AndroidAssetUtil.initializeNativeAssetManager(this);
|
||||
```
|
||||
|
||||
Now, we need to setup a [`FrameProcessor`] object that sends camera frames
|
||||
prepared by the `converter` to the MediaPipe graph and runs the graph, prepares
|
||||
the output and then updates the `previewDisplayView` to display the output. Add
|
||||
the following code to declare the `FrameProcessor`:
|
||||
|
||||
```
|
||||
private FrameProcessor processor;
|
||||
```
|
||||
|
||||
and initialize it in `onCreate(Bundle)` after initializing `eglManager`:
|
||||
|
||||
```
|
||||
processor =
|
||||
new FrameProcessor(
|
||||
this,
|
||||
eglManager.getNativeContext(),
|
||||
applicationInfo.metaData.getString("binaryGraphName"),
|
||||
applicationInfo.metaData.getString("inputVideoStreamName"),
|
||||
applicationInfo.metaData.getString("outputVideoStreamName"));
|
||||
```
|
||||
|
||||
The `processor` needs to consume the converted frames from the `converter` for
|
||||
processing. Add the following line to `onResume()` after initializing the
|
||||
`converter`:
|
||||
|
||||
```
|
||||
converter.setConsumer(processor);
|
||||
```
|
||||
|
||||
The `processor` should send its output to `previewDisplayView` To do this, add
|
||||
the following function definitions to our custom [`SurfaceHolder.Callback`]:
|
||||
|
||||
```
|
||||
@Override
|
||||
public void surfaceCreated(SurfaceHolder holder) {
|
||||
processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void surfaceDestroyed(SurfaceHolder holder) {
|
||||
processor.getVideoSurfaceOutput().setSurface(null);
|
||||
}
|
||||
```
|
||||
|
||||
When the `SurfaceHolder` is created, we had the `Surface` to the
|
||||
`VideoSurfaceOutput` of the `processor`. When it is destroyed, we remove it from
|
||||
the `VideoSurfaceOutput` of the `processor`.
|
||||
|
||||
And that's it! You should now be able to successfully build and run the
|
||||
application on the device and see Sobel edge detection running on a live camera
|
||||
feed! Congrats!
|
||||
|
||||
|
||||
|
||||
If you ran into any issues, please see the full code of the tutorial
|
||||
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic).
|
||||
|
||||
[`ApplicationInfo`]:https://developer.android.com/reference/android/content/pm/ApplicationInfo
|
||||
[`AndroidAssetUtil`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/framework/AndroidAssetUtil.java
|
||||
[Bazel]:https://bazel.build/
|
||||
[`CameraHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraHelper.java
|
||||
[CameraX]:https://developer.android.com/training/camerax
|
||||
[`CameraXPreviewHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraXPreviewHelper.java
|
||||
[developer options]:https://developer.android.com/studio/debug/dev-options
|
||||
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
|
||||
[`EglManager`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/glutil/EglManager.java
|
||||
[`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java
|
||||
[`FrameLayout`]:https://developer.android.com/reference/android/widget/FrameLayout
|
||||
[`FrameProcessor`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/FrameProcessor.java
|
||||
[MediaPipe installation guide]:./install.md
|
||||
[`PermissionHelper`]: https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/PermissionHelper.java
|
||||
[`SurfaceHolder.Callback`]:https://developer.android.com/reference/android/view/SurfaceHolder.Callback.html
|
||||
[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
|
||||
[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
|
||||
[`SurfaceTexture`]:https://developer.android.com/reference/android/graphics/SurfaceTexture
|
||||
[`TextView`]:https://developer.android.com/reference/android/widget/TextView
|
||||
@@ -1,116 +0,0 @@
|
||||
## Hello World for C++
|
||||
|
||||
1. Ensure you have a working version of MediaPipe. See
|
||||
[installation instructions](./install.md).
|
||||
|
||||
2. To run the [`hello world`] example:
|
||||
|
||||
```bash
|
||||
$ git clone https://github.com/google/mediapipe/mediapipe.git
|
||||
$ cd mediapipe
|
||||
|
||||
$ export GLOG_logtostderr=1
|
||||
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is not supported currently.
|
||||
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
|
||||
mediapipe/examples/desktop/hello_world:hello_world
|
||||
|
||||
# It should print 10 rows of Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
# Hello World!
|
||||
```
|
||||
|
||||
3. The [`hello world`] example uses a simple MediaPipe graph in the
|
||||
`PrintHelloWorld()` function, defined in a [`CalculatorGraphConfig`] proto.
|
||||
|
||||
```C++
|
||||
::mediapipe::Status PrintHelloWorld() {
|
||||
// Configures a simple graph, which concatenates 2 PassThroughCalculators.
|
||||
CalculatorGraphConfig config = ParseTextProtoOrDie<CalculatorGraphConfig>(R"(
|
||||
input_stream: "in"
|
||||
output_stream: "out"
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "in"
|
||||
output_stream: "out1"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out1"
|
||||
output_stream: "out"
|
||||
}
|
||||
)");
|
||||
```
|
||||
|
||||
You can visualize this graph using
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev) by pasting the
|
||||
CalculatorGraphConfig content below into the visualizer. See
|
||||
[here](./visualizer.md) for help on the visualizer.
|
||||
|
||||
```bash
|
||||
input_stream: "in"
|
||||
output_stream: "out"
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "in"
|
||||
output_stream: "out1"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out1"
|
||||
output_stream: "out"
|
||||
}
|
||||
```
|
||||
|
||||
This graph consists of 1 graph input stream (`in`) and 1 graph output stream
|
||||
(`out`), and 2 [`PassThroughCalculator`]s connected serially.
|
||||
|
||||
|
||||
|
||||
4. Before running the graph, an `OutputStreamPoller` object is connected to the
|
||||
output stream in order to later retrieve the graph output, and a graph run
|
||||
is started with [`StartRun`].
|
||||
|
||||
```c++
|
||||
CalculatorGraph graph;
|
||||
RETURN_IF_ERROR(graph.Initialize(config));
|
||||
ASSIGN_OR_RETURN(OutputStreamPoller poller,
|
||||
graph.AddOutputStreamPoller("out"));
|
||||
RETURN_IF_ERROR(graph.StartRun({}));
|
||||
```
|
||||
|
||||
5. The example then creates 10 packets (each packet contains a string "Hello
|
||||
World!" with Timestamp values ranging from 0, 1, ... 9) using the
|
||||
[`MakePacket`] function, adds each packet into the graph through the `in`
|
||||
input stream, and finally closes the input stream to finish the graph run.
|
||||
|
||||
```c++
|
||||
for (int i = 0; i < 10; ++i) {
|
||||
RETURN_IF_ERROR(graph.AddPacketToInputStream("in", MakePacket<std::string>("Hello World!").At(Timestamp(i))));
|
||||
}
|
||||
RETURN_IF_ERROR(graph.CloseInputStream("in"));
|
||||
```
|
||||
|
||||
6. Through the `OutputStreamPoller` object the example then retrieves all 10
|
||||
packets from the output stream, gets the string content out of each packet
|
||||
and prints it to the output log.
|
||||
|
||||
```c++
|
||||
mediapipe::Packet packet;
|
||||
while (poller.Next(&packet)) {
|
||||
LOG(INFO) << packet.Get<string>();
|
||||
}
|
||||
```
|
||||
|
||||
[`hello world`]: https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hello_world/hello_world.cc
|
||||
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
|
||||
[`PassThroughCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/pass_through_calculator.cc
|
||||
[`MakePacket`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/packet.h
|
||||
[`StartRun`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
|
||||
@@ -1,548 +0,0 @@
|
||||
# Hello World! in MediaPipe on iOS
|
||||
|
||||
## Introduction
|
||||
|
||||
This codelab uses MediaPipe on an iOS device.
|
||||
|
||||
### What you will learn
|
||||
|
||||
How to develop an iOS application that uses MediaPipe and run a MediaPipe
|
||||
graph on iOS.
|
||||
|
||||
### What you will build
|
||||
|
||||
A simple camera app for real-time Sobel edge detection applied to a live video
|
||||
stream on an iOS device.
|
||||
|
||||
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install MediaPipe on your system, see [MediaPipe installation guide] for
|
||||
details.
|
||||
2. Setup your iOS device for development.
|
||||
3. Setup [Bazel] on your system to build and deploy the iOS app.
|
||||
|
||||
## Graph for edge detection
|
||||
|
||||
We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
|
||||
|
||||
```
|
||||
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
|
||||
# Used in the examples
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/edgedetectiongpu.
|
||||
# mediapipe/examples/ios/edgedetectiongpu.
|
||||
|
||||
# Images coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Converts RGB images into luminance images, still stored in RGB format.
|
||||
node: {
|
||||
calculator: "LuminanceCalculator"
|
||||
input_stream: "input_video"
|
||||
output_stream: "luma_video"
|
||||
}
|
||||
|
||||
# Applies the Sobel filter to luminance images sotred in RGB format.
|
||||
node: {
|
||||
calculator: "SobelEdgesCalculator"
|
||||
input_stream: "luma_video"
|
||||
output_stream: "output_video"
|
||||
}
|
||||
```
|
||||
|
||||
A visualization of the graph is shown below:
|
||||
|
||||
|
||||
|
||||
This graph has a single input stream named `input_video` for all incoming frames
|
||||
that will be provided by your device's camera.
|
||||
|
||||
The first node in the graph, `LuminanceCalculator`, takes a single packet (image
|
||||
frame) and applies a change in luminance using an OpenGL shader. The resulting
|
||||
image frame is sent to the `luma_video` output stream.
|
||||
|
||||
The second node, `SobelEdgesCalculator` applies edge detection to incoming
|
||||
packets in the `luma_video` stream and outputs results in `output_video` output
|
||||
stream.
|
||||
|
||||
Our iOS application will display the output image frames of the `output_video`
|
||||
stream.
|
||||
|
||||
## Initial minimal application setup
|
||||
|
||||
We first start with a simple iOS application and demonstrate how to use `bazel`
|
||||
to build it.
|
||||
|
||||
First, create an XCode project via File > New > Single View App.
|
||||
|
||||
Set the product name to "EdgeDetectionGpu", and use an appropriate organization
|
||||
identifier, such as `com.google.mediapipe`. The organization identifier
|
||||
alongwith the product name will be the `bundle_id` for the application, such as
|
||||
`com.google.mediapipe.EdgeDetectionGpu`.
|
||||
|
||||
Set the language to Objective-C.
|
||||
|
||||
Save the project to an appropriate location. Let's call this
|
||||
`$PROJECT_TEMPLATE_LOC`. So your project will be in the
|
||||
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu` directory. This directory will contain
|
||||
another directory named `EdgeDetectionGpu` and an `EdgeDetectionGpu.xcodeproj` file.
|
||||
|
||||
The `EdgeDetectionGpu.xcodeproj` will not be useful for this tutorial, as we will
|
||||
use bazel to build the iOS application. The content of the
|
||||
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu/EdgeDetectionGpu` directory is listed below:
|
||||
|
||||
1. `AppDelegate.h` and `AppDelegate.m`
|
||||
2. `ViewController.h` and `ViewController.m`
|
||||
3. `main.m`
|
||||
4. `Info.plist`
|
||||
5. `Main.storyboard` and `Launch.storyboard`
|
||||
6. `Assets.xcassets` directory.
|
||||
|
||||
Copy these files to a directory named `EdgeDetectionGpu` to a location that can
|
||||
access the MediaPipe source code. For example, the source code of the
|
||||
application that we will build in this tutorial is located in
|
||||
`mediapipe/examples/ios/EdgeDetectionGpu`. We will refer to this path as the
|
||||
`$APPLICATION_PATH` throughout the codelab.
|
||||
|
||||
Note: MediaPipe provides Objective-C bindings for iOS. The edge detection
|
||||
application in this tutorial and all iOS examples using MediaPipe use
|
||||
Objective-C with C++ in `.mm` files.
|
||||
|
||||
Create a `BUILD` file in the `$APPLICATION_PATH` and add the following build
|
||||
rules:
|
||||
|
||||
```
|
||||
MIN_IOS_VERSION = "10.0"
|
||||
|
||||
load(
|
||||
"@build_bazel_rules_apple//apple:ios.bzl",
|
||||
"ios_application",
|
||||
)
|
||||
|
||||
ios_application(
|
||||
name = "EdgeDetectionGpuApp",
|
||||
bundle_id = "com.google.mediapipe.EdgeDetectionGpu",
|
||||
families = [
|
||||
"iphone",
|
||||
"ipad",
|
||||
],
|
||||
infoplists = ["Info.plist"],
|
||||
minimum_os_version = MIN_IOS_VERSION,
|
||||
provisioning_profile = "//mediapipe/examples/ios:developer_provisioning_profile",
|
||||
deps = [":EdgeDetectionGpuAppLibrary"],
|
||||
)
|
||||
|
||||
objc_library(
|
||||
name = "EdgeDetectionGpuAppLibrary",
|
||||
srcs = [
|
||||
"AppDelegate.m",
|
||||
"ViewController.m",
|
||||
"main.m",
|
||||
],
|
||||
hdrs = [
|
||||
"AppDelegate.h",
|
||||
"ViewController.h",
|
||||
],
|
||||
data = [
|
||||
"Base.lproj/LaunchScreen.storyboard",
|
||||
"Base.lproj/Main.storyboard",
|
||||
],
|
||||
sdk_frameworks = [
|
||||
"UIKit",
|
||||
],
|
||||
deps = [],
|
||||
)
|
||||
```
|
||||
|
||||
The `objc_library` rule adds dependencies for the `AppDelegate` and
|
||||
`ViewController` classes, `main.m` and the application storyboards. The
|
||||
templated app depends only on the `UIKit` SDK.
|
||||
|
||||
The `ios_application` rule uses the `EdgeDetectionGpuAppLibrary` Objective-C
|
||||
library generated to build an iOS application for installation on your iOS
|
||||
device.
|
||||
|
||||
Note: You need to point to your own iOS developer provisioning profile to be
|
||||
able to run the application on your iOS device.
|
||||
|
||||
To build the app, use the following command in a terminal:
|
||||
|
||||
```
|
||||
bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
|
||||
```
|
||||
|
||||
For example, to build the `EdgeDetectionGpuApp` application in
|
||||
`mediapipe/examples/ios/edgedetectiongpu`, use the following
|
||||
command:
|
||||
|
||||
```
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp
|
||||
```
|
||||
|
||||
Then, go back to XCode, open Window > Devices and Simulators, select your
|
||||
device, and add the `.ipa` file generated by the command above to your device.
|
||||
Here is the document on [setting up and compiling](./building_examples.md#ios) iOS
|
||||
MediaPipe apps.
|
||||
|
||||
Open the application on your device. Since it is empty, it should display a
|
||||
blank white screen.
|
||||
|
||||
## Use the camera for the live view feed
|
||||
|
||||
In this tutorial, we will use the `MPPCameraInputSource` class to access and
|
||||
grab frames from the camera. This class uses the `AVCaptureSession` API to get
|
||||
the frames from the camera.
|
||||
|
||||
But before using this class, change the `Info.plist` file to support camera
|
||||
usage in the app.
|
||||
|
||||
In `ViewController.m`, add the following import line:
|
||||
|
||||
```
|
||||
#import "mediapipe/objc/MPPCameraInputSource.h"
|
||||
```
|
||||
|
||||
Add the following to its implementation block to create an object
|
||||
`_cameraSource`:
|
||||
|
||||
```
|
||||
@implementation ViewController {
|
||||
// Handles camera access via AVCaptureSession library.
|
||||
MPPCameraInputSource* _cameraSource;
|
||||
}
|
||||
```
|
||||
|
||||
Add the following code to `viewDidLoad()`:
|
||||
|
||||
```
|
||||
-(void)viewDidLoad {
|
||||
[super viewDidLoad];
|
||||
|
||||
_cameraSource = [[MPPCameraInputSource alloc] init];
|
||||
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
|
||||
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
|
||||
// The frame's native format is rotated with respect to the portrait orientation.
|
||||
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
|
||||
}
|
||||
```
|
||||
|
||||
The code initializes `_cameraSource`, sets the capture session preset, and which
|
||||
camera to use.
|
||||
|
||||
We need to get frames from the `_cameraSource` into our application
|
||||
`ViewController` to display them. `MPPCameraInputSource` is a subclass of
|
||||
`MPPInputSource`, which provides a protocol for its delegates, namely the
|
||||
`MPPInputSourceDelegate`. So our application `ViewController` can be a delegate
|
||||
of `_cameraSource`.
|
||||
|
||||
To handle camera setup and process incoming frames, we should use a queue
|
||||
different from the main queue. Add the following to the implementation block of
|
||||
the `ViewController`:
|
||||
|
||||
```
|
||||
// Process camera frames on this queue.
|
||||
dispatch_queue_t _videoQueue;
|
||||
```
|
||||
|
||||
In `viewDidLoad()`, add the following line after initializing the
|
||||
`_cameraSource` object:
|
||||
|
||||
```
|
||||
[_cameraSource setDelegate:self queue:_videoQueue];
|
||||
```
|
||||
|
||||
And add the following code to initialize the queue before setting up the
|
||||
`_cameraSource` object:
|
||||
|
||||
```
|
||||
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
|
||||
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
|
||||
_videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
|
||||
```
|
||||
|
||||
We will use a serial queue with the priority `QOS_CLASS_USER_INTERACTIVE` for
|
||||
processing camera frames.
|
||||
|
||||
Add the following line after the header imports at the top of the file, before
|
||||
the interface/implementation of the `ViewController`:
|
||||
|
||||
```
|
||||
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
|
||||
```
|
||||
|
||||
Before implementing any method from `MPPInputSourceDelegate` protocol, we must
|
||||
first set up a way to display the camera frames. MediaPipe provides another
|
||||
utility called `MPPLayerRenderer` to display images on the screen. This utility
|
||||
can be used to display `CVPixelBufferRef` objects, which is the type of the
|
||||
images provided by `MPPCameraInputSource` to its delegates.
|
||||
|
||||
To display images of the screen, we need to add a new `UIView` object called
|
||||
`_liveView` to the `ViewController`.
|
||||
|
||||
Add the following lines to the implementation block of the `ViewController`:
|
||||
|
||||
```
|
||||
// Display the camera preview frames.
|
||||
IBOutlet UIView* _liveView;
|
||||
// Render frames in a layer.
|
||||
MPPLayerRenderer* _renderer;
|
||||
```
|
||||
|
||||
Go to `Main.storyboard`, add a `UIView` object from the object library to the
|
||||
`View` of the `ViewController` class. Add a referencing outlet from this view to
|
||||
the `_liveView` object you just added to the `ViewController` class. Resize the
|
||||
view so that it is centered and covers the entire application screen.
|
||||
|
||||
Go back to `ViewController.m` and add the following code to `viewDidLoad()` to
|
||||
initialize the `_renderer` object:
|
||||
|
||||
```
|
||||
_renderer = [[MPPLayerRenderer alloc] init];
|
||||
_renderer.layer.frame = _liveView.layer.bounds;
|
||||
[_liveView.layer addSublayer:_renderer.layer];
|
||||
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
|
||||
```
|
||||
|
||||
To get frames from the camera, we will implement the following method:
|
||||
|
||||
```
|
||||
// Must be invoked on _videoQueue.
|
||||
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
||||
timestamp:(CMTime)timestamp
|
||||
fromSource:(MPPInputSource*)source {
|
||||
if (source != _cameraSource) {
|
||||
NSLog(@"Unknown source: %@", source);
|
||||
return;
|
||||
}
|
||||
// Display the captured image on the screen.
|
||||
CFRetain(imageBuffer);
|
||||
dispatch_async(dispatch_get_main_queue(), ^{
|
||||
[_renderer renderPixelBuffer:imageBuffer];
|
||||
CFRelease(imageBuffer);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
This is a delegate method of `MPPInputSource`. We first check that we are
|
||||
getting frames from the right source, i.e. the `_cameraSource`. Then we display
|
||||
the frame received from the camera via `_renderer` on the main queue.
|
||||
|
||||
Now, we need to start the camera as soon as the view to display the frames is
|
||||
about to appear. To do this, we will implement the
|
||||
`viewWillAppear:(BOOL)animated` function:
|
||||
|
||||
```
|
||||
-(void)viewWillAppear:(BOOL)animated {
|
||||
[super viewWillAppear:animated];
|
||||
}
|
||||
```
|
||||
|
||||
Before we start running the camera, we need the user's permission to access it.
|
||||
`MPPCameraInputSource` provides a function
|
||||
`requestCameraAccessWithCompletionHandler:(void (^_Nullable)(BOOL
|
||||
granted))handler` to request camera access and do some work when the user has
|
||||
responded. Add the following code to `viewWillAppear:animated`:
|
||||
|
||||
```
|
||||
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
|
||||
if (granted) {
|
||||
dispatch_async(_videoQueue, ^{
|
||||
[_cameraSource start];
|
||||
});
|
||||
}
|
||||
}];
|
||||
```
|
||||
|
||||
Before building the application, add the following dependencies to your `BUILD`
|
||||
file:
|
||||
|
||||
```
|
||||
sdk_frameworks = [
|
||||
"AVFoundation",
|
||||
"CoreGraphics",
|
||||
"CoreMedia",
|
||||
],
|
||||
deps = [
|
||||
"//mediapipe/objc:mediapipe_framework_ios",
|
||||
"//mediapipe/objc:mediapipe_input_sources_ios",
|
||||
"//mediapipe/objc:mediapipe_layer_renderer",
|
||||
],
|
||||
```
|
||||
|
||||
Now build and run the application on your iOS device. You should see a live
|
||||
camera view feed after accepting camera permissions.
|
||||
|
||||
We are now ready to use camera frames in a MediaPipe graph.
|
||||
|
||||
## Using a MediaPipe graph in iOS
|
||||
|
||||
### Add relevant dependencies
|
||||
|
||||
We already added the dependencies of the MediaPipe framework code which contains
|
||||
the iOS API to use a MediaPipe graph. To use a MediaPipe graph, we need to add a
|
||||
dependency on the graph we intend to use in our application. Add the following
|
||||
line to the `data` list in your `BUILD` file:
|
||||
|
||||
```
|
||||
"//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph",
|
||||
```
|
||||
|
||||
Now add the dependency to the calculators used in this graph in the `deps` field
|
||||
in the `BUILD` file:
|
||||
|
||||
```
|
||||
"//mediapipe/graphs/edge_detection:mobile_calculators",
|
||||
```
|
||||
|
||||
Finally, rename the file `ViewController.m` to `ViewController.mm` to support
|
||||
Objective-C++.
|
||||
|
||||
### Use the graph in `ViewController`
|
||||
|
||||
Declare a static constant with the name of the graph, the input stream and the
|
||||
output stream:
|
||||
|
||||
```
|
||||
static NSString* const kGraphName = @"mobile_gpu";
|
||||
|
||||
static const char* kInputStream = "input_video";
|
||||
static const char* kOutputStream = "output_video";
|
||||
```
|
||||
|
||||
Add the following property to the interface of the `ViewController`:
|
||||
|
||||
```
|
||||
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
|
||||
// sent video frames on _videoQueue.
|
||||
@property(nonatomic) MPPGraph* mediapipeGraph;
|
||||
```
|
||||
|
||||
As explained in the comment above, we will initialize this graph in
|
||||
`viewDidLoad` first. To do so, we need to load the graph from the `.pbtxt` file
|
||||
using the following function:
|
||||
|
||||
```
|
||||
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
|
||||
// Load the graph config resource.
|
||||
NSError* configLoadError = nil;
|
||||
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
|
||||
if (!resource || resource.length == 0) {
|
||||
return nil;
|
||||
}
|
||||
NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
|
||||
NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
|
||||
if (!data) {
|
||||
NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
|
||||
return nil;
|
||||
}
|
||||
|
||||
// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
|
||||
mediapipe::CalculatorGraphConfig config;
|
||||
config.ParseFromArray(data.bytes, data.length);
|
||||
|
||||
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
|
||||
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
|
||||
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MPPPacketTypePixelBuffer];
|
||||
return newGraph;
|
||||
}
|
||||
```
|
||||
|
||||
Use this function to initialize the graph in `viewDidLoad` as follows:
|
||||
|
||||
```
|
||||
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
|
||||
```
|
||||
|
||||
The graph should send the results of processing camera frames back to the
|
||||
`ViewController`. Add the following line after initializing the graph to set the
|
||||
`ViewController` as a delegate of the `mediapipeGraph` object:
|
||||
|
||||
```
|
||||
self.mediapipeGraph.delegate = self;
|
||||
```
|
||||
|
||||
To avoid memory contention while processing frames from the live video feed, add
|
||||
the following line:
|
||||
|
||||
```
|
||||
// Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
|
||||
self.mediapipeGraph.maxFramesInFlight = 2;
|
||||
```
|
||||
|
||||
Now, start the graph when the user has granted the permission to use the camera
|
||||
in our app:
|
||||
|
||||
```
|
||||
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
|
||||
if (granted) {
|
||||
// Start running self.mediapipeGraph.
|
||||
NSError* error;
|
||||
if (![self.mediapipeGraph startWithError:&error]) {
|
||||
NSLog(@"Failed to start graph: %@", error);
|
||||
}
|
||||
|
||||
dispatch_async(_videoQueue, ^{
|
||||
[_cameraSource start];
|
||||
});
|
||||
}
|
||||
}];
|
||||
```
|
||||
|
||||
Note: It is important to start the graph before starting the camera, so that
|
||||
the graph is ready to process frames as soon as the camera starts sending them.
|
||||
|
||||
Earlier, when we received frames from the camera in the `processVideoFrame`
|
||||
function, we displayed them in the `_liveView` using the `_renderer`. Now, we
|
||||
need to send those frames to the graph and render the results instead. Modify
|
||||
this function's implementation to do the following:
|
||||
|
||||
```
|
||||
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
||||
timestamp:(CMTime)timestamp
|
||||
fromSource:(MPPInputSource*)source {
|
||||
if (source != _cameraSource) {
|
||||
NSLog(@"Unknown source: %@", source);
|
||||
return;
|
||||
}
|
||||
[self.mediapipeGraph sendPixelBuffer:imageBuffer
|
||||
intoStream:kInputStream
|
||||
packetType:MPPPacketTypePixelBuffer];
|
||||
}
|
||||
```
|
||||
|
||||
We send the `imageBuffer` to `self.mediapipeGraph` as a packet of type
|
||||
`MPPPacketTypePixelBuffer` into the input stream `kInputStream`, i.e.
|
||||
"input_video".
|
||||
|
||||
The graph will run with this input packet and output a result in
|
||||
`kOutputStream`, i.e. "output_video". We can implement the following delegate
|
||||
method to receive packets on this output stream and display them on the screen:
|
||||
|
||||
```
|
||||
- (void)mediapipeGraph:(MPPGraph*)graph
|
||||
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
|
||||
fromStream:(const std::string&)streamName {
|
||||
if (streamName == kOutputStream) {
|
||||
// Display the captured image on the screen.
|
||||
CVPixelBufferRetain(pixelBuffer);
|
||||
dispatch_async(dispatch_get_main_queue(), ^{
|
||||
[_renderer renderPixelBuffer:pixelBuffer];
|
||||
CVPixelBufferRelease(pixelBuffer);
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
And that is all! Build and run the app on your iOS device. You should see the
|
||||
results of running the edge detection graph on a live video feed. Congrats!
|
||||
|
||||
|
||||
|
||||
If you ran into any issues, please see the full code of the tutorial
|
||||
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
|
||||
|
||||
[Bazel]:https://bazel.build/
|
||||
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
|
||||
[MediaPipe installation guide]:./install.md
|
||||
@@ -1,41 +0,0 @@
|
||||
## Getting Help
|
||||
|
||||
- [Technical questions](#technical-questions)
|
||||
- [Bugs and feature requests](#bugs-and-feature-requests)
|
||||
|
||||
Below are the various ways to get help:
|
||||
|
||||
### Technical questions
|
||||
|
||||
For help with technical or algorithmic questions, visit
|
||||
[Stack Overflow](https://stackoverflow.com/questions/tagged/mediapipe) to find
|
||||
answers and support from the MediaPipe community.
|
||||
|
||||
### Bugs and feature requests
|
||||
|
||||
To report bugs or make feature requests,
|
||||
[file an issue on GitHub](https://github.com/google/mediapipe/issues).
|
||||
|
||||
If you open a GitHub issue, here is our policy:
|
||||
|
||||
1. It must be a bug, a feature request, or a significant problem with documentation (for small doc fixes please send a PR instead).
|
||||
2. The form below must be filled out.
|
||||
|
||||
**Here's why we have that policy**: MediaPipe developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.
|
||||
|
||||
------------------------
|
||||
|
||||
### System information
|
||||
- **Have I written custom code**:
|
||||
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
|
||||
- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
|
||||
- **Bazel version**:
|
||||
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev enviroment)**:
|
||||
- **Xcode & Tulsi version (if issue is related to building in mobile dev enviroment)**:
|
||||
- **Exact steps to reproduce**:
|
||||
|
||||
### Describe the problem
|
||||
Describe the problem clearly here. Be sure to convey here why it's a bug in MediaPipe or a feature request.
|
||||
|
||||
### Source code / logs
|
||||
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached instead of being pasted into the issue as text.
|
||||
@@ -1,142 +0,0 @@
|
||||
## Questions and Answers
|
||||
|
||||
- [How to convert ImageFrames and GpuBuffers](#how-to-convert-imageframes-and-gpubuffers)
|
||||
- [How to visualize perceived results](#how-to-visualize-perception-results)
|
||||
- [How to run calculators in parallel](#how-to-run-calculators-in-parallel)
|
||||
- [Output timestamps when using ImmediateInputStreamHandler](#output-timestamps-when-using-immediateinputstreamhandler)
|
||||
- [How to change settings at runtime](#how-to-change-settings-at-runtime)
|
||||
- [How to process real-time input streams](#how-to-process-real-time-input-streams)
|
||||
- [Can I run MediaPipe on MS Windows?](#can-i-run-mediapipe-on-ms-windows)
|
||||
|
||||
### How to convert ImageFrames and GpuBuffers
|
||||
|
||||
The Calculators [`ImageFrameToGpuBufferCalculator`] and
|
||||
[`GpuBufferToImageFrameCalculator`] convert back and forth between packets of
|
||||
type [`ImageFrame`] and [`GpuBuffer`]. [`ImageFrame`] refers to image data in
|
||||
CPU memory in any of a number of bitmap image formats. [`GpuBuffer`] refers to
|
||||
image data in GPU memory. You can find more detail in the Framework Concepts
|
||||
section
|
||||
[GpuBuffer to ImageFrame Converters](./gpu.md#gpubuffer-to-imageframe-converters).
|
||||
You can see an example in:
|
||||
|
||||
* [`object_detection_mobile_cpu.pbtxt`]
|
||||
|
||||
### How to visualize perception results
|
||||
|
||||
The [`AnnotationOverlayCalculator`] allows perception results, such as bounding
|
||||
boxes, arrows, and ovals, to be superimposed on the video frames aligned with
|
||||
the recognized objects. The results can be displayed in a diagnostic window when
|
||||
running on a workstation, or in a texture frame when running on device. You can
|
||||
see an example use of [`AnnotationOverlayCalculator`] in:
|
||||
|
||||
* [`face_detection_mobile_gpu.pbtxt`].
|
||||
|
||||
### How to run calculators in parallel
|
||||
|
||||
Within a calculator graph, MediaPipe routinely runs separate calculator nodes
|
||||
in parallel. MediaPipe maintains a pool of threads, and runs each calculator
|
||||
as soon as a thread is available and all of it's inputs are ready. Each
|
||||
calculator instance is only run for one set of inputs at a time, so most
|
||||
calculators need only to be *thread-compatible* and not *thread-safe*.
|
||||
|
||||
In order to enable one calculator to process multiple inputs in parallel, there
|
||||
are two possible approaches:
|
||||
|
||||
1. Define multiple calulator nodes and dispatch input packets to all nodes.
|
||||
2. Make the calculator thread-safe and configure its [`max_in_flight`] setting.
|
||||
|
||||
The first approach can be followed using the calculators designed to distribute
|
||||
packets across other calculators, such as [`RoundRobinDemuxCalculator`]. A
|
||||
single [`RoundRobinDemuxCalculator`] can distribute successive packets across
|
||||
several identically configured [`ScaleImageCalculator`] nodes.
|
||||
|
||||
The second approach allows up to [`max_in_flight`] invocations of the
|
||||
[`CalculatorBase::Process`] method on the same calculator node. The output
|
||||
packets from [`CalculatorBase::Process`] are automatically ordered by timestamp
|
||||
before they are passed along to downstream calculators.
|
||||
|
||||
With either aproach, you must be aware that the calculator running in parallel
|
||||
cannot maintain internal state in the same way as a normal sequential
|
||||
calculator.
|
||||
|
||||
### Output timestamps when using ImmediateInputStreamHandler
|
||||
|
||||
The [`ImmediateInputStreamHandler`] delivers each packet as soon as it arrives
|
||||
at an input stream. As a result, it can deliver a packet
|
||||
with a higher timestamp from one input stream before delivering a packet with a
|
||||
lower timestamp from a different input stream. If these input timestamps are
|
||||
both used for packets sent to one output stream, that output stream will
|
||||
complain that the timestamps are not monotonically increasing. In order to
|
||||
remedy this, the calculator must take care to output a packet only after
|
||||
processing is complete for its timestamp. This could be accomplished by waiting
|
||||
until input packets have been received from all inputstreams for that timestamp,
|
||||
or by ignoring a packet that arrives with a timestamp that has already been
|
||||
processed.
|
||||
|
||||
### How to change settings at runtime
|
||||
|
||||
There are two main approaches to changing the settings of a calculator graph
|
||||
while the application is running:
|
||||
|
||||
1. Restart the calculator graph with modified [`CalculatorGraphConfig`].
|
||||
2. Send new calculator options through packets on graph input-streams.
|
||||
|
||||
The first approach has the advantage of leveraging [`CalculatorGraphConfig`]
|
||||
processing tools such as "subgraphs". The second approach has the advantage of
|
||||
allowing active calculators and packets to remain in-flight while settings
|
||||
change. Mediapipe contributors are currently investigating alternative approaches
|
||||
to achieve both of these advantages.
|
||||
|
||||
### How to process realtime input streams
|
||||
|
||||
The mediapipe framework can be used to process data streams either online or
|
||||
offline. For offline processing, packets are pushed into the graph as soon as
|
||||
calculators are ready to process those packets. For online processing, one
|
||||
packet for each frame is pushed into the graph as that frame is recorded.
|
||||
|
||||
The MediaPipe framework requires only that successive packets be assigned
|
||||
monotonically increasing timestamps. By convention, realtime calculators and
|
||||
graphs use the recording time or the presentation time as the timestamp for each
|
||||
packet, with each timestamp representing microseconds since
|
||||
`Jan/1/1970:00:00:00`. This allows packets from various sources to be processed
|
||||
in a gloablly consistent order.
|
||||
|
||||
Normally for offline processing, every input packet is processed and processing
|
||||
continues as long as necessary. For online processing, it is often necessary to
|
||||
drop input packets in order to keep pace with the arrival of input data frames.
|
||||
When inputs arrive too frequently, the recommended technique for dropping
|
||||
packets is to use the MediaPipe calculators designed specifically for this
|
||||
purpose such as [`FlowLimiterCalculator`] and [`PacketClonerCalculator`].
|
||||
|
||||
For online processing, it is also necessary to promptly determine when processing
|
||||
can proceed. MediaPipe supports this by propagating timestamp bounds between
|
||||
calculators. Timestamp bounds indicate timestamp intervals that will contain no
|
||||
input packets, and they allow calculators to begin processing for those
|
||||
timestamps immediately. Calculators designed for realtime processing should
|
||||
carefully calculate timestamp bounds in order to begin processing as promptly as
|
||||
possible. For example, the [`MakePairCalculator`] uses the `SetOffset` API to
|
||||
propagate timestamp bounds from input streams to output streams.
|
||||
|
||||
### Can I run MediaPipe on MS Windows?
|
||||
|
||||
Currently MediaPipe portability supports Debian Linux, Ubuntu Linux,
|
||||
MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library
|
||||
conforming to the C++11 standard, so it is relatively easy to port to
|
||||
additional platforms.
|
||||
|
||||
[`object_detection_mobile_cpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt
|
||||
[`ImageFrame`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/formats/image_frame.h
|
||||
[`GpuBuffer`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer.h
|
||||
[`GpuBufferToImageFrameCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer_to_image_frame_calculator.cc
|
||||
[`ImageFrameToGpuBufferCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/image_frame_to_gpu_buffer_calculator.cc
|
||||
[`AnnotationOverlayCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/util/annotation_overlay_calculator.cc
|
||||
[`face_detection_mobile_gpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt
|
||||
[`CalculatorBase::Process`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
|
||||
[`max_in_flight`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
|
||||
[`RoundRobinDemuxCalculator`]: https://github.com/google/mediapipe/tree/master//mediapipe/calculators/core/round_robin_demux_calculator.cc
|
||||
[`ScaleImageCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/image/scale_image_calculator.cc
|
||||
[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.cc
|
||||
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
|
||||
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
|
||||
[`PacketClonerCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/packet_cloner_calculator.cc
|
||||
[`MakePairCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/make_pair_calculator.cc
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 51 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 885 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 66 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 8.2 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 170 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.5 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 361 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
File diff suppressed because one or more lines are too long
|
Before Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 195 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.0 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 118 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.1 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 923 B |
Binary file not shown.
|
Before Width: | Height: | Size: 46 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 217 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 163 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 9.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.8 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 302 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 9.4 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 35 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 75 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 56 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 666 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 529 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 808 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 121 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 121 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.0 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 59 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 460 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 64 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 299 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.1 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 383 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 84 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 293 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 93 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.6 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.7 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 448 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 150 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 20 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 149 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 193 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 213 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.5 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 475 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 112 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 100 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1004 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 945 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 336 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user