talksik/mediapipe
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
PiperOrigin-RevId: 263889205
This commit is contained in:
MediaPipe Team
2019-08-16 18:56:48 -07:00
committed by jqtang
parent dc40414468
commit 294687295d
443 changed files with 33160 additions and 2011 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mediapipe/docs/examples.md
+50 -25
View File
@@ -9,47 +9,72 @@ for Objective-C shortly.
### Hello World! on Android
[Hello World! on Android](./hello_world_android.md) should be the first mobile
example users go through in detail. It teaches the following:
Android example users go through in detail. It teaches the following:
* Introduction of a simple MediaPipe graph running on mobile GPUs for
[Sobel edge detection].
[Sobel edge detection](https://en.wikipedia.org/wiki/Sobel_operator).
* Building a simple baseline Android application that displays "Hello World!".
* Adding camera preview support into the baseline application using the
Android [CameraX] API.
* Incorporating the Sobel edge detection graph to process the live camera
preview and display the processed video in real-time.
### Object Detection with GPU on Android
### Hello World! on iOS
[Object Detection on GPU on Android](./object_detection_android_gpu.md)
illustrates how to use MediaPipe with a TFLite model for object detection in a
GPU-accelerated pipeline.
[Hello World! on iOS](./hello_world_ios.md) is the iOS version of Sobel edge
detection example
### Object Detection with CPU on Android
### Object Detection with GPU
[Object Detection on CPU on Android](./object_detection_android_cpu.md)
illustrates using the same TFLite model in a CPU-based pipeline. This example
highlights how graphs can be easily adapted to run on CPU v.s. GPU.
### Face Detection on Android
[Face Detection on Android](./face_detection_android_gpu.md) illustrates how to
use MediaPipe with a TFLite model for face detection in a GPU-accelerated
[Object Detection with GPU](./object_detection_mobile_gpu.md) illustrates how to
use MediaPipe with a TFLite model for object detection in a GPU-accelerated
pipeline.
* The selfie face detection TFLite model is based on
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface).
* [Model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
* [Android](./object_detection_mobile_gpu.md#android)
* [iOS](./object_detection_mobile_gpu.md#ios)
### Hair Segmentation on Android
### Object Detection with CPU
[Hair Segmentation on Android](./hair_segmentation_android_gpu.md) illustrates
how to use MediaPipe with a TFLite model for hair segmentation in a
GPU-accelerated pipeline.
[Object Detection with CPU](./object_detection_mobile_cpu.md) illustrates using
the same TFLite model in a CPU-based pipeline. This example highlights how
graphs can be easily adapted to run on CPU v.s. GPU.
* The selfie hair segmentation TFLite model is based on
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation).
* [Model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
### Face Detection with GPU
[Face Detection with GPU](./face_detection_mobile_gpu.md) illustrates how to use
MediaPipe with a TFLite model for face detection in a GPU-accelerated pipeline.
The selfie face detection TFLite model is based on
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface).
[Model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
* [Android](./face_detection_mobile_gpu.md#android)
* [iOS](./face_detection_mobile_gpu.md#ios)
### Hand Detection with GPU
[Hand Detection with GPU](./hand_detection_mobile_gpu.md) illustrates how to use
MediaPipe with a TFLite model for hand detection in a GPU-accelerated pipeline.
* [Android](./hand_detection_mobile_gpu.md#android)
* [iOS](./hand_detection_mobile_gpu.md#ios)
### Hand Tracking with GPU
[Hand Tracking with GPU](./hand_tracking_mobile_gpu.md) illustrates how to use
MediaPipe with a TFLite model for hand tracking in a GPU-accelerated pipeline.
* [Android](./hand_tracking_mobile_gpu.md#android)
* [iOS](./hand_tracking_mobile_gpu.md#ios)
### Hair Segmentation with GPU
[Hair Segmentation on GPU](./hair_segmentation_mobile_gpu.md) illustrates how to
use MediaPipe with a TFLite model for hair segmentation in a GPU-accelerated
pipeline. The selfie hair segmentation TFLite model is based on
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation).
[Model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
* [Android](./hair_segmentation_mobile_gpu.md#android)
## Desktop
mediapipe/docs/face_detection_android_gpu.md → mediapipe/docs/face_detection_mobile_gpu.md
+86 -80
View File
@@ -1,17 +1,18 @@
# Face Detection on Android
# Face Detection (GPU)
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe. This
doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_android_gpu.pbtxt)
This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
that performs face detection with TensorFlow Lite on GPU.
![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif){width="300"}
## App
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Face Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu)
[Face Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu)
example app. To build the app, run:
```bash
@@ -24,17 +25,33 @@ To further install the app on android device, run:
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.apk
```
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Face Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu).
To build the iOS app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp
```
## Graph
![face_detection_android_gpu_graph](images/mobile/face_detection_android_gpu.png){width="400"}
![face_detection_mobile_gpu_graph](images/mobile/face_detection_mobile_gpu.png){width="400"}
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
```bash
# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
# Used in the example in
# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectiongpu.
# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectiongpu and
# mediapipie/examples/ios/facedetectiongpu.
# Images on GPU coming into and out of the graph.
input_stream: "input_video"
@@ -54,7 +71,7 @@ output_stream: "output_video"
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
# processing previous inputs.
node {
calculator: "RealTimeFlowLimiterCalculator"
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:detections"
input_stream_info: {
@@ -64,40 +81,29 @@ node {
output_stream: "throttled_input_video"
}
# Transforms the input image on GPU to a 320x320 image. To scale the image, by
# default it uses the STRETCH scale mode that maps the entire input image to the
# entire transformed image. As a result, image aspect ratio may be changed and
# objects in the image may be deformed (stretched or squeezed), but the object
# detection model used in this graph is agnostic to that deformation.
# Transforms the input image on GPU to a 128x128 image. To scale the input
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
# resulting in potential letterboxing in the transformed image.
node: {
calculator: "ImageTransformationCalculator"
input_stream: "IMAGE_GPU:throttled_input_video"
output_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "LETTERBOX_PADDING:letterbox_padding"
node_options: {
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
output_width: 320
output_height: 320
output_width: 128
output_height: 128
scale_mode: FIT
}
}
}
# Converts the transformed input image on GPU into an image tensor stored in
# tflite::gpu::GlBuffer. The zero_center option is set to true to normalize the
# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f]. The flip_vertically
# option is set to true to account for the descrepancy between the
# representation of the input image (origin at the bottom-left corner, the
# OpenGL convention) and what the model used in this graph is expecting (origin
# at the top-left corner).
# Converts the transformed input image on GPU into an image tensor stored as a
# TfLiteTensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "TENSORS_GPU:image_tensor"
node_options: {
[type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
zero_center: true
flip_vertically: true
}
}
}
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
@@ -106,10 +112,10 @@ node {
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS_GPU:image_tensor"
output_stream: "TENSORS_GPU:detection_tensors"
output_stream: "TENSORS:detection_tensors"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "ssdlite_object_detection.tflite"
model_path: "face_detection_front.tflite"
}
}
}
@@ -121,25 +127,19 @@ node {
output_side_packet: "anchors"
node_options: {
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
input_size_height: 320
input_size_width: 320
num_layers: 4
min_scale: 0.1484375
max_scale: 0.75
input_size_height: 128
input_size_width: 128
anchor_offset_x: 0.5
anchor_offset_y: 0.5
strides: 8
strides: 16
strides: 16
strides: 16
strides: 32
strides: 64
strides: 128
strides: 256
strides: 512
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.3333
reduce_boxes_in_lowest_layer: true
fixed_anchor_size: true
}
}
}
@@ -149,22 +149,26 @@ node {
# detections. Each detection describes a detected object.
node {
calculator: "TfLiteTensorsToDetectionsCalculator"
input_stream: "TENSORS_GPU:detection_tensors"
input_stream: "TENSORS:detection_tensors"
input_side_packet: "ANCHORS:anchors"
output_stream: "DETECTIONS:detections"
node_options: {
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
num_classes: 91
num_boxes: 2034
num_coords: 4
ignore_classes: 0
num_classes: 1
num_boxes: 896
num_coords: 16
box_coord_offset: 0
keypoint_coord_offset: 4
num_keypoints: 6
num_values_per_keypoint: 2
sigmoid_score: true
apply_exponential_on_box_size: true
x_scale: 10.0
y_scale: 10.0
h_scale: 5.0
w_scale: 5.0
flip_vertically: true
score_clipping_thresh: 100.0
reverse_output_order: true
x_scale: 128.0
y_scale: 128.0
h_scale: 128.0
w_scale: 128.0
min_score_thresh: 0.75
}
}
}
@@ -176,56 +180,58 @@ node {
output_stream: "filtered_detections"
node_options: {
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
min_suppression_threshold: 0.4
min_score_threshold: 0.6
max_num_detections: 3
min_suppression_threshold: 0.3
overlap_type: INTERSECTION_OVER_UNION
algorithm: WEIGHTED
return_empty_detections: true
}
}
}
# Maps detection label IDs to the corresponding label text. The label map is
# provided in the label_map_path option.
# Maps detection label IDs to the corresponding label text ("Face"). The label
# map is provided in the label_map_path option.
node {
calculator: "DetectionLabelIdToTextCalculator"
input_stream: "filtered_detections"
output_stream: "output_detections"
output_stream: "labeled_detections"
node_options: {
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
label_map_path: "ssdlite_object_detection_labelmap.txt"
label_map_path: "face_detection_front_labelmap.txt"
}
}
}
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
# letterboxed image (after image transformation with the FIT scale mode) to the
# corresponding locations on the same image with the letterbox removed (the
# input image to the graph before image transformation).
node {
calculator: "DetectionLetterboxRemovalCalculator"
input_stream: "DETECTIONS:labeled_detections"
input_stream: "LETTERBOX_PADDING:letterbox_padding"
output_stream: "DETECTIONS:output_detections"
}
# Converts the detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTION_VECTOR:output_detections"
input_stream: "DETECTIONS:output_detections"
output_stream: "RENDER_DATA:render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
thickness: 10.0
color { r: 255 g: 0 b: 0 }
}
}
}
# Draws annotations and overlays them on top of the original image coming into
# the graph. Annotation drawing is performed on CPU, and the result is
# transferred to GPU and overlaid on the input image. The calculator assumes
# that image origin is always at the top-left corner and renders text
# accordingly. However, the input image has its origin at the bottom-left corner
# (OpenGL convention) and the flip_text_vertically option is set to true to
# compensate that.
# Draws annotations and overlays them on top of a GPU copy of the original
# image coming into the graph. The calculator assumes that image origin is
# always at the top-left corner and renders text accordingly.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "INPUT_FRAME_GPU:throttled_input_video"
input_stream: "render_data"
output_stream: "OUTPUT_FRAME_GPU:output_video"
node_options: {
[type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
flip_text_vertically: true
}
}
}
```
mediapipe/docs/framework_concepts.md
+18 -120
View File
@@ -252,21 +252,23 @@ node {
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
public interface to a subgraph consists of a set of input and output streams
similar to the public interface of a calculator. The subgraph can then be
public interface of a subgraph consists of a set of input and output streams
similar to a calculator's public interface. The subgraph can then be
included in an `CalculatorGraphConfig` as if it were a calculator. When a
MediaPipe graph is loaded from a `CalculatorGraphConfig`, each subgraph node is
replaced by the corresponding graph of calculators. As a result, the semantics
and performance of the subgraph is identical to the corresponding graph of
calculators.
Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`
Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`.
1. Defining the subgraph.
1. Defining the subgraph.
```proto
# This subgraph is defined in two_pass_through_subgraph.pbtxt
# that is registered in the BUILD file as "TwoPassThroughSubgraph"
# and is registered as "TwoPassThroughSubgraph"
type: "TwoPassThroughSubgraph"
input_stream: "out1"
output_stream: "out3"
@@ -282,19 +284,20 @@ Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`
}
```
The public interface to the graph that consist of:
* Graph input streams
* Graph output streams
* Graph input side packets
* Graph output side packets
The public interface to the subgraph consists of:
2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`
* The parameter `register_as` defines the component name for the new subgraph
* Graph input streams
* Graph output streams
* Graph input side packets
* Graph output side packets
2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`. The
parameter `register_as` defines the component name for the new subgraph.
```proto
# Small section of BUILD file for registering the "TwoPassThroughSubgraph"
# subgraph for use by main graph main_pass_throughcals.pbtxt
#
mediapipe_simple_subgraph(
name = "twopassthrough_subgraph",
graph = "twopassthrough_subgraph.pbtxt",
@@ -306,12 +309,12 @@ The public interface to the graph that consist of:
)
```
3. Use the subgraph in the main graph
3. Use the subgraph in the main graph.
```proto
# This main graph is defined in main_pass_throughcals.pbtxt
# using subgraph called "TwoPassThroughSubgraph"
#
input_stream: "in"
node {
calculator: "PassThroughCalculator"
@@ -329,108 +332,3 @@ The public interface to the graph that consist of:
output_stream: "out4"
}
```
<!---
### Graph Templates
A MediaPipe graph template looks exactly like a calculator graph .pbtxt file with some embedded parameters like `%num_detectors%`. When the template parameters are replaced by argument values, the template defines a complete `CalculatorGraphConfig`.
1. Defining and using a graph template by writing a `CalculatorGraphConfig` text protobuf file containing template parameters. The file extension .textpbt stands for "text protobuf template".
```proto
# Test graph with an iteration template directive
node: {
name: %name_1%
calculator: "IntervalFilterCalculator"
options: {
[mediapipe.IntervalFilterCalculatorOptions.ext] {
intervals {
% for (interval : intervals_1) %
interval {
start_us: %interval.begin%
end_us: %interval.end%
}
%end%
}
}
}
}
```
2. Specify values for the template parameters as name-value pairs in a [`TemplateDict protobuf`](http://github.com/mediapipe/framework/tool/calculator_graph_template.proto)
```proto
# Some test template arguments for iteration_test_template.textpbt
arg: {key: "name_1" value: {str: "hooloo"}}
arg: {key: "intervals_1" value: {
element: { dict: {
arg: {key: "begin" value: {num:33}}
arg: {key: "end" value: {num:44}}
}}
element: { dict: {
arg: {key: "begin" value: {num:55}}
arg: {key: "end" value: {num:66}}
}}
element: { dict: {
arg: {key: "begin" value: {num:77}}
arg: {key: "end" value: {num:88}}
}}
}}
```
3. Register the subgraph using the build rule: `mediapipe_template_subgraph`.
```proto
mediapipe_template_graph(
name = "iteration_test_subgraph",
register_as = "IterationTestTemplateSubgraph",
template = "iteration_test_template.textpbt",
deps = [
"//mediapipe/core:interval_filter_calculator",
],
)
```
Alternatively, the template and the parameter values can be combined using the build rule: `mediapipe_template_graph`.
```proto
mediapipe_template_graph(
name = "iteration_test_2_graph",
arg_file = "iteration_test_arg.pbtxt",
template = "iteration_test_template.textpbt",
)
```
4. The result is a complete CalculatorGraphConfig protobuf, such as the following:
```proto
node {
name: "hooloo"
calculator: "IntervalFilterCalculator"
options {
[mediapipe.IntervalFilterCalculatorOptions.ext] {
intervals {
interval {
start_us: 33
end_us: 44
}
interval {
start_us: 55
end_us: 66
}
interval {
start_us: 77
end_us: 88
}
}
}
}
}
```
Graph template parameters
* For a template parameter, you can specify a simple parameter name such as `%end_time%`
```proto
interval {
start_us: 10
end_us: %end_time%
}
```
or a more complex expression, such as `% begin_time + duration %`
```proto
interval {
start_us: %begin_time%
end_us: % begin_time + duration %
}
```
--->
mediapipe/docs/hair_segmentation_android_gpu.md → mediapipe/docs/hair_segmentation_mobile_gpu.md
+33 -21
View File
@@ -1,14 +1,15 @@
# Hair Segmentation on Android
# Hair Segmentation (GPU)
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe. This
doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_android_gpu.pbtxt)
This doc focuses on the
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_android_gpu.pbtxt)
that performs hair segmentation with TensorFlow Lite on GPU.
![hair_segmentation_android_gpu_gif](images/mobile/hair_segmentation_android_gpu.gif){width="300"}
## App
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Hair Segmentation GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu)
@@ -26,7 +27,7 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## Graph
![hair_segmentation_android_gpu_graph](images/mobile/hair_segmentation_android_gpu.png){width="600"}
![hair_segmentation_mobile_gpu_graph](images/mobile/hair_segmentation_mobile_gpu.png){width="600"}
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
@@ -34,7 +35,7 @@ below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
```bash
# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
# Used in the example in
# mediapipie/examples/android/src/java/com/mediapipe/apps/hairsegmentationgpu.
# mediapipie/examples/ios/hairsegmentationgpu.
# Images on GPU coming into and out of the graph.
input_stream: "input_video"
@@ -54,7 +55,7 @@ output_stream: "output_video"
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
# processing previous inputs.
node {
calculator: "RealTimeFlowLimiterCalculator"
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:hair_mask"
input_stream_info: {
@@ -111,12 +112,9 @@ node {
# Converts the transformed input image on GPU into an image tensor stored in
# tflite::gpu::GlBuffer. The zero_center option is set to false to normalize the
# pixel values to [0.f, 1.f] as opposed to [-1.f, 1.f]. The flip_vertically
# option is set to true to account for the descrepancy between the
# representation of the input image (origin at the bottom-left corner, the
# OpenGL convention) and what the model used in this graph is expecting (origin
# at the top-left corner). With the max_num_channels option set to 4, all 4 RGBA
# channels are contained in the image tensor.
# pixel values to [0.f, 1.f] as opposed to [-1.f, 1.f].
# With the max_num_channels option set to 4, all 4 RGBA channels are contained
# in the image tensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE_GPU:mask_embedded_input_video"
@@ -124,7 +122,6 @@ node {
node_options: {
[type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
zero_center: false
flip_vertically: true
max_num_channels: 4
}
}
@@ -148,7 +145,7 @@ node {
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS_GPU:image_tensor"
output_stream: "TENSORS_GPU:segmentation_tensor"
output_stream: "TENSORS:segmentation_tensor"
input_side_packet: "CUSTOM_OP_RESOLVER:op_resolver"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
@@ -158,15 +155,23 @@ node {
}
}
# The next step (tensors to segmentation) is not yet supported on iOS GPU.
# Convert the previous segmentation mask to CPU for processing.
node: {
calculator: "GpuBufferToImageFrameCalculator"
input_stream: "previous_hair_mask"
output_stream: "previous_hair_mask_cpu"
}
# Decodes the segmentation tensor generated by the TensorFlow Lite model into a
# mask of values in [0.f, 1.f], stored in the R channel of a GPU buffer. It also
# mask of values in [0.f, 1.f], stored in the R channel of a CPU buffer. It also
# takes the mask generated previously as another input to improve the temporal
# consistency.
node {
calculator: "TfLiteTensorsToSegmentationCalculator"
input_stream: "TENSORS_GPU:segmentation_tensor"
input_stream: "PREV_MASK_GPU:previous_hair_mask"
output_stream: "MASK_GPU:hair_mask"
input_stream: "TENSORS:segmentation_tensor"
input_stream: "PREV_MASK:previous_hair_mask_cpu"
output_stream: "MASK:hair_mask_cpu"
node_options: {
[type.googleapis.com/mediapipe.TfLiteTensorsToSegmentationCalculatorOptions] {
tensor_width: 512
@@ -178,6 +183,13 @@ node {
}
}
# Send the current segmentation mask to GPU for the last step, blending.
node: {
calculator: "ImageFrameToGpuBufferCalculator"
input_stream: "hair_mask_cpu"
output_stream: "hair_mask"
}
# Colors the hair segmentation with the color specified in the option.
node {
calculator: "RecolorCalculator"
mediapipe/docs/hand_detection_mobile_gpu.md
+326
View File
@@ -0,0 +1,326 @@
# Hand Detection (GPU)
This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
that performs hand detection with TensorFlow Lite on GPU. This hand detection
example is related to
[hand tracking GPU example](./hand_tracking_mobile_gpu.md). Here is the
[model card](https://mediapipe.page.link/handmc) for hand detection.
For overall context on hand detection and hand tracking, please read
[this Google AI blog post](https://mediapipe.page.link/handgoogleaiblog).
![hand_detection_android_gpu_gif](images/mobile/hand_detection_android_gpu.gif){width="300"}
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Hand Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu)
example app. To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu
```
To further install the app on android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu/handdetectiongpu.apk
```
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Hand Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu)
To build the iOS app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp
```
## Graph
The hand detection graph is
[hand_detection_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
and it includes a [HandDetectionSubgraph](./framework_concepts.md#subgraph) with
filename
[hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
shown as a box called `HandDetection` in purple
For more information on how to visualize a graph that includes subgraphs, see
[subgraph documentation](./visualizer.md#visualizing-subgraphs) for Visualizer.
![hand_detection_mobile_graph](images/mobile/hand_detection_mobile.png){width="500"}
```bash
# MediaPipe graph that performs hand detection with TensorFlow Lite on GPU.
# Used in the example in
# mediapipie/examples/android/src/java/com/mediapipe/apps/handdetectiongpu.
# mediapipie/examples/ios/handdetectiongpu.
# Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"
node {
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:hand_rect_from_palm_detections"
input_stream_info: {
tag_index: "FINISHED"
back_edge: true
}
output_stream: "throttled_input_video"
}
node {
calculator: "HandDetectionSubgraph"
input_stream: "throttled_input_video"
output_stream: "DETECTIONS:palm_detections"
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
}
# Converts detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTIONS:palm_detections"
output_stream: "RENDER_DATA:detection_render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
color { r: 0 g: 255 b: 0 }
}
}
}
# Converts normalized rects to drawing primitives for annotation overlay.
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:hand_rect_from_palm_detections"
output_stream: "RENDER_DATA:rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
# Draws annotations and overlays them on top of the input image into the graph.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "INPUT_FRAME_GPU:throttled_input_video"
input_stream: "detection_render_data"
input_stream: "rect_render_data"
output_stream: "OUTPUT_FRAME_GPU:output_video"
}
```
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png){width="500"}
```bash
type: "HandDetectionSubgraph"
input_stream: "input_video"
output_stream: "DETECTIONS:palm_detections"
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
# Transforms the input image on GPU to a 256x256 image. To scale the input
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
# resulting in potential letterboxing in the transformed image.
node: {
calculator: "ImageTransformationCalculator"
input_stream: "IMAGE_GPU:input_video"
output_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "LETTERBOX_PADDING:letterbox_padding"
node_options: {
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
output_width: 256
output_height: 256
scale_mode: FIT
}
}
}
# Generates a single side packet containing a TensorFlow Lite op resolver that
# supports custom ops needed by the model used in this graph.
node {
calculator: "TfLiteCustomOpResolverCalculator"
output_side_packet: "opresolver"
node_options: {
[type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
use_gpu: true
}
}
}
# Converts the transformed input image on GPU into an image tensor stored as a
# TfLiteTensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "TENSORS_GPU:image_tensor"
}
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
# vector of tensors representing, for instance, detection boxes/keypoints and
# scores.
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS_GPU:image_tensor"
output_stream: "TENSORS:detection_tensors"
input_side_packet: "CUSTOM_OP_RESOLVER:opresolver"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "palm_detection.tflite"
use_gpu: true
}
}
}
# Generates a single side packet containing a vector of SSD anchors based on
# the specification in the options.
node {
calculator: "SsdAnchorsCalculator"
output_side_packet: "anchors"
node_options: {
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
num_layers: 5
min_scale: 0.1171875
max_scale: 0.75
input_size_height: 256
input_size_width: 256
anchor_offset_x: 0.5
anchor_offset_y: 0.5
strides: 8
strides: 16
strides: 32
strides: 32
strides: 32
aspect_ratios: 1.0
fixed_anchor_size: true
}
}
}
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
# the SSD anchors and the specification in the options, into a vector of
# detections. Each detection describes a detected object.
node {
calculator: "TfLiteTensorsToDetectionsCalculator"
input_stream: "TENSORS:detection_tensors"
input_side_packet: "ANCHORS:anchors"
output_stream: "DETECTIONS:detections"
node_options: {
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
num_classes: 1
num_boxes: 2944
num_coords: 18
box_coord_offset: 0
keypoint_coord_offset: 4
num_keypoints: 7
num_values_per_keypoint: 2
sigmoid_score: true
score_clipping_thresh: 100.0
reverse_output_order: true
x_scale: 256.0
y_scale: 256.0
h_scale: 256.0
w_scale: 256.0
min_score_thresh: 0.7
}
}
}
# Performs non-max suppression to remove excessive detections.
node {
calculator: "NonMaxSuppressionCalculator"
input_stream: "detections"
output_stream: "filtered_detections"
node_options: {
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
min_suppression_threshold: 0.3
overlap_type: INTERSECTION_OVER_UNION
algorithm: WEIGHTED
return_empty_detections: true
}
}
}
# Maps detection label IDs to the corresponding label text. The label map is
# provided in the label_map_path option.
node {
calculator: "DetectionLabelIdToTextCalculator"
input_stream: "filtered_detections"
output_stream: "labeled_detections"
node_options: {
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
label_map_path: "palm_detection_labelmap.txt"
}
}
}
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
# letterboxed image (after image transformation with the FIT scale mode) to the
# corresponding locations on the same image with the letterbox removed (the
# input image to the graph before image transformation).
node {
calculator: "DetectionLetterboxRemovalCalculator"
input_stream: "DETECTIONS:labeled_detections"
input_stream: "LETTERBOX_PADDING:letterbox_padding"
output_stream: "DETECTIONS:palm_detections"
}
# Extracts image size from the input images.
node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE_GPU:input_video"
output_stream: "SIZE:image_size"
}
# Converts results of palm detection into a rectangle (normalized by image size)
# that encloses the palm and is rotated such that the line connecting center of
# the wrist and MCP of the middle finger is aligned with the Y-axis of the
# rectangle.
node {
calculator: "DetectionsToRectsCalculator"
input_stream: "DETECTIONS:palm_detections"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "NORM_RECT:palm_rect"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRectsCalculatorOptions] {
rotation_vector_start_keypoint_index: 0 # Center of wrist.
rotation_vector_end_keypoint_index: 2 # MCP of middle finger.
rotation_vector_target_angle_degrees: 90
output_zero_rect_for_empty_detections: true
}
}
}
# Expands and shifts the rectangle that contains the palm so that it's likely
# to cover the entire hand.
node {
calculator: "RectTransformationCalculator"
input_stream: "NORM_RECT:palm_rect"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "hand_rect_from_palm_detections"
node_options: {
[type.googleapis.com/mediapipe.RectTransformationCalculatorOptions] {
scale_x: 2.6
scale_y: 2.6
shift_y: -0.5
square_long: true
}
}
}
```
mediapipe/docs/hand_tracking_mobile_gpu.md
+640
View File
@@ -0,0 +1,640 @@
# Hand Tracking (GPU)
This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_android_gpu.pbtxt)
that performs hand tracking with TensorFlow Lite on GPU. This hand tracking
example is related to
[hand detection GPU example](./hand_detection_mobile_gpu.md). We recommend users
to review the hand detection GPU example first. Here is the
[model card](https://mediapipe.page.link/handmc) for hand tracking.
For overall context on hand detection and hand tracking, please read
[this Google AI blog post](https://mediapipe.page.link/handgoogleaiblog).
![hand_tracking_android_gpu.gif](images/mobile/hand_tracking_android_gpu.gif){width="300"}
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Hand Tracking GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu)
example app. To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu
```
To further install the app on android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
```
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Hand Tracking GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu)
To build the iOS app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
```
## Graph
For more information on how to visualize a graph that includes subgraphs, see
[subgraph documentation](./visualizer.md#visualizing-subgraphs) for Visualizer.
The hand tracking graph is
[hand_tracking_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
and it includes 3 [subgraphs](./framework_concepts.md#subgraph):
* [HandDetectionSubgraph - hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
* [HandLandmarkSubgraph - hand_landmark_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_landmark_gpu.pbtxt)
* [RendererSubgraph - renderer_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/renderer_gpu.pbtxt)
![hand_tracking_mobile_graph](images/mobile/hand_tracking_mobile.png){width="400"}
```bash
# MediaPipe graph that performs hand tracking with TensorFlow Lite on GPU.
# Used in the example in
# mediapipie/examples/android/src/java/com/mediapipe/apps/handtrackinggpu.
# Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"
node {
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:hand_rect"
input_stream_info: {
tag_index: "FINISHED"
back_edge: true
}
output_stream: "throttled_input_video"
}
node {
calculator: "PreviousLoopbackCalculator"
input_stream: "MAIN:throttled_input_video"
input_stream: "LOOP:hand_presence"
input_stream_info: {
tag_index: "LOOP"
back_edge: true
}
output_stream: "PREV_LOOP:prev_hand_presence"
}
node {
calculator: "GateCalculator"
input_stream: "throttled_input_video"
input_stream: "DISALLOW:prev_hand_presence"
output_stream: "hand_detection_input_video"
node_options: {
[type.googleapis.com/mediapipe.GateCalculatorOptions] {
empty_packets_as_allow: true
}
}
}
node {
calculator: "HandDetectionSubgraph"
input_stream: "hand_detection_input_video"
output_stream: "DETECTIONS:palm_detections"
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
}
node {
calculator: "HandLandmarkSubgraph"
input_stream: "IMAGE:throttled_input_video"
input_stream: "NORM_RECT:hand_rect"
output_stream: "LANDMARKS:hand_landmarks"
output_stream: "NORM_RECT:hand_rect_from_landmarks"
output_stream: "PRESENCE:hand_presence"
}
node {
calculator: "PreviousLoopbackCalculator"
input_stream: "MAIN:throttled_input_video"
input_stream: "LOOP:hand_rect_from_landmarks"
input_stream_info: {
tag_index: "LOOP"
back_edge: true
}
output_stream: "PREV_LOOP:prev_hand_rect_from_landmarks"
}
node {
calculator: "MergeCalculator"
input_stream: "hand_rect_from_palm_detections"
input_stream: "prev_hand_rect_from_landmarks"
output_stream: "hand_rect"
}
node {
calculator: "RendererSubgraph"
input_stream: "IMAGE:throttled_input_video"
input_stream: "LANDMARKS:hand_landmarks"
input_stream: "NORM_RECT:hand_rect"
input_stream: "DETECTIONS:palm_detections"
output_stream: "IMAGE:output_video"
}
```
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png){width="500"}
```bash
type: "HandDetectionSubgraph"
input_stream: "input_video"
output_stream: "DETECTIONS:palm_detections"
output_stream: "NORM_RECT:hand_rect_from_palm_detections"
# Transforms the input image on GPU to a 256x256 image. To scale the input
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
# resulting in potential letterboxing in the transformed image.
node: {
calculator: "ImageTransformationCalculator"
input_stream: "IMAGE_GPU:input_video"
output_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "LETTERBOX_PADDING:letterbox_padding"
node_options: {
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
output_width: 256
output_height: 256
scale_mode: FIT
}
}
}
# Generates a single side packet containing a TensorFlow Lite op resolver that
# supports custom ops needed by the model used in this graph.
node {
calculator: "TfLiteCustomOpResolverCalculator"
output_side_packet: "opresolver"
node_options: {
[type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
use_gpu: true
}
}
}
# Converts the transformed input image on GPU into an image tensor stored as a
# TfLiteTensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "TENSORS_GPU:image_tensor"
}
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
# vector of tensors representing, for instance, detection boxes/keypoints and
# scores.
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS_GPU:image_tensor"
output_stream: "TENSORS:detection_tensors"
input_side_packet: "CUSTOM_OP_RESOLVER:opresolver"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "palm_detection.tflite"
use_gpu: true
}
}
}
# Generates a single side packet containing a vector of SSD anchors based on
# the specification in the options.
node {
calculator: "SsdAnchorsCalculator"
output_side_packet: "anchors"
node_options: {
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
num_layers: 5
min_scale: 0.1171875
max_scale: 0.75
input_size_height: 256
input_size_width: 256
anchor_offset_x: 0.5
anchor_offset_y: 0.5
strides: 8
strides: 16
strides: 32
strides: 32
strides: 32
aspect_ratios: 1.0
fixed_anchor_size: true
}
}
}
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
# the SSD anchors and the specification in the options, into a vector of
# detections. Each detection describes a detected object.
node {
calculator: "TfLiteTensorsToDetectionsCalculator"
input_stream: "TENSORS:detection_tensors"
input_side_packet: "ANCHORS:anchors"
output_stream: "DETECTIONS:detections"
node_options: {
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
num_classes: 1
num_boxes: 2944
num_coords: 18
box_coord_offset: 0
keypoint_coord_offset: 4
num_keypoints: 7
num_values_per_keypoint: 2
sigmoid_score: true
score_clipping_thresh: 100.0
reverse_output_order: true
x_scale: 256.0
y_scale: 256.0
h_scale: 256.0
w_scale: 256.0
min_score_thresh: 0.7
}
}
}
# Performs non-max suppression to remove excessive detections.
node {
calculator: "NonMaxSuppressionCalculator"
input_stream: "detections"
output_stream: "filtered_detections"
node_options: {
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
min_suppression_threshold: 0.3
overlap_type: INTERSECTION_OVER_UNION
algorithm: WEIGHTED
return_empty_detections: true
}
}
}
# Maps detection label IDs to the corresponding label text. The label map is
# provided in the label_map_path option.
node {
calculator: "DetectionLabelIdToTextCalculator"
input_stream: "filtered_detections"
output_stream: "labeled_detections"
node_options: {
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
label_map_path: "palm_detection_labelmap.txt"
}
}
}
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
# letterboxed image (after image transformation with the FIT scale mode) to the
# corresponding locations on the same image with the letterbox removed (the
# input image to the graph before image transformation).
node {
calculator: "DetectionLetterboxRemovalCalculator"
input_stream: "DETECTIONS:labeled_detections"
input_stream: "LETTERBOX_PADDING:letterbox_padding"
output_stream: "DETECTIONS:palm_detections"
}
# Extracts image size from the input images.
node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE_GPU:input_video"
output_stream: "SIZE:image_size"
}
# Converts results of palm detection into a rectangle (normalized by image size)
# that encloses the palm and is rotated such that the line connecting center of
# the wrist and MCP of the middle finger is aligned with the Y-axis of the
# rectangle.
node {
calculator: "DetectionsToRectsCalculator"
input_stream: "DETECTIONS:palm_detections"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "NORM_RECT:palm_rect"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRectsCalculatorOptions] {
rotation_vector_start_keypoint_index: 0 # Center of wrist.
rotation_vector_end_keypoint_index: 2 # MCP of middle finger.
rotation_vector_target_angle_degrees: 90
output_zero_rect_for_empty_detections: true
}
}
}
# Expands and shifts the rectangle that contains the palm so that it's likely
# to cover the entire hand.
node {
calculator: "RectTransformationCalculator"
input_stream: "NORM_RECT:palm_rect"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "hand_rect_from_palm_detections"
node_options: {
[type.googleapis.com/mediapipe.RectTransformationCalculatorOptions] {
scale_x: 2.6
scale_y: 2.6
shift_y: -0.5
square_long: true
}
}
}
```
![hand_landmark_gpu_subgraph.pbtxt](images/mobile/hand_landmark_gpu_subgraph.png){width="400"}
```bash
# MediaPipe hand landmark localization subgraph.
type: "HandLandmarkSubgraph"
input_stream: "IMAGE:input_video"
input_stream: "NORM_RECT:hand_rect"
output_stream: "LANDMARKS:hand_landmarks"
output_stream: "NORM_RECT:hand_rect_for_next_frame"
output_stream: "PRESENCE:hand_presence"
# Crops the rectangle that contains a hand from the input image.
node {
calculator: "ImageCroppingCalculator"
input_stream: "IMAGE_GPU:input_video"
input_stream: "NORM_RECT:hand_rect"
output_stream: "IMAGE_GPU:hand_image"
}
# Transforms the input image on GPU to a 256x256 image. To scale the input
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
# resulting in potential letterboxing in the transformed image.
node: {
calculator: "ImageTransformationCalculator"
input_stream: "IMAGE_GPU:hand_image"
output_stream: "IMAGE_GPU:transformed_hand_image"
output_stream: "LETTERBOX_PADDING:letterbox_padding"
node_options: {
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
output_width: 256
output_height: 256
scale_mode: FIT
}
}
}
# Converts the transformed input image on GPU into an image tensor stored as a
# TfLiteTensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE_GPU:transformed_hand_image"
output_stream: "TENSORS_GPU:image_tensor"
}
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
# vector of tensors representing, for instance, detection boxes/keypoints and
# scores.
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS_GPU:image_tensor"
output_stream: "TENSORS:output_tensors"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "hand_landmark.tflite"
use_gpu: true
}
}
}
# Splits a vector of tensors into multiple vectors.
node {
calculator: "SplitTfLiteTensorVectorCalculator"
input_stream: "output_tensors"
output_stream: "landmark_tensors"
output_stream: "hand_flag_tensor"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
ranges: { begin: 1 end: 2 }
}
}
}
# Converts the hand-flag tensor into a float that represents the confidence
# score of hand presence.
node {
calculator: "TfLiteTensorsToFloatsCalculator"
input_stream: "TENSORS:hand_flag_tensor"
output_stream: "FLOAT:hand_presence_score"
}
# Applies a threshold to the confidence score to determine whether a hand is
# present.
node {
calculator: "ThresholdingCalculator"
input_stream: "FLOAT:hand_presence_score"
output_stream: "FLAG:hand_presence"
node_options: {
[type.googleapis.com/mediapipe.ThresholdingCalculatorOptions] {
threshold: 0.1
}
}
}
# Decodes the landmark tensors into a vector of lanmarks, where the landmark
# coordinates are normalized by the size of the input image to the model.
node {
calculator: "TfLiteTensorsToLandmarksCalculator"
input_stream: "TENSORS:landmark_tensors"
output_stream: "NORM_LANDMARKS:landmarks"
node_options: {
[type.googleapis.com/mediapipe.TfLiteTensorsToLandmarksCalculatorOptions] {
num_landmarks: 21
input_image_width: 256
input_image_height: 256
}
}
}
# Adjusts landmarks (already normalized to [0.f, 1.f]) on the letterboxed hand
# image (after image transformation with the FIT scale mode) to the
# corresponding locations on the same image with the letterbox removed (hand
# image before image transformation).
node {
calculator: "LandmarkLetterboxRemovalCalculator"
input_stream: "LANDMARKS:landmarks"
input_stream: "LETTERBOX_PADDING:letterbox_padding"
output_stream: "LANDMARKS:scaled_landmarks"
}
# Projects the landmarks from the cropped hand image to the corresponding
# locations on the full image before cropping (input to the graph).
node {
calculator: "LandmarkProjectionCalculator"
input_stream: "NORM_LANDMARKS:scaled_landmarks"
input_stream: "NORM_RECT:hand_rect"
output_stream: "NORM_LANDMARKS:hand_landmarks"
}
# Extracts image size from the input images.
node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE_GPU:input_video"
output_stream: "SIZE:image_size"
}
# Converts hand landmarks to a detection that tightly encloses all landmarks.
node {
calculator: "LandmarksToDetectionCalculator"
input_stream: "NORM_LANDMARKS:hand_landmarks"
output_stream: "DETECTION:hand_detection"
}
# Converts the hand detection into a rectangle (normalized by image size)
# that encloses the hand and is rotated such that the line connecting center of
# the wrist and MCP of the middle finger is aligned with the Y-axis of the
# rectangle.
node {
calculator: "DetectionsToRectsCalculator"
input_stream: "DETECTION:hand_detection"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "NORM_RECT:hand_rect_from_landmarks"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRectsCalculatorOptions] {
rotation_vector_start_keypoint_index: 0 # Center of wrist.
rotation_vector_end_keypoint_index: 9 # MCP of middle finger.
rotation_vector_target_angle_degrees: 90
}
}
}
# Expands the hand rectangle so that in the next video frame it's likely to
# still contain the hand even with some motion.
node {
calculator: "RectTransformationCalculator"
input_stream: "NORM_RECT:hand_rect_from_landmarks"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "hand_rect_for_next_frame"
node_options: {
[type.googleapis.com/mediapipe.RectTransformationCalculatorOptions] {
scale_x: 1.6
scale_y: 1.6
square_long: true
}
}
}
```
![hand_renderer_gpu_subgraph.pbtxt](images/mobile/hand_renderer_gpu_subgraph.png){width="500"}
```bash
# MediaPipe hand tracking rendering subgraph.
type: "RendererSubgraph"
input_stream: "IMAGE:input_image"
input_stream: "DETECTIONS:detections"
input_stream: "LANDMARKS:landmarks"
input_stream: "NORM_RECT:rect"
output_stream: "IMAGE:output_image"
# Converts detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTIONS:detections"
output_stream: "RENDER_DATA:detection_render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
color { r: 0 g: 255 b: 0 }
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:landmarks"
output_stream: "RENDER_DATA:landmark_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 0
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 5
landmark_connections: 9
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 9
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 14
landmark_connections: 15
landmark_connections: 15
landmark_connections: 16
landmark_connections: 13
landmark_connections: 17
landmark_connections: 0
landmark_connections: 17
landmark_connections: 17
landmark_connections: 18
landmark_connections: 18
landmark_connections: 19
landmark_connections: 19
landmark_connections: 20
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 0 g: 255 b: 0 }
thickness: 5.0
}
}
}
# Converts normalized rects to drawing primitives for annotation overlay.
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:rect"
output_stream: "RENDER_DATA:rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
# Draws annotations and overlays them on top of the input images.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "INPUT_FRAME_GPU:input_image"
input_stream: "detection_render_data"
input_stream: "landmark_render_data"
input_stream: "rect_render_data"
output_stream: "OUTPUT_FRAME_GPU:output_image"
}
```
mediapipe/docs/hello_world_android.md
+14 -6
View File
@@ -27,18 +27,26 @@ stream on an Android device.
## Graph for edge detection
We will be using the following graph, [`edge_detection_android_gpu.pbtxt`]:
We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
```
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
# Used in the examples
# mediapipe/examples/android/src/java/com/mediapipe/apps/edgedetectiongpu.
# mediapipe/examples/ios/edgedetectiongpu.
# Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"
# Converts RGB images into luminance images, still stored in RGB format.
node: {
calculator: "LuminanceCalculator"
input_stream: "input_video"
output_stream: "luma_video"
}
# Applies the Sobel filter to luminance images sotred in RGB format.
node: {
calculator: "SobelEdgesCalculator"
input_stream: "luma_video"
@@ -48,7 +56,7 @@ node: {
A visualization of the graph is shown below:
![edge_detection_android_gpu_graph](images/mobile/edge_detection_android_graph_gpu.png){width="200"}
![edge_detection_mobile_gpu_graph](images/mobile/edge_detection_mobile_graph_gpu.png){width="200"}
This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera.
@@ -62,7 +70,7 @@ packets in the `luma_video` stream and outputs results in `output_video` output
stream.
Our Android application will display the output image frames of the
`sobel_video` stream.
`output_video` stream.
## Initial minimal application setup
@@ -582,7 +590,7 @@ First, add dependencies to all calculator code in the `libmediapipe_jni.so`
build rule:
```
"//mediapipe/graphs/edge_detection:android_calculators",
"//mediapipe/graphs/edge_detection:mobile_calculators",
```
MediaPipe graphs are `.pbtxt` files, but to use them in the application, we need
@@ -594,7 +602,7 @@ graph:
```
genrule(
name = "binary_graph",
srcs = ["//mediapipe/graphs/edge_detection:android_gpu_binary_graph"],
srcs = ["//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph"],
outs = ["edgedetectiongpu.binarypb"],
cmd = "cp $< $@",
)
@@ -712,7 +720,7 @@ If you ran into any issues, please see the full code of the tutorial
[CameraX]:https://developer.android.com/training/camerax
[`CameraXPreviewHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraXPreviewHelper.java
[developer options]:https://developer.android.com/studio/debug/dev-options
[`edge_detection_android_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_gpu.pbtxt
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
[`EdgeDetectionGPU` example]:https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu/
[`EglManager`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/glutil/EglManager.java
[`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java
mediapipe/docs/hello_world_desktop.md
+1 -1
View File
@@ -11,7 +11,7 @@
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is not supported currently.
$ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# It should print 10 rows of Hello World!
mediapipe/docs/hello_world_ios.md
+545
View File
@@ -0,0 +1,545 @@
# Hello World! in MediaPipe on iOS
## Introduction
This codelab uses MediaPipe on an iOS device.
### What you will learn
How to develop an Android application that uses MediaPipe and run a MediaPipe
graph on iOS.
### What you will build
A simple camera app for real-time Sobel edge detection applied to a live video
stream on an iOS device.
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif){width="300"}
## Setup
1. Install MediaPipe on your system, see [MediaPipe installation guide] for
details.
2. Setup your iOS device for development.
3. Setup [Bazel] on your system to build and deploy the iOS app.
## Graph for edge detection
We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
```
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
# Used in the examples
# mediapipe/examples/android/src/java/com/mediapipe/apps/edgedetectiongpu.
# mediapipe/examples/ios/edgedetectiongpu.
# Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"
# Converts RGB images into luminance images, still stored in RGB format.
node: {
calculator: "LuminanceCalculator"
input_stream: "input_video"
output_stream: "luma_video"
}
# Applies the Sobel filter to luminance images sotred in RGB format.
node: {
calculator: "SobelEdgesCalculator"
input_stream: "luma_video"
output_stream: "output_video"
}
```
A visualization of the graph is shown below:
![edge_detection_mobile_gpu_graph](images/mobile/edge_detection_mobile_graph_gpu.png){width="200"}
This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera.
The first node in the graph, `LuminanceCalculator`, takes a single packet (image
frame) and applies a change in luminance using an OpenGL shader. The resulting
image frame is sent to the `luma_video` output stream.
The second node, `SobelEdgesCalculator` applies edge detection to incoming
packets in the `luma_video` stream and outputs results in `output_video` output
stream.
Our iOS application will display the output image frames of the `output_video`
stream.
## Initial minimal application setup
We first start with a simple iOS application and demonstrate how to use `bazel`
to build it.
First, create an XCode project via File > New > Single View App.
Set the product name to "EdgeDetectionGpu", and use an appropriate organization
identifier, such as `com.google.mediapipe`. The organization identifier
alongwith the product name will be the `bundle_id` for the application, such as
`com.google.mediapipe.EdgeDetectionGpu`.
Set the language to Objective-C.
Save the project to an appropriate location. Let's call this
`$PROJECT_TEMPLATE_LOC`. So your project will be in the
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu` directory. This directory will contain
another directory named `EdgeDetectionGpu` and an `EdgeDetectionGpu.xcodeproj` file.
The `EdgeDetectionGpu.xcodeproj` will not be useful for this tutorial, as we will
use bazel to build the iOS application. The content of the
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu/EdgeDetectionGpu` directory is listed below:
1. `AppDelegate.h` and `AppDelegate.m`
2. `ViewController.h` and `ViewController.m`
3. `main.m`
4. `Info.plist`
5. `Main.storyboard` and `Launch.storyboard`
6. `Assets.xcassets` directory.
Copy these files to a directory named `EdgeDetectionGpu` to a location that can
access the MediaPipe source code. For example, the source code of the
application that we will build in this tutorial is located in
`mediapipe/examples/ios/EdgeDetectionGpu`. We will refer to this path as the
`$APPLICATION_PATH` throughout the codelab.
Note: MediaPipe provides Objective-C bindings for iOS. The edge detection
application in this tutorial and all iOS examples using MediaPipe use
Objective-C with C++ in `.mm` files.
Create a `BUILD` file in the `$APPLICATION_PATH` and add the following build
rules:
```
MIN_IOS_VERSION = "10.0"
load(
"@build_bazel_rules_apple//apple:ios.bzl",
"ios_application",
)
ios_application(
name = "EdgeDetectionGpuApp",
bundle_id = "com.google.mediapipe.EdgeDetectionGpu",
families = [
"iphone",
"ipad",
],
infoplists = ["Info.plist"],
minimum_os_version = MIN_IOS_VERSION,
provisioning_profile = "//mediapipe/examples/ios:developer_provisioning_profile",
deps = [":EdgeDetectionGpuAppLibrary"],
)
objc_library(
name = "EdgeDetectionGpuAppLibrary",
srcs = [
"AppDelegate.m",
"ViewController.m",
"main.m",
],
hdrs = [
"AppDelegate.h",
"ViewController.h",
],
data = [
"Base.lproj/LaunchScreen.storyboard",
"Base.lproj/Main.storyboard",
],
sdk_frameworks = [
"UIKit",
],
deps = [],
)
```
The `objc_library` rule adds dependencies for the `AppDelegate` and
`ViewController` classes, `main.m` and the application storyboards. The
templated app depends only on the `UIKit` SDK.
The `ios_application` rule uses the `EdgeDetectionGpuAppLibrary` Objective-C
library generated to build an iOS application for installation on your iOS
device.
Note: You need to point to your own iOS developer provisioning profile to be
able to run the application on your iOS device.
To build the app, use the following command in a terminal:
```
bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
```
For example, to build the `EdgeDetectionGpuApp` application in
`mediapipe/examples/ios/edgedetection`, use the following command:
```
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetection:EdgeDetectionGpuApp
```
Then, go back to XCode, open Window > Devices and Simulators, select your
device, and add the `.ipa` file generated by the command above to your device.
Open the application on your device. Since it is empty, it should display a
blank white screen.
## Use the camera for the live view feed
In this tutorial, we will use the `MediaPipeCameraInputSource` class to access
and grab frames from the camera. This class uses the `AVCaptureSession` API to
get the frames from the camera.
But before using this class, change the `Info.plist` file to support camera
usage in the app.
In `ViewController.m`, add the following import line:
```
#import "mediapipe/objc/MediaPipeCameraInputSource.h"
```
Add the following to its implementation block to create an object
`_cameraSource`:
```
@implementation ViewController {
// Handles camera access via AVCaptureSession library.
MediaPipeCameraInputSource* _cameraSource;
}
```
Add the following code to `viewDidLoad()`:
```
-(void)viewDidLoad {
[super viewDidLoad];
_cameraSource = [[MediaPipeCameraInputSource alloc] init];
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
}
```
The code initializes `_cameraSource`, sets the capture session preset, and which
camera to use.
We need to get frames from the `_cameraSource` into our application
`ViewController` to display them. `MediaPipeCameraInputSource` is a subclass of
`MediaPipeInputSource`, which provides a protocol for its delegates, namely the
`MediaPipeInputSourceDelegate`. So our application `ViewController` can be a
delegate of `_cameraSource`.
To handle camera setup and process incoming frames, we should use a queue
different from the main queue. Add the following to the implementation block of
the `ViewController`:
```
// Process camera frames on this queue.
dispatch_queue_t _videoQueue;
```
In `viewDidLoad()`, add the following line after initializing the
`_cameraSource` object:
```
[_cameraSource setDelegate:self queue:_videoQueue];
```
And add the following code to initialize the queue before setting up the
`_cameraSource` object:
```
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
_videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
```
We will use a serial queue with the priority `QOS_CLASS_USER_INTERACTIVE` for
processing camera frames.
Add the following line after the header imports at the top of the file, before
the interface/implementation of the `ViewController`:
```
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
```
Before implementing any method from `MediaPipeInputSourceDelegate` protocol, we
must first set up a way to display the camera frames. MediaPipe provides another
utility called `MediaPipeLayerRenderer` to display images on the screen. This
utility can be used to display `CVPixelBufferRef` objects, which is the type of
the images provided by `MediaPipeCameraInputSource` to its delegates.
To display images of the screen, we need to add a new `UIView` object called
`_liveView` to the `ViewController`.
Add the following lines to the implementation block of the `ViewController`:
```
// Display the camera preview frames.
IBOutlet UIView* _liveView;
// Render frames in a layer.
MediaPipeLayerRenderer* _renderer;
```
Go to `Main.storyboard`, add a `UIView` object from the object library to the
`View` of the `ViewController` class. Add a referencing outlet from this view to
the `_liveView` object you just added to the `ViewController` class. Resize the
view so that it is centered and covers the entire application screen.
Go back to `ViewController.m` and add the following code to `viewDidLoad()` to
initialize the `_renderer` object:
```
_renderer = [[MediaPipeLayerRenderer alloc] init];
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MediaPipeFrameScaleFillAndCrop;
```
To get frames from the camera, we will implement the following method:
```
// Must be invoked on _videoQueue.
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MediaPipeInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
}
// Display the captured image on the screen.
CFRetain(imageBuffer);
dispatch_async(dispatch_get_main_queue(), ^{
[_renderer renderPixelBuffer:imageBuffer];
CFRelease(imageBuffer);
});
}
```
This is a delegate method of `MediaPipeInputSource`. We first check that we are
getting frames from the right source, i.e. the `_cameraSource`. Then we display
the frame received from the camera via `_renderer` on the main queue.
Now, we need to start the camera as soon as the view to display the frames is
about to appear. To do this, we will implement the
`viewWillAppear:(BOOL)animated` function:
```
-(void)viewWillAppear:(BOOL)animated {
[super viewWillAppear:animated];
}
```
Before we start running the camera, we need the user's permission to access it.
`MediaPipeCameraInputSource` provides a function
`requestCameraAccessWithCompletionHandler:(void (^_Nullable)(BOOL
granted))handler` to request camera access and do some work when the user has
responded. Add the following code to `viewWillAppear:animated`:
```
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
if (granted) {
dispatch_queue(_videoQueue, ^{
[_cameraSource start];
});
}
}];
```
Before building the application, add the following dependencies to your `BUILD`
file:
```
sdk_frameworks = [
"AVFoundation",
"CoreGraphics",
"CoreMedia",
],
deps = [
"//mediapipe/objc:mediapipe_framework_ios",
"//mediapipe/objc:mediapipe_input_sources_ios",
"//mediapipe/objc:mediapipe_layer_renderer",
],
```
Now build and run the application on your iOS device. You should see a live
camera view feed after accepting camera permissions.
We are now ready to use camera frames in a MediaPipe graph.
## Using a MediaPipe graph in iOS
### Add relevant dependencies
We already added the dependencies of the MediaPipe framework code which contains
the iOS API to use a MediaPipe graph. To use a MediaPipe graph, we need to add a
dependency on the graph we intend to use in our application. Add the following
line to the `data` list in your `BUILD` file:
```
"//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph",
```
Now add the dependency to the calculators used in this graph in the `deps` field
in the `BUILD` file:
```
"//mediapipe/graphs/edge_detection:mobile_calculators",
```
Finally, rename the file `ViewController.m` to `ViewController.mm` to support
Objective-C++.
### Use the graph in `ViewController`
Declare a static constant with the name of the graph, the input stream and the
output stream:
```
static NSString* const kGraphName = @"android_gpu";
static const char* kInputStream = "input_video";
static const char* kOutputStream = "output_video";
```
Add the following property to the interface of the `ViewController`:
```
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
// sent video frames on _videoQueue.
@property(nonatomic) MediaPipeGraph* mediapipeGraph;
```
As explained in the comment above, we will initialize this graph in
`viewDidLoad` first. To do so, we need to load the graph from the `.pbtxt` file
using the following function:
```
+ (MediaPipeGraph*)loadGraphFromResource:(NSString*)resource {
// Load the graph config resource.
NSError* configLoadError = nil;
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
if (!resource || resource.length == 0) {
return nil;
}
NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
if (!data) {
NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
return nil;
}
// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
mediapipe::CalculatorGraphConfig config;
config.ParseFromArray(data.bytes, data.length);
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
MediaPipeGraph* newGraph = [[MediaPipeGraph alloc] initWithGraphConfig:config];
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MediaPipePacketPixelBuffer];
return newGraph;
}
```
Use this function to initialize the graph in `viewDidLoad` as follows:
```
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
```
The graph should send the results of processing camera frames back to the
`ViewController`. Add the following line after initializing the graph to set the
`ViewController` as a delegate of the `mediapipeGraph` object:
```
self.mediapipeGraph.delegate = self;
```
To avoid memory contention while processing frames from the live video feed, add
the following line:
```
// Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
self.mediapipeGraph.maxFramesInFlight = 2;
```
Now, start the graph when the user has granted the permission to use the camera
in our app:
```
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
if (granted) {
// Start running self.mediapipeGraph.
NSError* error;
if (![self.mediapipeGraph startWithError:&error]) {
NSLog(@"Failed to start graph: %@", error);
}
dispatch_queue(_videoQueue, ^{
[_cameraSource start];
});
}
}];
```
Note: It is important to start the graph before starting the camera, so that
the graph is ready to process frames as soon as the camera starts sending them.
Earlier, when we received frames from the camera in the `processVideoFrame`
function, we displayed them in the `_liveView` using the `_renderer`. Now, we
need to send those frames to the graph and render the results instead. Modify
this function's implementation to do the following:
```
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MediaPipeInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
}
[self.mediapipeGraph sendPixelBuffer:imageBuffer
intoStream:kInputStream
packetType:MediaPipePacketPixelBuffer];
}
```
We send the `imageBuffer` to `self.mediapipeGraph` as a packet of type
`MediaPipePacketPixelBuffer` into the input stream `kInputStream`, i.e.
"input_video".
The graph will run with this input packet and output a result in
`kOutputStream`, i.e. "output_video". We can implement the following delegate
method to receive packets on this output stream and display them on the screen:
```
- (void)mediapipeGraph:(MediaPipeGraph*)graph
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
fromStream:(const std::string&)streamName {
if (streamName == kOutputStream) {
// Display the captured image on the screen.
CVPixelBufferRetain(pixelBuffer);
dispatch_async(dispatch_get_main_queue(), ^{
[_renderer renderPixelBuffer:pixelBuffer];
CVPixelBufferRelease(pixelBuffer);
});
}
}
```
And that is all! Build and run the app on your iOS device. You should see the
results of running the edge detection graph on a live video feed. Congrats!
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif){width="300"}
If you ran into any issues, please see the full code of the tutorial
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
[Bazel]:https://bazel.build/
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
[MediaPipe installation guide]:./install.md
mediapipe/docs/how_to_questions.md
+6 -7
View File
@@ -19,7 +19,7 @@ section
[GpuBuffer to ImageFrame converters](./gpu.md).
You can see an example in:
* [`object_detection_android_cpu.pbtxt`]
* [`object_detection_mobile_cpu.pbtxt`]
### How to visualize perception results
@@ -29,7 +29,7 @@ the recognized objects. The results can be displayed in a diagnostic window when
running on a workstation, or in a texture frame when running on device. You can
see an example use of [`AnnotationOverlayCalculator`] in:
* [`face_detection_android_gpu.pbtxt`].
* [`face_detection_mobile_gpu.pbtxt`].
### How to run calculators in parallel
@@ -106,7 +106,7 @@ continues as long as necessary. For online processing, it is often necessary to
drop input packets in order to keep pace with the arrival of input data frames.
When inputs arrive too frequently, the recommended technique for dropping
packets is to use the MediaPipe calculators designed specifically for this
purpose such as [`RealTimeFlowLimiterCalculator`] and [`PacketClonerCalculator`].
purpose such as [`FlowLimiterCalculator`] and [`PacketClonerCalculator`].
For online processing, it is also necessary to promptly determine when processing
can proceed. MediaPipe supports this by propagating timestamp bounds between
@@ -124,20 +124,19 @@ MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library
conforming to the C++11 standard, so it is relatively easy to port to
additional platforms.
[`object_detection_android_cpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_cpu.pbtxt
[`object_detection_mobile_cpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt
[`ImageFrame`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/formats/image_frame.h
[`GpuBuffer`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer.h
[`GpuBufferToImageFrameCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer_to_image_frame_calculator.cc
[`ImageFrameToGpuBufferCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/image_frame_to_gpu_buffer_calculator.cc
[`AnnotationOverlayCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/util/annotation_overlay_calculator.cc
[`face_detection_android_gpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_android_gpu.pbtxt
[`face_detection_mobile_gpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt
[`CalculatorBase::Process`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
[`max_in_flight`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`RoundRobinDemuxCalculator`]: https://github.com/google/mediapipe/tree/master//mediapipe/calculators/core/round_robin_demux_calculator.cc
[`ScaleImageCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/image/scale_image_calculator.cc
[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.cc
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`RealTimeFlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/real_time_flow_limiter_calculator.cc
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
[`PacketClonerCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/packet_cloner_calculator.cc
[`MakePairCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/make_pair_calculator.cc
mediapipe/docs/images/add_ipa.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

mediapipe/docs/images/app_ipa.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 885 KiB

mediapipe/docs/images/app_ipa_added.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

mediapipe/docs/images/bazel_permission.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

mediapipe/docs/images/click_subgraph_handdetection.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 361 KiB

mediapipe/docs/images/device.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 195 KiB

mediapipe/docs/images/maingraph_visualizer.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 302 KiB

mediapipe/docs/images/mobile/edge_detection_ios_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 964 KiB

mediapipe/docs/images/mobile/edge_detection_android_graph_gpu.png → mediapipe/docs/images/mobile/edge_detection_mobile_graph_gpu.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 37 KiB

mediapipe/docs/images/mobile/face_detection_android_gpu.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 148 KiB

mediapipe/docs/images/mobile/face_detection_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

mediapipe/docs/images/mobile/hair_segmentation_android_gpu.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 171 KiB

mediapipe/docs/images/mobile/hair_segmentation_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

mediapipe/docs/images/mobile/hand_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.8 MiB

mediapipe/docs/images/mobile/hand_detection_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 105 KiB

mediapipe/docs/images/mobile/hand_detection_mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

mediapipe/docs/images/mobile/hand_landmark_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 107 KiB

mediapipe/docs/images/mobile/hand_renderer_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

mediapipe/docs/images/mobile/hand_tracking_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 MiB

mediapipe/docs/images/mobile/hand_tracking_mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 65 KiB

mediapipe/docs/images/mobile/object_detection_android_cpu.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 125 KiB

mediapipe/docs/images/mobile/object_detection_android_gpu.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 133 KiB

mediapipe/docs/images/mobile/object_detection_desktop_tensorflow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 94 KiB

mediapipe/docs/images/mobile/object_detection_desktop_tflite.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

mediapipe/docs/images/mobile/object_detection_mobile_cpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 68 KiB

mediapipe/docs/images/mobile/object_detection_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

mediapipe/docs/images/mobile/renderer_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 126 KiB

mediapipe/docs/images/upload_2pbtxt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 108 KiB

mediapipe/docs/images/upload_graph_button.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

mediapipe/docs/install.md
+54 -32
View File
@@ -2,15 +2,25 @@
Choose your operating system:
- [Prework](#prework)
- [Dependences](#dependences)
- [Installing on Debian and Ubuntu](#installing-on-debian-and-ubuntu)
- [Installing on CentOS](#installing-on-centos)
- [Installing on macOS](#installing-on-macos)
- [Installing on Windows Subsystem for Linux (WSL)](#installing-on-windows-subsystem-for-linux-wsl)
- [Installing using Docker](#installing-using-docker)
- [Setting up Android Studio with MediaPipe](#setting-up-android-studio-with-mediapipe)
- [Setting up Android SDK and NDK](#setting-up-android-sdk-and-ndk)
### Dependences
### Prework
* Install a package manager, e.g., Homebrew for macOS, and APT for Debian and Ubuntu
* Install Xcode for the iOS apps (macOS only)
* Install Android Studio for the Android apps
### Dependencies
Required libraries
@@ -56,8 +66,8 @@ Required libraries
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries.
Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to
take option 2 or 3 to install OpenCV 3 or above.
Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to take
option 2 or 3 to install OpenCV 3 or above.
```bash
$ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \
@@ -71,11 +81,11 @@ Required libraries
[documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
to manually build OpenCV from source code.
Note: You may need to modify [`WORKSAPCE`] and [`opencv_linux.BUILD`] to point
MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is
installed in "/usr/local/", you need to update the "linux_opencv"
new_local_repository rule in [`WORKSAPCE`] and "opencv" cc_library rule in
[`opencv_linux.BUILD`] like the following:
Note: You may need to modify [`WORKSAPCE`] and [`opencv_linux.BUILD`] to
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
rule in [`WORKSAPCE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
like the following:
```bash
new_local_repository(
@@ -110,7 +120,7 @@ Required libraries
```bash
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
$ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
@@ -156,11 +166,11 @@ Required libraries
Option 2. Build OpenCV from source code.
Note: You may need to modify [`WORKSAPCE`] and [`opencv_linux.BUILD`] to point
MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is
installed in "/usr/local/", you need to update the "linux_opencv"
new_local_repository rule in [`WORKSAPCE`] and "opencv" cc_library rule in
[`opencv_linux.BUILD`] like the following:
Note: You may need to modify [`WORKSAPCE`] and [`opencv_linux.BUILD`] to
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
rule in [`WORKSAPCE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
like the following:
```bash
new_local_repository(
@@ -195,7 +205,7 @@ Required libraries
```bash
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
$ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
@@ -284,7 +294,7 @@ Required libraries
```bash
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
$ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
@@ -403,7 +413,7 @@ Required libraries
username@DESKTOP-TMVLBJ1:~/mediapipe$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
username@DESKTOP-TMVLBJ1:~/mediapipe$ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
username@DESKTOP-TMVLBJ1:~/mediapipe$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
@@ -454,7 +464,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
```bash
$ docker run -it --name mediapipe mediapipe:latest
root@bca08b91ff63:/mediapipe# GLOG_logtostderr=1 bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' mediapipe/examples/desktop/hello_world:hello_world
root@bca08b91ff63:/mediapipe# GLOG_logtostderr=1 bazel run --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world:hello_world
# Should print:
# Hello World!
@@ -472,7 +482,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
<!-- 4. Uncomment the last line of the Dockerfile
```bash
RUN bazel build -c opt --define 'MEDIAPIPE_DISABLE_GPU=1' mediapipe/examples/desktop/demo:object_detection_tensorflow_demo
RUN bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/demo:object_detection_tensorflow_demo
```
and rebuild the image and then run the docker image
@@ -482,7 +492,6 @@ This will use a Docker image that will isolate mediapipe's installation from the
docker run -i -t mediapipe:latest
``` -->
### Setting up Android Studio with MediaPipe
The steps below use Android Studio to build and install a MediaPipe demo app.
@@ -491,13 +500,16 @@ The steps below use Android Studio to build and install a MediaPipe demo app.
2. Select `Configure` | `SDK Manager` | `SDK Platforms`
* verify that an Android SDK is installed
* note the Android SDK Location such as `/usr/local/home/Android/Sdk`
* Verify that Android SDK Platform API Level 28 or 29 is installed
* Note the Android SDK Location such as `/usr/local/home/Android/Sdk`
3. Select `Configure` | `SDK Manager` | `SDK Tools`
* verify that an Android NDK is installed
* note the Android NDK Location such as `/usr/local/home/Android/Sdk/ndk-bundle`
* Verify that Android SDK Build-Tools 28 or 29 is installed
* Verify that Android SDK Platform-Tools 28 or 29 is installed
* Verify that Android SDK Tools 26.1.1 is installed
* Verify that Android NDK 17c or above is installed
* Note the Android NDK Location such as `/usr/local/home/Android/Sdk/ndk-bundle`
4. Set environment variables `$ANDROID_HOME` and `$ANDROID_NDK_HOME` to point to
the installed SDK and NDK.
@@ -511,18 +523,18 @@ The steps below use Android Studio to build and install a MediaPipe demo app.
6. Select `Import Bazel Project`
* select `Workspace`: `/path/to/mediapipe`
* select `Generate from BUILD file`: `/path/to/mediapipe/BUILD`
* select `Finish`
* Select `Workspace`: `/path/to/mediapipe`
* Select `Generate from BUILD file`: `/path/to/mediapipe/BUILD`
* Select `Finish`
7. Connect an android device to the workstation.
8. Select `Run...` | `Edit Configurations...`
* enter Target Expression:
* Enter Target Expression:
`//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu`
* enter Bazel command: `mobile-install`
* enter Bazel flags: `-c opt --config=android_arm64` select `Run`
* Enter Bazel command: `mobile-install`
* Enter Bazel flags: `-c opt --config=android_arm64` select `Run`
### Setting up Android SDK and NDK
@@ -534,8 +546,18 @@ export ANDROID_HOME=<path to the Android SDK>
export ANDROID_NDK_HOME=<path to the Android NDK>
```
Otherwise, please run [`setup_android_sdk_and_ndk.sh`] to download and setup
Android SDK and NDK for MediaPipe before building any Android demos.
Please verify all the necessary packages are installed
* Android SDK Platform API Level 28 or 29
* Android SDK Build-Tools 28 or 29
* Android SDK Platform-Tools 28 or 29
* Android SDK Tools 26.1.1
* Android NDK 17c or above
MediaPipe prefers to use the Android SDK and NDK from Android Studio. See
[the previous section](#setting-up-android-studio-with-mediapipe) for the
Android Studio setup. If you prefer to try MediaPipe without Android Studio, please run [`setup_android_sdk_and_ndk.sh`] to download and setup Android SDK and NDK for
MediaPipe before building any Android demos.
[`WORKSAPCE`]: https://github.com/google/mediapipe/tree/master/WORKSPACE
[`opencv_linux.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_linux.BUILD
mediapipe/docs/media_sequence.md
+22 -16
View File
@@ -13,7 +13,8 @@ tasks like video object detection, but very difficult to encode in
TensorFlow.Examples. The goal of MediaSequence is to simplify working with
SequenceExamples and to automate common preparation tasks. Much more information
is available about the MediaSequence pipeline, including how to use it to
process new data sets, in the [documentation](https://github.com/google/mediapipe/tree/master/mediapipe/util/sequence/README.md).
process new data sets, in the documentation of
[MediaSequence](https://github.com/google/mediapipe/tree/master/mediapipe/util/sequence).
### Preparing an example data set
@@ -27,20 +28,20 @@ process new data sets, in the [documentation](https://github.com/google/mediapip
1. Compile the MediaSequence demo C++ binary
```bash
bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo --define 'MEDIAPIPE_DISABLE_GPU=1'
bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo --define MEDIAPIPE_DISABLE_GPU=1
```
MediaSequence uses C++ binaries to improve multimedia processing speed and
encourage a strong separation between annotations and the image data or
other features. The binary code is very general in that it reads from files
into input side packets and writes output side packets to files when
completed, but it also links in all of the calculators for necessary for
the MediaPipe graphs preparing the Charades data set.
completed, but it also links in all of the calculators for necessary for the
MediaPipe graphs preparing the Charades data set.
1. Download and prepare the data set through Python
To run this step, you must have Python 2.7 or 3.5+ installed with the
TensorFlow 1.19+ package installed.
TensorFlow 1.14+ package installed.
```bash
python -m mediapipe.examples.desktop.media_sequence.demo_dataset \
@@ -56,10 +57,11 @@ process new data sets, in the [documentation](https://github.com/google/mediapip
MediaPipe graphs during processing.
Running this module
1. Downloads videos from the internet.
1. For each annotation in a CSV, creates a structured metadata file.
1. Runs MediaPipe to extract images as defined by the metadata.
1. Stores the results in numbered set of TFRecords files.
1. Downloads videos from the internet.
1. For each annotation in a CSV, creates a structured metadata file.
1. Runs MediaPipe to extract images as defined by the metadata.
1. Stores the results in numbered set of TFRecords files.
MediaSequence uses SequenceExamples as the format of both inputs and
outputs. Annotations are encoded as inputs in a SequenceExample of metadata
@@ -84,12 +86,16 @@ process new data sets, in the [documentation](https://github.com/google/mediapip
demo_data_path = '/tmp/demo_data/'
with tf.Graph().as_default():
d = DemoDataset(demo_data_path)
dataset = d.as_dataset("test")
dataset = d.as_dataset('test')
# implement additional processing and batching here
output = dataset.make_one_shot_iterator().get_next()
dataset_output = dataset.make_one_shot_iterator().get_next()
images = dataset_output=['images']
labels = dataset_output=['labels']
with tf.Session() as sess:
output_ = sess.run(output)
images_, labels_ = sess.run(images, labels)
print('The shape of images_ is %s' % str(images_.shape))
print('The shape of labels_ is %s' % str(labels_.shape))
```
### Preparing a practical data set
@@ -104,9 +110,9 @@ The Charades data set is large (~150 GB), and will take considerable time to
download and process (4-8 hours).
```bash
bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo --define 'MEDIAPIPE_DISABLE_GPU=1'
bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo --define MEDIAPIPE_DISABLE_GPU=1
python -m mediapipe.examples.desktop.media_sequence.demo_dataset \
python -m mediapipe.examples.desktop.media_sequence.charades_dataset \
--alsologtostderr \
--path_to_charades_data=/tmp/demo_data/ \
--path_to_mediapipe_binary=bazel-bin/mediapipe/examples/desktop/media_sequence/media_sequence_demo \
@@ -115,7 +121,7 @@ python -m mediapipe.examples.desktop.media_sequence.demo_dataset \
### Preparing your own data set
The process for preparing your own data set is described in the [MediaSequence
documentation](https://github.com/google/mediapipe/blob/master/mediapipe/util/sequence/README.md).
documentation](https://github.com/google/mediapipe/tree/master/mediapipe/util/sequence).
The Python code for Charades can easily be modified to process most annotations,
but the MediaPipe processing warrants further discussion. MediaSequence uses
MediaPipe graphs to extract features related to the metadata or previously
@@ -145,7 +151,7 @@ node {
output_side_packet: "DATA_PATH:input_video_path"
output_side_packet: "RESAMPLER_OPTIONS:packet_resampler_options"
options {
[mediapipe.UnpackMediaSequenceCalculatorOptions.ext]: {
[type.googleapis.com/mediapipe.UnpackMediaSequenceCalculatorOptions]: {
base_packet_resampler_options {
frame_rate: 24.0
base_timestamp: 0
mediapipe/docs/mediapipe_ios_setup.md
+48
View File
@@ -0,0 +1,48 @@
## Setting up MediaPipe for iOS
1. Install [Xcode](https://developer.apple.com/xcode/).
Follow Apple's instructions to obtain the required developemnt certificates
and provisioning profiles for your iOS device.
2. Install [Bazel](https://bazel.build/).
See their [instructions](https://docs.bazel.build/versions/master/install-os-x.html).
We recommend using [Homebrew](https://brew.sh/):
```bash
brew tap bazelbuild/tap
brew install bazelbuild/tap/bazel
```
3. Clone the MediaPipe repository.
```bash
git clone https://github.com/google/mediapipe.git
```
4. Symlink or copy your provisioning profile to `mediapipe/mediapipe/provisioning_profile.mobileprovision`.
```bash
cd mediapipe
ln -s ~/Downloads/MyProvisioningProfile.mobileprovision mediapipe/provisioning_profile.mobileprovision
```
## Building an iOS app from the command line
1. Build one of the example apps for iOS. We will be using the
[Face Detection GPU App example](./face_detection_mobile_gpu.md)
```bash
bazel build --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp
```
You may see a permission request from `codesign` in order to sign the app.
2. In Xcode, open the `Devices and Simulators` window (command-shift-2).
3. Make sure your device is connected. You will see a list of installed apps.
Press the "+" button under the list, and select the `.ipa` file built by
Bazel.
4. You can now run the app on your device.
mediapipe/docs/object_detection_desktop.md
+3 -3
View File
@@ -23,8 +23,8 @@ To build and run the TensorFlow example on desktop, run:
# Note that this command also builds TensorFlow targets from scratch, it may
# take a long time (e.g., up to 30 mins) to build for the first time.
$ bazel build -c opt \
--define 'MEDIAPIPE_DISABLE_GPU=1' \
--define 'no_aws_support=true' \
--define MEDIAPIPE_DISABLE_GPU=1 \
--define no_aws_support=true \
mediapipe/examples/desktop/object_detection:object_detection_tensorflow
# It should print:
@@ -189,7 +189,7 @@ node {
To build and run the TensorFlow Lite example on desktop, run:
```bash
$ bazel build -c opt --define 'MEDIAPIPE_DISABLE_GPU=1' \
$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/object_detection:object_detection_tflite
# It should print:
mediapipe/docs/object_detection_android_cpu.md → mediapipe/docs/object_detection_mobile_cpu.md
+28 -29
View File
@@ -1,9 +1,9 @@
# Object Detection on CPU on Android
# Object Detection (CPU)
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe. This
doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_cpu.pbtxt)
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt)
that performs object detection with TensorFlow Lite on CPU.
This is very similar to the
@@ -16,7 +16,7 @@ CPU.
![object_detection_android_cpu_gif](images/mobile/object_detection_android_cpu.gif){width="300"}
## App
## Android
The graph is used in the
[Object Detection CPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu)
@@ -32,9 +32,24 @@ To further install the app on android device, run:
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu/objectdetectioncpu.apk
```
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectioncpu).
To build the iOS app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp
```
## Graph
![object_detection_android_cpu_graph](images/mobile/object_detection_android_cpu.png){width="400"}
![object_detection_mobile_cpu_graph](images/mobile/object_detection_mobile_cpu.png){width="400"}
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
@@ -42,7 +57,8 @@ below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
```bash
# MediaPipe graph that performs object detection with TensorFlow Lite on CPU.
# Used in the example in
# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectioncpu.
# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectioncpu and
# mediapipie/examples/ios/objectdetectioncpu.
# Images on GPU coming into and out of the graph.
input_stream: "input_video"
@@ -72,7 +88,7 @@ node: {
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
# processing previous inputs.
node {
calculator: "RealTimeFlowLimiterCalculator"
calculator: "FlowLimiterCalculator"
input_stream: "input_video_cpu"
input_stream: "FINISHED:detections"
input_stream_info: {
@@ -99,22 +115,12 @@ node: {
}
}
# Converts the transformed input image on CPU into an image tensor as a
# TfLiteTensor. The zero_center option is set to true to normalize the
# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f]. The flip_vertically
# option is set to true to account for the descrepancy between the
# representation of the input image (origin at the bottom-left corner) and what
# the model used in this graph is expecting (origin at the top-left corner).
# Converts the transformed input image on CPU into an image tensor stored as a
# TfLiteTensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE:transformed_input_video_cpu"
output_stream: "TENSORS:image_tensor"
node_options: {
[type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
zero_center: true
flip_vertically: true
}
}
}
# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
@@ -181,7 +187,7 @@ node {
y_scale: 10.0
h_scale: 5.0
w_scale: 5.0
flip_vertically: true
min_score_thresh: 0.6
}
}
}
@@ -194,9 +200,9 @@ node {
node_options: {
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
min_suppression_threshold: 0.4
min_score_threshold: 0.6
max_num_detections: 3
overlap_type: INTERSECTION_OVER_UNION
return_empty_detections: true
}
}
}
@@ -217,7 +223,7 @@ node {
# Converts the detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTION_VECTOR:output_detections"
input_stream: "DETECTIONS:output_detections"
output_stream: "RENDER_DATA:render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
@@ -229,19 +235,12 @@ node {
# Draws annotations and overlays them on top of the CPU copy of the original
# image coming into the graph. The calculator assumes that image origin is
# always at the top-left corner and renders text accordingly. However, the input
# image has its origin at the bottom-left corner (OpenGL convention) and the
# flip_text_vertically option is set to true to compensate that.
# always at the top-left corner and renders text accordingly.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "INPUT_FRAME:throttled_input_video_cpu"
input_stream: "render_data"
output_stream: "OUTPUT_FRAME:output_video_cpu"
node_options: {
[type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
flip_text_vertically: true
}
}
}
# Transfers the annotated image from CPU back to GPU memory, to be sent out of
mediapipe/docs/object_detection_android_gpu.md → mediapipe/docs/object_detection_mobile_gpu.md
+37 -40
View File
@@ -1,16 +1,17 @@
# Object Detection on GPU on Android
# Object Detection (GPU)
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe. This
doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_gpu.pbtxt)
This doc focuses on the
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt)
that performs object detection with TensorFlow Lite on GPU.
![object_detection_android_gpu_gif](images/mobile/object_detection_android_gpu.gif){width="300"}
## App
## Android
The graph is used in the
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph below is used in the
[Object Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu)
example app. To build the app, run:
@@ -24,9 +25,24 @@ To further install the app on android device, run:
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
```
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectiongpu)
To build the iOS app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp
```
## Graph
![object_detection_android_gpu_graph](images/mobile/object_detection_android_gpu.png){width="400"}
![object_detection_mobile_gpu_graph](images/mobile/object_detection_mobile_gpu.png){width="400"}
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
@@ -34,7 +50,8 @@ below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
```bash
# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
# Used in the example in
# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectiongpu.
# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectiongpu and
# mediapipie/examples/ios/objectdetectiongpu.
# Images on GPU coming into and out of the graph.
input_stream: "input_video"
@@ -54,7 +71,7 @@ output_stream: "output_video"
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
# processing previous inputs.
node {
calculator: "RealTimeFlowLimiterCalculator"
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:detections"
input_stream_info: {
@@ -81,23 +98,12 @@ node: {
}
}
# Converts the transformed input image on GPU into an image tensor stored in
# tflite::gpu::GlBuffer. The zero_center option is set to true to normalize the
# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f]. The flip_vertically
# option is set to true to account for the descrepancy between the
# representation of the input image (origin at the bottom-left corner, the
# OpenGL convention) and what the model used in this graph is expecting (origin
# at the top-left corner).
# Converts the transformed input image on GPU into an image tensor stored as a
# TfLiteTensor.
node {
calculator: "TfLiteConverterCalculator"
input_stream: "IMAGE_GPU:transformed_input_video"
output_stream: "TENSORS_GPU:image_tensor"
node_options: {
[type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
zero_center: true
flip_vertically: true
}
}
}
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
@@ -106,7 +112,7 @@ node {
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS_GPU:image_tensor"
output_stream: "TENSORS_GPU:detection_tensors"
output_stream: "TENSORS:detection_tensors"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "ssdlite_object_detection.tflite"
@@ -149,7 +155,7 @@ node {
# detections. Each detection describes a detected object.
node {
calculator: "TfLiteTensorsToDetectionsCalculator"
input_stream: "TENSORS_GPU:detection_tensors"
input_stream: "TENSORS:detection_tensors"
input_side_packet: "ANCHORS:anchors"
output_stream: "DETECTIONS:detections"
node_options: {
@@ -164,7 +170,7 @@ node {
y_scale: 10.0
h_scale: 5.0
w_scale: 5.0
flip_vertically: true
min_score_thresh: 0.6
}
}
}
@@ -177,9 +183,9 @@ node {
node_options: {
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
min_suppression_threshold: 0.4
min_score_threshold: 0.6
max_num_detections: 3
overlap_type: INTERSECTION_OVER_UNION
return_empty_detections: true
}
}
}
@@ -200,7 +206,7 @@ node {
# Converts the detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTION_VECTOR:output_detections"
input_stream: "DETECTIONS:output_detections"
output_stream: "RENDER_DATA:render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
@@ -210,22 +216,13 @@ node {
}
}
# Draws annotations and overlays them on top of the original image coming into
# the graph. Annotation drawing is performed on CPU, and the result is
# transferred to GPU and overlaid on the input image. The calculator assumes
# that image origin is always at the top-left corner and renders text
# accordingly. However, the input image has its origin at the bottom-left corner
# (OpenGL convention) and the flip_text_vertically option is set to true to
# compensate that.
# Draws annotations and overlays them on top of a GPU copy of the original
# image coming into the graph. The calculator assumes that image origin is
# always at the top-left corner and renders text accordingly.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "INPUT_FRAME_GPU:throttled_input_video"
input_stream: "render_data"
output_stream: "OUTPUT_FRAME_GPU:output_video"
node_options: {
[type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
flip_text_vertically: true
}
}
}
```
mediapipe/docs/scheduling_sync.md
+8 -9
View File
@@ -143,14 +143,13 @@ system that relaxes configured limits when needed.
The second system consists of inserting special nodes which can drop packets
according to real-time constraints (typically using custom input policies)
defined by [`RealTimeFlowLimiterCalculator`]. For example, a common pattern
places a flow-control node at the input of a subgraph, with a loopback
connection from the final output to the flow-control node. The flow-control node
is thus able to keep track of how many timestamps are being processed in the
downstream graph, and drop packets if this count hits a (configurable) limit;
and since packets are dropped upstream, we avoid the wasted work that would
result from partially processing a timestamp and then dropping packets between
intermediate stages.
defined by [`FlowLimiterCalculator`]. For example, a common pattern places a
flow-control node at the input of a subgraph, with a loopback connection from
the final output to the flow-control node. The flow-control node is thus able to
keep track of how many timestamps are being processed in the downstream graph,
and drop packets if this count hits a (configurable) limit; and since packets
are dropped upstream, we avoid the wasted work that would result from partially
processing a timestamp and then dropping packets between intermediate stages.
This calculator-based approach gives the graph author control of where packets
can be dropped, and allows flexibility in adapting and customizing the graph’s
@@ -161,4 +160,4 @@ behavior depending on resource constraints.
[`SyncSetInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/sync_set_input_stream_handler.h
[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.h
[`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`RealTimeFlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/real_time_flow_limiter_calculator.cc
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
mediapipe/docs/tracer.md
View File
mediapipe/docs/troubleshooting.md
+2 -2
View File
@@ -128,7 +128,7 @@ If some of the calculators in the graph cannot keep pace with the realtime input
streams, then latency will continue to increase, and it becomes necessary to
drop some input packets. The recommended technique is to use the MediaPipe
calculators designed specifically for this purpose such as
[`RealTimeFlowLimiterCalculator`] as described in
[`FlowLimiterCalculator`] as described in
[How to process realtime input streams](how_to_questions.md#how-to-process-realtime-input-streams).
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
@@ -141,4 +141,4 @@ calculators designed specifically for this purpose such as
[`CalculatorGraph::WaitUntilDone`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
[`Timestamp::Done`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/timestamp.h
[`CalculatorBase::Close`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
[`RealTimeFlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/real_time_flow_limiter_calculator.cc
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
mediapipe/docs/visualizer.md
+21
View File
@@ -59,3 +59,24 @@ The visualizer graph shows the connections between calculator nodes.
![Special nodes](./images/special_nodes.png){width="350"}
![Special nodes](./images/special_nodes_code.png){width="350"}
### Visualizing subgraphs
The MediaPipe visualizer can display multiple graphs. If a graph has a name (designated by assigning a string to the "type" field in the top level of the graph's proto file) and that name is used as a calculator name in a separate graph, it is considered a subgraph and colored appropriately where it is used. Clicking on a subgraph will navigate to the corresponding tab which holds the subgraph's definition. In this example, for hand detection GPU we have 2 pbtxt files:
[hand_detection_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
and its associated [subgraph](./framework_concepts.md#subgraph) called
[hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
* In the default MediaPipe visualizer, click on upload graph button and select
the 2 pbtxt files to visualize (main graph and all its associated subgraphs)
![Upload graph button](./images/upload_button.png){width="250"}
![Choose the 2 files](./images/upload_2pbtxt.png){width="400"}
* You will see 3 tabs. The main graph tab is `hand_detection_mobile.pbtxt`
![hand_detection_mobile_gpu.pbtxt](./images/maingraph_visualizer.png){width="1500"}
* Click on the subgraph block in purple `Hand Detection` and the
`hand_detection_gpu.pbtxt` tab will open
![Hand detection subgraph](./images/clicksubgraph_handdetection.png){width="1500"}