talksik/mediapipe
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
PiperOrigin-RevId: 263982686
This commit is contained in:
MediaPipe Team
2019-08-17 20:08:42 -07:00
committed by chuoling
parent 7d4ec1e24d
commit c27a7c1e10
71 changed files with 2066 additions and 153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mediapipe/docs/examples.md
+7 -5
View File
@@ -22,7 +22,7 @@ Android example users go through in detail. It teaches the following:
### Hello World! on iOS
[Hello World! on iOS](./hello_world_ios.md) is the iOS version of Sobel edge
detection example
detection example.
### Object Detection with GPU
@@ -44,8 +44,9 @@ graphs can be easily adapted to run on CPU v.s. GPU.
[Face Detection with GPU](./face_detection_mobile_gpu.md) illustrates how to use
MediaPipe with a TFLite model for face detection in a GPU-accelerated pipeline.
The selfie face detection TFLite model is based on
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface).
[Model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface),
and model details are described in the
[model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
* [Android](./face_detection_mobile_gpu.md#android)
* [iOS](./face_detection_mobile_gpu.md#ios)
@@ -71,8 +72,9 @@ MediaPipe with a TFLite model for hand tracking in a GPU-accelerated pipeline.
[Hair Segmentation on GPU](./hair_segmentation_mobile_gpu.md) illustrates how to
use MediaPipe with a TFLite model for hair segmentation in a GPU-accelerated
pipeline. The selfie hair segmentation TFLite model is based on
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation).
[Model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation),
and model details are described in the
[model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
* [Android](./hair_segmentation_mobile_gpu.md#android)
mediapipe/docs/face_detection_mobile_gpu.md
+13 -11
View File
@@ -4,22 +4,22 @@ This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
that performs face detection with TensorFlow Lite on GPU.
![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif){width="300"}
![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif)
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Face Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu)
example app. To build the app, run:
The graph below is used in the
[Face Detection GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu).
To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu
```
To further install the app on android device, run:
To further install the app on an Android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.apk
@@ -28,13 +28,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Face Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu).
instructions to develop an iOS application that uses MediaPipe.
To build the iOS app, please see the general
The graph below is used in the
[Face Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu).
To build the app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
Specific to this example, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp
@@ -42,11 +42,13 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:Fa
## Graph
![face_detection_mobile_gpu_graph](images/mobile/face_detection_mobile_gpu.png){width="400"}
![face_detection_mobile_gpu_graph](images/mobile/face_detection_mobile_gpu.png)
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
```bash
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
# Used in the example in
mediapipe/docs/hair_segmentation_mobile_gpu.md
+9 -7
View File
@@ -1,25 +1,25 @@
# Hair Segmentation (GPU)
This doc focuses on the
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_android_gpu.pbtxt)
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt)
that performs hair segmentation with TensorFlow Lite on GPU.
![hair_segmentation_android_gpu_gif](images/mobile/hair_segmentation_android_gpu.gif){width="300"}
![hair_segmentation_android_gpu_gif](images/mobile/hair_segmentation_android_gpu.gif)
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Hair Segmentation GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu)
example app. To build the app, run:
The graph below is used in the
[Hair Segmentation GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu).
To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu
```
To further install the app on android device, run:
To further install the app on an Android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu/hairsegmentationgpu.apk
@@ -27,11 +27,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## Graph
![hair_segmentation_mobile_gpu_graph](images/mobile/hair_segmentation_mobile_gpu.png){width="600"}
![hair_segmentation_mobile_gpu_graph](images/mobile/hair_segmentation_mobile_gpu.png)
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt)
```bash
# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
# Used in the example in
mediapipe/docs/hand_detection_mobile_gpu.md
+39 -26
View File
@@ -2,30 +2,36 @@
This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
that performs hand detection with TensorFlow Lite on GPU. This hand detection
example is related to
[hand tracking GPU example](./hand_tracking_mobile_gpu.md). Here is the
[model card](https://mediapipe.page.link/handmc) for hand detection.
that performs hand detection with TensorFlow Lite on GPU. It is related to the
[hand tracking example](./hand_tracking_mobile_gpu.md).
For overall context on hand detection and hand tracking, please read
[this Google AI blog post](https://mediapipe.page.link/handgoogleaiblog).
For overall context on hand detection and hand tracking, please read this
[Google AI Blog post](https://mediapipe.page.link/handgoogleaiblog).
![hand_detection_android_gpu_gif](images/mobile/hand_detection_android_gpu.gif){width="300"}
![hand_detection_android_gpu_gif](images/mobile/hand_detection_android_gpu.gif)
In the visualization above, green boxes represent the results of palm detection,
and the red box represents the extended hand rectangle designed to cover the
entire hand. The palm detection ML model (see also
[model card](https://mediapipe.page.link/handmc)) supports detection of multiple
palms, and this example selects only the one with the highest detection
confidence score to generate the hand rectangle, to be further utilized in the
[hand tracking example](./hand_tracking_mobile_gpu.md).
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Hand Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu)
example app. To build the app, run:
The graph below is used in the
[Hand Detection GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu).
To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu
```
To further install the app on android device, run:
To further install the app on an Android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu/handdetectiongpu.apk
@@ -34,13 +40,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Hand Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu)
instructions to develop an iOS application that uses MediaPipe.
To build the iOS app, please see the general
The graph below is used in the
[Hand Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu).
To build the app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
Specific to this example, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp
@@ -48,17 +54,18 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:Ha
## Graph
The hand detection graph is
[hand_detection_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
and it includes a [HandDetectionSubgraph](./framework_concepts.md#subgraph) with
filename
[hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
shown as a box called `HandDetection` in purple
The hand detection [main graph](#main-graph) internally utilizes a
[hand detection subgraph](#hand-detection-subgraph). The subgraph shows up in
the main graph visualization as the `HandDetection` node colored in purple, and
the subgraph itself can also be visualized just like a regular graph. For more
information on how to visualize a graph that includes subgraphs, see
[visualizing subgraphs](./visualizer.md#visualizing-subgraphs).
For more information on how to visualize a graph that includes subgraphs, see
[subgraph documentation](./visualizer.md#visualizing-subgraphs) for Visualizer.
### Main Graph
![hand_detection_mobile_graph](images/mobile/hand_detection_mobile.png){width="500"}
![hand_detection_mobile_graph](images/mobile/hand_detection_mobile.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
```bash
# MediaPipe graph that performs hand detection with TensorFlow Lite on GPU.
@@ -125,9 +132,15 @@ node {
}
```
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png){width="500"}
### Hand Detection Subgraph
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
```bash
# MediaPipe hand detection subgraph.
type: "HandDetectionSubgraph"
input_stream: "input_video"
mediapipe/docs/hand_tracking_mobile_gpu.md
+55 -31
View File
@@ -1,32 +1,41 @@
# Hand Tracking (GPU)
This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_android_gpu.pbtxt)
that performs hand tracking with TensorFlow Lite on GPU. This hand tracking
example is related to
[hand detection GPU example](./hand_detection_mobile_gpu.md). We recommend users
to review the hand detection GPU example first. Here is the
[model card](https://mediapipe.page.link/handmc) for hand tracking.
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
that performs hand tracking with TensorFlow Lite on GPU. It is related to the
[hand detection example](./hand_detection_mobile_gpu.md), and we recommend users
to review the hand detection example first.
For overall context on hand detection and hand tracking, please read
[this Google AI blog post](https://mediapipe.page.link/handgoogleaiblog).
For overall context on hand detection and hand tracking, please read this
[Google AI Blog post](https://mediapipe.page.link/handgoogleaiblog).
![hand_tracking_android_gpu.gif](images/mobile/hand_tracking_android_gpu.gif){width="300"}
![hand_tracking_android_gpu.gif](images/mobile/hand_tracking_android_gpu.gif)
In the visualization above, the red dots represent the localized hand landmarks,
and the green lines are simply connections between selected landmark pairs for
visualization of the hand skeleton. The red box represents a hand rectangle that
covers the entire hand, derived either from hand detection (see
[hand detection example](./hand_detection_mobile_gpu.md)) or from the pervious
round of hand landmark localization using an ML model (see also
[model card](https://mediapipe.page.link/handmc)). Hand landmark localization is
performed only within the hand rectangle for computational efficiency and
accuracy, and hand detection is only invoked when landmark localization could
not identify hand presence in the previous iteration.
## Android
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph is used in the
[Hand Tracking GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu)
example app. To build the app, run:
The graph below is used in the
[Hand Tracking GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu).
To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu
```
To further install the app on android device, run:
To further install the app on an Android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
@@ -35,13 +44,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Hand Tracking GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu)
instructions to develop an iOS application that uses MediaPipe.
To build the iOS app, please see the general
The graph below is used in the
[Hand Tracking GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu).
To build the app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
Specific to this example, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
@@ -49,20 +58,21 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:Han
## Graph
For more information on how to visualize a graph that includes subgraphs, see
[subgraph documentation](./visualizer.md#visualizing-subgraphs) for Visualizer.
The hand tracking [main graph](#main-graph) internally utilizes a
[hand detection subgraph](#hand-detection-subgraph), a
[hand landmark subgraph](#hand-landmark-subgraph) and a
[renderer subgraph](#renderer-subgraph).
The hand tracking graph is
[hand_tracking_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
and it includes 3 [subgraphs](./framework_concepts.md#subgraph):
The subgraphs show up in the main graph visualization as nodes colored in
purple, and the subgraph itself can also be visualized just like a regular
graph. For more information on how to visualize a graph that includes subgraphs,
see [visualizing subgraphs](./visualizer.md#visualizing-subgraphs).
* [HandDetectionSubgraph - hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
### Main Graph
* [HandLandmarkSubgraph - hand_landmark_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_landmark_gpu.pbtxt)
![hand_tracking_mobile_graph](images/mobile/hand_tracking_mobile.png)
* [RendererSubgraph - renderer_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/renderer_gpu.pbtxt)
![hand_tracking_mobile_graph](images/mobile/hand_tracking_mobile.png){width="400"}
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
```bash
# MediaPipe graph that performs hand tracking with TensorFlow Lite on GPU.
@@ -152,9 +162,15 @@ node {
}
```
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png){width="500"}
### Hand Detection Subgraph
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
```bash
# MediaPipe hand detection subgraph.
type: "HandDetectionSubgraph"
input_stream: "input_video"
@@ -352,7 +368,11 @@ node {
}
```
![hand_landmark_gpu_subgraph.pbtxt](images/mobile/hand_landmark_gpu_subgraph.png){width="400"}
### Hand Landmark Subgraph
![hand_landmark_gpu_subgraph.pbtxt](images/mobile/hand_landmark_gpu_subgraph.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_landmark_gpu.pbtxt)
```bash
# MediaPipe hand landmark localization subgraph.
@@ -532,7 +552,11 @@ node {
}
```
![hand_renderer_gpu_subgraph.pbtxt](images/mobile/hand_renderer_gpu_subgraph.png){width="500"}
### Renderer Subgraph
![hand_renderer_gpu_subgraph.pbtxt](images/mobile/hand_renderer_gpu_subgraph.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/renderer_gpu.pbtxt)
```bash
# MediaPipe hand tracking rendering subgraph.
mediapipe/docs/hello_world_android.md
+5 -5
View File
@@ -14,7 +14,7 @@ graph on Android.
A simple camera app for real-time Sobel edge detection applied to a live video
stream on an Android device.
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif){width="300"}
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif)
## Setup
@@ -56,7 +56,7 @@ node: {
A visualization of the graph is shown below:
![edge_detection_mobile_gpu_graph](images/mobile/edge_detection_mobile_graph_gpu.png){width="200"}
![edge_detection_mobile_gpu](images/mobile/edge_detection_mobile_gpu.png)
This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera.
@@ -252,7 +252,7 @@ adb install bazel-bin/$APPLICATION_PATH/edgedetectiongpu.apk
Open the application on your device. It should display a screen with the text
`Hello World!`.
![bazel_hello_world_android](images/mobile/bazel_hello_world_android.png){width="300"}
![bazel_hello_world_android](images/mobile/bazel_hello_world_android.png)
## Using the camera via `CameraX`
@@ -369,7 +369,7 @@ Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file:
When the user doesn't grant camera permission, the screen will now look like
this:
![missing_camera_permission_android](images/mobile/missing_camera_permission_android.png){width="300"}
![missing_camera_permission_android](images/mobile/missing_camera_permission_android.png)
Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
`MainActivity`:
@@ -709,7 +709,7 @@ And that's it! You should now be able to successfully build and run the
application on the device and see Sobel edge detection running on a live camera
feed! Congrats!
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif){width="300"}
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif)
If you ran into any issues, please see the full code of the tutorial
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu).
mediapipe/docs/hello_world_desktop.md
+1 -1
View File
@@ -72,7 +72,7 @@
This graph consists of 1 graph input stream (`in`) and 1 graph output stream
(`out`), and 2 [`PassThroughCalculator`]s connected serially.
![hello_world.cc graph](./images/hello_world_graph.png){width="200"}
![hello_world graph](./images/hello_world.png)
4. Before running the graph, an `OutputStreamPoller` object is connected to the
output stream in order to later retrieve the graph output, and a graph run
mediapipe/docs/hello_world_ios.md
+33 -32
View File
@@ -14,7 +14,7 @@ graph on iOS.
A simple camera app for real-time Sobel edge detection applied to a live video
stream on an iOS device.
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif){width="300"}
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif)
## Setup
@@ -54,7 +54,7 @@ node: {
A visualization of the graph is shown below:
![edge_detection_mobile_gpu_graph](images/mobile/edge_detection_mobile_graph_gpu.png){width="200"}
![edge_detection_mobile_gpu](images/mobile/edge_detection_mobile_gpu.png)
This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera.
@@ -174,10 +174,11 @@ bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
```
For example, to build the `EdgeDetectionGpuApp` application in
`mediapipe/examples/ios/edgedetection`, use the following command:
`mediapipe/examples/ios/edgedetectiongpu`, use the following
command:
```
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetection:EdgeDetectionGpuApp
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp
```
Then, go back to XCode, open Window > Devices and Simulators, select your
@@ -188,9 +189,9 @@ blank white screen.
## Use the camera for the live view feed
In this tutorial, we will use the `MediaPipeCameraInputSource` class to access
and grab frames from the camera. This class uses the `AVCaptureSession` API to
get the frames from the camera.
In this tutorial, we will use the `MPPCameraInputSource` class to access and
grab frames from the camera. This class uses the `AVCaptureSession` API to get
the frames from the camera.
But before using this class, change the `Info.plist` file to support camera
usage in the app.
@@ -198,7 +199,7 @@ usage in the app.
In `ViewController.m`, add the following import line:
```
#import "mediapipe/objc/MediaPipeCameraInputSource.h"
#import "mediapipe/objc/MPPCameraInputSource.h"
```
Add the following to its implementation block to create an object
@@ -207,7 +208,7 @@ Add the following to its implementation block to create an object
```
@implementation ViewController {
// Handles camera access via AVCaptureSession library.
MediaPipeCameraInputSource* _cameraSource;
MPPCameraInputSource* _cameraSource;
}
```
@@ -217,7 +218,7 @@ Add the following code to `viewDidLoad()`:
-(void)viewDidLoad {
[super viewDidLoad];
_cameraSource = [[MediaPipeCameraInputSource alloc] init];
_cameraSource = [[MPPCameraInputSource alloc] init];
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
// The frame's native format is rotated with respect to the portrait orientation.
@@ -229,10 +230,10 @@ The code initializes `_cameraSource`, sets the capture session preset, and which
camera to use.
We need to get frames from the `_cameraSource` into our application
`ViewController` to display them. `MediaPipeCameraInputSource` is a subclass of
`MediaPipeInputSource`, which provides a protocol for its delegates, namely the
`MediaPipeInputSourceDelegate`. So our application `ViewController` can be a
delegate of `_cameraSource`.
`ViewController` to display them. `MPPCameraInputSource` is a subclass of
`MPPInputSource`, which provides a protocol for its delegates, namely the
`MPPInputSourceDelegate`. So our application `ViewController` can be a delegate
of `_cameraSource`.
To handle camera setup and process incoming frames, we should use a queue
different from the main queue. Add the following to the implementation block of
@@ -269,11 +270,11 @@ the interface/implementation of the `ViewController`:
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
```
Before implementing any method from `MediaPipeInputSourceDelegate` protocol, we
must first set up a way to display the camera frames. MediaPipe provides another
utility called `MediaPipeLayerRenderer` to display images on the screen. This
utility can be used to display `CVPixelBufferRef` objects, which is the type of
the images provided by `MediaPipeCameraInputSource` to its delegates.
Before implementing any method from `MPPInputSourceDelegate` protocol, we must
first set up a way to display the camera frames. MediaPipe provides another
utility called `MPPLayerRenderer` to display images on the screen. This utility
can be used to display `CVPixelBufferRef` objects, which is the type of the
images provided by `MPPCameraInputSource` to its delegates.
To display images of the screen, we need to add a new `UIView` object called
`_liveView` to the `ViewController`.
@@ -284,7 +285,7 @@ Add the following lines to the implementation block of the `ViewController`:
// Display the camera preview frames.
IBOutlet UIView* _liveView;
// Render frames in a layer.
MediaPipeLayerRenderer* _renderer;
MPPLayerRenderer* _renderer;
```
Go to `Main.storyboard`, add a `UIView` object from the object library to the
@@ -296,7 +297,7 @@ Go back to `ViewController.m` and add the following code to `viewDidLoad()` to
initialize the `_renderer` object:
```
_renderer = [[MediaPipeLayerRenderer alloc] init];
_renderer = [[MPPLayerRenderer alloc] init];
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MediaPipeFrameScaleFillAndCrop;
@@ -308,7 +309,7 @@ To get frames from the camera, we will implement the following method:
// Must be invoked on _videoQueue.
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MediaPipeInputSource*)source {
fromSource:(MPPInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
@@ -322,7 +323,7 @@ To get frames from the camera, we will implement the following method:
}
```
This is a delegate method of `MediaPipeInputSource`. We first check that we are
This is a delegate method of `MPPInputSource`. We first check that we are
getting frames from the right source, i.e. the `_cameraSource`. Then we display
the frame received from the camera via `_renderer` on the main queue.
@@ -337,7 +338,7 @@ about to appear. To do this, we will implement the
```
Before we start running the camera, we need the user's permission to access it.
`MediaPipeCameraInputSource` provides a function
`MPPCameraInputSource` provides a function
`requestCameraAccessWithCompletionHandler:(void (^_Nullable)(BOOL
granted))handler` to request camera access and do some work when the user has
responded. Add the following code to `viewWillAppear:animated`:
@@ -413,7 +414,7 @@ Add the following property to the interface of the `ViewController`:
```
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
// sent video frames on _videoQueue.
@property(nonatomic) MediaPipeGraph* mediapipeGraph;
@property(nonatomic) MPPGraph* mediapipeGraph;
```
As explained in the comment above, we will initialize this graph in
@@ -421,7 +422,7 @@ As explained in the comment above, we will initialize this graph in
using the following function:
```
+ (MediaPipeGraph*)loadGraphFromResource:(NSString*)resource {
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
// Load the graph config resource.
NSError* configLoadError = nil;
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
@@ -440,7 +441,7 @@ using the following function:
config.ParseFromArray(data.bytes, data.length);
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
MediaPipeGraph* newGraph = [[MediaPipeGraph alloc] initWithGraphConfig:config];
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MediaPipePacketPixelBuffer];
return newGraph;
}
@@ -498,7 +499,7 @@ this function's implementation to do the following:
```
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MediaPipeInputSource*)source {
fromSource:(MPPInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
@@ -518,9 +519,9 @@ The graph will run with this input packet and output a result in
method to receive packets on this output stream and display them on the screen:
```
- (void)mediapipeGraph:(MediaPipeGraph*)graph
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
fromStream:(const std::string&)streamName {
- (void)mediapipeGraph:(MPPGraph*)graph
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
fromStream:(const std::string&)streamName {
if (streamName == kOutputStream) {
// Display the captured image on the screen.
CVPixelBufferRetain(pixelBuffer);
@@ -535,7 +536,7 @@ method to receive packets on this output stream and display them on the screen:
And that is all! Build and run the app on your iOS device. You should see the
results of running the edge detection graph on a live video feed. Congrats!
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif){width="300"}
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif)
If you ran into any issues, please see the full code of the tutorial
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
mediapipe/docs/images/hello_world.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.6 KiB

mediapipe/docs/images/hello_world_graph.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 39 KiB

mediapipe/docs/images/mobile/bazel_hello_world_android.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 24 KiB

After

Width:  |  Height:  |  Size: 18 KiB

mediapipe/docs/images/mobile/edge_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.4 MiB

After

Width:  |  Height:  |  Size: 666 KiB

mediapipe/docs/images/mobile/edge_detection_ios_gpu.gif
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 964 KiB

After

Width:  |  Height:  |  Size: 529 KiB

mediapipe/docs/images/mobile/edge_detection_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

mediapipe/docs/images/mobile/edge_detection_mobile_graph_gpu.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 37 KiB

mediapipe/docs/images/mobile/face_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.8 MiB

After

Width:  |  Height:  |  Size: 1.8 MiB

mediapipe/docs/images/mobile/hair_segmentation_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 80 KiB

After

Width:  |  Height:  |  Size: 64 KiB

mediapipe/docs/images/mobile/hand_detection_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 105 KiB

After

Width:  |  Height:  |  Size: 84 KiB

mediapipe/docs/images/mobile/hand_tracking_mobile.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 65 KiB

After

Width:  |  Height:  |  Size: 49 KiB

mediapipe/docs/images/mobile/missing_camera_permission_android.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 20 KiB

mediapipe/docs/images/mobile/object_detection_desktop_tensorflow.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 94 KiB

mediapipe/docs/images/mobile/object_detection_desktop_tflite.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 80 KiB

mediapipe/docs/images/object_detection_desktop_tensorflow.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 220 KiB

After

Width:  |  Height:  |  Size: 94 KiB

mediapipe/docs/images/object_detection_desktop_tflite.png
BIN
View File
Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 156 KiB

After

Width:  |  Height:  |  Size: 64 KiB

mediapipe/docs/index.rst
+4 -4
View File
@@ -8,9 +8,9 @@ machine learning pipeline can be built as a graph of modular components,
including, for instance, inference models and media processing functions. Sensory
data such as audio and video streams enter the graph, and perceived descriptions
such as object-localization and face-landmark streams exit the graph. An example
graph that performs real-time hair segmentation on mobile GPU is shown below.
graph that performs real-time hand tracking on mobile GPU is shown below.
.. image:: images/mobile/hair_segmentation_android_gpu.png
.. image:: images/mobile/hand_tracking_mobile.png
:width: 400
:alt: Example MediaPipe graph
@@ -29,11 +29,11 @@ APIs for MediaPipe
* (Coming Soon) Graph Construction API in C++
* Graph Execution API in C++
* Graph Execution API in Java (Android)
* (Coming Soon) Graph Execution API in Objective-C (iOS)
* Graph Execution API in Objective-C (iOS)
Alpha Disclaimer
==================
MediaPipe is currently in alpha for v0.5. We are still making breaking API changes and expect to get to stable API by v1.0. We recommend that you target a specific version of MediaPipe, and periodically bump to the latest release. That way you have control over when a breaking change affects you.
MediaPipe is currently in alpha for v0.6. We are still making breaking API changes and expect to get to stable API by v1.0. We recommend that you target a specific version of MediaPipe, and periodically bump to the latest release. That way you have control over when a breaking change affects you.
User Documentation
==================
mediapipe/docs/object_detection_desktop.md
+2 -2
View File
@@ -44,7 +44,7 @@ $ bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tensorf
#### Graph
![graph visualization](images/object_detection_desktop_tensorflow.png){width="800"}
![graph visualization](images/object_detection_desktop_tensorflow.png)
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into
@@ -209,7 +209,7 @@ $ bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite
#### Graph
![graph visualization](images/object_detection_desktop_tflite.png){width="400"}
![graph visualization](images/object_detection_desktop_tflite.png)
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into
mediapipe/docs/object_detection_mobile_cpu.md
+18 -15
View File
@@ -1,8 +1,6 @@
# Object Detection (CPU)
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe. This
doc focuses on the
This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt)
that performs object detection with TensorFlow Lite on CPU.
@@ -11,22 +9,25 @@ This is very similar to the
except that at the beginning and the end of the graph it performs GPU-to-CPU and
CPU-to-GPU image transfer respectively. As a result, the rest of graph, which
shares the same configuration as the
[GPU graph](images/mobile/object_detection_android_gpu.png), runs entirely on
[GPU graph](images/mobile/object_detection_mobile_gpu.png), runs entirely on
CPU.
![object_detection_android_cpu_gif](images/mobile/object_detection_android_cpu.gif){width="300"}
![object_detection_android_cpu_gif](images/mobile/object_detection_android_cpu.gif)
## Android
The graph is used in the
[Object Detection CPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu)
example app. To build the app, run:
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph below is used in the
[Object Detection CPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu).
To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu
```
To further install the app on android device, run:
To further install the app on an Android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu/objectdetectioncpu.apk
@@ -35,13 +36,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectioncpu).
instructions to develop an iOS application that uses MediaPipe.
To build the iOS app, please see the general
The graph below is used in the
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectioncpu).
To build the app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
Specific to this example, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp
@@ -49,11 +50,13 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectioncpu:
## Graph
![object_detection_mobile_cpu_graph](images/mobile/object_detection_mobile_cpu.png){width="400"}
![object_detection_mobile_cpu_graph](images/mobile/object_detection_mobile_cpu.png)
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt)
```bash
# MediaPipe graph that performs object detection with TensorFlow Lite on CPU.
# Used in the example in
mediapipe/docs/object_detection_mobile_gpu.md
+12 -10
View File
@@ -4,7 +4,7 @@ This doc focuses on the
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt)
that performs object detection with TensorFlow Lite on GPU.
![object_detection_android_gpu_gif](images/mobile/object_detection_android_gpu.gif){width="300"}
![object_detection_android_gpu_gif](images/mobile/object_detection_android_gpu.gif)
## Android
@@ -12,14 +12,14 @@ Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
general instructions to develop an Android application that uses MediaPipe.
The graph below is used in the
[Object Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu)
example app. To build the app, run:
[Object Detection GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu).
To build the app, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu
```
To further install the app on android device, run:
To further install the app on an Android device, run:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
@@ -28,13 +28,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
## iOS
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
instructions to develop an iOS application that uses MediaPipe. The graph below
is used in the
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectiongpu)
instructions to develop an iOS application that uses MediaPipe.
To build the iOS app, please see the general
The graph below is used in the
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectiongpu).
To build the app, please see the general
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
Specifically, run:
Specific to this example, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp
@@ -42,11 +42,13 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectiongpu:
## Graph
![object_detection_mobile_gpu_graph](images/mobile/object_detection_mobile_gpu.png){width="400"}
![object_detection_mobile_gpu_graph](images/mobile/object_detection_mobile_gpu.png)
To visualize the graph as shown above, copy the text specification of the graph
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt)
```bash
# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
# Used in the example in
mediapipe/docs/visualizer.md
+1 -1
View File
@@ -79,4 +79,4 @@ and its associated [subgraph](./framework_concepts.md#subgraph) called
* Click on the subgraph block in purple `Hand Detection` and the
`hand_detection_gpu.pbtxt` tab will open
![Hand detection subgraph](./images/clicksubgraph_handdetection.png){width="1500"}
![Hand detection subgraph](./images/click_subgraph_handdetection.png){width="1500"}