Project import generated by Copybara.

GitOrigin-RevId: 72ff4ae24943c2ccf9905bc9e516042b0aa3dd86
This commit is contained in:
MediaPipe Team
2020-04-13 20:15:59 -04:00
committed by chuoling
parent 4c68eb4a70
commit 16e5d7242d
112 changed files with 4762 additions and 217 deletions
+26 -5
View File
@@ -73,6 +73,18 @@ can be easily adapted to run on CPU v.s. GPU.
* [Android](./face_detection_mobile_cpu.md)
* [iOS](./face_detection_mobile_cpu.md)
### Face Mesh with GPU
[Face Mesh with GPU](./face_mesh_mobile_gpu.md) illustrates how to run the
MediaPipe Face Mesh pipeline to perform 3D face landmark estimation in real-time
on mobile devices, utilizing GPU acceleration. The pipeline is based on
["Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs"](https://arxiv.org/abs/1907.06724),
and details of the underlying ML models are described in the
[model card](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view).
* [Android](./face_mesh_mobile_gpu.md)
* [iOS](./face_mesh_mobile_gpu.md)
### Hand Detection with GPU
[Hand Detection with GPU](./hand_detection_mobile_gpu.md) illustrates how to use
@@ -84,7 +96,7 @@ MediaPipe with a TFLite model for hand detection in a GPU-accelerated pipeline.
### Hand Tracking with GPU
[Hand Tracking with GPU](./hand_tracking_mobile_gpu.md) illustrates how to use
MediaPipe with a TFLite model for hand tracking in a GPU-accelerated pipeline.
MediaPipe with TFLite models for hand tracking in a GPU-accelerated pipeline.
* [Android](./hand_tracking_mobile_gpu.md)
* [iOS](./hand_tracking_mobile_gpu.md)
@@ -92,7 +104,7 @@ MediaPipe with a TFLite model for hand tracking in a GPU-accelerated pipeline.
### Multi-Hand Tracking with GPU
[Multi-Hand Tracking with GPU](./multi_hand_tracking_mobile_gpu.md) illustrates
how to use MediaPipe with a TFLite model for multi-hand tracking in a
how to use MediaPipe with TFLite models for multi-hand tracking in a
GPU-accelerated pipeline.
* [Android](./multi_hand_tracking_mobile_gpu.md)
@@ -150,11 +162,20 @@ GPU with live video from a webcam.
* [Desktop GPU](./face_detection_desktop.md)
* [Desktop CPU](./face_detection_desktop.md)
### Face Mesh on Desktop with Webcam
[Face Mesh on Desktop with Webcam](./face_mesh_desktop.md) shows how to run the
MediaPipe Face Mesh pipeline to perform 3D face landmark estimation in real-time
on desktop with webcam input.
* [Desktop GPU](./face_mesh_desktop.md)
* [Desktop CPU](./face_mesh_desktop.md)
### Hand Tracking on Desktop with Webcam
[Hand Tracking on Desktop with Webcam](./hand_tracking_desktop.md) shows how to
use MediaPipe with a TFLite model for hand tracking on desktop using CPU or GPU
use MediaPipe with TFLite models for hand tracking on desktop using CPU or GPU
with live video from a webcam.
* [Desktop GPU](./hand_tracking_desktop.md)
@@ -163,8 +184,8 @@ with live video from a webcam.
### Multi-Hand Tracking on Desktop with Webcam
[Multi-Hand Tracking on Desktop with Webcam](./multi_hand_tracking_desktop.md)
shows how to use MediaPipe with a TFLite model for multi-hand tracking on
desktop using CPU or GPU with live video from a webcam.
shows how to use MediaPipe with TFLite models for multi-hand tracking on desktop
using CPU or GPU with live video from a webcam.
* [Desktop GPU](./multi_hand_tracking_desktop.md)
* [Desktop CPU](./multi_hand_tracking_desktop.md)
@@ -4,6 +4,8 @@ This doc focuses on the
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_cpu.pbtxt)
that performs face detection with TensorFlow Lite on CPU.
![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif)
## Android
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu)
+58
View File
@@ -0,0 +1,58 @@
## Face Mesh on Desktop with Webcam
This doc focuses on running the **MediaPipe Face Mesh** pipeline to perform 3D
face landmark estimation in real-time on desktop with webcam input. The pipeline
internally incorporates TensorFlow Lite models. To know more about the models,
please refer to the model
[README file](https://github.com/google/mediapipe/tree/master/mediapipe/models/README.md#face-mesh).
Moreover, if you are interested in running the same pipeline on Android/iOS,
please see [Face Mesh on Android/iOS](face_mesh_mobile_gpu.md).
- [Face Mesh on Desktop with Webcam (CPU)](#face-mesh-on-desktop-with-webcam-cpu)
- [Face Mesh on Desktop with Webcam (GPU)](#face-mesh-on-desktop-with-webcam-gpu)
Note: Desktop GPU works only on Linux. Mesa drivers need to be installed. Please
see
[step 4 of "Installing on Debian and Ubuntu" in the installation guide](./install.md).
Note: If MediaPipe depends on OpenCV 2, please see the [known issues with OpenCV 2](#known-issues-with-opencv-2) section.
### Face Mesh on Desktop with Webcam (CPU)
To build and run Face Mesh on desktop with webcam (CPU), run:
```bash
$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/face_mesh:face_mesh_cpu
# It should print:
# Target //mediapipe/examples/desktop/face_mesh:face_mesh_cpu up-to-date:
# bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_cpu
# This will open up your webcam as long as it is connected. Errors are likely
# due to your webcam being not accessible.
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_cpu \
--calculator_graph_config_file=mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt
```
### Face Mesh on Desktop with Webcam (GPU)
Note: please first [check that your GPU is supported](gpu.md#desktop-gpu-linux).
To build and run Face Mesh on desktop with webcam (GPU), run:
```bash
# This works only for linux currently
$ bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
mediapipe/examples/desktop/face_mesh:face_mesh_gpu
# It should print:
# Target //mediapipe/examples/desktop/face_mesh:face_mesh_gpu up-to-date:
# bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_gpu
# This will open up your webcam as long as it is connected. Errors are likely
# due to your webcam being not accessible, or GPU drivers not setup properly.
$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/face_mesh/face_mesh_gpu \
--calculator_graph_config_file=mediapipe/graphs/face_mesh/face_mesh_desktop_live_gpu.pbtxt
```
+90
View File
@@ -0,0 +1,90 @@
# Face Mesh (GPU)
This example focuses on running the **MediaPipe Face Mesh** pipeline on mobile
devices to perform 3D face landmark estimation in real-time, utilizing GPU
acceleration. The pipeline internally incorporates TensorFlow Lite models. To
know more about the models, please refer to the model
[README file](https://github.com/google/mediapipe/tree/master/mediapipe/models/README.md#face-mesh).
The pipeline is related to the
[face detection example](./face_detection_mobile_gpu.md) as it internally
utilizes face detection and performs landmark estimation only within the
detected region.
![face_mesh_android_gpu.gif](images/mobile/face_mesh_android_gpu.gif)
**MediaPipe Face Mesh** generates 468 3D face landmarks in real-time on mobile
devices. In the visualization above, the red dots represent the landmarks, and
the green lines connecting landmarks illustrate the contours around the eyes,
eyebrows, lips and the entire face.
## Android
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu)
A prebuilt arm64 APK can be
[downloaded here](https://drive.google.com/open?id=1pUmd7CXCL_onYMbsZo5p91cH0oNnR4gi).
To build the app yourself, run:
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu
```
Once the app is built, install it on Android device with:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/facemeshgpu.apk
```
## iOS
[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facemeshgpu).
See the general [instructions](./mediapipe_ios_setup.md) for building iOS
examples and generating an Xcode project. This will be the FaceMeshGpuApp
target.
To build on the command line:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp
```
## Graph
The face mesh [main graph](#main-graph) utilizes a
[face landmark subgraph](#face-landmark-subgraph) from the
[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark),
and renders using a dedicated [face renderer subgraph](#face-renderer-subgraph).
The subgraphs show up in the main graph visualization as nodes colored in
purple, and the subgraph itself can also be visualized just like a regular
graph. For more information on how to visualize a graph that includes subgraphs,
see the Visualizing Subgraphs section in the
[visualizer documentation](./visualizer.md).
### Main Graph
![face_mesh_mobile_graph](images/mobile/face_mesh_mobile.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt)
### Face Landmark Subgraph
The
[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark)
contains several subgraphs that can be used to detect and track face landmarks.
In particular, in this example the
[FaceLandmarkFrontGPU](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
subgraph, suitable for images from front-facing cameras (i.e., selfie images)
and utilizing GPU acceleration, is selected.
![face_landmark_front_gpu_subgraph](images/mobile/face_landmark_front_gpu_subgraph.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
### Face Renderer Subgraph
![face_renderer_gpu_subgraph](images/mobile/face_renderer_gpu_subgraph.png)
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/subgraphs/face_renderer_gpu.pbtxt)
Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.8 KiB

After

Width:  |  Height:  |  Size: 9.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.8 MiB

After

Width:  |  Height:  |  Size: 2.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 350 KiB

After

Width:  |  Height:  |  Size: 808 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 121 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.1 MiB

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 430 KiB

After

Width:  |  Height:  |  Size: 460 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.8 MiB

After

Width:  |  Height:  |  Size: 3.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 383 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.5 MiB

After

Width:  |  Height:  |  Size: 5.6 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.6 MiB

After

Width:  |  Height:  |  Size: 4.7 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.4 MiB

After

Width:  |  Height:  |  Size: 448 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.2 MiB

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 475 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.6 MiB

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.6 MiB

After

Width:  |  Height:  |  Size: 1004 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.5 MiB

After

Width:  |  Height:  |  Size: 945 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 282 KiB

After

Width:  |  Height:  |  Size: 336 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.6 MiB

After

Width:  |  Height:  |  Size: 2.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

After

Width:  |  Height:  |  Size: 730 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 670 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 661 KiB

+1 -1
View File
@@ -12,7 +12,7 @@ We show the object detection demo with both TensorFlow model and TensorFlow Lite
- [TensorFlow Object Detection Demo](#tensorflow-object-detection-demo)
- [TensorFlow Lite Object Detection Demo](#tensorflow-lite-object-detection-demo)
- [TensorFlow Lite Object Detection Demo with Webcam (CPU)](#tensorflow-lite-object-detection-demo)
- [TensorFlow Lite Object Detection Demo with Webcam (CPU)](#tensorflow-lite-object-detection-demo-with-webcam-cpu)
Note: If MediaPipe depends on OpenCV 2, please see the [known issues with OpenCV 2](#known-issues-with-opencv-2) section.