Project import generated by Copybara.
GitOrigin-RevId: 73d686c40057684f8bfaca285368bf1813f9fc26
This commit is contained in:
+29
-29
@@ -20,34 +20,34 @@ nav_order: 2
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe Face Mesh is a face geometry solution that estimates 468 3D face
|
||||
landmarks in real-time even on mobile devices. It employs machine learning (ML)
|
||||
to infer the 3D surface geometry, requiring only a single camera input without
|
||||
the need for a dedicated depth sensor. Utilizing lightweight model architectures
|
||||
together with GPU acceleration throughout the pipeline, the solution delivers
|
||||
real-time performance critical for live experiences.
|
||||
MediaPipe Face Mesh is a solution that estimates 468 3D face landmarks in
|
||||
real-time even on mobile devices. It employs machine learning (ML) to infer the
|
||||
3D facial surface, requiring only a single camera input without the need for a
|
||||
dedicated depth sensor. Utilizing lightweight model architectures together with
|
||||
GPU acceleration throughout the pipeline, the solution delivers real-time
|
||||
performance critical for live experiences.
|
||||
|
||||
Additionally, the solution is bundled with the Face Geometry module that bridges
|
||||
the gap between the face landmark estimation and useful real-time augmented
|
||||
reality (AR) applications. It establishes a metric 3D space and uses the face
|
||||
landmark screen positions to estimate face geometry within that space. The face
|
||||
geometry data consists of common 3D geometry primitives, including a face pose
|
||||
transformation matrix and a triangular face mesh. Under the hood, a lightweight
|
||||
statistical analysis method called
|
||||
Additionally, the solution is bundled with the Face Transform module that
|
||||
bridges the gap between the face landmark estimation and useful real-time
|
||||
augmented reality (AR) applications. It establishes a metric 3D space and uses
|
||||
the face landmark screen positions to estimate a face transform within that
|
||||
space. The face transform data consists of common 3D primitives, including a
|
||||
face pose transformation matrix and a triangular face mesh. Under the hood, a
|
||||
lightweight statistical analysis method called
|
||||
[Procrustes Analysis](https://en.wikipedia.org/wiki/Procrustes_analysis) is
|
||||
employed to drive a robust, performant and portable logic. The analysis runs on
|
||||
CPU and has a minimal speed/memory footprint on top of the ML model inference.
|
||||
|
||||
 |
|
||||
:-------------------------------------------------------------: |
|
||||
*Fig 1. AR effects utilizing facial surface geometry.* |
|
||||
*Fig 1. AR effects utilizing the 3D facial surface.* |
|
||||
|
||||
## ML Pipeline
|
||||
|
||||
Our ML pipeline consists of two real-time deep neural network models that work
|
||||
together: A detector that operates on the full image and computes face locations
|
||||
and a 3D face landmark model that operates on those locations and predicts the
|
||||
approximate surface geometry via regression. Having the face accurately cropped
|
||||
approximate 3D surface via regression. Having the face accurately cropped
|
||||
drastically reduces the need for common data augmentations like affine
|
||||
transformations consisting of rotations, translation and scale changes. Instead
|
||||
it allows the network to dedicate most of its capacity towards coordinate
|
||||
@@ -55,8 +55,8 @@ prediction accuracy. In addition, in our pipeline the crops can also be
|
||||
generated based on the face landmarks identified in the previous frame, and only
|
||||
when the landmark model could no longer identify face presence is the face
|
||||
detector invoked to relocalize the face. This strategy is similar to that
|
||||
employed in our [MediaPipe Hands](./hands.md) solution, which uses a palm detector
|
||||
together with a hand landmark model.
|
||||
employed in our [MediaPipe Hands](./hands.md) solution, which uses a palm
|
||||
detector together with a hand landmark model.
|
||||
|
||||
The pipeline is implemented as a MediaPipe
|
||||
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt)
|
||||
@@ -128,7 +128,7 @@ about the model in this [paper](https://arxiv.org/abs/2006.10962).
|
||||
:---------------------------------------------------------------------------: |
|
||||
*Fig 3. Attention Mesh: Overview of model architecture.* |
|
||||
|
||||
## Face Geometry Module
|
||||
## Face Transform Module
|
||||
|
||||
The [Face Landmark Model](#face-landmark-model) performs a single-camera face landmark
|
||||
detection in the screen coordinate space: the X- and Y- coordinates are
|
||||
@@ -140,7 +140,7 @@ enable the full spectrum of augmented reality (AR) features like aligning a
|
||||
virtual 3D object with a detected face.
|
||||
|
||||
The
|
||||
[Face Geometry module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry)
|
||||
[Face Transform module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry)
|
||||
moves away from the screen coordinate space towards a metric 3D space and
|
||||
provides necessary primitives to handle a detected face as a regular 3D object.
|
||||
By design, you'll be able to use a perspective camera to project the final 3D
|
||||
@@ -151,7 +151,7 @@ landmark positions are not changed.
|
||||
|
||||
#### Metric 3D Space
|
||||
|
||||
The **Metric 3D space** established within the Face Geometry module is a
|
||||
The **Metric 3D space** established within the Face Transform module is a
|
||||
right-handed orthonormal metric 3D coordinate space. Within the space, there is
|
||||
a **virtual perspective camera** located at the space origin and pointed in the
|
||||
negative direction of the Z-axis. In the current pipeline, it is assumed that
|
||||
@@ -184,11 +184,11 @@ functions:
|
||||
|
||||
### Components
|
||||
|
||||
#### Geometry Pipeline
|
||||
#### Transform Pipeline
|
||||
|
||||
The **Geometry Pipeline** is a key component, which is responsible for
|
||||
estimating face geometry objects within the Metric 3D space. On each frame, the
|
||||
following steps are executed in the given order:
|
||||
The **Transform Pipeline** is a key component, which is responsible for
|
||||
estimating the face transform objects within the Metric 3D space. On each frame,
|
||||
the following steps are executed in the given order:
|
||||
|
||||
- Face landmark screen coordinates are converted into the Metric 3D space
|
||||
coordinates;
|
||||
@@ -199,12 +199,12 @@ following steps are executed in the given order:
|
||||
positions (XYZ), while both the vertex texture coordinates (UV) and the
|
||||
triangular topology are inherited from the canonical face model.
|
||||
|
||||
The geometry pipeline is implemented as a MediaPipe
|
||||
The transform pipeline is implemented as a MediaPipe
|
||||
[calculator](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/geometry_pipeline_calculator.cc).
|
||||
For your convenience, the face geometry pipeline calculator is bundled together
|
||||
with corresponding metadata into a unified MediaPipe
|
||||
For your convenience, this calculator is bundled together with corresponding
|
||||
metadata into a unified MediaPipe
|
||||
[subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/face_geometry_from_landmarks.pbtxt).
|
||||
The face geometry format is defined as a Protocol Buffer
|
||||
The face transform format is defined as a Protocol Buffer
|
||||
[message](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/protos/face_geometry.proto).
|
||||
|
||||
#### Effect Renderer
|
||||
@@ -227,7 +227,7 @@ The effect renderer is implemented as a MediaPipe
|
||||
|
||||
|  |
|
||||
| :---------------------------------------------------------------------: |
|
||||
| *Fig 5. An example of face effects rendered by the Face Geometry Effect Renderer.* |
|
||||
| *Fig 5. An example of face effects rendered by the Face Transform Effect Renderer.* |
|
||||
|
||||
## Solution APIs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user