Project import generated by Copybara.
GitOrigin-RevId: 1e13be30e2c6838d4a2ff768a39c414bc80534bb
This commit is contained in:
MediaPipe Team
committed by
Sebastian Schmidt
Sebastian Schmidt
parent
63e679d99c
commit
4dc4b19ddb
@@ -27,7 +27,7 @@ to arbitrary aspect ratios.
|
||||
For overall context on AutoFlip, please read this
|
||||
[Google AI Blog](https://ai.googleblog.com/2020/02/autoflip-open-source-framework-for.html).
|
||||
|
||||
|
||||
|
||||
|
||||
## Building
|
||||
|
||||
@@ -61,7 +61,7 @@ command above accordingly to run AutoFlip against the videos.
|
||||
|
||||
## MediaPipe Graph
|
||||
|
||||
|
||||
|
||||
|
||||
To visualize the graph as shown above, copy the text specification of the graph
|
||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev).
|
||||
@@ -297,7 +297,7 @@ the required features cannot be all covered (for example, when they are too
|
||||
spread out in the video), AutoFlip will apply a padding effect to cover as much
|
||||
salient content as possible. See an illustration below.
|
||||
|
||||
|
||||
|
||||
|
||||
### Stable vs Tracking Camera Motion
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@ frame (e.g., [MediaPipe Object Detection](./object_detection.md)):
|
||||
* Object localization is temporally consistent with the help of tracking,
|
||||
meaning less jitter is observable across frames.
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------------------: |
|
||||
*Fig 1. Box tracking paired with ML-based object detection.* |
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ improved tie resolution strategy alternative to non-maximum suppression. For
|
||||
more information about BlazeFace, please see the [Resources](#resources)
|
||||
section.
|
||||
|
||||
|
||||
|
||||
|
||||
## Solution APIs
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ lightweight statistical analysis method called
|
||||
employed to drive a robust, performant and portable logic. The analysis runs on
|
||||
CPU and has a minimal speed/memory footprint on top of the ML model inference.
|
||||
|
||||
 |
|
||||
 |
|
||||
:-------------------------------------------------------------: |
|
||||
*Fig 1. AR effects utilizing the 3D facial surface.* |
|
||||
|
||||
@@ -107,7 +107,7 @@ angle and occlusions.
|
||||
You can find more information about the face landmark model in this
|
||||
[paper](https://arxiv.org/abs/1907.06724).
|
||||
|
||||
 |
|
||||
 |
|
||||
:------------------------------------------------------------------------: |
|
||||
*Fig 2. Face landmarks: the red box indicates the cropped area as input to the landmark model, the red dots represent the 468 landmarks in 3D, and the green lines connecting landmarks illustrate the contours around the eyes, eyebrows, lips and the entire face.* |
|
||||
|
||||
@@ -124,7 +124,7 @@ The attention mesh model can be selected in the Solution APIs via the
|
||||
[refine_landmarks](#refine_landmarks) option. You can also find more information
|
||||
about the model in this [paper](https://arxiv.org/abs/2006.10962).
|
||||
|
||||
 |
|
||||
 |
|
||||
:---------------------------------------------------------------------------: |
|
||||
*Fig 3. Attention Mesh: Overview of model architecture.* |
|
||||
|
||||
@@ -161,7 +161,7 @@ coordinates back into the Metric 3D space. The *virtual camera parameters* can
|
||||
be set freely, however for better results it is advised to set them as close to
|
||||
the *real physical camera parameters* as possible.
|
||||
|
||||
 |
|
||||
 |
|
||||
:-------------------------------------------------------------------------------: |
|
||||
*Fig 4. A visualization of multiple key elements in the Metric 3D space.* |
|
||||
|
||||
@@ -225,7 +225,7 @@ hiding invisible elements behind the face surface.
|
||||
The effect renderer is implemented as a MediaPipe
|
||||
[calculator](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/effect_renderer_calculator.cc).
|
||||
|
||||
|  |
|
||||
|  |
|
||||
| :---------------------------------------------------------------------: |
|
||||
| *Fig 5. An example of face effects rendered by the Face Transform Effect Renderer.* |
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ nav_order: 8
|
||||
</details>
|
||||
---
|
||||
|
||||
|
||||
|
||||
|
||||
## Example Apps
|
||||
|
||||
@@ -58,7 +58,7 @@ processed all locally in real-time and never leaves your device. Please see
|
||||
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
|
||||
in Google Developers Blog for details.
|
||||
|
||||
|
||||
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ hand perception functionality to the wider research and development community
|
||||
will result in an emergence of creative use cases, stimulating new applications
|
||||
and new research avenues.
|
||||
|
||||
 |
|
||||
 |
|
||||
:------------------------------------------------------------------------------------: |
|
||||
*Fig 1. Tracked 3D hand landmarks are represented by dots in different shades, with the brighter ones denoting landmarks closer to the camera.* |
|
||||
|
||||
@@ -91,9 +91,9 @@ To detect initial hand locations, we designed a
|
||||
mobile real-time uses in a manner similar to the face detection model in
|
||||
[MediaPipe Face Mesh](./face_mesh.md). Detecting hands is a decidedly complex
|
||||
task: our
|
||||
[lite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/palm_detection/palm_detection_lite.tflite)
|
||||
[lite model](https://storage.googleapis.com/mediapipe-assets/palm_detection_lite.tflite)
|
||||
and
|
||||
[full model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/palm_detection/palm_detection_full.tflite)
|
||||
[full model](https://storage.googleapis.com/mediapipe-assets/palm_detection_full.tflite)
|
||||
have to work across a variety of hand sizes with a large scale span (~20x)
|
||||
relative to the image frame and be able to detect occluded and self-occluded
|
||||
hands. Whereas faces have high contrast patterns, e.g., in the eye and mouth
|
||||
@@ -122,7 +122,7 @@ just 86.22%.
|
||||
### Hand Landmark Model
|
||||
|
||||
After the palm detection over the whole image our subsequent hand landmark
|
||||
[model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/hand_landmark/hand_landmark_full.tflite)
|
||||
[model](https://storage.googleapis.com/mediapipe-assets/hand_landmark_full.tflite)
|
||||
performs precise keypoint localization of 21 3D hand-knuckle coordinates inside
|
||||
the detected hand regions via regression, that is direct coordinate prediction.
|
||||
The model learns a consistent internal hand pose representation and is robust
|
||||
@@ -135,11 +135,11 @@ and provide additional supervision on the nature of hand geometry, we also
|
||||
render a high-quality synthetic hand model over various backgrounds and map it
|
||||
to the corresponding 3D coordinates.
|
||||
|
||||
 |
|
||||
 |
|
||||
:--------------------------------------------------------: |
|
||||
*Fig 2. 21 hand landmarks.* |
|
||||
|
||||
 |
|
||||
 |
|
||||
:-------------------------------------------------------------------------: |
|
||||
*Fig 3. Top: Aligned hand crops passed to the tracking network with ground truth annotation. Bottom: Rendered synthetic hand images with ground truth annotation.* |
|
||||
|
||||
|
||||
@@ -29,7 +29,7 @@ solutions for these tasks. Combining them all in real-time into a semantically
|
||||
consistent end-to-end solution is a uniquely difficult problem requiring
|
||||
simultaneous inference of multiple, dependent neural networks.
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------------------: |
|
||||
*Fig 1. Example of MediaPipe Holistic.* |
|
||||
|
||||
@@ -54,7 +54,7 @@ full-resolution input frame to these ROIs and apply task-specific face and hand
|
||||
models to estimate their corresponding landmarks. Finally, we merge all
|
||||
landmarks with those of the pose model to yield the full 540+ landmarks.
|
||||
|
||||
 |
|
||||
 |
|
||||
:------------------------------------------------------------------------------: |
|
||||
*Fig 2. MediaPipe Holistic Pipeline Overview.* |
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ platforms without initialization or calibration. It is built upon the
|
||||
Tracking, you can easily place virtual 2D and 3D content on static or moving
|
||||
surfaces, allowing them to seamlessly interact with the real-world environment.
|
||||
|
||||
 |
|
||||
 |
|
||||
:-----------------------------------------------------------------------: |
|
||||
*Fig 1. Instant Motion Tracking is used to augment the world with a 3D sticker.* |
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ of the MediaPipe framework, MediaPipe Iris can run on most modern
|
||||
[mobile phones](#mobile), [desktops/laptops](#desktop) and even on the
|
||||
[web](#web).
|
||||
|
||||
 |
|
||||
 |
|
||||
:------------------------------------------------------------------------: |
|
||||
*Fig 1. Example of MediaPipe Iris: eyelid (red) and iris (blue) contours.* |
|
||||
|
||||
@@ -102,7 +102,7 @@ The iris model takes an image patch of the eye region and estimates both the eye
|
||||
landmarks (along the eyelid) and iris landmarks (along ths iris contour). You
|
||||
can find more details in this [paper](https://arxiv.org/abs/2006.11341).
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------------------: |
|
||||
*Fig 2. Eye landmarks (red) and iris landmarks (green).* |
|
||||
|
||||
@@ -115,7 +115,7 @@ human eye remains roughly constant at 11.7±0.5 mm across a wide population,
|
||||
along with some simple geometric arguments. For more details please refer to our
|
||||
[Google AI Blog post](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html).
|
||||
|
||||
 |
|
||||
 |
|
||||
:--------------------------------------------------------------------------------------------: |
|
||||
*Fig 3. (Left) MediaPipe Iris predicting metric distance in cm on a Pixel 2 from iris tracking without use of a depth sensor. (Right) Ground-truth depth.* |
|
||||
|
||||
@@ -200,7 +200,7 @@ never leaves your device. Please see
|
||||
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
|
||||
in Google Developers Blog for details.
|
||||
|
||||
|
||||
|
||||
|
||||
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
|
||||
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
|
||||
|
||||
@@ -23,7 +23,7 @@ nav_order: 13
|
||||
MediaPipe KNIFT is a template-based feature matching solution using KNIFT
|
||||
(Keypoint Neural Invariant Feature Transform).
|
||||
|
||||
 |
|
||||
 |
|
||||
:-----------------------------------------------------------------------: |
|
||||
*Fig 1. Matching a real Stop Sign with a Stop Sign template using KNIFT.* |
|
||||
|
||||
@@ -56,7 +56,7 @@ For more information, please see
|
||||
[MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
|
||||
in Google Developers Blog.
|
||||
|
||||
 |
|
||||
 |
|
||||
:-------------------------------------------------------------------------------------: |
|
||||
*Fig 2. Matching US dollar bills using KNIFT.* |
|
||||
|
||||
@@ -70,7 +70,7 @@ pre-computed from the 3 template images (of US dollar bills) shown below. If
|
||||
you'd like to use your own template images, see
|
||||
[Matching Your Own Template Images](#matching-your-own-template-images).
|
||||
|
||||
|
||||
|
||||
|
||||
Please first see general instructions for
|
||||
[Android](../getting_started/android.md) on how to build MediaPipe examples.
|
||||
|
||||
+28
-28
@@ -15,14 +15,14 @@ nav_order: 30
|
||||
### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
||||
|
||||
* Short-range model (best for faces within 2 meters from the camera):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite),
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/face_detection_short_range.tflite),
|
||||
[TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite),
|
||||
[Model card](https://mediapipe.page.link/blazeface-mc)
|
||||
* Full-range model (dense, best for faces within 5 meters from the camera):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range.tflite),
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/face_detection_full_range.tflite),
|
||||
[Model card](https://mediapipe.page.link/blazeface-back-mc)
|
||||
* Full-range model (sparse, best for faces within 5 meters from the camera):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite),
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/face_detection_full_range_sparse.tflite),
|
||||
[Model card](https://mediapipe.page.link/blazeface-back-sparse-mc)
|
||||
|
||||
Full-range dense and sparse models have the same quality in terms of
|
||||
@@ -39,77 +39,77 @@ one over the other.
|
||||
### [Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)
|
||||
|
||||
* Face landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/face_landmark.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
|
||||
* Face landmark model w/ attention (aka Attention Mesh):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_with_attention.tflite)
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/face_landmark_with_attention.tflite)
|
||||
* [Model card](https://mediapipe.page.link/facemesh-mc),
|
||||
[Model card (w/ attention)](https://mediapipe.page.link/attentionmesh-mc)
|
||||
|
||||
### [Iris](https://google.github.io/mediapipe/solutions/iris)
|
||||
|
||||
* Iris landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark.tflite)
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/iris_landmark.tflite)
|
||||
* [Model card](https://mediapipe.page.link/iris-mc)
|
||||
|
||||
### [Hands](https://google.github.io/mediapipe/solutions/hands)
|
||||
|
||||
* Palm detection model:
|
||||
[TFLite model (lite)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/palm_detection/palm_detection_lite.tflite),
|
||||
[TFLite model (full)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/palm_detection/palm_detection_full.tflite),
|
||||
[TFLite model (lite)](https://storage.googleapis.com/mediapipe-assets/palm_detection_lite.tflite),
|
||||
[TFLite model (full)](https://storage.googleapis.com/mediapipe-assets/palm_detection_full.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/handdetector/1)
|
||||
* Hand landmark model:
|
||||
[TFLite model (lite)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/hand_landmark/hand_landmark_lite.tflite),
|
||||
[TFLite model (full)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/hand_landmark/hand_landmark_full.tflite),
|
||||
[TFLite model (lite)](https://storage.googleapis.com/mediapipe-assets/hand_landmark_lite.tflite),
|
||||
[TFLite model (full)](https://storage.googleapis.com/mediapipe-assets/hand_landmark_full.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/handskeleton/1)
|
||||
* [Model card](https://mediapipe.page.link/handmc)
|
||||
|
||||
### [Pose](https://google.github.io/mediapipe/solutions/pose)
|
||||
|
||||
* Pose detection model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection/pose_detection.tflite)
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/pose_detection.tflite)
|
||||
* Pose landmark model:
|
||||
[TFLite model (lite)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_lite.tflite),
|
||||
[TFLite model (full)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_full.tflite),
|
||||
[TFLite model (heavy)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_heavy.tflite)
|
||||
[TFLite model (lite)](https://storage.googleapis.com/mediapipe-assets/pose_landmark_lite.tflite),
|
||||
[TFLite model (full)](https://storage.googleapis.com/mediapipe-assets/pose_landmark_full.tflite),
|
||||
[TFLite model (heavy)](https://storage.googleapis.com/mediapipe-assets/pose_landmark_heavy.tflite)
|
||||
* [Model card](https://mediapipe.page.link/blazepose-mc)
|
||||
|
||||
### [Holistic](https://google.github.io/mediapipe/solutions/holistic)
|
||||
|
||||
* Hand recrop model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/holistic_landmark/hand_recrop.tflite)
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/hand_recrop.tflite)
|
||||
|
||||
### [Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)
|
||||
|
||||
* [TFLite model (general)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation.tflite)
|
||||
* [TFLite model (landscape)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation_landscape.tflite)
|
||||
* [TFLite model (general)](https://storage.googleapis.com/mediapipe-assets/selfie_segmentation.tflite)
|
||||
* [TFLite model (landscape)](https://storage.googleapis.com/mediapipe-assets/selfie_segmentation_landscape.tflite)
|
||||
* [Model card](https://mediapipe.page.link/selfiesegmentation-mc)
|
||||
|
||||
### [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)
|
||||
|
||||
* [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite)
|
||||
* [TFLite model](https://storage.googleapis.com/mediapipe-assets/hair_segmentation.tflite)
|
||||
* [Model card](https://mediapipe.page.link/hairsegmentation-mc)
|
||||
|
||||
### [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)
|
||||
|
||||
* [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/ssdlite_object_detection.tflite)
|
||||
* [TFLite model](https://storage.googleapis.com/mediapipe-assets/ssdlite_object_detection.tflite)
|
||||
* [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/object-detector-quantized_edgetpu.tflite)
|
||||
* [TensorFlow model](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model)
|
||||
* [Model information](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md)
|
||||
|
||||
### [Objectron](https://google.github.io/mediapipe/solutions/objectron)
|
||||
|
||||
* [TFLite model for shoes](https://github.com/google/mediapipe/tree/master/mediapipe/modules/objectron/object_detection_3d_sneakers.tflite)
|
||||
* [TFLite model for chairs](https://github.com/google/mediapipe/tree/master/mediapipe/modules/objectron/object_detection_3d_chair.tflite)
|
||||
* [TFLite model for cameras](https://github.com/google/mediapipe/tree/master/mediapipe/modules/objectron/object_detection_3d_camera.tflite)
|
||||
* [TFLite model for cups](https://github.com/google/mediapipe/tree/master/mediapipe/modules/objectron/object_detection_3d_cup.tflite)
|
||||
* [Single-stage TFLite model for shoes](https://github.com/google/mediapipe/tree/master/mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite)
|
||||
* [Single-stage TFLite model for chairs](https://github.com/google/mediapipe/tree/master/mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite)
|
||||
* [TFLite model for shoes](https://storage.googleapis.com/mediapipe-assets/object_detection_3d_sneakers.tflite)
|
||||
* [TFLite model for chairs](https://storage.googleapis.com/mediapipe-assets/object_detection_3d_chair.tflite)
|
||||
* [TFLite model for cameras](https://storage.googleapis.com/mediapipe-assets/object_detection_3d_camera.tflite)
|
||||
* [TFLite model for cups](https://storage.googleapis.com/mediapipe-assets/object_detection_3d_cup.tflite)
|
||||
* [Single-stage TFLite model for shoes](https://storage.googleapis.com/mediapipe-assets/object_detection_3d_sneakers_1stage.tflite)
|
||||
* [Single-stage TFLite model for chairs](https://storage.googleapis.com/mediapipe-assets/object_detection_3d_chair_1stage.tflite)
|
||||
* [Model card](https://mediapipe.page.link/objectron-mc)
|
||||
|
||||
### [KNIFT](https://google.github.io/mediapipe/solutions/knift)
|
||||
|
||||
* [TFLite model for up to 200 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float.tflite)
|
||||
* [TFLite model for up to 400 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float_400.tflite)
|
||||
* [TFLite model for up to 1000 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float_1k.tflite)
|
||||
* [TFLite model for up to 200 keypoints](https://storage.googleapis.com/mediapipe-assets/knift_float.tflite)
|
||||
* [TFLite model for up to 400 keypoints](https://storage.googleapis.com/mediapipe-assets/knift_float_400.tflite)
|
||||
* [TFLite model for up to 1000 keypoints](https://storage.googleapis.com/mediapipe-assets/knift_float_1k.tflite)
|
||||
* [Model card](https://mediapipe.page.link/knift-mc)
|
||||
|
||||
@@ -18,7 +18,7 @@ nav_order: 9
|
||||
</details>
|
||||
---
|
||||
|
||||
|
||||
|
||||
|
||||
## Example Apps
|
||||
|
||||
@@ -75,7 +75,7 @@ on how to build MediaPipe examples.
|
||||
* With a TFLite Model
|
||||
|
||||
This uses the same
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/ssdlite_object_detection.tflite)
|
||||
[TFLite model](https://storage.googleapis.com/mediapipe-assets/ssdlite_object_detection.tflite)
|
||||
(see also
|
||||
[model info](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md))
|
||||
as in [Live Camera Input](#live-camera-input) above. The pipeline is
|
||||
|
||||
+10
-10
@@ -24,7 +24,7 @@ MediaPipe Objectron is a mobile real-time 3D object detection solution for
|
||||
everyday objects. It detects objects in 2D images, and estimates their poses
|
||||
through a machine learning (ML) model, trained on the [Objectron dataset](https://github.com/google-research-datasets/Objectron).
|
||||
|
||||
 |  |  | 
|
||||
 |  |  | 
|
||||
:--------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------:
|
||||
*Fig 1a. Shoe Objectron* | *Fig 1b. Chair Objectron* | *Fig 1c. Camera Objectron* | *Fig 1d. Cup Objectron*
|
||||
|
||||
@@ -39,7 +39,7 @@ mature and has been widely used in the industry, 3D object detection from 2D
|
||||
imagery is a challenging problem, due to the lack of data and diversity of
|
||||
appearances and shapes of objects within a category.
|
||||
|
||||
 |
|
||||
 |
|
||||
:-----------------------------------------------------------------------: |
|
||||
*Fig 2. Objectron example results.* |
|
||||
|
||||
@@ -68,7 +68,7 @@ object in a single frame and propagate its location to all frames using the
|
||||
ground truth camera pose information from the AR session data, which makes the
|
||||
procedure highly efficient.
|
||||
|
||||
|  |
|
||||
|  |
|
||||
| :--------------------------------------------------------------------------: |
|
||||
| *Fig 3. Real-world data annotation for 3D object detection. (Right) 3D bounding boxes are annotated in the 3D world with detected surfaces and point clouds. (Left) Projections of annotated 3D bounding boxes are overlaid on top of video frames making it easy to validate the annotation.* |
|
||||
|
||||
@@ -86,7 +86,7 @@ with rendered objects that respect the scene geometry and fit seamlessly into
|
||||
real backgrounds. By combining real-world data and AR synthetic data, we are
|
||||
able to increase the accuracy by about 10%.
|
||||
|
||||
 |
|
||||
 |
|
||||
:-------------------------------------------------------------------------------------------: |
|
||||
*Fig 4. An example of AR synthetic data generation. The virtual white-brown cereal box is rendered into the real scene, next to the real blue book.* |
|
||||
|
||||
@@ -107,7 +107,7 @@ takes the image crop and estimates the 3D bounding box. At the same time, it
|
||||
also computes the 2D crop of the object for the next frame, such that the object
|
||||
detector does not need to run every frame.
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------: |
|
||||
*Fig 5. Network architecture and post-processing for two-stage 3D object detection.* |
|
||||
|
||||
@@ -119,7 +119,7 @@ mobile GPU.
|
||||
|
||||
### Single-stage Pipeline
|
||||
|
||||
 |
|
||||
 |
|
||||
:---------------------------------------------------------------------------------: |
|
||||
*Fig 6. Network architecture and post-processing for single-stage 3D object detection.* |
|
||||
|
||||
@@ -144,7 +144,7 @@ object dimensions. Given the 3D bounding box, we can easily compute pose and
|
||||
size of the object. The model is light enough to run real-time on mobile devices
|
||||
(at 26 FPS on an Adreno 650 mobile GPU).
|
||||
|
||||
 |
|
||||
 |
|
||||
:-------------------------------------------------------------------------------------: |
|
||||
*Fig 7. Sample results of our network — (Left) original 2D image with estimated bounding boxes, (Middle) object detection by Gaussian distribution, (Right) predicted segmentation mask.* |
|
||||
|
||||
@@ -573,7 +573,7 @@ Each object has its object coordinate frame. We use the below object coordinate
|
||||
definition, with `+x` pointing right, `+y` pointing up and `+z` pointing front,
|
||||
origin is at the center of the 3D bounding box.
|
||||
|
||||
|
||||
|
||||
|
||||
### Camera Coordinate
|
||||
|
||||
@@ -582,7 +582,7 @@ regard to the camera coordinate frame. In this API we use the below camera
|
||||
coordinate definition, with `+x` pointing right, `+y` pointing up and `-z`
|
||||
pointing to the scene.
|
||||
|
||||
|
||||
|
||||
|
||||
To work with box landmarks, one can first derive landmark coordinates in object
|
||||
frame by scaling a origin centered unit box with `scale`, then transform to
|
||||
@@ -599,7 +599,7 @@ In this API we use
|
||||
as an intermediate space when projecting points from 3D to 2D. In NDC space,
|
||||
`x`, `y` are confined to `[-1, 1]`.
|
||||
|
||||
|
||||
|
||||
|
||||
By default the camera parameters `(fx, fy)` and `(px, py)` are defined in NDC
|
||||
space. Given `(X, Y, Z)` of 3D points in camera coordinate, one can project 3D
|
||||
|
||||
@@ -40,7 +40,7 @@ environments for inference, whereas our method achieves real-time performance on
|
||||
most modern [mobile phones](#mobile), [desktops/laptops](#desktop), in
|
||||
[python](#python-solution-api) and even on the [web](#javascript-solution-api).
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------: |
|
||||
*Fig 1. Example of MediaPipe Pose for pose tracking.* |
|
||||
|
||||
@@ -94,7 +94,7 @@ BlazePose GHUM Lite
|
||||
[AlphaPose ResNet50](https://github.com/MVIG-SJTU/AlphaPose) | 63.4 | **96.0** | 57.8 | **95.5** | 63.4 | **96.0**
|
||||
[Apple Vision](https://developer.apple.com/documentation/vision/detecting_human_body_poses_in_images) | 32.8 | **82.7** | 36.4 | **91.4** | 44.5 | **88.6**
|
||||
|
||||
 |
|
||||
 |
|
||||
:--------------------------------------------------------------------------: |
|
||||
*Fig 2. Quality evaluation in [`[email protected]`].* |
|
||||
|
||||
@@ -121,7 +121,7 @@ predict the midpoint of a person's hips, the radius of a circle circumscribing
|
||||
the whole person, and the incline angle of the line connecting the shoulder and
|
||||
hip midpoints.
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------------------: |
|
||||
*Fig 3. Vitruvian man aligned via two virtual keypoints predicted by BlazePose detector in addition to the face bounding box.* |
|
||||
|
||||
@@ -130,7 +130,7 @@ hip midpoints.
|
||||
The landmark model in MediaPipe Pose predicts the location of 33 pose landmarks
|
||||
(see figure below).
|
||||
|
||||
 |
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------------: |
|
||||
*Fig 4. 33 pose landmarks.* |
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@ demo within
|
||||
Push-ups and squats are used for demonstration purposes as the most common
|
||||
exercises.
|
||||
|
||||
 |
|
||||
 |
|
||||
:--------------------------------------------------------------------------------------------------------: |
|
||||
*Fig 1. Pose classification and repetition counting with MediaPipe Pose.* |
|
||||
|
||||
@@ -58,7 +58,7 @@ exercise (e.g., "up" and "down" positions for push-ups). It's important that
|
||||
collected samples cover different camera angles, environment conditions, body
|
||||
shapes, and exercise variations.
|
||||
|
||||
 |
|
||||
 |
|
||||
:--------------------------------------------------------------------------------------------------------------------------: |
|
||||
*Fig 2. Two terminal states of push-ups.* |
|
||||
|
||||
@@ -90,7 +90,7 @@ ankle and hip, and two wrists. Since the algorithm relies on distances, all
|
||||
poses are normalized to have the same torso size and vertical torso orientation
|
||||
before the conversion.
|
||||
|
||||
 |
|
||||
 |
|
||||
:--------------------------------------------------------------------------------------------------------: |
|
||||
*Fig 3. Main pairwise distances used for the pose feature vector.* |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user