Project import generated by Copybara.

GitOrigin-RevId: 65b427572550bd9c5bc5f053eeea0f44340d5673

.github/ISSUE_TEMPLATE/00-build-installation-issue.md

@@ -1,3 +1,9 @@
+---
+name: "Build/Installation Issue"
+about: Use this template for build/installation issues
+labels: type:build/install
+
+---
 <em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
 
 **System information** (Please provide as much relevant information as possible)

.github/ISSUE_TEMPLATE/10-solution-issue.md

@@ -1,3 +1,9 @@
+---
+name: "Solution Issue"
+about: Use this template for assistance with a specific mediapipe solution, such as "Pose" or "Iris", including inference model usage/training, solution-specific calculators, etc.
+labels: type:support
+
+---
 <em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
 
 **System information** (Please provide as much relevant information as possible)

.github/ISSUE_TEMPLATE/20-documentation-issue.md

@@ -1,3 +1,9 @@
+---
+name: "Documentation Issue"
+about: Use this template for documentation related issues
+labels: type:docs
+
+---
 Thank you for submitting a MediaPipe documentation issue.
 The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
 ## URL(s) with the issue:

.github/ISSUE_TEMPLATE/30-bug-issue.md

@@ -1,3 +1,9 @@
+---
+name: "Bug Issue"
+about: Use this template for reporting a bug
+labels: type:bug
+
+---
 <em>Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.</em>
 
 **System information** (Please provide as much relevant information as possible)

.github/ISSUE_TEMPLATE/40-feature-request.md

@@ -1,3 +1,9 @@
+---
+name: "Feature Request"
+about: Use this template for raising a feature request
+labels: type:feature
+
+---
 <em>Please make sure that this is a feature request.</em>
 
 **System information** (Please provide as much relevant information as possible)

.github/ISSUE_TEMPLATE/50-other-issues.md

@@ -1,3 +1,9 @@
+---
+name: "Other Issue"
+about: Use this template for any other non-support related issues.
+labels: type:others
+
+---
 This template is for miscellaneous issues not covered by the other issue categories
 
 For questions on how to work with MediaPipe, or support for problems that are not verified bugs in MediaPipe, please go to [StackOverflow](https://stackoverflow.com/questions/tagged/mediapipe) and [Slack](https://mediapipe.page.link/joinslack) communities.

MANIFEST.in

@@ -8,6 +8,7 @@ include README.md
 include requirements.txt
 
 recursive-include mediapipe/modules *.tflite *.txt *.binarypb
+exclude mediapipe/modules/face_detection/face_detection_full_range.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers.tflite

README.md

@@ -55,46 +55,22 @@ See also
 [MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
 for ML models released in MediaPipe.
 
-## MediaPipe in Python
-
-MediaPipe offers customizable Python solutions as a prebuilt Python package on
-[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
-`pip install mediapipe`. It also provides tools for users to build their own
-solutions. Please see
-[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
-for more info.
-
-## MediaPipe on the Web
-
-MediaPipe on the Web is an effort to run the same ML solutions built for mobile
-and desktop also in web browsers. The official API is under construction, but
-the core technology has been proven effective. Please see
-[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
-in Google Developers Blog for details.
-
-You can use the following links to load a demo in the MediaPipe Visualizer, and
-over there click the "Runner" icon in the top bar like shown below. The demos
-use your webcam video as input, which is processed all locally in real-time and
-never leaves your device.
-
-![visualizer_runner](docs/images/visualizer_runner.png)
-
-*   [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
-*   [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
-*   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
-*   [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
-*   [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
-*   [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
-*   [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
-
 ## Getting started
 
-Learn how to [install](https://google.github.io/mediapipe/getting_started/install)
+To start using MediaPipe
-MediaPipe and
+[solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
-[build example applications](https://google.github.io/mediapipe/getting_started/building_examples),
+lines code, see example code and demos in
-and start exploring our ready-to-use
+[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
-[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
+[MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
-further extend and customize.
+
+To use MediaPipe in C++, Android and iOS, which allow further customization of
+the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
+building your own, learn how to
+[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
+start building example applications in
+[C++](https://google.github.io/mediapipe/getting_started/cpp),
+[Android](https://google.github.io/mediapipe/getting_started/android) and
+[iOS](https://google.github.io/mediapipe/getting_started/ios).
 
 The source code is hosted in the
 [MediaPipe Github repository](https://github.com/google/mediapipe), and you can

WORKSPACE

@@ -35,8 +35,8 @@ http_archive(
 
 http_archive(
     name = "rules_cc",
-    strip_prefix = "rules_cc-master",
+    strip_prefix = "rules_cc-main",
-    urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
+    urls = ["https://github.com/bazelbuild/rules_cc/archive/main.zip"],
 )
 
 http_archive(
@@ -242,6 +242,20 @@ http_archive(
     url = "https://github.com/opencv/opencv/releases/download/3.2.0/opencv-3.2.0-ios-framework.zip",
 )
 
+http_archive(
+    name = "stblib",
+    strip_prefix = "stb-b42009b3b9d4ca35bc703f5310eedc74f584be58",
+    sha256 = "13a99ad430e930907f5611325ec384168a958bf7610e63e60e2fd8e7b7379610",
+    urls = ["https://github.com/nothings/stb/archive/b42009b3b9d4ca35bc703f5310eedc74f584be58.tar.gz"],
+    build_file = "@//third_party:stblib.BUILD",
+    patches = [
+        "@//third_party:stb_image_impl.diff"
+    ],
+    patch_args = [
+        "-p1",
+    ],
+)
+
 # You may run setup_android.sh to install Android SDK and NDK.
 android_ndk_repository(
     name = "androidndk",
@@ -337,8 +351,8 @@ maven_install(
         "androidx.test.espresso:espresso-core:3.1.1",
         "com.github.bumptech.glide:glide:4.11.0",
         "com.google.android.material:material:aar:1.0.0-rc01",
-        "com.google.auto.value:auto-value:1.6.4",
+        "com.google.auto.value:auto-value:1.8.1",
-        "com.google.auto.value:auto-value-annotations:1.6.4",
+        "com.google.auto.value:auto-value-annotations:1.8.1",
         "com.google.code.findbugs:jsr305:3.0.2",
         "com.google.flogger:flogger-system-backend:0.3.1",
         "com.google.flogger:flogger:0.3.1",
@@ -369,9 +383,9 @@ http_archive(
 )
 
 # Tensorflow repo should always go after the other external dependencies.
-# 2021-05-27
+# 2021-06-07
-_TENSORFLOW_GIT_COMMIT = "d6bfcdb0926173dbb7aa02ceba5aae6250b8aaa6"
+_TENSORFLOW_GIT_COMMIT = "700533808e6016dc458bb2eeecfca4babfc482ec"
-_TENSORFLOW_SHA256 = "ec40e1462239d8783d02f76a43412c8f80bac71ea20e41e1b7729b990aad6923"
+_TENSORFLOW_SHA256 = "b6edd7f4039bfc19f3e77594ecff558ba620091d0dc48181484b3d9085026126"
 http_archive(
     name = "org_tensorflow",
     urls = [

docs/framework_concepts/calculators.md

@@ -262,7 +262,7 @@ specified, appear as literal values in the `node_options` field of the
     output_stream: "TENSORS:main_model_output"
     node_options: {
       [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
-        model_path: "mediapipe/models/active_speaker_detection/audio_visual_model.tflite"
+        model_path: "mediapipe/models/detection_model.tflite"
       }
     }
   }
@@ -272,14 +272,13 @@ The `node_options` field accepts the proto3 syntax.  Alternatively, calculator
 options can be specified in the `options` field using proto2 syntax.
 
 ```
-  node: {
+  node {
-    calculator: "IntervalFilterCalculator"
+    calculator: "TfLiteInferenceCalculator"
+    input_stream: "TENSORS:main_model_input"
+    output_stream: "TENSORS:main_model_output"
     node_options: {
-      [type.googleapis.com/mediapipe.IntervalFilterCalculatorOptions] {
+      [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
-        intervals {
+        model_path: "mediapipe/models/detection_model.tflite"
-          start_us: 20000
-          end_us: 40000
-        }
       }
     }
   }
@@ -287,13 +286,26 @@ options can be specified in the `options` field using proto2 syntax.
 
 Not all calculators accept calcuator options. In order to accept options, a
 calculator will normally define a new protobuf message type to represent its
-options, such as `IntervalFilterCalculatorOptions`. The calculator will then
+options, such as `PacketClonerCalculatorOptions`. The calculator will then
 read that protobuf message in its `CalculatorBase::Open` method, and possibly
-also in the `CalculatorBase::GetContract` function or its
+also in its `CalculatorBase::GetContract` function or its
 `CalculatorBase::Process` method. Normally, the new protobuf message type will
 be defined as a protobuf schema using a ".proto" file and a
 `mediapipe_proto_library()` build rule.
 
+```
+  mediapipe_proto_library(
+      name = "packet_cloner_calculator_proto",
+      srcs = ["packet_cloner_calculator.proto"],
+      visibility = ["//visibility:public"],
+      deps = [
+          "//mediapipe/framework:calculator_options_proto",
+          "//mediapipe/framework:calculator_proto",
+      ],
+  )
+```
+
+
 ## Example calculator
 
 This section discusses the implementation of `PacketClonerCalculator`, which

docs/getting_started/android_archive_library.md

@@ -92,12 +92,12 @@ each project.
     and copy
     [the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41)
     and
-    [the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite).
+    [the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite).
 
     ```bash
     bazel build -c opt mediapipe/graphs/face_detection:face_detection_mobile_gpu_binary_graph
     cp bazel-bin/mediapipe/graphs/face_detection/face_detection_mobile_gpu.binarypb /path/to/your/app/src/main/assets/
-    cp mediapipe/modules/face_detection/face_detection_front.tflite /path/to/your/app/src/main/assets/
+    cp mediapipe/modules/face_detection/face_detection_short_range.tflite /path/to/your/app/src/main/assets/
     ```
 
     ![Screenshot](../images/mobile/assets_location.png)
@@ -117,7 +117,6 @@ each project.
         implementation 'com.google.flogger:flogger-system-backend:0.3.1'
         implementation 'com.google.code.findbugs:jsr305:3.0.2'
         implementation 'com.google.guava:guava:27.0.1-android'
-        implementation 'com.google.guava:guava:27.0.1-android'
         implementation 'com.google.protobuf:protobuf-java:3.11.4'
         // CameraX core library
         def camerax_version = "1.0.0-beta10"
@@ -125,7 +124,7 @@ each project.
         implementation "androidx.camera:camera-camera2:$camerax_version"
         implementation "androidx.camera:camera-lifecycle:$camerax_version"
         // AutoValue
-        def auto_value_version = "1.6.4"
+        def auto_value_version = "1.8.1"
         implementation "com.google.auto.value:auto-value-annotations:$auto_value_version"
         annotationProcessor "com.google.auto.value:auto-value:$auto_value_version"
     }

docs/index.md

@@ -55,46 +55,22 @@ See also
 [MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
 for ML models released in MediaPipe.
 
-## MediaPipe in Python
-
-MediaPipe offers customizable Python solutions as a prebuilt Python package on
-[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
-`pip install mediapipe`. It also provides tools for users to build their own
-solutions. Please see
-[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
-for more info.
-
-## MediaPipe on the Web
-
-MediaPipe on the Web is an effort to run the same ML solutions built for mobile
-and desktop also in web browsers. The official API is under construction, but
-the core technology has been proven effective. Please see
-[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
-in Google Developers Blog for details.
-
-You can use the following links to load a demo in the MediaPipe Visualizer, and
-over there click the "Runner" icon in the top bar like shown below. The demos
-use your webcam video as input, which is processed all locally in real-time and
-never leaves your device.
-
-![visualizer_runner](images/visualizer_runner.png)
-
-*   [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
-*   [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
-*   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
-*   [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
-*   [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
-*   [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
-*   [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
-
 ## Getting started
 
-Learn how to [install](https://google.github.io/mediapipe/getting_started/install)
+To start using MediaPipe
-MediaPipe and
+[solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
-[build example applications](https://google.github.io/mediapipe/getting_started/building_examples),
+lines code, see example code and demos in
-and start exploring our ready-to-use
+[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
-[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
+[MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
-further extend and customize.
+
+To use MediaPipe in C++, Android and iOS, which allow further customization of
+the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
+building your own, learn how to
+[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
+start building example applications in
+[C++](https://google.github.io/mediapipe/getting_started/cpp),
+[Android](https://google.github.io/mediapipe/getting_started/android) and
+[iOS](https://google.github.io/mediapipe/getting_started/ios).
 
 The source code is hosted in the
 [MediaPipe Github repository](https://github.com/google/mediapipe), and you can

docs/solutions/face_detection.md

@@ -45,6 +45,15 @@ section.
 
 Naming style and availability may differ slightly across platforms/languages.
 
+#### model_selection
+
+An integer index `0` or `1`. Use `0` to select a short-range model that works
+best for faces within 2 meters from the camera, and `1` for a full-range model
+best for faces within 5 meters. For the full-range option, a sparse model is
+used for its improved inference speed. Please refer to the
+[model cards](./models.md#face_detection) for details. Default to `0` if not
+specified.
+
 #### min_detection_confidence
 
 Minimum confidence value (`[0.0, 1.0]`) from the face detection model for the
@@ -72,6 +81,7 @@ install MediaPipe Python package, then learn more in the companion
 
 Supported configuration options:
 
+*   [model_selection](#model_selection)
 *   [min_detection_confidence](#min_detection_confidence)
 
 ```python
@@ -83,7 +93,7 @@ mp_drawing = mp.solutions.drawing_utils
 # For static images:
 IMAGE_FILES = []
 with mp_face_detection.FaceDetection(
-    min_detection_confidence=0.5) as face_detection:
+    model_selection=1, min_detection_confidence=0.5) as face_detection:
   for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB and process it with MediaPipe Face Detection.
@@ -103,7 +113,7 @@ with mp_face_detection.FaceDetection(
 # For webcam input:
 cap = cv2.VideoCapture(0)
 with mp_face_detection.FaceDetection(
-    min_detection_confidence=0.5) as face_detection:
+    model_selection=0, min_detection_confidence=0.5) as face_detection:
   while cap.isOpened():
     success, image = cap.read()
     if not success:
@@ -139,6 +149,7 @@ and the following usage example.
 
 Supported configuration options:
 
+*   [modelSelection](#model_selection)
 *   [minDetectionConfidence](#min_detection_confidence)
 
 ```html
@@ -189,6 +200,7 @@ const faceDetection = new FaceDetection({locateFile: (file) => {
   return `https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/${file}`;
 }});
 faceDetection.setOptions({
+  modelSelection: 0
   minDetectionConfidence: 0.5
 });
 faceDetection.onResults(onResults);
@@ -255,10 +267,6 @@ same configuration as the GPU pipeline, runs entirely on CPU.
     *   Target:
         [`mediapipe/examples/desktop/face_detection:face_detection_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_detection/BUILD)
 
-### Web
-
-Please refer to [these instructions](../index.md#mediapipe-on-the-web).
-
 ### Coral
 
 Please refer to

docs/solutions/face_mesh.md

@@ -69,7 +69,7 @@ and renders using a dedicated
 The
 [face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
 internally uses a
-[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
+[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range_gpu.pbtxt)
 from the
 [face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
 

docs/solutions/hair_segmentation.md

@@ -51,7 +51,14 @@ to visualize its associated subgraphs, please see
 
 ### Web
 
-Please refer to [these instructions](../index.md#mediapipe-on-the-web).
+Use [this link](https://viz.mediapipe.dev/demo/hair_segmentation) to load a demo
+in the MediaPipe Visualizer, and over there click the "Runner" icon in the top
+bar like shown below. The demos use your webcam video as input, which is
+processed all locally in real-time and never leaves your device. Please see
+[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
+in Google Developers Blog for details.
+
+![visualizer_runner](../images/visualizer_runner.png)
 
 ## Resources
 

docs/solutions/holistic.md

@@ -176,6 +176,16 @@ A list of pose landmarks. Each landmark consists of the following:
 *   `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
     landmark being visible (present and not occluded) in the image.
 
+#### pose_world_landmarks
+
+Another list of pose landmarks in world coordinates. Each landmark consists of
+the following:
+
+*   `x`, `y` and `z`: Real-world 3D coordinates in meters with the origin at the
+    center between hips.
+*   `visibility`: Identical to that defined in the corresponding
+    [pose_landmarks](#pose_landmarks).
+
 #### face_landmarks
 
 A list of 468 face landmarks. Each landmark consists of `x`, `y` and `z`. `x`
@@ -245,6 +255,9 @@ with mp_holistic.Holistic(
     mp_drawing.draw_landmarks(
         annotated_image, results.pose_landmarks, mp_holistic.POSE_CONNECTIONS)
     cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
+    # Plot pose world landmarks.
+    mp_drawing.plot_landmarks(
+        results.pose_world_landmarks, mp_holistic.POSE_CONNECTIONS)
 
 # For webcam input:
 cap = cv2.VideoCapture(0)

docs/solutions/iris.md

@@ -69,7 +69,7 @@ and renders using a dedicated
 The
 [face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
 internally uses a
-[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
+[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range_gpu.pbtxt)
 from the
 [face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
 
@@ -193,7 +193,17 @@ on how to build MediaPipe examples.
 
 ### Web
 
-Please refer to [these instructions](../index.md#mediapipe-on-the-web).
+You can use the following links to load a demo in the MediaPipe Visualizer, and
+over there click the "Runner" icon in the top bar like shown below. The demos
+use your webcam video as input, which is processed all locally in real-time and
+never leaves your device. Please see
+[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
+in Google Developers Blog for details.
+
+![visualizer_runner](../images/visualizer_runner.png)
+
+*   [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
+*   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
 
 ## Resources
 

docs/solutions/models.md

@@ -14,17 +14,27 @@ nav_order: 30
 
 ### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
 
-*   Face detection model for front-facing/selfie camera:
+*   Short-range model (best for faces within 2 meters from the camera):
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite),
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite),
     [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite),
     [Model card](https://mediapipe.page.link/blazeface-mc)
-*   Face detection model for back-facing camera:
+*   Full-range model (dense, best for faces within 5 meters from the camera):
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back.tflite),
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range.tflite),
     [Model card](https://mediapipe.page.link/blazeface-back-mc)
-*   Face detection model for back-facing camera (sparse):
+*   Full-range model (sparse, best for faces within 5 meters from the camera):
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back_sparse.tflite),
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite),
     [Model card](https://mediapipe.page.link/blazeface-back-sparse-mc)
 
+Full-range dense and sparse models have the same quality in terms of
+[F-score](https://en.wikipedia.org/wiki/F-score) however differ in underlying
+metrics. The dense model is slightly better in
+[Recall](https://en.wikipedia.org/wiki/Precision_and_recall) whereas the sparse
+model outperforms the dense one in
+[Precision](https://en.wikipedia.org/wiki/Precision_and_recall). Speed-wise
+sparse model is ~30% faster when executing on CPU via
+[XNNPACK](https://github.com/google/XNNPACK) whereas on GPU the models
+demonstrate comparable latencies. Depending on your application, you may prefer
+one over the other.
 
 ### [Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)
 

docs/solutions/pose.md

@@ -194,10 +194,23 @@ A list of pose landmarks. Each landmark consists of the following:
 *   `z`: Represents the landmark depth with the depth at the midpoint of hips
     being the origin, and the smaller the value the closer the landmark is to
     the camera. The magnitude of `z` uses roughly the same scale as `x`.
-
 *   `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
     landmark being visible (present and not occluded) in the image.
 
+#### pose_world_landmarks
+
+*Fig 5. Example of MediaPipe Pose real-world 3D coordinates.* |
+:-----------------------------------------------------------: |
+<video autoplay muted loop preload style="height: auto; width: 480px"><source src="../images/mobile/pose_world_landmarks.mp4" type="video/mp4"></video> |
+
+Another list of pose landmarks in world coordinates. Each landmark consists of
+the following:
+
+*   `x`, `y` and `z`: Real-world 3D coordinates in meters with the origin at the
+    center between hips.
+*   `visibility`: Identical to that defined in the corresponding
+    [pose_landmarks](#pose_landmarks).
+
 ### Python Solution API
 
 Please first follow general [instructions](../getting_started/python.md) to
@@ -242,6 +255,9 @@ with mp_pose.Pose(
     mp_drawing.draw_landmarks(
         annotated_image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
     cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
+    # Plot pose world landmarks.
+    mp_drawing.plot_landmarks(
+        results.pose_world_landmarks, mp_pose.POSE_CONNECTIONS)
 
 # For webcam input:
 cap = cv2.VideoCapture(0)
@@ -294,6 +310,7 @@ Supported configuration options:
   <meta charset="utf-8">
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/control_utils_3d.js" crossorigin="anonymous"></script>
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/pose/pose.js" crossorigin="anonymous"></script>
 </head>
@@ -312,8 +329,15 @@ Supported configuration options:
 const videoElement = document.getElementsByClassName('input_video')[0];
 const canvasElement = document.getElementsByClassName('output_canvas')[0];
 const canvasCtx = canvasElement.getContext('2d');
+const landmarkContainer = document.getElementsByClassName('landmark-grid-container')[0];
+const grid = new LandmarkGrid(landmarkContainer);
 
 function onResults(results) {
+  if (!results.poseLandmarks) {
+    grid.updateLandmarks([]);
+    return;
+  }
+
   canvasCtx.save();
   canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
   canvasCtx.drawImage(
@@ -323,6 +347,8 @@ function onResults(results) {
   drawLandmarks(canvasCtx, results.poseLandmarks,
                 {color: '#FF0000', lineWidth: 2});
   canvasCtx.restore();
+
+  grid.updateLandmarks(results.poseWorldLandmarks);
 }
 
 const pose = new Pose({locateFile: (file) => {

docs/solutions/selfie_segmentation.md

@@ -284,6 +284,6 @@ on how to build MediaPipe examples.
 *   Google AI Blog:
     [Background Features in Google Meet, Powered by Web ML](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html)
 *   [ML Kit Selfie Segmentation API](https://developers.google.com/ml-kit/vision/selfie-segmentation)
-*   [Models and model cards](./models.md#selfie_segmentation)
+*   [Models and model cards](./models.md#selfie-segmentation)
 *   [Web demo](https://code.mediapipe.dev/codepen/selfie_segmentation)
 *   [Python Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)

mediapipe/calculators/core/BUILD

@@ -419,6 +419,23 @@ cc_library(
     alwayslink = 1,
 )
 
+cc_test(
+    name = "make_pair_calculator_test",
+    size = "small",
+    srcs = ["make_pair_calculator_test.cc"],
+    deps = [
+        ":make_pair_calculator",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:calculator_runner",
+        "//mediapipe/framework:timestamp",
+        "//mediapipe/framework/port:gtest_main",
+        "//mediapipe/framework/port:status",
+        "//mediapipe/framework/tool:validate_type",
+        "//mediapipe/util:packet_test_util",
+        "//mediapipe/util:time_series_test_util",
+    ],
+)
+
 cc_library(
     name = "matrix_multiply_calculator",
     srcs = ["matrix_multiply_calculator.cc"],
@@ -933,8 +950,8 @@ cc_test(
 )
 
 cc_library(
-    name = "split_normalized_landmark_list_calculator",
+    name = "split_landmarks_calculator",
-    srcs = ["split_normalized_landmark_list_calculator.cc"],
+    srcs = ["split_landmarks_calculator.cc"],
     visibility = ["//visibility:public"],
     deps = [
         ":split_vector_calculator_cc_proto",
@@ -948,10 +965,10 @@ cc_library(
 )
 
 cc_test(
-    name = "split_normalized_landmark_list_calculator_test",
+    name = "split_landmarks_calculator_test",
-    srcs = ["split_normalized_landmark_list_calculator_test.cc"],
+    srcs = ["split_landmarks_calculator_test.cc"],
     deps = [
-        ":split_normalized_landmark_list_calculator",
+        ":split_landmarks_calculator",
         ":split_vector_calculator_cc_proto",
         "//mediapipe/framework:calculator_framework",
         "//mediapipe/framework:calculator_runner",

mediapipe/calculators/core/end_loop_calculator.cc

@@ -28,6 +28,10 @@ typedef EndLoopCalculator<std::vector<::mediapipe::NormalizedRect>>
     EndLoopNormalizedRectCalculator;
 REGISTER_CALCULATOR(EndLoopNormalizedRectCalculator);
 
+typedef EndLoopCalculator<std::vector<::mediapipe::LandmarkList>>
+    EndLoopLandmarkListVectorCalculator;
+REGISTER_CALCULATOR(EndLoopLandmarkListVectorCalculator);
+
 typedef EndLoopCalculator<std::vector<::mediapipe::NormalizedLandmarkList>>
     EndLoopNormalizedLandmarkListVectorCalculator;
 REGISTER_CALCULATOR(EndLoopNormalizedLandmarkListVectorCalculator);

mediapipe/calculators/core/make_pair_calculator_test.cc

@@ -0,0 +1,70 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_runner.h"
+#include "mediapipe/framework/port/canonical_errors.h"
+#include "mediapipe/framework/port/gmock.h"
+#include "mediapipe/framework/port/gtest.h"
+#include "mediapipe/framework/port/status.h"
+#include "mediapipe/framework/port/status_matchers.h"
+#include "mediapipe/framework/timestamp.h"
+#include "mediapipe/framework/tool/validate_type.h"
+#include "mediapipe/util/packet_test_util.h"
+#include "mediapipe/util/time_series_test_util.h"
+
+namespace mediapipe {
+
+class MakePairCalculatorTest
+    : public mediapipe::TimeSeriesCalculatorTest<mediapipe::NoOptions> {
+ protected:
+  void SetUp() override {
+    calculator_name_ = "MakePairCalculator";
+    num_input_streams_ = 2;
+  }
+};
+
+TEST_F(MakePairCalculatorTest, ProducesExpectedPairs) {
+  InitializeGraph();
+  AppendInputPacket(new std::string("first packet"), Timestamp(1),
+                    /* input_index= */ 0);
+  AppendInputPacket(new std::string("second packet"), Timestamp(5),
+                    /* input_index= */ 0);
+  AppendInputPacket(new int(10), Timestamp(1), /* input_index= */ 1);
+  AppendInputPacket(new int(20), Timestamp(5), /* input_index= */ 1);
+
+  MP_ASSERT_OK(RunGraph());
+
+  EXPECT_THAT(
+      output().packets,
+      ::testing::ElementsAre(
+          mediapipe::PacketContainsTimestampAndPayload<
+              std::pair<Packet, Packet>>(
+              Timestamp(1),
+              ::testing::Pair(
+                  mediapipe::PacketContainsTimestampAndPayload<std::string>(
+                      Timestamp(1), std::string("first packet")),
+                  mediapipe::PacketContainsTimestampAndPayload<int>(
+                      Timestamp(1), 10))),
+          mediapipe::PacketContainsTimestampAndPayload<
+              std::pair<Packet, Packet>>(
+              Timestamp(5),
+              ::testing::Pair(
+                  mediapipe::PacketContainsTimestampAndPayload<std::string>(
+                      Timestamp(5), std::string("second packet")),
+                  mediapipe::PacketContainsTimestampAndPayload<int>(
+                      Timestamp(5), 20)))));
+}
+
+}  // namespace mediapipe

mediapipe/calculators/core/split_landmarks_calculator.cc

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-#ifndef MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_  // NOLINT
+#ifndef MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_  // NOLINT
-#define MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_  // NOLINT
+#define MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_  // NOLINT
 
 #include "mediapipe/calculators/core/split_vector_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
@@ -24,29 +24,30 @@
 
 namespace mediapipe {
 
-// Splits an input packet with NormalizedLandmarkList into
+// Splits an input packet with LandmarkListType into
-// multiple NormalizedLandmarkList output packets using the [begin, end) ranges
+// multiple LandmarkListType output packets using the [begin, end) ranges
 // specified in SplitVectorCalculatorOptions. If the option "element_only" is
 // set to true, all ranges should be of size 1 and all outputs will be elements
-// of type NormalizedLandmark. If "element_only" is false, ranges can be
+// of type LandmarkType. If "element_only" is false, ranges can be
-// non-zero in size and all outputs will be of type NormalizedLandmarkList.
+// non-zero in size and all outputs will be of type LandmarkListType.
 // If the option "combine_outputs" is set to true, only one output stream can be
 // specified and all ranges of elements will be combined into one
-// NormalizedLandmarkList.
+// LandmarkListType.
-class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
+template <typename LandmarkType, typename LandmarkListType>
+class SplitLandmarksCalculator : public CalculatorBase {
  public:
   static absl::Status GetContract(CalculatorContract* cc) {
     RET_CHECK(cc->Inputs().NumEntries() == 1);
     RET_CHECK(cc->Outputs().NumEntries() != 0);
 
-    cc->Inputs().Index(0).Set<NormalizedLandmarkList>();
+    cc->Inputs().Index(0).Set<LandmarkListType>();
 
     const auto& options =
         cc->Options<::mediapipe::SplitVectorCalculatorOptions>();
 
     if (options.combine_outputs()) {
       RET_CHECK_EQ(cc->Outputs().NumEntries(), 1);
-      cc->Outputs().Index(0).Set<NormalizedLandmarkList>();
+      cc->Outputs().Index(0).Set<LandmarkListType>();
       for (int i = 0; i < options.ranges_size() - 1; ++i) {
         for (int j = i + 1; j < options.ranges_size(); ++j) {
           const auto& range_0 = options.ranges(i);
@@ -81,9 +82,9 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
             return absl::InvalidArgumentError(
                 "Since element_only is true, all ranges should be of size 1.");
           }
-          cc->Outputs().Index(i).Set<NormalizedLandmark>();
+          cc->Outputs().Index(i).Set<LandmarkType>();
         } else {
-          cc->Outputs().Index(i).Set<NormalizedLandmarkList>();
+          cc->Outputs().Index(i).Set<LandmarkListType>();
         }
       }
     }
@@ -110,40 +111,39 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
   }
 
   absl::Status Process(CalculatorContext* cc) override {
-    const NormalizedLandmarkList& input =
+    const LandmarkListType& input =
-        cc->Inputs().Index(0).Get<NormalizedLandmarkList>();
+        cc->Inputs().Index(0).Get<LandmarkListType>();
     RET_CHECK_GE(input.landmark_size(), max_range_end_)
         << "Max range end " << max_range_end_ << " exceeds landmarks size "
         << input.landmark_size();
 
     if (combine_outputs_) {
-      NormalizedLandmarkList output;
+      LandmarkListType output;
       for (int i = 0; i < ranges_.size(); ++i) {
         for (int j = ranges_[i].first; j < ranges_[i].second; ++j) {
-          const NormalizedLandmark& input_landmark = input.landmark(j);
+          const LandmarkType& input_landmark = input.landmark(j);
           *output.add_landmark() = input_landmark;
         }
       }
       RET_CHECK_EQ(output.landmark_size(), total_elements_);
       cc->Outputs().Index(0).AddPacket(
-          MakePacket<NormalizedLandmarkList>(output).At(cc->InputTimestamp()));
+          MakePacket<LandmarkListType>(output).At(cc->InputTimestamp()));
     } else {
       if (element_only_) {
         for (int i = 0; i < ranges_.size(); ++i) {
           cc->Outputs().Index(i).AddPacket(
-              MakePacket<NormalizedLandmark>(input.landmark(ranges_[i].first))
+              MakePacket<LandmarkType>(input.landmark(ranges_[i].first))
                   .At(cc->InputTimestamp()));
         }
       } else {
         for (int i = 0; i < ranges_.size(); ++i) {
-          NormalizedLandmarkList output;
+          LandmarkListType output;
           for (int j = ranges_[i].first; j < ranges_[i].second; ++j) {
-            const NormalizedLandmark& input_landmark = input.landmark(j);
+            const LandmarkType& input_landmark = input.landmark(j);
             *output.add_landmark() = input_landmark;
           }
           cc->Outputs().Index(i).AddPacket(
-              MakePacket<NormalizedLandmarkList>(output).At(
+              MakePacket<LandmarkListType>(output).At(cc->InputTimestamp()));
-                  cc->InputTimestamp()));
         }
       }
     }
@@ -159,9 +159,15 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
   bool combine_outputs_ = false;
 };
 
+typedef SplitLandmarksCalculator<NormalizedLandmark, NormalizedLandmarkList>
+    SplitNormalizedLandmarkListCalculator;
 REGISTER_CALCULATOR(SplitNormalizedLandmarkListCalculator);
 
+typedef SplitLandmarksCalculator<Landmark, LandmarkList>
+    SplitLandmarkListCalculator;
+REGISTER_CALCULATOR(SplitLandmarkListCalculator);
+
 }  // namespace mediapipe
 
 // NOLINTNEXTLINE
-#endif  // MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_
+#endif  // MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_

mediapipe/calculators/image/BUILD

@@ -80,6 +80,16 @@ mediapipe_proto_library(
     ],
 )
 
+mediapipe_proto_library(
+    name = "segmentation_smoothing_calculator_proto",
+    srcs = ["segmentation_smoothing_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
 cc_library(
     name = "color_convert_calculator",
     srcs = ["color_convert_calculator.cc"],
@@ -602,3 +612,52 @@ cc_test(
         "//mediapipe/framework/port:parse_text_proto",
     ],
 )
+
+cc_library(
+    name = "segmentation_smoothing_calculator",
+    srcs = ["segmentation_smoothing_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":segmentation_smoothing_calculator_cc_proto",
+        "//mediapipe/framework:calculator_options_cc_proto",
+        "//mediapipe/framework/formats:image_format_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_frame_opencv",
+        "//mediapipe/framework/formats:image",
+        "//mediapipe/framework/formats:image_opencv",
+        "//mediapipe/framework/port:logging",
+        "//mediapipe/framework/port:opencv_core",
+        "//mediapipe/framework/port:status",
+        "//mediapipe/framework/port:vector",
+    ] + select({
+        "//mediapipe/gpu:disable_gpu": [],
+        "//conditions:default": [
+            "//mediapipe/gpu:gl_calculator_helper",
+            "//mediapipe/gpu:gl_simple_shaders",
+            "//mediapipe/gpu:gl_quad_renderer",
+            "//mediapipe/gpu:shader_util",
+        ],
+    }),
+    alwayslink = 1,
+)
+
+cc_test(
+    name = "segmentation_smoothing_calculator_test",
+    srcs = ["segmentation_smoothing_calculator_test.cc"],
+    deps = [
+        ":image_clone_calculator",
+        ":image_clone_calculator_cc_proto",
+        ":segmentation_smoothing_calculator",
+        ":segmentation_smoothing_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:calculator_runner",
+        "//mediapipe/framework/deps:file_path",
+        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_opencv",
+        "//mediapipe/framework/port:gtest_main",
+        "//mediapipe/framework/port:opencv_imgcodecs",
+        "//mediapipe/framework/port:opencv_imgproc",
+        "//mediapipe/framework/port:parse_text_proto",
+    ],
+)

mediapipe/calculators/image/segmentation_smoothing_calculator.cc

@@ -0,0 +1,429 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <algorithm>
+#include <memory>
+
+#include "mediapipe/calculators/image/segmentation_smoothing_calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_options.pb.h"
+#include "mediapipe/framework/formats/image.h"
+#include "mediapipe/framework/formats/image_format.pb.h"
+#include "mediapipe/framework/formats/image_frame.h"
+#include "mediapipe/framework/formats/image_frame_opencv.h"
+#include "mediapipe/framework/formats/image_opencv.h"
+#include "mediapipe/framework/port/logging.h"
+#include "mediapipe/framework/port/opencv_core_inc.h"
+#include "mediapipe/framework/port/status.h"
+#include "mediapipe/framework/port/vector.h"
+
+#if !MEDIAPIPE_DISABLE_GPU
+#include "mediapipe/gpu/gl_calculator_helper.h"
+#include "mediapipe/gpu/gl_simple_shaders.h"
+#include "mediapipe/gpu/shader_util.h"
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+namespace mediapipe {
+
+namespace {
+constexpr char kCurrentMaskTag[] = "MASK";
+constexpr char kPreviousMaskTag[] = "MASK_PREVIOUS";
+constexpr char kOutputMaskTag[] = "MASK_SMOOTHED";
+
+enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
+}  // namespace
+
+// A calculator for mixing two segmentation masks together,
+// based on an uncertantity probability estimate.
+//
+// Inputs:
+//   MASK - Image containing the new/current mask.
+//          [ImageFormat::VEC32F1, or
+//           GpuBufferFormat::kBGRA32/kRGB24/kGrayHalf16/kGrayFloat32]
+//   MASK_PREVIOUS - Image containing previous mask.
+//                   [Same format as MASK_CURRENT]
+//   * If input channels is >1, only the first channel (R) is used as the mask.
+//
+// Output:
+//   MASK_SMOOTHED - Blended mask.
+//                   [Same format as MASK_CURRENT]
+//   * The resulting filtered mask will be stored in R channel,
+//     and duplicated in A if 4 channels.
+//
+// Options:
+//   combine_with_previous_ratio - Amount of previous to blend with current.
+//
+// Example:
+//  node {
+//    calculator: "SegmentationSmoothingCalculator"
+//    input_stream: "MASK:mask"
+//    input_stream: "MASK_PREVIOUS:mask_previous"
+//    output_stream: "MASK_SMOOTHED:mask_smoothed"
+//    options: {
+//      [mediapipe.SegmentationSmoothingCalculatorOptions.ext] {
+//        combine_with_previous_ratio: 0.9
+//      }
+//    }
+//  }
+//
+class SegmentationSmoothingCalculator : public CalculatorBase {
+ public:
+  SegmentationSmoothingCalculator() = default;
+
+  static absl::Status GetContract(CalculatorContract* cc);
+
+  // From Calculator.
+  absl::Status Open(CalculatorContext* cc) override;
+  absl::Status Process(CalculatorContext* cc) override;
+  absl::Status Close(CalculatorContext* cc) override;
+
+ private:
+  absl::Status RenderGpu(CalculatorContext* cc);
+  absl::Status RenderCpu(CalculatorContext* cc);
+
+  absl::Status GlSetup(CalculatorContext* cc);
+  void GlRender(CalculatorContext* cc);
+
+  float combine_with_previous_ratio_;
+
+  bool gpu_initialized_ = false;
+#if !MEDIAPIPE_DISABLE_GPU
+  mediapipe::GlCalculatorHelper gpu_helper_;
+  GLuint program_ = 0;
+#endif  // !MEDIAPIPE_DISABLE_GPU
+};
+REGISTER_CALCULATOR(SegmentationSmoothingCalculator);
+
+absl::Status SegmentationSmoothingCalculator::GetContract(
+    CalculatorContract* cc) {
+  CHECK_GE(cc->Inputs().NumEntries(), 1);
+
+  cc->Inputs().Tag(kCurrentMaskTag).Set<Image>();
+  cc->Inputs().Tag(kPreviousMaskTag).Set<Image>();
+  cc->Outputs().Tag(kOutputMaskTag).Set<Image>();
+
+#if !MEDIAPIPE_DISABLE_GPU
+  MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::Open(CalculatorContext* cc) {
+  cc->SetOffset(TimestampDiff(0));
+
+  auto options =
+      cc->Options<mediapipe::SegmentationSmoothingCalculatorOptions>();
+  combine_with_previous_ratio_ = options.combine_with_previous_ratio();
+
+#if !MEDIAPIPE_DISABLE_GPU
+  MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
+#endif  //  !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::Process(CalculatorContext* cc) {
+  if (cc->Inputs().Tag(kCurrentMaskTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+  if (cc->Inputs().Tag(kPreviousMaskTag).IsEmpty()) {
+    // Pass through current image if previous is not available.
+    cc->Outputs()
+        .Tag(kOutputMaskTag)
+        .AddPacket(cc->Inputs().Tag(kCurrentMaskTag).Value());
+    return absl::OkStatus();
+  }
+
+  // Run on GPU if incoming data is on GPU.
+  const bool use_gpu = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>().UsesGpu();
+
+  if (use_gpu) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this, cc]() -> absl::Status {
+      if (!gpu_initialized_) {
+        MP_RETURN_IF_ERROR(GlSetup(cc));
+        gpu_initialized_ = true;
+      }
+      MP_RETURN_IF_ERROR(RenderGpu(cc));
+      return absl::OkStatus();
+    }));
+#else
+    return absl::InternalError("GPU processing is disabled.");
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  } else {
+    MP_RETURN_IF_ERROR(RenderCpu(cc));
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::Close(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  gpu_helper_.RunInGlContext([this] {
+    if (program_) glDeleteProgram(program_);
+    program_ = 0;
+  });
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::RenderCpu(CalculatorContext* cc) {
+  // Setup source images.
+  const auto& current_frame = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>();
+  const cv::Mat current_mat = mediapipe::formats::MatView(&current_frame);
+  RET_CHECK_EQ(current_mat.type(), CV_32FC1)
+      << "Only 1-channel float input image is supported.";
+
+  const auto& previous_frame = cc->Inputs().Tag(kPreviousMaskTag).Get<Image>();
+  const cv::Mat previous_mat = mediapipe::formats::MatView(&previous_frame);
+  RET_CHECK_EQ(previous_mat.type(), current_mat.type())
+      << "Warning: mixing input format types: " << previous_mat.type()
+      << " != " << previous_mat.type();
+
+  RET_CHECK_EQ(current_mat.rows, previous_mat.rows);
+  RET_CHECK_EQ(current_mat.cols, previous_mat.cols);
+
+  // Setup destination image.
+  auto output_frame = std::make_shared<ImageFrame>(
+      current_frame.image_format(), current_mat.cols, current_mat.rows);
+  cv::Mat output_mat = mediapipe::formats::MatView(output_frame.get());
+  output_mat.setTo(cv::Scalar(0));
+
+  // Blending function.
+  const auto blending_fn = [&](const float prev_mask_value,
+                               const float new_mask_value) {
+    /*
+     * Assume p := new_mask_value
+     * H(p) := 1 + (p * log(p) + (1-p) * log(1-p)) / log(2)
+     * uncertainty alpha(p) =
+     *   Clamp(1 - (1 - H(p)) * (1 - H(p)), 0, 1) [squaring the uncertainty]
+     *
+     * The following polynomial approximates uncertainty alpha as a function
+     * of (p + 0.5):
+     */
+    const float c1 = 5.68842;
+    const float c2 = -0.748699;
+    const float c3 = -57.8051;
+    const float c4 = 291.309;
+    const float c5 = -624.717;
+    const float t = new_mask_value - 0.5f;
+    const float x = t * t;
+
+    const float uncertainty =
+        1.0f -
+        std::min(1.0f, x * (c1 + x * (c2 + x * (c3 + x * (c4 + x * c5)))));
+
+    return new_mask_value + (prev_mask_value - new_mask_value) *
+                                (uncertainty * combine_with_previous_ratio_);
+  };
+
+  // Write directly to the first channel of output.
+  for (int i = 0; i < output_mat.rows; ++i) {
+    float* out_ptr = output_mat.ptr<float>(i);
+    const float* curr_ptr = current_mat.ptr<float>(i);
+    const float* prev_ptr = previous_mat.ptr<float>(i);
+    for (int j = 0; j < output_mat.cols; ++j) {
+      const float new_mask_value = curr_ptr[j];
+      const float prev_mask_value = prev_ptr[j];
+      out_ptr[j] = blending_fn(prev_mask_value, new_mask_value);
+    }
+  }
+
+  cc->Outputs()
+      .Tag(kOutputMaskTag)
+      .AddPacket(MakePacket<Image>(output_frame).At(cc->InputTimestamp()));
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::RenderGpu(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  // Setup source textures.
+  const auto& current_frame = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>();
+  RET_CHECK(
+      (current_frame.format() == mediapipe::GpuBufferFormat::kBGRA32 ||
+       current_frame.format() == mediapipe::GpuBufferFormat::kGrayHalf16 ||
+       current_frame.format() == mediapipe::GpuBufferFormat::kGrayFloat32 ||
+       current_frame.format() == mediapipe::GpuBufferFormat::kRGB24))
+      << "Only RGBA, RGB, or 1-channel Float input image supported.";
+
+  auto current_texture = gpu_helper_.CreateSourceTexture(current_frame);
+
+  const auto& previous_frame = cc->Inputs().Tag(kPreviousMaskTag).Get<Image>();
+  if (previous_frame.format() != current_frame.format()) {
+    LOG(ERROR) << "Warning: mixing input format types. ";
+  }
+  auto previous_texture = gpu_helper_.CreateSourceTexture(previous_frame);
+
+  // Setup destination texture.
+  const int width = current_frame.width(), height = current_frame.height();
+  auto output_texture = gpu_helper_.CreateDestinationTexture(
+      width, height, current_frame.format());
+
+  // Process shader.
+  {
+    gpu_helper_.BindFramebuffer(output_texture);
+    glActiveTexture(GL_TEXTURE1);
+    glBindTexture(GL_TEXTURE_2D, current_texture.name());
+    glActiveTexture(GL_TEXTURE2);
+    glBindTexture(GL_TEXTURE_2D, previous_texture.name());
+    GlRender(cc);
+    glActiveTexture(GL_TEXTURE2);
+    glBindTexture(GL_TEXTURE_2D, 0);
+    glActiveTexture(GL_TEXTURE1);
+    glBindTexture(GL_TEXTURE_2D, 0);
+  }
+  glFlush();
+
+  // Send out image as GPU packet.
+  auto output_frame = output_texture.GetFrame<Image>();
+  cc->Outputs()
+      .Tag(kOutputMaskTag)
+      .Add(output_frame.release(), cc->InputTimestamp());
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+void SegmentationSmoothingCalculator::GlRender(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  static const GLfloat square_vertices[] = {
+      -1.0f, -1.0f,  // bottom left
+      1.0f,  -1.0f,  // bottom right
+      -1.0f, 1.0f,   // top left
+      1.0f,  1.0f,   // top right
+  };
+  static const GLfloat texture_vertices[] = {
+      0.0f, 0.0f,  // bottom left
+      1.0f, 0.0f,  // bottom right
+      0.0f, 1.0f,  // top left
+      1.0f, 1.0f,  // top right
+  };
+
+  // program
+  glUseProgram(program_);
+
+  // vertex storage
+  GLuint vbo[2];
+  glGenBuffers(2, vbo);
+  GLuint vao;
+  glGenVertexArrays(1, &vao);
+  glBindVertexArray(vao);
+
+  // vbo 0
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_VERTEX);
+  glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // vbo 1
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // draw
+  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
+
+  // cleanup
+  glDisableVertexAttribArray(ATTRIB_VERTEX);
+  glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glBindBuffer(GL_ARRAY_BUFFER, 0);
+  glBindVertexArray(0);
+  glDeleteVertexArrays(1, &vao);
+  glDeleteBuffers(2, vbo);
+
+#endif  // !MEDIAPIPE_DISABLE_GPU
+}
+
+absl::Status SegmentationSmoothingCalculator::GlSetup(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  const GLint attr_location[NUM_ATTRIBUTES] = {
+      ATTRIB_VERTEX,
+      ATTRIB_TEXTURE_POSITION,
+  };
+  const GLchar* attr_name[NUM_ATTRIBUTES] = {
+      "position",
+      "texture_coordinate",
+  };
+
+  // Shader to blend in previous mask based on computed uncertainty probability.
+  const std::string frag_src =
+      absl::StrCat(std::string(mediapipe::kMediaPipeFragmentShaderPreamble),
+                   R"(
+    DEFAULT_PRECISION(mediump, float)
+
+    #ifdef GL_ES
+    #define fragColor gl_FragColor
+    #else
+    out vec4 fragColor;
+    #endif  // defined(GL_ES);
+
+    in vec2 sample_coordinate;
+    uniform sampler2D current_mask;
+    uniform sampler2D previous_mask;
+    uniform float combine_with_previous_ratio;
+
+    void main() {
+      vec4 current_pix = texture2D(current_mask, sample_coordinate);
+      vec4 previous_pix = texture2D(previous_mask, sample_coordinate);
+      float new_mask_value = current_pix.r;
+      float prev_mask_value = previous_pix.r;
+
+      // Assume p := new_mask_value
+      // H(p) := 1 + (p * log(p) + (1-p) * log(1-p)) / log(2)
+      // uncertainty alpha(p) =
+      //   Clamp(1 - (1 - H(p)) * (1 - H(p)), 0, 1) [squaring the uncertainty]
+      //
+      // The following polynomial approximates uncertainty alpha as a function
+      // of (p + 0.5):
+      const float c1 = 5.68842;
+      const float c2 = -0.748699;
+      const float c3 = -57.8051;
+      const float c4 = 291.309;
+      const float c5 = -624.717;
+      float t = new_mask_value - 0.5;
+      float x = t * t;
+
+      float uncertainty =
+        1.0 - min(1.0, x * (c1 + x * (c2 + x * (c3 + x * (c4 + x * c5)))));
+
+      new_mask_value +=
+        (prev_mask_value - new_mask_value) * (uncertainty * combine_with_previous_ratio);
+
+      fragColor = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
+    }
+  )");
+
+  // Create shader program and set parameters.
+  mediapipe::GlhCreateProgram(mediapipe::kBasicVertexShader, frag_src.c_str(),
+                              NUM_ATTRIBUTES, (const GLchar**)&attr_name[0],
+                              attr_location, &program_);
+  RET_CHECK(program_) << "Problem initializing the program.";
+  glUseProgram(program_);
+  glUniform1i(glGetUniformLocation(program_, "current_mask"), 1);
+  glUniform1i(glGetUniformLocation(program_, "previous_mask"), 2);
+  glUniform1f(glGetUniformLocation(program_, "combine_with_previous_ratio"),
+              combine_with_previous_ratio_);
+
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+}  // namespace mediapipe

mediapipe/calculators/image/segmentation_smoothing_calculator.proto

@@ -0,0 +1,35 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto2";
+
+package mediapipe;
+
+import "mediapipe/framework/calculator.proto";
+
+message SegmentationSmoothingCalculatorOptions {
+  extend CalculatorOptions {
+    optional SegmentationSmoothingCalculatorOptions ext = 377425128;
+  }
+
+  // How much to blend in previous mask, based on a probability estimate.
+  // Range: [0-1]
+  // 0 = Use only current frame (no blending).
+  // 1 = Blend in the previous mask based on uncertainty estimate.
+  //     With ratio at 1, the uncertainty estimate is trusted completely.
+  //     When uncertainty is high, the previous mask is given higher weight.
+  //     Therefore, if both ratio and uncertainty are 1, only old mask is used.
+  //   A pixel is 'uncertain' if its value is close to the middle (0.5 or 127).
+  optional float combine_with_previous_ratio = 1 [default = 0.0];
+}

mediapipe/calculators/image/segmentation_smoothing_calculator_test.cc

@@ -0,0 +1,206 @@
+// Copyright 2018 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <memory>
+
+#include "mediapipe/calculators/image/segmentation_smoothing_calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_runner.h"
+#include "mediapipe/framework/deps/file_path.h"
+#include "mediapipe/framework/formats/image_frame.h"
+#include "mediapipe/framework/formats/image_opencv.h"
+#include "mediapipe/framework/port/gmock.h"
+#include "mediapipe/framework/port/gtest.h"
+#include "mediapipe/framework/port/opencv_imgcodecs_inc.h"
+#include "mediapipe/framework/port/opencv_imgproc_inc.h"
+#include "mediapipe/framework/port/parse_text_proto.h"
+#include "mediapipe/framework/port/status_matchers.h"
+
+namespace mediapipe {
+
+namespace {
+
+// 4x4 VEC32F1, center 2x2 block set at ~250
+const float mask_data[] = {
+    0.00, 0.00, 0.00, 0.00,  //
+    0.00, 0.98, 0.98, 0.00,  //
+    0.00, 0.98, 0.98, 0.00,  //
+    0.00, 0.00, 0.00, 0.00,  //
+};
+
+void RunGraph(Packet curr_packet, Packet prev_packet, bool use_gpu, float ratio,
+              cv::Mat* result) {
+  CalculatorGraphConfig graph_config;
+  if (use_gpu) {
+    graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
+        R"pb(
+          input_stream: "curr_mask"
+          input_stream: "prev_mask"
+          output_stream: "new_mask"
+          node {
+            calculator: "ImageCloneCalculator"
+            input_stream: "curr_mask"
+            output_stream: "curr_mask_gpu"
+            options: {
+              [mediapipe.ImageCloneCalculatorOptions.ext] {
+                output_on_gpu: true
+              }
+            }
+          }
+          node {
+            calculator: "ImageCloneCalculator"
+            input_stream: "prev_mask"
+            output_stream: "prev_mask_gpu"
+            options: {
+              [mediapipe.ImageCloneCalculatorOptions.ext] {
+                output_on_gpu: true
+              }
+            }
+          }
+          node {
+            calculator: "SegmentationSmoothingCalculator"
+            input_stream: "MASK:curr_mask_gpu"
+            input_stream: "MASK_PREVIOUS:prev_mask_gpu"
+            output_stream: "MASK_SMOOTHED:new_mask"
+            node_options {
+              [type.googleapis.com/
+               mediapipe.SegmentationSmoothingCalculatorOptions]: {
+                combine_with_previous_ratio: $0
+              }
+            }
+          }
+        )pb",
+        ratio));
+  } else {
+    graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
+        R"pb(
+          input_stream: "curr_mask"
+          input_stream: "prev_mask"
+          output_stream: "new_mask"
+          node {
+            calculator: "SegmentationSmoothingCalculator"
+            input_stream: "MASK:curr_mask"
+            input_stream: "MASK_PREVIOUS:prev_mask"
+            output_stream: "MASK_SMOOTHED:new_mask"
+            node_options {
+              [type.googleapis.com/
+               mediapipe.SegmentationSmoothingCalculatorOptions]: {
+                combine_with_previous_ratio: $0
+              }
+            }
+          }
+        )pb",
+        ratio));
+  }
+  std::vector<Packet> output_packets;
+  tool::AddVectorSink("new_mask", &graph_config, &output_packets);
+  CalculatorGraph graph(graph_config);
+  MP_ASSERT_OK(graph.StartRun({}));
+
+  MP_ASSERT_OK(
+      graph.AddPacketToInputStream("curr_mask", curr_packet.At(Timestamp(0))));
+  MP_ASSERT_OK(
+      graph.AddPacketToInputStream("prev_mask", prev_packet.At(Timestamp(0))));
+  MP_ASSERT_OK(graph.WaitUntilIdle());
+  ASSERT_EQ(1, output_packets.size());
+
+  Image result_image = output_packets[0].Get<Image>();
+  cv::Mat result_mat = formats::MatView(&result_image);
+  result_mat.copyTo(*result);
+
+  // Fully close graph at end, otherwise calculator+Images are destroyed
+  // after calling WaitUntilDone().
+  MP_ASSERT_OK(graph.CloseInputStream("curr_mask"));
+  MP_ASSERT_OK(graph.CloseInputStream("prev_mask"));
+  MP_ASSERT_OK(graph.WaitUntilDone());
+}
+
+void RunTest(bool use_gpu, float mix_ratio, cv::Mat& test_result) {
+  cv::Mat mask_mat(cv::Size(4, 4), CV_32FC1, const_cast<float*>(mask_data));
+  cv::Mat curr_mat = mask_mat;
+  // 3x3 blur of 250 block produces all pixels '111'.
+  cv::Mat prev_mat;
+  cv::blur(mask_mat, prev_mat, cv::Size(3, 3));
+
+  Packet curr_packet = MakePacket<Image>(std::make_unique<ImageFrame>(
+      ImageFormat::VEC32F1, curr_mat.size().width, curr_mat.size().height));
+  curr_mat.copyTo(formats::MatView(&(curr_packet.Get<Image>())));
+  Packet prev_packet = MakePacket<Image>(std::make_unique<ImageFrame>(
+      ImageFormat::VEC32F1, prev_mat.size().width, prev_mat.size().height));
+  prev_mat.copyTo(formats::MatView(&(prev_packet.Get<Image>())));
+
+  cv::Mat result;
+  RunGraph(curr_packet, prev_packet, use_gpu, mix_ratio, &result);
+
+  ASSERT_EQ(curr_mat.rows, result.rows);
+  ASSERT_EQ(curr_mat.cols, result.cols);
+  ASSERT_EQ(curr_mat.type(), result.type());
+  result.copyTo(test_result);
+
+  if (mix_ratio == 1.0) {
+    for (int i = 0; i < 4; ++i) {
+      for (int j = 0; j < 4; ++j) {
+        float in = curr_mat.at<float>(i, j);
+        float out = result.at<float>(i, j);
+        // Since the input has high value (250), it has low uncertainty.
+        // So the output should have changed lower (towards prev),
+        // but not too much.
+        if (in > 0) EXPECT_NE(in, out);
+        EXPECT_NEAR(in, out, 3.0 / 255.0);
+      }
+    }
+  } else if (mix_ratio == 0.0) {
+    for (int i = 0; i < 4; ++i) {
+      for (int j = 0; j < 4; ++j) {
+        float in = curr_mat.at<float>(i, j);
+        float out = result.at<float>(i, j);
+        EXPECT_EQ(in, out);  // Output should match current.
+      }
+    }
+  } else {
+    LOG(ERROR) << "invalid ratio";
+  }
+}
+
+TEST(SegmentationSmoothingCalculatorTest, TestSmoothing) {
+  bool use_gpu;
+  float mix_ratio;
+
+  use_gpu = false;
+  mix_ratio = 0.0;
+  cv::Mat cpu_0;
+  RunTest(use_gpu, mix_ratio, cpu_0);
+
+  use_gpu = false;
+  mix_ratio = 1.0;
+  cv::Mat cpu_1;
+  RunTest(use_gpu, mix_ratio, cpu_1);
+
+  use_gpu = true;
+  mix_ratio = 1.0;
+  cv::Mat gpu_1;
+  RunTest(use_gpu, mix_ratio, gpu_1);
+
+  // CPU & GPU should match.
+  for (int i = 0; i < 4; ++i) {
+    for (int j = 0; j < 4; ++j) {
+      float gpu = gpu_1.at<float>(i, j);
+      float cpu = cpu_1.at<float>(i, j);
+      EXPECT_EQ(cpu, gpu);
+    }
+  }
+}
+
+}  // namespace
+}  // namespace mediapipe

mediapipe/calculators/tensor/BUILD

@@ -109,6 +109,8 @@ cc_library(
         "//mediapipe/gpu:MPPMetalUtil",
         "//mediapipe/gpu:gpu_buffer",
         "//mediapipe/objc:mediapipe_framework_ios",
+        "//mediapipe/util/tflite:config",
+        "@com_google_absl//absl/memory",
         "@org_tensorflow//tensorflow/lite/delegates/gpu:metal_delegate",
         "@org_tensorflow//tensorflow/lite/delegates/gpu:metal_delegate_internal",
         "@org_tensorflow//tensorflow/lite/delegates/gpu/common:shape",
@@ -478,7 +480,6 @@ cc_library(
     deps = [
         ":image_to_tensor_calculator_cc_proto",
         ":image_to_tensor_converter",
-        ":image_to_tensor_converter_opencv",
         ":image_to_tensor_utils",
         "//mediapipe/framework/api2:node",
         "//mediapipe/framework/formats:image",
@@ -494,6 +495,9 @@ cc_library(
     ] + select({
         "//mediapipe/gpu:disable_gpu": [],
         "//conditions:default": [":image_to_tensor_calculator_gpu_deps"],
+    }) + select({
+        "//mediapipe/framework/port:disable_opencv": [],
+        "//conditions:default": [":image_to_tensor_converter_opencv"],
     }),
     alwayslink = 1,
 )

mediapipe/calculators/tensor/image_to_tensor_calculator.cc

@@ -18,7 +18,6 @@
 
 #include "mediapipe/calculators/tensor/image_to_tensor_calculator.pb.h"
 #include "mediapipe/calculators/tensor/image_to_tensor_converter.h"
-#include "mediapipe/calculators/tensor/image_to_tensor_converter_opencv.h"
 #include "mediapipe/calculators/tensor/image_to_tensor_utils.h"
 #include "mediapipe/framework/api2/node.h"
 #include "mediapipe/framework/calculator_framework.h"
@@ -33,6 +32,10 @@
 #include "mediapipe/framework/port/statusor.h"
 #include "mediapipe/gpu/gpu_origin.pb.h"
 
+#if !MEDIAPIPE_DISABLE_OPENCV
+#include "mediapipe/calculators/tensor/image_to_tensor_converter_opencv.h"
+#endif
+
 #if !MEDIAPIPE_DISABLE_GPU
 #include "mediapipe/gpu/gpu_buffer.h"
 
@@ -301,8 +304,13 @@ class ImageToTensorCalculator : public Node {
       }
     } else {
       if (!cpu_converter_) {
+#if !MEDIAPIPE_DISABLE_OPENCV
         ASSIGN_OR_RETURN(cpu_converter_,
                          CreateOpenCvConverter(cc, GetBorderMode()));
+#else
+        LOG(FATAL) << "Cannot create image to tensor opencv converter since "
+                      "MEDIAPIPE_DISABLE_OPENCV is defined.";
+#endif  // !MEDIAPIPE_DISABLE_OPENCV
       }
     }
     return absl::OkStatus();

mediapipe/calculators/tensor/image_to_tensor_converter_gl_buffer.cc

@@ -312,7 +312,7 @@ class GlProcessor : public ImageToTensorConverter {
       return absl::OkStatus();
     }));
 
-    return std::move(tensor);
+    return tensor;
   }
 
   ~GlProcessor() override {
@@ -338,8 +338,7 @@ CreateImageToGlBufferTensorConverter(CalculatorContext* cc,
   auto result = absl::make_unique<GlProcessor>();
   MP_RETURN_IF_ERROR(result->Init(cc, input_starts_at_bottom, border_mode));
 
-  // Simply "return std::move(result)" failed to build on macOS with bazel.
+  return result;
-  return std::unique_ptr<ImageToTensorConverter>(std::move(result));
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/image_to_tensor_converter_gl_texture.cc

@@ -334,9 +334,7 @@ CreateImageToGlTextureTensorConverter(CalculatorContext* cc,
                                       BorderMode border_mode) {
   auto result = absl::make_unique<GlProcessor>();
   MP_RETURN_IF_ERROR(result->Init(cc, input_starts_at_bottom, border_mode));
-
+  return result;
-  // Simply "return std::move(result)" failed to build on macOS with bazel.
-  return std::unique_ptr<ImageToTensorConverter>(std::move(result));
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/image_to_tensor_converter_metal.cc

@@ -383,7 +383,7 @@ class MetalProcessor : public ImageToTensorConverter {
           tflite::gpu::HW(output_dims.height, output_dims.width),
           command_buffer, buffer_view.buffer()));
       [command_buffer commit];
-      return std::move(tensor);
+      return tensor;
     }
   }
 
@@ -399,8 +399,7 @@ absl::StatusOr<std::unique_ptr<ImageToTensorConverter>> CreateMetalConverter(
   auto result = absl::make_unique<MetalProcessor>();
   MP_RETURN_IF_ERROR(result->Init(cc, border_mode));
 
-  // Simply "return std::move(result)" failed to build on macOS with bazel.
+  return result;
-  return std::unique_ptr<ImageToTensorConverter>(std::move(result));
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/image_to_tensor_converter_opencv.cc

@@ -103,7 +103,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
         GetValueRangeTransformation(kInputImageRangeMin, kInputImageRangeMax,
                                     range_min, range_max));
     transformed.convertTo(dst, CV_32FC3, transform.scale, transform.offset);
-    return std::move(tensor);
+    return tensor;
   }
 
  private:
@@ -114,10 +114,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
 
 absl::StatusOr<std::unique_ptr<ImageToTensorConverter>> CreateOpenCvConverter(
     CalculatorContext* cc, BorderMode border_mode) {
-  // Simply "return absl::make_unique<OpenCvProcessor>()" failed to build on
+  return absl::make_unique<OpenCvProcessor>(border_mode);
-  // macOS with bazel.
-  return std::unique_ptr<ImageToTensorConverter>(
-      absl::make_unique<OpenCvProcessor>(border_mode));
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/inference_calculator_cpu.cc

@@ -35,20 +35,28 @@ namespace api2 {
 
 namespace {
 
+int GetXnnpackDefaultNumThreads() {
+#if defined(MEDIAPIPE_ANDROID) || defined(MEDIAPIPE_IOS) || \
+    defined(__EMSCRIPTEN_PTHREADS__)
+  constexpr int kMinNumThreadsByDefault = 1;
+  constexpr int kMaxNumThreadsByDefault = 4;
+  return std::clamp(NumCPUCores() / 2, kMinNumThreadsByDefault,
+                    kMaxNumThreadsByDefault);
+#else
+  return 1;
+#endif  // MEDIAPIPE_ANDROID || MEDIAPIPE_IOS || __EMSCRIPTEN_PTHREADS__
+}
+
 // Returns number of threads to configure XNNPACK delegate with.
-// (Equal to user provided value if specified.  Otherwise, it returns number of
+// Returns user provided value if specified. Otherwise, tries to choose optimal
-// high cores (hard-coded to 1 for Emscripten without Threads extension))
+// number of threads depending on the device.
 int GetXnnpackNumThreads(const mediapipe::InferenceCalculatorOptions& opts) {
   static constexpr int kDefaultNumThreads = -1;
   if (opts.has_delegate() && opts.delegate().has_xnnpack() &&
       opts.delegate().xnnpack().num_threads() != kDefaultNumThreads) {
     return opts.delegate().xnnpack().num_threads();
   }
-#if !defined(__EMSCRIPTEN__) || defined(__EMSCRIPTEN_PTHREADS__)
+  return GetXnnpackDefaultNumThreads();
-  return InferHigherCoreIds().size();
-#else
-  return 1;
-#endif  // !__EMSCRIPTEN__ || __EMSCRIPTEN_PTHREADS__
 }
 
 }  // namespace

mediapipe/calculators/tensor/inference_calculator_gl.cc

@@ -269,8 +269,8 @@ absl::Status InferenceCalculatorGlImpl::InitTFLiteGPURunner(
       break;
     }
   }
-  MP_RETURN_IF_ERROR(
+  MP_RETURN_IF_ERROR(tflite_gpu_runner_->InitializeWithModel(
-      tflite_gpu_runner_->InitializeWithModel(model, op_resolver));
+      model, op_resolver, /*allow_quant_ops=*/true));
 
   // Create and bind OpenGL buffers for outputs.
   // The buffers are created once and their ids are passed to calculator outputs

mediapipe/calculators/tensor/inference_calculator_metal.cc

@@ -226,6 +226,10 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
 
   // Configure and create the delegate.
   TFLGpuDelegateOptions options;
+  // `enable_quantization` enables the run of sparse models i.e. the models with
+  // DENSIFY op preceding DEQUINTIZE op. Both ops get removed from the execution
+  // graph after the tensor of the weights is read.
+  options.enable_quantization = true;
   options.allow_precision_loss = allow_precision_loss_;
   options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeDoNotWait;
   delegate_ =

mediapipe/calculators/tensor/tensors_to_segmentation_calculator.cc

@@ -763,9 +763,13 @@ out vec4 fragColor;
 #endif  // defined(GL_ES);
 
 void main() {
-
+#ifdef FLIP_Y_COORD
-  vec4 input_value = texture2D(input_texture, sample_coordinate);
+  float y_coord = 1.0 - sample_coordinate.y;
-  vec2 gid = sample_coordinate;
+#else
+  float y_coord = sample_coordinate.y;
+#endif  // defined(FLIP_Y_COORD)
+  vec2 adjusted_coordinate = vec2(sample_coordinate.x, y_coord);
+  vec4 input_value = texture2D(input_texture, adjusted_coordinate);
 
   // Run activation function.
   // One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
@@ -787,13 +791,6 @@ void main() {
   float new_mask_value = input_value.r;
 #endif // FN_NONE
 
-#ifdef FLIP_Y_COORD
-  float y_coord = 1.0 - gid.y;
-#else
-  float y_coord = gid.y;
-#endif  // defined(FLIP_Y_COORD)
-  vec2 output_coordinate = vec2(gid.x, y_coord);
-
   vec4 out_value = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
   fragColor = out_value;
 })";

mediapipe/calculators/tensor/testdata/face_detection_test.pbtxt

@@ -4,7 +4,7 @@ output_stream: "detections"
 
 # Subgraph that detects faces.
 node {
-  calculator: "FaceDetectionFrontCpu"
+  calculator: "FaceDetectionShortRangeCpu"
   input_stream: "IMAGE:image"
   output_stream: "DETECTIONS:detections"
 }

mediapipe/calculators/tensorflow/tensorflow_inference_calculator.cc

@@ -490,7 +490,7 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
               << keyed_tensors.first;
         }
       } else {
-        // Pad by replicating the first tens  or, then ignore the values.
+        // Pad by replicating the first tensor, then ignore the values.
         keyed_tensors.second.resize(options_.batch_size());
         std::fill(keyed_tensors.second.begin() +
                       inference_state->batch_timestamps_.size(),

mediapipe/calculators/tflite/tflite_inference_calculator.cc

@@ -128,9 +128,23 @@ struct GPUData {
 }  // namespace
 #endif  // MEDIAPIPE_TFLITE_GPU_SUPPORTED
 
+namespace {
+
+int GetXnnpackDefaultNumThreads() {
+#if defined(MEDIAPIPE_ANDROID) || defined(MEDIAPIPE_IOS) || \
+    defined(__EMSCRIPTEN_PTHREADS__)
+  constexpr int kMinNumThreadsByDefault = 1;
+  constexpr int kMaxNumThreadsByDefault = 4;
+  return std::clamp(NumCPUCores() / 2, kMinNumThreadsByDefault,
+                    kMaxNumThreadsByDefault);
+#else
+  return 1;
+#endif  // MEDIAPIPE_ANDROID || MEDIAPIPE_IOS || __EMSCRIPTEN_PTHREADS__
+}
+
 // Returns number of threads to configure XNNPACK delegate with.
-// (Equal to user provided value if specified.  Otherwise, it returns number of
+// Returns user provided value if specified. Otherwise, tries to choose optimal
-// high cores (hard-coded to 1 for Emscripten without Threads extension))
+// number of threads depending on the device.
 int GetXnnpackNumThreads(
     const mediapipe::TfLiteInferenceCalculatorOptions& opts) {
   static constexpr int kDefaultNumThreads = -1;
@@ -138,13 +152,11 @@ int GetXnnpackNumThreads(
       opts.delegate().xnnpack().num_threads() != kDefaultNumThreads) {
     return opts.delegate().xnnpack().num_threads();
   }
-#if !defined(__EMSCRIPTEN__) || defined(__EMSCRIPTEN_PTHREADS__)
+  return GetXnnpackDefaultNumThreads();
-  return InferHigherCoreIds().size();
-#else
-  return 1;
-#endif  // !__EMSCRIPTEN__ || __EMSCRIPTEN_PTHREADS__
 }
 
+}  // namespace
+
 // Calculator Header Section
 
 // Runs inference on the provided input TFLite tensors and TFLite model.
@@ -737,8 +749,8 @@ absl::Status TfLiteInferenceCalculator::InitTFLiteGPURunner(
       break;
     }
   }
-  MP_RETURN_IF_ERROR(
+  MP_RETURN_IF_ERROR(tflite_gpu_runner_->InitializeWithModel(
-      tflite_gpu_runner_->InitializeWithModel(model, *op_resolver_ptr));
+      model, *op_resolver_ptr, /*allow_quant_ops=*/true));
 
   // Allocate interpreter memory for cpu output.
   if (!gpu_output_) {
@@ -969,6 +981,10 @@ absl::Status TfLiteInferenceCalculator::LoadDelegate(CalculatorContext* cc) {
   const int kHalfSize = 2;  // sizeof(half)
   // Configure and create the delegate.
   TFLGpuDelegateOptions options;
+  // `enable_quantization` enables the run of sparse models i.e. the models with
+  // DENSIFY op preceding DEQUINTIZE op. Both ops get removed from the execution
+  // graph after the tensor of the weights is read.
+  options.enable_quantization = true;
   options.allow_precision_loss = allow_precision_loss_;
   options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeActive;
   if (!delegate_)

mediapipe/calculators/util/BUILD

@@ -840,6 +840,20 @@ cc_test(
     ],
 )
 
+cc_library(
+    name = "world_landmark_projection_calculator",
+    srcs = ["world_landmark_projection_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/formats:rect_cc_proto",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
+    ],
+    alwayslink = 1,
+)
+
 mediapipe_proto_library(
     name = "landmarks_smoothing_calculator_proto",
     srcs = ["landmarks_smoothing_calculator.proto"],
@@ -894,6 +908,31 @@ cc_library(
     alwayslink = 1,
 )
 
+mediapipe_proto_library(
+    name = "visibility_copy_calculator_proto",
+    srcs = ["visibility_copy_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
+cc_library(
+    name = "visibility_copy_calculator",
+    srcs = ["visibility_copy_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":visibility_copy_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:timestamp",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/port:ret_check",
+        "@com_google_absl//absl/algorithm:container",
+    ],
+    alwayslink = 1,
+)
+
 cc_library(
     name = "landmarks_to_floats_calculator",
     srcs = ["landmarks_to_floats_calculator.cc"],

mediapipe/calculators/util/annotation_overlay_calculator.cc

@@ -272,6 +272,15 @@ absl::Status AnnotationOverlayCalculator::Open(CalculatorContext* cc) {
 }
 
 absl::Status AnnotationOverlayCalculator::Process(CalculatorContext* cc) {
+  if (cc->Inputs().HasTag(kGpuBufferTag) &&
+      cc->Inputs().Tag(kGpuBufferTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+  if (cc->Inputs().HasTag(kImageFrameTag) &&
+      cc->Inputs().Tag(kImageFrameTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+
   // Initialize render target, drawn with OpenCV.
   std::unique_ptr<cv::Mat> image_mat;
   ImageFormat::Format target_format;

mediapipe/calculators/util/detections_to_rects_calculator.cc

@@ -203,6 +203,9 @@ absl::Status DetectionsToRectsCalculator::Process(CalculatorContext* cc) {
       cc->Inputs().Tag(kDetectionsTag).IsEmpty()) {
     return absl::OkStatus();
   }
+  if (rotate_ && !HasTagValue(cc, kImageSizeTag)) {
+    return absl::OkStatus();
+  }
 
   std::vector<Detection> detections;
   if (cc->Inputs().HasTag(kDetectionTag)) {

mediapipe/calculators/util/filter_collection_calculator.cc

@@ -32,11 +32,15 @@ typedef FilterCollectionCalculator<std::vector<::mediapipe::NormalizedRect>>
     FilterNormalizedRectCollectionCalculator;
 REGISTER_CALCULATOR(FilterNormalizedRectCollectionCalculator);
 
-typedef FilterCollectionCalculator<
+typedef FilterCollectionCalculator<std::vector<::mediapipe::LandmarkList>>
-    std::vector<::mediapipe::NormalizedLandmarkList>>
     FilterLandmarkListCollectionCalculator;
 REGISTER_CALCULATOR(FilterLandmarkListCollectionCalculator);
 
+typedef FilterCollectionCalculator<
+    std::vector<::mediapipe::NormalizedLandmarkList>>
+    FilterNormalizedLandmarkListCollectionCalculator;
+REGISTER_CALCULATOR(FilterNormalizedLandmarkListCollectionCalculator);
+
 typedef FilterCollectionCalculator<std::vector<::mediapipe::ClassificationList>>
     FilterClassificationListCollectionCalculator;
 REGISTER_CALCULATOR(FilterClassificationListCollectionCalculator);

mediapipe/calculators/util/rect_transformation_calculator.cc

@@ -130,8 +130,8 @@ absl::Status RectTransformationCalculator::Process(CalculatorContext* cc) {
     }
     cc->Outputs().Index(0).Add(output_rects.release(), cc->InputTimestamp());
   }
-  if (cc->Inputs().HasTag(kNormRectTag) &&
+  if (HasTagValue(cc->Inputs(), kNormRectTag) &&
-      !cc->Inputs().Tag(kNormRectTag).IsEmpty()) {
+      HasTagValue(cc->Inputs(), kImageSizeTag)) {
     auto rect = cc->Inputs().Tag(kNormRectTag).Get<NormalizedRect>();
     const auto& image_size =
         cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
@@ -139,8 +139,8 @@ absl::Status RectTransformationCalculator::Process(CalculatorContext* cc) {
     cc->Outputs().Index(0).AddPacket(
         MakePacket<NormalizedRect>(rect).At(cc->InputTimestamp()));
   }
-  if (cc->Inputs().HasTag(kNormRectsTag) &&
+  if (HasTagValue(cc->Inputs(), kNormRectsTag) &&
-      !cc->Inputs().Tag(kNormRectsTag).IsEmpty()) {
+      HasTagValue(cc->Inputs(), kImageSizeTag)) {
     auto rects =
         cc->Inputs().Tag(kNormRectsTag).Get<std::vector<NormalizedRect>>();
     const auto& image_size =

mediapipe/calculators/video/motion_analysis_calculator.cc

@@ -549,7 +549,7 @@ absl::Status MotionAnalysisCalculator::Process(CalculatorContext* cc) {
     timestamp_buffer_.push_back(timestamp);
     ++frame_idx_;
 
-    VLOG_EVERY_N(0, 100) << "Analyzed frame " << frame_idx_;
+    VLOG_EVERY_N(1, 100) << "Analyzed frame " << frame_idx_;
 
     // Buffer input frames only if visualization is requested.
     if (visualize_output_ || video_output_) {

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu/BUILD

@@ -37,7 +37,7 @@ android_binary(
     srcs = glob(["*.java"]),
     assets = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_cpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectionfullrangegpu/BUILD

@@ -0,0 +1,60 @@
+# Copyright 2019 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:private"])
+
+cc_binary(
+    name = "libmediapipe_jni.so",
+    linkshared = 1,
+    linkstatic = 1,
+    deps = [
+        "//mediapipe/graphs/face_detection:face_detection_full_range_mobile_gpu_deps",
+        "//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
+    ],
+)
+
+cc_library(
+    name = "mediapipe_jni_lib",
+    srcs = [":libmediapipe_jni.so"],
+    alwayslink = 1,
+)
+
+android_binary(
+    name = "facedetectionfullrangegpu",
+    srcs = glob(["*.java"]),
+    assets = [
+        "//mediapipe/graphs/face_detection:face_detection_full_range_mobile_gpu.binarypb",
+        "//mediapipe/modules/face_detection:face_detection_full_range_sparse.tflite",
+    ],
+    assets_dir = "",
+    manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
+    manifest_values = {
+        "applicationId": "com.google.mediapipe.apps.facedetectionfullrangegpu",
+        "appName": "Face Detection Full-range (GPU)",
+        "mainActivity": "com.google.mediapipe.apps.basic.MainActivity",
+        "cameraFacingFront": "False",
+        "binaryGraphName": "face_detection_full_range_mobile_gpu.binarypb",
+        "inputVideoStreamName": "input_video",
+        "outputVideoStreamName": "output_video",
+        "flipFramesVertically": "True",
+        "converterNumBuffers": "2",
+    },
+    multidex = "native",
+    deps = [
+        ":mediapipe_jni_lib",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
+    ],
+)

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD

@@ -37,7 +37,7 @@ android_binary(
     srcs = glob(["*.java"]),
     assets = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/faceeffect/BUILD

@@ -42,7 +42,7 @@ android_binary(
         "//mediapipe/graphs/face_effect/data:glasses.binarypb",
         "//mediapipe/graphs/face_effect/data:glasses.pngblob",
         "//mediapipe/graphs/face_effect:face_effect_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/BUILD

@@ -38,7 +38,7 @@ android_binary(
     assets = [
         "//mediapipe/graphs/face_mesh:face_mesh_mobile_gpu.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/holistictrackinggpu/BUILD

@@ -37,7 +37,7 @@ android_binary(
     srcs = glob(["*.java"]),
     assets = [
         "//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
         "//mediapipe/modules/hand_landmark:hand_landmark.tflite",
         "//mediapipe/modules/hand_landmark:handedness.txt",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu/BUILD

@@ -39,7 +39,7 @@ android_binary(
         "//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
         "//mediapipe/modules/iris_landmark:iris_landmark.tflite",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator.cc

@@ -37,6 +37,13 @@ constexpr char kFirstCropRect[] = "FIRST_CROP_RECT";
 // (configured through option us_to_first_rect). If provided, a non-zero integer
 // will allow the animated zoom to be used when the first detections arrive.
 constexpr char kAnimateZoom[] = "ANIMATE_ZOOM";
+// Can be used to control the maximum zoom; note that it is re-evaluated only
+// upon change of input resolution. A value of 100 disables zooming and is the
+// smallest allowed value. A value of 200 allows zooming such that a pixel of
+// the input may cover up to four times its original area. Note that
+// max_zoom_value_deg from options is always respected; MAX_ZOOM_PCT can only be
+// used to limit zooming further.
+constexpr char kMaxZoomFactorPercent[] = "MAX_ZOOM_FACTOR_PCT";
 // Field-of-view (degrees) of the camera's x-axis (width).
 // TODO: Parameterize FOV based on camera specs.
 constexpr float kFieldOfView = 60;
@@ -75,11 +82,16 @@ class ContentZoomingCalculator : public CalculatorBase {
                               int frame_height);
   // Saves state to a state-cache, if provided.
   absl::Status SaveState(mediapipe::CalculatorContext* cc) const;
+  // Returns the factor for maximum zoom based on options and the
+  // kMaxZoomFactorPercent input (if present).
+  double GetMaxZoomFactor(mediapipe::CalculatorContext* cc) const;
   // Initializes the calculator for the given frame size, creating path solvers
   // and resetting history like last measured values.
-  absl::Status InitializeState(int frame_width, int frame_height);
+  absl::Status InitializeState(mediapipe::CalculatorContext* cc,
+                               int frame_width, int frame_height);
   // Adjusts state to work with an updated frame size.
-  absl::Status UpdateForResolutionChange(int frame_width, int frame_height);
+  absl::Status UpdateForResolutionChange(mediapipe::CalculatorContext* cc,
+                                         int frame_width, int frame_height);
   // Returns true if we are animating to the first rect.
   bool IsAnimatingToFirstRect(const Timestamp& timestamp) const;
   // Builds the output rectangle when animating to the first rect.
@@ -136,6 +148,9 @@ absl::Status ContentZoomingCalculator::GetContract(
     return mediapipe::UnknownErrorBuilder(MEDIAPIPE_LOC)
            << "Input VIDEO or VIDEO_SIZE must be provided.";
   }
+  if (cc->Inputs().HasTag(kMaxZoomFactorPercent)) {
+    cc->Inputs().Tag(kMaxZoomFactorPercent).Set<int>();
+  }
   if (cc->Inputs().HasTag(kSalientRegions)) {
     cc->Inputs().Tag(kSalientRegions).Set<DetectionSet>();
   }
@@ -330,7 +345,7 @@ absl::Status ContentZoomingCalculator::MaybeLoadState(
           ? cc->InputSidePackets().Tag(kStateCache).Get<StateCacheType*>()
           : nullptr;
   if (!state_cache || !state_cache->has_value()) {
-    return InitializeState(frame_width, frame_height);
+    return InitializeState(cc, frame_width, frame_height);
   }
 
   const ContentZoomingCalculatorState& state = state_cache->value();
@@ -350,7 +365,7 @@ absl::Status ContentZoomingCalculator::MaybeLoadState(
   last_measured_y_offset_ = state.last_measured_y_offset;
   MP_RETURN_IF_ERROR(UpdateAspectAndMax());
 
-  return UpdateForResolutionChange(frame_width, frame_height);
+  return UpdateForResolutionChange(cc, frame_width, frame_height);
 }
 
 absl::Status ContentZoomingCalculator::SaveState(
@@ -379,8 +394,20 @@ absl::Status ContentZoomingCalculator::SaveState(
   return absl::OkStatus();
 }
 
-absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
+double ContentZoomingCalculator::GetMaxZoomFactor(
-                                                       int frame_height) {
+    mediapipe::CalculatorContext* cc) const {
+  double max_zoom_value =
+      options_.max_zoom_value_deg() / static_cast<double>(kFieldOfView);
+  if (cc->Inputs().HasTag(kMaxZoomFactorPercent)) {
+    const double factor = std::max(
+        1.0, cc->Inputs().Tag(kMaxZoomFactorPercent).Get<int>() / 100.0);
+    max_zoom_value = std::max(max_zoom_value, 1.0 / factor);
+  }
+  return max_zoom_value;
+}
+
+absl::Status ContentZoomingCalculator::InitializeState(
+    mediapipe::CalculatorContext* cc, int frame_width, int frame_height) {
   frame_width_ = frame_width;
   frame_height_ = frame_height;
   path_solver_pan_ = std::make_unique<KinematicPathSolver>(
@@ -390,8 +417,7 @@ absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
       options_.kinematic_options_tilt(), 0, frame_height_,
       static_cast<float>(frame_height_) / kFieldOfView);
   MP_RETURN_IF_ERROR(UpdateAspectAndMax());
-  int min_zoom_size = frame_height_ * (options_.max_zoom_value_deg() /
+  int min_zoom_size = frame_height_ * GetMaxZoomFactor(cc);
-                                       static_cast<double>(kFieldOfView));
   path_solver_zoom_ = std::make_unique<KinematicPathSolver>(
       options_.kinematic_options_zoom(), min_zoom_size,
       max_frame_value_ * frame_height_,
@@ -405,7 +431,7 @@ absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
 }
 
 absl::Status ContentZoomingCalculator::UpdateForResolutionChange(
-    int frame_width, int frame_height) {
+    mediapipe::CalculatorContext* cc, int frame_width, int frame_height) {
   // Update state for change in input resolution.
   if (frame_width_ != frame_width || frame_height_ != frame_height) {
     double width_scale = frame_width / static_cast<double>(frame_width_);
@@ -419,8 +445,7 @@ absl::Status ContentZoomingCalculator::UpdateForResolutionChange(
     MP_RETURN_IF_ERROR(path_solver_pan_->UpdateMinMaxLocation(0, frame_width_));
     MP_RETURN_IF_ERROR(
         path_solver_tilt_->UpdateMinMaxLocation(0, frame_height_));
-    int min_zoom_size = frame_height_ * (options_.max_zoom_value_deg() /
+    int min_zoom_size = frame_height_ * GetMaxZoomFactor(cc);
-                                         static_cast<double>(kFieldOfView));
     MP_RETURN_IF_ERROR(path_solver_zoom_->UpdateMinMaxLocation(
         min_zoom_size, max_frame_value_ * frame_height_));
     MP_RETURN_IF_ERROR(path_solver_zoom_->UpdatePixelsPerDegree(
@@ -493,7 +518,8 @@ absl::Status ContentZoomingCalculator::Process(
     MP_RETURN_IF_ERROR(MaybeLoadState(cc, frame_width, frame_height));
     initialized_ = !options_.is_stateless();
   } else {
-    MP_RETURN_IF_ERROR(UpdateForResolutionChange(frame_width, frame_height));
+    MP_RETURN_IF_ERROR(
+        UpdateForResolutionChange(cc, frame_width, frame_height));
   }
 
   bool only_required_found = false;
@@ -634,14 +660,18 @@ absl::Status ContentZoomingCalculator::Process(
   // Prevent box from extending beyond the image after camera smoothing.
   if (path_offset_y - ceil(path_height / 2.0) < 0) {
     path_offset_y = ceil(path_height / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_tilt_->SetState(path_offset_y));
   } else if (path_offset_y + ceil(path_height / 2.0) > frame_height_) {
     path_offset_y = frame_height_ - ceil(path_height / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_tilt_->SetState(path_offset_y));
   }
 
   if (path_offset_x - ceil(path_width / 2.0) < 0) {
     path_offset_x = ceil(path_width / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_pan_->SetState(path_offset_x));
   } else if (path_offset_x + ceil(path_width / 2.0) > frame_width_) {
     path_offset_x = frame_width_ - ceil(path_width / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_pan_->SetState(path_offset_x));
   }
 
   // Convert to top/bottom borders to remove.

mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator_test.cc

@@ -150,6 +150,29 @@ const char kConfigE[] = R"(
     }
     )";
 
+const char kConfigF[] = R"(
+    calculator: "ContentZoomingCalculator"
+    input_stream: "VIDEO_SIZE:size"
+    input_stream: "DETECTIONS:detections"
+    input_stream: "MAX_ZOOM_FACTOR_PCT:max_zoom_factor_pct"
+    output_stream: "CROP_RECT:rect"
+    output_stream: "FIRST_CROP_RECT:first_rect"
+    options: {
+      [mediapipe.autoflip.ContentZoomingCalculatorOptions.ext]: {
+        max_zoom_value_deg: 0
+        kinematic_options_zoom {
+          min_motion_to_reframe: 1.2
+        }
+        kinematic_options_tilt {
+          min_motion_to_reframe: 1.2
+        }
+        kinematic_options_pan {
+          min_motion_to_reframe: 1.2
+        }
+      }
+    }
+    )";
+
 void CheckBorder(const StaticFeatures& static_features, int width, int height,
                  int top_border, int bottom_border) {
   ASSERT_EQ(2, static_features.border().size());
@@ -170,6 +193,7 @@ void CheckBorder(const StaticFeatures& static_features, int width, int height,
 
 struct AddDetectionFlags {
   std::optional<bool> animated_zoom;
+  std::optional<int> max_zoom_factor_percent;
 };
 
 void AddDetectionFrameSize(const cv::Rect_<float>& position, const int64 time,
@@ -211,6 +235,14 @@ void AddDetectionFrameSize(const cv::Rect_<float>& position, const int64 time,
             mediapipe::MakePacket<bool>(flags.animated_zoom.value())
                 .At(Timestamp(time)));
   }
+
+  if (flags.max_zoom_factor_percent.has_value()) {
+    runner->MutableInputs()
+        ->Tag("MAX_ZOOM_FACTOR_PCT")
+        .packets.push_back(
+            mediapipe::MakePacket<int>(flags.max_zoom_factor_percent.value())
+                .At(Timestamp(time)));
+  }
 }
 
 void AddDetection(const cv::Rect_<float>& position, const int64 time,
@@ -259,6 +291,7 @@ TEST(ContentZoomingCalculatorTest, ZoomTest) {
   CheckBorder(static_features, 1000, 1000, 495, 395);
 }
 
+#if 0
 TEST(ContentZoomingCalculatorTest, ZoomTestFullPTZ) {
   auto runner = ::absl::make_unique<CalculatorRunner>(
       ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD));
@@ -694,8 +727,8 @@ TEST(ContentZoomingCalculatorTest, ResolutionChangeZoomingWithCache) {
     auto runner = ::absl::make_unique<CalculatorRunner>(config);
     runner->MutableSidePackets()->Tag("STATE_CACHE") = MakePacket<
         mediapipe::autoflip::ContentZoomingCalculatorStateCacheType*>(&cache);
-    AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1000, 1000,
+    AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1000,
-                          runner.get());
+                          1000, runner.get());
     AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 2000000, 500, 500,
                           runner.get());
     MP_ASSERT_OK(runner->Run());
@@ -719,6 +752,36 @@ TEST(ContentZoomingCalculatorTest, MaxZoomValue) {
   CheckCropRect(500, 500, 916, 916, 0,
                 runner->Outputs().Tag("CROP_RECT").packets);
 }
+#endif
+
+TEST(ContentZoomingCalculatorTest, MaxZoomValueOverride) {
+  auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigF);
+  auto* options = config.mutable_options()->MutableExtension(
+      ContentZoomingCalculatorOptions::ext);
+  options->set_max_zoom_value_deg(30);
+  auto runner = ::absl::make_unique<CalculatorRunner>(config);
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 0, 640, 480,
+                        runner.get(), {.max_zoom_factor_percent = 133});
+  // Change resolution and allow more zoom, and give time to use the new limit
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1280, 720,
+                        runner.get(), {.max_zoom_factor_percent = 166});
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 2000000, 1280, 720,
+                        runner.get(), {.max_zoom_factor_percent = 166});
+  // Switch back to a smaller resolution with a more limited zoom
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 3000000, 640, 480,
+                        runner.get(), {.max_zoom_factor_percent = 133});
+  MP_ASSERT_OK(runner->Run());
+  // Max. 133% zoomed in means min. (100/133) ~ 75% of height left: ~360
+  // Max. 166% zoomed in means min. (100/166) ~ 60% of height left: ~430
+  CheckCropRect(320, 240, 480, 360, 0,
+                runner->Outputs().Tag("CROP_RECT").packets);
+  CheckCropRect(640, 360, 769, 433, 2,
+                runner->Outputs().Tag("CROP_RECT").packets);
+  CheckCropRect(320, 240, 480, 360, 3,
+                runner->Outputs().Tag("CROP_RECT").packets);
+}
+
+#if 0
 TEST(ContentZoomingCalculatorTest, MaxZoomOutValue) {
   auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD);
   auto* options = config.mutable_options()->MutableExtension(
@@ -906,6 +969,7 @@ TEST(ContentZoomingCalculatorTest, ProvidesConstantFirstRect) {
     EXPECT_EQ(first_rect.height(), rect.height());
   }
 }
+#endif
 
 }  // namespace
 }  // namespace autoflip

mediapipe/examples/desktop/autoflip/calculators/face_box_adjuster_calculator.proto

@@ -68,5 +68,10 @@ message FaceBoxAdjusterCalculatorOptions {
 
   // The max amount of time to use an old eye distance when the face look angle
   // is unstable.
-  optional int32 max_facesize_history_us = 9 [default = 8000000];
+  optional int32 max_facesize_history_us = 9 [default = 300000000];
+
+  // Scale factor of face width to shift based on pan look angle.
+  optional float pan_position_shift_scale = 15 [default = 0.5];
+  // Scale factor of face height to shift based on tilt look angle.
+  optional float tilt_position_shift_scale = 16 [default = 0.5];
 }

mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver.cc

@@ -154,10 +154,18 @@ absl::Status KinematicPathSolver::AddObservation(int position,
 
   // Time and position updates.
   double delta_t = (time_us - current_time_) / 1000000.0;
+  // Time since last state/prediction update, smoothed by
+  // mean_period_update_rate.
+  if (mean_delta_t_ < 0) {
+    mean_delta_t_ = delta_t;
+  } else {
+    mean_delta_t_ = mean_delta_t_ * (1 - options_.mean_period_update_rate()) +
+                    delta_t * options_.mean_period_update_rate();
+  }
 
   // Observed velocity and then weighted update of this velocity.
   double observed_velocity = delta_degs / delta_t;
-  double update_rate = std::min(delta_t / options_.update_rate_seconds(),
+  double update_rate = std::min(mean_delta_t_ / options_.update_rate_seconds(),
                                 options_.max_update_rate());
   double updated_velocity = current_velocity_deg_per_s_ * (1 - update_rate) +
                             observed_velocity * update_rate;
@@ -174,16 +182,6 @@ absl::Status KinematicPathSolver::UpdatePrediction(const int64 time_us) {
   RET_CHECK(current_time_ < time_us)
       << "Prediction time added before a prior observation or prediction.";
 
-  // Time since last state/prediction update, smoothed by
-  // mean_period_update_rate.
-  double delta_t = (time_us - current_time_) / 1000000.0;
-  if (mean_delta_t_ < 0) {
-    mean_delta_t_ = delta_t;
-  } else {
-    mean_delta_t_ = mean_delta_t_ * (1 - options_.mean_period_update_rate()) +
-                    delta_t * options_.mean_period_update_rate();
-  }
-
   // Position update limited by min/max.
   double update_position_px =
       current_position_px_ +
@@ -211,6 +209,12 @@ absl::Status KinematicPathSolver::GetState(int* position) {
   return absl::OkStatus();
 }
 
+absl::Status KinematicPathSolver::SetState(const int position) {
+  RET_CHECK(initialized_) << "SetState called before first observation added.";
+  current_position_px_ = position;
+  return absl::OkStatus();
+}
+
 absl::Status KinematicPathSolver::GetTargetPosition(int* target_position) {
   RET_CHECK(initialized_)
       << "GetTargetPosition called before first observation added.";

mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver.h

@@ -48,6 +48,8 @@ class KinematicPathSolver {
   absl::Status UpdatePrediction(const int64 time_us);
   // Get the state at a time.
   absl::Status GetState(int* position);
+  // Overwrite the current state value.
+  absl::Status SetState(const int position);
   // Update PixelPerDegree value.
   absl::Status UpdatePixelsPerDegree(const float pixels_per_degree);
   // Provide the current target position of the reframe action.

mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver_test.cc

@@ -337,6 +337,61 @@ TEST(KinematicPathSolverTest, PassDegPerPxChange) {
   EXPECT_EQ(state, 516);
 }
 
+TEST(KinematicPathSolverTest, NoTimestampSmoothing) {
+  KinematicOptions options;
+  options.set_min_motion_to_reframe(1.0);
+  options.set_update_rate(1.0);
+  options.set_max_velocity(6);
+  options.set_mean_period_update_rate(1.0);
+  KinematicPathSolver solver(options, 0, 1000, 1000.0 / kWidthFieldOfView);
+  int state;
+  MP_ASSERT_OK(solver.AddObservation(500, 0));
+  MP_ASSERT_OK(solver.AddObservation(1000, 1000000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 600);
+  MP_ASSERT_OK(solver.AddObservation(1000, 2200000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 720);
+}
+
+TEST(KinematicPathSolverTest, TimestampSmoothing) {
+  KinematicOptions options;
+  options.set_min_motion_to_reframe(1.0);
+  options.set_update_rate(1.0);
+  options.set_max_velocity(6);
+  options.set_mean_period_update_rate(0.05);
+  KinematicPathSolver solver(options, 0, 1000, 1000.0 / kWidthFieldOfView);
+  int state;
+  MP_ASSERT_OK(solver.AddObservation(500, 0));
+  MP_ASSERT_OK(solver.AddObservation(1000, 1000000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 600);
+  MP_ASSERT_OK(solver.AddObservation(1000, 2200000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 701);
+}
+
+TEST(KinematicPathSolverTest, PassSetPosition) {
+  KinematicOptions options;
+  // Set min motion to 2deg
+  options.set_min_motion_to_reframe(1.0);
+  options.set_update_rate_seconds(.0000001);
+  options.set_max_update_rate(1.0);
+  options.set_max_velocity(18);
+  // Set degrees / pixel to 8.3
+  KinematicPathSolver solver(options, 0, 500, 500.0 / kWidthFieldOfView);
+  int state;
+  MP_ASSERT_OK(solver.AddObservation(400, kMicroSecInSec * 0));
+  // Move target by 10px / 8.3 = 1.2deg
+  MP_ASSERT_OK(solver.AddObservation(410, kMicroSecInSec * 1));
+  MP_ASSERT_OK(solver.GetState(&state));
+  // Expect cam to move.
+  EXPECT_EQ(state, 410);
+  MP_ASSERT_OK(solver.SetState(400));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 400);
+}
+
 }  // namespace
 }  // namespace autoflip
 }  // namespace mediapipe

mediapipe/examples/desktop/autoflip/subgraph/face_detection_subgraph.pbtxt

@@ -38,7 +38,7 @@ node {
   output_stream: "TENSORS:detection_tensors"
   options: {
     [mediapipe.TfLiteInferenceCalculatorOptions.ext] {
-      model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
+      model_path: "mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite"
     }
   }
 }

mediapipe/examples/desktop/face_detection/BUILD

@@ -16,6 +16,14 @@ licenses(["notice"])
 
 package(default_visibility = ["//mediapipe/examples:__subpackages__"])
 
+cc_binary(
+    name = "face_detection_full_range_cpu",
+    deps = [
+        "//mediapipe/examples/desktop:demo_run_graph_main",
+        "//mediapipe/graphs/face_detection:face_detection_full_range_desktop_live_deps",
+    ],
+)
+
 cc_binary(
     name = "face_detection_cpu",
     deps = [

mediapipe/examples/ios/facedetectioncpu/BUILD

@@ -55,7 +55,7 @@ objc_library(
     name = "FaceDetectionCpuAppLibrary",
     data = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_cpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     deps = [
         "//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",

mediapipe/examples/ios/facedetectiongpu/BUILD

@@ -55,7 +55,7 @@ objc_library(
     name = "FaceDetectionGpuAppLibrary",
     data = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     deps = [
         "//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",

mediapipe/examples/ios/faceeffect/BUILD

@@ -66,7 +66,7 @@ objc_library(
         "//mediapipe/graphs/face_effect/data:facepaint.pngblob",
         "//mediapipe/graphs/face_effect/data:glasses.binarypb",
         "//mediapipe/graphs/face_effect/data:glasses.pngblob",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
@@ -109,7 +109,7 @@ objc_library(
         "//mediapipe/graphs/face_effect/data:facepaint.pngblob",
         "//mediapipe/graphs/face_effect/data:glasses.binarypb",
         "//mediapipe/graphs/face_effect/data:glasses.pngblob",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
     ],

mediapipe/examples/ios/facemeshgpu/BUILD

@@ -62,7 +62,7 @@ objc_library(
     copts = ["-std=c++17"],
     data = [
         "//mediapipe/graphs/face_mesh:face_mesh_mobile_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
     ],
     deps = [

mediapipe/examples/ios/holistictrackinggpu/BUILD

@@ -55,7 +55,7 @@ objc_library(
     name = "HolisticTrackingGpuAppLibrary",
     data = [
         "//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
         "//mediapipe/modules/hand_landmark:hand_landmark.tflite",
         "//mediapipe/modules/hand_landmark:handedness.txt",

mediapipe/examples/ios/iristrackinggpu/BUILD

@@ -62,7 +62,7 @@ objc_library(
     copts = ["-std=c++17"],
     data = [
         "//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
         "//mediapipe/modules/iris_landmark:iris_landmark.tflite",
     ],

mediapipe/framework/BUILD

@@ -953,6 +953,9 @@ cc_library(
     }) + select({
         "//conditions:default": [],
         "//mediapipe/gpu:disable_gpu": ["MEDIAPIPE_DISABLE_GPU=1"],
+    }) + select({
+        "//conditions:default": [],
+        "//mediapipe/framework/port:disable_opencv": ["MEDIAPIPE_DISABLE_OPENCV=1"],
     }) + select({
         "//conditions:default": [],
         "//mediapipe/framework:disable_rtti_and_exceptions": [

mediapipe/framework/api2/builder.h

@@ -17,6 +17,14 @@ namespace mediapipe {
 namespace api2 {
 namespace builder {
 
+// Workaround for static_assert(false). Example:
+//   dependent_false<T>::value returns false.
+// For more information, see:
+// https://en.cppreference.com/w/cpp/language/if#Constexpr_If
+// TODO: migrate to a common utility when available.
+template <class T>
+struct dependent_false : std::false_type {};
+
 template <typename T>
 T& GetWithAutoGrow(std::vector<std::unique_ptr<T>>* vecp, int index) {
   auto& vec = *vecp;
@@ -209,6 +217,21 @@ class NodeBase {
     return SideDestination<true>(&in_sides_[tag]);
   }
 
+  template <typename B, typename T, bool kIsOptional, bool kIsMultiple>
+  auto operator[](const PortCommon<B, T, kIsOptional, kIsMultiple>& port) {
+    if constexpr (std::is_same_v<B, OutputBase>) {
+      return Source<kIsMultiple, T>(&out_streams_[port.Tag()]);
+    } else if constexpr (std::is_same_v<B, InputBase>) {
+      return Destination<kIsMultiple, T>(&in_streams_[port.Tag()]);
+    } else if constexpr (std::is_same_v<B, SideOutputBase>) {
+      return SideSource<kIsMultiple, T>(&out_sides_[port.Tag()]);
+    } else if constexpr (std::is_same_v<B, SideInputBase>) {
+      return SideDestination<kIsMultiple, T>(&in_sides_[port.Tag()]);
+    } else {
+      static_assert(dependent_false<B>::value, "Type not supported.");
+    }
+  }
+
   // Convenience methods for accessing purely index-based ports.
   Source<false> Out(int index) { return Out("")[index]; }
 
@@ -429,6 +452,24 @@ class Graph {
     return Dst(&graph_boundary_.in_sides_[graph_output.Tag()]);
   }
 
+  template <typename B, typename T, bool kIsOptional, bool kIsMultiple>
+  auto operator[](const PortCommon<B, T, kIsOptional, kIsMultiple>& port) {
+    if constexpr (std::is_same_v<B, OutputBase>) {
+      return Destination<kIsMultiple, T>(
+          &graph_boundary_.in_streams_[port.Tag()]);
+    } else if constexpr (std::is_same_v<B, InputBase>) {
+      return Source<kIsMultiple, T>(&graph_boundary_.out_streams_[port.Tag()]);
+    } else if constexpr (std::is_same_v<B, SideOutputBase>) {
+      return SideDestination<kIsMultiple, T>(
+          &graph_boundary_.in_sides_[port.Tag()]);
+    } else if constexpr (std::is_same_v<B, SideInputBase>) {
+      return SideSource<kIsMultiple, T>(
+          &graph_boundary_.out_sides_[port.Tag()]);
+    } else {
+      static_assert(dependent_false<B>::value, "Type not supported.");
+    }
+  }
+
   // Returns the graph config. This can be used to instantiate and run the
   // graph.
   CalculatorGraphConfig GetConfig() {

mediapipe/framework/api2/builder_test.cc

@@ -138,6 +138,35 @@ TEST(BuilderTest, TypedMultiple) {
   EXPECT_THAT(graph.GetConfig(), EqualsProto(expected));
 }
 
+TEST(BuilderTest, TypedByPorts) {
+  builder::Graph graph;
+  auto& foo = graph.AddNode<Foo>();
+  auto& adder = graph.AddNode<FloatAdder>();
+
+  graph[FooBar1::kIn].SetName("base") >> foo[Foo::kBase];
+  foo[Foo::kOut] >> adder[FloatAdder::kIn][0];
+  foo[Foo::kOut] >> adder[FloatAdder::kIn][1];
+  adder[FloatAdder::kOut].SetName("out") >> graph[FooBar1::kOut];
+
+  CalculatorGraphConfig expected =
+      mediapipe::ParseTextProtoOrDie<CalculatorGraphConfig>(R"pb(
+        input_stream: "IN:base"
+        output_stream: "OUT:out"
+        node {
+          calculator: "Foo"
+          input_stream: "BASE:base"
+          output_stream: "OUT:__stream_0"
+        }
+        node {
+          calculator: "FloatAdder"
+          input_stream: "IN:0:__stream_0"
+          input_stream: "IN:1:__stream_0"
+          output_stream: "OUT:out"
+        }
+      )pb");
+  EXPECT_THAT(graph.GetConfig(), EqualsProto(expected));
+}
+
 TEST(BuilderTest, PacketGenerator) {
   builder::Graph graph;
   auto& generator = graph.AddPacketGenerator("FloatGenerator");

mediapipe/framework/deps/mathutil_unittest.cc

@@ -262,7 +262,7 @@ TEST(MathUtil, IntRound) {
 
   // A double-precision number has a 53-bit mantissa (52 fraction bits),
   // so the following value can be represented exactly.
-  int64 value64 = GG_ULONGLONG(0x1234567890abcd00);
+  int64 value64 = static_cast<int64_t>(0x1234567890abcd00);
   EXPECT_EQ(mediapipe::MathUtil::Round<int64>(static_cast<double>(value64)),
             value64);
 }
@@ -369,7 +369,7 @@ class SafeCastTester {
     if (sizeof(FloatIn) >= 64) {
       // A double-precision number has a 53-bit mantissa (52 fraction bits),
       // so the following value can be represented exactly by a double.
-      int64 value64 = GG_ULONGLONG(0x1234567890abcd00);
+      int64 value64 = static_cast<int64_t>(0x1234567890abcd00);
       const IntOut expected =
           (sizeof(IntOut) >= 64) ? static_cast<IntOut>(value64) : imax;
       EXPECT_EQ(
@@ -565,7 +565,7 @@ TEST(MathUtil, SafeCast) {
       -12345);
   EXPECT_EQ(mediapipe::MathUtil::SafeCast<int>(1E47), 2147483647);
   EXPECT_EQ(mediapipe::MathUtil::SafeCast<int>(-1E47),
-            GG_LONGLONG(-2147483648));
+            static_cast<int64_t>(-2147483648));
 }
 
 template <class FloatIn, class IntOut>
@@ -682,7 +682,7 @@ class SafeRoundTester {
     if (sizeof(FloatIn) >= 64) {
       // A double-precision number has a 53-bit mantissa (52 fraction bits),
       // so the following value can be represented exactly by a double.
-      int64 value64 = GG_ULONGLONG(0x1234567890abcd00);
+      int64 value64 = static_cast<int64_t>(0x1234567890abcd00);
       const IntOut expected =
           (sizeof(IntOut) >= 64) ? static_cast<IntOut>(value64) : imax;
       EXPECT_EQ(
@@ -873,7 +873,7 @@ TEST(MathUtil, SafeRound) {
       -12345);
   EXPECT_EQ(mediapipe::MathUtil::SafeRound<int>(1E47), 2147483647);
   EXPECT_EQ(mediapipe::MathUtil::SafeRound<int>(-1E47),
-            GG_LONGLONG(-2147483648));
+            static_cast<int64_t>(-2147483648));
 }
 
 }  // namespace

mediapipe/framework/formats/BUILD

@@ -186,6 +186,7 @@ cc_library(
         "//conditions:default": [
             "//mediapipe/framework/port:opencv_imgproc",
         ],
+        "//mediapipe/framework/port:disable_opencv": [],
     }) + select({
         "//conditions:default": [
         ],

mediapipe/framework/formats/image.cc

@@ -76,10 +76,7 @@ bool Image::ConvertToGpu() const {
   gpu_buffer_ = mediapipe::GpuBuffer(std::move(buffer));
 #else
   // GlCalculatorHelperImpl::MakeGlTextureBuffer (CreateSourceTexture)
-  auto buffer = mediapipe::GlTextureBuffer::Create(
+  auto buffer = mediapipe::GlTextureBuffer::Create(*image_frame_);
-      image_frame_->Width(), image_frame_->Height(),
-      mediapipe::GpuBufferFormatForImageFormat(image_frame_->Format()),
-      image_frame_->PixelData());
   glBindTexture(GL_TEXTURE_2D, buffer->name());
   // See GlCalculatorHelperImpl::SetStandardTextureParams
   glTexParameteri(buffer->target(), GL_TEXTURE_MIN_FILTER, GL_LINEAR);

mediapipe/framework/formats/location.h

@@ -32,7 +32,12 @@
 
 // clang-format off
 #if !defined(LOCATION_OPENCV)
-#  define LOCATION_OPENCV 1
+#  if !MEDIAPIPE_DISABLE_OPENCV && \
+      (!defined(MEDIAPIPE_MOBILE) || defined(MEDIAPIPE_ANDROID_OPENCV))
+#    define LOCATION_OPENCV 1
+#  else
+#    define LOCATION_OPENCV 0
+#  endif
 #endif
 
 #if LOCATION_OPENCV

mediapipe/framework/mediapipe_cc_test.bzl

@@ -8,6 +8,7 @@ def mediapipe_cc_test(
         data = [],
         deps = [],
         size = None,
+        tags = [],
         timeout = None,
         additional_deps = DEFAULT_ADDITIONAL_TEST_DEPS,
         **kwargs):

mediapipe/framework/port/BUILD

@@ -158,12 +158,12 @@ cc_library(
     hdrs = [
         "gmock.h",
         "gtest.h",
+        "gtest-spi.h",
         "status_matchers.h",
     ],
     visibility = ["//visibility:public"],
     deps = [
         ":status_matchers",
-        "//mediapipe/framework:port",
         "@com_google_googletest//:gtest",
     ],
 )
@@ -174,12 +174,12 @@ cc_library(
     hdrs = [
         "gmock.h",
         "gtest.h",
+        "gtest-spi.h",
         "status_matchers.h",
     ],
     visibility = ["//visibility:public"],
     deps = [
         ":status_matchers",
-        "//mediapipe/framework:port",
         "//mediapipe/framework/deps:status_matchers",
         "@com_google_googletest//:gtest_main",
     ],
@@ -217,6 +217,16 @@ cc_library(
     deps = ["//mediapipe/framework/deps:numbers"],
 )
 
+# Disabling opencv when defining MEDIAPIPE_DISABLE_OPENCV to 1 in the bazel command.
+# Note that this only applies to a select few calculators/framework components currently.
+config_setting(
+    name = "disable_opencv",
+    define_values = {
+        "MEDIAPIPE_DISABLE_OPENCV": "1",
+    },
+    visibility = ["//visibility:public"],
+)
+
 cc_library(
     name = "opencv_core",
     hdrs = ["opencv_core_inc.h"],

mediapipe/framework/port/gtest-spi.h

@@ -0,0 +1,20 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef MEDIAPIPE_PORT_GTEST_SPI_H_
+#define MEDIAPIPE_PORT_GTEST_SPI_H_
+
+#include "gtest/gtest-spi.h"
+
+#endif  // MEDIAPIPE_PORT_GTEST_SPI_H_

mediapipe/framework/tool/BUILD

@@ -641,14 +641,20 @@ cc_library(
         "//mediapipe/framework:calculator_cc_proto",
         "//mediapipe/framework/deps:file_path",
         "//mediapipe/framework/deps:no_destructor",
+        "//mediapipe/framework/formats:image_format_cc_proto",
         "//mediapipe/framework/formats:image_frame",
         "//mediapipe/framework/port:advanced_proto",
         "//mediapipe/framework/port:file_helpers",
-        "//mediapipe/framework/port:gtest",
         "//mediapipe/framework/port:logging",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
         "@com_google_absl//absl/container:flat_hash_set",
         "@com_google_absl//absl/memory",
+        "@com_google_absl//absl/status",
+        "@com_google_absl//absl/status:statusor",
         "@com_google_absl//absl/strings",
+        "@stblib//:stb_image",
+        "@stblib//:stb_image_write",
     ],
 )
 

mediapipe/framework/tool/test_util.cc

@@ -18,18 +18,27 @@
 #include <unistd.h>
 
 #include <memory>
+#include <string>
 
 #include "absl/container/flat_hash_set.h"
 #include "absl/memory/memory.h"
+#include "absl/status/status.h"
 #include "absl/strings/match.h"
+#include "absl/strings/str_cat.h"
+#include "absl/strings/str_join.h"
 #include "absl/strings/substitute.h"
 #include "mediapipe/framework/calculator.pb.h"
 #include "mediapipe/framework/deps/file_path.h"
 #include "mediapipe/framework/deps/no_destructor.h"
+#include "mediapipe/framework/formats/image_format.pb.h"
 #include "mediapipe/framework/port/advanced_proto_inc.h"
 #include "mediapipe/framework/port/file_helpers.h"
 #include "mediapipe/framework/port/logging.h"
 #include "mediapipe/framework/port/proto_ns.h"
+#include "mediapipe/framework/port/ret_check.h"
+#include "mediapipe/framework/port/status_macros.h"
+#include "stb_image.h"
+#include "stb_image_write.h"
 
 namespace mediapipe {
 
@@ -43,15 +52,14 @@ bool EqualWithTolerance(const T value1, const T value2, const T max_diff) {
 }
 
 template <typename T>
-bool CompareDiff(const ImageFrame& image1, const ImageFrame& image2,
+absl::Status CompareDiff(const ImageFrame& image1, const ImageFrame& image2,
-                 const T max_color_diff, const T max_alpha_diff,
+                         const T max_color_diff, const T max_alpha_diff,
-                 const float max_avg_diff, std::string* error_message) {
+                         const float max_avg_diff,
+                         std::unique_ptr<ImageFrame>& diff_image) {
   // Verify image byte depth matches expected byte depth.
   CHECK_EQ(sizeof(T), image1.ByteDepth());
   CHECK_EQ(sizeof(T), image2.ByteDepth());
 
-  const bool return_error = error_message != nullptr;
-
   const int width = image1.Width();
   const int height = image1.Height();
   const int channels1 = image1.NumberOfChannels();
@@ -68,57 +76,64 @@ bool CompareDiff(const ImageFrame& image1, const ImageFrame& image2,
   const int width_padding2 =
       image2.WidthStep() / image2.ByteDepth() - width * channels2;
 
+  diff_image = std::make_unique<ImageFrame>(image1.Format(), width, height);
+  T* pixel_diff = reinterpret_cast<T*>(diff_image->MutablePixelData());
+  const int width_padding_diff =
+      diff_image->WidthStep() / diff_image->ByteDepth() - width * channels1;
+
   float avg_diff = 0;
-  uint diff_count = 0;
+  uint total_count = 0;
+  int different_color_components = 0;
+  float max_color_diff_found = 0;
+  int different_alpha_components = 0;
+  float max_alpha_diff_found = 0;
   for (int row = 0; row < height; ++row) {
     for (int col = 0; col < width; ++col) {
       for (int channel = 0; channel < num_channels; ++channel) {
         // Check local difference.
-        const T max_diff = channel < 3 ? max_color_diff : max_alpha_diff;
         const T value1 = pixel1[channel];
         const T value2 = pixel2[channel];
-        if (!EqualWithTolerance<T>(value1, value2, max_diff)) {
-          // We cast uint8 to int using this type (and leave other values as-is)
-          // to avoid printing as a single char.
-          using TypeToPrint =
-              typename std::conditional<std::is_same<T, uint8>::value, int,
-                                        T>::type;
-          std::string error = absl::Substitute(
-              "images differ: row = $0 col = $1 channel = $2 : pixel1 = $3, "
-              "pixel2 = $4",
-              row, col, channel, static_cast<TypeToPrint>(value1),
-              static_cast<TypeToPrint>(value2));
-          if (return_error) {
-            *error_message = error;
-          } else {
-            LOG(ERROR) << error;
-          }
-          return false;
-        }
-        // Check global average difference.
         const float diff =
             std::abs(static_cast<float>(value1) - static_cast<float>(value2));
-        avg_diff += (diff - avg_diff) / ++diff_count;
+        if (channel < 3) {
+          different_color_components += diff > max_color_diff;
+          max_color_diff_found = std::max(max_color_diff_found, diff);
+          pixel_diff[channel] = diff;
+        } else {
+          different_alpha_components += diff > max_alpha_diff;
+          max_alpha_diff_found = std::max(max_alpha_diff_found, diff);
+          pixel_diff[channel] = 255;  // opaque to see color difference
+        }
+        // Check global average difference.
+        avg_diff += (diff - avg_diff) / ++total_count;
       }
       pixel1 += channels1;
       pixel2 += channels2;
+      pixel_diff += channels1;
     }
     pixel1 += width_padding1;
     pixel2 += width_padding2;
+    pixel_diff += width_padding_diff;
   }
 
-  if (avg_diff > max_avg_diff) {
+  std::vector<std::string> errors;
-    std::string error =
+  if (different_color_components)
-        absl::Substitute("images differ: avg pixel error = $0", avg_diff);
+    errors.push_back(absl::Substitute(
-    if (return_error) {
+        "$0 color components differences above limit of $1, max found was $2",
-      *error_message = error;
+        different_color_components, max_color_diff, max_color_diff_found));
-    } else {
+  if (different_alpha_components)
-      LOG(ERROR) << error;
+    errors.push_back(absl::Substitute(
-    }
+        "$0 alpha components differences above limit of $1, max found was $2",
-    return false;
+        different_alpha_components, max_alpha_diff, max_alpha_diff_found));
-  }
+  if (avg_diff > max_avg_diff)
+    errors.push_back(
+        absl::Substitute("the average component difference is $0 (limit: $1)",
+                         avg_diff, max_avg_diff));
 
-  return true;
+  if (!errors.empty())
+    return absl::InternalError(
+        absl::StrCat("images differ: ", absl::StrJoin(errors, "; ")));
+  return absl::OkStatus();
 }
 
 #if defined(__linux__)
@@ -134,77 +149,32 @@ std::string GetBinaryDirectory() {
 
 }  // namespace
 
-bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
+absl::Status CompareImageFrames(const ImageFrame& image1,
-                        const float max_color_diff, const float max_alpha_diff,
+                                const ImageFrame& image2,
-                        const float max_avg_diff, std::string* error_message) {
+                                const float max_color_diff,
-  const bool return_error = error_message != nullptr;
+                                const float max_alpha_diff,
-
+                                const float max_avg_diff,
-  auto IsSupportedImageFormatComparison = [](const ImageFrame& image1,
+                                std::unique_ptr<ImageFrame>& diff_image) {
-                                             const ImageFrame& image2) {
+  auto IsSupportedImageFormatComparison = [](ImageFormat::Format one,
-    // Pairs of non-equal image formats that can be compared against each other.
+                                             ImageFormat::Format two) {
-    static const mediapipe::NoDestructor<absl::flat_hash_set<
+    auto both = std::minmax(one, two);
-        std::pair<ImageFormat::Format, ImageFormat::Format>>>
+    return one == two ||
-        kCompatibleImageFormats({
+           both == std::minmax(ImageFormat::SRGB, ImageFormat::SRGBA) ||
-            {ImageFormat::SRGB, ImageFormat::SRGBA},
+           both == std::minmax(ImageFormat::SRGB48, ImageFormat::SRGBA64);
-            {ImageFormat::SRGB48, ImageFormat::SRGBA64},
-        });
-
-    auto* compatible_image_formats = kCompatibleImageFormats.get();
-
-    return image1.Format() == image2.Format() ||
-           compatible_image_formats->contains(
-               {image1.Format(), image2.Format()}) ||
-           compatible_image_formats->contains(
-               {image2.Format(), image1.Format()});
   };
 
-  if (!IsSupportedImageFormatComparison(image1, image2)) {
+  RET_CHECK(IsSupportedImageFormatComparison(image1.Format(), image2.Format()))
-    std::string error = absl::Substitute(
+      << "unsupported image format comparison; image1 = " << image1.Format()
-        "unsupported image format comparison; image1 = $0, image2 = $1",
+      << ", image2 = " << image2.Format();
-        image1.Format(), image2.Format());
-    if (return_error) {
-      *error_message = error;
-    } else {
-      LOG(ERROR) << error;
-    }
-    return false;
-  }
 
-  if (image1.Width() != image2.Width()) {
+  // Cannot use RET_CHECK_EQ because pair is not printable.
-    std::string error =
+  RET_CHECK(std::make_pair(image1.Width(), image1.Height()) ==
-        absl::Substitute("image width mismatch: image1 = $0, image2 = $1",
+            std::make_pair(image2.Width(), image2.Height()))
-                         image1.Width(), image2.Width());
+      << "image size mismatch: " << image1.Width() << "x" << image1.Height()
-    if (return_error) {
+      << " != " << image2.Width() << "x" << image2.Height();
-      *error_message = error;
-    } else {
-      LOG(ERROR) << error;
-    }
-    return false;
-  }
 
-  if (image1.Height() != image2.Height()) {
+  RET_CHECK_EQ(image1.ByteDepth(), image2.ByteDepth())
-    std::string error =
+      << "image byte depth mismatch";
-        absl::Substitute("image height mismatch: image1 = $0, image2 = $1",
-                         image1.Height(), image2.Height());
-    if (return_error) {
-      *error_message = error;
-    } else {
-      LOG(ERROR) << error;
-    }
-    return false;
-  }
-
-  if (image1.ByteDepth() != image2.ByteDepth()) {
-    std::string error =
-        absl::Substitute("image byte depth mismatch: image1 = $0, image2 = $1",
-                         image1.ByteDepth(), image2.ByteDepth());
-    if (return_error) {
-      *error_message = error;
-    } else {
-      LOG(ERROR) << error;
-    }
-    return false;
-  }
 
   switch (image1.Format()) {
     case ImageFormat::GRAY8:
@@ -212,45 +182,87 @@ bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
     case ImageFormat::SRGBA:
     case ImageFormat::LAB8:
       return CompareDiff<uint8>(image1, image2, max_color_diff, max_alpha_diff,
-                                max_avg_diff, error_message);
+                                max_avg_diff, diff_image);
     case ImageFormat::GRAY16:
     case ImageFormat::SRGB48:
     case ImageFormat::SRGBA64:
       return CompareDiff<uint16>(image1, image2, max_color_diff, max_alpha_diff,
-                                 max_avg_diff, error_message);
+                                 max_avg_diff, diff_image);
     case ImageFormat::VEC32F1:
     case ImageFormat::VEC32F2:
       return CompareDiff<float>(image1, image2, max_color_diff, max_alpha_diff,
-                                max_avg_diff, error_message);
+                                max_avg_diff, diff_image);
     default:
       LOG(FATAL) << ImageFrame::InvalidFormatString(image1.Format());
   }
 }
 
+bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
+                        const float max_color_diff, const float max_alpha_diff,
+                        const float max_avg_diff, std::string* error_message) {
+  std::unique_ptr<ImageFrame> diff_image;
+  auto status = CompareImageFrames(image1, image2, max_color_diff,
+                                   max_alpha_diff, max_avg_diff, diff_image);
+  if (status.ok()) return true;
+  if (error_message) *error_message = std::string(status.message());
+  return false;
+}
+
 std::string GetTestRootDir() {
-#if defined(__ANDROID__)
+  return file::JoinPath(std::getenv("TEST_SRCDIR"), "mediapipe");
-  char path[1024];
+}
-  char* ptr = getcwd(path, sizeof(path));
+
-  CHECK_EQ(ptr, path);
+std::string GetTestOutputsDir() {
-  return path;
+  const char* output_dir = getenv("TEST_UNDECLARED_OUTPUTS_DIR");
-#else
+  if (!output_dir) {
-  return ::mediapipe::file::JoinPath(std::getenv("TEST_SRCDIR"), "mediapipe");
+    output_dir = "/tmp";
-#endif  // defined(__ANDROID__)
+  }
+  return output_dir;
 }
 
 std::string GetTestDataDir(const std::string& package_base_path) {
-#if defined(__ANDROID__)
+  return file::JoinPath(GetTestRootDir(), package_base_path, "testdata/");
-  std::string data_dir = GetTestRootDir();
+}
-  std::string binary_dir = GetBinaryDirectory();
+
-  // In Mobile Harness, the cwd is "/" and the run dir is "/data/local/tmp".
+std::string GetTestFilePath(absl::string_view relative_path) {
-  if (data_dir == "/" && absl::StartsWith(binary_dir, "/data")) {
+  return file::JoinPath(GetTestRootDir(), relative_path);
-    data_dir = binary_dir;
+}
+
+absl::StatusOr<std::unique_ptr<ImageFrame>> LoadTestImage(
+    absl::string_view path, ImageFormat::Format format) {
+  std::string encoded;
+  MP_RETURN_IF_ERROR(mediapipe::file::GetContents(path, &encoded));
+
+  // stbi_load determines the output pixel format based on the desired channels.
+  // 0 means "use whatever's in the file".
+  int desired_channels = format == ImageFormat::UNKNOWN ? 0
+                         : format == ImageFormat::SRGBA ? 4
+                         : format == ImageFormat::SRGB  ? 3
+                         : format == ImageFormat::GRAY8 ? 1
+                                                        : -1;
+  RET_CHECK(desired_channels >= 0)
+      << "unsupported output format requested: " << format;
+
+  int width, height, channels_in_file;
+  auto data = stbi_load_from_memory(reinterpret_cast<stbi_uc*>(encoded.data()),
+                                    encoded.size(), &width, &height,
+                                    &channels_in_file, desired_channels);
+  RET_CHECK(data) << "failed to decode image data from: " << path;
+
+  // If we didn't specify a desired format, it will be determined by what the
+  // file contains.
+  int output_channels = desired_channels ? desired_channels : channels_in_file;
+  if (format == ImageFormat::UNKNOWN) {
+    format = output_channels == 4   ? ImageFormat::SRGBA
+             : output_channels == 3 ? ImageFormat::SRGB
+             : output_channels == 1 ? ImageFormat::GRAY8
+                                    : ImageFormat::UNKNOWN;
+    RET_CHECK(format != ImageFormat::UNKNOWN)
+        << "unsupported number of channels: " << output_channels;
   }
-  return ::mediapipe::file::JoinPath(data_dir, package_base_path, "testdata/");
+
-#else
+  return absl::make_unique<ImageFrame>(
-  return ::mediapipe::file::JoinPath(GetTestRootDir(), package_base_path,
+      format, width, height, width * output_channels, data, stbi_image_free);
-                                     "testdata/");
-#endif  // defined(__APPLE__)
 }
 
 std::unique_ptr<ImageFrame> LoadTestPng(const std::string& path,

mediapipe/framework/tool/test_util.h

@@ -15,6 +15,7 @@
 #ifndef MEDIAPIPE_FRAMEWORK_TEST_UTIL_H_
 #define MEDIAPIPE_FRAMEWORK_TEST_UTIL_H_
 
+#include "absl/status/statusor.h"
 #include "mediapipe/framework/calculator.pb.h"
 #include "mediapipe/framework/formats/image_frame.h"
 
@@ -35,14 +36,29 @@ using mediapipe::CalculatorGraphConfig;
 // Note: Although max_color_diff and max_alpha_diff are floats, all uint8/uint16
 // values are exactly representable. (2^24 + 1 is the first non-representable
 // positive integral value.)
+absl::Status CompareImageFrames(const ImageFrame& image1,
+                                const ImageFrame& image2,
+                                const float max_color_diff,
+                                const float max_alpha_diff,
+                                const float max_avg_diff,
+                                std::unique_ptr<ImageFrame>& diff_image);
+
 bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
                         const float max_color_diff, const float max_alpha_diff,
                         const float max_avg_diff = 1.0,
                         std::string* error_message = nullptr);
 
-// Returns the absolute path to the directory that contains test source code.
+// Returns the absolute path to the directory that contains test source code
+// (TEST_SRCDIR).
 std::string GetTestRootDir();
 
+// Returns the absolute path to a directory where tests can write outputs to
+// be sent to bazel (TEST_UNDECLARED_OUTPUTS_DIR or a fallback).
+std::string GetTestOutputsDir();
+
+// Returns the absolute path to a file within TEST_SRCDIR.
+std::string GetTestFilePath(absl::string_view relative_path);
+
 // Returns the absolute path to the contents of the package's "testdata"
 // directory.
 // This handles the different paths where test data ends up when using
@@ -52,6 +68,10 @@ std::string GetTestDataDir(const std::string& package_base_path);
 // Loads a binary graph from path. Returns true iff successful.
 bool LoadTestGraph(CalculatorGraphConfig* proto, const std::string& path);
 
+// Loads an image from path.
+absl::StatusOr<std::unique_ptr<ImageFrame>> LoadTestImage(
+    absl::string_view path, ImageFormat::Format format = ImageFormat::SRGBA);
+
 // Loads a PNG image from path using the given ImageFormat. Returns nullptr in
 // case of failure.
 std::unique_ptr<ImageFrame> LoadTestPng(

mediapipe/gpu/BUILD

@@ -585,6 +585,7 @@ cc_library(
         "//mediapipe:apple": [
             ":gl_calculator_helper_ios",
             "//mediapipe/objc:util",
+            "//mediapipe/objc:CFHolder",
         ],
     }),
 )
@@ -714,11 +715,12 @@ cc_library(
     deps = [
         ":gl_calculator_helper",
         "//mediapipe/framework:calculator_framework",
-        "//mediapipe/framework:timestamp",
         "//mediapipe/framework/formats:image_frame",
-        "//mediapipe/framework/port:ret_check",
         "//mediapipe/framework/port:status",
-    ],
+    ] + select({
+        "//conditions:default": [],
+        "//mediapipe:apple": ["//mediapipe/objc:util"],
+    }),
     alwayslink = 1,
 )
 

mediapipe/gpu/gl_calculator_helper_impl_common.cc

@@ -12,6 +12,9 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+#include <memory>
+
+#include "mediapipe/framework/formats/image_frame.h"
 #include "mediapipe/gpu/gl_calculator_helper_impl.h"
 #include "mediapipe/gpu/gpu_buffer_format.h"
 #include "mediapipe/gpu/gpu_shared_data_internal.h"
@@ -176,10 +179,8 @@ GlTexture GlCalculatorHelperImpl::MapGlTextureBuffer(
 GlTextureBufferSharedPtr GlCalculatorHelperImpl::MakeGlTextureBuffer(
     const ImageFrame& image_frame) {
   CHECK(gl_context_->IsCurrent());
-  auto buffer = GlTextureBuffer::Create(
+
-      image_frame.Width(), image_frame.Height(),
+  auto buffer = GlTextureBuffer::Create(image_frame);
-      GpuBufferFormatForImageFormat(image_frame.Format()),
-      image_frame.PixelData());
 
   if (buffer->format_ != GpuBufferFormat::kUnknown) {
     glBindTexture(GL_TEXTURE_2D, buffer->name_);

mediapipe/gpu/gl_context_webgl.cc

@@ -75,26 +75,30 @@ absl::Status GlContext::CreateContextInternal(
   // TODO: Ensure this works with all options (in particular,
   //   multithreading options, like the special-case combination of USE_PTHREADS
   //   and OFFSCREEN_FRAMEBUFFER)
-  EM_ASM(let init_once = true; if (init_once) {
+  // clang-format off
-    const cachedFindCanvasEventTarget = findCanvasEventTarget;
+  EM_ASM(
+    let init_once = true;
+    if (init_once) {
+      const cachedFindCanvasEventTarget = findCanvasEventTarget;
 
-    if (typeof cachedFindCanvasEventTarget != = 'function') {
+      if (typeof cachedFindCanvasEventTarget !== 'function') {
-      if (typeof console != = 'undefined') {
+        if (typeof console !== 'undefined') {
-        console.error(
+          console.error('Expected Emscripten global function '
-            'Expected Emscripten global function ' +
+              + '"findCanvasEventTarget" not found. WebGL context creation '
-            '"findCanvasEventTarget" not found. WebGL context creation ' +
+              + 'may fail.');
-            'may fail.');
+        }
+        return;
       }
-      return;
-    }
 
-    findCanvasEventTarget = function(target) {
+      findCanvasEventTarget = function(target) {
-      if (Module && Module.canvas) {
+        if (target == 0) {
-        return Module.canvas;
+          if (Module && Module.canvas) {
-      } else if (Module && Module.canvasCssSelector) {
+            return Module.canvas;
-        return cachedFindCanvasEventTarget(Module.canvasCssSelector);
+          } else if (Module && Module.canvasCssSelector) {
-      } else {
+            return cachedFindCanvasEventTarget(Module.canvasCssSelector);
-        if (typeof console != = 'undefined') {
+          }
+        }
+        if (typeof console !== 'undefined') {
           console.warn('Module properties canvas and canvasCssSelector not ' +
                        'found during WebGL context creation.');
         }
@@ -102,15 +106,14 @@ absl::Status GlContext::CreateContextInternal(
         // cases it will not succeed, just in case the user does want to fall-
         // back.
         return cachedFindCanvasEventTarget(target);
-      }
+      };  // NOLINT: Necessary semicolon.
-    };  // NOLINT: Necessary semicolon.
+      init_once = false;
-    init_once = false;
+    }
-  });
+  );
+  // clang-format on
 
-  // Note: below id parameter is only actually used if both Module.canvas and
-  // Module.canvasCssSelector are undefined.
   EMSCRIPTEN_WEBGL_CONTEXT_HANDLE context_handle =
-      emscripten_webgl_create_context(0 /* id */, &attrs);
+      emscripten_webgl_create_context(nullptr, &attrs);
 
   // Check for failure
   if (context_handle <= 0) {

mediapipe/gpu/gl_texture_buffer.cc

@@ -32,15 +32,56 @@ std::unique_ptr<GlTextureBuffer> GlTextureBuffer::Wrap(
 
 std::unique_ptr<GlTextureBuffer> GlTextureBuffer::Create(int width, int height,
                                                          GpuBufferFormat format,
-                                                         const void* data) {
+                                                         const void* data,
+                                                         int alignment) {
   auto buf = absl::make_unique<GlTextureBuffer>(GL_TEXTURE_2D, 0, width, height,
                                                 format, nullptr);
-  if (!buf->CreateInternal(data)) {
+  if (!buf->CreateInternal(data, alignment)) {
     return nullptr;
   }
   return buf;
 }
 
+static inline int AlignedToPowerOf2(int value, int alignment) {
+  // alignment must be a power of 2
+  return ((value - 1) | (alignment - 1)) + 1;
+}
+
+std::unique_ptr<GlTextureBuffer> GlTextureBuffer::Create(
+    const ImageFrame& image_frame) {
+  int base_ws = image_frame.Width() * image_frame.NumberOfChannels() *
+                image_frame.ByteDepth();
+  int actual_ws = image_frame.WidthStep();
+  int alignment = 0;
+  std::unique_ptr<ImageFrame> temp;
+  const uint8* data = image_frame.PixelData();
+
+  // Let's see if the pixel data is tightly aligned to one of the alignments
+  // supported by OpenGL, preferring 4 if possible since it's the default.
+  if (actual_ws == AlignedToPowerOf2(base_ws, 4))
+    alignment = 4;
+  else if (actual_ws == AlignedToPowerOf2(base_ws, 1))
+    alignment = 1;
+  else if (actual_ws == AlignedToPowerOf2(base_ws, 2))
+    alignment = 2;
+  else if (actual_ws == AlignedToPowerOf2(base_ws, 8))
+    alignment = 8;
+
+  // If no GL-compatible alignment was found, we copy the data to a temporary
+  // buffer, aligned to 4. We do this using another ImageFrame purely for
+  // convenience.
+  if (!alignment) {
+    temp = std::make_unique<ImageFrame>();
+    temp->CopyFrom(image_frame, 4);
+    data = temp->PixelData();
+    alignment = 4;
+  }
+
+  return Create(image_frame.Width(), image_frame.Height(),
+                GpuBufferFormatForImageFormat(image_frame.Format()), data,
+                alignment);
+}
+
 GlTextureBuffer::GlTextureBuffer(GLenum target, GLuint name, int width,
                                  int height, GpuBufferFormat format,
                                  DeletionCallback deletion_callback,
@@ -53,7 +94,7 @@ GlTextureBuffer::GlTextureBuffer(GLenum target, GLuint name, int width,
       deletion_callback_(deletion_callback),
       producer_context_(producer_context) {}
 
-bool GlTextureBuffer::CreateInternal(const void* data) {
+bool GlTextureBuffer::CreateInternal(const void* data, int alignment) {
   auto context = GlContext::GetCurrent();
   if (!context) return false;
 
@@ -66,8 +107,11 @@ bool GlTextureBuffer::CreateInternal(const void* data) {
   GlTextureInfo info =
       GlTextureInfoForGpuBufferFormat(format_, 0, context->GetGlVersion());
 
+  if (alignment != 4 && data) glPixelStorei(GL_UNPACK_ALIGNMENT, alignment);
+
   // See b/70294573 for details about this.
   if (info.gl_internal_format == GL_RGBA16F &&
+      context->GetGlVersion() != GlVersion::kGLES2 &&
       SymbolAvailable(&glTexStorage2D)) {
     CHECK(data == nullptr) << "unimplemented";
     glTexStorage2D(target_, 1, info.gl_internal_format, width_, height_);
@@ -76,6 +120,8 @@ bool GlTextureBuffer::CreateInternal(const void* data) {
                  height_, 0 /* border */, info.gl_format, info.gl_type, data);
   }
 
+  if (alignment != 4 && data) glPixelStorei(GL_UNPACK_ALIGNMENT, 4);
+
   glBindTexture(target_, 0);
 
   // Use the deletion callback to delete the texture on the context

mediapipe/gpu/gl_texture_buffer.h

@@ -21,6 +21,7 @@
 #include <atomic>
 
 #include "absl/memory/memory.h"
+#include "mediapipe/framework/formats/image_frame.h"
 #include "mediapipe/gpu/gl_base.h"
 #include "mediapipe/gpu/gl_context.h"
 #include "mediapipe/gpu/gpu_buffer_format.h"
@@ -60,7 +61,11 @@ class GlTextureBuffer {
   // provided later via glTexSubImage2D.
   static std::unique_ptr<GlTextureBuffer> Create(int width, int height,
                                                  GpuBufferFormat format,
-                                                 const void* data = nullptr);
+                                                 const void* data = nullptr,
+                                                 int alignment = 4);
+
+  // Create a texture with a copy of the data in image_frame.
+  static std::unique_ptr<GlTextureBuffer> Create(const ImageFrame& image_frame);
 
   // Wraps an existing texture, but does not take ownership of it.
   // deletion_callback is invoked when the GlTextureBuffer is released, so
@@ -127,7 +132,7 @@ class GlTextureBuffer {
   // If data is provided, it is uploaded to the texture; otherwise, it can be
   // provided later via glTexSubImage2D.
   // Returns true on success.
-  bool CreateInternal(const void* data = nullptr);
+  bool CreateInternal(const void* data, int alignment = 4);
 
   friend class GlCalculatorHelperImpl;
 

mediapipe/gpu/gpu_buffer_format.cc

@@ -164,7 +164,9 @@ const GlTextureInfo& GlTextureInfoForGpuBufferFormat(GpuBufferFormat format,
   }
 
   auto iter = format_info->find(format);
-  CHECK(iter != format_info->end()) << "unsupported format";
+  CHECK(iter != format_info->end())
+      << "unsupported format: "
+      << static_cast<std::underlying_type_t<decltype(format)>>(format);
   const auto& planes = iter->second;
 #ifndef __APPLE__
   CHECK_EQ(planes.size(), 1)

mediapipe/gpu/shader_util.cc

@@ -51,8 +51,6 @@ namespace mediapipe {
 constexpr int kMaxShaderInfoLength = 1024;
 
 GLint GlhCompileShader(GLenum target, const GLchar* source, GLuint* shader) {
-  GLint status;
-
   *shader = glCreateShader(target);
   if (*shader == 0) {
     return GL_FALSE;
@@ -62,6 +60,11 @@ GLint GlhCompileShader(GLenum target, const GLchar* source, GLuint* shader) {
 
   GL_DEBUG_LOG(Shader, *shader, "compile");
 
+#if UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
+  return GL_TRUE;
+#else
+  GLint status;
+
   glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
   LOG_IF(ERROR, status == GL_FALSE) << "Failed to compile shader:\n" << source;
 
@@ -72,19 +75,24 @@ GLint GlhCompileShader(GLenum target, const GLchar* source, GLuint* shader) {
     LOG(ERROR) << "Error message: " << std::string(cmessage, length);
   }
   return status;
+#endif  // UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
 }
 
 GLint GlhLinkProgram(GLuint program) {
-  GLint status;
-
   glLinkProgram(program);
 
+#if UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
+  return GL_TRUE;
+#else
+  GLint status;
+
   GL_DEBUG_LOG(Program, program, "link");
 
   glGetProgramiv(program, GL_LINK_STATUS, &status);
   LOG_IF(ERROR, status == GL_FALSE) << "Failed to link program " << program;
 
   return status;
+#endif  // UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
 }
 
 GLint GlhValidateProgram(GLuint program) {

mediapipe/graphs/face_detection/BUILD

@@ -11,6 +11,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+load(
+    "//mediapipe/framework/tool:mediapipe_graph.bzl",
+    "mediapipe_binary_graph",
+)
 
 licenses(["notice"])
 
@@ -24,8 +28,8 @@ cc_library(
         "//mediapipe/calculators/util:detections_to_render_data_calculator",
         "//mediapipe/gpu:gpu_buffer_to_image_frame_calculator",
         "//mediapipe/gpu:image_frame_to_gpu_buffer_calculator",
-        "//mediapipe/modules/face_detection:face_detection_front_cpu",
+        "//mediapipe/modules/face_detection:face_detection_short_range_cpu",
-        "//mediapipe/modules/face_detection:face_detection_front_gpu",
+        "//mediapipe/modules/face_detection:face_detection_short_range_gpu",
     ],
 )
 
@@ -35,7 +39,7 @@ cc_library(
         "//mediapipe/calculators/core:flow_limiter_calculator",
         "//mediapipe/calculators/util:annotation_overlay_calculator",
         "//mediapipe/calculators/util:detections_to_render_data_calculator",
-        "//mediapipe/modules/face_detection:face_detection_front_cpu",
+        "//mediapipe/modules/face_detection:face_detection_short_range_cpu",
     ],
 )
 
@@ -45,15 +49,10 @@ cc_library(
         "//mediapipe/calculators/core:flow_limiter_calculator",
         "//mediapipe/calculators/util:annotation_overlay_calculator",
         "//mediapipe/calculators/util:detections_to_render_data_calculator",
-        "//mediapipe/modules/face_detection:face_detection_front_gpu",
+        "//mediapipe/modules/face_detection:face_detection_short_range_gpu",
     ],
 )
 
-load(
-    "//mediapipe/framework/tool:mediapipe_graph.bzl",
-    "mediapipe_binary_graph",
-)
-
 mediapipe_binary_graph(
     name = "face_detection_mobile_cpu_binary_graph",
     graph = "face_detection_mobile_cpu.pbtxt",
@@ -67,3 +66,30 @@ mediapipe_binary_graph(
     output_name = "face_detection_mobile_gpu.binarypb",
     deps = [":mobile_calculators"],
 )
+
+cc_library(
+    name = "face_detection_full_range_mobile_gpu_deps",
+    deps = [
+        "//mediapipe/calculators/core:flow_limiter_calculator",
+        "//mediapipe/calculators/util:annotation_overlay_calculator",
+        "//mediapipe/calculators/util:detections_to_render_data_calculator",
+        "//mediapipe/modules/face_detection:face_detection_full_range_gpu",
+    ],
+)
+
+mediapipe_binary_graph(
+    name = "face_detection_full_range_mobile_gpu_binary_graph",
+    graph = "face_detection_full_range_mobile_gpu.pbtxt",
+    output_name = "face_detection_full_range_mobile_gpu.binarypb",
+    deps = [":face_detection_full_range_mobile_gpu_deps"],
+)
+
+cc_library(
+    name = "face_detection_full_range_desktop_live_deps",
+    deps = [
+        "//mediapipe/calculators/core:flow_limiter_calculator",
+        "//mediapipe/calculators/util:annotation_overlay_calculator",
+        "//mediapipe/calculators/util:detections_to_render_data_calculator",
+        "//mediapipe/modules/face_detection:face_detection_full_range_cpu",
+    ],
+)

mediapipe/graphs/face_detection/face_detection_back_desktop_live.pbtxt

@@ -1,169 +0,0 @@
-# MediaPipe graph that performs face detection with TensorFlow Lite on CPU.
-# Used in the examples in
-# mediapipe/examples/desktop/face_detection:face_detection_cpu.
-
-# Images on GPU coming into and out of the graph.
-input_stream: "input_video"
-output_stream: "output_video"
-
-# Throttles the images flowing downstream for flow control. It passes through
-# the very first incoming image unaltered, and waits for
-# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
-# generating the corresponding detections before it passes through another
-# image. All images that come in while waiting are dropped, limiting the number
-# of in-flight images between this calculator and
-# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
-# from queuing up incoming images and data excessively, which leads to increased
-# latency and memory usage, unwanted in real-time mobile applications. It also
-# eliminates unnecessarily computation, e.g., a transformed image produced by
-# ImageTransformationCalculator may get dropped downstream if the subsequent
-# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
-# processing previous inputs.
-node {
-  calculator: "FlowLimiterCalculator"
-  input_stream: "input_video"
-  input_stream: "FINISHED:detections"
-  input_stream_info: {
-    tag_index: "FINISHED"
-    back_edge: true
-  }
-  output_stream: "throttled_input_video"
-}
-
-# Transforms the input image on CPU to a 128x128 image. To scale the input
-# image, the scale_mode option is set to FIT to preserve the aspect ratio,
-# resulting in potential letterboxing in the transformed image.
-node: {
-  calculator: "ImageTransformationCalculator"
-  input_stream: "IMAGE:throttled_input_video"
-  output_stream: "IMAGE:transformed_input_video_cpu"
-  output_stream: "LETTERBOX_PADDING:letterbox_padding"
-  node_options: {
-    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
-      output_width: 192
-      output_height: 192
-      scale_mode: FIT
-    }
-  }
-}
-
-# Converts the transformed input image on CPU into an image tensor stored as a
-# TfLiteTensor.
-node {
-  calculator: "TfLiteConverterCalculator"
-  input_stream: "IMAGE:transformed_input_video_cpu"
-  output_stream: "TENSORS:image_tensor"
-}
-
-# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
-# vector of tensors representing, for instance, detection boxes/keypoints and
-# scores.
-node {
-  calculator: "TfLiteInferenceCalculator"
-  input_stream: "TENSORS:image_tensor"
-  output_stream: "TENSORS:detection_tensors"
-  node_options: {
-    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
-      model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
-    }
-  }
-}
-
-# Generates a single side packet containing a vector of SSD anchors based on
-# the specification in the options.
-node {
-  calculator: "SsdAnchorsCalculator"
-  output_side_packet: "anchors"
-  node_options: {
-    [type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
-      num_layers: 1
-      min_scale: 0.1484375
-      max_scale: 0.75
-      input_size_height: 192
-      input_size_width: 192
-      anchor_offset_x: 0.5
-      anchor_offset_y: 0.5
-      strides: 4
-      aspect_ratios: 1.0
-      fixed_anchor_size: true
-      interpolated_scale_aspect_ratio: 0.0
-    }
-  }
-}
-
-# Decodes the detection tensors generated by the TensorFlow Lite model, based on
-# the SSD anchors and the specification in the options, into a vector of
-# detections. Each detection describes a detected object.
-node {
-  calculator: "TfLiteTensorsToDetectionsCalculator"
-  input_stream: "TENSORS:detection_tensors"
-  input_side_packet: "ANCHORS:anchors"
-  output_stream: "DETECTIONS:detections"
-  node_options: {
-    [type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
-      num_classes: 1
-      num_boxes: 2304
-      num_coords: 16
-      box_coord_offset: 0
-      keypoint_coord_offset: 4
-      num_keypoints: 6
-      num_values_per_keypoint: 2
-      sigmoid_score: true
-      score_clipping_thresh: 100.0
-      reverse_output_order: true
-      x_scale: 192.0
-      y_scale: 192.0
-      h_scale: 192.0
-      w_scale: 192.0
-      min_score_thresh: 0.6
-    }
-  }
-}
-
-# Performs non-max suppression to remove excessive detections.
-node {
-  calculator: "NonMaxSuppressionCalculator"
-  input_stream: "detections"
-  output_stream: "filtered_detections"
-  node_options: {
-    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
-      min_suppression_threshold: 0.3
-      overlap_type: INTERSECTION_OVER_UNION
-      algorithm: WEIGHTED
-      return_empty_detections: true
-    }
-  }
-}
-
-# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
-# letterboxed image (after image transformation with the FIT scale mode) to the
-# corresponding locations on the same image with the letterbox removed (the
-# input image to the graph before image transformation).
-node {
-  calculator: "DetectionLetterboxRemovalCalculator"
-  input_stream: "DETECTIONS:filtered_detections"
-  input_stream: "LETTERBOX_PADDING:letterbox_padding"
-  output_stream: "DETECTIONS:output_detections"
-}
-
-# Converts the detections to drawing primitives for annotation overlay.
-node {
-  calculator: "DetectionsToRenderDataCalculator"
-  input_stream: "DETECTIONS:output_detections"
-  output_stream: "RENDER_DATA:render_data"
-  node_options: {
-    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
-      thickness: 4.0
-      color { r: 255 g: 0 b: 0 }
-    }
-  }
-}
-
-# Draws annotations and overlays them on top of the input images.
-node {
-  calculator: "AnnotationOverlayCalculator"
-  input_stream: "IMAGE:throttled_input_video"
-  input_stream: "render_data"
-  output_stream: "IMAGE:output_video"
-}
-

mediapipe/graphs/face_detection/face_detection_back_mobile_gpu.pbtxt

@@ -1,169 +0,0 @@
-# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
-# Used in the examples in
-# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectiongpu and
-# mediapipie/examples/ios/facedetectiongpu.
-
-# Images on GPU coming into and out of the graph.
-input_stream: "input_video"
-output_stream: "output_video"
-
-# Throttles the images flowing downstream for flow control. It passes through
-# the very first incoming image unaltered, and waits for
-# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
-# generating the corresponding detections before it passes through another
-# image. All images that come in while waiting are dropped, limiting the number
-# of in-flight images between this calculator and
-# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
-# from queuing up incoming images and data excessively, which leads to increased
-# latency and memory usage, unwanted in real-time mobile applications. It also
-# eliminates unnecessarily computation, e.g., a transformed image produced by
-# ImageTransformationCalculator may get dropped downstream if the subsequent
-# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
-# processing previous inputs.
-node {
-  calculator: "FlowLimiterCalculator"
-  input_stream: "input_video"
-  input_stream: "FINISHED:detections"
-  input_stream_info: {
-    tag_index: "FINISHED"
-    back_edge: true
-  }
-  output_stream: "throttled_input_video"
-}
-
-# Transforms the input image on GPU to a 128x128 image. To scale the input
-# image, the scale_mode option is set to FIT to preserve the aspect ratio,
-# resulting in potential letterboxing in the transformed image.
-node: {
-  calculator: "ImageTransformationCalculator"
-  input_stream: "IMAGE_GPU:throttled_input_video"
-  output_stream: "IMAGE_GPU:transformed_input_video"
-  output_stream: "LETTERBOX_PADDING:letterbox_padding"
-  node_options: {
-    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
-      output_width: 192
-      output_height: 192
-      scale_mode: FIT
-    }
-  }
-}
-
-# Converts the transformed input image on GPU into an image tensor stored as a
-# TfLiteTensor.
-node {
-  calculator: "TfLiteConverterCalculator"
-  input_stream: "IMAGE_GPU:transformed_input_video"
-  output_stream: "TENSORS_GPU:image_tensor"
-}
-
-# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
-# vector of tensors representing, for instance, detection boxes/keypoints and
-# scores.
-node {
-  calculator: "TfLiteInferenceCalculator"
-  input_stream: "TENSORS_GPU:image_tensor"
-  output_stream: "TENSORS_GPU:detection_tensors"
-  node_options: {
-    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
-      model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
-    }
-  }
-}
-
-# Generates a single side packet containing a vector of SSD anchors based on
-# the specification in the options.
-node {
-  calculator: "SsdAnchorsCalculator"
-  output_side_packet: "anchors"
-  node_options: {
-    [type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
-      num_layers: 1
-      min_scale: 0.1484375
-      max_scale: 0.75
-      input_size_height: 192
-      input_size_width: 192
-      anchor_offset_x: 0.5
-      anchor_offset_y: 0.5
-      strides: 4
-      aspect_ratios: 1.0
-      fixed_anchor_size: true
-      interpolated_scale_aspect_ratio: 0.0
-    }
-  }
-}
-
-# Decodes the detection tensors generated by the TensorFlow Lite model, based on
-# the SSD anchors and the specification in the options, into a vector of
-# detections. Each detection describes a detected object.
-node {
-  calculator: "TfLiteTensorsToDetectionsCalculator"
-  input_stream: "TENSORS_GPU:detection_tensors"
-  input_side_packet: "ANCHORS:anchors"
-  output_stream: "DETECTIONS:detections"
-  node_options: {
-    [type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
-      num_classes: 1
-      num_boxes: 2304
-      num_coords: 16
-      box_coord_offset: 0
-      keypoint_coord_offset: 4
-      num_keypoints: 6
-      num_values_per_keypoint: 2
-      sigmoid_score: true
-      score_clipping_thresh: 100.0
-      reverse_output_order: true
-      x_scale: 192.0
-      y_scale: 192.0
-      h_scale: 192.0
-      w_scale: 192.0
-      min_score_thresh: 0.6
-    }
-  }
-}
-
-# Performs non-max suppression to remove excessive detections.
-node {
-  calculator: "NonMaxSuppressionCalculator"
-  input_stream: "detections"
-  output_stream: "filtered_detections"
-  node_options: {
-    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
-      min_suppression_threshold: 0.3
-      overlap_type: INTERSECTION_OVER_UNION
-      algorithm: WEIGHTED
-      return_empty_detections: true
-    }
-  }
-}
-
-# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
-# letterboxed image (after image transformation with the FIT scale mode) to the
-# corresponding locations on the same image with the letterbox removed (the
-# input image to the graph before image transformation).
-node {
-  calculator: "DetectionLetterboxRemovalCalculator"
-  input_stream: "DETECTIONS:filtered_detections"
-  input_stream: "LETTERBOX_PADDING:letterbox_padding"
-  output_stream: "DETECTIONS:output_detections"
-}
-
-# Converts the detections to drawing primitives for annotation overlay.
-node {
-  calculator: "DetectionsToRenderDataCalculator"
-  input_stream: "DETECTIONS:output_detections"
-  output_stream: "RENDER_DATA:render_data"
-  node_options: {
-    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
-      thickness: 4.0
-      color { r: 255 g: 0 b: 0 }
-    }
-  }
-}
-
-# Draws annotations and overlays them on top of the input images.
-node {
-  calculator: "AnnotationOverlayCalculator"
-  input_stream: "IMAGE_GPU:throttled_input_video"
-  input_stream: "render_data"
-  output_stream: "IMAGE_GPU:output_video"
-}

mediapipe/graphs/face_detection/face_detection_desktop_live.pbtxt

@@ -31,7 +31,7 @@ node {
 
 # Subgraph that detects faces.
 node {
-  calculator: "FaceDetectionFrontCpu"
+  calculator: "FaceDetectionShortRangeCpu"
   input_stream: "IMAGE:throttled_input_video"
   output_stream: "DETECTIONS:face_detections"
 }

mediapipe/graphs/face_detection/face_detection_full_range_desktop_live.pbtxt

@@ -0,0 +1,60 @@
+# MediaPipe graph that performs face detection with TensorFlow Lite on CPU.
+# Used in the examples in
+# mediapipe/examples/desktop/face_detection:face_detection_cpu.
+
+# Images on GPU coming into and out of the graph.
+input_stream: "input_video"
+output_stream: "output_video"
+
+# Throttles the images flowing downstream for flow control. It passes through
+# the very first incoming image unaltered, and waits for
+# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
+# generating the corresponding detections before it passes through another
+# image. All images that come in while waiting are dropped, limiting the number
+# of in-flight images between this calculator and
+# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
+# from queuing up incoming images and data excessively, which leads to increased
+# latency and memory usage, unwanted in real-time mobile applications. It also
+# eliminates unnecessarily computation, e.g., a transformed image produced by
+# ImageTransformationCalculator may get dropped downstream if the subsequent
+# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
+# processing previous inputs.
+node {
+  calculator: "FlowLimiterCalculator"
+  input_stream: "input_video"
+  input_stream: "FINISHED:detections"
+  input_stream_info: {
+    tag_index: "FINISHED"
+    back_edge: true
+  }
+  output_stream: "throttled_input_video"
+}
+
+# Detects faces.
+node {
+  calculator: "FaceDetectionFullRangeCpu"
+  input_stream: "IMAGE:throttled_input_video"
+  output_stream: "DETECTIONS:detections"
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTIONS:detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the input images.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "IMAGE:throttled_input_video"
+  input_stream: "render_data"
+  output_stream: "IMAGE:output_video"
+}
+

mediapipe/graphs/face_detection/face_detection_full_range_mobile_gpu.pbtxt

@@ -0,0 +1,60 @@
+# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
+# Used in the examples in
+# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectiongpu and
+# mediapipie/examples/ios/facedetectiongpu.
+
+# Images on GPU coming into and out of the graph.
+input_stream: "input_video"
+output_stream: "output_video"
+
+# Throttles the images flowing downstream for flow control. It passes through
+# the very first incoming image unaltered, and waits for
+# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
+# generating the corresponding detections before it passes through another
+# image. All images that come in while waiting are dropped, limiting the number
+# of in-flight images between this calculator and
+# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
+# from queuing up incoming images and data excessively, which leads to increased
+# latency and memory usage, unwanted in real-time mobile applications. It also
+# eliminates unnecessarily computation, e.g., a transformed image produced by
+# ImageTransformationCalculator may get dropped downstream if the subsequent
+# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
+# processing previous inputs.
+node {
+  calculator: "FlowLimiterCalculator"
+  input_stream: "input_video"
+  input_stream: "FINISHED:output_video"
+  input_stream_info: {
+    tag_index: "FINISHED"
+    back_edge: true
+  }
+  output_stream: "throttled_input_video"
+}
+
+# Detects faces.
+node {
+  calculator: "FaceDetectionFullRangeGpu"
+  input_stream: "IMAGE:throttled_input_video"
+  output_stream: "DETECTIONS:detections"
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTIONS:detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the input images.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "IMAGE_GPU:throttled_input_video"
+  input_stream: "render_data"
+  output_stream: "IMAGE_GPU:output_video"
+}

mediapipe/graphs/face_detection/face_detection_mobile_cpu.pbtxt

@@ -41,7 +41,7 @@ node: {
 
 # Subgraph that detects faces.
 node {
-  calculator: "FaceDetectionFrontCpu"
+  calculator: "FaceDetectionShortRangeCpu"
   input_stream: "IMAGE:input_video_cpu"
   output_stream: "DETECTIONS:face_detections"
 }

mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt

@@ -31,7 +31,7 @@ node {
 
 # Subgraph that detects faces.
 node {
-  calculator: "FaceDetectionFrontGpu"
+  calculator: "FaceDetectionShortRangeGpu"
   input_stream: "IMAGE:throttled_input_video"
   output_stream: "DETECTIONS:face_detections"
 }

mediapipe/graphs/face_effect/subgraphs/BUILD

@@ -39,7 +39,7 @@ mediapipe_simple_subgraph(
         "//mediapipe/calculators/core:concatenate_detection_vector_calculator",
         "//mediapipe/calculators/core:split_vector_calculator",
         "//mediapipe/calculators/image:image_properties_calculator",
-        "//mediapipe/modules/face_detection:face_detection_front_gpu",
+        "//mediapipe/modules/face_detection:face_detection_short_range_gpu",
         "//mediapipe/modules/face_geometry:face_geometry_from_detection",
     ],
 )

mediapipe/graphs/face_effect/subgraphs/single_face_geometry_from_detection_gpu.pbtxt

@@ -24,7 +24,7 @@ output_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
 # Subgraph that detects faces and corresponding landmarks using the face
 # detection pipeline.
 node {
-  calculator: "FaceDetectionFrontGpu"
+  calculator: "FaceDetectionShortRangeGpu"
   input_stream: "IMAGE:input_image"
   output_stream: "DETECTIONS:multi_face_detection"
 }