Project import generated by Copybara.

GitOrigin-RevId: 65b427572550bd9c5bc5f053eeea0f44340d5673

.github/ISSUE_TEMPLATE/00-build-installation-issue.md

@@ -0,0 +1,27 @@
+---
+name: "Build/Installation Issue"
+about: Use this template for build/installation issues
+labels: type:build/install
+
+---
+<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
+
+**System information** (Please provide as much relevant information as possible)
+- OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
+- Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
+- Installed using virtualenv? pip? Conda? (if python):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version:
+- XCode and Tulsi versions (if iOS):
+- Android SDK and NDK versions (if android):
+- Android [AAR](https://google.github.io/mediapipe/getting_started/android_archive_library.html) ( if android):
+- OpenCV version (if running on desktop):
+
+**Describe the problem**:
+
+
+**[Provide the exact sequence of commands / steps that you executed before running into the problem](https://google.github.io/mediapipe/getting_started/getting_started.html):**
+
+**Complete Logs:**
+Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:

.github/ISSUE_TEMPLATE/10-solution-issue.md

@@ -0,0 +1,26 @@
+---
+name: "Solution Issue"
+about: Use this template for assistance with a specific mediapipe solution, such as "Pose" or "Iris", including inference model usage/training, solution-specific calculators, etc.
+labels: type:support
+
+---
+<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
+
+**System information** (Please provide as much relevant information as possible)
+- Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
+- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version:
+- Solution (e.g. FaceMesh, Pose, Holistic):
+- Programming Language and version ( e.g. C++, Python, Java):
+
+**Describe the expected behavior:**
+
+**Standalone code you may have used to try to get what you need :**
+
+If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
+
+**Other info / Complete Logs :**
+Include any logs or source code that would be helpful to
+diagnose the problem. If including tracebacks, please include the full
+traceback. Large logs and files should be attached:

.github/ISSUE_TEMPLATE/20-documentation-issue.md

@@ -0,0 +1,51 @@
+---
+name: "Documentation Issue"
+about: Use this template for documentation related issues
+labels: type:docs
+
+---
+Thank you for submitting a MediaPipe documentation issue.
+The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
+## URL(s) with the issue:
+
+Please provide a link to the documentation entry, for example: https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
+
+## Description of issue (what needs changing):
+
+Kinds of documentation problems:
+
+### Clear description
+
+For example, why should someone use this method? How is it useful?
+
+### Correct links
+
+Is the link to the source code correct?
+
+### Parameters defined
+Are all parameters defined and formatted correctly?
+
+### Returns defined
+
+Are return values defined?
+
+### Raises listed and defined
+
+Are the errors defined? For example,
+
+### Usage example
+
+Is there a usage example?
+
+See the API guide:
+on how to write testable usage examples.
+
+### Request visuals, if applicable
+
+Are there currently visuals? If not, will it clarify the content?
+
+### Submit a pull request?
+
+Are you planning to also submit a pull request to fix the issue? See the docs
+https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md
+

.github/ISSUE_TEMPLATE/30-bug-issue.md

@@ -0,0 +1,32 @@
+---
+name: "Bug Issue"
+about: Use this template for reporting a bug
+labels: type:bug
+
+---
+<em>Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.</em>
+
+**System information** (Please provide as much relevant information as possible)
+
+- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
+- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
+- Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser:
+- Programming Language and version ( e.g. C++, Python, Java):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version (if compiling from source):
+- Solution ( e.g. FaceMesh, Pose, Holistic ):
+- Android Studio, NDK, SDK versions (if issue is related to building in Android environment):
+- Xcode & Tulsi version (if issue is related to building for iOS):
+
+**Describe the current behavior:**
+
+**Describe the expected behavior:**
+
+**Standalone code to reproduce the issue:**
+Provide a reproducible test case that is the bare minimum necessary to replicate the problem. If possible, please share a link to Colab/repo link /any notebook:
+
+**Other info / Complete Logs :**
+ Include any logs or source code that would be helpful to
+diagnose the problem. If including tracebacks, please include the full
+traceback. Large logs and files should be attached

.github/ISSUE_TEMPLATE/40-feature-request.md

@@ -0,0 +1,24 @@
+---
+name: "Feature Request"
+about: Use this template for raising a feature request
+labels: type:feature
+
+---
+<em>Please make sure that this is a feature request.</em>
+
+**System information** (Please provide as much relevant information as possible)
+
+- MediaPipe Solution (you are using):
+- Programming language : C++/typescript/Python/Objective C/Android Java
+- Are you willing to contribute it (Yes/No):
+
+
+**Describe the feature and the current behavior/state:**
+
+**Will this change the current api? How?**
+
+**Who will benefit with this feature?**
+
+**Please specify the use cases for this feature:**
+
+**Any Other info:**

.github/ISSUE_TEMPLATE/50-other-issues.md

@@ -0,0 +1,14 @@
+---
+name: "Other Issue"
+about: Use this template for any other non-support related issues.
+labels: type:others
+
+---
+This template is for miscellaneous issues not covered by the other issue categories
+
+For questions on how to work with MediaPipe, or support for problems that are not verified bugs in MediaPipe, please go to [StackOverflow](https://stackoverflow.com/questions/tagged/mediapipe) and [Slack](https://mediapipe.page.link/joinslack) communities.
+
+If you are reporting a vulnerability, please use the [dedicated reporting process](https://github.com/google/mediapipe/security).
+
+For high-level discussions about MediaPipe, please post to discuss@mediapipe.org, for questions about the development or internal workings of MediaPipe, or if you would like to know how to contribute to MediaPipe, please post to developers@mediapipe.org.
+

.github/bot_config.yml

@@ -0,0 +1,18 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+# A list of assignees
+assignees:
+   - sgowroji

.github/stale.yml

@@ -0,0 +1,34 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 7
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 7
+# Only issues or pull requests with all of these labels are checked if stale. Defaults to `[]` (disabled)
+onlyLabels:
+ - stat:awaiting response
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you.
+# Comment to post when removing the stale label. Set to `false` to disable
+unmarkComment: false
+closeComment: >
+  Closing as stale. Please reopen if you'd like to work on this further.

MANIFEST.in

@@ -8,6 +8,7 @@ include README.md
 include requirements.txt
 
 recursive-include mediapipe/modules *.tflite *.txt *.binarypb
+exclude mediapipe/modules/face_detection/face_detection_full_range.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers.tflite

README.md

@@ -40,11 +40,12 @@ Hair Segmentation
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                        | ✅                                                         | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
@@ -54,46 +55,22 @@ See also
 [MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
 for ML models released in MediaPipe.
 
-## MediaPipe in Python
-
-MediaPipe offers customizable Python solutions as a prebuilt Python package on
-[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
-`pip install mediapipe`. It also provides tools for users to build their own
-solutions. Please see
-[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
-for more info.
-
-## MediaPipe on the Web
-
-MediaPipe on the Web is an effort to run the same ML solutions built for mobile
-and desktop also in web browsers. The official API is under construction, but
-the core technology has been proven effective. Please see
-[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
-in Google Developers Blog for details.
-
-You can use the following links to load a demo in the MediaPipe Visualizer, and
-over there click the "Runner" icon in the top bar like shown below. The demos
-use your webcam video as input, which is processed all locally in real-time and
-never leaves your device.
-
-![visualizer_runner](docs/images/visualizer_runner.png)
-
-*   [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
-*   [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
-*   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
-*   [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
-*   [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
-*   [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
-*   [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
-
 ## Getting started
 
-Learn how to [install](https://google.github.io/mediapipe/getting_started/install)
-MediaPipe and
-[build example applications](https://google.github.io/mediapipe/getting_started/building_examples),
-and start exploring our ready-to-use
-[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
-further extend and customize.
+To start using MediaPipe
+[solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
+lines code, see example code and demos in
+[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
+[MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
+
+To use MediaPipe in C++, Android and iOS, which allow further customization of
+the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
+building your own, learn how to
+[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
+start building example applications in
+[C++](https://google.github.io/mediapipe/getting_started/cpp),
+[Android](https://google.github.io/mediapipe/getting_started/android) and
+[iOS](https://google.github.io/mediapipe/getting_started/ios).
 
 The source code is hosted in the
 [MediaPipe Github repository](https://github.com/google/mediapipe), and you can

WORKSPACE

@@ -35,8 +35,8 @@ http_archive(
 
 http_archive(
     name = "rules_cc",
-    strip_prefix = "rules_cc-master",
-    urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
+    strip_prefix = "rules_cc-main",
+    urls = ["https://github.com/bazelbuild/rules_cc/archive/main.zip"],
 )
 
 http_archive(
@@ -71,8 +71,8 @@ http_archive(
 # Google Benchmark library.
 http_archive(
     name = "com_google_benchmark",
-    urls = ["https://github.com/google/benchmark/archive/master.zip"],
-    strip_prefix = "benchmark-master",
+    urls = ["https://github.com/google/benchmark/archive/main.zip"],
+    strip_prefix = "benchmark-main",
     build_file = "@//third_party:benchmark.BUILD",
 )
 
@@ -242,6 +242,20 @@ http_archive(
     url = "https://github.com/opencv/opencv/releases/download/3.2.0/opencv-3.2.0-ios-framework.zip",
 )
 
+http_archive(
+    name = "stblib",
+    strip_prefix = "stb-b42009b3b9d4ca35bc703f5310eedc74f584be58",
+    sha256 = "13a99ad430e930907f5611325ec384168a958bf7610e63e60e2fd8e7b7379610",
+    urls = ["https://github.com/nothings/stb/archive/b42009b3b9d4ca35bc703f5310eedc74f584be58.tar.gz"],
+    build_file = "@//third_party:stblib.BUILD",
+    patches = [
+        "@//third_party:stb_image_impl.diff"
+    ],
+    patch_args = [
+        "-p1",
+    ],
+)
+
 # You may run setup_android.sh to install Android SDK and NDK.
 android_ndk_repository(
     name = "androidndk",
@@ -337,8 +351,8 @@ maven_install(
         "androidx.test.espresso:espresso-core:3.1.1",
         "com.github.bumptech.glide:glide:4.11.0",
         "com.google.android.material:material:aar:1.0.0-rc01",
-        "com.google.auto.value:auto-value:1.6.4",
-        "com.google.auto.value:auto-value-annotations:1.6.4",
+        "com.google.auto.value:auto-value:1.8.1",
+        "com.google.auto.value:auto-value-annotations:1.8.1",
         "com.google.code.findbugs:jsr305:3.0.2",
         "com.google.flogger:flogger-system-backend:0.3.1",
         "com.google.flogger:flogger:0.3.1",
@@ -369,9 +383,9 @@ http_archive(
 )
 
 # Tensorflow repo should always go after the other external dependencies.
-# 2021-04-30
-_TENSORFLOW_GIT_COMMIT = "5bd3c57ef184543d22e34e36cff9d9bea608e06d"
-_TENSORFLOW_SHA256= "9a45862834221aafacf6fb275f92b3876bc89443cbecc51be93f13839a6609f0"
+# 2021-06-07
+_TENSORFLOW_GIT_COMMIT = "700533808e6016dc458bb2eeecfca4babfc482ec"
+_TENSORFLOW_SHA256 = "b6edd7f4039bfc19f3e77594ecff558ba620091d0dc48181484b3d9085026126"
 http_archive(
     name = "org_tensorflow",
     urls = [

build_desktop_examples.sh

@@ -97,6 +97,7 @@ for app in ${apps}; do
       if [[ ${target_name} == "holistic_tracking" ||
             ${target_name} == "iris_tracking" ||
             ${target_name} == "pose_tracking" ||
+            ${target_name} == "selfie_segmentation" ||
             ${target_name} == "upper_body_pose_tracking" ]]; then
         graph_suffix="cpu"
       else

docs/framework_concepts/calculators.md

@@ -248,12 +248,70 @@ absl::Status MyCalculator::Process() {
 }
 ```
 
+## Calculator options
+
+Calculators accept processing parameters through (1) input stream packets (2)
+input side packets, and (3) calculator options. Calculator options, if
+specified, appear as literal values in the `node_options` field of the
+`CalculatorGraphConfiguration.Node` message.
+
+```
+  node {
+    calculator: "TfLiteInferenceCalculator"
+    input_stream: "TENSORS:main_model_input"
+    output_stream: "TENSORS:main_model_output"
+    node_options: {
+      [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+        model_path: "mediapipe/models/detection_model.tflite"
+      }
+    }
+  }
+```
+
+The `node_options` field accepts the proto3 syntax.  Alternatively, calculator
+options can be specified in the `options` field using proto2 syntax.
+
+```
+  node {
+    calculator: "TfLiteInferenceCalculator"
+    input_stream: "TENSORS:main_model_input"
+    output_stream: "TENSORS:main_model_output"
+    node_options: {
+      [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+        model_path: "mediapipe/models/detection_model.tflite"
+      }
+    }
+  }
+```
+
+Not all calculators accept calcuator options. In order to accept options, a
+calculator will normally define a new protobuf message type to represent its
+options, such as `PacketClonerCalculatorOptions`. The calculator will then
+read that protobuf message in its `CalculatorBase::Open` method, and possibly
+also in its `CalculatorBase::GetContract` function or its
+`CalculatorBase::Process` method. Normally, the new protobuf message type will
+be defined as a protobuf schema using a ".proto" file and a
+`mediapipe_proto_library()` build rule.
+
+```
+  mediapipe_proto_library(
+      name = "packet_cloner_calculator_proto",
+      srcs = ["packet_cloner_calculator.proto"],
+      visibility = ["//visibility:public"],
+      deps = [
+          "//mediapipe/framework:calculator_options_proto",
+          "//mediapipe/framework:calculator_proto",
+      ],
+  )
+```
+
+
 ## Example calculator
 
 This section discusses the implementation of `PacketClonerCalculator`, which
 does a relatively simple job, and is used in many calculator graphs.
-`PacketClonerCalculator` simply produces a copy of its most recent input
-packets on demand.
+`PacketClonerCalculator` simply produces a copy of its most recent input packets
+on demand.
 
 `PacketClonerCalculator` is useful when the timestamps of arriving data packets
 are not aligned perfectly. Suppose we have a room with a microphone, light
@@ -279,8 +337,8 @@ input streams:
     imageframe of video data representing video collected from camera in the
     room with timestamp.
 
-Below is the implementation of the `PacketClonerCalculator`.  You can see
-the `GetContract()`, `Open()`, and `Process()` methods as well as the instance
+Below is the implementation of the `PacketClonerCalculator`. You can see the
+`GetContract()`, `Open()`, and `Process()` methods as well as the instance
 variable `current_` which holds the most recent input packets.
 
 ```c++
@@ -401,6 +459,6 @@ node {
 The diagram below shows how the `PacketClonerCalculator` defines its output
 packets (bottom) based on its series of input packets (top).
 
-| ![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png) |
-| :---------------------------------------------------------------------------: |
-| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |
+![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png)  |
+:--------------------------------------------------------------------------: |
+*Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |

docs/framework_concepts/framework_concepts.md

@@ -111,11 +111,11 @@ component known as an InputStreamHandler.
 
 See [Synchronization](synchronization.md) for more details.
 
-### Realtime data streams
+### Real-time streams
 
 MediaPipe calculator graphs are often used to process streams of video or audio
 frames for interactive applications. Normally, each Calculator runs as soon as
 all of its input packets for a given timestamp become available. Calculators
-used in realtime graphs need to define output timestamp bounds based on input
+used in real-time graphs need to define output timestamp bounds based on input
 timestamp bounds in order to allow downstream calculators to be scheduled
-promptly. See [Realtime data streams](realtime.md) for details.
+promptly. See [Real-time Streams](realtime_streams.md) for details.

docs/framework_concepts/realtime_streams.md

@@ -1,29 +1,28 @@
 ---
 layout: default
-title: Processing real-time data streams
+title: Real-time Streams
+parent: Framework Concepts
 nav_order: 6
-has_children: true
-has_toc: false
 ---
 
-# Processing real-time data streams
+# Real-time Streams
 {: .no_toc }
 
 1. TOC
 {:toc}
 ---
 
-## Realtime timestamps
+## Real-time timestamps
 
 MediaPipe calculator graphs are often used to process streams of video or audio
 frames for interactive applications. The MediaPipe framework requires only that
 successive packets be assigned monotonically increasing timestamps. By
-convention, realtime calculators and graphs use the recording time or the
+convention, real-time calculators and graphs use the recording time or the
 presentation time of each frame as its timestamp, with each timestamp indicating
 the microseconds since `Jan/1/1970:00:00:00`. This allows packets from various
 sources to be processed in a globally consistent sequence.
 
-## Realtime scheduling
+## Real-time scheduling
 
 Normally, each Calculator runs as soon as all of its input packets for a given
 timestamp become available. Normally, this happens when the calculator has
@@ -38,7 +37,7 @@ When a calculator does not produce any output packets for a given timestamp, it
 can instead output a "timestamp bound" indicating that no packet will be
 produced for that timestamp. This indication is necessary to allow downstream
 calculators to run at that timestamp, even though no packet has arrived for
-certain streams for that timestamp. This is especially important for realtime
+certain streams for that timestamp. This is especially important for real-time
 graphs in interactive applications, where it is crucial that each calculator
 begin processing as soon as possible.
 
@@ -83,12 +82,12 @@ For example, `Timestamp(1).NextAllowedInStream() == Timestamp(2)`.
 
 ## Propagating timestamp bounds
 
-Calculators that will be used in realtime graphs need to define output timestamp
-bounds based on input timestamp bounds in order to allow downstream calculators
-to be scheduled promptly. A common pattern is for calculators to output packets
-with the same timestamps as their input packets. In this case, simply outputting
-a packet on every call to `Calculator::Process` is sufficient to define output
-timestamp bounds.
+Calculators that will be used in real-time graphs need to define output
+timestamp bounds based on input timestamp bounds in order to allow downstream
+calculators to be scheduled promptly. A common pattern is for calculators to
+output packets with the same timestamps as their input packets. In this case,
+simply outputting a packet on every call to `Calculator::Process` is sufficient
+to define output timestamp bounds.
 
 However, calculators are not required to follow this common pattern for output
 timestamps, they are only required to choose monotonically increasing output

docs/getting_started/android_archive_library.md

@@ -92,12 +92,12 @@ each project.
     and copy
     [the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41)
     and
-    [the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite).
+    [the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite).
 
     ```bash
     bazel build -c opt mediapipe/graphs/face_detection:face_detection_mobile_gpu_binary_graph
     cp bazel-bin/mediapipe/graphs/face_detection/face_detection_mobile_gpu.binarypb /path/to/your/app/src/main/assets/
-    cp mediapipe/modules/face_detection/face_detection_front.tflite /path/to/your/app/src/main/assets/
+    cp mediapipe/modules/face_detection/face_detection_short_range.tflite /path/to/your/app/src/main/assets/
     ```
 
     ![Screenshot](../images/mobile/assets_location.png)
@@ -117,7 +117,6 @@ each project.
         implementation 'com.google.flogger:flogger-system-backend:0.3.1'
         implementation 'com.google.code.findbugs:jsr305:3.0.2'
         implementation 'com.google.guava:guava:27.0.1-android'
-        implementation 'com.google.guava:guava:27.0.1-android'
         implementation 'com.google.protobuf:protobuf-java:3.11.4'
         // CameraX core library
         def camerax_version = "1.0.0-beta10"
@@ -125,7 +124,7 @@ each project.
         implementation "androidx.camera:camera-camera2:$camerax_version"
         implementation "androidx.camera:camera-lifecycle:$camerax_version"
         // AutoValue
-        def auto_value_version = "1.6.4"
+        def auto_value_version = "1.8.1"
         implementation "com.google.auto.value:auto-value-annotations:$auto_value_version"
         annotationProcessor "com.google.auto.value:auto-value:$auto_value_version"
     }

docs/getting_started/javascript.md

@@ -16,13 +16,14 @@ nav_order: 4
 
 MediaPipe currently offers the following solutions:
 
-Solution          | NPM Package                   | Example
------------------ | ----------------------------- | -------
-[Face Mesh][F-pg] | [@mediapipe/face_mesh][F-npm] | [mediapipe.dev/demo/face_mesh][F-demo]
-[Face Detection][Fd-pg] | [@mediapipe/face_detection][Fd-npm] | [mediapipe.dev/demo/face_detection][Fd-demo]
-[Hands][H-pg]     | [@mediapipe/hands][H-npm]     | [mediapipe.dev/demo/hands][H-demo]
-[Holistic][Ho-pg] | [@mediapipe/holistic][Ho-npm] | [mediapipe.dev/demo/holistic][Ho-demo]
-[Pose][P-pg]      | [@mediapipe/pose][P-npm]      | [mediapipe.dev/demo/pose][P-demo]
+Solution                    | NPM Package                             | Example
+--------------------------- | --------------------------------------- | -------
+[Face Mesh][F-pg]           | [@mediapipe/face_mesh][F-npm]           | [mediapipe.dev/demo/face_mesh][F-demo]
+[Face Detection][Fd-pg]     | [@mediapipe/face_detection][Fd-npm]     | [mediapipe.dev/demo/face_detection][Fd-demo]
+[Hands][H-pg]               | [@mediapipe/hands][H-npm]               | [mediapipe.dev/demo/hands][H-demo]
+[Holistic][Ho-pg]           | [@mediapipe/holistic][Ho-npm]           | [mediapipe.dev/demo/holistic][Ho-demo]
+[Pose][P-pg]                | [@mediapipe/pose][P-npm]                | [mediapipe.dev/demo/pose][P-demo]
+[Selfie Segmentation][S-pg] | [@mediapipe/selfie_segmentation][S-npm] | [mediapipe.dev/demo/selfie_segmentation][S-demo]
 
 Click on a solution link above for more information, including API and code
 snippets.
@@ -67,11 +68,13 @@ affecting your work, restrict your request to a `<minor>` number. e.g.,
 [Fd-pg]: ../solutions/face_detection#javascript-solution-api
 [H-pg]: ../solutions/hands#javascript-solution-api
 [P-pg]: ../solutions/pose#javascript-solution-api
+[S-pg]: ../solutions/selfie_segmentation#javascript-solution-api
 [Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic
 [F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh
 [Fd-npm]: https://www.npmjs.com/package/@mediapipe/face_detection
 [H-npm]: https://www.npmjs.com/package/@mediapipe/hands
 [P-npm]: https://www.npmjs.com/package/@mediapipe/pose
+[S-npm]: https://www.npmjs.com/package/@mediapipe/selfie_segmentation
 [draw-npm]: https://www.npmjs.com/package/@mediapipe/drawing_utils
 [cam-npm]: https://www.npmjs.com/package/@mediapipe/camera_utils
 [ctrl-npm]: https://www.npmjs.com/package/@mediapipe/control_utils
@@ -80,15 +83,18 @@ affecting your work, restrict your request to a `<minor>` number. e.g.,
 [Fd-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/face_detection
 [H-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/hands
 [P-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/pose
+[P-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/selfie_segmentation
 [Ho-pen]: https://code.mediapipe.dev/codepen/holistic
 [F-pen]: https://code.mediapipe.dev/codepen/face_mesh
 [Fd-pen]: https://code.mediapipe.dev/codepen/face_detection
 [H-pen]: https://code.mediapipe.dev/codepen/hands
 [P-pen]: https://code.mediapipe.dev/codepen/pose
+[S-pen]: https://code.mediapipe.dev/codepen/selfie_segmentation
 [Ho-demo]: https://mediapipe.dev/demo/holistic
 [F-demo]: https://mediapipe.dev/demo/face_mesh
 [Fd-demo]: https://mediapipe.dev/demo/face_detection
 [H-demo]: https://mediapipe.dev/demo/hands
 [P-demo]: https://mediapipe.dev/demo/pose
+[S-demo]: https://mediapipe.dev/demo/selfie_segmentation
 [npm]: https://www.npmjs.com/package/@mediapipe
 [codepen]: https://code.mediapipe.dev/codepen

docs/getting_started/python.md

@@ -51,6 +51,7 @@ details in each solution via the links below:
 *   [MediaPipe Holistic](../solutions/holistic#python-solution-api)
 *   [MediaPipe Objectron](../solutions/objectron#python-solution-api)
 *   [MediaPipe Pose](../solutions/pose#python-solution-api)
+*   [MediaPipe Selfie Segmentation](../solutions/selfie_segmentation#python-solution-api)
 
 ## MediaPipe on Google Colab
 
@@ -62,6 +63,7 @@ details in each solution via the links below:
 *   [MediaPipe Pose Colab](https://mediapipe.page.link/pose_py_colab)
 *   [MediaPipe Pose Classification Colab (Basic)](https://mediapipe.page.link/pose_classification_basic)
 *   [MediaPipe Pose Classification Colab (Extended)](https://mediapipe.page.link/pose_classification_extended)
+*   [MediaPipe Selfie Segmentation Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)
 
 ## MediaPipe Python Framework
 

docs/index.md

@@ -40,11 +40,12 @@ Hair Segmentation
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                        | ✅                                                         | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
@@ -54,46 +55,22 @@ See also
 [MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
 for ML models released in MediaPipe.
 
-## MediaPipe in Python
-
-MediaPipe offers customizable Python solutions as a prebuilt Python package on
-[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
-`pip install mediapipe`. It also provides tools for users to build their own
-solutions. Please see
-[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
-for more info.
-
-## MediaPipe on the Web
-
-MediaPipe on the Web is an effort to run the same ML solutions built for mobile
-and desktop also in web browsers. The official API is under construction, but
-the core technology has been proven effective. Please see
-[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
-in Google Developers Blog for details.
-
-You can use the following links to load a demo in the MediaPipe Visualizer, and
-over there click the "Runner" icon in the top bar like shown below. The demos
-use your webcam video as input, which is processed all locally in real-time and
-never leaves your device.
-
-![visualizer_runner](images/visualizer_runner.png)
-
-*   [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
-*   [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
-*   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
-*   [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
-*   [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
-*   [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
-*   [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
-
 ## Getting started
 
-Learn how to [install](https://google.github.io/mediapipe/getting_started/install)
-MediaPipe and
-[build example applications](https://google.github.io/mediapipe/getting_started/building_examples),
-and start exploring our ready-to-use
-[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
-further extend and customize.
+To start using MediaPipe
+[solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
+lines code, see example code and demos in
+[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
+[MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
+
+To use MediaPipe in C++, Android and iOS, which allow further customization of
+the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
+building your own, learn how to
+[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
+start building example applications in
+[C++](https://google.github.io/mediapipe/getting_started/cpp),
+[Android](https://google.github.io/mediapipe/getting_started/android) and
+[iOS](https://google.github.io/mediapipe/getting_started/ios).
 
 The source code is hosted in the
 [MediaPipe Github repository](https://github.com/google/mediapipe), and you can

docs/solutions/autoflip.md

@@ -2,7 +2,7 @@
 layout: default
 title: AutoFlip (Saliency-aware Video Cropping)
 parent: Solutions
-nav_order: 13
+nav_order: 14
 ---
 
 # AutoFlip: Saliency-aware Video Cropping

docs/solutions/box_tracking.md

@@ -2,7 +2,7 @@
 layout: default
 title: Box Tracking
 parent: Solutions
-nav_order: 9
+nav_order: 10
 ---
 
 # MediaPipe Box Tracking

docs/solutions/face_detection.md

@@ -45,6 +45,15 @@ section.
 
 Naming style and availability may differ slightly across platforms/languages.
 
+#### model_selection
+
+An integer index `0` or `1`. Use `0` to select a short-range model that works
+best for faces within 2 meters from the camera, and `1` for a full-range model
+best for faces within 5 meters. For the full-range option, a sparse model is
+used for its improved inference speed. Please refer to the
+[model cards](./models.md#face_detection) for details. Default to `0` if not
+specified.
+
 #### min_detection_confidence
 
 Minimum confidence value (`[0.0, 1.0]`) from the face detection model for the
@@ -68,10 +77,11 @@ normalized to `[0.0, 1.0]` by the image width and height respectively.
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
+*   [model_selection](#model_selection)
 *   [min_detection_confidence](#min_detection_confidence)
 
 ```python
@@ -81,9 +91,10 @@ mp_face_detection = mp.solutions.face_detection
 mp_drawing = mp.solutions.drawing_utils
 
 # For static images:
+IMAGE_FILES = []
 with mp_face_detection.FaceDetection(
-    min_detection_confidence=0.5) as face_detection:
-  for idx, file in enumerate(file_list):
+    model_selection=1, min_detection_confidence=0.5) as face_detection:
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB and process it with MediaPipe Face Detection.
     results = face_detection.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
@@ -102,7 +113,7 @@ with mp_face_detection.FaceDetection(
 # For webcam input:
 cap = cv2.VideoCapture(0)
 with mp_face_detection.FaceDetection(
-    min_detection_confidence=0.5) as face_detection:
+    model_selection=0, min_detection_confidence=0.5) as face_detection:
   while cap.isOpened():
     success, image = cap.read()
     if not success:
@@ -138,6 +149,7 @@ and the following usage example.
 
 Supported configuration options:
 
+*   [modelSelection](#model_selection)
 *   [minDetectionConfidence](#min_detection_confidence)
 
 ```html
@@ -188,6 +200,7 @@ const faceDetection = new FaceDetection({locateFile: (file) => {
   return `https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/${file}`;
 }});
 faceDetection.setOptions({
+  modelSelection: 0
   minDetectionConfidence: 0.5
 });
 faceDetection.onResults(onResults);
@@ -254,10 +267,6 @@ same configuration as the GPU pipeline, runs entirely on CPU.
     *   Target:
         [`mediapipe/examples/desktop/face_detection:face_detection_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_detection/BUILD)
 
-### Web
-
-Please refer to [these instructions](../index.md#mediapipe-on-the-web).
-
 ### Coral
 
 Please refer to

docs/solutions/face_mesh.md

@@ -69,7 +69,7 @@ and renders using a dedicated
 The
 [face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
 internally uses a
-[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
+[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range_gpu.pbtxt)
 from the
 [face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
 
@@ -265,7 +265,7 @@ magnitude of `z` uses roughly the same scale as `x`.
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -281,12 +281,13 @@ mp_drawing = mp.solutions.drawing_utils
 mp_face_mesh = mp.solutions.face_mesh
 
 # For static images:
+IMAGE_FILES = []
 drawing_spec = mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
 with mp_face_mesh.FaceMesh(
     static_image_mode=True,
     max_num_faces=1,
     min_detection_confidence=0.5) as face_mesh:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB before processing.
     results = face_mesh.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

docs/solutions/hair_segmentation.md

@@ -2,7 +2,7 @@
 layout: default
 title: Hair Segmentation
 parent: Solutions
-nav_order: 7
+nav_order: 8
 ---
 
 # MediaPipe Hair Segmentation
@@ -51,7 +51,14 @@ to visualize its associated subgraphs, please see
 
 ### Web
 
-Please refer to [these instructions](../index.md#mediapipe-on-the-web).
+Use [this link](https://viz.mediapipe.dev/demo/hair_segmentation) to load a demo
+in the MediaPipe Visualizer, and over there click the "Runner" icon in the top
+bar like shown below. The demos use your webcam video as input, which is
+processed all locally in real-time and never leaves your device. Please see
+[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
+in Google Developers Blog for details.
+
+![visualizer_runner](../images/visualizer_runner.png)
 
 ## Resources
 

docs/solutions/hands.md

@@ -206,7 +206,7 @@ is not the case, please swap the handedness output in the application.
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -222,11 +222,12 @@ mp_drawing = mp.solutions.drawing_utils
 mp_hands = mp.solutions.hands
 
 # For static images:
+IMAGE_FILES = []
 with mp_hands.Hands(
     static_image_mode=True,
     max_num_hands=2,
     min_detection_confidence=0.5) as hands:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     # Read an image, flip it around y-axis for correct handedness output (see
     # above).
     image = cv2.flip(cv2.imread(file), 1)

docs/solutions/holistic.md

@@ -176,6 +176,16 @@ A list of pose landmarks. Each landmark consists of the following:
 *   `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
     landmark being visible (present and not occluded) in the image.
 
+#### pose_world_landmarks
+
+Another list of pose landmarks in world coordinates. Each landmark consists of
+the following:
+
+*   `x`, `y` and `z`: Real-world 3D coordinates in meters with the origin at the
+    center between hips.
+*   `visibility`: Identical to that defined in the corresponding
+    [pose_landmarks](#pose_landmarks).
+
 #### face_landmarks
 
 A list of 468 face landmarks. Each landmark consists of `x`, `y` and `z`. `x`
@@ -201,7 +211,7 @@ A list of 21 hand landmarks on the right hand, in the same representation as
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -218,10 +228,11 @@ mp_drawing = mp.solutions.drawing_utils
 mp_holistic = mp.solutions.holistic
 
 # For static images:
+IMAGE_FILES = []
 with mp_holistic.Holistic(
     static_image_mode=True,
     model_complexity=2) as holistic:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     image_height, image_width, _ = image.shape
     # Convert the BGR image to RGB before processing.
@@ -244,6 +255,9 @@ with mp_holistic.Holistic(
     mp_drawing.draw_landmarks(
         annotated_image, results.pose_landmarks, mp_holistic.POSE_CONNECTIONS)
     cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
+    # Plot pose world landmarks.
+    mp_drawing.plot_landmarks(
+        results.pose_world_landmarks, mp_holistic.POSE_CONNECTIONS)
 
 # For webcam input:
 cap = cv2.VideoCapture(0)

docs/solutions/instant_motion_tracking.md

@@ -2,7 +2,7 @@
 layout: default
 title: Instant Motion Tracking
 parent: Solutions
-nav_order: 10
+nav_order: 11
 ---
 
 # MediaPipe Instant Motion Tracking

docs/solutions/iris.md

@@ -69,7 +69,7 @@ and renders using a dedicated
 The
 [face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
 internally uses a
-[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
+[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range_gpu.pbtxt)
 from the
 [face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
 
@@ -193,7 +193,17 @@ on how to build MediaPipe examples.
 
 ### Web
 
-Please refer to [these instructions](../index.md#mediapipe-on-the-web).
+You can use the following links to load a demo in the MediaPipe Visualizer, and
+over there click the "Runner" icon in the top bar like shown below. The demos
+use your webcam video as input, which is processed all locally in real-time and
+never leaves your device. Please see
+[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
+in Google Developers Blog for details.
+
+![visualizer_runner](../images/visualizer_runner.png)
+
+*   [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
+*   [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
 
 ## Resources
 

docs/solutions/knift.md

@@ -2,7 +2,7 @@
 layout: default
 title: KNIFT (Template-based Feature Matching)
 parent: Solutions
-nav_order: 12
+nav_order: 13
 ---
 
 # MediaPipe KNIFT

docs/solutions/media_sequence.md

@@ -2,7 +2,7 @@
 layout: default
 title: Dataset Preparation with MediaSequence
 parent: Solutions
-nav_order: 14
+nav_order: 15
 ---
 
 # Dataset Preparation with MediaSequence

docs/solutions/models.md

@@ -14,12 +14,27 @@ nav_order: 30
 
 ### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
 
-*   Face detection model for front-facing/selfie camera:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite),
-    [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
-*   Face detection model for back-facing camera:
-    [TFLite model ](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back.tflite)
-*   [Model card](https://mediapipe.page.link/blazeface-mc)
+*   Short-range model (best for faces within 2 meters from the camera):
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite),
+    [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite),
+    [Model card](https://mediapipe.page.link/blazeface-mc)
+*   Full-range model (dense, best for faces within 5 meters from the camera):
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range.tflite),
+    [Model card](https://mediapipe.page.link/blazeface-back-mc)
+*   Full-range model (sparse, best for faces within 5 meters from the camera):
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite),
+    [Model card](https://mediapipe.page.link/blazeface-back-sparse-mc)
+
+Full-range dense and sparse models have the same quality in terms of
+[F-score](https://en.wikipedia.org/wiki/F-score) however differ in underlying
+metrics. The dense model is slightly better in
+[Recall](https://en.wikipedia.org/wiki/Precision_and_recall) whereas the sparse
+model outperforms the dense one in
+[Precision](https://en.wikipedia.org/wiki/Precision_and_recall). Speed-wise
+sparse model is ~30% faster when executing on CPU via
+[XNNPACK](https://github.com/google/XNNPACK) whereas on GPU the models
+demonstrate comparable latencies. Depending on your application, you may prefer
+one over the other.
 
 ### [Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)
 
@@ -60,6 +75,12 @@ nav_order: 30
 *   Hand recrop model:
     [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/holistic_landmark/hand_recrop.tflite)
 
+### [Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)
+
+*   [TFLite model (general)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation.tflite)
+*   [TFLite model (landscape)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation_landscape.tflite)
+*   [Model card](https://mediapipe.page.link/selfiesegmentation-mc)
+
 ### [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)
 
 *   [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite)

docs/solutions/object_detection.md

@@ -2,7 +2,7 @@
 layout: default
 title: Object Detection
 parent: Solutions
-nav_order: 8
+nav_order: 9
 ---
 
 # MediaPipe Object Detection

docs/solutions/objectron.md

@@ -2,7 +2,7 @@
 layout: default
 title: Objectron (3D Object Detection)
 parent: Solutions
-nav_order: 11
+nav_order: 12
 ---
 
 # MediaPipe Objectron
@@ -277,7 +277,7 @@ following:
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -297,11 +297,12 @@ mp_drawing = mp.solutions.drawing_utils
 mp_objectron = mp.solutions.objectron
 
 # For static images:
+IMAGE_FILES = []
 with mp_objectron.Objectron(static_image_mode=True,
                             max_num_objects=5,
                             min_detection_confidence=0.5,
                             model_name='Shoe') as objectron:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB and process it with MediaPipe Objectron.
     results = objectron.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

docs/solutions/pose.md

@@ -187,22 +187,35 @@ Naming style may differ slightly across platforms/languages.
 
 #### pose_landmarks
 
-A list of pose landmarks. Each lanmark consists of the following:
+A list of pose landmarks. Each landmark consists of the following:
 
 *   `x` and `y`: Landmark coordinates normalized to `[0.0, 1.0]` by the image
     width and height respectively.
 *   `z`: Represents the landmark depth with the depth at the midpoint of hips
     being the origin, and the smaller the value the closer the landmark is to
     the camera. The magnitude of `z` uses roughly the same scale as `x`.
-
 *   `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
     landmark being visible (present and not occluded) in the image.
 
+#### pose_world_landmarks
+
+*Fig 5. Example of MediaPipe Pose real-world 3D coordinates.* |
+:-----------------------------------------------------------: |
+<video autoplay muted loop preload style="height: auto; width: 480px"><source src="../images/mobile/pose_world_landmarks.mp4" type="video/mp4"></video> |
+
+Another list of pose landmarks in world coordinates. Each landmark consists of
+the following:
+
+*   `x`, `y` and `z`: Real-world 3D coordinates in meters with the origin at the
+    center between hips.
+*   `visibility`: Identical to that defined in the corresponding
+    [pose_landmarks](#pose_landmarks).
+
 ### Python Solution API
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -219,11 +232,12 @@ mp_drawing = mp.solutions.drawing_utils
 mp_pose = mp.solutions.pose
 
 # For static images:
+IMAGE_FILES = []
 with mp_pose.Pose(
     static_image_mode=True,
     model_complexity=2,
     min_detection_confidence=0.5) as pose:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     image_height, image_width, _ = image.shape
     # Convert the BGR image to RGB before processing.
@@ -241,6 +255,9 @@ with mp_pose.Pose(
     mp_drawing.draw_landmarks(
         annotated_image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
     cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
+    # Plot pose world landmarks.
+    mp_drawing.plot_landmarks(
+        results.pose_world_landmarks, mp_pose.POSE_CONNECTIONS)
 
 # For webcam input:
 cap = cv2.VideoCapture(0)
@@ -293,6 +310,7 @@ Supported configuration options:
   <meta charset="utf-8">
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/control_utils_3d.js" crossorigin="anonymous"></script>
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
   <script src="https://cdn.jsdelivr.net/npm/@mediapipe/pose/pose.js" crossorigin="anonymous"></script>
 </head>
@@ -311,8 +329,15 @@ Supported configuration options:
 const videoElement = document.getElementsByClassName('input_video')[0];
 const canvasElement = document.getElementsByClassName('output_canvas')[0];
 const canvasCtx = canvasElement.getContext('2d');
+const landmarkContainer = document.getElementsByClassName('landmark-grid-container')[0];
+const grid = new LandmarkGrid(landmarkContainer);
 
 function onResults(results) {
+  if (!results.poseLandmarks) {
+    grid.updateLandmarks([]);
+    return;
+  }
+
   canvasCtx.save();
   canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
   canvasCtx.drawImage(
@@ -322,6 +347,8 @@ function onResults(results) {
   drawLandmarks(canvasCtx, results.poseLandmarks,
                 {color: '#FF0000', lineWidth: 2});
   canvasCtx.restore();
+
+  grid.updateLandmarks(results.poseWorldLandmarks);
 }
 
 const pose = new Pose({locateFile: (file) => {
@@ -399,4 +426,4 @@ on how to build MediaPipe examples.
 *   [Python Colab](https://mediapipe.page.link/pose_py_colab)
 
 [`mAP`]: https://cocodataset.org/#keypoints-eval
-[`[email protected]`]: https\://github.com/cbsudux/Human-Pose-Estimation-101
+[`[email protected]`]: https://github.com/cbsudux/Human-Pose-Estimation-101

docs/solutions/selfie_segmentation.md

@@ -0,0 +1,289 @@
+---
+layout: default
+title: Selfie Segmentation
+parent: Solutions
+nav_order: 7
+---
+
+# MediaPipe Selfie Segmentation
+{: .no_toc }
+
+<details close markdown="block">
+  <summary>
+    Table of contents
+  </summary>
+  {: .text-delta }
+1. TOC
+{:toc}
+</details>
+---
+
+## Overview
+
+*Fig 1. Example of MediaPipe Selfie Segmentation.* |
+:------------------------------------------------: |
+<video autoplay muted loop preload style="height: auto; width: 480px"><source src="../images/selfie_segmentation_web.mp4" type="video/mp4"></video> |
+
+MediaPipe Selfie Segmentation segments the prominent humans in the scene. It can
+run in real-time on both smartphones and laptops. The intended use cases include
+selfie effects and video conferencing, where the person is close (< 2m) to the
+camera.
+
+## Models
+
+In this solution, we provide two models: general and landscape. Both models are
+based on
+[MobileNetV3](https://ai.googleblog.com/2019/11/introducing-next-generation-on-device.html),
+with modifications to make them more efficient. The general model operates on a
+256x256x3 (HWC) tensor, and outputs a 256x256x1 tensor representing the
+segmentation mask. The landscape model is similar to the general model, but
+operates on a 144x256x3 (HWC) tensor. It has fewer FLOPs than the general model,
+and therefore, runs faster. Note that MediaPipe Selfie Segmentation
+automatically resizes the input image to the desired tensor dimension before
+feeding it into the ML models.
+
+The general model is also powering
+[ML Kit](https://developers.google.com/ml-kit/vision/selfie-segmentation), and a
+variant of the landscape model is powering
+[Google Meet](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html).
+Please find more detail about the models in the
+[model card](./models.md#selfie-segmentation).
+
+## ML Pipeline
+
+The pipeline is implemented as a MediaPipe
+[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+that uses a
+[selfie segmentation subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+from the
+[selfie segmentation module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation).
+
+Note: To visualize a graph, copy the graph and paste it into
+[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
+to visualize its associated subgraphs, please see
+[visualizer documentation](../tools/visualizer.md).
+
+## Solution APIs
+
+### Cross-platform Configuration Options
+
+Naming style and availability may differ slightly across platforms/languages.
+
+#### model_selection
+
+An integer index `0` or `1`. Use `0` to select the general model, and `1` to
+select the landscape model (see details in [Models](#models)). Default to `0` if
+not specified.
+
+### Output
+
+Naming style may differ slightly across platforms/languages.
+
+#### segmentation_mask
+
+The output segmentation mask, which has the same dimension as the input image.
+
+### Python Solution API
+
+Please first follow general [instructions](../getting_started/python.md) to
+install MediaPipe Python package, then learn more in the companion
+[Python Colab](#resources) and the usage example below.
+
+Supported configuration options:
+
+*   [model_selection](#model_selection)
+
+```python
+import cv2
+import mediapipe as mp
+mp_drawing = mp.solutions.drawing_utils
+mp_selfie_segmentation = mp.solutions.selfie_segmentation
+
+# For static images:
+IMAGE_FILES = []
+BG_COLOR = (192, 192, 192) # gray
+MASK_COLOR = (255, 255, 255) # white
+with mp_selfie_segmentation.SelfieSegmentation(
+    model_selection=0) as selfie_segmentation:
+  for idx, file in enumerate(IMAGE_FILES):
+    image = cv2.imread(file)
+    image_height, image_width, _ = image.shape
+    # Convert the BGR image to RGB before processing.
+    results = selfie_segmentation.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
+
+    # Draw selfie segmentation on the background image.
+    # To improve segmentation around boundaries, consider applying a joint
+    # bilateral filter to "results.segmentation_mask" with "image".
+    condition = np.stack((results.segmentation_mask,) * 3, axis=-1) > 0.1
+    # Generate solid color images for showing the output selfie segmentation mask.
+    fg_image = np.zeros(image.shape, dtype=np.uint8)
+    fg_image[:] = MASK_COLOR
+    bg_image = np.zeros(image.shape, dtype=np.uint8)
+    bg_image[:] = BG_COLOR
+    output_image = np.where(condition, fg_image, bg_image)
+    cv2.imwrite('/tmp/selfie_segmentation_output' + str(idx) + '.png', output_image)
+
+# For webcam input:
+BG_COLOR = (192, 192, 192) # gray
+cap = cv2.VideoCapture(0)
+with mp_selfie_segmentation.SelfieSegmentation(
+    model_selection=1) as selfie_segmentation:
+  bg_image = None
+  while cap.isOpened():
+    success, image = cap.read()
+    if not success:
+      print("Ignoring empty camera frame.")
+      # If loading a video, use 'break' instead of 'continue'.
+      continue
+
+    # Flip the image horizontally for a later selfie-view display, and convert
+    # the BGR image to RGB.
+    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
+    # To improve performance, optionally mark the image as not writeable to
+    # pass by reference.
+    image.flags.writeable = False
+    results = selfie_segmentation.process(image)
+
+    image.flags.writeable = True
+    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
+
+    # Draw selfie segmentation on the background image.
+    # To improve segmentation around boundaries, consider applying a joint
+    # bilateral filter to "results.segmentation_mask" with "image".
+    condition = np.stack(
+      (results.segmentation_mask,) * 3, axis=-1) > 0.1
+    # The background can be customized.
+    #   a) Load an image (with the same width and height of the input image) to
+    #      be the background, e.g., bg_image = cv2.imread('/path/to/image/file')
+    #   b) Blur the input image by applying image filtering, e.g.,
+    #      bg_image = cv2.GaussianBlur(image,(55,55),0)
+    if bg_image is None:
+      bg_image = np.zeros(image.shape, dtype=np.uint8)
+      bg_image[:] = BG_COLOR
+    output_image = np.where(condition, image, bg_image)
+
+    cv2.imshow('MediaPipe Selfie Segmentation', output_image)
+    if cv2.waitKey(5) & 0xFF == 27:
+      break
+cap.release()
+```
+
+### JavaScript Solution API
+
+Please first see general [introduction](../getting_started/javascript.md) on
+MediaPipe in JavaScript, then learn more in the companion [web demo](#resources)
+and the following usage example.
+
+Supported configuration options:
+
+*   [modelSelection](#model_selection)
+
+```html
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation.js" crossorigin="anonymous"></script>
+</head>
+
+<body>
+  <div class="container">
+    <video class="input_video"></video>
+    <canvas class="output_canvas" width="1280px" height="720px"></canvas>
+  </div>
+</body>
+</html>
+```
+
+```javascript
+<script type="module">
+const videoElement = document.getElementsByClassName('input_video')[0];
+const canvasElement = document.getElementsByClassName('output_canvas')[0];
+const canvasCtx = canvasElement.getContext('2d');
+
+function onResults(results) {
+  canvasCtx.save();
+  canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
+  canvasCtx.drawImage(results.segmentationMask, 0, 0,
+                      canvasElement.width, canvasElement.height);
+
+  // Only overwrite existing pixels.
+  canvasCtx.globalCompositeOperation = 'source-in';
+  canvasCtx.fillStyle = '#00FF00';
+  canvasCtx.fillRect(0, 0, canvasElement.width, canvasElement.height);
+
+  // Only overwrite missing pixels.
+  canvasCtx.globalCompositeOperation = 'destination-atop';
+  canvasCtx.drawImage(
+      results.image, 0, 0, canvasElement.width, canvasElement.height);
+
+  canvasCtx.restore();
+}
+
+const selfieSegmentation = new SelfieSegmentation({locateFile: (file) => {
+  return `https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/${file}`;
+}});
+selfieSegmentation.setOptions({
+  modelSelection: 1,
+});
+selfieSegmentation.onResults(onResults);
+
+const camera = new Camera(videoElement, {
+  onFrame: async () => {
+    await selfieSegmentation.send({image: videoElement});
+  },
+  width: 1280,
+  height: 720
+});
+camera.start();
+</script>
+```
+
+## Example Apps
+
+Please first see general instructions for
+[Android](../getting_started/android.md), [iOS](../getting_started/ios.md), and
+[desktop](../getting_started/cpp.md) on how to build MediaPipe examples.
+
+Note: To visualize a graph, copy the graph and paste it into
+[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
+to visualize its associated subgraphs, please see
+[visualizer documentation](../tools/visualizer.md).
+
+### Mobile
+
+*   Graph:
+    [`mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+*   Android target:
+    [(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1DoeyGzMmWUsjfVgZfGGecrn7GKzYcEAo/view?usp=sharing)
+    [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/selfiesegmentationgpu:selfiesegmentationgpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/selfiesegmentationgpu/BUILD)
+*   iOS target:
+    [`mediapipe/examples/ios/selfiesegmentationgpu:SelfieSegmentationGpuApp`](http:/mediapipe/examples/ios/selfiesegmentationgpu/BUILD)
+
+### Desktop
+
+Please first see general instructions for [desktop](../getting_started/cpp.md)
+on how to build MediaPipe examples.
+
+*   Running on CPU
+    *   Graph:
+        [`mediapipe/graphs/selfie_segmentation/selfie_segmentation_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_cpu.pbtxt)
+    *   Target:
+        [`mediapipe/examples/desktop/selfie_segmentation:selfie_segmentation_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/selfie_segmentation/BUILD)
+*   Running on GPU
+    *   Graph:
+        [`mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+    *   Target:
+        [`mediapipe/examples/desktop/selfie_segmentation:selfie_segmentation_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/selfie_segmentation/BUILD)
+
+## Resources
+
+*   Google AI Blog:
+    [Background Features in Google Meet, Powered by Web ML](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html)
+*   [ML Kit Selfie Segmentation API](https://developers.google.com/ml-kit/vision/selfie-segmentation)
+*   [Models and model cards](./models.md#selfie-segmentation)
+*   [Web demo](https://code.mediapipe.dev/codepen/selfie_segmentation)
+*   [Python Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)

docs/solutions/solutions.md

@@ -24,11 +24,12 @@ has_toc: false
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                        | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |

docs/solutions/youtube_8m.md

@@ -2,7 +2,7 @@
 layout: default
 title: YouTube-8M Feature Extraction and Model Inference
 parent: Solutions
-nav_order: 15
+nav_order: 16
 ---
 
 # YouTube-8M Feature Extraction and Model Inference

mediapipe/MediaPipe.tulsiproj/Configs/MediaPipe.tulsigen

@@ -16,6 +16,7 @@
     "mediapipe/examples/ios/objectdetectiongpu/BUILD",
     "mediapipe/examples/ios/objectdetectiontrackinggpu/BUILD",
     "mediapipe/examples/ios/posetrackinggpu/BUILD",
+    "mediapipe/examples/ios/selfiesegmentationgpu/BUILD",
     "mediapipe/framework/BUILD",
     "mediapipe/gpu/BUILD",
     "mediapipe/objc/BUILD",
@@ -35,6 +36,7 @@
     "//mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp",
     "//mediapipe/examples/ios/objectdetectiontrackinggpu:ObjectDetectionTrackingGpuApp",
     "//mediapipe/examples/ios/posetrackinggpu:PoseTrackingGpuApp",
+    "//mediapipe/examples/ios/selfiesegmentationgpu:SelfieSegmentationGpuApp",
     "//mediapipe/objc:mediapipe_framework_ios"
   ],
   "optionSet" : {
@@ -103,6 +105,7 @@
     "mediapipe/examples/ios/objectdetectioncpu",
     "mediapipe/examples/ios/objectdetectiongpu",
     "mediapipe/examples/ios/posetrackinggpu",
+    "mediapipe/examples/ios/selfiesegmentationgpu",
     "mediapipe/framework",
     "mediapipe/framework/deps",
     "mediapipe/framework/formats",
@@ -120,6 +123,7 @@
     "mediapipe/graphs/hand_tracking",
     "mediapipe/graphs/object_detection",
     "mediapipe/graphs/pose_tracking",
+    "mediapipe/graphs/selfie_segmentation",
     "mediapipe/models",
     "mediapipe/modules",
     "mediapipe/objc",

mediapipe/MediaPipe.tulsiproj/project.tulsiconf

@@ -22,6 +22,7 @@
     "mediapipe/examples/ios/objectdetectiongpu",
     "mediapipe/examples/ios/objectdetectiontrackinggpu",
     "mediapipe/examples/ios/posetrackinggpu",
+    "mediapipe/examples/ios/selfiesegmentationgpu",
     "mediapipe/objc"
   ],
   "projectName" : "Mediapipe",

mediapipe/calculators/core/BUILD

@@ -419,6 +419,23 @@ cc_library(
     alwayslink = 1,
 )
 
+cc_test(
+    name = "make_pair_calculator_test",
+    size = "small",
+    srcs = ["make_pair_calculator_test.cc"],
+    deps = [
+        ":make_pair_calculator",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:calculator_runner",
+        "//mediapipe/framework:timestamp",
+        "//mediapipe/framework/port:gtest_main",
+        "//mediapipe/framework/port:status",
+        "//mediapipe/framework/tool:validate_type",
+        "//mediapipe/util:packet_test_util",
+        "//mediapipe/util:time_series_test_util",
+    ],
+)
+
 cc_library(
     name = "matrix_multiply_calculator",
     srcs = ["matrix_multiply_calculator.cc"],
@@ -933,8 +950,8 @@ cc_test(
 )
 
 cc_library(
-    name = "split_normalized_landmark_list_calculator",
-    srcs = ["split_normalized_landmark_list_calculator.cc"],
+    name = "split_landmarks_calculator",
+    srcs = ["split_landmarks_calculator.cc"],
     visibility = ["//visibility:public"],
     deps = [
         ":split_vector_calculator_cc_proto",
@@ -948,10 +965,10 @@ cc_library(
 )
 
 cc_test(
-    name = "split_normalized_landmark_list_calculator_test",
-    srcs = ["split_normalized_landmark_list_calculator_test.cc"],
+    name = "split_landmarks_calculator_test",
+    srcs = ["split_landmarks_calculator_test.cc"],
     deps = [
-        ":split_normalized_landmark_list_calculator",
+        ":split_landmarks_calculator",
         ":split_vector_calculator_cc_proto",
         "//mediapipe/framework:calculator_framework",
         "//mediapipe/framework:calculator_runner",

mediapipe/calculators/core/end_loop_calculator.cc

@@ -28,6 +28,10 @@ typedef EndLoopCalculator<std::vector<::mediapipe::NormalizedRect>>
     EndLoopNormalizedRectCalculator;
 REGISTER_CALCULATOR(EndLoopNormalizedRectCalculator);
 
+typedef EndLoopCalculator<std::vector<::mediapipe::LandmarkList>>
+    EndLoopLandmarkListVectorCalculator;
+REGISTER_CALCULATOR(EndLoopLandmarkListVectorCalculator);
+
 typedef EndLoopCalculator<std::vector<::mediapipe::NormalizedLandmarkList>>
     EndLoopNormalizedLandmarkListVectorCalculator;
 REGISTER_CALCULATOR(EndLoopNormalizedLandmarkListVectorCalculator);

mediapipe/calculators/core/make_pair_calculator_test.cc

@@ -0,0 +1,70 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_runner.h"
+#include "mediapipe/framework/port/canonical_errors.h"
+#include "mediapipe/framework/port/gmock.h"
+#include "mediapipe/framework/port/gtest.h"
+#include "mediapipe/framework/port/status.h"
+#include "mediapipe/framework/port/status_matchers.h"
+#include "mediapipe/framework/timestamp.h"
+#include "mediapipe/framework/tool/validate_type.h"
+#include "mediapipe/util/packet_test_util.h"
+#include "mediapipe/util/time_series_test_util.h"
+
+namespace mediapipe {
+
+class MakePairCalculatorTest
+    : public mediapipe::TimeSeriesCalculatorTest<mediapipe::NoOptions> {
+ protected:
+  void SetUp() override {
+    calculator_name_ = "MakePairCalculator";
+    num_input_streams_ = 2;
+  }
+};
+
+TEST_F(MakePairCalculatorTest, ProducesExpectedPairs) {
+  InitializeGraph();
+  AppendInputPacket(new std::string("first packet"), Timestamp(1),
+                    /* input_index= */ 0);
+  AppendInputPacket(new std::string("second packet"), Timestamp(5),
+                    /* input_index= */ 0);
+  AppendInputPacket(new int(10), Timestamp(1), /* input_index= */ 1);
+  AppendInputPacket(new int(20), Timestamp(5), /* input_index= */ 1);
+
+  MP_ASSERT_OK(RunGraph());
+
+  EXPECT_THAT(
+      output().packets,
+      ::testing::ElementsAre(
+          mediapipe::PacketContainsTimestampAndPayload<
+              std::pair<Packet, Packet>>(
+              Timestamp(1),
+              ::testing::Pair(
+                  mediapipe::PacketContainsTimestampAndPayload<std::string>(
+                      Timestamp(1), std::string("first packet")),
+                  mediapipe::PacketContainsTimestampAndPayload<int>(
+                      Timestamp(1), 10))),
+          mediapipe::PacketContainsTimestampAndPayload<
+              std::pair<Packet, Packet>>(
+              Timestamp(5),
+              ::testing::Pair(
+                  mediapipe::PacketContainsTimestampAndPayload<std::string>(
+                      Timestamp(5), std::string("second packet")),
+                  mediapipe::PacketContainsTimestampAndPayload<int>(
+                      Timestamp(5), 20)))));
+}
+
+}  // namespace mediapipe

mediapipe/calculators/core/split_landmarks_calculator.cc

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-#ifndef MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_  // NOLINT
-#define MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_  // NOLINT
+#ifndef MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_  // NOLINT
+#define MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_  // NOLINT
 
 #include "mediapipe/calculators/core/split_vector_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
@@ -24,29 +24,30 @@
 
 namespace mediapipe {
 
-// Splits an input packet with NormalizedLandmarkList into
-// multiple NormalizedLandmarkList output packets using the [begin, end) ranges
+// Splits an input packet with LandmarkListType into
+// multiple LandmarkListType output packets using the [begin, end) ranges
 // specified in SplitVectorCalculatorOptions. If the option "element_only" is
 // set to true, all ranges should be of size 1 and all outputs will be elements
-// of type NormalizedLandmark. If "element_only" is false, ranges can be
-// non-zero in size and all outputs will be of type NormalizedLandmarkList.
+// of type LandmarkType. If "element_only" is false, ranges can be
+// non-zero in size and all outputs will be of type LandmarkListType.
 // If the option "combine_outputs" is set to true, only one output stream can be
 // specified and all ranges of elements will be combined into one
-// NormalizedLandmarkList.
-class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
+// LandmarkListType.
+template <typename LandmarkType, typename LandmarkListType>
+class SplitLandmarksCalculator : public CalculatorBase {
  public:
   static absl::Status GetContract(CalculatorContract* cc) {
     RET_CHECK(cc->Inputs().NumEntries() == 1);
     RET_CHECK(cc->Outputs().NumEntries() != 0);
 
-    cc->Inputs().Index(0).Set<NormalizedLandmarkList>();
+    cc->Inputs().Index(0).Set<LandmarkListType>();
 
     const auto& options =
         cc->Options<::mediapipe::SplitVectorCalculatorOptions>();
 
     if (options.combine_outputs()) {
       RET_CHECK_EQ(cc->Outputs().NumEntries(), 1);
-      cc->Outputs().Index(0).Set<NormalizedLandmarkList>();
+      cc->Outputs().Index(0).Set<LandmarkListType>();
       for (int i = 0; i < options.ranges_size() - 1; ++i) {
         for (int j = i + 1; j < options.ranges_size(); ++j) {
           const auto& range_0 = options.ranges(i);
@@ -81,9 +82,9 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
             return absl::InvalidArgumentError(
                 "Since element_only is true, all ranges should be of size 1.");
           }
-          cc->Outputs().Index(i).Set<NormalizedLandmark>();
+          cc->Outputs().Index(i).Set<LandmarkType>();
         } else {
-          cc->Outputs().Index(i).Set<NormalizedLandmarkList>();
+          cc->Outputs().Index(i).Set<LandmarkListType>();
         }
       }
     }
@@ -110,40 +111,39 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
   }
 
   absl::Status Process(CalculatorContext* cc) override {
-    const NormalizedLandmarkList& input =
-        cc->Inputs().Index(0).Get<NormalizedLandmarkList>();
+    const LandmarkListType& input =
+        cc->Inputs().Index(0).Get<LandmarkListType>();
     RET_CHECK_GE(input.landmark_size(), max_range_end_)
         << "Max range end " << max_range_end_ << " exceeds landmarks size "
         << input.landmark_size();
 
     if (combine_outputs_) {
-      NormalizedLandmarkList output;
+      LandmarkListType output;
       for (int i = 0; i < ranges_.size(); ++i) {
         for (int j = ranges_[i].first; j < ranges_[i].second; ++j) {
-          const NormalizedLandmark& input_landmark = input.landmark(j);
+          const LandmarkType& input_landmark = input.landmark(j);
           *output.add_landmark() = input_landmark;
         }
       }
       RET_CHECK_EQ(output.landmark_size(), total_elements_);
       cc->Outputs().Index(0).AddPacket(
-          MakePacket<NormalizedLandmarkList>(output).At(cc->InputTimestamp()));
+          MakePacket<LandmarkListType>(output).At(cc->InputTimestamp()));
     } else {
       if (element_only_) {
         for (int i = 0; i < ranges_.size(); ++i) {
           cc->Outputs().Index(i).AddPacket(
-              MakePacket<NormalizedLandmark>(input.landmark(ranges_[i].first))
+              MakePacket<LandmarkType>(input.landmark(ranges_[i].first))
                   .At(cc->InputTimestamp()));
         }
       } else {
         for (int i = 0; i < ranges_.size(); ++i) {
-          NormalizedLandmarkList output;
+          LandmarkListType output;
           for (int j = ranges_[i].first; j < ranges_[i].second; ++j) {
-            const NormalizedLandmark& input_landmark = input.landmark(j);
+            const LandmarkType& input_landmark = input.landmark(j);
             *output.add_landmark() = input_landmark;
           }
           cc->Outputs().Index(i).AddPacket(
-              MakePacket<NormalizedLandmarkList>(output).At(
-                  cc->InputTimestamp()));
+              MakePacket<LandmarkListType>(output).At(cc->InputTimestamp()));
         }
       }
     }
@@ -159,9 +159,15 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
   bool combine_outputs_ = false;
 };
 
+typedef SplitLandmarksCalculator<NormalizedLandmark, NormalizedLandmarkList>
+    SplitNormalizedLandmarkListCalculator;
 REGISTER_CALCULATOR(SplitNormalizedLandmarkListCalculator);
 
+typedef SplitLandmarksCalculator<Landmark, LandmarkList>
+    SplitLandmarkListCalculator;
+REGISTER_CALCULATOR(SplitLandmarkListCalculator);
+
 }  // namespace mediapipe
 
 // NOLINTNEXTLINE
-#endif  // MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_
+#endif  // MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_

mediapipe/calculators/image/BUILD

@@ -80,6 +80,16 @@ mediapipe_proto_library(
     ],
 )
 
+mediapipe_proto_library(
+    name = "segmentation_smoothing_calculator_proto",
+    srcs = ["segmentation_smoothing_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
 cc_library(
     name = "color_convert_calculator",
     srcs = ["color_convert_calculator.cc"],
@@ -405,6 +415,36 @@ cc_library(
     alwayslink = 1,
 )
 
+mediapipe_proto_library(
+    name = "image_clone_calculator_proto",
+    srcs = ["image_clone_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
+cc_library(
+    name = "image_clone_calculator",
+    srcs = ["image_clone_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":image_clone_calculator_cc_proto",
+        "//mediapipe/framework/api2:node",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:image",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
+    ] + select({
+        "//mediapipe/gpu:disable_gpu": [],
+        "//conditions:default": [
+            "//mediapipe/gpu:gl_calculator_helper",
+        ],
+    }),
+    alwayslink = 1,
+)
+
 cc_library(
     name = "image_properties_calculator",
     srcs = ["image_properties_calculator.cc"],
@@ -572,3 +612,52 @@ cc_test(
         "//mediapipe/framework/port:parse_text_proto",
     ],
 )
+
+cc_library(
+    name = "segmentation_smoothing_calculator",
+    srcs = ["segmentation_smoothing_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":segmentation_smoothing_calculator_cc_proto",
+        "//mediapipe/framework:calculator_options_cc_proto",
+        "//mediapipe/framework/formats:image_format_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_frame_opencv",
+        "//mediapipe/framework/formats:image",
+        "//mediapipe/framework/formats:image_opencv",
+        "//mediapipe/framework/port:logging",
+        "//mediapipe/framework/port:opencv_core",
+        "//mediapipe/framework/port:status",
+        "//mediapipe/framework/port:vector",
+    ] + select({
+        "//mediapipe/gpu:disable_gpu": [],
+        "//conditions:default": [
+            "//mediapipe/gpu:gl_calculator_helper",
+            "//mediapipe/gpu:gl_simple_shaders",
+            "//mediapipe/gpu:gl_quad_renderer",
+            "//mediapipe/gpu:shader_util",
+        ],
+    }),
+    alwayslink = 1,
+)
+
+cc_test(
+    name = "segmentation_smoothing_calculator_test",
+    srcs = ["segmentation_smoothing_calculator_test.cc"],
+    deps = [
+        ":image_clone_calculator",
+        ":image_clone_calculator_cc_proto",
+        ":segmentation_smoothing_calculator",
+        ":segmentation_smoothing_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:calculator_runner",
+        "//mediapipe/framework/deps:file_path",
+        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_opencv",
+        "//mediapipe/framework/port:gtest_main",
+        "//mediapipe/framework/port:opencv_imgcodecs",
+        "//mediapipe/framework/port:opencv_imgproc",
+        "//mediapipe/framework/port:parse_text_proto",
+    ],
+)

mediapipe/calculators/image/image_clone_calculator.cc

@@ -0,0 +1,125 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/calculators/image/image_clone_calculator.pb.h"
+#include "mediapipe/framework/api2/node.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/formats/image.h"
+#include "mediapipe/framework/port/status.h"
+
+#if !MEDIAPIPE_DISABLE_GPU
+#include "mediapipe/gpu/gl_calculator_helper.h"
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+namespace mediapipe {
+namespace api2 {
+
+#if MEDIAPIPE_DISABLE_GPU
+// Just a placeholder to not have to depend on mediapipe::GpuBuffer.
+using GpuBuffer = AnyType;
+#else
+using GpuBuffer = mediapipe::GpuBuffer;
+#endif  // MEDIAPIPE_DISABLE_GPU
+
+// Clones an input image and makes sure in the output clone the pixel data are
+// stored on the target storage (CPU vs GPU) specified in the calculator option.
+//
+// The clone shares ownership of the input pixel data on the existing storage.
+// If the target storage is diffrent from the existing one, then the data is
+// further copied there.
+//
+// Example usage:
+// node {
+//   calculator: "ImageCloneCalculator"
+//   input_stream: "input"
+//   output_stream: "output"
+//   options: {
+//     [mediapipe.ImageCloneCalculatorOptions.ext] {
+//       output_on_gpu: true
+//     }
+//   }
+// }
+class ImageCloneCalculator : public Node {
+ public:
+  static constexpr Input<Image> kIn{""};
+  static constexpr Output<Image> kOut{""};
+
+  MEDIAPIPE_NODE_CONTRACT(kIn, kOut);
+
+  static absl::Status UpdateContract(CalculatorContract* cc) {
+#if MEDIAPIPE_DISABLE_GPU
+    if (cc->Options<mediapipe::ImageCloneCalculatorOptions>().output_on_gpu()) {
+      return absl::UnimplementedError(
+          "GPU processing is disabled in build flags");
+    }
+#else
+    MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
+#endif  // MEDIAPIPE_DISABLE_GPU
+    return absl::OkStatus();
+  }
+
+  absl::Status Open(CalculatorContext* cc) override {
+    const auto& options = cc->Options<mediapipe::ImageCloneCalculatorOptions>();
+    output_on_gpu_ = options.output_on_gpu();
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
+#endif  // !MEDIAPIPE_DISABLE_GPU
+    return absl::OkStatus();
+  }
+
+  absl::Status Process(CalculatorContext* cc) override {
+    std::unique_ptr<Image> output;
+    const auto& input = *kIn(cc);
+    if (input.UsesGpu()) {
+#if !MEDIAPIPE_DISABLE_GPU
+      // Create an output Image that co-owns the underlying texture buffer as
+      // the input Image.
+      output = std::make_unique<Image>(input.GetGpuBuffer());
+#endif  // !MEDIAPIPE_DISABLE_GPU
+    } else {
+      // Make a copy of the input packet to co-own the input Image.
+      mediapipe::Packet* packet_copy_ptr =
+          new mediapipe::Packet(kIn(cc).packet());
+      // Create an output Image that (co-)owns a new ImageFrame that points to
+      // the same pixel data as the input Image and also owns the packet
+      // copy. As a result, the output Image indirectly co-owns the input
+      // Image. This ensures a correct life span of the shared pixel data.
+      output = std::make_unique<Image>(std::make_unique<mediapipe::ImageFrame>(
+          input.image_format(), input.width(), input.height(), input.step(),
+          const_cast<uint8*>(input.GetImageFrameSharedPtr()->PixelData()),
+          [packet_copy_ptr](uint8*) { delete packet_copy_ptr; }));
+    }
+
+    if (output_on_gpu_) {
+#if !MEDIAPIPE_DISABLE_GPU
+      gpu_helper_.RunInGlContext([&output]() { output->ConvertToGpu(); });
+#endif  // !MEDIAPIPE_DISABLE_GPU
+    } else {
+      output->ConvertToCpu();
+    }
+    kOut(cc).Send(std::move(output));
+
+    return absl::OkStatus();
+  }
+
+ private:
+  bool output_on_gpu_;
+#if !MEDIAPIPE_DISABLE_GPU
+  mediapipe::GlCalculatorHelper gpu_helper_;
+#endif  // !MEDIAPIPE_DISABLE_GPU
+};
+MEDIAPIPE_REGISTER_NODE(ImageCloneCalculator);
+
+}  // namespace api2
+}  // namespace mediapipe

mediapipe/calculators/image/image_clone_calculator.proto

@@ -0,0 +1,28 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto2";
+
+package mediapipe;
+
+import "mediapipe/framework/calculator.proto";
+
+message ImageCloneCalculatorOptions {
+  extend CalculatorOptions {
+    optional ImageCloneCalculatorOptions ext = 372781894;
+  }
+
+  // Whether the output clone should have pixel data already available on GPU.
+  optional bool output_on_gpu = 1 [default = false];
+}

mediapipe/calculators/image/recolor_calculator.cc

@@ -37,6 +37,22 @@ constexpr char kImageFrameTag[] = "IMAGE";
 constexpr char kMaskCpuTag[] = "MASK";
 constexpr char kGpuBufferTag[] = "IMAGE_GPU";
 constexpr char kMaskGpuTag[] = "MASK_GPU";
+
+inline cv::Vec3b Blend(const cv::Vec3b& color1, const cv::Vec3b& color2,
+                       float weight, int invert_mask,
+                       int adjust_with_luminance) {
+  weight = (1 - invert_mask) * weight + invert_mask * (1.0f - weight);
+
+  float luminance =
+      (1 - adjust_with_luminance) * 1.0f +
+      adjust_with_luminance *
+          (color1[0] * 0.299 + color1[1] * 0.587 + color1[2] * 0.114) / 255;
+
+  float mix_value = weight * luminance;
+
+  return color1 * (1.0 - mix_value) + color2 * mix_value;
+}
+
 }  // namespace
 
 namespace mediapipe {
@@ -44,15 +60,14 @@ namespace mediapipe {
 // A calculator to recolor a masked area of an image to a specified color.
 //
 // A mask image is used to specify where to overlay a user defined color.
-// The luminance of the input image is used to adjust the blending weight,
-// to help preserve image textures.
 //
 // Inputs:
 //   One of the following IMAGE tags:
-//   IMAGE: An ImageFrame input image, RGB or RGBA.
+//   IMAGE: An ImageFrame input image in ImageFormat::SRGB.
 //   IMAGE_GPU: A GpuBuffer input image, RGBA.
 //   One of the following MASK tags:
-//   MASK: An ImageFrame input mask, Gray, RGB or RGBA.
+//   MASK: An ImageFrame input mask in ImageFormat::GRAY8, SRGB, SRGBA, or
+//         VEC32F1
 //   MASK_GPU: A GpuBuffer input mask, RGBA.
 // Output:
 //   One of the following IMAGE tags:
@@ -98,10 +113,12 @@ class RecolorCalculator : public CalculatorBase {
   void GlRender();
 
   bool initialized_ = false;
-  std::vector<float> color_;
+  std::vector<uint8> color_;
   mediapipe::RecolorCalculatorOptions::MaskChannel mask_channel_;
 
   bool use_gpu_ = false;
+  bool invert_mask_ = false;
+  bool adjust_with_luminance_ = false;
 #if !MEDIAPIPE_DISABLE_GPU
   mediapipe::GlCalculatorHelper gpu_helper_;
   GLuint program_ = 0;
@@ -233,11 +250,15 @@ absl::Status RecolorCalculator::RenderCpu(CalculatorContext* cc) {
   }
   cv::Mat mask_full;
   cv::resize(mask_mat, mask_full, input_mat.size());
+  const cv::Vec3b recolor = {color_[0], color_[1], color_[2]};
 
   auto output_img = absl::make_unique<ImageFrame>(
       input_img.Format(), input_mat.cols, input_mat.rows);
   cv::Mat output_mat = mediapipe::formats::MatView(output_img.get());
 
+  const int invert_mask = invert_mask_ ? 1 : 0;
+  const int adjust_with_luminance = adjust_with_luminance_ ? 1 : 0;
+
   // From GPU shader:
   /*
       vec4 weight = texture2D(mask, sample_coordinate);
@@ -249,18 +270,23 @@ absl::Status RecolorCalculator::RenderCpu(CalculatorContext* cc) {
 
       fragColor = mix(color1, color2, mix_value);
   */
-  for (int i = 0; i < output_mat.rows; ++i) {
-    for (int j = 0; j < output_mat.cols; ++j) {
-      float weight = mask_full.at<uchar>(i, j) * (1.0 / 255.0);
-      cv::Vec3f color1 = input_mat.at<cv::Vec3b>(i, j);
-      cv::Vec3f color2 = {color_[0], color_[1], color_[2]};
-
-      float luminance =
-          (color1[0] * 0.299 + color1[1] * 0.587 + color1[2] * 0.114) / 255;
-      float mix_value = weight * luminance;
-
-      cv::Vec3b mix_color = color1 * (1.0 - mix_value) + color2 * mix_value;
-      output_mat.at<cv::Vec3b>(i, j) = mix_color;
+  if (mask_img.Format() == ImageFormat::VEC32F1) {
+    for (int i = 0; i < output_mat.rows; ++i) {
+      for (int j = 0; j < output_mat.cols; ++j) {
+        const float weight = mask_full.at<float>(i, j);
+        output_mat.at<cv::Vec3b>(i, j) =
+            Blend(input_mat.at<cv::Vec3b>(i, j), recolor, weight, invert_mask,
+                  adjust_with_luminance);
+      }
+    }
+  } else {
+    for (int i = 0; i < output_mat.rows; ++i) {
+      for (int j = 0; j < output_mat.cols; ++j) {
+        const float weight = mask_full.at<uchar>(i, j) * (1.0 / 255.0);
+        output_mat.at<cv::Vec3b>(i, j) =
+            Blend(input_mat.at<cv::Vec3b>(i, j), recolor, weight, invert_mask,
+                  adjust_with_luminance);
+      }
     }
   }
 
@@ -385,6 +411,9 @@ absl::Status RecolorCalculator::LoadOptions(CalculatorContext* cc) {
   color_.push_back(options.color().g());
   color_.push_back(options.color().b());
 
+  invert_mask_ = options.invert_mask();
+  adjust_with_luminance_ = options.adjust_with_luminance();
+
   return absl::OkStatus();
 }
 
@@ -435,13 +464,20 @@ absl::Status RecolorCalculator::InitGpu(CalculatorContext* cc) {
     uniform sampler2D frame;
     uniform sampler2D mask;
     uniform vec3 recolor;
+    uniform float invert_mask;
+    uniform float adjust_with_luminance;
 
     void main() {
       vec4 weight = texture2D(mask, sample_coordinate);
       vec4 color1 = texture2D(frame, sample_coordinate);
       vec4 color2 = vec4(recolor, 1.0);
 
-      float luminance = dot(color1.rgb, vec3(0.299, 0.587, 0.114));
+      weight = mix(weight, 1.0 - weight, invert_mask);
+
+      float luminance = mix(1.0,
+                            dot(color1.rgb, vec3(0.299, 0.587, 0.114)),
+                            adjust_with_luminance);
+
       float mix_value = weight.MASK_COMPONENT * luminance;
 
       fragColor = mix(color1, color2, mix_value);
@@ -458,6 +494,10 @@ absl::Status RecolorCalculator::InitGpu(CalculatorContext* cc) {
   glUniform1i(glGetUniformLocation(program_, "mask"), 2);
   glUniform3f(glGetUniformLocation(program_, "recolor"), color_[0] / 255.0,
               color_[1] / 255.0, color_[2] / 255.0);
+  glUniform1f(glGetUniformLocation(program_, "invert_mask"),
+              invert_mask_ ? 1.0f : 0.0f);
+  glUniform1f(glGetUniformLocation(program_, "adjust_with_luminance"),
+              adjust_with_luminance_ ? 1.0f : 0.0f);
 #endif  // !MEDIAPIPE_DISABLE_GPU
 
   return absl::OkStatus();

mediapipe/calculators/image/recolor_calculator.proto

@@ -36,4 +36,11 @@ message RecolorCalculatorOptions {
   // Color to blend into input image where mask is > 0.
   // The blending is based on the input image luminosity.
   optional Color color = 2;
+
+  // Swap the meaning of mask values for foreground/background.
+  optional bool invert_mask = 3 [default = false];
+
+  // Whether to use the luminance of the input image to further adjust the
+  // blending weight, to help preserve image textures.
+  optional bool adjust_with_luminance = 4 [default = true];
 }

mediapipe/calculators/image/segmentation_smoothing_calculator.cc

@@ -0,0 +1,429 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <algorithm>
+#include <memory>
+
+#include "mediapipe/calculators/image/segmentation_smoothing_calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_options.pb.h"
+#include "mediapipe/framework/formats/image.h"
+#include "mediapipe/framework/formats/image_format.pb.h"
+#include "mediapipe/framework/formats/image_frame.h"
+#include "mediapipe/framework/formats/image_frame_opencv.h"
+#include "mediapipe/framework/formats/image_opencv.h"
+#include "mediapipe/framework/port/logging.h"
+#include "mediapipe/framework/port/opencv_core_inc.h"
+#include "mediapipe/framework/port/status.h"
+#include "mediapipe/framework/port/vector.h"
+
+#if !MEDIAPIPE_DISABLE_GPU
+#include "mediapipe/gpu/gl_calculator_helper.h"
+#include "mediapipe/gpu/gl_simple_shaders.h"
+#include "mediapipe/gpu/shader_util.h"
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+namespace mediapipe {
+
+namespace {
+constexpr char kCurrentMaskTag[] = "MASK";
+constexpr char kPreviousMaskTag[] = "MASK_PREVIOUS";
+constexpr char kOutputMaskTag[] = "MASK_SMOOTHED";
+
+enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
+}  // namespace
+
+// A calculator for mixing two segmentation masks together,
+// based on an uncertantity probability estimate.
+//
+// Inputs:
+//   MASK - Image containing the new/current mask.
+//          [ImageFormat::VEC32F1, or
+//           GpuBufferFormat::kBGRA32/kRGB24/kGrayHalf16/kGrayFloat32]
+//   MASK_PREVIOUS - Image containing previous mask.
+//                   [Same format as MASK_CURRENT]
+//   * If input channels is >1, only the first channel (R) is used as the mask.
+//
+// Output:
+//   MASK_SMOOTHED - Blended mask.
+//                   [Same format as MASK_CURRENT]
+//   * The resulting filtered mask will be stored in R channel,
+//     and duplicated in A if 4 channels.
+//
+// Options:
+//   combine_with_previous_ratio - Amount of previous to blend with current.
+//
+// Example:
+//  node {
+//    calculator: "SegmentationSmoothingCalculator"
+//    input_stream: "MASK:mask"
+//    input_stream: "MASK_PREVIOUS:mask_previous"
+//    output_stream: "MASK_SMOOTHED:mask_smoothed"
+//    options: {
+//      [mediapipe.SegmentationSmoothingCalculatorOptions.ext] {
+//        combine_with_previous_ratio: 0.9
+//      }
+//    }
+//  }
+//
+class SegmentationSmoothingCalculator : public CalculatorBase {
+ public:
+  SegmentationSmoothingCalculator() = default;
+
+  static absl::Status GetContract(CalculatorContract* cc);
+
+  // From Calculator.
+  absl::Status Open(CalculatorContext* cc) override;
+  absl::Status Process(CalculatorContext* cc) override;
+  absl::Status Close(CalculatorContext* cc) override;
+
+ private:
+  absl::Status RenderGpu(CalculatorContext* cc);
+  absl::Status RenderCpu(CalculatorContext* cc);
+
+  absl::Status GlSetup(CalculatorContext* cc);
+  void GlRender(CalculatorContext* cc);
+
+  float combine_with_previous_ratio_;
+
+  bool gpu_initialized_ = false;
+#if !MEDIAPIPE_DISABLE_GPU
+  mediapipe::GlCalculatorHelper gpu_helper_;
+  GLuint program_ = 0;
+#endif  // !MEDIAPIPE_DISABLE_GPU
+};
+REGISTER_CALCULATOR(SegmentationSmoothingCalculator);
+
+absl::Status SegmentationSmoothingCalculator::GetContract(
+    CalculatorContract* cc) {
+  CHECK_GE(cc->Inputs().NumEntries(), 1);
+
+  cc->Inputs().Tag(kCurrentMaskTag).Set<Image>();
+  cc->Inputs().Tag(kPreviousMaskTag).Set<Image>();
+  cc->Outputs().Tag(kOutputMaskTag).Set<Image>();
+
+#if !MEDIAPIPE_DISABLE_GPU
+  MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::Open(CalculatorContext* cc) {
+  cc->SetOffset(TimestampDiff(0));
+
+  auto options =
+      cc->Options<mediapipe::SegmentationSmoothingCalculatorOptions>();
+  combine_with_previous_ratio_ = options.combine_with_previous_ratio();
+
+#if !MEDIAPIPE_DISABLE_GPU
+  MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
+#endif  //  !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::Process(CalculatorContext* cc) {
+  if (cc->Inputs().Tag(kCurrentMaskTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+  if (cc->Inputs().Tag(kPreviousMaskTag).IsEmpty()) {
+    // Pass through current image if previous is not available.
+    cc->Outputs()
+        .Tag(kOutputMaskTag)
+        .AddPacket(cc->Inputs().Tag(kCurrentMaskTag).Value());
+    return absl::OkStatus();
+  }
+
+  // Run on GPU if incoming data is on GPU.
+  const bool use_gpu = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>().UsesGpu();
+
+  if (use_gpu) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this, cc]() -> absl::Status {
+      if (!gpu_initialized_) {
+        MP_RETURN_IF_ERROR(GlSetup(cc));
+        gpu_initialized_ = true;
+      }
+      MP_RETURN_IF_ERROR(RenderGpu(cc));
+      return absl::OkStatus();
+    }));
+#else
+    return absl::InternalError("GPU processing is disabled.");
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  } else {
+    MP_RETURN_IF_ERROR(RenderCpu(cc));
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::Close(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  gpu_helper_.RunInGlContext([this] {
+    if (program_) glDeleteProgram(program_);
+    program_ = 0;
+  });
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::RenderCpu(CalculatorContext* cc) {
+  // Setup source images.
+  const auto& current_frame = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>();
+  const cv::Mat current_mat = mediapipe::formats::MatView(&current_frame);
+  RET_CHECK_EQ(current_mat.type(), CV_32FC1)
+      << "Only 1-channel float input image is supported.";
+
+  const auto& previous_frame = cc->Inputs().Tag(kPreviousMaskTag).Get<Image>();
+  const cv::Mat previous_mat = mediapipe::formats::MatView(&previous_frame);
+  RET_CHECK_EQ(previous_mat.type(), current_mat.type())
+      << "Warning: mixing input format types: " << previous_mat.type()
+      << " != " << previous_mat.type();
+
+  RET_CHECK_EQ(current_mat.rows, previous_mat.rows);
+  RET_CHECK_EQ(current_mat.cols, previous_mat.cols);
+
+  // Setup destination image.
+  auto output_frame = std::make_shared<ImageFrame>(
+      current_frame.image_format(), current_mat.cols, current_mat.rows);
+  cv::Mat output_mat = mediapipe::formats::MatView(output_frame.get());
+  output_mat.setTo(cv::Scalar(0));
+
+  // Blending function.
+  const auto blending_fn = [&](const float prev_mask_value,
+                               const float new_mask_value) {
+    /*
+     * Assume p := new_mask_value
+     * H(p) := 1 + (p * log(p) + (1-p) * log(1-p)) / log(2)
+     * uncertainty alpha(p) =
+     *   Clamp(1 - (1 - H(p)) * (1 - H(p)), 0, 1) [squaring the uncertainty]
+     *
+     * The following polynomial approximates uncertainty alpha as a function
+     * of (p + 0.5):
+     */
+    const float c1 = 5.68842;
+    const float c2 = -0.748699;
+    const float c3 = -57.8051;
+    const float c4 = 291.309;
+    const float c5 = -624.717;
+    const float t = new_mask_value - 0.5f;
+    const float x = t * t;
+
+    const float uncertainty =
+        1.0f -
+        std::min(1.0f, x * (c1 + x * (c2 + x * (c3 + x * (c4 + x * c5)))));
+
+    return new_mask_value + (prev_mask_value - new_mask_value) *
+                                (uncertainty * combine_with_previous_ratio_);
+  };
+
+  // Write directly to the first channel of output.
+  for (int i = 0; i < output_mat.rows; ++i) {
+    float* out_ptr = output_mat.ptr<float>(i);
+    const float* curr_ptr = current_mat.ptr<float>(i);
+    const float* prev_ptr = previous_mat.ptr<float>(i);
+    for (int j = 0; j < output_mat.cols; ++j) {
+      const float new_mask_value = curr_ptr[j];
+      const float prev_mask_value = prev_ptr[j];
+      out_ptr[j] = blending_fn(prev_mask_value, new_mask_value);
+    }
+  }
+
+  cc->Outputs()
+      .Tag(kOutputMaskTag)
+      .AddPacket(MakePacket<Image>(output_frame).At(cc->InputTimestamp()));
+
+  return absl::OkStatus();
+}
+
+absl::Status SegmentationSmoothingCalculator::RenderGpu(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  // Setup source textures.
+  const auto& current_frame = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>();
+  RET_CHECK(
+      (current_frame.format() == mediapipe::GpuBufferFormat::kBGRA32 ||
+       current_frame.format() == mediapipe::GpuBufferFormat::kGrayHalf16 ||
+       current_frame.format() == mediapipe::GpuBufferFormat::kGrayFloat32 ||
+       current_frame.format() == mediapipe::GpuBufferFormat::kRGB24))
+      << "Only RGBA, RGB, or 1-channel Float input image supported.";
+
+  auto current_texture = gpu_helper_.CreateSourceTexture(current_frame);
+
+  const auto& previous_frame = cc->Inputs().Tag(kPreviousMaskTag).Get<Image>();
+  if (previous_frame.format() != current_frame.format()) {
+    LOG(ERROR) << "Warning: mixing input format types. ";
+  }
+  auto previous_texture = gpu_helper_.CreateSourceTexture(previous_frame);
+
+  // Setup destination texture.
+  const int width = current_frame.width(), height = current_frame.height();
+  auto output_texture = gpu_helper_.CreateDestinationTexture(
+      width, height, current_frame.format());
+
+  // Process shader.
+  {
+    gpu_helper_.BindFramebuffer(output_texture);
+    glActiveTexture(GL_TEXTURE1);
+    glBindTexture(GL_TEXTURE_2D, current_texture.name());
+    glActiveTexture(GL_TEXTURE2);
+    glBindTexture(GL_TEXTURE_2D, previous_texture.name());
+    GlRender(cc);
+    glActiveTexture(GL_TEXTURE2);
+    glBindTexture(GL_TEXTURE_2D, 0);
+    glActiveTexture(GL_TEXTURE1);
+    glBindTexture(GL_TEXTURE_2D, 0);
+  }
+  glFlush();
+
+  // Send out image as GPU packet.
+  auto output_frame = output_texture.GetFrame<Image>();
+  cc->Outputs()
+      .Tag(kOutputMaskTag)
+      .Add(output_frame.release(), cc->InputTimestamp());
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+void SegmentationSmoothingCalculator::GlRender(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  static const GLfloat square_vertices[] = {
+      -1.0f, -1.0f,  // bottom left
+      1.0f,  -1.0f,  // bottom right
+      -1.0f, 1.0f,   // top left
+      1.0f,  1.0f,   // top right
+  };
+  static const GLfloat texture_vertices[] = {
+      0.0f, 0.0f,  // bottom left
+      1.0f, 0.0f,  // bottom right
+      0.0f, 1.0f,  // top left
+      1.0f, 1.0f,  // top right
+  };
+
+  // program
+  glUseProgram(program_);
+
+  // vertex storage
+  GLuint vbo[2];
+  glGenBuffers(2, vbo);
+  GLuint vao;
+  glGenVertexArrays(1, &vao);
+  glBindVertexArray(vao);
+
+  // vbo 0
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_VERTEX);
+  glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // vbo 1
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // draw
+  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
+
+  // cleanup
+  glDisableVertexAttribArray(ATTRIB_VERTEX);
+  glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glBindBuffer(GL_ARRAY_BUFFER, 0);
+  glBindVertexArray(0);
+  glDeleteVertexArrays(1, &vao);
+  glDeleteBuffers(2, vbo);
+
+#endif  // !MEDIAPIPE_DISABLE_GPU
+}
+
+absl::Status SegmentationSmoothingCalculator::GlSetup(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  const GLint attr_location[NUM_ATTRIBUTES] = {
+      ATTRIB_VERTEX,
+      ATTRIB_TEXTURE_POSITION,
+  };
+  const GLchar* attr_name[NUM_ATTRIBUTES] = {
+      "position",
+      "texture_coordinate",
+  };
+
+  // Shader to blend in previous mask based on computed uncertainty probability.
+  const std::string frag_src =
+      absl::StrCat(std::string(mediapipe::kMediaPipeFragmentShaderPreamble),
+                   R"(
+    DEFAULT_PRECISION(mediump, float)
+
+    #ifdef GL_ES
+    #define fragColor gl_FragColor
+    #else
+    out vec4 fragColor;
+    #endif  // defined(GL_ES);
+
+    in vec2 sample_coordinate;
+    uniform sampler2D current_mask;
+    uniform sampler2D previous_mask;
+    uniform float combine_with_previous_ratio;
+
+    void main() {
+      vec4 current_pix = texture2D(current_mask, sample_coordinate);
+      vec4 previous_pix = texture2D(previous_mask, sample_coordinate);
+      float new_mask_value = current_pix.r;
+      float prev_mask_value = previous_pix.r;
+
+      // Assume p := new_mask_value
+      // H(p) := 1 + (p * log(p) + (1-p) * log(1-p)) / log(2)
+      // uncertainty alpha(p) =
+      //   Clamp(1 - (1 - H(p)) * (1 - H(p)), 0, 1) [squaring the uncertainty]
+      //
+      // The following polynomial approximates uncertainty alpha as a function
+      // of (p + 0.5):
+      const float c1 = 5.68842;
+      const float c2 = -0.748699;
+      const float c3 = -57.8051;
+      const float c4 = 291.309;
+      const float c5 = -624.717;
+      float t = new_mask_value - 0.5;
+      float x = t * t;
+
+      float uncertainty =
+        1.0 - min(1.0, x * (c1 + x * (c2 + x * (c3 + x * (c4 + x * c5)))));
+
+      new_mask_value +=
+        (prev_mask_value - new_mask_value) * (uncertainty * combine_with_previous_ratio);
+
+      fragColor = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
+    }
+  )");
+
+  // Create shader program and set parameters.
+  mediapipe::GlhCreateProgram(mediapipe::kBasicVertexShader, frag_src.c_str(),
+                              NUM_ATTRIBUTES, (const GLchar**)&attr_name[0],
+                              attr_location, &program_);
+  RET_CHECK(program_) << "Problem initializing the program.";
+  glUseProgram(program_);
+  glUniform1i(glGetUniformLocation(program_, "current_mask"), 1);
+  glUniform1i(glGetUniformLocation(program_, "previous_mask"), 2);
+  glUniform1f(glGetUniformLocation(program_, "combine_with_previous_ratio"),
+              combine_with_previous_ratio_);
+
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+}  // namespace mediapipe

mediapipe/calculators/image/segmentation_smoothing_calculator.proto

@@ -0,0 +1,35 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto2";
+
+package mediapipe;
+
+import "mediapipe/framework/calculator.proto";
+
+message SegmentationSmoothingCalculatorOptions {
+  extend CalculatorOptions {
+    optional SegmentationSmoothingCalculatorOptions ext = 377425128;
+  }
+
+  // How much to blend in previous mask, based on a probability estimate.
+  // Range: [0-1]
+  // 0 = Use only current frame (no blending).
+  // 1 = Blend in the previous mask based on uncertainty estimate.
+  //     With ratio at 1, the uncertainty estimate is trusted completely.
+  //     When uncertainty is high, the previous mask is given higher weight.
+  //     Therefore, if both ratio and uncertainty are 1, only old mask is used.
+  //   A pixel is 'uncertain' if its value is close to the middle (0.5 or 127).
+  optional float combine_with_previous_ratio = 1 [default = 0.0];
+}

mediapipe/calculators/image/segmentation_smoothing_calculator_test.cc

@@ -0,0 +1,206 @@
+// Copyright 2018 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <memory>
+
+#include "mediapipe/calculators/image/segmentation_smoothing_calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_runner.h"
+#include "mediapipe/framework/deps/file_path.h"
+#include "mediapipe/framework/formats/image_frame.h"
+#include "mediapipe/framework/formats/image_opencv.h"
+#include "mediapipe/framework/port/gmock.h"
+#include "mediapipe/framework/port/gtest.h"
+#include "mediapipe/framework/port/opencv_imgcodecs_inc.h"
+#include "mediapipe/framework/port/opencv_imgproc_inc.h"
+#include "mediapipe/framework/port/parse_text_proto.h"
+#include "mediapipe/framework/port/status_matchers.h"
+
+namespace mediapipe {
+
+namespace {
+
+// 4x4 VEC32F1, center 2x2 block set at ~250
+const float mask_data[] = {
+    0.00, 0.00, 0.00, 0.00,  //
+    0.00, 0.98, 0.98, 0.00,  //
+    0.00, 0.98, 0.98, 0.00,  //
+    0.00, 0.00, 0.00, 0.00,  //
+};
+
+void RunGraph(Packet curr_packet, Packet prev_packet, bool use_gpu, float ratio,
+              cv::Mat* result) {
+  CalculatorGraphConfig graph_config;
+  if (use_gpu) {
+    graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
+        R"pb(
+          input_stream: "curr_mask"
+          input_stream: "prev_mask"
+          output_stream: "new_mask"
+          node {
+            calculator: "ImageCloneCalculator"
+            input_stream: "curr_mask"
+            output_stream: "curr_mask_gpu"
+            options: {
+              [mediapipe.ImageCloneCalculatorOptions.ext] {
+                output_on_gpu: true
+              }
+            }
+          }
+          node {
+            calculator: "ImageCloneCalculator"
+            input_stream: "prev_mask"
+            output_stream: "prev_mask_gpu"
+            options: {
+              [mediapipe.ImageCloneCalculatorOptions.ext] {
+                output_on_gpu: true
+              }
+            }
+          }
+          node {
+            calculator: "SegmentationSmoothingCalculator"
+            input_stream: "MASK:curr_mask_gpu"
+            input_stream: "MASK_PREVIOUS:prev_mask_gpu"
+            output_stream: "MASK_SMOOTHED:new_mask"
+            node_options {
+              [type.googleapis.com/
+               mediapipe.SegmentationSmoothingCalculatorOptions]: {
+                combine_with_previous_ratio: $0
+              }
+            }
+          }
+        )pb",
+        ratio));
+  } else {
+    graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
+        R"pb(
+          input_stream: "curr_mask"
+          input_stream: "prev_mask"
+          output_stream: "new_mask"
+          node {
+            calculator: "SegmentationSmoothingCalculator"
+            input_stream: "MASK:curr_mask"
+            input_stream: "MASK_PREVIOUS:prev_mask"
+            output_stream: "MASK_SMOOTHED:new_mask"
+            node_options {
+              [type.googleapis.com/
+               mediapipe.SegmentationSmoothingCalculatorOptions]: {
+                combine_with_previous_ratio: $0
+              }
+            }
+          }
+        )pb",
+        ratio));
+  }
+  std::vector<Packet> output_packets;
+  tool::AddVectorSink("new_mask", &graph_config, &output_packets);
+  CalculatorGraph graph(graph_config);
+  MP_ASSERT_OK(graph.StartRun({}));
+
+  MP_ASSERT_OK(
+      graph.AddPacketToInputStream("curr_mask", curr_packet.At(Timestamp(0))));
+  MP_ASSERT_OK(
+      graph.AddPacketToInputStream("prev_mask", prev_packet.At(Timestamp(0))));
+  MP_ASSERT_OK(graph.WaitUntilIdle());
+  ASSERT_EQ(1, output_packets.size());
+
+  Image result_image = output_packets[0].Get<Image>();
+  cv::Mat result_mat = formats::MatView(&result_image);
+  result_mat.copyTo(*result);
+
+  // Fully close graph at end, otherwise calculator+Images are destroyed
+  // after calling WaitUntilDone().
+  MP_ASSERT_OK(graph.CloseInputStream("curr_mask"));
+  MP_ASSERT_OK(graph.CloseInputStream("prev_mask"));
+  MP_ASSERT_OK(graph.WaitUntilDone());
+}
+
+void RunTest(bool use_gpu, float mix_ratio, cv::Mat& test_result) {
+  cv::Mat mask_mat(cv::Size(4, 4), CV_32FC1, const_cast<float*>(mask_data));
+  cv::Mat curr_mat = mask_mat;
+  // 3x3 blur of 250 block produces all pixels '111'.
+  cv::Mat prev_mat;
+  cv::blur(mask_mat, prev_mat, cv::Size(3, 3));
+
+  Packet curr_packet = MakePacket<Image>(std::make_unique<ImageFrame>(
+      ImageFormat::VEC32F1, curr_mat.size().width, curr_mat.size().height));
+  curr_mat.copyTo(formats::MatView(&(curr_packet.Get<Image>())));
+  Packet prev_packet = MakePacket<Image>(std::make_unique<ImageFrame>(
+      ImageFormat::VEC32F1, prev_mat.size().width, prev_mat.size().height));
+  prev_mat.copyTo(formats::MatView(&(prev_packet.Get<Image>())));
+
+  cv::Mat result;
+  RunGraph(curr_packet, prev_packet, use_gpu, mix_ratio, &result);
+
+  ASSERT_EQ(curr_mat.rows, result.rows);
+  ASSERT_EQ(curr_mat.cols, result.cols);
+  ASSERT_EQ(curr_mat.type(), result.type());
+  result.copyTo(test_result);
+
+  if (mix_ratio == 1.0) {
+    for (int i = 0; i < 4; ++i) {
+      for (int j = 0; j < 4; ++j) {
+        float in = curr_mat.at<float>(i, j);
+        float out = result.at<float>(i, j);
+        // Since the input has high value (250), it has low uncertainty.
+        // So the output should have changed lower (towards prev),
+        // but not too much.
+        if (in > 0) EXPECT_NE(in, out);
+        EXPECT_NEAR(in, out, 3.0 / 255.0);
+      }
+    }
+  } else if (mix_ratio == 0.0) {
+    for (int i = 0; i < 4; ++i) {
+      for (int j = 0; j < 4; ++j) {
+        float in = curr_mat.at<float>(i, j);
+        float out = result.at<float>(i, j);
+        EXPECT_EQ(in, out);  // Output should match current.
+      }
+    }
+  } else {
+    LOG(ERROR) << "invalid ratio";
+  }
+}
+
+TEST(SegmentationSmoothingCalculatorTest, TestSmoothing) {
+  bool use_gpu;
+  float mix_ratio;
+
+  use_gpu = false;
+  mix_ratio = 0.0;
+  cv::Mat cpu_0;
+  RunTest(use_gpu, mix_ratio, cpu_0);
+
+  use_gpu = false;
+  mix_ratio = 1.0;
+  cv::Mat cpu_1;
+  RunTest(use_gpu, mix_ratio, cpu_1);
+
+  use_gpu = true;
+  mix_ratio = 1.0;
+  cv::Mat gpu_1;
+  RunTest(use_gpu, mix_ratio, gpu_1);
+
+  // CPU & GPU should match.
+  for (int i = 0; i < 4; ++i) {
+    for (int j = 0; j < 4; ++j) {
+      float gpu = gpu_1.at<float>(i, j);
+      float cpu = cpu_1.at<float>(i, j);
+      EXPECT_EQ(cpu, gpu);
+    }
+  }
+}
+
+}  // namespace
+}  // namespace mediapipe

mediapipe/calculators/tensor/BUILD

@@ -109,6 +109,8 @@ cc_library(
         "//mediapipe/gpu:MPPMetalUtil",
         "//mediapipe/gpu:gpu_buffer",
         "//mediapipe/objc:mediapipe_framework_ios",
+        "//mediapipe/util/tflite:config",
+        "@com_google_absl//absl/memory",
         "@org_tensorflow//tensorflow/lite/delegates/gpu:metal_delegate",
         "@org_tensorflow//tensorflow/lite/delegates/gpu:metal_delegate_internal",
         "@org_tensorflow//tensorflow/lite/delegates/gpu/common:shape",
@@ -478,7 +480,6 @@ cc_library(
     deps = [
         ":image_to_tensor_calculator_cc_proto",
         ":image_to_tensor_converter",
-        ":image_to_tensor_converter_opencv",
         ":image_to_tensor_utils",
         "//mediapipe/framework/api2:node",
         "//mediapipe/framework/formats:image",
@@ -494,6 +495,9 @@ cc_library(
     ] + select({
         "//mediapipe/gpu:disable_gpu": [],
         "//conditions:default": [":image_to_tensor_calculator_gpu_deps"],
+    }) + select({
+        "//mediapipe/framework/port:disable_opencv": [],
+        "//conditions:default": [":image_to_tensor_converter_opencv"],
     }),
     alwayslink = 1,
 )
@@ -753,3 +757,76 @@ cc_test(
         "//mediapipe/framework/port:gtest_main",
     ],
 )
+
+# Copied from /mediapipe/calculators/tflite/BUILD
+selects.config_setting_group(
+    name = "gpu_inference_disabled",
+    match_any = [
+        "//mediapipe/gpu:disable_gpu",
+    ],
+)
+
+mediapipe_proto_library(
+    name = "tensors_to_segmentation_calculator_proto",
+    srcs = ["tensors_to_segmentation_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+        "//mediapipe/gpu:gpu_origin_proto",
+    ],
+)
+
+cc_library(
+    name = "tensors_to_segmentation_calculator",
+    srcs = ["tensors_to_segmentation_calculator.cc"],
+    copts = select({
+        "//mediapipe:apple": [
+            "-x objective-c++",
+            "-fobjc-arc",  # enable reference-counting
+        ],
+        "//conditions:default": [],
+    }),
+    visibility = ["//visibility:public"],
+    deps = [
+        ":tensors_to_segmentation_calculator_cc_proto",
+        "@com_google_absl//absl/strings:str_format",
+        "@com_google_absl//absl/strings",
+        "@com_google_absl//absl/types:span",
+        "//mediapipe/framework/formats:image",
+        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_opencv",
+        "//mediapipe/framework/formats:tensor",
+        "//mediapipe/framework/port:opencv_imgproc",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework:calculator_context",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:port",
+        "//mediapipe/util:resource_util",
+        "@org_tensorflow//tensorflow/lite:framework",
+        "//mediapipe/gpu:gpu_origin_cc_proto",
+        "//mediapipe/framework/port:statusor",
+    ] + selects.with_or({
+        "//mediapipe/gpu:disable_gpu": [],
+        "//conditions:default": [
+            "//mediapipe/gpu:gl_calculator_helper",
+            "//mediapipe/gpu:gl_simple_shaders",
+            "//mediapipe/gpu:gpu_buffer",
+            "//mediapipe/gpu:shader_util",
+        ],
+    }) + selects.with_or({
+        ":gpu_inference_disabled": [],
+        "//mediapipe:ios": [
+            "//mediapipe/gpu:MPPMetalUtil",
+            "//mediapipe/gpu:MPPMetalHelper",
+        ],
+        "//conditions:default": [
+            "@org_tensorflow//tensorflow/lite/delegates/gpu:gl_delegate",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_program",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_shader",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_texture",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl/converters:util",
+        ],
+    }),
+    alwayslink = 1,
+)

mediapipe/calculators/tensor/image_to_tensor_calculator.cc

@@ -18,7 +18,6 @@
 
 #include "mediapipe/calculators/tensor/image_to_tensor_calculator.pb.h"
 #include "mediapipe/calculators/tensor/image_to_tensor_converter.h"
-#include "mediapipe/calculators/tensor/image_to_tensor_converter_opencv.h"
 #include "mediapipe/calculators/tensor/image_to_tensor_utils.h"
 #include "mediapipe/framework/api2/node.h"
 #include "mediapipe/framework/calculator_framework.h"
@@ -33,6 +32,10 @@
 #include "mediapipe/framework/port/statusor.h"
 #include "mediapipe/gpu/gpu_origin.pb.h"
 
+#if !MEDIAPIPE_DISABLE_OPENCV
+#include "mediapipe/calculators/tensor/image_to_tensor_converter_opencv.h"
+#endif
+
 #if !MEDIAPIPE_DISABLE_GPU
 #include "mediapipe/gpu/gpu_buffer.h"
 
@@ -301,8 +304,13 @@ class ImageToTensorCalculator : public Node {
       }
     } else {
       if (!cpu_converter_) {
+#if !MEDIAPIPE_DISABLE_OPENCV
         ASSIGN_OR_RETURN(cpu_converter_,
                          CreateOpenCvConverter(cc, GetBorderMode()));
+#else
+        LOG(FATAL) << "Cannot create image to tensor opencv converter since "
+                      "MEDIAPIPE_DISABLE_OPENCV is defined.";
+#endif  // !MEDIAPIPE_DISABLE_OPENCV
       }
     }
     return absl::OkStatus();

mediapipe/calculators/tensor/image_to_tensor_converter_gl_buffer.cc

@@ -312,7 +312,7 @@ class GlProcessor : public ImageToTensorConverter {
       return absl::OkStatus();
     }));
 
-    return std::move(tensor);
+    return tensor;
   }
 
   ~GlProcessor() override {
@@ -338,8 +338,7 @@ CreateImageToGlBufferTensorConverter(CalculatorContext* cc,
   auto result = absl::make_unique<GlProcessor>();
   MP_RETURN_IF_ERROR(result->Init(cc, input_starts_at_bottom, border_mode));
 
-  // Simply "return std::move(result)" failed to build on macOS with bazel.
-  return std::unique_ptr<ImageToTensorConverter>(std::move(result));
+  return result;
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/image_to_tensor_converter_gl_texture.cc

@@ -334,9 +334,7 @@ CreateImageToGlTextureTensorConverter(CalculatorContext* cc,
                                       BorderMode border_mode) {
   auto result = absl::make_unique<GlProcessor>();
   MP_RETURN_IF_ERROR(result->Init(cc, input_starts_at_bottom, border_mode));
-
-  // Simply "return std::move(result)" failed to build on macOS with bazel.
-  return std::unique_ptr<ImageToTensorConverter>(std::move(result));
+  return result;
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/image_to_tensor_converter_metal.cc

@@ -383,7 +383,7 @@ class MetalProcessor : public ImageToTensorConverter {
           tflite::gpu::HW(output_dims.height, output_dims.width),
           command_buffer, buffer_view.buffer()));
       [command_buffer commit];
-      return std::move(tensor);
+      return tensor;
     }
   }
 
@@ -399,8 +399,7 @@ absl::StatusOr<std::unique_ptr<ImageToTensorConverter>> CreateMetalConverter(
   auto result = absl::make_unique<MetalProcessor>();
   MP_RETURN_IF_ERROR(result->Init(cc, border_mode));
 
-  // Simply "return std::move(result)" failed to build on macOS with bazel.
-  return std::unique_ptr<ImageToTensorConverter>(std::move(result));
+  return result;
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/image_to_tensor_converter_opencv.cc

@@ -103,7 +103,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
         GetValueRangeTransformation(kInputImageRangeMin, kInputImageRangeMax,
                                     range_min, range_max));
     transformed.convertTo(dst, CV_32FC3, transform.scale, transform.offset);
-    return std::move(tensor);
+    return tensor;
   }
 
  private:
@@ -114,10 +114,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
 
 absl::StatusOr<std::unique_ptr<ImageToTensorConverter>> CreateOpenCvConverter(
     CalculatorContext* cc, BorderMode border_mode) {
-  // Simply "return absl::make_unique<OpenCvProcessor>()" failed to build on
-  // macOS with bazel.
-  return std::unique_ptr<ImageToTensorConverter>(
-      absl::make_unique<OpenCvProcessor>(border_mode));
+  return absl::make_unique<OpenCvProcessor>(border_mode);
 }
 
 }  // namespace mediapipe

mediapipe/calculators/tensor/inference_calculator_cpu.cc

@@ -35,20 +35,28 @@ namespace api2 {
 
 namespace {
 
+int GetXnnpackDefaultNumThreads() {
+#if defined(MEDIAPIPE_ANDROID) || defined(MEDIAPIPE_IOS) || \
+    defined(__EMSCRIPTEN_PTHREADS__)
+  constexpr int kMinNumThreadsByDefault = 1;
+  constexpr int kMaxNumThreadsByDefault = 4;
+  return std::clamp(NumCPUCores() / 2, kMinNumThreadsByDefault,
+                    kMaxNumThreadsByDefault);
+#else
+  return 1;
+#endif  // MEDIAPIPE_ANDROID || MEDIAPIPE_IOS || __EMSCRIPTEN_PTHREADS__
+}
+
 // Returns number of threads to configure XNNPACK delegate with.
-// (Equal to user provided value if specified.  Otherwise, it returns number of
-// high cores (hard-coded to 1 for Emscripten without Threads extension))
+// Returns user provided value if specified. Otherwise, tries to choose optimal
+// number of threads depending on the device.
 int GetXnnpackNumThreads(const mediapipe::InferenceCalculatorOptions& opts) {
   static constexpr int kDefaultNumThreads = -1;
   if (opts.has_delegate() && opts.delegate().has_xnnpack() &&
       opts.delegate().xnnpack().num_threads() != kDefaultNumThreads) {
     return opts.delegate().xnnpack().num_threads();
   }
-#if !defined(__EMSCRIPTEN__) || defined(__EMSCRIPTEN_PTHREADS__)
-  return InferHigherCoreIds().size();
-#else
-  return 1;
-#endif  // !__EMSCRIPTEN__ || __EMSCRIPTEN_PTHREADS__
+  return GetXnnpackDefaultNumThreads();
 }
 
 }  // namespace

mediapipe/calculators/tensor/inference_calculator_gl.cc

@@ -269,8 +269,8 @@ absl::Status InferenceCalculatorGlImpl::InitTFLiteGPURunner(
       break;
     }
   }
-  MP_RETURN_IF_ERROR(
-      tflite_gpu_runner_->InitializeWithModel(model, op_resolver));
+  MP_RETURN_IF_ERROR(tflite_gpu_runner_->InitializeWithModel(
+      model, op_resolver, /*allow_quant_ops=*/true));
 
   // Create and bind OpenGL buffers for outputs.
   // The buffers are created once and their ids are passed to calculator outputs

mediapipe/calculators/tensor/inference_calculator_metal.cc

@@ -226,6 +226,10 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
 
   // Configure and create the delegate.
   TFLGpuDelegateOptions options;
+  // `enable_quantization` enables the run of sparse models i.e. the models with
+  // DENSIFY op preceding DEQUINTIZE op. Both ops get removed from the execution
+  // graph after the tensor of the weights is read.
+  options.enable_quantization = true;
   options.allow_precision_loss = allow_precision_loss_;
   options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeDoNotWait;
   delegate_ =

mediapipe/calculators/tensor/tensors_to_detections_calculator.cc

@@ -105,6 +105,15 @@ void ConvertAnchorsToRawValues(const std::vector<Anchor>& anchors,
 //            for anchors (e.g. for SSD models) depend on the outputs of the
 //            detection model. The size of anchor tensor must be (num_boxes *
 //            4).
+//
+// Input side packet:
+//  ANCHORS (optional) - The anchors used for decoding the bounding boxes, as a
+//      vector of `Anchor` protos. Not required if post-processing is built-in
+//      the model.
+//  IGNORE_CLASSES (optional) - The list of class ids that should be ignored, as
+//      a vector of integers. It overrides the corresponding field in the
+//      calculator options.
+//
 // Output:
 //  DETECTIONS - Result MediaPipe detections.
 //
@@ -132,8 +141,11 @@ class TensorsToDetectionsCalculator : public Node {
   static constexpr Input<std::vector<Tensor>> kInTensors{"TENSORS"};
   static constexpr SideInput<std::vector<Anchor>>::Optional kInAnchors{
       "ANCHORS"};
+  static constexpr SideInput<std::vector<int>>::Optional kSideInIgnoreClasses{
+      "IGNORE_CLASSES"};
   static constexpr Output<std::vector<Detection>> kOutDetections{"DETECTIONS"};
-  MEDIAPIPE_NODE_CONTRACT(kInTensors, kInAnchors, kOutDetections);
+  MEDIAPIPE_NODE_CONTRACT(kInTensors, kInAnchors, kSideInIgnoreClasses,
+                          kOutDetections);
   static absl::Status UpdateContract(CalculatorContract* cc);
 
   absl::Status Open(CalculatorContext* cc) override;
@@ -566,8 +578,15 @@ absl::Status TensorsToDetectionsCalculator::LoadOptions(CalculatorContext* cc) {
                kNumCoordsPerBox,
            num_coords_);
 
-  for (int i = 0; i < options_.ignore_classes_size(); ++i) {
-    ignore_classes_.insert(options_.ignore_classes(i));
+  if (kSideInIgnoreClasses(cc).IsConnected()) {
+    RET_CHECK(!kSideInIgnoreClasses(cc).IsEmpty());
+    for (int ignore_class : *kSideInIgnoreClasses(cc)) {
+      ignore_classes_.insert(ignore_class);
+    }
+  } else {
+    for (int i = 0; i < options_.ignore_classes_size(); ++i) {
+      ignore_classes_.insert(options_.ignore_classes(i));
+    }
   }
 
   return absl::OkStatus();

mediapipe/calculators/tensor/tensors_to_detections_calculator.proto

@@ -56,7 +56,7 @@ message TensorsToDetectionsCalculatorOptions {
   // [x_center, y_center, w, h].
   optional bool reverse_output_order = 14 [default = false];
   // The ids of classes that should be ignored during decoding the score for
-  // each predicted box.
+  // each predicted box. Can be overridden with IGNORE_CLASSES side packet.
   repeated int32 ignore_classes = 8;
 
   optional bool sigmoid_score = 15 [default = false];

mediapipe/calculators/tensor/tensors_to_segmentation_calculator.cc

@@ -0,0 +1,882 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <vector>
+
+#include "absl/strings/str_format.h"
+#include "absl/types/span.h"
+#include "mediapipe/calculators/tensor/tensors_to_segmentation_calculator.pb.h"
+#include "mediapipe/framework/calculator_context.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/formats/image.h"
+#include "mediapipe/framework/formats/image_opencv.h"
+#include "mediapipe/framework/formats/tensor.h"
+#include "mediapipe/framework/port.h"
+#include "mediapipe/framework/port/opencv_imgproc_inc.h"
+#include "mediapipe/framework/port/ret_check.h"
+#include "mediapipe/framework/port/statusor.h"
+#include "mediapipe/gpu/gpu_origin.pb.h"
+#include "mediapipe/util/resource_util.h"
+#include "tensorflow/lite/interpreter.h"
+
+#if !MEDIAPIPE_DISABLE_GPU
+#include "mediapipe/gpu/gl_calculator_helper.h"
+#include "mediapipe/gpu/gl_simple_shaders.h"
+#include "mediapipe/gpu/gpu_buffer.h"
+#include "mediapipe/gpu/shader_util.h"
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+#include "tensorflow/lite/delegates/gpu/gl/converters/util.h"
+#include "tensorflow/lite/delegates/gpu/gl/gl_program.h"
+#include "tensorflow/lite/delegates/gpu/gl/gl_shader.h"
+#include "tensorflow/lite/delegates/gpu/gl/gl_texture.h"
+#include "tensorflow/lite/delegates/gpu/gl_delegate.h"
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+#if MEDIAPIPE_METAL_ENABLED
+#import <CoreVideo/CoreVideo.h>
+#import <Metal/Metal.h>
+#import <MetalKit/MetalKit.h>
+
+#import "mediapipe/gpu/MPPMetalHelper.h"
+#include "mediapipe/gpu/MPPMetalUtil.h"
+#endif  // MEDIAPIPE_METAL_ENABLED
+
+namespace {
+constexpr int kWorkgroupSize = 8;  // Block size for GPU shader.
+enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
+
+// Commonly used to compute the number of blocks to launch in a kernel.
+int NumGroups(const int size, const int group_size) {  // NOLINT
+  return (size + group_size - 1) / group_size;
+}
+
+bool CanUseGpu() {
+#if !MEDIAPIPE_DISABLE_GPU || MEDIAPIPE_METAL_ENABLED
+  // TODO: Configure GPU usage policy in individual calculators.
+  constexpr bool kAllowGpuProcessing = true;
+  return kAllowGpuProcessing;
+#else
+  return false;
+#endif  // !MEDIAPIPE_DISABLE_GPU || MEDIAPIPE_METAL_ENABLED
+}
+
+constexpr char kTensorsTag[] = "TENSORS";
+constexpr char kOutputSizeTag[] = "OUTPUT_SIZE";
+constexpr char kMaskTag[] = "MASK";
+
+absl::StatusOr<std::tuple<int, int, int>> GetHwcFromDims(
+    const std::vector<int>& dims) {
+  if (dims.size() == 3) {
+    return std::make_tuple(dims[0], dims[1], dims[2]);
+  } else if (dims.size() == 4) {
+    // BHWC format check B == 1
+    RET_CHECK_EQ(1, dims[0]) << "Expected batch to be 1 for BHWC heatmap";
+    return std::make_tuple(dims[1], dims[2], dims[3]);
+  } else {
+    RET_CHECK(false) << "Invalid shape for segmentation tensor " << dims.size();
+  }
+}
+}  // namespace
+
+namespace mediapipe {
+
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+using ::tflite::gpu::gl::GlProgram;
+using ::tflite::gpu::gl::GlShader;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+// Converts Tensors from a tflite segmentation model to an image mask.
+//
+// Performs optional upscale to OUTPUT_SIZE dimensions if provided,
+// otherwise the mask is the same size as input tensor.
+//
+// If at least one input tensor is already on GPU, processing happens on GPU and
+// the output mask is also stored on GPU. Otherwise, processing and the output
+// mask are both on CPU.
+//
+// On GPU, the mask is an RGBA image, in both the R & A channels, scaled 0-1.
+// On CPU, the mask is a ImageFormat::VEC32F1 image, with values scaled 0-1.
+//
+//
+// Inputs:
+//   One of the following TENSORS tags:
+//   TENSORS: Vector of Tensor,
+//            The tensor dimensions are specified in this calculator's options.
+//   OUTPUT_SIZE(optional): std::pair<int, int>,
+//                          If provided, the size to upscale mask to.
+//
+// Output:
+//   MASK: An Image output mask, RGBA(GPU) / VEC32F1(CPU).
+//
+// Options:
+//   See tensors_to_segmentation_calculator.proto
+//
+// Usage example:
+// node {
+//   calculator: "TensorsToSegmentationCalculator"
+//   input_stream: "TENSORS:tensors"
+//   input_stream: "OUTPUT_SIZE:size"
+//   output_stream: "MASK:hair_mask"
+//   node_options: {
+//     [mediapipe.TensorsToSegmentationCalculatorOptions] {
+//       output_layer_index: 1
+//       # gpu_origin: CONVENTIONAL # or TOP_LEFT
+//     }
+//   }
+// }
+//
+// Currently only OpenGLES 3.1 and CPU backends supported.
+// TODO Refactor and add support for other backends/platforms.
+//
+class TensorsToSegmentationCalculator : public CalculatorBase {
+ public:
+  static absl::Status GetContract(CalculatorContract* cc);
+
+  absl::Status Open(CalculatorContext* cc) override;
+  absl::Status Process(CalculatorContext* cc) override;
+  absl::Status Close(CalculatorContext* cc) override;
+
+ private:
+  absl::Status LoadOptions(CalculatorContext* cc);
+  absl::Status InitGpu(CalculatorContext* cc);
+  absl::Status ProcessGpu(CalculatorContext* cc);
+  absl::Status ProcessCpu(CalculatorContext* cc);
+  void GlRender();
+
+  bool DoesGpuTextureStartAtBottom() {
+    return options_.gpu_origin() != mediapipe::GpuOrigin_Mode_TOP_LEFT;
+  }
+
+  template <class T>
+  absl::Status ApplyActivation(cv::Mat& tensor_mat, cv::Mat* small_mask_mat);
+
+  ::mediapipe::TensorsToSegmentationCalculatorOptions options_;
+
+#if !MEDIAPIPE_DISABLE_GPU
+  mediapipe::GlCalculatorHelper gpu_helper_;
+  GLuint upsample_program_;
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+  std::unique_ptr<GlProgram> mask_program_31_;
+#else
+  GLuint mask_program_20_;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+#if MEDIAPIPE_METAL_ENABLED
+  MPPMetalHelper* metal_helper_ = nullptr;
+  id<MTLComputePipelineState> mask_program_;
+#endif  // MEDIAPIPE_METAL_ENABLED
+#endif  // !MEDIAPIPE_DISABLE_GPU
+};
+REGISTER_CALCULATOR(TensorsToSegmentationCalculator);
+
+// static
+absl::Status TensorsToSegmentationCalculator::GetContract(
+    CalculatorContract* cc) {
+  RET_CHECK(!cc->Inputs().GetTags().empty());
+  RET_CHECK(!cc->Outputs().GetTags().empty());
+
+  // Inputs.
+  cc->Inputs().Tag(kTensorsTag).Set<std::vector<Tensor>>();
+  if (cc->Inputs().HasTag(kOutputSizeTag)) {
+    cc->Inputs().Tag(kOutputSizeTag).Set<std::pair<int, int>>();
+  }
+
+  // Outputs.
+  cc->Outputs().Tag(kMaskTag).Set<Image>();
+
+  if (CanUseGpu()) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
+#if MEDIAPIPE_METAL_ENABLED
+    MP_RETURN_IF_ERROR([MPPMetalHelper updateContract:cc]);
+#endif  // MEDIAPIPE_METAL_ENABLED
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::Open(CalculatorContext* cc) {
+  cc->SetOffset(TimestampDiff(0));
+  bool use_gpu = false;
+
+  if (CanUseGpu()) {
+#if !MEDIAPIPE_DISABLE_GPU
+    use_gpu = true;
+    MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
+#if MEDIAPIPE_METAL_ENABLED
+    metal_helper_ = [[MPPMetalHelper alloc] initWithCalculatorContext:cc];
+    RET_CHECK(metal_helper_);
+#endif  // MEDIAPIPE_METAL_ENABLED
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  }
+
+  MP_RETURN_IF_ERROR(LoadOptions(cc));
+
+  if (use_gpu) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(InitGpu(cc));
+#else
+    RET_CHECK_FAIL() << "GPU processing disabled.";
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::Process(CalculatorContext* cc) {
+  if (cc->Inputs().Tag(kTensorsTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+
+  const auto& input_tensors =
+      cc->Inputs().Tag(kTensorsTag).Get<std::vector<Tensor>>();
+
+  bool use_gpu = false;
+  if (CanUseGpu()) {
+    // Use GPU processing only if at least one input tensor is already on GPU.
+    for (const auto& tensor : input_tensors) {
+      if (tensor.ready_on_gpu()) {
+        use_gpu = true;
+        break;
+      }
+    }
+  }
+
+  // Validate tensor channels and activation type.
+  {
+    RET_CHECK(!input_tensors.empty());
+    ASSIGN_OR_RETURN(auto hwc, GetHwcFromDims(input_tensors[0].shape().dims));
+    int tensor_channels = std::get<2>(hwc);
+    typedef mediapipe::TensorsToSegmentationCalculatorOptions Options;
+    switch (options_.activation()) {
+      case Options::NONE:
+        RET_CHECK_EQ(tensor_channels, 1);
+        break;
+      case Options::SIGMOID:
+        RET_CHECK_EQ(tensor_channels, 1);
+        break;
+      case Options::SOFTMAX:
+        RET_CHECK_EQ(tensor_channels, 2);
+        break;
+    }
+  }
+
+  if (use_gpu) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this, cc]() -> absl::Status {
+      MP_RETURN_IF_ERROR(ProcessGpu(cc));
+      return absl::OkStatus();
+    }));
+#else
+    RET_CHECK_FAIL() << "GPU processing disabled.";
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  } else {
+    MP_RETURN_IF_ERROR(ProcessCpu(cc));
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::Close(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  gpu_helper_.RunInGlContext([this] {
+    if (upsample_program_) glDeleteProgram(upsample_program_);
+    upsample_program_ = 0;
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    mask_program_31_.reset();
+#else
+    if (mask_program_20_) glDeleteProgram(mask_program_20_);
+    mask_program_20_ = 0;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+#if MEDIAPIPE_METAL_ENABLED
+    mask_program_ = nil;
+#endif  // MEDIAPIPE_METAL_ENABLED
+  });
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::ProcessCpu(
+    CalculatorContext* cc) {
+  // Get input streams, and dimensions.
+  const auto& input_tensors =
+      cc->Inputs().Tag(kTensorsTag).Get<std::vector<Tensor>>();
+  ASSIGN_OR_RETURN(auto hwc, GetHwcFromDims(input_tensors[0].shape().dims));
+  auto [tensor_height, tensor_width, tensor_channels] = hwc;
+  int output_width = tensor_width, output_height = tensor_height;
+  if (cc->Inputs().HasTag(kOutputSizeTag)) {
+    const auto& size =
+        cc->Inputs().Tag(kOutputSizeTag).Get<std::pair<int, int>>();
+    output_width = size.first;
+    output_height = size.second;
+  }
+
+  // Create initial working mask.
+  cv::Mat small_mask_mat(cv::Size(tensor_width, tensor_height), CV_32FC1);
+
+  // Wrap input tensor.
+  auto raw_input_tensor = &input_tensors[0];
+  auto raw_input_view = raw_input_tensor->GetCpuReadView();
+  const float* raw_input_data = raw_input_view.buffer<float>();
+  cv::Mat tensor_mat(cv::Size(tensor_width, tensor_height),
+                     CV_MAKETYPE(CV_32F, tensor_channels),
+                     const_cast<float*>(raw_input_data));
+
+  // Process mask tensor and apply activation function.
+  if (tensor_channels == 2) {
+    MP_RETURN_IF_ERROR(ApplyActivation<cv::Vec2f>(tensor_mat, &small_mask_mat));
+  } else if (tensor_channels == 1) {
+    RET_CHECK(mediapipe::TensorsToSegmentationCalculatorOptions::SOFTMAX !=
+              options_.activation());  // Requires 2 channels.
+    if (mediapipe::TensorsToSegmentationCalculatorOptions::NONE ==
+        options_.activation())  // Pass-through optimization.
+      tensor_mat.copyTo(small_mask_mat);
+    else
+      MP_RETURN_IF_ERROR(ApplyActivation<float>(tensor_mat, &small_mask_mat));
+  } else {
+    RET_CHECK_FAIL() << "Unsupported number of tensor channels "
+                     << tensor_channels;
+  }
+
+  // Send out image as CPU packet.
+  std::shared_ptr<ImageFrame> mask_frame = std::make_shared<ImageFrame>(
+      ImageFormat::VEC32F1, output_width, output_height);
+  std::unique_ptr<Image> output_mask = absl::make_unique<Image>(mask_frame);
+  cv::Mat output_mat = formats::MatView(output_mask.get());
+  // Upsample small mask into output.
+  cv::resize(small_mask_mat, output_mat, cv::Size(output_width, output_height));
+  cc->Outputs().Tag(kMaskTag).Add(output_mask.release(), cc->InputTimestamp());
+
+  return absl::OkStatus();
+}
+
+template <class T>
+absl::Status TensorsToSegmentationCalculator::ApplyActivation(
+    cv::Mat& tensor_mat, cv::Mat* small_mask_mat) {
+  // Configure activation function.
+  const int output_layer_index = options_.output_layer_index();
+  typedef mediapipe::TensorsToSegmentationCalculatorOptions Options;
+  const auto activation_fn = [&](const cv::Vec2f& mask_value) {
+    float new_mask_value = 0;
+    // TODO consider moving switch out of the loop,
+    // and also avoid float/Vec2f casting.
+    switch (options_.activation()) {
+      case Options::NONE: {
+        new_mask_value = mask_value[0];
+        break;
+      }
+      case Options::SIGMOID: {
+        const float pixel0 = mask_value[0];
+        new_mask_value = 1.0 / (std::exp(-pixel0) + 1.0);
+        break;
+      }
+      case Options::SOFTMAX: {
+        const float pixel0 = mask_value[0];
+        const float pixel1 = mask_value[1];
+        const float max_pixel = std::max(pixel0, pixel1);
+        const float min_pixel = std::min(pixel0, pixel1);
+        const float softmax_denom =
+            /*exp(max_pixel - max_pixel)=*/1.0f +
+            std::exp(min_pixel - max_pixel);
+        new_mask_value = std::exp(mask_value[output_layer_index] - max_pixel) /
+                         softmax_denom;
+        break;
+      }
+    }
+    return new_mask_value;
+  };
+
+  // Process mask tensor.
+  for (int i = 0; i < tensor_mat.rows; ++i) {
+    for (int j = 0; j < tensor_mat.cols; ++j) {
+      const T& input_pix = tensor_mat.at<T>(i, j);
+      const float mask_value = activation_fn(input_pix);
+      small_mask_mat->at<float>(i, j) = mask_value;
+    }
+  }
+
+  return absl::OkStatus();
+}
+
+// Steps:
+// 1. receive tensor
+// 2. process segmentation tensor into small mask
+// 3. upsample small mask into output mask to be same size as input image
+absl::Status TensorsToSegmentationCalculator::ProcessGpu(
+    CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  // Get input streams, and dimensions.
+  const auto& input_tensors =
+      cc->Inputs().Tag(kTensorsTag).Get<std::vector<Tensor>>();
+  ASSIGN_OR_RETURN(auto hwc, GetHwcFromDims(input_tensors[0].shape().dims));
+  auto [tensor_height, tensor_width, tensor_channels] = hwc;
+  int output_width = tensor_width, output_height = tensor_height;
+  if (cc->Inputs().HasTag(kOutputSizeTag)) {
+    const auto& size =
+        cc->Inputs().Tag(kOutputSizeTag).Get<std::pair<int, int>>();
+    output_width = size.first;
+    output_height = size.second;
+  }
+
+  // Create initial working mask texture.
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+  tflite::gpu::gl::GlTexture small_mask_texture;
+#else
+  mediapipe::GlTexture small_mask_texture;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+  // Run shader, process mask tensor.
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+  {
+    MP_RETURN_IF_ERROR(CreateReadWriteRgbaImageTexture(
+        tflite::gpu::DataType::UINT8,  // GL_RGBA8
+        {tensor_width, tensor_height}, &small_mask_texture));
+
+    const int output_index = 0;
+    glBindImageTexture(output_index, small_mask_texture.id(), 0, GL_FALSE, 0,
+                       GL_WRITE_ONLY, GL_RGBA8);
+
+    auto read_view = input_tensors[0].GetOpenGlBufferReadView();
+    glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 2, read_view.name());
+
+    const tflite::gpu::uint3 workgroups = {
+        NumGroups(tensor_width, kWorkgroupSize),
+        NumGroups(tensor_height, kWorkgroupSize), 1};
+
+    glUseProgram(mask_program_31_->id());
+    glUniform2i(glGetUniformLocation(mask_program_31_->id(), "out_size"),
+                tensor_width, tensor_height);
+
+    MP_RETURN_IF_ERROR(mask_program_31_->Dispatch(workgroups));
+  }
+#elif MEDIAPIPE_METAL_ENABLED
+  {
+    id<MTLCommandBuffer> command_buffer = [metal_helper_ commandBuffer];
+    command_buffer.label = @"SegmentationKernel";
+    id<MTLComputeCommandEncoder> command_encoder =
+        [command_buffer computeCommandEncoder];
+    [command_encoder setComputePipelineState:mask_program_];
+
+    auto read_view = input_tensors[0].GetMtlBufferReadView(command_buffer);
+    [command_encoder setBuffer:read_view.buffer() offset:0 atIndex:0];
+
+    mediapipe::GpuBuffer small_mask_buffer = [metal_helper_
+        mediapipeGpuBufferWithWidth:tensor_width
+                             height:tensor_height
+                             format:mediapipe::GpuBufferFormat::kBGRA32];
+    id<MTLTexture> small_mask_texture_metal =
+        [metal_helper_ metalTextureWithGpuBuffer:small_mask_buffer];
+    [command_encoder setTexture:small_mask_texture_metal atIndex:1];
+
+    unsigned int out_size[] = {static_cast<unsigned int>(tensor_width),
+                               static_cast<unsigned int>(tensor_height)};
+    [command_encoder setBytes:&out_size length:sizeof(out_size) atIndex:2];
+
+    MTLSize threads_per_group = MTLSizeMake(kWorkgroupSize, kWorkgroupSize, 1);
+    MTLSize threadgroups =
+        MTLSizeMake(NumGroups(tensor_width, kWorkgroupSize),
+                    NumGroups(tensor_height, kWorkgroupSize), 1);
+    [command_encoder dispatchThreadgroups:threadgroups
+                    threadsPerThreadgroup:threads_per_group];
+    [command_encoder endEncoding];
+    [command_buffer commit];
+
+    small_mask_texture = gpu_helper_.CreateSourceTexture(small_mask_buffer);
+  }
+#else
+  {
+    small_mask_texture = gpu_helper_.CreateDestinationTexture(
+        tensor_width, tensor_height,
+        mediapipe::GpuBufferFormat::kBGRA32);  // actually GL_RGBA8
+
+    // Go through CPU if not already texture 2D (no direct conversion yet).
+    // Tensor::GetOpenGlTexture2dReadView() doesn't automatically convert types.
+    if (!input_tensors[0].ready_as_opengl_texture_2d()) {
+      (void)input_tensors[0].GetCpuReadView();
+    }
+
+    auto read_view = input_tensors[0].GetOpenGlTexture2dReadView();
+
+    gpu_helper_.BindFramebuffer(small_mask_texture);
+    glActiveTexture(GL_TEXTURE1);
+    glBindTexture(GL_TEXTURE_2D, read_view.name());
+    glUseProgram(mask_program_20_);
+    GlRender();
+    glBindTexture(GL_TEXTURE_2D, 0);
+    glFlush();
+  }
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+  // Upsample small mask into output.
+  mediapipe::GlTexture output_texture = gpu_helper_.CreateDestinationTexture(
+      output_width, output_height,
+      mediapipe::GpuBufferFormat::kBGRA32);  // actually GL_RGBA8
+
+  // Run shader, upsample result.
+  {
+    gpu_helper_.BindFramebuffer(output_texture);
+    glActiveTexture(GL_TEXTURE1);
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    glBindTexture(GL_TEXTURE_2D, small_mask_texture.id());
+#else
+    glBindTexture(GL_TEXTURE_2D, small_mask_texture.name());
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    glUseProgram(upsample_program_);
+    GlRender();
+    glBindTexture(GL_TEXTURE_2D, 0);
+    glFlush();
+  }
+
+  // Send out image as GPU packet.
+  auto output_image = output_texture.GetFrame<Image>();
+  cc->Outputs().Tag(kMaskTag).Add(output_image.release(), cc->InputTimestamp());
+
+  // Cleanup
+  output_texture.Release();
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+void TensorsToSegmentationCalculator::GlRender() {
+#if !MEDIAPIPE_DISABLE_GPU
+  static const GLfloat square_vertices[] = {
+      -1.0f, -1.0f,  // bottom left
+      1.0f,  -1.0f,  // bottom right
+      -1.0f, 1.0f,   // top left
+      1.0f,  1.0f,   // top right
+  };
+  static const GLfloat texture_vertices[] = {
+      0.0f, 0.0f,  // bottom left
+      1.0f, 0.0f,  // bottom right
+      0.0f, 1.0f,  // top left
+      1.0f, 1.0f,  // top right
+  };
+
+  // vertex storage
+  GLuint vbo[2];
+  glGenBuffers(2, vbo);
+  GLuint vao;
+  glGenVertexArrays(1, &vao);
+  glBindVertexArray(vao);
+
+  // vbo 0
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_VERTEX);
+  glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // vbo 1
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // draw
+  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
+
+  // cleanup
+  glDisableVertexAttribArray(ATTRIB_VERTEX);
+  glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glBindBuffer(GL_ARRAY_BUFFER, 0);
+  glBindVertexArray(0);
+  glDeleteVertexArrays(1, &vao);
+  glDeleteBuffers(2, vbo);
+#endif  // !MEDIAPIPE_DISABLE_GPU
+}
+
+absl::Status TensorsToSegmentationCalculator::LoadOptions(
+    CalculatorContext* cc) {
+  // Get calculator options specified in the graph.
+  options_ = cc->Options<::mediapipe::TensorsToSegmentationCalculatorOptions>();
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::InitGpu(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this]() -> absl::Status {
+  // A shader to process a segmentation tensor into an output mask.
+  // Currently uses 4 channels for output, and sets R+A channels as mask value.
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    // GLES 3.1
+    const tflite::gpu::uint3 workgroup_size = {kWorkgroupSize, kWorkgroupSize,
+                                               1};
+    const std::string shader_header =
+        absl::StrCat(tflite::gpu::gl::GetShaderHeader(workgroup_size), R"(
+precision highp float;
+
+layout(rgba8, binding = 0) writeonly uniform highp image2D output_texture;
+
+uniform ivec2 out_size;
+)");
+    /* Shader defines will be inserted here. */
+
+    const std::string shader_src_main = R"(
+layout(std430, binding = 2) readonly buffer B0 {
+#ifdef TWO_CHANNEL_INPUT
+  vec2 elements[];
+#else
+  float elements[];
+#endif // TWO_CHANNEL_INPUT
+} input_data;   // data tensor
+
+void main() {
+  int out_width = out_size.x;
+  int out_height = out_size.y;
+
+  ivec2 gid = ivec2(gl_GlobalInvocationID.xy);
+  if (gid.x >= out_width || gid.y >= out_height) { return; }
+  int linear_index = gid.y * out_width + gid.x;
+
+#ifdef TWO_CHANNEL_INPUT
+  vec2 input_value = input_data.elements[linear_index];
+#else
+  vec2 input_value = vec2(input_data.elements[linear_index], 0.0);
+#endif // TWO_CHANNEL_INPUT
+
+// Run activation function.
+// One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
+#ifdef FN_SOFTMAX
+  // Only two channel input tensor is supported.
+  vec2 input_px = input_value.rg;
+  float shift = max(input_px.r, input_px.g);
+  float softmax_denom = exp(input_px.r - shift) + exp(input_px.g - shift);
+  float new_mask_value =
+      exp(input_px[OUTPUT_LAYER_INDEX] - shift) / softmax_denom;
+#endif // FN_SOFTMAX
+
+#ifdef FN_SIGMOID
+  float new_mask_value = 1.0 / (exp(-input_value.r) + 1.0);
+#endif // FN_SIGMOID
+
+#ifdef FN_NONE
+  float new_mask_value = input_value.r;
+#endif // FN_NONE
+
+#ifdef FLIP_Y_COORD
+  int y_coord = out_height - gid.y - 1;
+#else
+  int y_coord = gid.y;
+#endif  // defined(FLIP_Y_COORD)
+  ivec2 output_coordinate = ivec2(gid.x, y_coord);
+
+  vec4 out_value = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
+  imageStore(output_texture, output_coordinate, out_value);
+})";
+
+#elif MEDIAPIPE_METAL_ENABLED
+    // METAL
+    const std::string shader_header = R"(
+#include <metal_stdlib>
+using namespace metal;
+)";
+    /* Shader defines will be inserted here. */
+
+    const std::string shader_src_main = R"(
+kernel void segmentationKernel(
+#ifdef TWO_CHANNEL_INPUT
+    device float2*     elements        [[ buffer(0) ]],
+#else
+    device float*      elements        [[ buffer(0) ]],
+#endif // TWO_CHANNEL_INPUT
+    texture2d<float, access::write>  output_texture  [[ texture(1) ]],
+    constant uint*      out_size        [[ buffer(2) ]],
+    uint2               gid             [[ thread_position_in_grid ]])
+{
+  uint out_width = out_size[0];
+  uint out_height = out_size[1];
+
+  if (gid.x >= out_width || gid.y >= out_height) { return; }
+  uint linear_index = gid.y * out_width + gid.x;
+
+#ifdef TWO_CHANNEL_INPUT
+  float2 input_value = elements[linear_index];
+#else
+  float2 input_value = float2(elements[linear_index], 0.0);
+#endif // TWO_CHANNEL_INPUT
+
+// Run activation function.
+// One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
+#ifdef FN_SOFTMAX
+  // Only two channel input tensor is supported.
+  float2 input_px = input_value.xy;
+  float shift = max(input_px.x, input_px.y);
+  float softmax_denom = exp(input_px.r - shift) + exp(input_px.g - shift);
+  float new_mask_value =
+      exp(input_px[OUTPUT_LAYER_INDEX] - shift) / softmax_denom;
+#endif // FN_SOFTMAX
+
+#ifdef FN_SIGMOID
+  float new_mask_value = 1.0 / (exp(-input_value.x) + 1.0);
+#endif // FN_SIGMOID
+
+#ifdef FN_NONE
+  float new_mask_value = input_value.x;
+#endif // FN_NONE
+
+#ifdef FLIP_Y_COORD
+  int y_coord = out_height - gid.y - 1;
+#else
+  int y_coord = gid.y;
+#endif  // defined(FLIP_Y_COORD)
+  uint2 output_coordinate = uint2(gid.x, y_coord);
+
+  float4 out_value = float4(new_mask_value, 0.0, 0.0, new_mask_value);
+  output_texture.write(out_value, output_coordinate);
+}
+)";
+
+#else
+    // GLES 2.0
+    const std::string shader_header = absl::StrCat(
+        std::string(mediapipe::kMediaPipeFragmentShaderPreamble), R"(
+DEFAULT_PRECISION(mediump, float)
+)");
+    /* Shader defines will be inserted here. */
+
+    const std::string shader_src_main = R"(
+in vec2 sample_coordinate;
+
+uniform sampler2D input_texture;
+
+#ifdef GL_ES
+#define fragColor gl_FragColor
+#else
+out vec4 fragColor;
+#endif  // defined(GL_ES);
+
+void main() {
+#ifdef FLIP_Y_COORD
+  float y_coord = 1.0 - sample_coordinate.y;
+#else
+  float y_coord = sample_coordinate.y;
+#endif  // defined(FLIP_Y_COORD)
+  vec2 adjusted_coordinate = vec2(sample_coordinate.x, y_coord);
+  vec4 input_value = texture2D(input_texture, adjusted_coordinate);
+
+  // Run activation function.
+  // One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
+
+#ifdef FN_SOFTMAX
+  // Only two channel input tensor is supported.
+  vec2 input_px = input_value.rg;
+  float shift = max(input_px.r, input_px.g);
+  float softmax_denom = exp(input_px.r - shift) + exp(input_px.g - shift);
+  float new_mask_value =
+      exp(mix(input_px.r, input_px.g, float(OUTPUT_LAYER_INDEX)) - shift) / softmax_denom;
+#endif // FN_SOFTMAX
+
+#ifdef FN_SIGMOID
+  float new_mask_value = 1.0 / (exp(-input_value.r) + 1.0);
+#endif // FN_SIGMOID
+
+#ifdef FN_NONE
+  float new_mask_value = input_value.r;
+#endif // FN_NONE
+
+  vec4 out_value = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
+  fragColor = out_value;
+})";
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+    // Shader defines.
+    typedef mediapipe::TensorsToSegmentationCalculatorOptions Options;
+    const std::string output_layer_index =
+        "\n#define OUTPUT_LAYER_INDEX int(" +
+        std::to_string(options_.output_layer_index()) + ")";
+    const std::string flip_y_coord =
+        DoesGpuTextureStartAtBottom() ? "\n#define FLIP_Y_COORD" : "";
+    const std::string fn_none =
+        options_.activation() == Options::NONE ? "\n#define FN_NONE" : "";
+    const std::string fn_sigmoid =
+        options_.activation() == Options::SIGMOID ? "\n#define FN_SIGMOID" : "";
+    const std::string fn_softmax =
+        options_.activation() == Options::SOFTMAX ? "\n#define FN_SOFTMAX" : "";
+    const std::string two_channel = options_.activation() == Options::SOFTMAX
+                                        ? "\n#define TWO_CHANNEL_INPUT"
+                                        : "";
+    const std::string shader_defines =
+        absl::StrCat(output_layer_index, flip_y_coord, fn_softmax, fn_sigmoid,
+                     fn_none, two_channel);
+
+    // Build full shader.
+    const std::string shader_src_no_previous =
+        absl::StrCat(shader_header, shader_defines, shader_src_main);
+
+    // Vertex shader attributes.
+    const GLint attr_location[NUM_ATTRIBUTES] = {
+        ATTRIB_VERTEX,
+        ATTRIB_TEXTURE_POSITION,
+    };
+    const GLchar* attr_name[NUM_ATTRIBUTES] = {
+        "position",
+        "texture_coordinate",
+    };
+
+    // Main shader program & parameters
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    GlShader shader_without_previous;
+    MP_RETURN_IF_ERROR(GlShader::CompileShader(
+        GL_COMPUTE_SHADER, shader_src_no_previous, &shader_without_previous));
+    mask_program_31_ = absl::make_unique<GlProgram>();
+    MP_RETURN_IF_ERROR(GlProgram::CreateWithShader(shader_without_previous,
+                                                   mask_program_31_.get()));
+#elif MEDIAPIPE_METAL_ENABLED
+    id<MTLDevice> device = metal_helper_.mtlDevice;
+    NSString* library_source =
+        [NSString stringWithUTF8String:shader_src_no_previous.c_str()];
+    NSError* error = nil;
+    id<MTLLibrary> library = [device newLibraryWithSource:library_source
+                                                  options:nullptr
+                                                    error:&error];
+    RET_CHECK(library != nil) << "Couldn't create shader library "
+                              << [[error localizedDescription] UTF8String];
+    id<MTLFunction> kernel_func = nil;
+    kernel_func = [library newFunctionWithName:@"segmentationKernel"];
+    RET_CHECK(kernel_func != nil) << "Couldn't create kernel function.";
+    mask_program_ =
+        [device newComputePipelineStateWithFunction:kernel_func error:&error];
+    RET_CHECK(mask_program_ != nil) << "Couldn't create pipeline state " <<
+        [[error localizedDescription] UTF8String];
+#else
+    mediapipe::GlhCreateProgram(
+        mediapipe::kBasicVertexShader, shader_src_no_previous.c_str(),
+        NUM_ATTRIBUTES, &attr_name[0], attr_location, &mask_program_20_);
+    RET_CHECK(mask_program_20_) << "Problem initializing the program.";
+    glUseProgram(mask_program_20_);
+    glUniform1i(glGetUniformLocation(mask_program_20_, "input_texture"), 1);
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+    // Simple pass-through program, used for hardware upsampling.
+    mediapipe::GlhCreateProgram(
+        mediapipe::kBasicVertexShader, mediapipe::kBasicTexturedFragmentShader,
+        NUM_ATTRIBUTES, &attr_name[0], attr_location, &upsample_program_);
+    RET_CHECK(upsample_program_) << "Problem initializing the program.";
+    glUseProgram(upsample_program_);
+    glUniform1i(glGetUniformLocation(upsample_program_, "video_frame"), 1);
+
+    return absl::OkStatus();
+  }));
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+}  // namespace mediapipe

mediapipe/calculators/tensor/tensors_to_segmentation_calculator.proto

@@ -0,0 +1,46 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto2";
+
+package mediapipe;
+
+import "mediapipe/framework/calculator.proto";
+import "mediapipe/gpu/gpu_origin.proto";
+
+message TensorsToSegmentationCalculatorOptions {
+  extend mediapipe.CalculatorOptions {
+    optional TensorsToSegmentationCalculatorOptions ext = 374311106;
+  }
+
+  // For CONVENTIONAL mode in OpenGL, textures start at bottom and needs
+  // to be flipped vertically as tensors are expected to start at top.
+  // (DEFAULT or unset is interpreted as CONVENTIONAL.)
+  optional GpuOrigin.Mode gpu_origin = 1;
+
+  // Supported activation functions for filtering.
+  enum Activation {
+    NONE = 0;     // Assumes 1-channel input tensor.
+    SIGMOID = 1;  // Assumes 1-channel input tensor.
+    SOFTMAX = 2;  // Assumes 2-channel input tensor.
+  }
+  // Activation function to apply to input tensor.
+  // Softmax requires a 2-channel tensor, see output_layer_index below.
+  optional Activation activation = 2 [default = NONE];
+
+  // Channel to use for processing tensor.
+  // Only applies when using activation=SOFTMAX.
+  // Works on two channel input tensor only.
+  optional int32 output_layer_index = 3 [default = 1];
+}

mediapipe/calculators/tensor/testdata/face_detection_test.pbtxt

@@ -4,7 +4,7 @@ output_stream: "detections"
 
 # Subgraph that detects faces.
 node {
-  calculator: "FaceDetectionFrontCpu"
+  calculator: "FaceDetectionShortRangeCpu"
   input_stream: "IMAGE:image"
   output_stream: "DETECTIONS:detections"
 }

mediapipe/calculators/tensorflow/tensorflow_inference_calculator.cc

@@ -490,7 +490,7 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
               << keyed_tensors.first;
         }
       } else {
-        // Pad by replicating the first tens  or, then ignore the values.
+        // Pad by replicating the first tensor, then ignore the values.
         keyed_tensors.second.resize(options_.batch_size());
         std::fill(keyed_tensors.second.begin() +
                       inference_state->batch_timestamps_.size(),

mediapipe/calculators/tflite/tflite_inference_calculator.cc

@@ -128,9 +128,23 @@ struct GPUData {
 }  // namespace
 #endif  // MEDIAPIPE_TFLITE_GPU_SUPPORTED
 
+namespace {
+
+int GetXnnpackDefaultNumThreads() {
+#if defined(MEDIAPIPE_ANDROID) || defined(MEDIAPIPE_IOS) || \
+    defined(__EMSCRIPTEN_PTHREADS__)
+  constexpr int kMinNumThreadsByDefault = 1;
+  constexpr int kMaxNumThreadsByDefault = 4;
+  return std::clamp(NumCPUCores() / 2, kMinNumThreadsByDefault,
+                    kMaxNumThreadsByDefault);
+#else
+  return 1;
+#endif  // MEDIAPIPE_ANDROID || MEDIAPIPE_IOS || __EMSCRIPTEN_PTHREADS__
+}
+
 // Returns number of threads to configure XNNPACK delegate with.
-// (Equal to user provided value if specified.  Otherwise, it returns number of
-// high cores (hard-coded to 1 for Emscripten without Threads extension))
+// Returns user provided value if specified. Otherwise, tries to choose optimal
+// number of threads depending on the device.
 int GetXnnpackNumThreads(
     const mediapipe::TfLiteInferenceCalculatorOptions& opts) {
   static constexpr int kDefaultNumThreads = -1;
@@ -138,13 +152,11 @@ int GetXnnpackNumThreads(
       opts.delegate().xnnpack().num_threads() != kDefaultNumThreads) {
     return opts.delegate().xnnpack().num_threads();
   }
-#if !defined(__EMSCRIPTEN__) || defined(__EMSCRIPTEN_PTHREADS__)
-  return InferHigherCoreIds().size();
-#else
-  return 1;
-#endif  // !__EMSCRIPTEN__ || __EMSCRIPTEN_PTHREADS__
+  return GetXnnpackDefaultNumThreads();
 }
 
+}  // namespace
+
 // Calculator Header Section
 
 // Runs inference on the provided input TFLite tensors and TFLite model.
@@ -737,8 +749,8 @@ absl::Status TfLiteInferenceCalculator::InitTFLiteGPURunner(
       break;
     }
   }
-  MP_RETURN_IF_ERROR(
-      tflite_gpu_runner_->InitializeWithModel(model, *op_resolver_ptr));
+  MP_RETURN_IF_ERROR(tflite_gpu_runner_->InitializeWithModel(
+      model, *op_resolver_ptr, /*allow_quant_ops=*/true));
 
   // Allocate interpreter memory for cpu output.
   if (!gpu_output_) {
@@ -969,6 +981,10 @@ absl::Status TfLiteInferenceCalculator::LoadDelegate(CalculatorContext* cc) {
   const int kHalfSize = 2;  // sizeof(half)
   // Configure and create the delegate.
   TFLGpuDelegateOptions options;
+  // `enable_quantization` enables the run of sparse models i.e. the models with
+  // DENSIFY op preceding DEQUINTIZE op. Both ops get removed from the execution
+  // graph after the tensor of the weights is read.
+  options.enable_quantization = true;
   options.allow_precision_loss = allow_precision_loss_;
   options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeActive;
   if (!delegate_)

mediapipe/calculators/util/BUILD

@@ -840,6 +840,20 @@ cc_test(
     ],
 )
 
+cc_library(
+    name = "world_landmark_projection_calculator",
+    srcs = ["world_landmark_projection_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/formats:rect_cc_proto",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
+    ],
+    alwayslink = 1,
+)
+
 mediapipe_proto_library(
     name = "landmarks_smoothing_calculator_proto",
     srcs = ["landmarks_smoothing_calculator.proto"],
@@ -859,6 +873,7 @@ cc_library(
         "//mediapipe/framework:calculator_framework",
         "//mediapipe/framework:timestamp",
         "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/formats:rect_cc_proto",
         "//mediapipe/framework/port:ret_check",
         "//mediapipe/util/filtering:one_euro_filter",
         "//mediapipe/util/filtering:relative_velocity_filter",
@@ -893,6 +908,31 @@ cc_library(
     alwayslink = 1,
 )
 
+mediapipe_proto_library(
+    name = "visibility_copy_calculator_proto",
+    srcs = ["visibility_copy_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
+cc_library(
+    name = "visibility_copy_calculator",
+    srcs = ["visibility_copy_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":visibility_copy_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:timestamp",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/port:ret_check",
+        "@com_google_absl//absl/algorithm:container",
+    ],
+    alwayslink = 1,
+)
+
 cc_library(
     name = "landmarks_to_floats_calculator",
     srcs = ["landmarks_to_floats_calculator.cc"],

mediapipe/calculators/util/annotation_overlay_calculator.cc

@@ -272,6 +272,15 @@ absl::Status AnnotationOverlayCalculator::Open(CalculatorContext* cc) {
 }
 
 absl::Status AnnotationOverlayCalculator::Process(CalculatorContext* cc) {
+  if (cc->Inputs().HasTag(kGpuBufferTag) &&
+      cc->Inputs().Tag(kGpuBufferTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+  if (cc->Inputs().HasTag(kImageFrameTag) &&
+      cc->Inputs().Tag(kImageFrameTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+
   // Initialize render target, drawn with OpenCV.
   std::unique_ptr<cv::Mat> image_mat;
   ImageFormat::Format target_format;

mediapipe/calculators/util/detections_to_rects_calculator.cc

@@ -203,6 +203,9 @@ absl::Status DetectionsToRectsCalculator::Process(CalculatorContext* cc) {
       cc->Inputs().Tag(kDetectionsTag).IsEmpty()) {
     return absl::OkStatus();
   }
+  if (rotate_ && !HasTagValue(cc, kImageSizeTag)) {
+    return absl::OkStatus();
+  }
 
   std::vector<Detection> detections;
   if (cc->Inputs().HasTag(kDetectionTag)) {
@@ -323,7 +326,7 @@ absl::Status DetectionsToRectsCalculator::ComputeRotation(
 DetectionSpec DetectionsToRectsCalculator::GetDetectionSpec(
     const CalculatorContext* cc) {
   absl::optional<std::pair<int, int>> image_size;
-  if (cc->Inputs().HasTag(kImageSizeTag)) {
+  if (HasTagValue(cc->Inputs(), kImageSizeTag)) {
     image_size = cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
   }
 

mediapipe/calculators/util/detections_to_rects_calculator_test.cc

@@ -157,6 +157,12 @@ TEST(DetectionsToRectsCalculatorTest, DetectionKeyPointsToRect) {
       /*image_size=*/{640, 480});
   MP_ASSERT_OK(status_or_value);
   EXPECT_THAT(status_or_value.value(), RectEq(480, 360, 320, 240));
+
+  status_or_value = RunDetectionKeyPointsToRectCalculation(
+      /*detection=*/DetectionWithKeyPoints({{0.25f, 0.25f}, {0.75f, 0.75f}}),
+      /*image_size=*/{0, 0});
+  MP_ASSERT_OK(status_or_value);
+  EXPECT_THAT(status_or_value.value(), RectEq(0, 0, 0, 0));
 }
 
 TEST(DetectionsToRectsCalculatorTest, DetectionToNormalizedRect) {

mediapipe/calculators/util/filter_collection_calculator.cc

@@ -32,11 +32,15 @@ typedef FilterCollectionCalculator<std::vector<::mediapipe::NormalizedRect>>
     FilterNormalizedRectCollectionCalculator;
 REGISTER_CALCULATOR(FilterNormalizedRectCollectionCalculator);
 
-typedef FilterCollectionCalculator<
-    std::vector<::mediapipe::NormalizedLandmarkList>>
+typedef FilterCollectionCalculator<std::vector<::mediapipe::LandmarkList>>
     FilterLandmarkListCollectionCalculator;
 REGISTER_CALCULATOR(FilterLandmarkListCollectionCalculator);
 
+typedef FilterCollectionCalculator<
+    std::vector<::mediapipe::NormalizedLandmarkList>>
+    FilterNormalizedLandmarkListCollectionCalculator;
+REGISTER_CALCULATOR(FilterNormalizedLandmarkListCollectionCalculator);
+
 typedef FilterCollectionCalculator<std::vector<::mediapipe::ClassificationList>>
     FilterClassificationListCollectionCalculator;
 REGISTER_CALCULATOR(FilterClassificationListCollectionCalculator);

mediapipe/calculators/util/landmarks_smoothing_calculator.cc

@@ -18,6 +18,7 @@
 #include "mediapipe/calculators/util/landmarks_smoothing_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/formats/landmark.pb.h"
+#include "mediapipe/framework/formats/rect.pb.h"
 #include "mediapipe/framework/port/ret_check.h"
 #include "mediapipe/framework/timestamp.h"
 #include "mediapipe/util/filtering/one_euro_filter.h"
@@ -30,6 +31,7 @@ namespace {
 constexpr char kNormalizedLandmarksTag[] = "NORM_LANDMARKS";
 constexpr char kLandmarksTag[] = "LANDMARKS";
 constexpr char kImageSizeTag[] = "IMAGE_SIZE";
+constexpr char kObjectScaleRoiTag[] = "OBJECT_SCALE_ROI";
 constexpr char kNormalizedFilteredLandmarksTag[] = "NORM_FILTERED_LANDMARKS";
 constexpr char kFilteredLandmarksTag[] = "FILTERED_LANDMARKS";
 
@@ -94,6 +96,18 @@ float GetObjectScale(const LandmarkList& landmarks) {
   return (object_width + object_height) / 2.0f;
 }
 
+float GetObjectScale(const NormalizedRect& roi, const int image_width,
+                     const int image_height) {
+  const float object_width = roi.width() * image_width;
+  const float object_height = roi.height() * image_height;
+
+  return (object_width + object_height) / 2.0f;
+}
+
+float GetObjectScale(const Rect& roi) {
+  return (roi.width() + roi.height()) / 2.0f;
+}
+
 // Abstract class for various landmarks filters.
 class LandmarksFilter {
  public:
@@ -103,6 +117,7 @@ class LandmarksFilter {
 
   virtual absl::Status Apply(const LandmarkList& in_landmarks,
                              const absl::Duration& timestamp,
+                             const absl::optional<float> object_scale_opt,
                              LandmarkList* out_landmarks) = 0;
 };
 
@@ -111,6 +126,7 @@ class NoFilter : public LandmarksFilter {
  public:
   absl::Status Apply(const LandmarkList& in_landmarks,
                      const absl::Duration& timestamp,
+                     const absl::optional<float> object_scale_opt,
                      LandmarkList* out_landmarks) override {
     *out_landmarks = in_landmarks;
     return absl::OkStatus();
@@ -136,13 +152,15 @@ class VelocityFilter : public LandmarksFilter {
 
   absl::Status Apply(const LandmarkList& in_landmarks,
                      const absl::Duration& timestamp,
+                     const absl::optional<float> object_scale_opt,
                      LandmarkList* out_landmarks) override {
     // Get value scale as inverse value of the object scale.
     // If value is too small smoothing will be disabled and landmarks will be
     // returned as is.
     float value_scale = 1.0f;
     if (!disable_value_scaling_) {
-      const float object_scale = GetObjectScale(in_landmarks);
+      const float object_scale =
+          object_scale_opt ? *object_scale_opt : GetObjectScale(in_landmarks);
       if (object_scale < min_allowed_object_scale_) {
         *out_landmarks = in_landmarks;
         return absl::OkStatus();
@@ -205,12 +223,14 @@ class VelocityFilter : public LandmarksFilter {
 class OneEuroFilterImpl : public LandmarksFilter {
  public:
   OneEuroFilterImpl(double frequency, double min_cutoff, double beta,
-                    double derivate_cutoff, float min_allowed_object_scale)
+                    double derivate_cutoff, float min_allowed_object_scale,
+                    bool disable_value_scaling)
       : frequency_(frequency),
         min_cutoff_(min_cutoff),
         beta_(beta),
         derivate_cutoff_(derivate_cutoff),
-        min_allowed_object_scale_(min_allowed_object_scale) {}
+        min_allowed_object_scale_(min_allowed_object_scale),
+        disable_value_scaling_(disable_value_scaling) {}
 
   absl::Status Reset() override {
     x_filters_.clear();
@@ -221,16 +241,24 @@ class OneEuroFilterImpl : public LandmarksFilter {
 
   absl::Status Apply(const LandmarkList& in_landmarks,
                      const absl::Duration& timestamp,
+                     const absl::optional<float> object_scale_opt,
                      LandmarkList* out_landmarks) override {
     // Initialize filters once.
     MP_RETURN_IF_ERROR(InitializeFiltersIfEmpty(in_landmarks.landmark_size()));
 
-    const float object_scale = GetObjectScale(in_landmarks);
-    if (object_scale < min_allowed_object_scale_) {
-      *out_landmarks = in_landmarks;
-      return absl::OkStatus();
+    // Get value scale as inverse value of the object scale.
+    // If value is too small smoothing will be disabled and landmarks will be
+    // returned as is.
+    float value_scale = 1.0f;
+    if (!disable_value_scaling_) {
+      const float object_scale =
+          object_scale_opt ? *object_scale_opt : GetObjectScale(in_landmarks);
+      if (object_scale < min_allowed_object_scale_) {
+        *out_landmarks = in_landmarks;
+        return absl::OkStatus();
+      }
+      value_scale = 1.0f / object_scale;
     }
-    const float value_scale = 1.0f / object_scale;
 
     // Filter landmarks. Every axis of every landmark is filtered separately.
     for (int i = 0; i < in_landmarks.landmark_size(); ++i) {
@@ -277,6 +305,7 @@ class OneEuroFilterImpl : public LandmarksFilter {
   double beta_;
   double derivate_cutoff_;
   double min_allowed_object_scale_;
+  bool disable_value_scaling_;
 
   std::vector<OneEuroFilter> x_filters_;
   std::vector<OneEuroFilter> y_filters_;
@@ -292,6 +321,10 @@ class OneEuroFilterImpl : public LandmarksFilter {
 //   IMAGE_SIZE: A std::pair<int, int> represention of image width and height.
 //     Required to perform all computations in absolute coordinates to avoid any
 //     influence of normalized values.
+//   OBJECT_SCALE_ROI (optional): A NormRect or Rect (depending on the format of
+//     input landmarks) used to determine the object scale for some of the
+//     filters. If not provided - object scale will be calculated from
+//     landmarks.
 //
 // Outputs:
 //   NORM_FILTERED_LANDMARKS: A NormalizedLandmarkList of smoothed landmarks.
@@ -301,6 +334,7 @@ class OneEuroFilterImpl : public LandmarksFilter {
 //     calculator: "LandmarksSmoothingCalculator"
 //     input_stream: "NORM_LANDMARKS:pose_landmarks"
 //     input_stream: "IMAGE_SIZE:image_size"
+//     input_stream: "OBJECT_SCALE_ROI:roi"
 //     output_stream: "NORM_FILTERED_LANDMARKS:pose_landmarks_filtered"
 //     options: {
 //       [mediapipe.LandmarksSmoothingCalculatorOptions.ext] {
@@ -330,9 +364,17 @@ absl::Status LandmarksSmoothingCalculator::GetContract(CalculatorContract* cc) {
     cc->Outputs()
         .Tag(kNormalizedFilteredLandmarksTag)
         .Set<NormalizedLandmarkList>();
+
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag)) {
+      cc->Inputs().Tag(kObjectScaleRoiTag).Set<NormalizedRect>();
+    }
   } else {
     cc->Inputs().Tag(kLandmarksTag).Set<LandmarkList>();
     cc->Outputs().Tag(kFilteredLandmarksTag).Set<LandmarkList>();
+
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag)) {
+      cc->Inputs().Tag(kObjectScaleRoiTag).Set<Rect>();
+    }
   }
 
   return absl::OkStatus();
@@ -357,7 +399,8 @@ absl::Status LandmarksSmoothingCalculator::Open(CalculatorContext* cc) {
         options.one_euro_filter().min_cutoff(),
         options.one_euro_filter().beta(),
         options.one_euro_filter().derivate_cutoff(),
-        options.one_euro_filter().min_allowed_object_scale());
+        options.one_euro_filter().min_allowed_object_scale(),
+        options.one_euro_filter().disable_value_scaling());
   } else {
     RET_CHECK_FAIL()
         << "Landmarks filter is either not specified or not supported";
@@ -389,13 +432,20 @@ absl::Status LandmarksSmoothingCalculator::Process(CalculatorContext* cc) {
     std::tie(image_width, image_height) =
         cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
 
+    absl::optional<float> object_scale;
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag) &&
+        !cc->Inputs().Tag(kObjectScaleRoiTag).IsEmpty()) {
+      auto& roi = cc->Inputs().Tag(kObjectScaleRoiTag).Get<NormalizedRect>();
+      object_scale = GetObjectScale(roi, image_width, image_height);
+    }
+
     auto in_landmarks = absl::make_unique<LandmarkList>();
     NormalizedLandmarksToLandmarks(in_norm_landmarks, image_width, image_height,
                                    in_landmarks.get());
 
     auto out_landmarks = absl::make_unique<LandmarkList>();
-    MP_RETURN_IF_ERROR(landmarks_filter_->Apply(*in_landmarks, timestamp,
-                                                out_landmarks.get()));
+    MP_RETURN_IF_ERROR(landmarks_filter_->Apply(
+        *in_landmarks, timestamp, object_scale, out_landmarks.get()));
 
     auto out_norm_landmarks = absl::make_unique<NormalizedLandmarkList>();
     LandmarksToNormalizedLandmarks(*out_landmarks, image_width, image_height,
@@ -408,9 +458,16 @@ absl::Status LandmarksSmoothingCalculator::Process(CalculatorContext* cc) {
     const auto& in_landmarks =
         cc->Inputs().Tag(kLandmarksTag).Get<LandmarkList>();
 
+    absl::optional<float> object_scale;
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag) &&
+        !cc->Inputs().Tag(kObjectScaleRoiTag).IsEmpty()) {
+      auto& roi = cc->Inputs().Tag(kObjectScaleRoiTag).Get<Rect>();
+      object_scale = GetObjectScale(roi);
+    }
+
     auto out_landmarks = absl::make_unique<LandmarkList>();
-    MP_RETURN_IF_ERROR(
-        landmarks_filter_->Apply(in_landmarks, timestamp, out_landmarks.get()));
+    MP_RETURN_IF_ERROR(landmarks_filter_->Apply(
+        in_landmarks, timestamp, object_scale, out_landmarks.get()));
 
     cc->Outputs()
         .Tag(kFilteredLandmarksTag)

mediapipe/calculators/util/landmarks_smoothing_calculator.proto

@@ -41,9 +41,9 @@ message LandmarksSmoothingCalculatorOptions {
     optional float min_allowed_object_scale = 3 [default = 1e-6];
 
     // Disable value scaling based on object size and use `1.0` instead.
-    // Value scale is calculated as inverse value of object size. Object size is
-    // calculated as maximum side of rectangular bounding box of the object in
-    // XY plane.
+    // If not disabled, value scale is calculated as inverse value of object
+    // size. Object size is calculated as maximum side of rectangular bounding
+    // box of the object in XY plane.
     optional bool disable_value_scaling = 4 [default = false];
   }
 
@@ -72,6 +72,12 @@ message LandmarksSmoothingCalculatorOptions {
     // If calculated object scale is less than given value smoothing will be
     // disabled and landmarks will be returned as is.
     optional float min_allowed_object_scale = 5 [default = 1e-6];
+
+    // Disable value scaling based on object size and use `1.0` instead.
+    // If not disabled, value scale is calculated as inverse value of object
+    // size. Object size is calculated as maximum side of rectangular bounding
+    // box of the object in XY plane.
+    optional bool disable_value_scaling = 6 [default = false];
   }
 
   oneof filter_options {

mediapipe/calculators/util/rect_transformation_calculator.cc

@@ -130,8 +130,8 @@ absl::Status RectTransformationCalculator::Process(CalculatorContext* cc) {
     }
     cc->Outputs().Index(0).Add(output_rects.release(), cc->InputTimestamp());
   }
-  if (cc->Inputs().HasTag(kNormRectTag) &&
-      !cc->Inputs().Tag(kNormRectTag).IsEmpty()) {
+  if (HasTagValue(cc->Inputs(), kNormRectTag) &&
+      HasTagValue(cc->Inputs(), kImageSizeTag)) {
     auto rect = cc->Inputs().Tag(kNormRectTag).Get<NormalizedRect>();
     const auto& image_size =
         cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
@@ -139,8 +139,8 @@ absl::Status RectTransformationCalculator::Process(CalculatorContext* cc) {
     cc->Outputs().Index(0).AddPacket(
         MakePacket<NormalizedRect>(rect).At(cc->InputTimestamp()));
   }
-  if (cc->Inputs().HasTag(kNormRectsTag) &&
-      !cc->Inputs().Tag(kNormRectsTag).IsEmpty()) {
+  if (HasTagValue(cc->Inputs(), kNormRectsTag) &&
+      HasTagValue(cc->Inputs(), kImageSizeTag)) {
     auto rects =
         cc->Inputs().Tag(kNormRectsTag).Get<std::vector<NormalizedRect>>();
     const auto& image_size =

mediapipe/calculators/util/world_landmark_projection_calculator.cc

@@ -40,7 +40,7 @@ constexpr char kRectTag[] = "NORM_RECT";
 // Input:
 //   LANDMARKS: A LandmarkList representing world landmarks in the rectangle.
 //   NORM_RECT: An NormalizedRect representing a normalized rectangle in image
-//              coordinates.
+//              coordinates. (Optional)
 //
 // Output:
 //   LANDMARKS: A LandmarkList representing world landmarks projected (rotated
@@ -59,7 +59,9 @@ class WorldLandmarkProjectionCalculator : public CalculatorBase {
  public:
   static absl::Status GetContract(CalculatorContract* cc) {
     cc->Inputs().Tag(kLandmarksTag).Set<LandmarkList>();
-    cc->Inputs().Tag(kRectTag).Set<NormalizedRect>();
+    if (cc->Inputs().HasTag(kRectTag)) {
+      cc->Inputs().Tag(kRectTag).Set<NormalizedRect>();
+    }
     cc->Outputs().Tag(kLandmarksTag).Set<LandmarkList>();
 
     return absl::OkStatus();
@@ -74,13 +76,24 @@ class WorldLandmarkProjectionCalculator : public CalculatorBase {
   absl::Status Process(CalculatorContext* cc) override {
     // Check that landmarks and rect are not empty.
     if (cc->Inputs().Tag(kLandmarksTag).IsEmpty() ||
-        cc->Inputs().Tag(kRectTag).IsEmpty()) {
+        (cc->Inputs().HasTag(kRectTag) &&
+         cc->Inputs().Tag(kRectTag).IsEmpty())) {
       return absl::OkStatus();
     }
 
     const auto& in_landmarks =
         cc->Inputs().Tag(kLandmarksTag).Get<LandmarkList>();
-    const auto& in_rect = cc->Inputs().Tag(kRectTag).Get<NormalizedRect>();
+    std::function<void(const Landmark&, Landmark*)> rotate_fn;
+    if (cc->Inputs().HasTag(kRectTag)) {
+      const auto& in_rect = cc->Inputs().Tag(kRectTag).Get<NormalizedRect>();
+      const float cosa = std::cos(in_rect.rotation());
+      const float sina = std::sin(in_rect.rotation());
+      rotate_fn = [cosa, sina](const Landmark& in_landmark,
+                               Landmark* out_landmark) {
+        out_landmark->set_x(cosa * in_landmark.x() - sina * in_landmark.y());
+        out_landmark->set_y(sina * in_landmark.x() + cosa * in_landmark.y());
+      };
+    }
 
     auto out_landmarks = absl::make_unique<LandmarkList>();
     for (int i = 0; i < in_landmarks.landmark_size(); ++i) {
@@ -89,11 +102,9 @@ class WorldLandmarkProjectionCalculator : public CalculatorBase {
       Landmark* out_landmark = out_landmarks->add_landmark();
       *out_landmark = in_landmark;
 
-      const float angle = in_rect.rotation();
-      out_landmark->set_x(std::cos(angle) * in_landmark.x() -
-                          std::sin(angle) * in_landmark.y());
-      out_landmark->set_y(std::sin(angle) * in_landmark.x() +
-                          std::cos(angle) * in_landmark.y());
+      if (rotate_fn) {
+        rotate_fn(in_landmark, out_landmark);
+      }
     }
 
     cc->Outputs()

mediapipe/calculators/video/motion_analysis_calculator.cc

@@ -549,7 +549,7 @@ absl::Status MotionAnalysisCalculator::Process(CalculatorContext* cc) {
     timestamp_buffer_.push_back(timestamp);
     ++frame_idx_;
 
-    VLOG_EVERY_N(0, 100) << "Analyzed frame " << frame_idx_;
+    VLOG_EVERY_N(1, 100) << "Analyzed frame " << frame_idx_;
 
     // Buffer input frames only if visualization is requested.
     if (visualize_output_ || video_output_) {

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu/BUILD

@@ -37,7 +37,7 @@ android_binary(
     srcs = glob(["*.java"]),
     assets = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_cpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectionfullrangegpu/BUILD

@@ -0,0 +1,60 @@
+# Copyright 2019 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:private"])
+
+cc_binary(
+    name = "libmediapipe_jni.so",
+    linkshared = 1,
+    linkstatic = 1,
+    deps = [
+        "//mediapipe/graphs/face_detection:face_detection_full_range_mobile_gpu_deps",
+        "//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
+    ],
+)
+
+cc_library(
+    name = "mediapipe_jni_lib",
+    srcs = [":libmediapipe_jni.so"],
+    alwayslink = 1,
+)
+
+android_binary(
+    name = "facedetectionfullrangegpu",
+    srcs = glob(["*.java"]),
+    assets = [
+        "//mediapipe/graphs/face_detection:face_detection_full_range_mobile_gpu.binarypb",
+        "//mediapipe/modules/face_detection:face_detection_full_range_sparse.tflite",
+    ],
+    assets_dir = "",
+    manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
+    manifest_values = {
+        "applicationId": "com.google.mediapipe.apps.facedetectionfullrangegpu",
+        "appName": "Face Detection Full-range (GPU)",
+        "mainActivity": "com.google.mediapipe.apps.basic.MainActivity",
+        "cameraFacingFront": "False",
+        "binaryGraphName": "face_detection_full_range_mobile_gpu.binarypb",
+        "inputVideoStreamName": "input_video",
+        "outputVideoStreamName": "output_video",
+        "flipFramesVertically": "True",
+        "converterNumBuffers": "2",
+    },
+    multidex = "native",
+    deps = [
+        ":mediapipe_jni_lib",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
+    ],
+)

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD

@@ -37,7 +37,7 @@ android_binary(
     srcs = glob(["*.java"]),
     assets = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/faceeffect/BUILD

@@ -42,7 +42,7 @@ android_binary(
         "//mediapipe/graphs/face_effect/data:glasses.binarypb",
         "//mediapipe/graphs/face_effect/data:glasses.pngblob",
         "//mediapipe/graphs/face_effect:face_effect_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/BUILD

@@ -38,7 +38,7 @@ android_binary(
     assets = [
         "//mediapipe/graphs/face_mesh:face_mesh_mobile_gpu.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/holistictrackinggpu/BUILD

@@ -37,7 +37,7 @@ android_binary(
     srcs = glob(["*.java"]),
     assets = [
         "//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
         "//mediapipe/modules/hand_landmark:hand_landmark.tflite",
         "//mediapipe/modules/hand_landmark:handedness.txt",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu/BUILD

@@ -39,7 +39,7 @@ android_binary(
         "//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
         "//mediapipe/modules/iris_landmark:iris_landmark.tflite",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     assets_dir = "",
     manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",

mediapipe/examples/android/src/java/com/google/mediapipe/apps/selfiesegmentationgpu/BUILD

@@ -0,0 +1,60 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:private"])
+
+cc_binary(
+    name = "libmediapipe_jni.so",
+    linkshared = 1,
+    linkstatic = 1,
+    deps = [
+        "//mediapipe/graphs/selfie_segmentation:selfie_segmentation_gpu_deps",
+        "//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
+    ],
+)
+
+cc_library(
+    name = "mediapipe_jni_lib",
+    srcs = [":libmediapipe_jni.so"],
+    alwayslink = 1,
+)
+
+android_binary(
+    name = "selfiesegmentationgpu",
+    srcs = glob(["*.java"]),
+    assets = [
+        "//mediapipe/graphs/selfie_segmentation:selfie_segmentation_gpu.binarypb",
+        "//mediapipe/modules/selfie_segmentation:selfie_segmentation.tflite",
+    ],
+    assets_dir = "",
+    manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
+    manifest_values = {
+        "applicationId": "com.google.mediapipe.apps.selfiesegmentationgpu",
+        "appName": "Selfie Segmentation",
+        "mainActivity": "com.google.mediapipe.apps.basic.MainActivity",
+        "cameraFacingFront": "True",
+        "binaryGraphName": "selfie_segmentation_gpu.binarypb",
+        "inputVideoStreamName": "input_video",
+        "outputVideoStreamName": "output_video",
+        "flipFramesVertically": "True",
+        "converterNumBuffers": "2",
+    },
+    multidex = "native",
+    deps = [
+        ":mediapipe_jni_lib",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
+    ],
+)

mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator.cc

@@ -37,6 +37,13 @@ constexpr char kFirstCropRect[] = "FIRST_CROP_RECT";
 // (configured through option us_to_first_rect). If provided, a non-zero integer
 // will allow the animated zoom to be used when the first detections arrive.
 constexpr char kAnimateZoom[] = "ANIMATE_ZOOM";
+// Can be used to control the maximum zoom; note that it is re-evaluated only
+// upon change of input resolution. A value of 100 disables zooming and is the
+// smallest allowed value. A value of 200 allows zooming such that a pixel of
+// the input may cover up to four times its original area. Note that
+// max_zoom_value_deg from options is always respected; MAX_ZOOM_PCT can only be
+// used to limit zooming further.
+constexpr char kMaxZoomFactorPercent[] = "MAX_ZOOM_FACTOR_PCT";
 // Field-of-view (degrees) of the camera's x-axis (width).
 // TODO: Parameterize FOV based on camera specs.
 constexpr float kFieldOfView = 60;
@@ -75,11 +82,16 @@ class ContentZoomingCalculator : public CalculatorBase {
                               int frame_height);
   // Saves state to a state-cache, if provided.
   absl::Status SaveState(mediapipe::CalculatorContext* cc) const;
+  // Returns the factor for maximum zoom based on options and the
+  // kMaxZoomFactorPercent input (if present).
+  double GetMaxZoomFactor(mediapipe::CalculatorContext* cc) const;
   // Initializes the calculator for the given frame size, creating path solvers
   // and resetting history like last measured values.
-  absl::Status InitializeState(int frame_width, int frame_height);
+  absl::Status InitializeState(mediapipe::CalculatorContext* cc,
+                               int frame_width, int frame_height);
   // Adjusts state to work with an updated frame size.
-  absl::Status UpdateForResolutionChange(int frame_width, int frame_height);
+  absl::Status UpdateForResolutionChange(mediapipe::CalculatorContext* cc,
+                                         int frame_width, int frame_height);
   // Returns true if we are animating to the first rect.
   bool IsAnimatingToFirstRect(const Timestamp& timestamp) const;
   // Builds the output rectangle when animating to the first rect.
@@ -136,6 +148,9 @@ absl::Status ContentZoomingCalculator::GetContract(
     return mediapipe::UnknownErrorBuilder(MEDIAPIPE_LOC)
            << "Input VIDEO or VIDEO_SIZE must be provided.";
   }
+  if (cc->Inputs().HasTag(kMaxZoomFactorPercent)) {
+    cc->Inputs().Tag(kMaxZoomFactorPercent).Set<int>();
+  }
   if (cc->Inputs().HasTag(kSalientRegions)) {
     cc->Inputs().Tag(kSalientRegions).Set<DetectionSet>();
   }
@@ -330,7 +345,7 @@ absl::Status ContentZoomingCalculator::MaybeLoadState(
           ? cc->InputSidePackets().Tag(kStateCache).Get<StateCacheType*>()
           : nullptr;
   if (!state_cache || !state_cache->has_value()) {
-    return InitializeState(frame_width, frame_height);
+    return InitializeState(cc, frame_width, frame_height);
   }
 
   const ContentZoomingCalculatorState& state = state_cache->value();
@@ -350,7 +365,7 @@ absl::Status ContentZoomingCalculator::MaybeLoadState(
   last_measured_y_offset_ = state.last_measured_y_offset;
   MP_RETURN_IF_ERROR(UpdateAspectAndMax());
 
-  return UpdateForResolutionChange(frame_width, frame_height);
+  return UpdateForResolutionChange(cc, frame_width, frame_height);
 }
 
 absl::Status ContentZoomingCalculator::SaveState(
@@ -379,8 +394,20 @@ absl::Status ContentZoomingCalculator::SaveState(
   return absl::OkStatus();
 }
 
-absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
-                                                       int frame_height) {
+double ContentZoomingCalculator::GetMaxZoomFactor(
+    mediapipe::CalculatorContext* cc) const {
+  double max_zoom_value =
+      options_.max_zoom_value_deg() / static_cast<double>(kFieldOfView);
+  if (cc->Inputs().HasTag(kMaxZoomFactorPercent)) {
+    const double factor = std::max(
+        1.0, cc->Inputs().Tag(kMaxZoomFactorPercent).Get<int>() / 100.0);
+    max_zoom_value = std::max(max_zoom_value, 1.0 / factor);
+  }
+  return max_zoom_value;
+}
+
+absl::Status ContentZoomingCalculator::InitializeState(
+    mediapipe::CalculatorContext* cc, int frame_width, int frame_height) {
   frame_width_ = frame_width;
   frame_height_ = frame_height;
   path_solver_pan_ = std::make_unique<KinematicPathSolver>(
@@ -390,8 +417,7 @@ absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
       options_.kinematic_options_tilt(), 0, frame_height_,
       static_cast<float>(frame_height_) / kFieldOfView);
   MP_RETURN_IF_ERROR(UpdateAspectAndMax());
-  int min_zoom_size = frame_height_ * (options_.max_zoom_value_deg() /
-                                       static_cast<double>(kFieldOfView));
+  int min_zoom_size = frame_height_ * GetMaxZoomFactor(cc);
   path_solver_zoom_ = std::make_unique<KinematicPathSolver>(
       options_.kinematic_options_zoom(), min_zoom_size,
       max_frame_value_ * frame_height_,
@@ -405,7 +431,7 @@ absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
 }
 
 absl::Status ContentZoomingCalculator::UpdateForResolutionChange(
-    int frame_width, int frame_height) {
+    mediapipe::CalculatorContext* cc, int frame_width, int frame_height) {
   // Update state for change in input resolution.
   if (frame_width_ != frame_width || frame_height_ != frame_height) {
     double width_scale = frame_width / static_cast<double>(frame_width_);
@@ -419,8 +445,7 @@ absl::Status ContentZoomingCalculator::UpdateForResolutionChange(
     MP_RETURN_IF_ERROR(path_solver_pan_->UpdateMinMaxLocation(0, frame_width_));
     MP_RETURN_IF_ERROR(
         path_solver_tilt_->UpdateMinMaxLocation(0, frame_height_));
-    int min_zoom_size = frame_height_ * (options_.max_zoom_value_deg() /
-                                         static_cast<double>(kFieldOfView));
+    int min_zoom_size = frame_height_ * GetMaxZoomFactor(cc);
     MP_RETURN_IF_ERROR(path_solver_zoom_->UpdateMinMaxLocation(
         min_zoom_size, max_frame_value_ * frame_height_));
     MP_RETURN_IF_ERROR(path_solver_zoom_->UpdatePixelsPerDegree(
@@ -493,7 +518,8 @@ absl::Status ContentZoomingCalculator::Process(
     MP_RETURN_IF_ERROR(MaybeLoadState(cc, frame_width, frame_height));
     initialized_ = !options_.is_stateless();
   } else {
-    MP_RETURN_IF_ERROR(UpdateForResolutionChange(frame_width, frame_height));
+    MP_RETURN_IF_ERROR(
+        UpdateForResolutionChange(cc, frame_width, frame_height));
   }
 
   bool only_required_found = false;
@@ -634,14 +660,18 @@ absl::Status ContentZoomingCalculator::Process(
   // Prevent box from extending beyond the image after camera smoothing.
   if (path_offset_y - ceil(path_height / 2.0) < 0) {
     path_offset_y = ceil(path_height / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_tilt_->SetState(path_offset_y));
   } else if (path_offset_y + ceil(path_height / 2.0) > frame_height_) {
     path_offset_y = frame_height_ - ceil(path_height / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_tilt_->SetState(path_offset_y));
   }
 
   if (path_offset_x - ceil(path_width / 2.0) < 0) {
     path_offset_x = ceil(path_width / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_pan_->SetState(path_offset_x));
   } else if (path_offset_x + ceil(path_width / 2.0) > frame_width_) {
     path_offset_x = frame_width_ - ceil(path_width / 2.0);
+    MP_RETURN_IF_ERROR(path_solver_pan_->SetState(path_offset_x));
   }
 
   // Convert to top/bottom borders to remove.

mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator_test.cc

@@ -150,6 +150,29 @@ const char kConfigE[] = R"(
     }
     )";
 
+const char kConfigF[] = R"(
+    calculator: "ContentZoomingCalculator"
+    input_stream: "VIDEO_SIZE:size"
+    input_stream: "DETECTIONS:detections"
+    input_stream: "MAX_ZOOM_FACTOR_PCT:max_zoom_factor_pct"
+    output_stream: "CROP_RECT:rect"
+    output_stream: "FIRST_CROP_RECT:first_rect"
+    options: {
+      [mediapipe.autoflip.ContentZoomingCalculatorOptions.ext]: {
+        max_zoom_value_deg: 0
+        kinematic_options_zoom {
+          min_motion_to_reframe: 1.2
+        }
+        kinematic_options_tilt {
+          min_motion_to_reframe: 1.2
+        }
+        kinematic_options_pan {
+          min_motion_to_reframe: 1.2
+        }
+      }
+    }
+    )";
+
 void CheckBorder(const StaticFeatures& static_features, int width, int height,
                  int top_border, int bottom_border) {
   ASSERT_EQ(2, static_features.border().size());
@@ -170,6 +193,7 @@ void CheckBorder(const StaticFeatures& static_features, int width, int height,
 
 struct AddDetectionFlags {
   std::optional<bool> animated_zoom;
+  std::optional<int> max_zoom_factor_percent;
 };
 
 void AddDetectionFrameSize(const cv::Rect_<float>& position, const int64 time,
@@ -211,6 +235,14 @@ void AddDetectionFrameSize(const cv::Rect_<float>& position, const int64 time,
             mediapipe::MakePacket<bool>(flags.animated_zoom.value())
                 .At(Timestamp(time)));
   }
+
+  if (flags.max_zoom_factor_percent.has_value()) {
+    runner->MutableInputs()
+        ->Tag("MAX_ZOOM_FACTOR_PCT")
+        .packets.push_back(
+            mediapipe::MakePacket<int>(flags.max_zoom_factor_percent.value())
+                .At(Timestamp(time)));
+  }
 }
 
 void AddDetection(const cv::Rect_<float>& position, const int64 time,
@@ -259,6 +291,7 @@ TEST(ContentZoomingCalculatorTest, ZoomTest) {
   CheckBorder(static_features, 1000, 1000, 495, 395);
 }
 
+#if 0
 TEST(ContentZoomingCalculatorTest, ZoomTestFullPTZ) {
   auto runner = ::absl::make_unique<CalculatorRunner>(
       ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD));
@@ -694,8 +727,8 @@ TEST(ContentZoomingCalculatorTest, ResolutionChangeZoomingWithCache) {
     auto runner = ::absl::make_unique<CalculatorRunner>(config);
     runner->MutableSidePackets()->Tag("STATE_CACHE") = MakePacket<
         mediapipe::autoflip::ContentZoomingCalculatorStateCacheType*>(&cache);
-    AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1000, 1000,
-                          runner.get());
+    AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1000,
+                          1000, runner.get());
     AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 2000000, 500, 500,
                           runner.get());
     MP_ASSERT_OK(runner->Run());
@@ -719,6 +752,36 @@ TEST(ContentZoomingCalculatorTest, MaxZoomValue) {
   CheckCropRect(500, 500, 916, 916, 0,
                 runner->Outputs().Tag("CROP_RECT").packets);
 }
+#endif
+
+TEST(ContentZoomingCalculatorTest, MaxZoomValueOverride) {
+  auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigF);
+  auto* options = config.mutable_options()->MutableExtension(
+      ContentZoomingCalculatorOptions::ext);
+  options->set_max_zoom_value_deg(30);
+  auto runner = ::absl::make_unique<CalculatorRunner>(config);
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 0, 640, 480,
+                        runner.get(), {.max_zoom_factor_percent = 133});
+  // Change resolution and allow more zoom, and give time to use the new limit
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1280, 720,
+                        runner.get(), {.max_zoom_factor_percent = 166});
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 2000000, 1280, 720,
+                        runner.get(), {.max_zoom_factor_percent = 166});
+  // Switch back to a smaller resolution with a more limited zoom
+  AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 3000000, 640, 480,
+                        runner.get(), {.max_zoom_factor_percent = 133});
+  MP_ASSERT_OK(runner->Run());
+  // Max. 133% zoomed in means min. (100/133) ~ 75% of height left: ~360
+  // Max. 166% zoomed in means min. (100/166) ~ 60% of height left: ~430
+  CheckCropRect(320, 240, 480, 360, 0,
+                runner->Outputs().Tag("CROP_RECT").packets);
+  CheckCropRect(640, 360, 769, 433, 2,
+                runner->Outputs().Tag("CROP_RECT").packets);
+  CheckCropRect(320, 240, 480, 360, 3,
+                runner->Outputs().Tag("CROP_RECT").packets);
+}
+
+#if 0
 TEST(ContentZoomingCalculatorTest, MaxZoomOutValue) {
   auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD);
   auto* options = config.mutable_options()->MutableExtension(
@@ -906,6 +969,7 @@ TEST(ContentZoomingCalculatorTest, ProvidesConstantFirstRect) {
     EXPECT_EQ(first_rect.height(), rect.height());
   }
 }
+#endif
 
 }  // namespace
 }  // namespace autoflip

mediapipe/examples/desktop/autoflip/calculators/face_box_adjuster_calculator.proto

@@ -49,8 +49,29 @@ message FaceBoxAdjusterCalculatorOptions {
   optional float ipd_face_box_height_ratio = 7 [default = 0.3131];
 
   // The max look up angle before considering the eye distance unstable.
-  optional float max_head_tilt_angle_deg = 8 [default = 12.0];
+  optional float max_head_tilt_angle_deg = 8 [default = 5.0];
+  // The min look up angle (i.e. looking down) before considering the eye
+  // distance unstable.
+  optional float min_head_tilt_angle_deg = 10 [default = -18.0];
+  // The max look right angle before considering the eye distance unstable.
+  optional float max_head_pan_angle_deg = 11 [default = 25.0];
+  // The min look right angle (i.e. looking left) before considering the eye
+  // distance unstable.
+  optional float min_head_pan_angle_deg = 12 [default = -25.0];
+
+  // Update rate for motion history, valid values [0.0, 1.0].
+  optional float motion_history_alpha = 13 [default = 0.5];
+
+  // Max value of head motion (max of current or history) to be considered still
+  // stable.
+  optional float head_motion_threshold = 14 [default = 10.0];
+
   // The max amount of time to use an old eye distance when the face look angle
   // is unstable.
-  optional int32 max_facesize_history_us = 9 [default = 8000000];
+  optional int32 max_facesize_history_us = 9 [default = 300000000];
+
+  // Scale factor of face width to shift based on pan look angle.
+  optional float pan_position_shift_scale = 15 [default = 0.5];
+  // Scale factor of face height to shift based on tilt look angle.
+  optional float tilt_position_shift_scale = 16 [default = 0.5];
 }

mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver.cc

@@ -154,10 +154,18 @@ absl::Status KinematicPathSolver::AddObservation(int position,
 
   // Time and position updates.
   double delta_t = (time_us - current_time_) / 1000000.0;
+  // Time since last state/prediction update, smoothed by
+  // mean_period_update_rate.
+  if (mean_delta_t_ < 0) {
+    mean_delta_t_ = delta_t;
+  } else {
+    mean_delta_t_ = mean_delta_t_ * (1 - options_.mean_period_update_rate()) +
+                    delta_t * options_.mean_period_update_rate();
+  }
 
   // Observed velocity and then weighted update of this velocity.
   double observed_velocity = delta_degs / delta_t;
-  double update_rate = std::min(delta_t / options_.update_rate_seconds(),
+  double update_rate = std::min(mean_delta_t_ / options_.update_rate_seconds(),
                                 options_.max_update_rate());
   double updated_velocity = current_velocity_deg_per_s_ * (1 - update_rate) +
                             observed_velocity * update_rate;
@@ -174,16 +182,6 @@ absl::Status KinematicPathSolver::UpdatePrediction(const int64 time_us) {
   RET_CHECK(current_time_ < time_us)
       << "Prediction time added before a prior observation or prediction.";
 
-  // Time since last state/prediction update, smoothed by
-  // mean_period_update_rate.
-  double delta_t = (time_us - current_time_) / 1000000.0;
-  if (mean_delta_t_ < 0) {
-    mean_delta_t_ = delta_t;
-  } else {
-    mean_delta_t_ = mean_delta_t_ * (1 - options_.mean_period_update_rate()) +
-                    delta_t * options_.mean_period_update_rate();
-  }
-
   // Position update limited by min/max.
   double update_position_px =
       current_position_px_ +
@@ -211,6 +209,12 @@ absl::Status KinematicPathSolver::GetState(int* position) {
   return absl::OkStatus();
 }
 
+absl::Status KinematicPathSolver::SetState(const int position) {
+  RET_CHECK(initialized_) << "SetState called before first observation added.";
+  current_position_px_ = position;
+  return absl::OkStatus();
+}
+
 absl::Status KinematicPathSolver::GetTargetPosition(int* target_position) {
   RET_CHECK(initialized_)
       << "GetTargetPosition called before first observation added.";

mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver.h

@@ -48,6 +48,8 @@ class KinematicPathSolver {
   absl::Status UpdatePrediction(const int64 time_us);
   // Get the state at a time.
   absl::Status GetState(int* position);
+  // Overwrite the current state value.
+  absl::Status SetState(const int position);
   // Update PixelPerDegree value.
   absl::Status UpdatePixelsPerDegree(const float pixels_per_degree);
   // Provide the current target position of the reframe action.

mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver_test.cc

@@ -337,6 +337,61 @@ TEST(KinematicPathSolverTest, PassDegPerPxChange) {
   EXPECT_EQ(state, 516);
 }
 
+TEST(KinematicPathSolverTest, NoTimestampSmoothing) {
+  KinematicOptions options;
+  options.set_min_motion_to_reframe(1.0);
+  options.set_update_rate(1.0);
+  options.set_max_velocity(6);
+  options.set_mean_period_update_rate(1.0);
+  KinematicPathSolver solver(options, 0, 1000, 1000.0 / kWidthFieldOfView);
+  int state;
+  MP_ASSERT_OK(solver.AddObservation(500, 0));
+  MP_ASSERT_OK(solver.AddObservation(1000, 1000000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 600);
+  MP_ASSERT_OK(solver.AddObservation(1000, 2200000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 720);
+}
+
+TEST(KinematicPathSolverTest, TimestampSmoothing) {
+  KinematicOptions options;
+  options.set_min_motion_to_reframe(1.0);
+  options.set_update_rate(1.0);
+  options.set_max_velocity(6);
+  options.set_mean_period_update_rate(0.05);
+  KinematicPathSolver solver(options, 0, 1000, 1000.0 / kWidthFieldOfView);
+  int state;
+  MP_ASSERT_OK(solver.AddObservation(500, 0));
+  MP_ASSERT_OK(solver.AddObservation(1000, 1000000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 600);
+  MP_ASSERT_OK(solver.AddObservation(1000, 2200000));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 701);
+}
+
+TEST(KinematicPathSolverTest, PassSetPosition) {
+  KinematicOptions options;
+  // Set min motion to 2deg
+  options.set_min_motion_to_reframe(1.0);
+  options.set_update_rate_seconds(.0000001);
+  options.set_max_update_rate(1.0);
+  options.set_max_velocity(18);
+  // Set degrees / pixel to 8.3
+  KinematicPathSolver solver(options, 0, 500, 500.0 / kWidthFieldOfView);
+  int state;
+  MP_ASSERT_OK(solver.AddObservation(400, kMicroSecInSec * 0));
+  // Move target by 10px / 8.3 = 1.2deg
+  MP_ASSERT_OK(solver.AddObservation(410, kMicroSecInSec * 1));
+  MP_ASSERT_OK(solver.GetState(&state));
+  // Expect cam to move.
+  EXPECT_EQ(state, 410);
+  MP_ASSERT_OK(solver.SetState(400));
+  MP_ASSERT_OK(solver.GetState(&state));
+  EXPECT_EQ(state, 400);
+}
+
 }  // namespace
 }  // namespace autoflip
 }  // namespace mediapipe

mediapipe/examples/desktop/autoflip/subgraph/face_detection_subgraph.pbtxt

@@ -14,8 +14,8 @@ node: {
   output_stream: "LETTERBOX_PADDING:letterbox_padding"
   options: {
     [mediapipe.ImageTransformationCalculatorOptions.ext] {
-      output_width: 256
-      output_height: 256
+      output_width: 192
+      output_height: 192
       scale_mode: FIT
     }
   }
@@ -38,7 +38,7 @@ node {
   output_stream: "TENSORS:detection_tensors"
   options: {
     [mediapipe.TfLiteInferenceCalculatorOptions.ext] {
-      model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
+      model_path: "mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite"
     }
   }
 }
@@ -50,19 +50,17 @@ node {
   output_side_packet: "anchors"
   options: {
     [mediapipe.SsdAnchorsCalculatorOptions.ext] {
-      num_layers: 4
-      min_scale: 0.15625
+      num_layers: 1
+      min_scale: 0.1484375
       max_scale: 0.75
-      input_size_height: 256
-      input_size_width: 256
+      input_size_height: 192
+      input_size_width: 192
       anchor_offset_x: 0.5
       anchor_offset_y: 0.5
-      strides: 16
-      strides: 32
-      strides: 32
-      strides: 32
+      strides: 4
       aspect_ratios: 1.0
       fixed_anchor_size: true
+      interpolated_scale_aspect_ratio: 0.0
     }
   }
 }
@@ -78,7 +76,7 @@ node {
   options: {
     [mediapipe.TfLiteTensorsToDetectionsCalculatorOptions.ext] {
       num_classes: 1
-      num_boxes: 896
+      num_boxes: 2304
       num_coords: 16
       box_coord_offset: 0
       keypoint_coord_offset: 4
@@ -87,11 +85,11 @@ node {
       sigmoid_score: true
       score_clipping_thresh: 100.0
       reverse_output_order: true
-      x_scale: 256.0
-      y_scale: 256.0
-      h_scale: 256.0
-      w_scale: 256.0
-      min_score_thresh: 0.65
+      x_scale: 192.0
+      y_scale: 192.0
+      h_scale: 192.0
+      w_scale: 192.0
+      min_score_thresh: 0.6
     }
   }
 }

mediapipe/examples/desktop/face_detection/BUILD

@@ -16,6 +16,14 @@ licenses(["notice"])
 
 package(default_visibility = ["//mediapipe/examples:__subpackages__"])
 
+cc_binary(
+    name = "face_detection_full_range_cpu",
+    deps = [
+        "//mediapipe/examples/desktop:demo_run_graph_main",
+        "//mediapipe/graphs/face_detection:face_detection_full_range_desktop_live_deps",
+    ],
+)
+
 cc_binary(
     name = "face_detection_cpu",
     deps = [

mediapipe/examples/desktop/selfie_segmentation/BUILD

@@ -0,0 +1,34 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//mediapipe/examples:__subpackages__"])
+
+cc_binary(
+    name = "selfie_segmentation_cpu",
+    deps = [
+        "//mediapipe/examples/desktop:demo_run_graph_main",
+        "//mediapipe/graphs/selfie_segmentation:selfie_segmentation_cpu_deps",
+    ],
+)
+
+# Linux only
+cc_binary(
+    name = "selfie_segmentation_gpu",
+    deps = [
+        "//mediapipe/examples/desktop:demo_run_graph_main_gpu",
+        "//mediapipe/graphs/selfie_segmentation:selfie_segmentation_gpu_deps",
+    ],
+)

mediapipe/examples/ios/facedetectioncpu/BUILD

@@ -55,7 +55,7 @@ objc_library(
     name = "FaceDetectionCpuAppLibrary",
     data = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_cpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     deps = [
         "//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",

mediapipe/examples/ios/facedetectiongpu/BUILD

@@ -55,7 +55,7 @@ objc_library(
     name = "FaceDetectionGpuAppLibrary",
     data = [
         "//mediapipe/graphs/face_detection:face_detection_mobile_gpu.binarypb",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
     ],
     deps = [
         "//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",

mediapipe/examples/ios/faceeffect/BUILD

@@ -66,7 +66,7 @@ objc_library(
         "//mediapipe/graphs/face_effect/data:facepaint.pngblob",
         "//mediapipe/graphs/face_effect/data:glasses.binarypb",
         "//mediapipe/graphs/face_effect/data:glasses.pngblob",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
@@ -109,7 +109,7 @@ objc_library(
         "//mediapipe/graphs/face_effect/data:facepaint.pngblob",
         "//mediapipe/graphs/face_effect/data:glasses.binarypb",
         "//mediapipe/graphs/face_effect/data:glasses.pngblob",
-        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
         "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
         "//mediapipe/modules/face_landmark:face_landmark.tflite",
     ],