Project import generated by Copybara.

GitOrigin-RevId: 1e221238b0bc717115c8152ad3092da3309a63a1

.github/ISSUE_TEMPLATE/00-build-installation-issue.md

@@ -0,0 +1,21 @@
+<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
+
+**System information** (Please provide as much relevant information as possible)
+- OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
+- Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
+- Installed using virtualenv? pip? Conda? (if python):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version:
+- XCode and Tulsi versions (if iOS):
+- Android SDK and NDK versions (if android):
+- Android [AAR](https://google.github.io/mediapipe/getting_started/android_archive_library.html) ( if android):
+- OpenCV version (if running on desktop):
+
+**Describe the problem**:
+
+
+**[Provide the exact sequence of commands / steps that you executed before running into the problem](https://google.github.io/mediapipe/getting_started/getting_started.html):**
+
+**Complete Logs:**
+Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:

.github/ISSUE_TEMPLATE/10-solution-issue.md

@@ -0,0 +1,20 @@
+<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
+
+**System information** (Please provide as much relevant information as possible)
+- Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
+- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
+- [MediaPipe version](https://github.com/google/mediapipe/releases):
+- Bazel version:
+- Solution (e.g. FaceMesh, Pose, Holistic):
+- Programming Language and version ( e.g. C++, Python, Java):
+
+**Describe the expected behavior:**
+
+**Standalone code you may have used to try to get what you need :**
+
+If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
+
+**Other info / Complete Logs :**
+Include any logs or source code that would be helpful to
+diagnose the problem. If including tracebacks, please include the full
+traceback. Large logs and files should be attached:

.github/ISSUE_TEMPLATE/20-documentation-issue.md

@@ -0,0 +1,45 @@
+Thank you for submitting a MediaPipe documentation issue.
+The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
+## URL(s) with the issue:
+
+Please provide a link to the documentation entry, for example: https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
+
+## Description of issue (what needs changing):
+
+Kinds of documentation problems:
+
+### Clear description
+
+For example, why should someone use this method? How is it useful?
+
+### Correct links
+
+Is the link to the source code correct?
+
+### Parameters defined
+Are all parameters defined and formatted correctly?
+
+### Returns defined
+
+Are return values defined?
+
+### Raises listed and defined
+
+Are the errors defined? For example,
+
+### Usage example
+
+Is there a usage example?
+
+See the API guide:
+on how to write testable usage examples.
+
+### Request visuals, if applicable
+
+Are there currently visuals? If not, will it clarify the content?
+
+### Submit a pull request?
+
+Are you planning to also submit a pull request to fix the issue? See the docs
+https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md
+

.github/bot_config.yml

@@ -0,0 +1,18 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+# A list of assignees
+assignees:
+   - sgowroji

.github/stale.yml

@@ -0,0 +1,34 @@
+# Copyright 2021 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+# Number of days of inactivity before an Issue or Pull Request becomes stale
+daysUntilStale: 7
+# Number of days of inactivity before a stale Issue or Pull Request is closed
+daysUntilClose: 7
+# Only issues or pull requests with all of these labels are checked if stale. Defaults to `[]` (disabled)
+onlyLabels:
+ - stat:awaiting response
+# Comment to post when marking as stale. Set to `false` to disable
+markComment: >
+  This issue has been automatically marked as stale because it has not had
+  recent activity. It will be closed if no further activity occurs. Thank you.
+# Comment to post when removing the stale label. Set to `false` to disable
+unmarkComment: false
+closeComment: >
+  Closing as stale. Please reopen if you'd like to work on this further.

Dockerfile

@@ -23,6 +23,7 @@ ENV DEBIAN_FRONTEND=noninteractive
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
         build-essential \
+        gcc-8 g++-8 \
         ca-certificates \
         curl \
         ffmpeg \
@@ -44,6 +45,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
+RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 100 --slave /usr/bin/g++ g++ /usr/bin/g++-8
 RUN pip3 install --upgrade setuptools
 RUN pip3 install wheel
 RUN pip3 install future

MANIFEST.in

@@ -14,3 +14,6 @@ exclude mediapipe/modules/objectron/object_detection_3d_sneakers.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_chair.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_camera.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_cup.tflite
+exclude mediapipe/modules/objectron/object_detection_ssd_mobilenetv2_oidv4_fp16.tflite
+exclude mediapipe/modules/pose_landmark/pose_landmark_lite.tflite
+exclude mediapipe/modules/pose_landmark/pose_landmark_heavy.tflite

README.md

@@ -40,11 +40,12 @@ Hair Segmentation
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                        | ✅                                                         | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |

WORKSPACE

@@ -71,8 +71,8 @@ http_archive(
 # Google Benchmark library.
 http_archive(
     name = "com_google_benchmark",
-    urls = ["https://github.com/google/benchmark/archive/master.zip"],
-    strip_prefix = "benchmark-master",
+    urls = ["https://github.com/google/benchmark/archive/main.zip"],
+    strip_prefix = "benchmark-main",
     build_file = "@//third_party:benchmark.BUILD",
 )
 
@@ -337,6 +337,8 @@ maven_install(
         "androidx.test.espresso:espresso-core:3.1.1",
         "com.github.bumptech.glide:glide:4.11.0",
         "com.google.android.material:material:aar:1.0.0-rc01",
+        "com.google.auto.value:auto-value:1.6.4",
+        "com.google.auto.value:auto-value-annotations:1.6.4",
         "com.google.code.findbugs:jsr305:3.0.2",
         "com.google.flogger:flogger-system-backend:0.3.1",
         "com.google.flogger:flogger:0.3.1",
@@ -367,9 +369,9 @@ http_archive(
 )
 
 # Tensorflow repo should always go after the other external dependencies.
-# 2021-03-25
-_TENSORFLOW_GIT_COMMIT = "c67f68021824410ebe9f18513b8856ac1c6d4887"
-_TENSORFLOW_SHA256= "fd07d0b39422dc435e268c5e53b2646a8b4b1e3151b87837b43f86068faae87f"
+# 2021-05-27
+_TENSORFLOW_GIT_COMMIT = "d6bfcdb0926173dbb7aa02ceba5aae6250b8aaa6"
+_TENSORFLOW_SHA256 = "ec40e1462239d8783d02f76a43412c8f80bac71ea20e41e1b7729b990aad6923"
 http_archive(
     name = "org_tensorflow",
     urls = [

build_desktop_examples.sh

@@ -17,15 +17,15 @@
 # Script to build/run all MediaPipe desktop example apps (with webcam input).
 #
 # To build and run all apps and store them in out_dir:
-#   $ ./build_ios_examples.sh -d out_dir
+#   $ ./build_desktop_examples.sh -d out_dir
 #   Omitting -d and the associated directory saves all generated apps in the
 #   current directory.
 # To build all apps and store them in out_dir:
-#   $ ./build_ios_examples.sh -d out_dir -b
+#   $ ./build_desktop_examples.sh -d out_dir -b
 #   Omitting -d and the associated directory saves all generated apps in the
 #   current directory.
 # To run all apps already stored in out_dir:
-#   $ ./build_ios_examples.sh -d out_dir -r
+#   $ ./build_desktop_examples.sh -d out_dir -r
 #   Omitting -d and the associated directory assumes all apps are in the current
 #   directory.
 
@@ -97,6 +97,7 @@ for app in ${apps}; do
       if [[ ${target_name} == "holistic_tracking" ||
             ${target_name} == "iris_tracking" ||
             ${target_name} == "pose_tracking" ||
+            ${target_name} == "selfie_segmentation" ||
             ${target_name} == "upper_body_pose_tracking" ]]; then
         graph_suffix="cpu"
       else

docs/framework_concepts/calculators.md

@@ -187,7 +187,7 @@ node {
 ```
 
 In the calculator implementation, inputs and outputs are also identified by tag
-name and index number. In the function below input are output are identified:
+name and index number. In the function below input and output are identified:
 
 *   By index number: The combined input stream is identified simply by index
     `0`.
@@ -248,12 +248,58 @@ absl::Status MyCalculator::Process() {
 }
 ```
 
+## Calculator options
+
+Calculators accept processing parameters through (1) input stream packets (2)
+input side packets, and (3) calculator options. Calculator options, if
+specified, appear as literal values in the `node_options` field of the
+`CalculatorGraphConfiguration.Node` message.
+
+```
+  node {
+    calculator: "TfLiteInferenceCalculator"
+    input_stream: "TENSORS:main_model_input"
+    output_stream: "TENSORS:main_model_output"
+    node_options: {
+      [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+        model_path: "mediapipe/models/active_speaker_detection/audio_visual_model.tflite"
+      }
+    }
+  }
+```
+
+The `node_options` field accepts the proto3 syntax.  Alternatively, calculator
+options can be specified in the `options` field using proto2 syntax.
+
+```
+  node: {
+    calculator: "IntervalFilterCalculator"
+    node_options: {
+      [type.googleapis.com/mediapipe.IntervalFilterCalculatorOptions] {
+        intervals {
+          start_us: 20000
+          end_us: 40000
+        }
+      }
+    }
+  }
+```
+
+Not all calculators accept calcuator options. In order to accept options, a
+calculator will normally define a new protobuf message type to represent its
+options, such as `IntervalFilterCalculatorOptions`. The calculator will then
+read that protobuf message in its `CalculatorBase::Open` method, and possibly
+also in the `CalculatorBase::GetContract` function or its
+`CalculatorBase::Process` method. Normally, the new protobuf message type will
+be defined as a protobuf schema using a ".proto" file and a
+`mediapipe_proto_library()` build rule.
+
 ## Example calculator
 
 This section discusses the implementation of `PacketClonerCalculator`, which
 does a relatively simple job, and is used in many calculator graphs.
-`PacketClonerCalculator` simply produces a copy of its most recent input
-packets on demand.
+`PacketClonerCalculator` simply produces a copy of its most recent input packets
+on demand.
 
 `PacketClonerCalculator` is useful when the timestamps of arriving data packets
 are not aligned perfectly. Suppose we have a room with a microphone, light
@@ -279,8 +325,8 @@ input streams:
     imageframe of video data representing video collected from camera in the
     room with timestamp.
 
-Below is the implementation of the `PacketClonerCalculator`.  You can see
-the `GetContract()`, `Open()`, and `Process()` methods as well as the instance
+Below is the implementation of the `PacketClonerCalculator`. You can see the
+`GetContract()`, `Open()`, and `Process()` methods as well as the instance
 variable `current_` which holds the most recent input packets.
 
 ```c++
@@ -355,7 +401,6 @@ class PacketClonerCalculator : public CalculatorBase {
               current_[i].At(cc->InputTimestamp()));
           // Add a packet to output stream of index i a packet from inputstream i
           // with timestamp common to all present inputs
-          //
         } else {
           cc->Outputs().Index(i).SetNextTimestampBound(
               cc->InputTimestamp().NextAllowedInStream());
@@ -382,7 +427,7 @@ defined your calculator class, register it with a macro invocation
 REGISTER_CALCULATOR(calculator_class_name).
 
 Below is a trivial MediaPipe graph that has 3 input streams, 1 node
-(PacketClonerCalculator) and 3 output streams.
+(PacketClonerCalculator) and 2 output streams.
 
 ```proto
 input_stream: "room_mic_signal"
@@ -402,6 +447,6 @@ node {
 The diagram below shows how the `PacketClonerCalculator` defines its output
 packets (bottom) based on its series of input packets (top).
 
-| ![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png) |
-| :---------------------------------------------------------------------------: |
-| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |
+![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png)  |
+:--------------------------------------------------------------------------: |
+*Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |

docs/framework_concepts/framework_concepts.md

@@ -110,3 +110,12 @@ Other policies are also available, implemented using a separate kind of
 component known as an InputStreamHandler.
 
 See [Synchronization](synchronization.md) for more details.
+
+### Real-time streams
+
+MediaPipe calculator graphs are often used to process streams of video or audio
+frames for interactive applications. Normally, each Calculator runs as soon as
+all of its input packets for a given timestamp become available. Calculators
+used in real-time graphs need to define output timestamp bounds based on input
+timestamp bounds in order to allow downstream calculators to be scheduled
+promptly. See [Real-time Streams](realtime_streams.md) for details.

docs/framework_concepts/graphs.md

@@ -83,12 +83,12 @@ Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`.
     output_stream: "out3"
 
     node {
-        calculator: "PassThroughculator"
+        calculator: "PassThroughCalculator"
         input_stream: "out1"
         output_stream: "out2"
     }
     node {
-        calculator: "PassThroughculator"
+        calculator: "PassThroughCalculator"
         input_stream: "out2"
         output_stream: "out3"
     }

docs/framework_concepts/realtime_streams.md

@@ -0,0 +1,186 @@
+---
+layout: default
+title: Real-time Streams
+parent: Framework Concepts
+nav_order: 6
+---
+
+# Real-time Streams
+{: .no_toc }
+
+1. TOC
+{:toc}
+---
+
+## Real-time timestamps
+
+MediaPipe calculator graphs are often used to process streams of video or audio
+frames for interactive applications. The MediaPipe framework requires only that
+successive packets be assigned monotonically increasing timestamps. By
+convention, real-time calculators and graphs use the recording time or the
+presentation time of each frame as its timestamp, with each timestamp indicating
+the microseconds since `Jan/1/1970:00:00:00`. This allows packets from various
+sources to be processed in a globally consistent sequence.
+
+## Real-time scheduling
+
+Normally, each Calculator runs as soon as all of its input packets for a given
+timestamp become available. Normally, this happens when the calculator has
+finished processing the previous frame, and each of the calculators producing
+its inputs have finished processing the current frame. The MediaPipe scheduler
+invokes each calculator as soon as these conditions are met. See
+[Synchronization](synchronization.md) for more details.
+
+## Timestamp bounds
+
+When a calculator does not produce any output packets for a given timestamp, it
+can instead output a "timestamp bound" indicating that no packet will be
+produced for that timestamp. This indication is necessary to allow downstream
+calculators to run at that timestamp, even though no packet has arrived for
+certain streams for that timestamp. This is especially important for real-time
+graphs in interactive applications, where it is crucial that each calculator
+begin processing as soon as possible.
+
+Consider a graph like the following:
+
+```
+node {
+   calculator: "A"
+   input_stream: "alpha_in"
+   output_stream: "alpha"
+}
+node {
+   calculator: "B"
+   input_stream: "alpha"
+   input_stream: "foo"
+   output_stream: "beta"
+}
+```
+
+Suppose: at timestamp `T`, node `A` doesn't send a packet in its output stream
+`alpha`. Node `B` gets a packet in `foo` at timestamp `T` and is waiting for a
+packet in `alpha` at timestamp `T`. If `A` doesn't send `B` a timestamp bound
+update for `alpha`, `B` will keep waiting for a packet to arrive in `alpha`.
+Meanwhile, the packet queue of `foo` will accumulate packets at `T`, `T+1` and
+so on.
+
+To output a packet on a stream, a calculator uses the API functions
+`CalculatorContext::Outputs` and `OutputStream::Add`. To instead output a
+timestamp bound on a stream, a calculator can use the API functions
+`CalculatorContext::Outputs` and `CalculatorContext::SetNextTimestampBound`. The
+specified bound is the lowest allowable timestamp for the next packet on the
+specified output stream. When no packet is output, a calculator will typically
+do something like:
+
+```
+cc->Outputs().Tag("output_frame").SetNextTimestampBound(
+  cc->InputTimestamp().NextAllowedInStream());
+```
+
+The function `Timestamp::NextAllowedInStream` returns the successive timestamp.
+For example, `Timestamp(1).NextAllowedInStream() == Timestamp(2)`.
+
+## Propagating timestamp bounds
+
+Calculators that will be used in real-time graphs need to define output
+timestamp bounds based on input timestamp bounds in order to allow downstream
+calculators to be scheduled promptly. A common pattern is for calculators to
+output packets with the same timestamps as their input packets. In this case,
+simply outputting a packet on every call to `Calculator::Process` is sufficient
+to define output timestamp bounds.
+
+However, calculators are not required to follow this common pattern for output
+timestamps, they are only required to choose monotonically increasing output
+timestamps. As a result, certain calculators must calculate timestamp bounds
+explicitly. MediaPipe provides several tools for computing appropriate timestamp
+bound for each calculator.
+
+1\. **SetNextTimestampBound()** can be used to specify the timestamp bound, `t +
+1`, for an output stream.
+
+```
+cc->Outputs.Tag("OUT").SetNextTimestampBound(t.NextAllowedInStream());
+```
+
+Alternatively, an empty packet with timestamp `t` can be produced to specify the
+timestamp bound `t + 1`.
+
+```
+cc->Outputs.Tag("OUT").Add(Packet(), t);
+```
+
+The timestamp bound of an input stream is indicated by the packet or the empty
+packet on the input stream.
+
+```
+Timestamp bound = cc->Inputs().Tag("IN").Value().Timestamp();
+```
+
+2\. **TimestampOffset()** can be specified in order to automatically copy the
+timestamp bound from input streams to output streams.
+
+```
+cc->SetTimestampOffset(0);
+```
+
+This setting has the advantage of propagating timestamp bounds automatically,
+even when only timestamp bounds arrive and Calculator::Process is not invoked.
+
+3\. **ProcessTimestampBounds()** can be specified in order to invoke
+`Calculator::Process` for each new "settled timestamp", where the "settled
+timestamp" is the new highest timestamp below the current timestamp bounds.
+Without `ProcessTimestampBounds()`, `Calculator::Process` is invoked only with
+one or more arriving packets.
+
+```
+cc->SetProcessTimestampBounds(true);
+```
+
+This setting allows a calculator to perform its own timestamp bounds calculation
+and propagation, even when only input timestamps are updated. It can be used to
+replicate the effect of `TimestampOffset()`, but it can also be used to
+calculate a timestamp bound that takes into account additional factors.
+
+For example, in order to replicate `SetTimestampOffset(0)`, a calculator could
+do the following:
+
+```
+absl::Status Open(CalculatorContext* cc) {
+  cc->SetProcessTimestampBounds(true);
+}
+
+absl::Status Process(CalculatorContext* cc) {
+  cc->Outputs.Tag("OUT").SetNextTimestampBound(
+      cc->InputTimestamp().NextAllowedInStream());
+}
+```
+
+## Scheduling of Calculator::Open and Calculator::Close
+
+`Calculator::Open` is invoked when all required input side-packets have been
+produced. Input side-packets can be provided by the enclosing application or by
+"side-packet calculators" inside the graph. Side-packets can be specified from
+outside the graph using the API's `CalculatorGraph::Initialize` and
+`CalculatorGraph::StartRun`. Side packets can be specified by calculators within
+the graph using `CalculatorGraphConfig::OutputSidePackets` and
+`OutputSidePacket::Set`.
+
+Calculator::Close is invoked when all of the input streams have become `Done` by
+being closed or reaching timestamp bound `Timestamp::Done`.
+
+**Note:** If the graph finishes all pending calculator execution and becomes
+`Done`, before some streams become `Done`, then MediaPipe will invoke the
+remaining calls to `Calculator::Close`, so that every calculator can produce its
+final outputs.
+
+The use of `TimestampOffset` has some implications for `Calculator::Close`. A
+calculator specifying `SetTimestampOffset(0)` will by design signal that all of
+its output streams have reached `Timestamp::Done` when all of its input streams
+have reached `Timestamp::Done`, and therefore no further outputs are possible.
+This prevents such a calculator from emitting any packets during
+`Calculator::Close`. If a calculator needs to produce a summary packet during
+`Calculator::Close`, `Calculator::Process` must specify timestamp bounds such
+that at least one timestamp (such as `Timestamp::Max`) remains available during
+`Calculator::Close`. This means that such a calculator normally cannot rely upon
+`SetTimestampOffset(0)` and must instead specify timestamp bounds explicitly
+using `SetNextTimestampBounds()`.

docs/getting_started/android.md

@@ -57,7 +57,7 @@ Please verify all the necessary packages are installed.
 *   Android SDK Build-Tools 28 or 29
 *   Android SDK Platform-Tools 28 or 29
 *   Android SDK Tools 26.1.1
-*   Android NDK 17c or above
+*   Android NDK 19c or above
 
 ### Option 1: Build with Bazel in Command Line
 
@@ -111,7 +111,7 @@ app:
     *   Verify that Android SDK Build-Tools 28 or 29 is installed.
     *   Verify that Android SDK Platform-Tools 28 or 29 is installed.
     *   Verify that Android SDK Tools 26.1.1 is installed.
-    *   Verify that Android NDK 17c or above is installed.
+    *   Verify that Android NDK 19c or above is installed.
     *   Take note of the Android NDK Location, e.g.,
         `/usr/local/home/Android/Sdk/ndk-bundle` or
         `/usr/local/home/Android/Sdk/ndk/20.0.5594570`.

docs/getting_started/android_archive_library.md

@@ -37,7 +37,7 @@ each project.
     load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")
 
     mediapipe_aar(
-        name = "mp_face_detection_aar",
+        name = "mediapipe_face_detection",
         calculators = ["//mediapipe/graphs/face_detection:mobile_calculators"],
     )
     ```
@@ -45,26 +45,29 @@ each project.
 2.  Run the Bazel build command to generate the AAR.
 
     ```bash
-    bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
-        --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS \
-        //path/to/the/aar/build/file:aar_name
+    bazel build -c opt --strip=ALWAYS \
+        --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
+        --fat_apk_cpu=arm64-v8a,armeabi-v7a \
+        //path/to/the/aar/build/file:aar_name.aar
     ```
 
-    For the face detection AAR target we made in the step 1, run:
+    For the face detection AAR target we made in step 1, run:
 
     ```bash
-    bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \
-        //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar
+    bazel build -c opt --strip=ALWAYS \
+        --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
+        --fat_apk_cpu=arm64-v8a,armeabi-v7a \
+        //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mediapipe_face_detection.aar
 
     # It should print:
-    # Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar up-to-date:
-    # bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
+    # Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mediapipe_face_detection.aar up-to-date:
+    # bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mediapipe_face_detection.aar
     ```
 
 3.  (Optional) Save the AAR to your preferred location.
 
     ```bash
-    cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
+    cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mediapipe_face_detection.aar
     /absolute/path/to/your/preferred/location
     ```
 
@@ -75,7 +78,7 @@ each project.
 2.  Copy the AAR into app/libs.
 
     ```bash
-    cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
+    cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mediapipe_face_detection.aar
     /path/to/your/app/libs/
     ```
 
@@ -92,29 +95,14 @@ each project.
     [the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite).
 
     ```bash
-    bazel build -c opt mediapipe/mediapipe/graphs/face_detection:mobile_gpu_binary_graph
-    cp bazel-bin/mediapipe/graphs/face_detection/mobile_gpu.binarypb /path/to/your/app/src/main/assets/
+    bazel build -c opt mediapipe/graphs/face_detection:face_detection_mobile_gpu_binary_graph
+    cp bazel-bin/mediapipe/graphs/face_detection/face_detection_mobile_gpu.binarypb /path/to/your/app/src/main/assets/
     cp mediapipe/modules/face_detection/face_detection_front.tflite /path/to/your/app/src/main/assets/
     ```
 
     ![Screenshot](../images/mobile/assets_location.png)
 
-4.  Make app/src/main/jniLibs and copy OpenCV JNI libraries into
-    app/src/main/jniLibs.
-
-    MediaPipe depends on OpenCV, you will need to copy the precompiled OpenCV so
-    files into app/src/main/jniLibs. You can download the official OpenCV
-    Android SDK from
-    [here](https://github.com/opencv/opencv/releases/download/3.4.3/opencv-3.4.3-android-sdk.zip)
-    and run:
-
-    ```bash
-    cp -R ~/Downloads/OpenCV-android-sdk/sdk/native/libs/arm* /path/to/your/app/src/main/jniLibs/
-    ```
-
-    ![Screenshot](../images/mobile/android_studio_opencv_location.png)
-
-5.  Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR.
+4.  Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR.
 
     ```
     dependencies {
@@ -136,10 +124,14 @@ each project.
         implementation "androidx.camera:camera-core:$camerax_version"
         implementation "androidx.camera:camera-camera2:$camerax_version"
         implementation "androidx.camera:camera-lifecycle:$camerax_version"
+        // AutoValue
+        def auto_value_version = "1.6.4"
+        implementation "com.google.auto.value:auto-value-annotations:$auto_value_version"
+        annotationProcessor "com.google.auto.value:auto-value:$auto_value_version"
     }
     ```
 
-6.  Follow our Android app examples to use MediaPipe in Android Studio for your
+5.  Follow our Android app examples to use MediaPipe in Android Studio for your
     use case. If you are looking for an example, a face detection example can be
     found
     [here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and

docs/getting_started/install.md

@@ -471,7 +471,7 @@ next section.
 4.  Install Visual C++ Build Tools 2019 and WinSDK
 
     Go to
-    [the VisualStudio website](ttps://visualstudio.microsoft.com/visual-cpp-build-tools),
+    [the VisualStudio website](https://visualstudio.microsoft.com/visual-cpp-build-tools),
     download build tools, and install Microsoft Visual C++ 2019 Redistributable
     and Microsoft Build Tools 2019.
 
@@ -738,7 +738,7 @@ common build issues.
     root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk.sh
 
     # Should print:
-    # Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r18b
+    # Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r19c
     # Set android_ndk_repository and android_sdk_repository in WORKSPACE
     # Done
 

docs/getting_started/javascript.md

@@ -16,13 +16,14 @@ nav_order: 4
 
 MediaPipe currently offers the following solutions:
 
-Solution          | NPM Package                   | Example
------------------ | ----------------------------- | -------
-[Face Mesh][F-pg] | [@mediapipe/face_mesh][F-npm] | [mediapipe.dev/demo/face_mesh][F-demo]
-[Face Detection][Fd-pg] | [@mediapipe/face_detection][Fd-npm] | [mediapipe.dev/demo/face_detection][Fd-demo]
-[Hands][H-pg]     | [@mediapipe/hands][H-npm]     | [mediapipe.dev/demo/hands][H-demo]
-[Holistic][Ho-pg] | [@mediapipe/holistic][Ho-npm] | [mediapipe.dev/demo/holistic][Ho-demo]
-[Pose][P-pg]      | [@mediapipe/pose][P-npm]      | [mediapipe.dev/demo/pose][P-demo]
+Solution                    | NPM Package                             | Example
+--------------------------- | --------------------------------------- | -------
+[Face Mesh][F-pg]           | [@mediapipe/face_mesh][F-npm]           | [mediapipe.dev/demo/face_mesh][F-demo]
+[Face Detection][Fd-pg]     | [@mediapipe/face_detection][Fd-npm]     | [mediapipe.dev/demo/face_detection][Fd-demo]
+[Hands][H-pg]               | [@mediapipe/hands][H-npm]               | [mediapipe.dev/demo/hands][H-demo]
+[Holistic][Ho-pg]           | [@mediapipe/holistic][Ho-npm]           | [mediapipe.dev/demo/holistic][Ho-demo]
+[Pose][P-pg]                | [@mediapipe/pose][P-npm]                | [mediapipe.dev/demo/pose][P-demo]
+[Selfie Segmentation][S-pg] | [@mediapipe/selfie_segmentation][S-npm] | [mediapipe.dev/demo/selfie_segmentation][S-demo]
 
 Click on a solution link above for more information, including API and code
 snippets.
@@ -67,11 +68,13 @@ affecting your work, restrict your request to a `<minor>` number. e.g.,
 [Fd-pg]: ../solutions/face_detection#javascript-solution-api
 [H-pg]: ../solutions/hands#javascript-solution-api
 [P-pg]: ../solutions/pose#javascript-solution-api
+[S-pg]: ../solutions/selfie_segmentation#javascript-solution-api
 [Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic
 [F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh
 [Fd-npm]: https://www.npmjs.com/package/@mediapipe/face_detection
 [H-npm]: https://www.npmjs.com/package/@mediapipe/hands
 [P-npm]: https://www.npmjs.com/package/@mediapipe/pose
+[S-npm]: https://www.npmjs.com/package/@mediapipe/selfie_segmentation
 [draw-npm]: https://www.npmjs.com/package/@mediapipe/drawing_utils
 [cam-npm]: https://www.npmjs.com/package/@mediapipe/camera_utils
 [ctrl-npm]: https://www.npmjs.com/package/@mediapipe/control_utils
@@ -80,15 +83,18 @@ affecting your work, restrict your request to a `<minor>` number. e.g.,
 [Fd-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/face_detection
 [H-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/hands
 [P-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/pose
+[P-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/selfie_segmentation
 [Ho-pen]: https://code.mediapipe.dev/codepen/holistic
 [F-pen]: https://code.mediapipe.dev/codepen/face_mesh
 [Fd-pen]: https://code.mediapipe.dev/codepen/face_detection
 [H-pen]: https://code.mediapipe.dev/codepen/hands
 [P-pen]: https://code.mediapipe.dev/codepen/pose
+[S-pen]: https://code.mediapipe.dev/codepen/selfie_segmentation
 [Ho-demo]: https://mediapipe.dev/demo/holistic
 [F-demo]: https://mediapipe.dev/demo/face_mesh
 [Fd-demo]: https://mediapipe.dev/demo/face_detection
 [H-demo]: https://mediapipe.dev/demo/hands
 [P-demo]: https://mediapipe.dev/demo/pose
+[S-demo]: https://mediapipe.dev/demo/selfie_segmentation
 [npm]: https://www.npmjs.com/package/@mediapipe
 [codepen]: https://code.mediapipe.dev/codepen

docs/getting_started/python.md

@@ -26,7 +26,7 @@ You can, for instance, activate a Python virtual environment:
 $ python3 -m venv mp_env && source mp_env/bin/activate
 ```
 
-Install MediaPipe Python package and start Python intepreter:
+Install MediaPipe Python package and start Python interpreter:
 
 ```bash
 (mp_env)$ pip install mediapipe
@@ -51,6 +51,7 @@ details in each solution via the links below:
 *   [MediaPipe Holistic](../solutions/holistic#python-solution-api)
 *   [MediaPipe Objectron](../solutions/objectron#python-solution-api)
 *   [MediaPipe Pose](../solutions/pose#python-solution-api)
+*   [MediaPipe Selfie Segmentation](../solutions/selfie_segmentation#python-solution-api)
 
 ## MediaPipe on Google Colab
 
@@ -62,6 +63,7 @@ details in each solution via the links below:
 *   [MediaPipe Pose Colab](https://mediapipe.page.link/pose_py_colab)
 *   [MediaPipe Pose Classification Colab (Basic)](https://mediapipe.page.link/pose_classification_basic)
 *   [MediaPipe Pose Classification Colab (Extended)](https://mediapipe.page.link/pose_classification_extended)
+*   [MediaPipe Selfie Segmentation Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)
 
 ## MediaPipe Python Framework
 

docs/getting_started/troubleshooting.md

@@ -97,6 +97,49 @@ linux_opencv/macos_opencv/windows_opencv.BUILD files for your local opencv
 libraries. [This GitHub issue](https://github.com/google/mediapipe/issues/666)
 may also help.
 
+## Python pip install failure
+
+The error message:
+
+```
+ERROR: Could not find a version that satisfies the requirement mediapipe
+ERROR: No matching distribution found for mediapipe
+```
+
+after running `pip install mediapipe` usually indicates that there is no qualified MediaPipe Python for your system.
+Please note that MediaPipe Python PyPI officially supports the **64-bit**
+version of Python 3.7 and above on the following OS:
+
+-   x86_64 Linux
+-   x86_64 macOS 10.15+
+-   amd64 Windows
+
+If the OS is currently supported and you still see this error, please make sure
+that both the Python and pip binary are for Python 3.7 and above. Otherwise,
+please consider building the MediaPipe Python package locally by following the
+instructions [here](python.md#building-mediapipe-python-package).
+
+## Python DLL load failure on Windows
+
+The error message:
+
+```
+ImportError: DLL load failed: The specified module could not be found
+```
+
+usually indicates that the local Windows system is missing Visual C++
+redistributable packages and/or Visual C++ runtime DLLs. This can be solved by
+either installing the official
+[vc_redist.x64.exe](https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0)
+or installing the "msvc-runtime" Python package by running
+
+```bash
+$ python -m pip install msvc-runtime
+```
+
+Please note that the "msvc-runtime" Python package is not released or maintained
+by Microsoft.
+
 ## Native method not found
 
 The error message:

docs/index.md

@@ -40,11 +40,12 @@ Hair Segmentation
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                        | ✅                                                         | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |

docs/solutions/autoflip.md

@@ -2,7 +2,7 @@
 layout: default
 title: AutoFlip (Saliency-aware Video Cropping)
 parent: Solutions
-nav_order: 13
+nav_order: 14
 ---
 
 # AutoFlip: Saliency-aware Video Cropping

docs/solutions/box_tracking.md

@@ -2,7 +2,7 @@
 layout: default
 title: Box Tracking
 parent: Solutions
-nav_order: 9
+nav_order: 10
 ---
 
 # MediaPipe Box Tracking

docs/solutions/face_detection.md

@@ -68,7 +68,7 @@ normalized to `[0.0, 1.0]` by the image width and height respectively.
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -77,13 +77,14 @@ Supported configuration options:
 ```python
 import cv2
 import mediapipe as mp
-mp_face_detction = mp.solutions.face_detection
+mp_face_detection = mp.solutions.face_detection
 mp_drawing = mp.solutions.drawing_utils
 
 # For static images:
+IMAGE_FILES = []
 with mp_face_detection.FaceDetection(
     min_detection_confidence=0.5) as face_detection:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB and process it with MediaPipe Face Detection.
     results = face_detection.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

docs/solutions/face_mesh.md

@@ -265,7 +265,7 @@ magnitude of `z` uses roughly the same scale as `x`.
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -281,12 +281,13 @@ mp_drawing = mp.solutions.drawing_utils
 mp_face_mesh = mp.solutions.face_mesh
 
 # For static images:
+IMAGE_FILES = []
 drawing_spec = mp_drawing.DrawingSpec(thickness=1, circle_radius=1)
 with mp_face_mesh.FaceMesh(
     static_image_mode=True,
     max_num_faces=1,
     min_detection_confidence=0.5) as face_mesh:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB before processing.
     results = face_mesh.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

docs/solutions/hair_segmentation.md

@@ -2,7 +2,7 @@
 layout: default
 title: Hair Segmentation
 parent: Solutions
-nav_order: 7
+nav_order: 8
 ---
 
 # MediaPipe Hair Segmentation

docs/solutions/hands.md

@@ -206,7 +206,7 @@ is not the case, please swap the handedness output in the application.
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -222,11 +222,12 @@ mp_drawing = mp.solutions.drawing_utils
 mp_hands = mp.solutions.hands
 
 # For static images:
+IMAGE_FILES = []
 with mp_hands.Hands(
     static_image_mode=True,
     max_num_hands=2,
     min_detection_confidence=0.5) as hands:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     # Read an image, flip it around y-axis for correct handedness output (see
     # above).
     image = cv2.flip(cv2.imread(file), 1)

docs/solutions/holistic.md

@@ -135,12 +135,11 @@ another detection until it loses track, on reducing computation and latency. If
 set to `true`, person detection runs every input image, ideal for processing a
 batch of static, possibly unrelated, images. Default to `false`.
 
-#### upper_body_only
+#### model_complexity
 
-If set to `true`, the solution outputs only the 25 upper-body pose landmarks
-(535 in total) instead of the full set of 33 pose landmarks (543 in total). Note
-that upper-body-only prediction may be more accurate for use cases where the
-lower-body parts are mostly out of view. Default to `false`.
+Complexity of the pose landmark model: `0`, `1` or `2`. Landmark accuracy as
+well as inference latency generally go up with the model complexity. Default to
+`1`.
 
 #### smooth_landmarks
 
@@ -202,12 +201,12 @@ A list of 21 hand landmarks on the right hand, in the same representation as
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
 *   [static_image_mode](#static_image_mode)
-*   [upper_body_only](#upper_body_only)
+*   [model_complexity](#model_complexity)
 *   [smooth_landmarks](#smooth_landmarks)
 *   [min_detection_confidence](#min_detection_confidence)
 *   [min_tracking_confidence](#min_tracking_confidence)
@@ -219,8 +218,11 @@ mp_drawing = mp.solutions.drawing_utils
 mp_holistic = mp.solutions.holistic
 
 # For static images:
-with mp_holistic.Holistic(static_image_mode=True) as holistic:
-  for idx, file in enumerate(file_list):
+IMAGE_FILES = []
+with mp_holistic.Holistic(
+    static_image_mode=True,
+    model_complexity=2) as holistic:
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     image_height, image_width, _ = image.shape
     # Convert the BGR image to RGB before processing.
@@ -240,8 +242,6 @@ with mp_holistic.Holistic(static_image_mode=True) as holistic:
         annotated_image, results.left_hand_landmarks, mp_holistic.HAND_CONNECTIONS)
     mp_drawing.draw_landmarks(
         annotated_image, results.right_hand_landmarks, mp_holistic.HAND_CONNECTIONS)
-    # Use mp_holistic.UPPER_BODY_POSE_CONNECTIONS for drawing below when
-    # upper_body_only is set to True.
     mp_drawing.draw_landmarks(
         annotated_image, results.pose_landmarks, mp_holistic.POSE_CONNECTIONS)
     cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
@@ -291,7 +291,7 @@ and the following usage example.
 
 Supported configuration options:
 
-*   [upperBodyOnly](#upper_body_only)
+*   [modelComplexity](#model_complexity)
 *   [smoothLandmarks](#smooth_landmarks)
 *   [minDetectionConfidence](#min_detection_confidence)
 *   [minTrackingConfidence](#min_tracking_confidence)
@@ -348,7 +348,7 @@ const holistic = new Holistic({locateFile: (file) => {
   return `https://cdn.jsdelivr.net/npm/@mediapipe/holistic/${file}`;
 }});
 holistic.setOptions({
-  upperBodyOnly: false,
+  modelComplexity: 1,
   smoothLandmarks: true,
   minDetectionConfidence: 0.5,
   minTrackingConfidence: 0.5

docs/solutions/instant_motion_tracking.md

@@ -2,7 +2,7 @@
 layout: default
 title: Instant Motion Tracking
 parent: Solutions
-nav_order: 10
+nav_order: 11
 ---
 
 # MediaPipe Instant Motion Tracking

docs/solutions/knift.md

@@ -2,7 +2,7 @@
 layout: default
 title: KNIFT (Template-based Feature Matching)
 parent: Solutions
-nav_order: 12
+nav_order: 13
 ---
 
 # MediaPipe KNIFT

docs/solutions/media_sequence.md

@@ -2,7 +2,7 @@
 layout: default
 title: Dataset Preparation with MediaSequence
 parent: Solutions
-nav_order: 14
+nav_order: 15
 ---
 
 # Dataset Preparation with MediaSequence

docs/solutions/models.md

@@ -15,11 +15,16 @@ nav_order: 30
 ### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
 
 *   Face detection model for front-facing/selfie camera:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite),
-    [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite),
+    [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite),
+    [Model card](https://mediapipe.page.link/blazeface-mc)
 *   Face detection model for back-facing camera:
-    [TFLite model ](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_back.tflite)
-*   [Model card](https://mediapipe.page.link/blazeface-mc)
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back.tflite),
+    [Model card](https://mediapipe.page.link/blazeface-back-mc)
+*   Face detection model for back-facing camera (sparse):
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back_sparse.tflite),
+    [Model card](https://mediapipe.page.link/blazeface-back-sparse-mc)
+
 
 ### [Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)
 
@@ -49,10 +54,10 @@ nav_order: 30
 
 *   Pose detection model:
     [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection/pose_detection.tflite)
-*   Full-body pose landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_full_body.tflite)
-*   Upper-body pose landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_upper_body.tflite)
+*   Pose landmark model:
+    [TFLite model (lite)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_lite.tflite),
+    [TFLite model (full)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_full.tflite),
+    [TFLite model (heavy)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_heavy.tflite)
 *   [Model card](https://mediapipe.page.link/blazepose-mc)
 
 ### [Holistic](https://google.github.io/mediapipe/solutions/holistic)
@@ -60,6 +65,12 @@ nav_order: 30
 *   Hand recrop model:
     [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/holistic_landmark/hand_recrop.tflite)
 
+### [Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)
+
+*   [TFLite model (general)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation.tflite)
+*   [TFLite model (landscape)](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation_landscape.tflite)
+*   [Model card](https://mediapipe.page.link/selfiesegmentation-mc)
+
 ### [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)
 
 *   [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite)

docs/solutions/object_detection.md

@@ -2,7 +2,7 @@
 layout: default
 title: Object Detection
 parent: Solutions
-nav_order: 8
+nav_order: 9
 ---
 
 # MediaPipe Object Detection

docs/solutions/objectron.md

@@ -2,7 +2,7 @@
 layout: default
 title: Objectron (3D Object Detection)
 parent: Solutions
-nav_order: 11
+nav_order: 12
 ---
 
 # MediaPipe Objectron
@@ -277,7 +277,7 @@ following:
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
@@ -297,11 +297,12 @@ mp_drawing = mp.solutions.drawing_utils
 mp_objectron = mp.solutions.objectron
 
 # For static images:
+IMAGE_FILES = []
 with mp_objectron.Objectron(static_image_mode=True,
                             max_num_objects=5,
                             min_detection_confidence=0.5,
                             model_name='Shoe') as objectron:
-  for idx, file in enumerate(file_list):
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     # Convert the BGR image to RGB and process it with MediaPipe Objectron.
     results = objectron.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

docs/solutions/pose.md

@@ -30,8 +30,7 @@ overlay of digital content and information on top of the physical world in
 augmented reality.
 
 MediaPipe Pose is a ML solution for high-fidelity body pose tracking, inferring
-33 3D landmarks on the whole body (or 25 upper-body landmarks) from RGB video
-frames utilizing our
+33 3D landmarks on the whole body from RGB video frames utilizing our
 [BlazePose](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
 research that also powers the
 [ML Kit Pose Detection API](https://developers.google.com/ml-kit/vision/pose-detection).
@@ -40,9 +39,9 @@ environments for inference, whereas our method achieves real-time performance on
 most modern [mobile phones](#mobile), [desktops/laptops](#desktop), in
 [python](#python-solution-api) and even on the [web](#javascript-solution-api).
 
-![pose_tracking_upper_body_example.gif](../images/mobile/pose_tracking_upper_body_example.gif) |
-:--------------------------------------------------------------------------------------------: |
-*Fig 1. Example of MediaPipe Pose for upper-body pose tracking.*                               |
+![pose_tracking_example.gif](../images/mobile/pose_tracking_example.gif) |
+:----------------------------------------------------------------------: |
+*Fig 1. Example of MediaPipe Pose for pose tracking.*                    |
 
 ## ML Pipeline
 
@@ -77,6 +76,36 @@ Note: To visualize a graph, copy the graph and paste it into
 to visualize its associated subgraphs, please see
 [visualizer documentation](../tools/visualizer.md).
 
+## Pose Estimation Quality
+
+To evaluate the quality of our [models](./models.md#pose) against other
+well-performing publicly available solutions, we use three different validation
+datasets, representing different verticals: Yoga, Dance and HIIT. Each image
+contains only a single person located 2-4 meters from the camera. To be
+consistent with other solutions, we perform evaluation only for 17 keypoints
+from [COCO topology](https://cocodataset.org/#keypoints-2020).
+
+Method                                                                                                | Yoga <br/> [`mAP`] | Yoga <br/> [`[email protected]`] | Dance <br/> [`mAP`] | Dance <br/> [`[email protected]`] | HIIT <br/> [`mAP`] | HIIT <br/> [`[email protected]`]
+----------------------------------------------------------------------------------------------------- | -----------------: | ---------------------: | ------------------: | ----------------------: | -----------------: | ---------------------:
+BlazePose.Heavy                                                                                       | 68.1               | **96.4**               | 73.0                | **97.2**                | 74.0               | **97.5**
+BlazePose.Full                                                                                        | 62.6               | **95.5**               | 67.4                | **96.3**                | 68.0               | **95.7**
+BlazePose.Lite                                                                                        | 45.0               | **90.2**               | 53.6                | **92.5**                | 53.8               | **93.5**
+[AlphaPose.ResNet50](https://github.com/MVIG-SJTU/AlphaPose)                                          | 63.4               | **96.0**               | 57.8                | **95.5**                | 63.4               | **96.0**
+[Apple.Vision](https://developer.apple.com/documentation/vision/detecting_human_body_poses_in_images) | 32.8               | **82.7**               | 36.4                | **91.4**                | 44.5               | **88.6**
+
+![pose_tracking_pck_chart.png](../images/mobile/pose_tracking_pck_chart.png) |
+:--------------------------------------------------------------------------: |
+*Fig 2. Quality evaluation in [`[email protected]`].*                                  |
+
+We designed our models specifically for live perception use cases, so all of
+them work in real-time on the majority of modern devices.
+
+Method          | Latency <br/> Pixel 3 [TFLite GPU](https://www.tensorflow.org/lite/performance/gpu_advanced) | Latency <br/> MacBook Pro (15-inch 2017)
+--------------- | -------------------------------------------------------------------------------------------: | ---------------------------------------:
+BlazePose.Heavy | 53 ms                                                                                        | 38 ms
+BlazePose.Full  | 25 ms                                                                                        | 27 ms
+BlazePose.Lite  | 20 ms                                                                                        | 25 ms
+
 ## Models
 
 ### Person/pose Detection Model (BlazePose Detector)
@@ -93,15 +122,12 @@ hip midpoints.
 
 ![pose_tracking_detector_vitruvian_man.png](../images/mobile/pose_tracking_detector_vitruvian_man.png) |
 :----------------------------------------------------------------------------------------------------: |
-*Fig 2. Vitruvian man aligned via two virtual keypoints predicted by BlazePose detector in addition to the face bounding box.* |
+*Fig 3. Vitruvian man aligned via two virtual keypoints predicted by BlazePose detector in addition to the face bounding box.* |
 
 ### Pose Landmark Model (BlazePose GHUM 3D)
 
-The landmark model in MediaPipe Pose comes in two versions: a full-body model
-that predicts the location of 33 pose landmarks (see figure below), and an
-upper-body version that only predicts the first 25. The latter may be more
-accurate than the former in scenarios where the lower-body parts are mostly out
-of view.
+The landmark model in MediaPipe Pose predicts the location of 33 pose landmarks
+(see figure below).
 
 Please find more detail in the
 [BlazePose Google AI Blog](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html),
@@ -111,7 +137,7 @@ this [paper](https://arxiv.org/abs/2006.10204) and
 
 ![pose_tracking_full_body_landmarks.png](../images/mobile/pose_tracking_full_body_landmarks.png) |
 :----------------------------------------------------------------------------------------------: |
-*Fig 3. 33 pose landmarks.*                                                                      |
+*Fig 4. 33 pose landmarks.*                                                                      |
 
 ## Solution APIs
 
@@ -129,12 +155,11 @@ until it loses track, on reducing computation and latency. If set to `true`,
 person detection runs every input image, ideal for processing a batch of static,
 possibly unrelated, images. Default to `false`.
 
-#### upper_body_only
+#### model_complexity
 
-If set to `true`, the solution outputs only the 25 upper-body pose landmarks.
-Otherwise, it outputs the full set of 33 pose landmarks. Note that
-upper-body-only prediction may be more accurate for use cases where the
-lower-body parts are mostly out of view. Default to `false`.
+Complexity of the pose landmark model: `0`, `1` or `2`. Landmark accuracy as
+well as inference latency generally go up with the model complexity. Default to
+`1`.
 
 #### smooth_landmarks
 
@@ -162,7 +187,7 @@ Naming style may differ slightly across platforms/languages.
 
 #### pose_landmarks
 
-A list of pose landmarks. Each lanmark consists of the following:
+A list of pose landmarks. Each landmark consists of the following:
 
 *   `x` and `y`: Landmark coordinates normalized to `[0.0, 1.0]` by the image
     width and height respectively.
@@ -170,9 +195,6 @@ A list of pose landmarks. Each lanmark consists of the following:
     being the origin, and the smaller the value the closer the landmark is to
     the camera. The magnitude of `z` uses roughly the same scale as `x`.
 
-    Note: `z` is predicted only in full-body mode, and should be discarded when
-    [upper_body_only](#upper_body_only) is `true`.
-
 *   `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
     landmark being visible (present and not occluded) in the image.
 
@@ -180,12 +202,12 @@ A list of pose landmarks. Each lanmark consists of the following:
 
 Please first follow general [instructions](../getting_started/python.md) to
 install MediaPipe Python package, then learn more in the companion
-[Python Colab](#resources) and the following usage example.
+[Python Colab](#resources) and the usage example below.
 
 Supported configuration options:
 
 *   [static_image_mode](#static_image_mode)
-*   [upper_body_only](#upper_body_only)
+*   [model_complexity](#model_complexity)
 *   [smooth_landmarks](#smooth_landmarks)
 *   [min_detection_confidence](#min_detection_confidence)
 *   [min_tracking_confidence](#min_tracking_confidence)
@@ -197,9 +219,12 @@ mp_drawing = mp.solutions.drawing_utils
 mp_pose = mp.solutions.pose
 
 # For static images:
+IMAGE_FILES = []
 with mp_pose.Pose(
-    static_image_mode=True, min_detection_confidence=0.5) as pose:
-  for idx, file in enumerate(file_list):
+    static_image_mode=True,
+    model_complexity=2,
+    min_detection_confidence=0.5) as pose:
+  for idx, file in enumerate(IMAGE_FILES):
     image = cv2.imread(file)
     image_height, image_width, _ = image.shape
     # Convert the BGR image to RGB before processing.
@@ -214,8 +239,6 @@ with mp_pose.Pose(
     )
     # Draw pose landmarks on the image.
     annotated_image = image.copy()
-    # Use mp_pose.UPPER_BODY_POSE_CONNECTIONS for drawing below when
-    # upper_body_only is set to True.
     mp_drawing.draw_landmarks(
         annotated_image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
     cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
@@ -259,7 +282,7 @@ and the following usage example.
 
 Supported configuration options:
 
-*   [upperBodyOnly](#upper_body_only)
+*   [modelComplexity](#model_complexity)
 *   [smoothLandmarks](#smooth_landmarks)
 *   [minDetectionConfidence](#min_detection_confidence)
 *   [minTrackingConfidence](#min_tracking_confidence)
@@ -306,7 +329,7 @@ const pose = new Pose({locateFile: (file) => {
   return `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`;
 }});
 pose.setOptions({
-  upperBodyOnly: false,
+  modelComplexity: 1,
   smoothLandmarks: true,
   minDetectionConfidence: 0.5,
   minTrackingConfidence: 0.5
@@ -347,16 +370,6 @@ to visualize its associated subgraphs, please see
 *   iOS target:
     [`mediapipe/examples/ios/posetrackinggpu:PoseTrackingGpuApp`](http:/mediapipe/examples/ios/posetrackinggpu/BUILD)
 
-#### Upper-body Only
-
-*   Graph:
-    [`mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt)
-*   Android target:
-    [(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1uKc6T7KSuA0Mlq2URi5YookHu0U3yoh_/view?usp=sharing)
-    [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/upperbodyposetrackinggpu:upperbodyposetrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/upperbodyposetrackinggpu/BUILD)
-*   iOS target:
-    [`mediapipe/examples/ios/upperbodyposetrackinggpu:UpperBodyPoseTrackingGpuApp`](http:/mediapipe/examples/ios/upperbodyposetrackinggpu/BUILD)
-
 ### Desktop
 
 Please first see general instructions for [desktop](../getting_started/cpp.md)
@@ -375,19 +388,6 @@ on how to build MediaPipe examples.
     *   Target:
         [`mediapipe/examples/desktop/pose_tracking:pose_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/pose_tracking/BUILD)
 
-#### Upper-body Only
-
-*   Running on CPU
-    *   Graph:
-        [`mediapipe/graphs/pose_tracking/upper_body_pose_tracking_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_cpu.pbtxt)
-    *   Target:
-        [`mediapipe/examples/desktop/upper_body_pose_tracking:upper_body_pose_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/upper_body_pose_tracking/BUILD)
-*   Running on GPU
-    *   Graph:
-        [`mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt)
-    *   Target:
-        [`mediapipe/examples/desktop/upper_body_pose_tracking:upper_body_pose_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/upper_body_pose_tracking/BUILD)
-
 ## Resources
 
 *   Google AI Blog:
@@ -398,3 +398,6 @@ on how to build MediaPipe examples.
 *   [Models and model cards](./models.md#pose)
 *   [Web demo](https://code.mediapipe.dev/codepen/pose)
 *   [Python Colab](https://mediapipe.page.link/pose_py_colab)
+
+[`mAP`]: https://cocodataset.org/#keypoints-eval
+[`[email protected]`]: https://github.com/cbsudux/Human-Pose-Estimation-101

docs/solutions/selfie_segmentation.md

@@ -0,0 +1,289 @@
+---
+layout: default
+title: Selfie Segmentation
+parent: Solutions
+nav_order: 7
+---
+
+# MediaPipe Selfie Segmentation
+{: .no_toc }
+
+<details close markdown="block">
+  <summary>
+    Table of contents
+  </summary>
+  {: .text-delta }
+1. TOC
+{:toc}
+</details>
+---
+
+## Overview
+
+*Fig 1. Example of MediaPipe Selfie Segmentation.* |
+:------------------------------------------------: |
+<video autoplay muted loop preload style="height: auto; width: 480px"><source src="../images/selfie_segmentation_web.mp4" type="video/mp4"></video> |
+
+MediaPipe Selfie Segmentation segments the prominent humans in the scene. It can
+run in real-time on both smartphones and laptops. The intended use cases include
+selfie effects and video conferencing, where the person is close (< 2m) to the
+camera.
+
+## Models
+
+In this solution, we provide two models: general and landscape. Both models are
+based on
+[MobileNetV3](https://ai.googleblog.com/2019/11/introducing-next-generation-on-device.html),
+with modifications to make them more efficient. The general model operates on a
+256x256x3 (HWC) tensor, and outputs a 256x256x1 tensor representing the
+segmentation mask. The landscape model is similar to the general model, but
+operates on a 144x256x3 (HWC) tensor. It has fewer FLOPs than the general model,
+and therefore, runs faster. Note that MediaPipe Selfie Segmentation
+automatically resizes the input image to the desired tensor dimension before
+feeding it into the ML models.
+
+The general model is also powering
+[ML Kit](https://developers.google.com/ml-kit/vision/selfie-segmentation), and a
+variant of the landscape model is powering
+[Google Meet](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html).
+Please find more detail about the models in the
+[model card](./models.md#selfie-segmentation).
+
+## ML Pipeline
+
+The pipeline is implemented as a MediaPipe
+[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+that uses a
+[selfie segmentation subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+from the
+[selfie segmentation module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/selfie_segmentation).
+
+Note: To visualize a graph, copy the graph and paste it into
+[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
+to visualize its associated subgraphs, please see
+[visualizer documentation](../tools/visualizer.md).
+
+## Solution APIs
+
+### Cross-platform Configuration Options
+
+Naming style and availability may differ slightly across platforms/languages.
+
+#### model_selection
+
+An integer index `0` or `1`. Use `0` to select the general model, and `1` to
+select the landscape model (see details in [Models](#models)). Default to `0` if
+not specified.
+
+### Output
+
+Naming style may differ slightly across platforms/languages.
+
+#### segmentation_mask
+
+The output segmentation mask, which has the same dimension as the input image.
+
+### Python Solution API
+
+Please first follow general [instructions](../getting_started/python.md) to
+install MediaPipe Python package, then learn more in the companion
+[Python Colab](#resources) and the usage example below.
+
+Supported configuration options:
+
+*   [model_selection](#model_selection)
+
+```python
+import cv2
+import mediapipe as mp
+mp_drawing = mp.solutions.drawing_utils
+mp_selfie_segmentation = mp.solutions.selfie_segmentation
+
+# For static images:
+IMAGE_FILES = []
+BG_COLOR = (192, 192, 192) # gray
+MASK_COLOR = (255, 255, 255) # white
+with mp_selfie_segmentation.SelfieSegmentation(
+    model_selection=0) as selfie_segmentation:
+  for idx, file in enumerate(IMAGE_FILES):
+    image = cv2.imread(file)
+    image_height, image_width, _ = image.shape
+    # Convert the BGR image to RGB before processing.
+    results = selfie_segmentation.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
+
+    # Draw selfie segmentation on the background image.
+    # To improve segmentation around boundaries, consider applying a joint
+    # bilateral filter to "results.segmentation_mask" with "image".
+    condition = np.stack((results.segmentation_mask,) * 3, axis=-1) > 0.1
+    # Generate solid color images for showing the output selfie segmentation mask.
+    fg_image = np.zeros(image.shape, dtype=np.uint8)
+    fg_image[:] = MASK_COLOR
+    bg_image = np.zeros(image.shape, dtype=np.uint8)
+    bg_image[:] = BG_COLOR
+    output_image = np.where(condition, fg_image, bg_image)
+    cv2.imwrite('/tmp/selfie_segmentation_output' + str(idx) + '.png', output_image)
+
+# For webcam input:
+BG_COLOR = (192, 192, 192) # gray
+cap = cv2.VideoCapture(0)
+with mp_selfie_segmentation.SelfieSegmentation(
+    model_selection=1) as selfie_segmentation:
+  bg_image = None
+  while cap.isOpened():
+    success, image = cap.read()
+    if not success:
+      print("Ignoring empty camera frame.")
+      # If loading a video, use 'break' instead of 'continue'.
+      continue
+
+    # Flip the image horizontally for a later selfie-view display, and convert
+    # the BGR image to RGB.
+    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
+    # To improve performance, optionally mark the image as not writeable to
+    # pass by reference.
+    image.flags.writeable = False
+    results = selfie_segmentation.process(image)
+
+    image.flags.writeable = True
+    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
+
+    # Draw selfie segmentation on the background image.
+    # To improve segmentation around boundaries, consider applying a joint
+    # bilateral filter to "results.segmentation_mask" with "image".
+    condition = np.stack(
+      (results.segmentation_mask,) * 3, axis=-1) > 0.1
+    # The background can be customized.
+    #   a) Load an image (with the same width and height of the input image) to
+    #      be the background, e.g., bg_image = cv2.imread('/path/to/image/file')
+    #   b) Blur the input image by applying image filtering, e.g.,
+    #      bg_image = cv2.GaussianBlur(image,(55,55),0)
+    if bg_image is None:
+      bg_image = np.zeros(image.shape, dtype=np.uint8)
+      bg_image[:] = BG_COLOR
+    output_image = np.where(condition, image, bg_image)
+
+    cv2.imshow('MediaPipe Selfie Segmentation', output_image)
+    if cv2.waitKey(5) & 0xFF == 27:
+      break
+cap.release()
+```
+
+### JavaScript Solution API
+
+Please first see general [introduction](../getting_started/javascript.md) on
+MediaPipe in JavaScript, then learn more in the companion [web demo](#resources)
+and the following usage example.
+
+Supported configuration options:
+
+*   [modelSelection](#model_selection)
+
+```html
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
+  <script src="https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation.js" crossorigin="anonymous"></script>
+</head>
+
+<body>
+  <div class="container">
+    <video class="input_video"></video>
+    <canvas class="output_canvas" width="1280px" height="720px"></canvas>
+  </div>
+</body>
+</html>
+```
+
+```javascript
+<script type="module">
+const videoElement = document.getElementsByClassName('input_video')[0];
+const canvasElement = document.getElementsByClassName('output_canvas')[0];
+const canvasCtx = canvasElement.getContext('2d');
+
+function onResults(results) {
+  canvasCtx.save();
+  canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
+  canvasCtx.drawImage(results.segmentationMask, 0, 0,
+                      canvasElement.width, canvasElement.height);
+
+  // Only overwrite existing pixels.
+  canvasCtx.globalCompositeOperation = 'source-in';
+  canvasCtx.fillStyle = '#00FF00';
+  canvasCtx.fillRect(0, 0, canvasElement.width, canvasElement.height);
+
+  // Only overwrite missing pixels.
+  canvasCtx.globalCompositeOperation = 'destination-atop';
+  canvasCtx.drawImage(
+      results.image, 0, 0, canvasElement.width, canvasElement.height);
+
+  canvasCtx.restore();
+}
+
+const selfieSegmentation = new SelfieSegmentation({locateFile: (file) => {
+  return `https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/${file}`;
+}});
+selfieSegmentation.setOptions({
+  modelSelection: 1,
+});
+selfieSegmentation.onResults(onResults);
+
+const camera = new Camera(videoElement, {
+  onFrame: async () => {
+    await selfieSegmentation.send({image: videoElement});
+  },
+  width: 1280,
+  height: 720
+});
+camera.start();
+</script>
+```
+
+## Example Apps
+
+Please first see general instructions for
+[Android](../getting_started/android.md), [iOS](../getting_started/ios.md), and
+[desktop](../getting_started/cpp.md) on how to build MediaPipe examples.
+
+Note: To visualize a graph, copy the graph and paste it into
+[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
+to visualize its associated subgraphs, please see
+[visualizer documentation](../tools/visualizer.md).
+
+### Mobile
+
+*   Graph:
+    [`mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+*   Android target:
+    [(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1DoeyGzMmWUsjfVgZfGGecrn7GKzYcEAo/view?usp=sharing)
+    [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/selfiesegmentationgpu:selfiesegmentationgpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/selfiesegmentationgpu/BUILD)
+*   iOS target:
+    [`mediapipe/examples/ios/selfiesegmentationgpu:SelfieSegmentationGpuApp`](http:/mediapipe/examples/ios/selfiesegmentationgpu/BUILD)
+
+### Desktop
+
+Please first see general instructions for [desktop](../getting_started/cpp.md)
+on how to build MediaPipe examples.
+
+*   Running on CPU
+    *   Graph:
+        [`mediapipe/graphs/selfie_segmentation/selfie_segmentation_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_cpu.pbtxt)
+    *   Target:
+        [`mediapipe/examples/desktop/selfie_segmentation:selfie_segmentation_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/selfie_segmentation/BUILD)
+*   Running on GPU
+    *   Graph:
+        [`mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/selfie_segmentation/selfie_segmentation_gpu.pbtxt)
+    *   Target:
+        [`mediapipe/examples/desktop/selfie_segmentation:selfie_segmentation_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/selfie_segmentation/BUILD)
+
+## Resources
+
+*   Google AI Blog:
+    [Background Features in Google Meet, Powered by Web ML](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html)
+*   [ML Kit Selfie Segmentation API](https://developers.google.com/ml-kit/vision/selfie-segmentation)
+*   [Models and model cards](./models.md#selfie_segmentation)
+*   [Web demo](https://code.mediapipe.dev/codepen/selfie_segmentation)
+*   [Python Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)

docs/solutions/solutions.md

@@ -24,11 +24,12 @@ has_toc: false
 [Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Holistic](https://google.github.io/mediapipe/solutions/holistic)                               | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
+[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation)         | ✅                                                               | ✅                                                       | ✅                                                       | ✅                                                             | ✅                                                             |
 [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅                                                               |                                                         | ✅                                                       |                                                               |                                                               |
 [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               | ✅
 [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅                                                               | ✅                                                       | ✅                                                       |                                                               |                                                               |
 [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                        | ✅                                                             |                                                               |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅                                                               |                                                         | ✅                                                       | ✅                                                             |                                                               |
 [KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅                                                               |                                                         |                                                         |                                                               |                                                               |
 [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |
 [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |                                                                 |                                                         | ✅                                                       |                                                               |                                                               |

docs/solutions/youtube_8m.md

@@ -2,7 +2,7 @@
 layout: default
 title: YouTube-8M Feature Extraction and Model Inference
 parent: Solutions
-nav_order: 15
+nav_order: 16
 ---
 
 # YouTube-8M Feature Extraction and Model Inference

mediapipe/MediaPipe.tulsiproj/Configs/MediaPipe.tulsigen

@@ -16,7 +16,7 @@
     "mediapipe/examples/ios/objectdetectiongpu/BUILD",
     "mediapipe/examples/ios/objectdetectiontrackinggpu/BUILD",
     "mediapipe/examples/ios/posetrackinggpu/BUILD",
-    "mediapipe/examples/ios/upperbodyposetrackinggpu/BUILD",
+    "mediapipe/examples/ios/selfiesegmentationgpu/BUILD",
     "mediapipe/framework/BUILD",
     "mediapipe/gpu/BUILD",
     "mediapipe/objc/BUILD",
@@ -36,7 +36,7 @@
     "//mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp",
     "//mediapipe/examples/ios/objectdetectiontrackinggpu:ObjectDetectionTrackingGpuApp",
     "//mediapipe/examples/ios/posetrackinggpu:PoseTrackingGpuApp",
-    "//mediapipe/examples/ios/upperbodyposetrackinggpu:UpperBodyPoseTrackingGpuApp",
+    "//mediapipe/examples/ios/selfiesegmentationgpu:SelfieSegmentationGpuApp",
     "//mediapipe/objc:mediapipe_framework_ios"
   ],
   "optionSet" : {
@@ -105,7 +105,7 @@
     "mediapipe/examples/ios/objectdetectioncpu",
     "mediapipe/examples/ios/objectdetectiongpu",
     "mediapipe/examples/ios/posetrackinggpu",
-    "mediapipe/examples/ios/upperbodyposetrackinggpu",
+    "mediapipe/examples/ios/selfiesegmentationgpu",
     "mediapipe/framework",
     "mediapipe/framework/deps",
     "mediapipe/framework/formats",
@@ -123,6 +123,7 @@
     "mediapipe/graphs/hand_tracking",
     "mediapipe/graphs/object_detection",
     "mediapipe/graphs/pose_tracking",
+    "mediapipe/graphs/selfie_segmentation",
     "mediapipe/models",
     "mediapipe/modules",
     "mediapipe/objc",

mediapipe/MediaPipe.tulsiproj/project.tulsiconf

@@ -22,7 +22,7 @@
     "mediapipe/examples/ios/objectdetectiongpu",
     "mediapipe/examples/ios/objectdetectiontrackinggpu",
     "mediapipe/examples/ios/posetrackinggpu",
-    "mediapipe/examples/ios/upperbodyposetrackinggpu",
+    "mediapipe/examples/ios/selfiesegmentationgpu",
     "mediapipe/objc"
   ],
   "projectName" : "Mediapipe",

mediapipe/calculators/core/BUILD

@@ -233,6 +233,22 @@ cc_test(
     ],
 )
 
+cc_library(
+    name = "concatenate_vector_calculator_hdr",
+    hdrs = ["concatenate_vector_calculator.h"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":concatenate_vector_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/api2:node",
+        "//mediapipe/framework/api2:port",
+        "//mediapipe/framework/port:integral_types",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
+    ],
+    alwayslink = 1,
+)
+
 cc_library(
     name = "concatenate_vector_calculator",
     srcs = ["concatenate_vector_calculator.cc"],
@@ -451,8 +467,8 @@ cc_library(
 )
 
 cc_library(
-    name = "nonzero_calculator",
-    srcs = ["nonzero_calculator.cc"],
+    name = "non_zero_calculator",
+    srcs = ["non_zero_calculator.cc"],
     visibility = [
         "//visibility:public",
     ],
@@ -464,6 +480,21 @@ cc_library(
     alwayslink = 1,
 )
 
+cc_test(
+    name = "non_zero_calculator_test",
+    size = "small",
+    srcs = ["non_zero_calculator_test.cc"],
+    deps = [
+        ":non_zero_calculator",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:calculator_runner",
+        "//mediapipe/framework:timestamp",
+        "//mediapipe/framework/port:gtest_main",
+        "//mediapipe/framework/port:status",
+        "//mediapipe/framework/tool:validate_type",
+    ],
+)
+
 cc_test(
     name = "mux_calculator_test",
     srcs = ["mux_calculator_test.cc"],
@@ -665,6 +696,18 @@ cc_library(
     alwayslink = 1,
 )
 
+cc_library(
+    name = "default_side_packet_calculator",
+    srcs = ["default_side_packet_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
+    ],
+    alwayslink = 1,
+)
+
 cc_library(
     name = "side_packet_to_stream_calculator",
     srcs = ["side_packet_to_stream_calculator.cc"],

mediapipe/calculators/core/default_side_packet_calculator.cc

@@ -0,0 +1,104 @@
+// Copyright 2019 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/port/ret_check.h"
+#include "mediapipe/framework/port/status.h"
+
+namespace mediapipe {
+
+namespace {
+
+constexpr char kOptionalValueTag[] = "OPTIONAL_VALUE";
+constexpr char kDefaultValueTag[] = "DEFAULT_VALUE";
+constexpr char kValueTag[] = "VALUE";
+
+}  // namespace
+
+// Outputs side packet default value if optional value is not provided.
+//
+// This calculator utilizes the fact that MediaPipe automatically removes
+// optional side packets of the calculator configuration (i.e. OPTIONAL_VALUE).
+// And if it happens - returns default value, otherwise - returns optional
+// value.
+//
+// Input:
+//   OPTIONAL_VALUE (optional) - AnyType (but same type as DEFAULT_VALUE)
+//     Optional side packet value that is outputted by the calculator as is if
+//     provided.
+//
+//   DEFAULT_VALUE - AnyType
+//     Default side pack value that is outputted by the calculator if
+//     OPTIONAL_VALUE is not provided.
+//
+// Output:
+//   VALUE - AnyType (but same type as DEFAULT_VALUE)
+//     Either OPTIONAL_VALUE (if provided) or DEFAULT_VALUE (otherwise).
+//
+// Usage example:
+//   node {
+//     calculator: "DefaultSidePacketCalculator"
+//     input_side_packet: "OPTIONAL_VALUE:segmentation_mask_enabled_optional"
+//     input_side_packet: "DEFAULT_VALUE:segmentation_mask_enabled_default"
+//     output_side_packet: "VALUE:segmentation_mask_enabled"
+//   }
+class DefaultSidePacketCalculator : public CalculatorBase {
+ public:
+  static absl::Status GetContract(CalculatorContract* cc);
+  absl::Status Open(CalculatorContext* cc) override;
+  absl::Status Process(CalculatorContext* cc) override;
+};
+REGISTER_CALCULATOR(DefaultSidePacketCalculator);
+
+absl::Status DefaultSidePacketCalculator::GetContract(CalculatorContract* cc) {
+  RET_CHECK(cc->InputSidePackets().HasTag(kDefaultValueTag))
+      << "Default value must be provided";
+  cc->InputSidePackets().Tag(kDefaultValueTag).SetAny();
+
+  // Optional input side packet can be unspecified. In this case MediaPipe will
+  // remove it from the calculator config.
+  if (cc->InputSidePackets().HasTag(kOptionalValueTag)) {
+    cc->InputSidePackets()
+        .Tag(kOptionalValueTag)
+        .SetSameAs(&cc->InputSidePackets().Tag(kDefaultValueTag))
+        .Optional();
+  }
+
+  RET_CHECK(cc->OutputSidePackets().HasTag(kValueTag));
+  cc->OutputSidePackets().Tag(kValueTag).SetSameAs(
+      &cc->InputSidePackets().Tag(kDefaultValueTag));
+
+  return absl::OkStatus();
+}
+
+absl::Status DefaultSidePacketCalculator::Open(CalculatorContext* cc) {
+  // If optional value is provided it is returned as the calculator output.
+  if (cc->InputSidePackets().HasTag(kOptionalValueTag)) {
+    auto& packet = cc->InputSidePackets().Tag(kOptionalValueTag);
+    cc->OutputSidePackets().Tag(kValueTag).Set(packet);
+    return absl::OkStatus();
+  }
+
+  // If no optional value
+  auto& packet = cc->InputSidePackets().Tag(kDefaultValueTag);
+  cc->OutputSidePackets().Tag(kValueTag).Set(packet);
+
+  return absl::OkStatus();
+}
+
+absl::Status DefaultSidePacketCalculator::Process(CalculatorContext* cc) {
+  return absl::OkStatus();
+}
+
+}  // namespace mediapipe

mediapipe/calculators/core/non_zero_calculator.cc

@@ -23,14 +23,26 @@ namespace api2 {
 class NonZeroCalculator : public Node {
  public:
   static constexpr Input<int>::SideFallback kIn{"INPUT"};
-  static constexpr Output<int> kOut{"OUTPUT"};
+  static constexpr Output<int>::Optional kOut{"OUTPUT"};
+  static constexpr Output<bool>::Optional kBooleanOut{"OUTPUT_BOOL"};
 
-  MEDIAPIPE_NODE_CONTRACT(kIn, kOut);
+  MEDIAPIPE_NODE_CONTRACT(kIn, kOut, kBooleanOut);
+
+  absl::Status UpdateContract(CalculatorContract* cc) {
+    RET_CHECK(kOut(cc).IsConnected() || kBooleanOut(cc).IsConnected())
+        << "At least one output stream is expected.";
+    return absl::OkStatus();
+  }
 
   absl::Status Process(CalculatorContext* cc) final {
     if (!kIn(cc).IsEmpty()) {
-      auto output = std::make_unique<int>((*kIn(cc) != 0) ? 1 : 0);
-      kOut(cc).Send(std::move(output));
+      bool isNonZero = *kIn(cc) != 0;
+      if (kOut(cc).IsConnected()) {
+        kOut(cc).Send(std::make_unique<int>(isNonZero ? 1 : 0));
+      }
+      if (kBooleanOut(cc).IsConnected()) {
+        kBooleanOut(cc).Send(std::make_unique<bool>(isNonZero));
+      }
     }
     return absl::OkStatus();
   }

mediapipe/calculators/core/non_zero_calculator_test.cc

@@ -0,0 +1,93 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_runner.h"
+#include "mediapipe/framework/port/canonical_errors.h"
+#include "mediapipe/framework/port/gmock.h"
+#include "mediapipe/framework/port/gtest.h"
+#include "mediapipe/framework/port/status.h"
+#include "mediapipe/framework/port/status_matchers.h"
+#include "mediapipe/framework/timestamp.h"
+#include "mediapipe/framework/tool/validate_type.h"
+
+namespace mediapipe {
+
+class NonZeroCalculatorTest : public ::testing::Test {
+ protected:
+  NonZeroCalculatorTest()
+      : runner_(
+            R"pb(
+              calculator: "NonZeroCalculator"
+              input_stream: "INPUT:input"
+              output_stream: "OUTPUT:output"
+              output_stream: "OUTPUT_BOOL:output_bool"
+            )pb") {}
+
+  void SetInput(const std::vector<int>& inputs) {
+    int timestamp = 0;
+    for (const auto input : inputs) {
+      runner_.MutableInputs()
+          ->Get("INPUT", 0)
+          .packets.push_back(MakePacket<int>(input).At(Timestamp(timestamp++)));
+    }
+  }
+
+  std::vector<int> GetOutput() {
+    std::vector<int> result;
+    for (const auto output : runner_.Outputs().Get("OUTPUT", 0).packets) {
+      result.push_back(output.Get<int>());
+    }
+    return result;
+  }
+
+  std::vector<bool> GetOutputBool() {
+    std::vector<bool> result;
+    for (const auto output : runner_.Outputs().Get("OUTPUT_BOOL", 0).packets) {
+      result.push_back(output.Get<bool>());
+    }
+    return result;
+  }
+
+  CalculatorRunner runner_;
+};
+
+TEST_F(NonZeroCalculatorTest, ProducesZeroOutputForZeroInput) {
+  SetInput({0});
+
+  MP_ASSERT_OK(runner_.Run());
+
+  EXPECT_THAT(GetOutput(), ::testing::ElementsAre(0));
+  EXPECT_THAT(GetOutputBool(), ::testing::ElementsAre(false));
+}
+
+TEST_F(NonZeroCalculatorTest, ProducesNonZeroOutputForNonZeroInput) {
+  SetInput({1, 2, 3, -4, 5});
+
+  MP_ASSERT_OK(runner_.Run());
+
+  EXPECT_THAT(GetOutput(), ::testing::ElementsAre(1, 1, 1, 1, 1));
+  EXPECT_THAT(GetOutputBool(),
+              ::testing::ElementsAre(true, true, true, true, true));
+}
+
+TEST_F(NonZeroCalculatorTest, SwitchesBetweenNonZeroAndZeroOutput) {
+  SetInput({1, 0, 3, 0, 5});
+  MP_ASSERT_OK(runner_.Run());
+  EXPECT_THAT(GetOutput(), ::testing::ElementsAre(1, 0, 1, 0, 1));
+  EXPECT_THAT(GetOutputBool(),
+              ::testing::ElementsAre(true, false, true, false, true));
+}
+
+}  // namespace mediapipe

mediapipe/calculators/image/BUILD

@@ -405,12 +405,44 @@ cc_library(
     alwayslink = 1,
 )
 
+mediapipe_proto_library(
+    name = "image_clone_calculator_proto",
+    srcs = ["image_clone_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
+cc_library(
+    name = "image_clone_calculator",
+    srcs = ["image_clone_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":image_clone_calculator_cc_proto",
+        "//mediapipe/framework/api2:node",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:image",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:status",
+    ] + select({
+        "//mediapipe/gpu:disable_gpu": [],
+        "//conditions:default": [
+            "//mediapipe/gpu:gl_calculator_helper",
+        ],
+    }),
+    alwayslink = 1,
+)
+
 cc_library(
     name = "image_properties_calculator",
     srcs = ["image_properties_calculator.cc"],
     visibility = ["//visibility:public"],
     deps = [
+        "//mediapipe/framework/api2:node",
         "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:image",
         "//mediapipe/framework/formats:image_frame",
         "//mediapipe/framework/port:ret_check",
         "//mediapipe/framework/port:status",

mediapipe/calculators/image/image_clone_calculator.cc

@@ -0,0 +1,125 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/calculators/image/image_clone_calculator.pb.h"
+#include "mediapipe/framework/api2/node.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/formats/image.h"
+#include "mediapipe/framework/port/status.h"
+
+#if !MEDIAPIPE_DISABLE_GPU
+#include "mediapipe/gpu/gl_calculator_helper.h"
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+namespace mediapipe {
+namespace api2 {
+
+#if MEDIAPIPE_DISABLE_GPU
+// Just a placeholder to not have to depend on mediapipe::GpuBuffer.
+using GpuBuffer = AnyType;
+#else
+using GpuBuffer = mediapipe::GpuBuffer;
+#endif  // MEDIAPIPE_DISABLE_GPU
+
+// Clones an input image and makes sure in the output clone the pixel data are
+// stored on the target storage (CPU vs GPU) specified in the calculator option.
+//
+// The clone shares ownership of the input pixel data on the existing storage.
+// If the target storage is diffrent from the existing one, then the data is
+// further copied there.
+//
+// Example usage:
+// node {
+//   calculator: "ImageCloneCalculator"
+//   input_stream: "input"
+//   output_stream: "output"
+//   options: {
+//     [mediapipe.ImageCloneCalculatorOptions.ext] {
+//       output_on_gpu: true
+//     }
+//   }
+// }
+class ImageCloneCalculator : public Node {
+ public:
+  static constexpr Input<Image> kIn{""};
+  static constexpr Output<Image> kOut{""};
+
+  MEDIAPIPE_NODE_CONTRACT(kIn, kOut);
+
+  static absl::Status UpdateContract(CalculatorContract* cc) {
+#if MEDIAPIPE_DISABLE_GPU
+    if (cc->Options<mediapipe::ImageCloneCalculatorOptions>().output_on_gpu()) {
+      return absl::UnimplementedError(
+          "GPU processing is disabled in build flags");
+    }
+#else
+    MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
+#endif  // MEDIAPIPE_DISABLE_GPU
+    return absl::OkStatus();
+  }
+
+  absl::Status Open(CalculatorContext* cc) override {
+    const auto& options = cc->Options<mediapipe::ImageCloneCalculatorOptions>();
+    output_on_gpu_ = options.output_on_gpu();
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
+#endif  // !MEDIAPIPE_DISABLE_GPU
+    return absl::OkStatus();
+  }
+
+  absl::Status Process(CalculatorContext* cc) override {
+    std::unique_ptr<Image> output;
+    const auto& input = *kIn(cc);
+    if (input.UsesGpu()) {
+#if !MEDIAPIPE_DISABLE_GPU
+      // Create an output Image that co-owns the underlying texture buffer as
+      // the input Image.
+      output = std::make_unique<Image>(input.GetGpuBuffer());
+#endif  // !MEDIAPIPE_DISABLE_GPU
+    } else {
+      // Make a copy of the input packet to co-own the input Image.
+      mediapipe::Packet* packet_copy_ptr =
+          new mediapipe::Packet(kIn(cc).packet());
+      // Create an output Image that (co-)owns a new ImageFrame that points to
+      // the same pixel data as the input Image and also owns the packet
+      // copy. As a result, the output Image indirectly co-owns the input
+      // Image. This ensures a correct life span of the shared pixel data.
+      output = std::make_unique<Image>(std::make_unique<mediapipe::ImageFrame>(
+          input.image_format(), input.width(), input.height(), input.step(),
+          const_cast<uint8*>(input.GetImageFrameSharedPtr()->PixelData()),
+          [packet_copy_ptr](uint8*) { delete packet_copy_ptr; }));
+    }
+
+    if (output_on_gpu_) {
+#if !MEDIAPIPE_DISABLE_GPU
+      gpu_helper_.RunInGlContext([&output]() { output->ConvertToGpu(); });
+#endif  // !MEDIAPIPE_DISABLE_GPU
+    } else {
+      output->ConvertToCpu();
+    }
+    kOut(cc).Send(std::move(output));
+
+    return absl::OkStatus();
+  }
+
+ private:
+  bool output_on_gpu_;
+#if !MEDIAPIPE_DISABLE_GPU
+  mediapipe::GlCalculatorHelper gpu_helper_;
+#endif  // !MEDIAPIPE_DISABLE_GPU
+};
+MEDIAPIPE_REGISTER_NODE(ImageCloneCalculator);
+
+}  // namespace api2
+}  // namespace mediapipe

mediapipe/calculators/image/image_clone_calculator.proto

@@ -0,0 +1,28 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto2";
+
+package mediapipe;
+
+import "mediapipe/framework/calculator.proto";
+
+message ImageCloneCalculatorOptions {
+  extend CalculatorOptions {
+    optional ImageCloneCalculatorOptions ext = 372781894;
+  }
+
+  // Whether the output clone should have pixel data already available on GPU.
+  optional bool output_on_gpu = 1 [default = false];
+}

mediapipe/calculators/image/image_cropping_calculator.cc

@@ -285,7 +285,7 @@ absl::Status ImageCroppingCalculator::RenderGpu(CalculatorContext* cc) {
 
   // Run cropping shader on GPU.
   {
-    gpu_helper_.BindFramebuffer(dst_tex);  // GL_TEXTURE0
+    gpu_helper_.BindFramebuffer(dst_tex);
 
     glActiveTexture(GL_TEXTURE1);
     glBindTexture(src_tex.target(), src_tex.name());

mediapipe/calculators/image/image_properties_calculator.cc

@@ -12,25 +12,32 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+#include "mediapipe/framework/api2/node.h"
 #include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/formats/image.h"
 #include "mediapipe/framework/formats/image_frame.h"
 
 #if !MEDIAPIPE_DISABLE_GPU
 #include "mediapipe/gpu/gpu_buffer.h"
 #endif  // !MEDIAPIPE_DISABLE_GPU
 
-namespace {
-constexpr char kImageFrameTag[] = "IMAGE";
-constexpr char kGpuBufferTag[] = "IMAGE_GPU";
-}  // namespace
-
 namespace mediapipe {
+namespace api2 {
+
+#if MEDIAPIPE_DISABLE_GPU
+// Just a placeholder to not have to depend on mediapipe::GpuBuffer.
+using GpuBuffer = AnyType;
+#else
+using GpuBuffer = mediapipe::GpuBuffer;
+#endif  // MEDIAPIPE_DISABLE_GPU
 
 // Extracts image properties from the input image and outputs the properties.
 // Currently only supports image size.
 // Input:
 //   One of the following:
-//   IMAGE: An ImageFrame
+//   IMAGE: An Image or ImageFrame (for backward compatibility with existing
+//          graphs that use IMAGE for ImageFrame input)
+//   IMAGE_CPU: An ImageFrame
 //   IMAGE_GPU: A GpuBuffer
 //
 // Output:
@@ -42,59 +49,64 @@ namespace mediapipe {
 //   input_stream: "IMAGE:image"
 //   output_stream: "SIZE:size"
 // }
-class ImagePropertiesCalculator : public CalculatorBase {
+class ImagePropertiesCalculator : public Node {
  public:
-  static absl::Status GetContract(CalculatorContract* cc) {
-    RET_CHECK(cc->Inputs().HasTag(kImageFrameTag) ^
-              cc->Inputs().HasTag(kGpuBufferTag));
-    if (cc->Inputs().HasTag(kImageFrameTag)) {
-      cc->Inputs().Tag(kImageFrameTag).Set<ImageFrame>();
-    }
-#if !MEDIAPIPE_DISABLE_GPU
-    if (cc->Inputs().HasTag(kGpuBufferTag)) {
-      cc->Inputs().Tag(kGpuBufferTag).Set<::mediapipe::GpuBuffer>();
-    }
-#endif  // !MEDIAPIPE_DISABLE_GPU
+  static constexpr Input<
+      OneOf<mediapipe::Image, mediapipe::ImageFrame>>::Optional kIn{"IMAGE"};
+  // IMAGE_CPU, dedicated to ImageFrame input, is only needed in some top-level
+  // graphs for the Python Solution APIs to figure out the type of input stream
+  // without running into ambiguities from IMAGE.
+  // TODO: Remove IMAGE_CPU once Python Solution APIs adopt Image.
+  static constexpr Input<mediapipe::ImageFrame>::Optional kInCpu{"IMAGE_CPU"};
+  static constexpr Input<GpuBuffer>::Optional kInGpu{"IMAGE_GPU"};
+  static constexpr Output<std::pair<int, int>> kOut{"SIZE"};
 
-    if (cc->Outputs().HasTag("SIZE")) {
-      cc->Outputs().Tag("SIZE").Set<std::pair<int, int>>();
-    }
+  MEDIAPIPE_NODE_CONTRACT(kIn, kInCpu, kInGpu, kOut);
 
-    return absl::OkStatus();
-  }
+  static absl::Status UpdateContract(CalculatorContract* cc) {
+    RET_CHECK_EQ(kIn(cc).IsConnected() + kInCpu(cc).IsConnected() +
+                     kInGpu(cc).IsConnected(),
+                 1)
+        << "One and only one of IMAGE, IMAGE_CPU and IMAGE_GPU input is "
+           "expected.";
 
-  absl::Status Open(CalculatorContext* cc) override {
-    cc->SetOffset(TimestampDiff(0));
     return absl::OkStatus();
   }
 
   absl::Status Process(CalculatorContext* cc) override {
-    int width;
-    int height;
+    std::pair<int, int> size;
 
-    if (cc->Inputs().HasTag(kImageFrameTag) &&
-        !cc->Inputs().Tag(kImageFrameTag).IsEmpty()) {
-      const auto& image = cc->Inputs().Tag(kImageFrameTag).Get<ImageFrame>();
-      width = image.Width();
-      height = image.Height();
+    if (kIn(cc).IsConnected()) {
+      kIn(cc).Visit(
+          [&size](const mediapipe::Image& value) {
+            size.first = value.width();
+            size.second = value.height();
+          },
+          [&size](const mediapipe::ImageFrame& value) {
+            size.first = value.Width();
+            size.second = value.Height();
+          });
+    }
+    if (kInCpu(cc).IsConnected()) {
+      const auto& image = *kInCpu(cc);
+      size.first = image.Width();
+      size.second = image.Height();
     }
 #if !MEDIAPIPE_DISABLE_GPU
-    if (cc->Inputs().HasTag(kGpuBufferTag) &&
-        !cc->Inputs().Tag(kGpuBufferTag).IsEmpty()) {
-      const auto& image =
-          cc->Inputs().Tag(kGpuBufferTag).Get<mediapipe::GpuBuffer>();
-      width = image.width();
-      height = image.height();
+    if (kInGpu(cc).IsConnected()) {
+      const auto& image = *kInGpu(cc);
+      size.first = image.width();
+      size.second = image.height();
     }
 #endif  // !MEDIAPIPE_DISABLE_GPU
 
-    cc->Outputs().Tag("SIZE").AddPacket(
-        MakePacket<std::pair<int, int>>(width, height)
-            .At(cc->InputTimestamp()));
+    kOut(cc).Send(size);
 
     return absl::OkStatus();
   }
 };
-REGISTER_CALCULATOR(ImagePropertiesCalculator);
 
+MEDIAPIPE_REGISTER_NODE(ImagePropertiesCalculator);
+
+}  // namespace api2
 }  // namespace mediapipe

mediapipe/calculators/image/image_transformation_calculator.cc

@@ -546,7 +546,7 @@ absl::Status ImageTransformationCalculator::RenderGpu(CalculatorContext* cc) {
   auto dst = gpu_helper_.CreateDestinationTexture(output_width, output_height,
                                                   input.format());
 
-  gpu_helper_.BindFramebuffer(dst);  // GL_TEXTURE0
+  gpu_helper_.BindFramebuffer(dst);
   glActiveTexture(GL_TEXTURE1);
   glBindTexture(src1.target(), src1.name());
 

mediapipe/calculators/image/recolor_calculator.cc

@@ -37,6 +37,22 @@ constexpr char kImageFrameTag[] = "IMAGE";
 constexpr char kMaskCpuTag[] = "MASK";
 constexpr char kGpuBufferTag[] = "IMAGE_GPU";
 constexpr char kMaskGpuTag[] = "MASK_GPU";
+
+inline cv::Vec3b Blend(const cv::Vec3b& color1, const cv::Vec3b& color2,
+                       float weight, int invert_mask,
+                       int adjust_with_luminance) {
+  weight = (1 - invert_mask) * weight + invert_mask * (1.0f - weight);
+
+  float luminance =
+      (1 - adjust_with_luminance) * 1.0f +
+      adjust_with_luminance *
+          (color1[0] * 0.299 + color1[1] * 0.587 + color1[2] * 0.114) / 255;
+
+  float mix_value = weight * luminance;
+
+  return color1 * (1.0 - mix_value) + color2 * mix_value;
+}
+
 }  // namespace
 
 namespace mediapipe {
@@ -44,15 +60,14 @@ namespace mediapipe {
 // A calculator to recolor a masked area of an image to a specified color.
 //
 // A mask image is used to specify where to overlay a user defined color.
-// The luminance of the input image is used to adjust the blending weight,
-// to help preserve image textures.
 //
 // Inputs:
 //   One of the following IMAGE tags:
-//   IMAGE: An ImageFrame input image, RGB or RGBA.
+//   IMAGE: An ImageFrame input image in ImageFormat::SRGB.
 //   IMAGE_GPU: A GpuBuffer input image, RGBA.
 //   One of the following MASK tags:
-//   MASK: An ImageFrame input mask, Gray, RGB or RGBA.
+//   MASK: An ImageFrame input mask in ImageFormat::GRAY8, SRGB, SRGBA, or
+//         VEC32F1
 //   MASK_GPU: A GpuBuffer input mask, RGBA.
 // Output:
 //   One of the following IMAGE tags:
@@ -98,10 +113,12 @@ class RecolorCalculator : public CalculatorBase {
   void GlRender();
 
   bool initialized_ = false;
-  std::vector<float> color_;
+  std::vector<uint8> color_;
   mediapipe::RecolorCalculatorOptions::MaskChannel mask_channel_;
 
   bool use_gpu_ = false;
+  bool invert_mask_ = false;
+  bool adjust_with_luminance_ = false;
 #if !MEDIAPIPE_DISABLE_GPU
   mediapipe::GlCalculatorHelper gpu_helper_;
   GLuint program_ = 0;
@@ -209,6 +226,9 @@ absl::Status RecolorCalculator::Close(CalculatorContext* cc) {
 
 absl::Status RecolorCalculator::RenderCpu(CalculatorContext* cc) {
   if (cc->Inputs().Tag(kMaskCpuTag).IsEmpty()) {
+    cc->Outputs()
+        .Tag(kImageFrameTag)
+        .AddPacket(cc->Inputs().Tag(kImageFrameTag).Value());
     return absl::OkStatus();
   }
   // Get inputs and setup output.
@@ -230,11 +250,15 @@ absl::Status RecolorCalculator::RenderCpu(CalculatorContext* cc) {
   }
   cv::Mat mask_full;
   cv::resize(mask_mat, mask_full, input_mat.size());
+  const cv::Vec3b recolor = {color_[0], color_[1], color_[2]};
 
   auto output_img = absl::make_unique<ImageFrame>(
       input_img.Format(), input_mat.cols, input_mat.rows);
   cv::Mat output_mat = mediapipe::formats::MatView(output_img.get());
 
+  const int invert_mask = invert_mask_ ? 1 : 0;
+  const int adjust_with_luminance = adjust_with_luminance_ ? 1 : 0;
+
   // From GPU shader:
   /*
       vec4 weight = texture2D(mask, sample_coordinate);
@@ -246,18 +270,23 @@ absl::Status RecolorCalculator::RenderCpu(CalculatorContext* cc) {
 
       fragColor = mix(color1, color2, mix_value);
   */
-  for (int i = 0; i < output_mat.rows; ++i) {
-    for (int j = 0; j < output_mat.cols; ++j) {
-      float weight = mask_full.at<uchar>(i, j) * (1.0 / 255.0);
-      cv::Vec3f color1 = input_mat.at<cv::Vec3b>(i, j);
-      cv::Vec3f color2 = {color_[0], color_[1], color_[2]};
-
-      float luminance =
-          (color1[0] * 0.299 + color1[1] * 0.587 + color1[2] * 0.114) / 255;
-      float mix_value = weight * luminance;
-
-      cv::Vec3b mix_color = color1 * (1.0 - mix_value) + color2 * mix_value;
-      output_mat.at<cv::Vec3b>(i, j) = mix_color;
+  if (mask_img.Format() == ImageFormat::VEC32F1) {
+    for (int i = 0; i < output_mat.rows; ++i) {
+      for (int j = 0; j < output_mat.cols; ++j) {
+        const float weight = mask_full.at<float>(i, j);
+        output_mat.at<cv::Vec3b>(i, j) =
+            Blend(input_mat.at<cv::Vec3b>(i, j), recolor, weight, invert_mask,
+                  adjust_with_luminance);
+      }
+    }
+  } else {
+    for (int i = 0; i < output_mat.rows; ++i) {
+      for (int j = 0; j < output_mat.cols; ++j) {
+        const float weight = mask_full.at<uchar>(i, j) * (1.0 / 255.0);
+        output_mat.at<cv::Vec3b>(i, j) =
+            Blend(input_mat.at<cv::Vec3b>(i, j), recolor, weight, invert_mask,
+                  adjust_with_luminance);
+      }
     }
   }
 
@@ -270,6 +299,9 @@ absl::Status RecolorCalculator::RenderCpu(CalculatorContext* cc) {
 
 absl::Status RecolorCalculator::RenderGpu(CalculatorContext* cc) {
   if (cc->Inputs().Tag(kMaskGpuTag).IsEmpty()) {
+    cc->Outputs()
+        .Tag(kGpuBufferTag)
+        .AddPacket(cc->Inputs().Tag(kGpuBufferTag).Value());
     return absl::OkStatus();
   }
 #if !MEDIAPIPE_DISABLE_GPU
@@ -287,7 +319,7 @@ absl::Status RecolorCalculator::RenderGpu(CalculatorContext* cc) {
 
   // Run recolor shader on GPU.
   {
-    gpu_helper_.BindFramebuffer(dst_tex);  // GL_TEXTURE0
+    gpu_helper_.BindFramebuffer(dst_tex);
 
     glActiveTexture(GL_TEXTURE1);
     glBindTexture(img_tex.target(), img_tex.name());
@@ -379,6 +411,9 @@ absl::Status RecolorCalculator::LoadOptions(CalculatorContext* cc) {
   color_.push_back(options.color().g());
   color_.push_back(options.color().b());
 
+  invert_mask_ = options.invert_mask();
+  adjust_with_luminance_ = options.adjust_with_luminance();
+
   return absl::OkStatus();
 }
 
@@ -429,13 +464,20 @@ absl::Status RecolorCalculator::InitGpu(CalculatorContext* cc) {
     uniform sampler2D frame;
     uniform sampler2D mask;
     uniform vec3 recolor;
+    uniform float invert_mask;
+    uniform float adjust_with_luminance;
 
     void main() {
       vec4 weight = texture2D(mask, sample_coordinate);
       vec4 color1 = texture2D(frame, sample_coordinate);
       vec4 color2 = vec4(recolor, 1.0);
 
-      float luminance = dot(color1.rgb, vec3(0.299, 0.587, 0.114));
+      weight = mix(weight, 1.0 - weight, invert_mask);
+
+      float luminance = mix(1.0,
+                            dot(color1.rgb, vec3(0.299, 0.587, 0.114)),
+                            adjust_with_luminance);
+
       float mix_value = weight.MASK_COMPONENT * luminance;
 
       fragColor = mix(color1, color2, mix_value);
@@ -452,6 +494,10 @@ absl::Status RecolorCalculator::InitGpu(CalculatorContext* cc) {
   glUniform1i(glGetUniformLocation(program_, "mask"), 2);
   glUniform3f(glGetUniformLocation(program_, "recolor"), color_[0] / 255.0,
               color_[1] / 255.0, color_[2] / 255.0);
+  glUniform1f(glGetUniformLocation(program_, "invert_mask"),
+              invert_mask_ ? 1.0f : 0.0f);
+  glUniform1f(glGetUniformLocation(program_, "adjust_with_luminance"),
+              adjust_with_luminance_ ? 1.0f : 0.0f);
 #endif  // !MEDIAPIPE_DISABLE_GPU
 
   return absl::OkStatus();

mediapipe/calculators/image/recolor_calculator.proto

@@ -36,4 +36,11 @@ message RecolorCalculatorOptions {
   // Color to blend into input image where mask is > 0.
   // The blending is based on the input image luminosity.
   optional Color color = 2;
+
+  // Swap the meaning of mask values for foreground/background.
+  optional bool invert_mask = 3 [default = false];
+
+  // Whether to use the luminance of the input image to further adjust the
+  // blending weight, to help preserve image textures.
+  optional bool adjust_with_luminance = 4 [default = true];
 }

mediapipe/calculators/image/set_alpha_calculator.cc

@@ -323,7 +323,7 @@ absl::Status SetAlphaCalculator::RenderGpu(CalculatorContext* cc) {
     const auto& alpha_mask =
         cc->Inputs().Tag(kInputAlphaTagGpu).Get<mediapipe::GpuBuffer>();
     auto alpha_texture = gpu_helper_.CreateSourceTexture(alpha_mask);
-    gpu_helper_.BindFramebuffer(output_texture);  // GL_TEXTURE0
+    gpu_helper_.BindFramebuffer(output_texture);
     glActiveTexture(GL_TEXTURE1);
     glBindTexture(GL_TEXTURE_2D, input_texture.name());
     glActiveTexture(GL_TEXTURE2);
@@ -335,7 +335,7 @@ absl::Status SetAlphaCalculator::RenderGpu(CalculatorContext* cc) {
     glBindTexture(GL_TEXTURE_2D, 0);
     alpha_texture.Release();
   } else {
-    gpu_helper_.BindFramebuffer(output_texture);  // GL_TEXTURE0
+    gpu_helper_.BindFramebuffer(output_texture);
     glActiveTexture(GL_TEXTURE1);
     glBindTexture(GL_TEXTURE_2D, input_texture.name());
     GlRender(cc);  // use value from options

mediapipe/calculators/tensor/BUILD

@@ -490,6 +490,7 @@ cc_library(
         "//mediapipe/framework/port:statusor",
         "//mediapipe/framework:calculator_framework",
         "//mediapipe/framework:port",
+        "//mediapipe/gpu:gpu_origin_cc_proto",
     ] + select({
         "//mediapipe/gpu:disable_gpu": [],
         "//conditions:default": [":image_to_tensor_calculator_gpu_deps"],
@@ -526,6 +527,7 @@ mediapipe_proto_library(
     deps = [
         "//mediapipe/framework:calculator_options_proto",
         "//mediapipe/framework:calculator_proto",
+        "//mediapipe/gpu:gpu_origin_proto",
     ],
 )
 
@@ -583,6 +585,7 @@ cc_library(
         ],
         "//conditions:default": [],
     }),
+    visibility = ["//visibility:public"],
     deps = [
         ":image_to_tensor_utils",
         "//mediapipe/framework/formats:image",
@@ -750,3 +753,76 @@ cc_test(
         "//mediapipe/framework/port:gtest_main",
     ],
 )
+
+# Copied from /mediapipe/calculators/tflite/BUILD
+selects.config_setting_group(
+    name = "gpu_inference_disabled",
+    match_any = [
+        "//mediapipe/gpu:disable_gpu",
+    ],
+)
+
+mediapipe_proto_library(
+    name = "tensors_to_segmentation_calculator_proto",
+    srcs = ["tensors_to_segmentation_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+        "//mediapipe/gpu:gpu_origin_proto",
+    ],
+)
+
+cc_library(
+    name = "tensors_to_segmentation_calculator",
+    srcs = ["tensors_to_segmentation_calculator.cc"],
+    copts = select({
+        "//mediapipe:apple": [
+            "-x objective-c++",
+            "-fobjc-arc",  # enable reference-counting
+        ],
+        "//conditions:default": [],
+    }),
+    visibility = ["//visibility:public"],
+    deps = [
+        ":tensors_to_segmentation_calculator_cc_proto",
+        "@com_google_absl//absl/strings:str_format",
+        "@com_google_absl//absl/strings",
+        "@com_google_absl//absl/types:span",
+        "//mediapipe/framework/formats:image",
+        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_opencv",
+        "//mediapipe/framework/formats:tensor",
+        "//mediapipe/framework/port:opencv_imgproc",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework:calculator_context",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:port",
+        "//mediapipe/util:resource_util",
+        "@org_tensorflow//tensorflow/lite:framework",
+        "//mediapipe/gpu:gpu_origin_cc_proto",
+        "//mediapipe/framework/port:statusor",
+    ] + selects.with_or({
+        "//mediapipe/gpu:disable_gpu": [],
+        "//conditions:default": [
+            "//mediapipe/gpu:gl_calculator_helper",
+            "//mediapipe/gpu:gl_simple_shaders",
+            "//mediapipe/gpu:gpu_buffer",
+            "//mediapipe/gpu:shader_util",
+        ],
+    }) + selects.with_or({
+        ":gpu_inference_disabled": [],
+        "//mediapipe:ios": [
+            "//mediapipe/gpu:MPPMetalUtil",
+            "//mediapipe/gpu:MPPMetalHelper",
+        ],
+        "//conditions:default": [
+            "@org_tensorflow//tensorflow/lite/delegates/gpu:gl_delegate",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_program",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_shader",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_texture",
+            "@org_tensorflow//tensorflow/lite/delegates/gpu/gl/converters:util",
+        ],
+    }),
+    alwayslink = 1,
+)

mediapipe/calculators/tensor/image_to_tensor_calculator.cc

@@ -31,6 +31,7 @@
 #include "mediapipe/framework/port/ret_check.h"
 #include "mediapipe/framework/port/status.h"
 #include "mediapipe/framework/port/statusor.h"
+#include "mediapipe/gpu/gpu_origin.pb.h"
 
 #if !MEDIAPIPE_DISABLE_GPU
 #include "mediapipe/gpu/gpu_buffer.h"
@@ -236,7 +237,7 @@ class ImageToTensorCalculator : public Node {
   }
 
  private:
-  bool DoesInputStartAtBottom() {
+  bool DoesGpuInputStartAtBottom() {
     return options_.gpu_origin() != mediapipe::GpuOrigin_Mode_TOP_LEFT;
   }
 
@@ -290,11 +291,11 @@ class ImageToTensorCalculator : public Node {
 #elif MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
         ASSIGN_OR_RETURN(gpu_converter_,
                          CreateImageToGlBufferTensorConverter(
-                             cc, DoesInputStartAtBottom(), GetBorderMode()));
+                             cc, DoesGpuInputStartAtBottom(), GetBorderMode()));
 #else
         ASSIGN_OR_RETURN(gpu_converter_,
                          CreateImageToGlTextureTensorConverter(
-                             cc, DoesInputStartAtBottom(), GetBorderMode()));
+                             cc, DoesGpuInputStartAtBottom(), GetBorderMode()));
 #endif  // MEDIAPIPE_METAL_ENABLED
 #endif  // !MEDIAPIPE_DISABLE_GPU
       }

mediapipe/calculators/tensor/image_to_tensor_calculator.proto

@@ -17,20 +17,7 @@ syntax = "proto2";
 package mediapipe;
 
 import "mediapipe/framework/calculator.proto";
-
-message GpuOrigin {
-  enum Mode {
-    DEFAULT = 0;
-
-    // OpenGL: bottom-left origin
-    // Metal : top-left origin
-    CONVENTIONAL = 1;
-
-    // OpenGL: top-left origin
-    // Metal : top-left origin
-    TOP_LEFT = 2;
-  }
-}
+import "mediapipe/gpu/gpu_origin.proto";
 
 message ImageToTensorCalculatorOptions {
   extend mediapipe.CalculatorOptions {

mediapipe/calculators/tensor/image_to_tensor_converter_gl_buffer.cc

@@ -312,7 +312,7 @@ class GlProcessor : public ImageToTensorConverter {
       return absl::OkStatus();
     }));
 
-    return tensor;
+    return std::move(tensor);
   }
 
   ~GlProcessor() override {

mediapipe/calculators/tensor/image_to_tensor_converter_metal.cc

@@ -383,7 +383,7 @@ class MetalProcessor : public ImageToTensorConverter {
           tflite::gpu::HW(output_dims.height, output_dims.width),
           command_buffer, buffer_view.buffer()));
       [command_buffer commit];
-      return tensor;
+      return std::move(tensor);
     }
   }
 

mediapipe/calculators/tensor/image_to_tensor_converter_opencv.cc

@@ -103,7 +103,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
         GetValueRangeTransformation(kInputImageRangeMin, kInputImageRangeMax,
                                     range_min, range_max));
     transformed.convertTo(dst, CV_32FC3, transform.scale, transform.offset);
-    return tensor;
+    return std::move(tensor);
   }
 
  private:

mediapipe/calculators/tensor/inference_calculator_gl.cc

@@ -317,7 +317,8 @@ absl::Status InferenceCalculatorGlImpl::LoadModel(CalculatorContext* cc) {
 absl::Status InferenceCalculatorGlImpl::LoadDelegate(CalculatorContext* cc) {
   // Configure and create the delegate.
   TfLiteGpuDelegateOptions options = TfLiteGpuDelegateOptionsDefault();
-  options.compile_options.precision_loss_allowed = 1;
+  options.compile_options.precision_loss_allowed =
+      allow_precision_loss_ ? 1 : 0;
   options.compile_options.preferred_gl_object_type =
       TFLITE_GL_OBJECT_TYPE_FASTEST;
   options.compile_options.dynamic_batch_enabled = 0;

mediapipe/calculators/tensor/inference_calculator_metal.cc

@@ -97,6 +97,7 @@ class InferenceCalculatorMetalImpl
   Packet<TfLiteModelPtr> model_packet_;
   std::unique_ptr<tflite::Interpreter> interpreter_;
   TfLiteDelegatePtr delegate_;
+  bool allow_precision_loss_ = false;
 
 #if MEDIAPIPE_TFLITE_METAL_INFERENCE
   MPPMetalHelper* gpu_helper_ = nullptr;
@@ -122,6 +123,9 @@ absl::Status InferenceCalculatorMetalImpl::UpdateContract(
 }
 
 absl::Status InferenceCalculatorMetalImpl::Open(CalculatorContext* cc) {
+  const auto& options = cc->Options<::mediapipe::InferenceCalculatorOptions>();
+  allow_precision_loss_ = options.delegate().gpu().allow_precision_loss();
+
   MP_RETURN_IF_ERROR(LoadModel(cc));
 
   gpu_helper_ = [[MPPMetalHelper alloc] initWithCalculatorContext:cc];
@@ -222,7 +226,7 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
 
   // Configure and create the delegate.
   TFLGpuDelegateOptions options;
-  options.allow_precision_loss = true;
+  options.allow_precision_loss = allow_precision_loss_;
   options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeDoNotWait;
   delegate_ =
       TfLiteDelegatePtr(TFLGpuDelegateCreate(&options), &TFLGpuDelegateDelete);
@@ -239,7 +243,9 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
                           tensor->dims->data + tensor->dims->size};
     dims.back() = RoundUp(dims.back(), 4);
     gpu_buffers_in_.emplace_back(absl::make_unique<Tensor>(
-        Tensor::ElementType::kFloat16, Tensor::Shape{dims}));
+        allow_precision_loss_ ? Tensor::ElementType::kFloat16
+                              : Tensor::ElementType::kFloat32,
+        Tensor::Shape{dims}));
     auto buffer_view =
         gpu_buffers_in_[i]->GetMtlBufferWriteView(gpu_helper_.mtlDevice);
     RET_CHECK_EQ(TFLGpuDelegateBindMetalBufferToTensor(
@@ -261,7 +267,9 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
     output_shapes_[i] = {dims};
     dims.back() = RoundUp(dims.back(), 4);
     gpu_buffers_out_.emplace_back(absl::make_unique<Tensor>(
-        Tensor::ElementType::kFloat16, Tensor::Shape{dims}));
+        allow_precision_loss_ ? Tensor::ElementType::kFloat16
+                              : Tensor::ElementType::kFloat32,
+        Tensor::Shape{dims}));
     RET_CHECK_EQ(TFLGpuDelegateBindMetalBufferToTensor(
                      delegate_.get(), output_indices[i],
                      gpu_buffers_out_[i]
@@ -271,17 +279,19 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
   }
 
   // Create converter for GPU input.
-  converter_to_BPHWC4_ = [[TFLBufferConvert alloc] initWithDevice:device
-                                                        isFloat16:true
-                                                  convertToPBHWC4:true];
+  converter_to_BPHWC4_ =
+      [[TFLBufferConvert alloc] initWithDevice:device
+                                     isFloat16:allow_precision_loss_
+                               convertToPBHWC4:true];
   if (converter_to_BPHWC4_ == nil) {
     return mediapipe::InternalError(
         "Error initializating input buffer converter");
   }
   // Create converter for GPU output.
-  converter_from_BPHWC4_ = [[TFLBufferConvert alloc] initWithDevice:device
-                                                          isFloat16:true
-                                                    convertToPBHWC4:false];
+  converter_from_BPHWC4_ =
+      [[TFLBufferConvert alloc] initWithDevice:device
+                                     isFloat16:allow_precision_loss_
+                               convertToPBHWC4:false];
   if (converter_from_BPHWC4_ == nil) {
     return absl::InternalError("Error initializating output buffer converter");
   }

mediapipe/calculators/tensor/tensors_to_classification_calculator.cc

@@ -89,7 +89,8 @@ absl::Status TensorsToClassificationCalculator::Open(CalculatorContext* cc) {
     ASSIGN_OR_RETURN(string_path,
                      PathToResourceAsFile(options_.label_map_path()));
     std::string label_map_string;
-    MP_RETURN_IF_ERROR(file::GetContents(string_path, &label_map_string));
+    MP_RETURN_IF_ERROR(
+        mediapipe::GetResourceContents(string_path, &label_map_string));
 
     std::istringstream stream(label_map_string);
     std::string line;
@@ -98,6 +99,14 @@ absl::Status TensorsToClassificationCalculator::Open(CalculatorContext* cc) {
       label_map_[i++] = line;
     }
     label_map_loaded_ = true;
+  } else if (options_.has_label_map()) {
+    for (int i = 0; i < options_.label_map().entries_size(); ++i) {
+      const auto& entry = options_.label_map().entries(i);
+      RET_CHECK(!label_map_.contains(entry.id()))
+          << "Duplicate id found: " << entry.id();
+      label_map_[entry.id()] = entry.label();
+    }
+    label_map_loaded_ = true;
   }
 
   return absl::OkStatus();

mediapipe/calculators/tensor/tensors_to_classification_calculator.proto

@@ -25,6 +25,14 @@ message TensorsToClassificationCalculatorOptions {
     optional TensorsToClassificationCalculatorOptions ext = 335742638;
   }
 
+  message LabelMap {
+    message Entry {
+      optional int32 id = 1;
+      optional string label = 2;
+    }
+    repeated Entry entries = 1;
+  }
+
   // Score threshold for perserving the class.
   optional float min_score_threshold = 1;
   // Number of highest scoring labels to output.  If top_k is not positive then
@@ -32,6 +40,10 @@ message TensorsToClassificationCalculatorOptions {
   optional int32 top_k = 2;
   // Path to a label map file for getting the actual name of class ids.
   optional string label_map_path = 3;
+  // Label map. (Can be used instead of label_map_path.)
+  // NOTE: "label_map_path", if specified, takes precedence over "label_map".
+  optional LabelMap label_map = 5;
+
   // Whether the input is a single float for binary classification.
   // When true, only a single float is expected in the input tensor and the
   // label map, if provided, is expected to have exactly two labels.

mediapipe/calculators/tensor/tensors_to_classification_calculator_test.cc

@@ -115,6 +115,41 @@ TEST_F(TensorsToClassificationCalculatorTest, CorrectOutputWithLabelMapPath) {
   }
 }
 
+TEST_F(TensorsToClassificationCalculatorTest, CorrectOutputWithLabelMap) {
+  mediapipe::CalculatorRunner runner(ParseTextProtoOrDie<Node>(R"pb(
+    calculator: "TensorsToClassificationCalculator"
+    input_stream: "TENSORS:tensors"
+    output_stream: "CLASSIFICATIONS:classifications"
+    options {
+      [mediapipe.TensorsToClassificationCalculatorOptions.ext] {
+        label_map {
+          entries { id: 0, label: "ClassA" }
+          entries { id: 1, label: "ClassB" }
+          entries { id: 2, label: "ClassC" }
+        }
+      }
+    }
+  )pb"));
+
+  BuildGraph(&runner, {0, 0.5, 1});
+  MP_ASSERT_OK(runner.Run());
+
+  const auto& output_packets_ = runner.Outputs().Tag("CLASSIFICATIONS").packets;
+
+  EXPECT_EQ(1, output_packets_.size());
+
+  const auto& classification_list =
+      output_packets_[0].Get<ClassificationList>();
+  EXPECT_EQ(3, classification_list.classification_size());
+
+  // Verify that the label field is set.
+  for (int i = 0; i < classification_list.classification_size(); ++i) {
+    EXPECT_EQ(i, classification_list.classification(i).index());
+    EXPECT_EQ(i * 0.5, classification_list.classification(i).score());
+    ASSERT_TRUE(classification_list.classification(i).has_label());
+  }
+}
+
 TEST_F(TensorsToClassificationCalculatorTest,
        CorrectOutputWithLabelMinScoreThreshold) {
   mediapipe::CalculatorRunner runner(ParseTextProtoOrDie<Node>(R"pb(

mediapipe/calculators/tensor/tensors_to_detections_calculator.cc

@@ -105,6 +105,15 @@ void ConvertAnchorsToRawValues(const std::vector<Anchor>& anchors,
 //            for anchors (e.g. for SSD models) depend on the outputs of the
 //            detection model. The size of anchor tensor must be (num_boxes *
 //            4).
+//
+// Input side packet:
+//  ANCHORS (optional) - The anchors used for decoding the bounding boxes, as a
+//      vector of `Anchor` protos. Not required if post-processing is built-in
+//      the model.
+//  IGNORE_CLASSES (optional) - The list of class ids that should be ignored, as
+//      a vector of integers. It overrides the corresponding field in the
+//      calculator options.
+//
 // Output:
 //  DETECTIONS - Result MediaPipe detections.
 //
@@ -132,8 +141,11 @@ class TensorsToDetectionsCalculator : public Node {
   static constexpr Input<std::vector<Tensor>> kInTensors{"TENSORS"};
   static constexpr SideInput<std::vector<Anchor>>::Optional kInAnchors{
       "ANCHORS"};
+  static constexpr SideInput<std::vector<int>>::Optional kSideInIgnoreClasses{
+      "IGNORE_CLASSES"};
   static constexpr Output<std::vector<Detection>> kOutDetections{"DETECTIONS"};
-  MEDIAPIPE_NODE_CONTRACT(kInTensors, kInAnchors, kOutDetections);
+  MEDIAPIPE_NODE_CONTRACT(kInTensors, kInAnchors, kSideInIgnoreClasses,
+                          kOutDetections);
   static absl::Status UpdateContract(CalculatorContract* cc);
 
   absl::Status Open(CalculatorContext* cc) override;
@@ -566,8 +578,15 @@ absl::Status TensorsToDetectionsCalculator::LoadOptions(CalculatorContext* cc) {
                kNumCoordsPerBox,
            num_coords_);
 
-  for (int i = 0; i < options_.ignore_classes_size(); ++i) {
-    ignore_classes_.insert(options_.ignore_classes(i));
+  if (kSideInIgnoreClasses(cc).IsConnected()) {
+    RET_CHECK(!kSideInIgnoreClasses(cc).IsEmpty());
+    for (int ignore_class : *kSideInIgnoreClasses(cc)) {
+      ignore_classes_.insert(ignore_class);
+    }
+  } else {
+    for (int i = 0; i < options_.ignore_classes_size(); ++i) {
+      ignore_classes_.insert(options_.ignore_classes(i));
+    }
   }
 
   return absl::OkStatus();

mediapipe/calculators/tensor/tensors_to_detections_calculator.proto

@@ -56,7 +56,7 @@ message TensorsToDetectionsCalculatorOptions {
   // [x_center, y_center, w, h].
   optional bool reverse_output_order = 14 [default = false];
   // The ids of classes that should be ignored during decoding the score for
-  // each predicted box.
+  // each predicted box. Can be overridden with IGNORE_CLASSES side packet.
   repeated int32 ignore_classes = 8;
 
   optional bool sigmoid_score = 15 [default = false];

mediapipe/calculators/tensor/tensors_to_segmentation_calculator.cc

@@ -0,0 +1,885 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <vector>
+
+#include "absl/strings/str_format.h"
+#include "absl/types/span.h"
+#include "mediapipe/calculators/tensor/tensors_to_segmentation_calculator.pb.h"
+#include "mediapipe/framework/calculator_context.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/formats/image.h"
+#include "mediapipe/framework/formats/image_opencv.h"
+#include "mediapipe/framework/formats/tensor.h"
+#include "mediapipe/framework/port.h"
+#include "mediapipe/framework/port/opencv_imgproc_inc.h"
+#include "mediapipe/framework/port/ret_check.h"
+#include "mediapipe/framework/port/statusor.h"
+#include "mediapipe/gpu/gpu_origin.pb.h"
+#include "mediapipe/util/resource_util.h"
+#include "tensorflow/lite/interpreter.h"
+
+#if !MEDIAPIPE_DISABLE_GPU
+#include "mediapipe/gpu/gl_calculator_helper.h"
+#include "mediapipe/gpu/gl_simple_shaders.h"
+#include "mediapipe/gpu/gpu_buffer.h"
+#include "mediapipe/gpu/shader_util.h"
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+#include "tensorflow/lite/delegates/gpu/gl/converters/util.h"
+#include "tensorflow/lite/delegates/gpu/gl/gl_program.h"
+#include "tensorflow/lite/delegates/gpu/gl/gl_shader.h"
+#include "tensorflow/lite/delegates/gpu/gl/gl_texture.h"
+#include "tensorflow/lite/delegates/gpu/gl_delegate.h"
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+#if MEDIAPIPE_METAL_ENABLED
+#import <CoreVideo/CoreVideo.h>
+#import <Metal/Metal.h>
+#import <MetalKit/MetalKit.h>
+
+#import "mediapipe/gpu/MPPMetalHelper.h"
+#include "mediapipe/gpu/MPPMetalUtil.h"
+#endif  // MEDIAPIPE_METAL_ENABLED
+
+namespace {
+constexpr int kWorkgroupSize = 8;  // Block size for GPU shader.
+enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
+
+// Commonly used to compute the number of blocks to launch in a kernel.
+int NumGroups(const int size, const int group_size) {  // NOLINT
+  return (size + group_size - 1) / group_size;
+}
+
+bool CanUseGpu() {
+#if !MEDIAPIPE_DISABLE_GPU || MEDIAPIPE_METAL_ENABLED
+  // TODO: Configure GPU usage policy in individual calculators.
+  constexpr bool kAllowGpuProcessing = true;
+  return kAllowGpuProcessing;
+#else
+  return false;
+#endif  // !MEDIAPIPE_DISABLE_GPU || MEDIAPIPE_METAL_ENABLED
+}
+
+constexpr char kTensorsTag[] = "TENSORS";
+constexpr char kOutputSizeTag[] = "OUTPUT_SIZE";
+constexpr char kMaskTag[] = "MASK";
+
+absl::StatusOr<std::tuple<int, int, int>> GetHwcFromDims(
+    const std::vector<int>& dims) {
+  if (dims.size() == 3) {
+    return std::make_tuple(dims[0], dims[1], dims[2]);
+  } else if (dims.size() == 4) {
+    // BHWC format check B == 1
+    RET_CHECK_EQ(1, dims[0]) << "Expected batch to be 1 for BHWC heatmap";
+    return std::make_tuple(dims[1], dims[2], dims[3]);
+  } else {
+    RET_CHECK(false) << "Invalid shape for segmentation tensor " << dims.size();
+  }
+}
+}  // namespace
+
+namespace mediapipe {
+
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+using ::tflite::gpu::gl::GlProgram;
+using ::tflite::gpu::gl::GlShader;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+// Converts Tensors from a tflite segmentation model to an image mask.
+//
+// Performs optional upscale to OUTPUT_SIZE dimensions if provided,
+// otherwise the mask is the same size as input tensor.
+//
+// If at least one input tensor is already on GPU, processing happens on GPU and
+// the output mask is also stored on GPU. Otherwise, processing and the output
+// mask are both on CPU.
+//
+// On GPU, the mask is an RGBA image, in both the R & A channels, scaled 0-1.
+// On CPU, the mask is a ImageFormat::VEC32F1 image, with values scaled 0-1.
+//
+//
+// Inputs:
+//   One of the following TENSORS tags:
+//   TENSORS: Vector of Tensor,
+//            The tensor dimensions are specified in this calculator's options.
+//   OUTPUT_SIZE(optional): std::pair<int, int>,
+//                          If provided, the size to upscale mask to.
+//
+// Output:
+//   MASK: An Image output mask, RGBA(GPU) / VEC32F1(CPU).
+//
+// Options:
+//   See tensors_to_segmentation_calculator.proto
+//
+// Usage example:
+// node {
+//   calculator: "TensorsToSegmentationCalculator"
+//   input_stream: "TENSORS:tensors"
+//   input_stream: "OUTPUT_SIZE:size"
+//   output_stream: "MASK:hair_mask"
+//   node_options: {
+//     [mediapipe.TensorsToSegmentationCalculatorOptions] {
+//       output_layer_index: 1
+//       # gpu_origin: CONVENTIONAL # or TOP_LEFT
+//     }
+//   }
+// }
+//
+// Currently only OpenGLES 3.1 and CPU backends supported.
+// TODO Refactor and add support for other backends/platforms.
+//
+class TensorsToSegmentationCalculator : public CalculatorBase {
+ public:
+  static absl::Status GetContract(CalculatorContract* cc);
+
+  absl::Status Open(CalculatorContext* cc) override;
+  absl::Status Process(CalculatorContext* cc) override;
+  absl::Status Close(CalculatorContext* cc) override;
+
+ private:
+  absl::Status LoadOptions(CalculatorContext* cc);
+  absl::Status InitGpu(CalculatorContext* cc);
+  absl::Status ProcessGpu(CalculatorContext* cc);
+  absl::Status ProcessCpu(CalculatorContext* cc);
+  void GlRender();
+
+  bool DoesGpuTextureStartAtBottom() {
+    return options_.gpu_origin() != mediapipe::GpuOrigin_Mode_TOP_LEFT;
+  }
+
+  template <class T>
+  absl::Status ApplyActivation(cv::Mat& tensor_mat, cv::Mat* small_mask_mat);
+
+  ::mediapipe::TensorsToSegmentationCalculatorOptions options_;
+
+#if !MEDIAPIPE_DISABLE_GPU
+  mediapipe::GlCalculatorHelper gpu_helper_;
+  GLuint upsample_program_;
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+  std::unique_ptr<GlProgram> mask_program_31_;
+#else
+  GLuint mask_program_20_;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+#if MEDIAPIPE_METAL_ENABLED
+  MPPMetalHelper* metal_helper_ = nullptr;
+  id<MTLComputePipelineState> mask_program_;
+#endif  // MEDIAPIPE_METAL_ENABLED
+#endif  // !MEDIAPIPE_DISABLE_GPU
+};
+REGISTER_CALCULATOR(TensorsToSegmentationCalculator);
+
+// static
+absl::Status TensorsToSegmentationCalculator::GetContract(
+    CalculatorContract* cc) {
+  RET_CHECK(!cc->Inputs().GetTags().empty());
+  RET_CHECK(!cc->Outputs().GetTags().empty());
+
+  // Inputs.
+  cc->Inputs().Tag(kTensorsTag).Set<std::vector<Tensor>>();
+  if (cc->Inputs().HasTag(kOutputSizeTag)) {
+    cc->Inputs().Tag(kOutputSizeTag).Set<std::pair<int, int>>();
+  }
+
+  // Outputs.
+  cc->Outputs().Tag(kMaskTag).Set<Image>();
+
+  if (CanUseGpu()) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
+#if MEDIAPIPE_METAL_ENABLED
+    MP_RETURN_IF_ERROR([MPPMetalHelper updateContract:cc]);
+#endif  // MEDIAPIPE_METAL_ENABLED
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::Open(CalculatorContext* cc) {
+  cc->SetOffset(TimestampDiff(0));
+  bool use_gpu = false;
+
+  if (CanUseGpu()) {
+#if !MEDIAPIPE_DISABLE_GPU
+    use_gpu = true;
+    MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
+#if MEDIAPIPE_METAL_ENABLED
+    metal_helper_ = [[MPPMetalHelper alloc] initWithCalculatorContext:cc];
+    RET_CHECK(metal_helper_);
+#endif  // MEDIAPIPE_METAL_ENABLED
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  }
+
+  MP_RETURN_IF_ERROR(LoadOptions(cc));
+
+  if (use_gpu) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(InitGpu(cc));
+#else
+    RET_CHECK_FAIL() << "GPU processing disabled.";
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::Process(CalculatorContext* cc) {
+  if (cc->Inputs().Tag(kTensorsTag).IsEmpty()) {
+    return absl::OkStatus();
+  }
+
+  const auto& input_tensors =
+      cc->Inputs().Tag(kTensorsTag).Get<std::vector<Tensor>>();
+
+  bool use_gpu = false;
+  if (CanUseGpu()) {
+    // Use GPU processing only if at least one input tensor is already on GPU.
+    for (const auto& tensor : input_tensors) {
+      if (tensor.ready_on_gpu()) {
+        use_gpu = true;
+        break;
+      }
+    }
+  }
+
+  // Validate tensor channels and activation type.
+  {
+    RET_CHECK(!input_tensors.empty());
+    ASSIGN_OR_RETURN(auto hwc, GetHwcFromDims(input_tensors[0].shape().dims));
+    int tensor_channels = std::get<2>(hwc);
+    typedef mediapipe::TensorsToSegmentationCalculatorOptions Options;
+    switch (options_.activation()) {
+      case Options::NONE:
+        RET_CHECK_EQ(tensor_channels, 1);
+        break;
+      case Options::SIGMOID:
+        RET_CHECK_EQ(tensor_channels, 1);
+        break;
+      case Options::SOFTMAX:
+        RET_CHECK_EQ(tensor_channels, 2);
+        break;
+    }
+  }
+
+  if (use_gpu) {
+#if !MEDIAPIPE_DISABLE_GPU
+    MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this, cc]() -> absl::Status {
+      MP_RETURN_IF_ERROR(ProcessGpu(cc));
+      return absl::OkStatus();
+    }));
+#else
+    RET_CHECK_FAIL() << "GPU processing disabled.";
+#endif  // !MEDIAPIPE_DISABLE_GPU
+  } else {
+    MP_RETURN_IF_ERROR(ProcessCpu(cc));
+  }
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::Close(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  gpu_helper_.RunInGlContext([this] {
+    if (upsample_program_) glDeleteProgram(upsample_program_);
+    upsample_program_ = 0;
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    mask_program_31_.reset();
+#else
+    if (mask_program_20_) glDeleteProgram(mask_program_20_);
+    mask_program_20_ = 0;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+#if MEDIAPIPE_METAL_ENABLED
+    mask_program_ = nil;
+#endif  // MEDIAPIPE_METAL_ENABLED
+  });
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::ProcessCpu(
+    CalculatorContext* cc) {
+  // Get input streams, and dimensions.
+  const auto& input_tensors =
+      cc->Inputs().Tag(kTensorsTag).Get<std::vector<Tensor>>();
+  ASSIGN_OR_RETURN(auto hwc, GetHwcFromDims(input_tensors[0].shape().dims));
+  auto [tensor_height, tensor_width, tensor_channels] = hwc;
+  int output_width = tensor_width, output_height = tensor_height;
+  if (cc->Inputs().HasTag(kOutputSizeTag)) {
+    const auto& size =
+        cc->Inputs().Tag(kOutputSizeTag).Get<std::pair<int, int>>();
+    output_width = size.first;
+    output_height = size.second;
+  }
+
+  // Create initial working mask.
+  cv::Mat small_mask_mat(cv::Size(tensor_width, tensor_height), CV_32FC1);
+
+  // Wrap input tensor.
+  auto raw_input_tensor = &input_tensors[0];
+  auto raw_input_view = raw_input_tensor->GetCpuReadView();
+  const float* raw_input_data = raw_input_view.buffer<float>();
+  cv::Mat tensor_mat(cv::Size(tensor_width, tensor_height),
+                     CV_MAKETYPE(CV_32F, tensor_channels),
+                     const_cast<float*>(raw_input_data));
+
+  // Process mask tensor and apply activation function.
+  if (tensor_channels == 2) {
+    MP_RETURN_IF_ERROR(ApplyActivation<cv::Vec2f>(tensor_mat, &small_mask_mat));
+  } else if (tensor_channels == 1) {
+    RET_CHECK(mediapipe::TensorsToSegmentationCalculatorOptions::SOFTMAX !=
+              options_.activation());  // Requires 2 channels.
+    if (mediapipe::TensorsToSegmentationCalculatorOptions::NONE ==
+        options_.activation())  // Pass-through optimization.
+      tensor_mat.copyTo(small_mask_mat);
+    else
+      MP_RETURN_IF_ERROR(ApplyActivation<float>(tensor_mat, &small_mask_mat));
+  } else {
+    RET_CHECK_FAIL() << "Unsupported number of tensor channels "
+                     << tensor_channels;
+  }
+
+  // Send out image as CPU packet.
+  std::shared_ptr<ImageFrame> mask_frame = std::make_shared<ImageFrame>(
+      ImageFormat::VEC32F1, output_width, output_height);
+  std::unique_ptr<Image> output_mask = absl::make_unique<Image>(mask_frame);
+  cv::Mat output_mat = formats::MatView(output_mask.get());
+  // Upsample small mask into output.
+  cv::resize(small_mask_mat, output_mat, cv::Size(output_width, output_height));
+  cc->Outputs().Tag(kMaskTag).Add(output_mask.release(), cc->InputTimestamp());
+
+  return absl::OkStatus();
+}
+
+template <class T>
+absl::Status TensorsToSegmentationCalculator::ApplyActivation(
+    cv::Mat& tensor_mat, cv::Mat* small_mask_mat) {
+  // Configure activation function.
+  const int output_layer_index = options_.output_layer_index();
+  typedef mediapipe::TensorsToSegmentationCalculatorOptions Options;
+  const auto activation_fn = [&](const cv::Vec2f& mask_value) {
+    float new_mask_value = 0;
+    // TODO consider moving switch out of the loop,
+    // and also avoid float/Vec2f casting.
+    switch (options_.activation()) {
+      case Options::NONE: {
+        new_mask_value = mask_value[0];
+        break;
+      }
+      case Options::SIGMOID: {
+        const float pixel0 = mask_value[0];
+        new_mask_value = 1.0 / (std::exp(-pixel0) + 1.0);
+        break;
+      }
+      case Options::SOFTMAX: {
+        const float pixel0 = mask_value[0];
+        const float pixel1 = mask_value[1];
+        const float max_pixel = std::max(pixel0, pixel1);
+        const float min_pixel = std::min(pixel0, pixel1);
+        const float softmax_denom =
+            /*exp(max_pixel - max_pixel)=*/1.0f +
+            std::exp(min_pixel - max_pixel);
+        new_mask_value = std::exp(mask_value[output_layer_index] - max_pixel) /
+                         softmax_denom;
+        break;
+      }
+    }
+    return new_mask_value;
+  };
+
+  // Process mask tensor.
+  for (int i = 0; i < tensor_mat.rows; ++i) {
+    for (int j = 0; j < tensor_mat.cols; ++j) {
+      const T& input_pix = tensor_mat.at<T>(i, j);
+      const float mask_value = activation_fn(input_pix);
+      small_mask_mat->at<float>(i, j) = mask_value;
+    }
+  }
+
+  return absl::OkStatus();
+}
+
+// Steps:
+// 1. receive tensor
+// 2. process segmentation tensor into small mask
+// 3. upsample small mask into output mask to be same size as input image
+absl::Status TensorsToSegmentationCalculator::ProcessGpu(
+    CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  // Get input streams, and dimensions.
+  const auto& input_tensors =
+      cc->Inputs().Tag(kTensorsTag).Get<std::vector<Tensor>>();
+  ASSIGN_OR_RETURN(auto hwc, GetHwcFromDims(input_tensors[0].shape().dims));
+  auto [tensor_height, tensor_width, tensor_channels] = hwc;
+  int output_width = tensor_width, output_height = tensor_height;
+  if (cc->Inputs().HasTag(kOutputSizeTag)) {
+    const auto& size =
+        cc->Inputs().Tag(kOutputSizeTag).Get<std::pair<int, int>>();
+    output_width = size.first;
+    output_height = size.second;
+  }
+
+  // Create initial working mask texture.
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+  tflite::gpu::gl::GlTexture small_mask_texture;
+#else
+  mediapipe::GlTexture small_mask_texture;
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+  // Run shader, process mask tensor.
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+  {
+    MP_RETURN_IF_ERROR(CreateReadWriteRgbaImageTexture(
+        tflite::gpu::DataType::UINT8,  // GL_RGBA8
+        {tensor_width, tensor_height}, &small_mask_texture));
+
+    const int output_index = 0;
+    glBindImageTexture(output_index, small_mask_texture.id(), 0, GL_FALSE, 0,
+                       GL_WRITE_ONLY, GL_RGBA8);
+
+    auto read_view = input_tensors[0].GetOpenGlBufferReadView();
+    glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 2, read_view.name());
+
+    const tflite::gpu::uint3 workgroups = {
+        NumGroups(tensor_width, kWorkgroupSize),
+        NumGroups(tensor_height, kWorkgroupSize), 1};
+
+    glUseProgram(mask_program_31_->id());
+    glUniform2i(glGetUniformLocation(mask_program_31_->id(), "out_size"),
+                tensor_width, tensor_height);
+
+    MP_RETURN_IF_ERROR(mask_program_31_->Dispatch(workgroups));
+  }
+#elif MEDIAPIPE_METAL_ENABLED
+  {
+    id<MTLCommandBuffer> command_buffer = [metal_helper_ commandBuffer];
+    command_buffer.label = @"SegmentationKernel";
+    id<MTLComputeCommandEncoder> command_encoder =
+        [command_buffer computeCommandEncoder];
+    [command_encoder setComputePipelineState:mask_program_];
+
+    auto read_view = input_tensors[0].GetMtlBufferReadView(command_buffer);
+    [command_encoder setBuffer:read_view.buffer() offset:0 atIndex:0];
+
+    mediapipe::GpuBuffer small_mask_buffer = [metal_helper_
+        mediapipeGpuBufferWithWidth:tensor_width
+                             height:tensor_height
+                             format:mediapipe::GpuBufferFormat::kBGRA32];
+    id<MTLTexture> small_mask_texture_metal =
+        [metal_helper_ metalTextureWithGpuBuffer:small_mask_buffer];
+    [command_encoder setTexture:small_mask_texture_metal atIndex:1];
+
+    unsigned int out_size[] = {static_cast<unsigned int>(tensor_width),
+                               static_cast<unsigned int>(tensor_height)};
+    [command_encoder setBytes:&out_size length:sizeof(out_size) atIndex:2];
+
+    MTLSize threads_per_group = MTLSizeMake(kWorkgroupSize, kWorkgroupSize, 1);
+    MTLSize threadgroups =
+        MTLSizeMake(NumGroups(tensor_width, kWorkgroupSize),
+                    NumGroups(tensor_height, kWorkgroupSize), 1);
+    [command_encoder dispatchThreadgroups:threadgroups
+                    threadsPerThreadgroup:threads_per_group];
+    [command_encoder endEncoding];
+    [command_buffer commit];
+
+    small_mask_texture = gpu_helper_.CreateSourceTexture(small_mask_buffer);
+  }
+#else
+  {
+    small_mask_texture = gpu_helper_.CreateDestinationTexture(
+        tensor_width, tensor_height,
+        mediapipe::GpuBufferFormat::kBGRA32);  // actually GL_RGBA8
+
+    // Go through CPU if not already texture 2D (no direct conversion yet).
+    // Tensor::GetOpenGlTexture2dReadView() doesn't automatically convert types.
+    if (!input_tensors[0].ready_as_opengl_texture_2d()) {
+      (void)input_tensors[0].GetCpuReadView();
+    }
+
+    auto read_view = input_tensors[0].GetOpenGlTexture2dReadView();
+
+    gpu_helper_.BindFramebuffer(small_mask_texture);
+    glActiveTexture(GL_TEXTURE1);
+    glBindTexture(GL_TEXTURE_2D, read_view.name());
+    glUseProgram(mask_program_20_);
+    GlRender();
+    glBindTexture(GL_TEXTURE_2D, 0);
+    glFlush();
+  }
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+  // Upsample small mask into output.
+  mediapipe::GlTexture output_texture = gpu_helper_.CreateDestinationTexture(
+      output_width, output_height,
+      mediapipe::GpuBufferFormat::kBGRA32);  // actually GL_RGBA8
+
+  // Run shader, upsample result.
+  {
+    gpu_helper_.BindFramebuffer(output_texture);
+    glActiveTexture(GL_TEXTURE1);
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    glBindTexture(GL_TEXTURE_2D, small_mask_texture.id());
+#else
+    glBindTexture(GL_TEXTURE_2D, small_mask_texture.name());
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    glUseProgram(upsample_program_);
+    GlRender();
+    glBindTexture(GL_TEXTURE_2D, 0);
+    glFlush();
+  }
+
+  // Send out image as GPU packet.
+  auto output_image = output_texture.GetFrame<Image>();
+  cc->Outputs().Tag(kMaskTag).Add(output_image.release(), cc->InputTimestamp());
+
+  // Cleanup
+  output_texture.Release();
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+void TensorsToSegmentationCalculator::GlRender() {
+#if !MEDIAPIPE_DISABLE_GPU
+  static const GLfloat square_vertices[] = {
+      -1.0f, -1.0f,  // bottom left
+      1.0f,  -1.0f,  // bottom right
+      -1.0f, 1.0f,   // top left
+      1.0f,  1.0f,   // top right
+  };
+  static const GLfloat texture_vertices[] = {
+      0.0f, 0.0f,  // bottom left
+      1.0f, 0.0f,  // bottom right
+      0.0f, 1.0f,  // top left
+      1.0f, 1.0f,  // top right
+  };
+
+  // vertex storage
+  GLuint vbo[2];
+  glGenBuffers(2, vbo);
+  GLuint vao;
+  glGenVertexArrays(1, &vao);
+  glBindVertexArray(vao);
+
+  // vbo 0
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_VERTEX);
+  glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // vbo 1
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // draw
+  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
+
+  // cleanup
+  glDisableVertexAttribArray(ATTRIB_VERTEX);
+  glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glBindBuffer(GL_ARRAY_BUFFER, 0);
+  glBindVertexArray(0);
+  glDeleteVertexArrays(1, &vao);
+  glDeleteBuffers(2, vbo);
+#endif  // !MEDIAPIPE_DISABLE_GPU
+}
+
+absl::Status TensorsToSegmentationCalculator::LoadOptions(
+    CalculatorContext* cc) {
+  // Get calculator options specified in the graph.
+  options_ = cc->Options<::mediapipe::TensorsToSegmentationCalculatorOptions>();
+
+  return absl::OkStatus();
+}
+
+absl::Status TensorsToSegmentationCalculator::InitGpu(CalculatorContext* cc) {
+#if !MEDIAPIPE_DISABLE_GPU
+  MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this]() -> absl::Status {
+  // A shader to process a segmentation tensor into an output mask.
+  // Currently uses 4 channels for output, and sets R+A channels as mask value.
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    // GLES 3.1
+    const tflite::gpu::uint3 workgroup_size = {kWorkgroupSize, kWorkgroupSize,
+                                               1};
+    const std::string shader_header =
+        absl::StrCat(tflite::gpu::gl::GetShaderHeader(workgroup_size), R"(
+precision highp float;
+
+layout(rgba8, binding = 0) writeonly uniform highp image2D output_texture;
+
+uniform ivec2 out_size;
+)");
+    /* Shader defines will be inserted here. */
+
+    const std::string shader_src_main = R"(
+layout(std430, binding = 2) readonly buffer B0 {
+#ifdef TWO_CHANNEL_INPUT
+  vec2 elements[];
+#else
+  float elements[];
+#endif // TWO_CHANNEL_INPUT
+} input_data;   // data tensor
+
+void main() {
+  int out_width = out_size.x;
+  int out_height = out_size.y;
+
+  ivec2 gid = ivec2(gl_GlobalInvocationID.xy);
+  if (gid.x >= out_width || gid.y >= out_height) { return; }
+  int linear_index = gid.y * out_width + gid.x;
+
+#ifdef TWO_CHANNEL_INPUT
+  vec2 input_value = input_data.elements[linear_index];
+#else
+  vec2 input_value = vec2(input_data.elements[linear_index], 0.0);
+#endif // TWO_CHANNEL_INPUT
+
+// Run activation function.
+// One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
+#ifdef FN_SOFTMAX
+  // Only two channel input tensor is supported.
+  vec2 input_px = input_value.rg;
+  float shift = max(input_px.r, input_px.g);
+  float softmax_denom = exp(input_px.r - shift) + exp(input_px.g - shift);
+  float new_mask_value =
+      exp(input_px[OUTPUT_LAYER_INDEX] - shift) / softmax_denom;
+#endif // FN_SOFTMAX
+
+#ifdef FN_SIGMOID
+  float new_mask_value = 1.0 / (exp(-input_value.r) + 1.0);
+#endif // FN_SIGMOID
+
+#ifdef FN_NONE
+  float new_mask_value = input_value.r;
+#endif // FN_NONE
+
+#ifdef FLIP_Y_COORD
+  int y_coord = out_height - gid.y - 1;
+#else
+  int y_coord = gid.y;
+#endif  // defined(FLIP_Y_COORD)
+  ivec2 output_coordinate = ivec2(gid.x, y_coord);
+
+  vec4 out_value = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
+  imageStore(output_texture, output_coordinate, out_value);
+})";
+
+#elif MEDIAPIPE_METAL_ENABLED
+    // METAL
+    const std::string shader_header = R"(
+#include <metal_stdlib>
+using namespace metal;
+)";
+    /* Shader defines will be inserted here. */
+
+    const std::string shader_src_main = R"(
+kernel void segmentationKernel(
+#ifdef TWO_CHANNEL_INPUT
+    device float2*     elements        [[ buffer(0) ]],
+#else
+    device float*      elements        [[ buffer(0) ]],
+#endif // TWO_CHANNEL_INPUT
+    texture2d<float, access::write>  output_texture  [[ texture(1) ]],
+    constant uint*      out_size        [[ buffer(2) ]],
+    uint2               gid             [[ thread_position_in_grid ]])
+{
+  uint out_width = out_size[0];
+  uint out_height = out_size[1];
+
+  if (gid.x >= out_width || gid.y >= out_height) { return; }
+  uint linear_index = gid.y * out_width + gid.x;
+
+#ifdef TWO_CHANNEL_INPUT
+  float2 input_value = elements[linear_index];
+#else
+  float2 input_value = float2(elements[linear_index], 0.0);
+#endif // TWO_CHANNEL_INPUT
+
+// Run activation function.
+// One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
+#ifdef FN_SOFTMAX
+  // Only two channel input tensor is supported.
+  float2 input_px = input_value.xy;
+  float shift = max(input_px.x, input_px.y);
+  float softmax_denom = exp(input_px.r - shift) + exp(input_px.g - shift);
+  float new_mask_value =
+      exp(input_px[OUTPUT_LAYER_INDEX] - shift) / softmax_denom;
+#endif // FN_SOFTMAX
+
+#ifdef FN_SIGMOID
+  float new_mask_value = 1.0 / (exp(-input_value.x) + 1.0);
+#endif // FN_SIGMOID
+
+#ifdef FN_NONE
+  float new_mask_value = input_value.x;
+#endif // FN_NONE
+
+#ifdef FLIP_Y_COORD
+  int y_coord = out_height - gid.y - 1;
+#else
+  int y_coord = gid.y;
+#endif  // defined(FLIP_Y_COORD)
+  uint2 output_coordinate = uint2(gid.x, y_coord);
+
+  float4 out_value = float4(new_mask_value, 0.0, 0.0, new_mask_value);
+  output_texture.write(out_value, output_coordinate);
+}
+)";
+
+#else
+    // GLES 2.0
+    const std::string shader_header = absl::StrCat(
+        std::string(mediapipe::kMediaPipeFragmentShaderPreamble), R"(
+DEFAULT_PRECISION(mediump, float)
+)");
+    /* Shader defines will be inserted here. */
+
+    const std::string shader_src_main = R"(
+in vec2 sample_coordinate;
+
+uniform sampler2D input_texture;
+
+#ifdef GL_ES
+#define fragColor gl_FragColor
+#else
+out vec4 fragColor;
+#endif  // defined(GL_ES);
+
+void main() {
+
+  vec4 input_value = texture2D(input_texture, sample_coordinate);
+  vec2 gid = sample_coordinate;
+
+  // Run activation function.
+  // One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
+
+#ifdef FN_SOFTMAX
+  // Only two channel input tensor is supported.
+  vec2 input_px = input_value.rg;
+  float shift = max(input_px.r, input_px.g);
+  float softmax_denom = exp(input_px.r - shift) + exp(input_px.g - shift);
+  float new_mask_value =
+      exp(mix(input_px.r, input_px.g, float(OUTPUT_LAYER_INDEX)) - shift) / softmax_denom;
+#endif // FN_SOFTMAX
+
+#ifdef FN_SIGMOID
+  float new_mask_value = 1.0 / (exp(-input_value.r) + 1.0);
+#endif // FN_SIGMOID
+
+#ifdef FN_NONE
+  float new_mask_value = input_value.r;
+#endif // FN_NONE
+
+#ifdef FLIP_Y_COORD
+  float y_coord = 1.0 - gid.y;
+#else
+  float y_coord = gid.y;
+#endif  // defined(FLIP_Y_COORD)
+  vec2 output_coordinate = vec2(gid.x, y_coord);
+
+  vec4 out_value = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
+  fragColor = out_value;
+})";
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+    // Shader defines.
+    typedef mediapipe::TensorsToSegmentationCalculatorOptions Options;
+    const std::string output_layer_index =
+        "\n#define OUTPUT_LAYER_INDEX int(" +
+        std::to_string(options_.output_layer_index()) + ")";
+    const std::string flip_y_coord =
+        DoesGpuTextureStartAtBottom() ? "\n#define FLIP_Y_COORD" : "";
+    const std::string fn_none =
+        options_.activation() == Options::NONE ? "\n#define FN_NONE" : "";
+    const std::string fn_sigmoid =
+        options_.activation() == Options::SIGMOID ? "\n#define FN_SIGMOID" : "";
+    const std::string fn_softmax =
+        options_.activation() == Options::SOFTMAX ? "\n#define FN_SOFTMAX" : "";
+    const std::string two_channel = options_.activation() == Options::SOFTMAX
+                                        ? "\n#define TWO_CHANNEL_INPUT"
+                                        : "";
+    const std::string shader_defines =
+        absl::StrCat(output_layer_index, flip_y_coord, fn_softmax, fn_sigmoid,
+                     fn_none, two_channel);
+
+    // Build full shader.
+    const std::string shader_src_no_previous =
+        absl::StrCat(shader_header, shader_defines, shader_src_main);
+
+    // Vertex shader attributes.
+    const GLint attr_location[NUM_ATTRIBUTES] = {
+        ATTRIB_VERTEX,
+        ATTRIB_TEXTURE_POSITION,
+    };
+    const GLchar* attr_name[NUM_ATTRIBUTES] = {
+        "position",
+        "texture_coordinate",
+    };
+
+    // Main shader program & parameters
+#if MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+    GlShader shader_without_previous;
+    MP_RETURN_IF_ERROR(GlShader::CompileShader(
+        GL_COMPUTE_SHADER, shader_src_no_previous, &shader_without_previous));
+    mask_program_31_ = absl::make_unique<GlProgram>();
+    MP_RETURN_IF_ERROR(GlProgram::CreateWithShader(shader_without_previous,
+                                                   mask_program_31_.get()));
+#elif MEDIAPIPE_METAL_ENABLED
+    id<MTLDevice> device = metal_helper_.mtlDevice;
+    NSString* library_source =
+        [NSString stringWithUTF8String:shader_src_no_previous.c_str()];
+    NSError* error = nil;
+    id<MTLLibrary> library = [device newLibraryWithSource:library_source
+                                                  options:nullptr
+                                                    error:&error];
+    RET_CHECK(library != nil) << "Couldn't create shader library "
+                              << [[error localizedDescription] UTF8String];
+    id<MTLFunction> kernel_func = nil;
+    kernel_func = [library newFunctionWithName:@"segmentationKernel"];
+    RET_CHECK(kernel_func != nil) << "Couldn't create kernel function.";
+    mask_program_ =
+        [device newComputePipelineStateWithFunction:kernel_func error:&error];
+    RET_CHECK(mask_program_ != nil) << "Couldn't create pipeline state " <<
+        [[error localizedDescription] UTF8String];
+#else
+    mediapipe::GlhCreateProgram(
+        mediapipe::kBasicVertexShader, shader_src_no_previous.c_str(),
+        NUM_ATTRIBUTES, &attr_name[0], attr_location, &mask_program_20_);
+    RET_CHECK(mask_program_20_) << "Problem initializing the program.";
+    glUseProgram(mask_program_20_);
+    glUniform1i(glGetUniformLocation(mask_program_20_, "input_texture"), 1);
+#endif  // MEDIAPIPE_OPENGL_ES_VERSION >= MEDIAPIPE_OPENGL_ES_31
+
+    // Simple pass-through program, used for hardware upsampling.
+    mediapipe::GlhCreateProgram(
+        mediapipe::kBasicVertexShader, mediapipe::kBasicTexturedFragmentShader,
+        NUM_ATTRIBUTES, &attr_name[0], attr_location, &upsample_program_);
+    RET_CHECK(upsample_program_) << "Problem initializing the program.";
+    glUseProgram(upsample_program_);
+    glUniform1i(glGetUniformLocation(upsample_program_, "video_frame"), 1);
+
+    return absl::OkStatus();
+  }));
+#endif  // !MEDIAPIPE_DISABLE_GPU
+
+  return absl::OkStatus();
+}
+
+}  // namespace mediapipe

mediapipe/calculators/tensor/tensors_to_segmentation_calculator.proto

@@ -0,0 +1,46 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+syntax = "proto2";
+
+package mediapipe;
+
+import "mediapipe/framework/calculator.proto";
+import "mediapipe/gpu/gpu_origin.proto";
+
+message TensorsToSegmentationCalculatorOptions {
+  extend mediapipe.CalculatorOptions {
+    optional TensorsToSegmentationCalculatorOptions ext = 374311106;
+  }
+
+  // For CONVENTIONAL mode in OpenGL, textures start at bottom and needs
+  // to be flipped vertically as tensors are expected to start at top.
+  // (DEFAULT or unset is interpreted as CONVENTIONAL.)
+  optional GpuOrigin.Mode gpu_origin = 1;
+
+  // Supported activation functions for filtering.
+  enum Activation {
+    NONE = 0;     // Assumes 1-channel input tensor.
+    SIGMOID = 1;  // Assumes 1-channel input tensor.
+    SOFTMAX = 2;  // Assumes 2-channel input tensor.
+  }
+  // Activation function to apply to input tensor.
+  // Softmax requires a 2-channel tensor, see output_layer_index below.
+  optional Activation activation = 2 [default = NONE];
+
+  // Channel to use for processing tensor.
+  // Only applies when using activation=SOFTMAX.
+  // Works on two channel input tensor only.
+  optional int32 output_layer_index = 3 [default = 1];
+}

mediapipe/calculators/tensorflow/tensor_to_image_frame_calculator.cc

@@ -34,15 +34,28 @@ constexpr char kTensor[] = "TENSOR";
 }  // namespace
 
 // Input:
-//  Tensor of type DT_FLOAT, with values between 0-255 (SRGB or GRAY8). The
-//  shape can be HxWx{3,1} or simply HxW.
+//  Tensor of type DT_FLOAT or DT_UINT8, with values between 0-255
+//  (SRGB or GRAY8). The shape can be HxWx{3,1} or simply HxW.
 //
-//  Optionally supports a scale factor that can scale 0-1 value ranges to 0-255.
+//  For DT_FLOAT tensors, optionally supports a scale factor that can scale 0-1
+//  value ranges to 0-255.
 //
 // Output:
 //  ImageFrame containing the values of the tensor cast as uint8 (SRGB or GRAY8)
 //
 // Possible extensions: support other input ranges, maybe 4D tensors.
+//
+// Example:
+// node {
+//   calculator: "TensorToImageFrameCalculator"
+//   input_stream: "TENSOR:3d_float_tensor"
+//   output_stream: "IMAGE:image_frame"
+//   options {
+//     [mediapipe.TensorToImageFrameCalculatorOptions.ext] {
+//       scale_factor: 1.0  # set to 255.0 for [0,1] -> [0,255] scaling
+//     }
+//   }
+// }
 class TensorToImageFrameCalculator : public CalculatorBase {
  public:
   static absl::Status GetContract(CalculatorContract* cc);
@@ -57,8 +70,8 @@ class TensorToImageFrameCalculator : public CalculatorBase {
 REGISTER_CALCULATOR(TensorToImageFrameCalculator);
 
 absl::Status TensorToImageFrameCalculator::GetContract(CalculatorContract* cc) {
-  RET_CHECK_EQ(cc->Inputs().NumEntries(), 1)
-      << "Only one input stream is supported.";
+  RET_CHECK_EQ(cc->Outputs().NumEntries(), 1)
+      << "Only one output stream is supported.";
   RET_CHECK_EQ(cc->Inputs().NumEntries(), 1)
       << "One input stream must be provided.";
   RET_CHECK(cc->Inputs().HasTag(kTensor))
@@ -91,29 +104,44 @@ absl::Status TensorToImageFrameCalculator::Process(CalculatorContext* cc) {
       RET_CHECK_EQ(depth, 3) << "Output tensor depth must be 3 or 1.";
     }
   }
-  const int32 total_size =
-      input_tensor.dim_size(0) * input_tensor.dim_size(1) * depth;
-  std::unique_ptr<uint8[]> buffer(new uint8[total_size]);
-  auto data = input_tensor.flat<float>().data();
-  for (int i = 0; i < total_size; ++i) {
-    float d = scale_factor_ * data[i];
-    if (d < 0) d = 0;
-    if (d > 255) d = 255;
-    buffer[i] = d;
+  int32 height = input_tensor.dim_size(0);
+  int32 width = input_tensor.dim_size(1);
+  auto format = (depth == 3 ? ImageFormat::SRGB : ImageFormat::GRAY8);
+  const int32 total_size = height * width * depth;
+
+  ::std::unique_ptr<const ImageFrame> output;
+  if (input_tensor.dtype() == tensorflow::DT_FLOAT) {
+    // Allocate buffer with alignments.
+    std::unique_ptr<uint8_t[]> buffer(
+        new (std::align_val_t(EIGEN_MAX_ALIGN_BYTES)) uint8_t[total_size]);
+    auto data = input_tensor.flat<float>().data();
+    for (int i = 0; i < total_size; ++i) {
+      float d = scale_factor_ * data[i];
+      if (d < 0) d = 0;
+      if (d > 255) d = 255;
+      buffer[i] = d;
+    }
+    output = ::absl::make_unique<ImageFrame>(format, width, height,
+                                             width * depth, buffer.release());
+  } else if (input_tensor.dtype() == tensorflow::DT_UINT8) {
+    if (scale_factor_ != 1.0) {
+      return absl::InvalidArgumentError("scale_factor_ given for uint8 tensor");
+    }
+    // tf::Tensor has internally ref-counted buffer. The following code make the
+    // ImageFrame own the copied Tensor through the deleter, which increases
+    // the refcount of the buffer and allow us to use the shared buffer as the
+    // image. This allows us to create an ImageFrame object without copying
+    // buffer. const ImageFrame prevents the buffer from being modified later.
+    auto copy = new tf::Tensor(input_tensor);
+    output = ::absl::make_unique<const ImageFrame>(
+        format, width, height, width * depth, copy->flat<uint8_t>().data(),
+        [copy](uint8*) { delete copy; });
+  } else {
+    return absl::InvalidArgumentError(
+        absl::StrCat("Expected float or uint8 tensor, received ",
+                     DataTypeString(input_tensor.dtype())));
   }
 
-  ::std::unique_ptr<ImageFrame> output;
-  if (depth == 3) {
-    output = ::absl::make_unique<ImageFrame>(
-        ImageFormat::SRGB, input_tensor.dim_size(1), input_tensor.dim_size(0),
-        input_tensor.dim_size(1) * 3, buffer.release());
-  } else if (depth == 1) {
-    output = ::absl::make_unique<ImageFrame>(
-        ImageFormat::GRAY8, input_tensor.dim_size(1), input_tensor.dim_size(0),
-        input_tensor.dim_size(1), buffer.release());
-  } else {
-    return absl::InvalidArgumentError("Unrecognized image depth.");
-  }
   cc->Outputs().Tag(kImage).Add(output.release(), cc->InputTimestamp());
 
   return absl::OkStatus();

mediapipe/calculators/tensorflow/tensor_to_image_frame_calculator_test.cc

@@ -29,6 +29,7 @@ constexpr char kImage[] = "IMAGE";
 
 }  // namespace
 
+template <class TypeParam>
 class TensorToImageFrameCalculatorTest : public ::testing::Test {
  protected:
   void SetUpRunner() {
@@ -42,14 +43,20 @@ class TensorToImageFrameCalculatorTest : public ::testing::Test {
   std::unique_ptr<CalculatorRunner> runner_;
 };
 
-TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrame) {
-  SetUpRunner();
+using TensorToImageFrameCalculatorTestTypes = ::testing::Types<float, uint8_t>;
+TYPED_TEST_CASE(TensorToImageFrameCalculatorTest,
+                TensorToImageFrameCalculatorTestTypes);
+
+TYPED_TEST(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrame) {
+  // TYPED_TEST requires explicit "this->"
+  this->SetUpRunner();
+  auto& runner = this->runner_;
   constexpr int kWidth = 16;
   constexpr int kHeight = 8;
-  const tf::TensorShape tensor_shape(
-      std::vector<tf::int64>{kHeight, kWidth, 3});
-  auto tensor = absl::make_unique<tf::Tensor>(tf::DT_FLOAT, tensor_shape);
-  auto tensor_vec = tensor->flat<float>().data();
+  const tf::TensorShape tensor_shape{kHeight, kWidth, 3};
+  auto tensor = absl::make_unique<tf::Tensor>(
+      tf::DataTypeToEnum<TypeParam>::v(), tensor_shape);
+  auto tensor_vec = tensor->template flat<TypeParam>().data();
 
   // Writing sequence of integers as floats which we want back (as they were
   // written).
@@ -58,15 +65,16 @@ TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrame) {
   }
 
   const int64 time = 1234;
-  runner_->MutableInputs()->Tag(kTensor).packets.push_back(
+  runner->MutableInputs()->Tag(kTensor).packets.push_back(
       Adopt(tensor.release()).At(Timestamp(time)));
 
-  EXPECT_TRUE(runner_->Run().ok());
+  EXPECT_TRUE(runner->Run().ok());
   const std::vector<Packet>& output_packets =
-      runner_->Outputs().Tag(kImage).packets;
+      runner->Outputs().Tag(kImage).packets;
   EXPECT_EQ(1, output_packets.size());
   EXPECT_EQ(time, output_packets[0].Timestamp().Value());
   const ImageFrame& output_image = output_packets[0].Get<ImageFrame>();
+  EXPECT_EQ(ImageFormat::SRGB, output_image.Format());
   EXPECT_EQ(kWidth, output_image.Width());
   EXPECT_EQ(kHeight, output_image.Height());
 
@@ -76,14 +84,15 @@ TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrame) {
   }
 }
 
-TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrameGray) {
-  SetUpRunner();
+TYPED_TEST(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrameGray) {
+  this->SetUpRunner();
+  auto& runner = this->runner_;
   constexpr int kWidth = 16;
   constexpr int kHeight = 8;
-  const tf::TensorShape tensor_shape(
-      std::vector<tf::int64>{kHeight, kWidth, 1});
-  auto tensor = absl::make_unique<tf::Tensor>(tf::DT_FLOAT, tensor_shape);
-  auto tensor_vec = tensor->flat<float>().data();
+  const tf::TensorShape tensor_shape{kHeight, kWidth, 1};
+  auto tensor = absl::make_unique<tf::Tensor>(
+      tf::DataTypeToEnum<TypeParam>::v(), tensor_shape);
+  auto tensor_vec = tensor->template flat<TypeParam>().data();
 
   // Writing sequence of integers as floats which we want back (as they were
   // written).
@@ -92,15 +101,16 @@ TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrameGray) {
   }
 
   const int64 time = 1234;
-  runner_->MutableInputs()->Tag(kTensor).packets.push_back(
+  runner->MutableInputs()->Tag(kTensor).packets.push_back(
       Adopt(tensor.release()).At(Timestamp(time)));
 
-  EXPECT_TRUE(runner_->Run().ok());
+  EXPECT_TRUE(runner->Run().ok());
   const std::vector<Packet>& output_packets =
-      runner_->Outputs().Tag(kImage).packets;
+      runner->Outputs().Tag(kImage).packets;
   EXPECT_EQ(1, output_packets.size());
   EXPECT_EQ(time, output_packets[0].Timestamp().Value());
   const ImageFrame& output_image = output_packets[0].Get<ImageFrame>();
+  EXPECT_EQ(ImageFormat::GRAY8, output_image.Format());
   EXPECT_EQ(kWidth, output_image.Width());
   EXPECT_EQ(kHeight, output_image.Height());
 
@@ -110,13 +120,16 @@ TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrameGray) {
   }
 }
 
-TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrame2DGray) {
-  SetUpRunner();
+TYPED_TEST(TensorToImageFrameCalculatorTest,
+           Converts3DTensorToImageFrame2DGray) {
+  this->SetUpRunner();
+  auto& runner = this->runner_;
   constexpr int kWidth = 16;
   constexpr int kHeight = 8;
-  const tf::TensorShape tensor_shape(std::vector<tf::int64>{kHeight, kWidth});
-  auto tensor = absl::make_unique<tf::Tensor>(tf::DT_FLOAT, tensor_shape);
-  auto tensor_vec = tensor->flat<float>().data();
+  const tf::TensorShape tensor_shape{kHeight, kWidth};
+  auto tensor = absl::make_unique<tf::Tensor>(
+      tf::DataTypeToEnum<TypeParam>::v(), tensor_shape);
+  auto tensor_vec = tensor->template flat<TypeParam>().data();
 
   // Writing sequence of integers as floats which we want back (as they were
   // written).
@@ -125,15 +138,16 @@ TEST_F(TensorToImageFrameCalculatorTest, Converts3DTensorToImageFrame2DGray) {
   }
 
   const int64 time = 1234;
-  runner_->MutableInputs()->Tag(kTensor).packets.push_back(
+  runner->MutableInputs()->Tag(kTensor).packets.push_back(
       Adopt(tensor.release()).At(Timestamp(time)));
 
-  EXPECT_TRUE(runner_->Run().ok());
+  EXPECT_TRUE(runner->Run().ok());
   const std::vector<Packet>& output_packets =
-      runner_->Outputs().Tag(kImage).packets;
+      runner->Outputs().Tag(kImage).packets;
   EXPECT_EQ(1, output_packets.size());
   EXPECT_EQ(time, output_packets[0].Timestamp().Value());
   const ImageFrame& output_image = output_packets[0].Get<ImageFrame>();
+  EXPECT_EQ(ImageFormat::GRAY8, output_image.Format());
   EXPECT_EQ(kWidth, output_image.Width());
   EXPECT_EQ(kHeight, output_image.Height());
 

mediapipe/calculators/tensorflow/tensor_to_matrix_calculator.cc

@@ -91,8 +91,6 @@ absl::Status FillTimeSeriesHeaderIfValid(const Packet& header_packet,
 // the input data when it arrives in Process(). In particular, if the header
 // states that we produce a 1xD column vector, the input tensor must also be 1xD
 //
-// This designed was discussed in http://g/speakeranalysis/4uyx7cNRwJY and
-// http://g/daredevil-project/VB26tcseUy8.
 // Example Config
 // node: {
 //   calculator: "TensorToMatrixCalculator"
@@ -158,22 +156,17 @@ absl::Status TensorToMatrixCalculator::Open(CalculatorContext* cc) {
   if (header_status.ok()) {
     if (cc->Options<TensorToMatrixCalculatorOptions>()
             .has_time_series_header_overrides()) {
-      // From design discussions with Daredevil, we only want to support single
-      // sample per packet for now, so we hardcode the sample_rate based on the
-      // packet_rate of the REFERENCE and fail noisily if we cannot. An
-      // alternative would be to calculate the sample_rate from the reference
-      // sample_rate and the change in num_samples between the reference and
-      // override headers:
-      // sample_rate_output = sample_rate_reference /
-      //                      (num_samples_override / num_samples_reference)
+      // This only supports a single sample per packet for now, so we hardcode
+      // the sample_rate based on the packet_rate of the REFERENCE and fail
+      // if we cannot.
       const TimeSeriesHeader& override_header =
           cc->Options<TensorToMatrixCalculatorOptions>()
               .time_series_header_overrides();
       input_header->MergeFrom(override_header);
-      CHECK(input_header->has_packet_rate())
+      RET_CHECK(input_header->has_packet_rate())
           << "The TimeSeriesHeader.packet_rate must be set.";
       if (!override_header.has_sample_rate()) {
-        CHECK_EQ(input_header->num_samples(), 1)
+        RET_CHECK_EQ(input_header->num_samples(), 1)
             << "Currently the time series can only output single samples.";
         input_header->set_sample_rate(input_header->packet_rate());
       }
@@ -186,20 +179,16 @@ absl::Status TensorToMatrixCalculator::Open(CalculatorContext* cc) {
 }
 
 absl::Status TensorToMatrixCalculator::Process(CalculatorContext* cc) {
-  // Daredevil requested CHECK for noisy failures rather than quieter RET_CHECK
-  // failures. These are absolute conditions of the graph for the graph to be
-  // valid, and if it is violated by any input anywhere, the graph will be
-  // invalid for all inputs. A hard CHECK will enable faster debugging by
-  // immediately exiting and more prominently displaying error messages.
-  // Do not replace with RET_CHECKs.
-
   // Verify that each reference stream packet corresponds to a tensor packet
   // otherwise the header information is invalid. If we don't have a reference
   // stream, Process() is only called when we have an input tensor and this is
   // always True.
-  CHECK(cc->Inputs().HasTag(kTensor))
+  RET_CHECK(cc->Inputs().HasTag(kTensor))
       << "Tensor stream not available at same timestamp as the reference "
          "stream.";
+  RET_CHECK(!cc->Inputs().Tag(kTensor).IsEmpty()) << "Tensor stream is empty.";
+  RET_CHECK_OK(cc->Inputs().Tag(kTensor).Value().ValidateAsType<tf::Tensor>())
+      << "Tensor stream packet does not contain a Tensor.";
 
   const tf::Tensor& input_tensor = cc->Inputs().Tag(kTensor).Get<tf::Tensor>();
   CHECK(1 == input_tensor.dims() || 2 == input_tensor.dims())
@@ -207,13 +196,12 @@ absl::Status TensorToMatrixCalculator::Process(CalculatorContext* cc) {
   const int32 length = input_tensor.dim_size(input_tensor.dims() - 1);
   const int32 width = (1 == input_tensor.dims()) ? 1 : input_tensor.dim_size(0);
   if (header_.has_num_channels()) {
-    CHECK_EQ(length, header_.num_channels())
+    RET_CHECK_EQ(length, header_.num_channels())
         << "The number of channels at runtime does not match the header.";
   }
   if (header_.has_num_samples()) {
-    CHECK_EQ(width, header_.num_samples())
+    RET_CHECK_EQ(width, header_.num_samples())
         << "The number of samples at runtime does not match the header.";
-    ;
   }
   auto output = absl::make_unique<Matrix>(width, length);
   *output =

mediapipe/calculators/tensorflow/tensorflow_inference_calculator.cc

@@ -98,388 +98,543 @@ class InferenceState {
 
 // This calculator performs inference on a trained TensorFlow model.
 //
-// A mediapipe::TensorFlowSession with a model loaded and ready for use.
-// For this calculator it must include a tag_to_tensor_map.
-cc->InputSidePackets().Tag("SESSION").Set<TensorFlowSession>();
-if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS")) {
-  cc->InputSidePackets()
-      .Tag("RECURRENT_INIT_TENSORS")
-      .Set<std::unique_ptr<std::map<std::string, tf::Tensor>>>();
-}
-return absl::OkStatus();
-}
+// TensorFlow Sessions can be created from checkpoint paths, frozen models, or
+// the SavedModel system. See the TensorFlowSessionFrom* packet generators for
+// details. Each of these methods defines a mapping between MediaPipe streams
+// and TensorFlow tensors. All of this information is passed in as an
+// input_side_packet.
+//
+// The input and output streams are TensorFlow tensors labeled by tags. The tags
+// for the streams are matched to feeds and fetchs in a TensorFlow session using
+// a named_signature.generic_signature in the ModelManifest. The
+// generic_signature is used as key-value pairs between the MediaPipe tag and
+// the TensorFlow tensor. The signature_name in the options proto determines
+// which named_signature is used. The keys in the generic_signature must be
+// valid MediaPipe tags ([A-Z0-9_]*, no lowercase or special characters). All of
+// the tensors corresponding to tags in the signature for input_streams are fed
+// to the model and for output_streams the tensors are fetched from the model.
+//
+// Other calculators are used to convert data to and from tensors, this op only
+// handles the TensorFlow session and batching. Batching occurs by concatenating
+// input tensors along the 0th dimension across timestamps. If the 0th dimension
+// is not a batch dimension, this calculator will add a 0th dimension by
+// default. Setting add_batch_dim_to_tensors to false disables the dimension
+// addition. Once batch_size inputs have been provided, the batch will be run
+// and the output tensors sent out on the output streams with timestamps
+// corresponding to the input stream packets. Setting the batch_size to 1
+// completely disables batching, but is indepdent of add_batch_dim_to_tensors.
+//
+// The TensorFlowInferenceCalculator also support feeding states recurrently for
+// RNNs and LSTMs. Simply set the recurrent_tag_pair options to define the
+// recurrent tensors. Initializing the recurrent state can be handled by the
+// GraphTensorsPacketGenerator.
+//
+// The calculator updates two Counters to report timing information:
+//   --<name>-TotalTimeUsecs = Total time spent running inference (in usecs),
+//   --<name>-TotalProcessedTimestamps = # of instances processed
+//         (approximately batches processed  * batch_size),
+// where <name> is replaced with CalculatorGraphConfig::Node::name() if it
+// exists, or with TensorFlowInferenceCalculator if the name is not set. The
+// name must be set for timing information to be instance-specific in graphs
+// with multiple TensorFlowInferenceCalculators.
+//
+// Example config:
+//   packet_generator {
+//     packet_generator: "TensorFlowSessionFromSavedModelGenerator"
+//     output_side_packet: "tensorflow_session"
+//     options {
+//       [mediapipe.TensorFlowSessionFromSavedModelGeneratorOptions.ext]: {
+//         saved_model_path: "/path/to/saved/model"
+//         signature_name: "mediapipe"
+//       }
+//     }
+//   }
+//   node {
+//     calculator: "TensorFlowInferenceCalculator"
+//     input_stream: "IMAGES:image_tensors_keyed_in_signature_by_tag"
+//     input_stream: "AUDIO:audio_tensors_keyed_in_signature_by_tag"
+//     output_stream: "LABELS:softmax_tensor_keyed_in_signature_by_tag"
+//     input_side_packet: "SESSION:tensorflow_session"
+//   }
+//
+//   Where the input and output streams are treated as Packet<tf::Tensor> and
+//   the mediapipe_signature has tensor bindings between "IMAGES", "AUDIO", and
+//   "LABELS" and their respective tensors exported to /path/to/bundle. For an
+//   example of how this model was exported, see
+//   tensorflow_inference_test_graph_generator.py
+//
+// It is possible to use a GraphDef proto that was not exported by exporter (i.e
+// without MetaGraph with bindings). Such GraphDef could contain all of its
+// parameters in-lined (for example, it can be the output of freeze_graph.py).
+// To instantiate a TensorFlow model from a GraphDef file, replace the
+// packet_factory above with TensorFlowSessionFromFrozenGraphGenerator:
+//
+//   packet_generator {
+//     packet_generator: "TensorFlowSessionFromFrozenGraphGenerator"
+//     output_side_packet: "SESSION:tensorflow_session"
+//     options {
+//       [mediapipe.TensorFlowSessionFromFrozenGraphGeneratorOptions.ext]: {
+//         graph_proto_path: "[PATH]"
+//         tag_to_tensor_names {
+//           key: "JPG_STRING"
+//           value: "input:0"
+//         }
+//         tag_to_tensor_names {
+//           key: "SOFTMAX"
+//           value: "softmax:0"
+//         }
+//       }
+//     }
+//   }
+//
+// It is also possible to use a GraphDef proto and checkpoint file that have not
+// been frozen. This can be used to load graphs directly as they have been
+// written from training. However, it is more brittle and you are encouraged to
+// use a one of the more perminent formats described above. To instantiate a
+// TensorFlow model from a GraphDef file and checkpoint, replace the
+// packet_factory above with TensorFlowSessionFromModelCheckpointGenerator:
+//
+//   packet_generator {
+//     packet_generator: "TensorFlowSessionFromModelCheckpointGenerator"
+//     output_side_packet: "SESSION:tensorflow_session"
+//     options {
+//       [mediapipe.TensorFlowSessionFromModelCheckpointGeneratorOptions.ext]: {
+//         graph_proto_path: "[PATH]"
+//         model_options {
+//           checkpoint_path: "[PATH2]"
+//         }
+//         tag_to_tensor_names {
+//           key: "JPG_STRING"
+//           value: "input:0"
+//         }
+//         tag_to_tensor_names {
+//           key: "SOFTMAX"
+//           value: "softmax:0"
+//         }
+//       }
+//     }
+//   }
+class TensorFlowInferenceCalculator : public CalculatorBase {
+ public:
+  // Counters for recording timing information. The actual names have the value
+  // of CalculatorGraphConfig::Node::name() prepended.
+  static constexpr char kTotalUsecsCounterSuffix[] = "TotalTimeUsecs";
+  static constexpr char kTotalProcessedTimestampsCounterSuffix[] =
+      "TotalProcessedTimestamps";
+  static constexpr char kTotalSessionRunsTimeUsecsCounterSuffix[] =
+      "TotalSessionRunsTimeUsecs";
+  static constexpr char kTotalNumSessionRunsCounterSuffix[] =
+      "TotalNumSessionRuns";
 
-std::unique_ptr<InferenceState> CreateInferenceState(CalculatorContext* cc)
-    ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
-  std::unique_ptr<InferenceState> inference_state =
-      absl::make_unique<InferenceState>();
-  if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS") &&
-      !cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS").IsEmpty()) {
-    std::map<std::string, tf::Tensor>* init_tensor_map;
-    init_tensor_map = GetFromUniquePtr<std::map<std::string, tf::Tensor>>(
-        cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS"));
-    for (const auto& p : *init_tensor_map) {
-      inference_state->input_tensor_batches_[p.first].emplace_back(p.second);
+  TensorFlowInferenceCalculator() : session_(nullptr) {
+    clock_ = std::unique_ptr<mediapipe::Clock>(
+        mediapipe::MonotonicClock::CreateSynchronizedMonotonicClock());
+  }
+
+  static absl::Status GetContract(CalculatorContract* cc) {
+    const auto& options = cc->Options<TensorFlowInferenceCalculatorOptions>();
+    RET_CHECK(!cc->Inputs().GetTags().empty());
+    for (const std::string& tag : cc->Inputs().GetTags()) {
+      // The tensorflow::Tensor with the tag equal to the graph node. May
+      // have a TimeSeriesHeader if all present TimeSeriesHeaders match.
+      if (!options.batched_input()) {
+        cc->Inputs().Tag(tag).Set<tf::Tensor>();
+      } else {
+        cc->Inputs().Tag(tag).Set<std::vector<mediapipe::Packet>>();
+      }
     }
-  }
-  return inference_state;
-}
-
-absl::Status Open(CalculatorContext* cc) override {
-  options_ = cc->Options<TensorFlowInferenceCalculatorOptions>();
-
-  RET_CHECK(cc->InputSidePackets().HasTag("SESSION"));
-  session_ = cc->InputSidePackets()
-                 .Tag("SESSION")
-                 .Get<TensorFlowSession>()
-                 .session.get();
-  tag_to_tensor_map_ = cc->InputSidePackets()
-                           .Tag("SESSION")
-                           .Get<TensorFlowSession>()
-                           .tag_to_tensor_map;
-
-  // Validate and store the recurrent tags
-  RET_CHECK(options_.has_batch_size());
-  RET_CHECK(options_.batch_size() == 1 || options_.recurrent_tag_pair().empty())
-      << "To use recurrent_tag_pairs, batch_size must be 1.";
-  for (const auto& tag_pair : options_.recurrent_tag_pair()) {
-    const std::vector<std::string> tags = absl::StrSplit(tag_pair, ':');
-    RET_CHECK_EQ(tags.size(), 2)
-        << "recurrent_tag_pair must be a colon "
-           "separated std::string with two components: "
-        << tag_pair;
-    RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tags[0]))
-        << "Can't find tag '" << tags[0] << "' in signature "
-        << options_.signature_name();
-    RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tags[1]))
-        << "Can't find tag '" << tags[1] << "' in signature "
-        << options_.signature_name();
-    recurrent_feed_tags_.insert(tags[0]);
-    recurrent_fetch_tags_to_feed_tags_[tags[1]] = tags[0];
-  }
-
-  // Check that all tags are present in this signature bound to tensors.
-  for (const std::string& tag : cc->Inputs().GetTags()) {
-    RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tag))
-        << "Can't find tag '" << tag << "' in signature "
-        << options_.signature_name();
-  }
-  for (const std::string& tag : cc->Outputs().GetTags()) {
-    RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tag))
-        << "Can't find tag '" << tag << "' in signature "
-        << options_.signature_name();
-  }
-
-  {
-    absl::WriterMutexLock l(&mutex_);
-    inference_state_ = std::unique_ptr<InferenceState>();
-  }
-
-  if (options_.batch_size() == 1 || options_.batched_input()) {
-    cc->SetOffset(0);
-  }
-
-  return absl::OkStatus();
-}
-
-// Adds a batch dimension to the input tensor if specified in the calculator
-// options.
-absl::Status AddBatchDimension(tf::Tensor* input_tensor) {
-  if (options_.add_batch_dim_to_tensors()) {
-    tf::TensorShape new_shape(input_tensor->shape());
-    new_shape.InsertDim(0, 1);
-    RET_CHECK(input_tensor->CopyFrom(*input_tensor, new_shape))
-        << "Could not add 0th dimension to tensor without changing its shape."
-        << " Current shape: " << input_tensor->shape().DebugString();
-  }
-  return absl::OkStatus();
-}
-
-absl::Status AggregateTensorPacket(
-    const std::string& tag_name, const Packet& packet,
-    std::map<Timestamp, std::map<std::string, tf::Tensor>>*
-        input_tensors_by_tag_by_timestamp,
-    InferenceState* inference_state) ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
-  tf::Tensor input_tensor(packet.Get<tf::Tensor>());
-  RET_CHECK_OK(AddBatchDimension(&input_tensor));
-  if (mediapipe::ContainsKey(recurrent_feed_tags_, tag_name)) {
-    // If we receive an input on a recurrent tag, override the state.
-    // It's OK to override the global state because there is just one
-    // input stream allowed for recurrent tensors.
-    inference_state_->input_tensor_batches_[tag_name].clear();
-  }
-  (*input_tensors_by_tag_by_timestamp)[packet.Timestamp()].insert(
-      std::make_pair(tag_name, input_tensor));
-  return absl::OkStatus();
-}
-
-// Removes the batch dimension of the output tensor if specified in the
-// calculator options.
-absl::Status RemoveBatchDimension(tf::Tensor* output_tensor) {
-  if (options_.add_batch_dim_to_tensors()) {
-    tf::TensorShape new_shape(output_tensor->shape());
-    new_shape.RemoveDim(0);
-    RET_CHECK(output_tensor->CopyFrom(*output_tensor, new_shape))
-        << "Could not remove 0th dimension from tensor without changing its "
-        << "shape. Current shape: " << output_tensor->shape().DebugString()
-        << " (The expected first dimension is 1 for a batch element.)";
-  }
-  return absl::OkStatus();
-}
-
-absl::Status Process(CalculatorContext* cc) override {
-  std::unique_ptr<InferenceState> inference_state_to_process;
-  {
-    absl::WriterMutexLock l(&mutex_);
-    if (inference_state_ == nullptr) {
-      inference_state_ = CreateInferenceState(cc);
+    RET_CHECK(!cc->Outputs().GetTags().empty());
+    for (const std::string& tag : cc->Outputs().GetTags()) {
+      // The tensorflow::Tensor with tag equal to the graph node to
+      // output.  Any TimeSeriesHeader from the inputs will be forwarded
+      // with channels set to 0.
+      cc->Outputs().Tag(tag).Set<tf::Tensor>();
     }
-    std::map<Timestamp, std::map<std::string, tf::Tensor>>
-        input_tensors_by_tag_by_timestamp;
-    for (const std::string& tag_as_node_name : cc->Inputs().GetTags()) {
-      if (cc->Inputs().Tag(tag_as_node_name).IsEmpty()) {
-        // Recurrent tensors can be empty.
-        if (!mediapipe::ContainsKey(recurrent_feed_tags_, tag_as_node_name)) {
-          if (options_.skip_on_missing_features()) {
-            return absl::OkStatus();
-          } else {
-            return absl::InvalidArgumentError(absl::StrCat(
-                "Tag ", tag_as_node_name,
-                " not present at timestamp: ", cc->InputTimestamp().Value()));
+    // A mediapipe::TensorFlowSession with a model loaded and ready for use.
+    // For this calculator it must include a tag_to_tensor_map.
+    cc->InputSidePackets().Tag("SESSION").Set<TensorFlowSession>();
+    if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS")) {
+      cc->InputSidePackets()
+          .Tag("RECURRENT_INIT_TENSORS")
+          .Set<std::unique_ptr<std::map<std::string, tf::Tensor>>>();
+    }
+    return absl::OkStatus();
+  }
+
+  std::unique_ptr<InferenceState> CreateInferenceState(CalculatorContext* cc)
+      ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
+    std::unique_ptr<InferenceState> inference_state =
+        absl::make_unique<InferenceState>();
+    if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS") &&
+        !cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS").IsEmpty()) {
+      std::map<std::string, tf::Tensor>* init_tensor_map;
+      init_tensor_map = GetFromUniquePtr<std::map<std::string, tf::Tensor>>(
+          cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS"));
+      for (const auto& p : *init_tensor_map) {
+        inference_state->input_tensor_batches_[p.first].emplace_back(p.second);
+      }
+    }
+    return inference_state;
+  }
+
+  absl::Status Open(CalculatorContext* cc) override {
+    options_ = cc->Options<TensorFlowInferenceCalculatorOptions>();
+
+    RET_CHECK(cc->InputSidePackets().HasTag("SESSION"));
+    session_ = cc->InputSidePackets()
+                   .Tag("SESSION")
+                   .Get<TensorFlowSession>()
+                   .session.get();
+    tag_to_tensor_map_ = cc->InputSidePackets()
+                             .Tag("SESSION")
+                             .Get<TensorFlowSession>()
+                             .tag_to_tensor_map;
+
+    // Validate and store the recurrent tags
+    RET_CHECK(options_.has_batch_size());
+    RET_CHECK(options_.batch_size() == 1 ||
+              options_.recurrent_tag_pair().empty())
+        << "To use recurrent_tag_pairs, batch_size must be 1.";
+    for (const auto& tag_pair : options_.recurrent_tag_pair()) {
+      const std::vector<std::string> tags = absl::StrSplit(tag_pair, ':');
+      RET_CHECK_EQ(tags.size(), 2)
+          << "recurrent_tag_pair must be a colon "
+             "separated std::string with two components: "
+          << tag_pair;
+      RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tags[0]))
+          << "Can't find tag '" << tags[0] << "' in signature "
+          << options_.signature_name();
+      RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tags[1]))
+          << "Can't find tag '" << tags[1] << "' in signature "
+          << options_.signature_name();
+      recurrent_feed_tags_.insert(tags[0]);
+      recurrent_fetch_tags_to_feed_tags_[tags[1]] = tags[0];
+    }
+
+    // Check that all tags are present in this signature bound to tensors.
+    for (const std::string& tag : cc->Inputs().GetTags()) {
+      RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tag))
+          << "Can't find tag '" << tag << "' in signature "
+          << options_.signature_name();
+    }
+    for (const std::string& tag : cc->Outputs().GetTags()) {
+      RET_CHECK(mediapipe::ContainsKey(tag_to_tensor_map_, tag))
+          << "Can't find tag '" << tag << "' in signature "
+          << options_.signature_name();
+    }
+
+    {
+      absl::WriterMutexLock l(&mutex_);
+      inference_state_ = std::unique_ptr<InferenceState>();
+    }
+
+    if (options_.batch_size() == 1 || options_.batched_input()) {
+      cc->SetOffset(0);
+    }
+
+    return absl::OkStatus();
+  }
+
+  // Adds a batch dimension to the input tensor if specified in the calculator
+  // options.
+  absl::Status AddBatchDimension(tf::Tensor* input_tensor) {
+    if (options_.add_batch_dim_to_tensors()) {
+      tf::TensorShape new_shape(input_tensor->shape());
+      new_shape.InsertDim(0, 1);
+      RET_CHECK(input_tensor->CopyFrom(*input_tensor, new_shape))
+          << "Could not add 0th dimension to tensor without changing its shape."
+          << " Current shape: " << input_tensor->shape().DebugString();
+    }
+    return absl::OkStatus();
+  }
+
+  absl::Status AggregateTensorPacket(
+      const std::string& tag_name, const Packet& packet,
+      std::map<Timestamp, std::map<std::string, tf::Tensor>>*
+          input_tensors_by_tag_by_timestamp,
+      InferenceState* inference_state) ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
+    tf::Tensor input_tensor(packet.Get<tf::Tensor>());
+    RET_CHECK_OK(AddBatchDimension(&input_tensor));
+    if (mediapipe::ContainsKey(recurrent_feed_tags_, tag_name)) {
+      // If we receive an input on a recurrent tag, override the state.
+      // It's OK to override the global state because there is just one
+      // input stream allowed for recurrent tensors.
+      inference_state_->input_tensor_batches_[tag_name].clear();
+    }
+    (*input_tensors_by_tag_by_timestamp)[packet.Timestamp()].insert(
+        std::make_pair(tag_name, input_tensor));
+    return absl::OkStatus();
+  }
+
+  // Removes the batch dimension of the output tensor if specified in the
+  // calculator options.
+  absl::Status RemoveBatchDimension(tf::Tensor* output_tensor) {
+    if (options_.add_batch_dim_to_tensors()) {
+      tf::TensorShape new_shape(output_tensor->shape());
+      new_shape.RemoveDim(0);
+      RET_CHECK(output_tensor->CopyFrom(*output_tensor, new_shape))
+          << "Could not remove 0th dimension from tensor without changing its "
+          << "shape. Current shape: " << output_tensor->shape().DebugString()
+          << " (The expected first dimension is 1 for a batch element.)";
+    }
+    return absl::OkStatus();
+  }
+
+  absl::Status Process(CalculatorContext* cc) override {
+    std::unique_ptr<InferenceState> inference_state_to_process;
+    {
+      absl::WriterMutexLock l(&mutex_);
+      if (inference_state_ == nullptr) {
+        inference_state_ = CreateInferenceState(cc);
+      }
+      std::map<Timestamp, std::map<std::string, tf::Tensor>>
+          input_tensors_by_tag_by_timestamp;
+      for (const std::string& tag_as_node_name : cc->Inputs().GetTags()) {
+        if (cc->Inputs().Tag(tag_as_node_name).IsEmpty()) {
+          // Recurrent tensors can be empty.
+          if (!mediapipe::ContainsKey(recurrent_feed_tags_, tag_as_node_name)) {
+            if (options_.skip_on_missing_features()) {
+              return absl::OkStatus();
+            } else {
+              return absl::InvalidArgumentError(absl::StrCat(
+                  "Tag ", tag_as_node_name,
+                  " not present at timestamp: ", cc->InputTimestamp().Value()));
+            }
           }
+        } else if (options_.batched_input()) {
+          const auto& tensor_packets =
+              cc->Inputs().Tag(tag_as_node_name).Get<std::vector<Packet>>();
+          if (tensor_packets.size() > options_.batch_size()) {
+            return absl::InvalidArgumentError(absl::StrCat(
+                "Batch for tag ", tag_as_node_name,
+                " has more packets than batch capacity. batch_size: ",
+                options_.batch_size(), " packets: ", tensor_packets.size()));
+          }
+          for (const auto& packet : tensor_packets) {
+            RET_CHECK_OK(AggregateTensorPacket(
+                tag_as_node_name, packet, &input_tensors_by_tag_by_timestamp,
+                inference_state_.get()));
+          }
+        } else {
+          RET_CHECK_OK(AggregateTensorPacket(
+              tag_as_node_name, cc->Inputs().Tag(tag_as_node_name).Value(),
+              &input_tensors_by_tag_by_timestamp, inference_state_.get()));
         }
-      } else if (options_.batched_input()) {
-        const auto& tensor_packets =
-            cc->Inputs().Tag(tag_as_node_name).Get<std::vector<Packet>>();
-        if (tensor_packets.size() > options_.batch_size()) {
-          return absl::InvalidArgumentError(absl::StrCat(
-              "Batch for tag ", tag_as_node_name,
-              " has more packets than batch capacity. batch_size: ",
-              options_.batch_size(), " packets: ", tensor_packets.size()));
+      }
+      for (const auto& timestamp_and_input_tensors_by_tag :
+           input_tensors_by_tag_by_timestamp) {
+        inference_state_->batch_timestamps_.emplace_back(
+            timestamp_and_input_tensors_by_tag.first);
+        for (const auto& input_tensor_and_tag :
+             timestamp_and_input_tensors_by_tag.second) {
+          inference_state_->input_tensor_batches_[input_tensor_and_tag.first]
+              .emplace_back(input_tensor_and_tag.second);
         }
-        for (const auto& packet : tensor_packets) {
-          RET_CHECK_OK(AggregateTensorPacket(tag_as_node_name, packet,
-                                             &input_tensors_by_tag_by_timestamp,
-                                             inference_state_.get()));
+      }
+      if (inference_state_->batch_timestamps_.size() == options_.batch_size() ||
+          options_.batched_input()) {
+        inference_state_to_process = std::move(inference_state_);
+        inference_state_ = std::unique_ptr<InferenceState>();
+      }
+    }
+
+    if (inference_state_to_process) {
+      MP_RETURN_IF_ERROR(
+          OutputBatch(cc, std::move(inference_state_to_process)));
+    }
+
+    return absl::OkStatus();
+  }
+
+  absl::Status Close(CalculatorContext* cc) override {
+    std::unique_ptr<InferenceState> inference_state_to_process = nullptr;
+    {
+      absl::WriterMutexLock l(&mutex_);
+      if (cc->GraphStatus().ok() && inference_state_ != nullptr &&
+          !inference_state_->batch_timestamps_.empty()) {
+        inference_state_to_process = std::move(inference_state_);
+        inference_state_ = std::unique_ptr<InferenceState>();
+      }
+    }
+    if (inference_state_to_process) {
+      MP_RETURN_IF_ERROR(
+          OutputBatch(cc, std::move(inference_state_to_process)));
+    }
+    return absl::OkStatus();
+  }
+
+  // When a batch of input tensors is ready to be run, runs TensorFlow and
+  // outputs the output tensors. The output tensors have timestamps matching
+  // the input tensor that formed that batch element. Any requested
+  // batch_dimension is added and removed. This code takes advantage of the fact
+  // that copying a tensor shares the same reference-counted, heap allocated
+  // memory buffer. Therefore, copies are cheap and should not cause the memory
+  // buffer to fall out of scope. In contrast, concat is only used where
+  // necessary.
+  absl::Status OutputBatch(CalculatorContext* cc,
+                           std::unique_ptr<InferenceState> inference_state) {
+    const int64 start_time = absl::ToUnixMicros(clock_->TimeNow());
+    std::vector<std::pair<mediapipe::ProtoString, tf::Tensor>> input_tensors;
+
+    for (auto& keyed_tensors : inference_state->input_tensor_batches_) {
+      if (options_.batch_size() == 1) {
+        // Short circuit to avoid the cost of deep copying tensors in concat.
+        if (!keyed_tensors.second.empty()) {
+          input_tensors.emplace_back(tag_to_tensor_map_[keyed_tensors.first],
+                                     keyed_tensors.second[0]);
+        } else {
+          // The input buffer can be empty for recurrent tensors.
+          RET_CHECK(
+              mediapipe::ContainsKey(recurrent_feed_tags_, keyed_tensors.first))
+              << "A non-recurrent tensor does not have an input: "
+              << keyed_tensors.first;
         }
       } else {
-        RET_CHECK_OK(AggregateTensorPacket(
-            tag_as_node_name, cc->Inputs().Tag(tag_as_node_name).Value(),
-            &input_tensors_by_tag_by_timestamp, inference_state_.get()));
-      }
-    }
-    for (const auto& timestamp_and_input_tensors_by_tag :
-         input_tensors_by_tag_by_timestamp) {
-      inference_state_->batch_timestamps_.emplace_back(
-          timestamp_and_input_tensors_by_tag.first);
-      for (const auto& input_tensor_and_tag :
-           timestamp_and_input_tensors_by_tag.second) {
-        inference_state_->input_tensor_batches_[input_tensor_and_tag.first]
-            .emplace_back(input_tensor_and_tag.second);
-      }
-    }
-    if (inference_state_->batch_timestamps_.size() == options_.batch_size() ||
-        options_.batched_input()) {
-      inference_state_to_process = std::move(inference_state_);
-      inference_state_ = std::unique_ptr<InferenceState>();
-    }
-  }
-
-  if (inference_state_to_process) {
-    MP_RETURN_IF_ERROR(OutputBatch(cc, std::move(inference_state_to_process)));
-  }
-
-  return absl::OkStatus();
-}
-
-absl::Status Close(CalculatorContext* cc) override {
-  std::unique_ptr<InferenceState> inference_state_to_process = nullptr;
-  {
-    absl::WriterMutexLock l(&mutex_);
-    if (cc->GraphStatus().ok() && inference_state_ != nullptr &&
-        !inference_state_->batch_timestamps_.empty()) {
-      inference_state_to_process = std::move(inference_state_);
-      inference_state_ = std::unique_ptr<InferenceState>();
-    }
-  }
-  if (inference_state_to_process) {
-    MP_RETURN_IF_ERROR(OutputBatch(cc, std::move(inference_state_to_process)));
-  }
-  return absl::OkStatus();
-}
-
-// When a batch of input tensors is ready to be run, runs TensorFlow and
-// outputs the output tensors. The output tensors have timestamps matching
-// the input tensor that formed that batch element. Any requested
-// batch_dimension is added and removed. This code takes advantage of the fact
-// that copying a tensor shares the same reference-counted, heap allocated
-// memory buffer. Therefore, copies are cheap and should not cause the memory
-// buffer to fall out of scope. In contrast, concat is only used where
-// necessary.
-absl::Status OutputBatch(CalculatorContext* cc,
-                         std::unique_ptr<InferenceState> inference_state) {
-  const int64 start_time = absl::ToUnixMicros(clock_->TimeNow());
-  std::vector<std::pair<mediapipe::ProtoString, tf::Tensor>> input_tensors;
-
-  for (auto& keyed_tensors : inference_state->input_tensor_batches_) {
-    if (options_.batch_size() == 1) {
-      // Short circuit to avoid the cost of deep copying tensors in concat.
-      if (!keyed_tensors.second.empty()) {
+        // Pad by replicating the first tens  or, then ignore the values.
+        keyed_tensors.second.resize(options_.batch_size());
+        std::fill(keyed_tensors.second.begin() +
+                      inference_state->batch_timestamps_.size(),
+                  keyed_tensors.second.end(), keyed_tensors.second[0]);
+        tf::Tensor concated;
+        const tf::Status concat_status =
+            tf::tensor::Concat(keyed_tensors.second, &concated);
+        CHECK(concat_status.ok()) << concat_status.ToString();
         input_tensors.emplace_back(tag_to_tensor_map_[keyed_tensors.first],
-                                   keyed_tensors.second[0]);
-      } else {
-        // The input buffer can be empty for recurrent tensors.
-        RET_CHECK(
-            mediapipe::ContainsKey(recurrent_feed_tags_, keyed_tensors.first))
-            << "A non-recurrent tensor does not have an input: "
-            << keyed_tensors.first;
+                                   concated);
       }
-    } else {
-      // Pad by replicating the first tens  or, then ignore the values.
-      keyed_tensors.second.resize(options_.batch_size());
-      std::fill(keyed_tensors.second.begin() +
-                    inference_state->batch_timestamps_.size(),
-                keyed_tensors.second.end(), keyed_tensors.second[0]);
-      tf::Tensor concated;
-      const tf::Status concat_status =
-          tf::tensor::Concat(keyed_tensors.second, &concated);
-      CHECK(concat_status.ok()) << concat_status.ToString();
-      input_tensors.emplace_back(tag_to_tensor_map_[keyed_tensors.first],
-                                 concated);
     }
-  }
-  inference_state->input_tensor_batches_.clear();
-  std::vector<mediapipe::ProtoString> output_tensor_names;
-  std::vector<std::string> output_name_in_signature;
-  for (const std::string& tag : cc->Outputs().GetTags()) {
-    output_tensor_names.emplace_back(tag_to_tensor_map_[tag]);
-    output_name_in_signature.emplace_back(tag);
-  }
-  for (const auto& tag_pair : recurrent_fetch_tags_to_feed_tags_) {
-    // Ensure that we always fetch the recurrent state tensors.
-    if (std::find(output_name_in_signature.begin(),
-                  output_name_in_signature.end(),
-                  tag_pair.first) == output_name_in_signature.end()) {
-      output_tensor_names.emplace_back(tag_to_tensor_map_[tag_pair.first]);
-      output_name_in_signature.emplace_back(tag_pair.first);
+    inference_state->input_tensor_batches_.clear();
+    std::vector<mediapipe::ProtoString> output_tensor_names;
+    std::vector<std::string> output_name_in_signature;
+    for (const std::string& tag : cc->Outputs().GetTags()) {
+      output_tensor_names.emplace_back(tag_to_tensor_map_[tag]);
+      output_name_in_signature.emplace_back(tag);
     }
-  }
-  std::vector<tf::Tensor> outputs;
+    for (const auto& tag_pair : recurrent_fetch_tags_to_feed_tags_) {
+      // Ensure that we always fetch the recurrent state tensors.
+      if (std::find(output_name_in_signature.begin(),
+                    output_name_in_signature.end(),
+                    tag_pair.first) == output_name_in_signature.end()) {
+        output_tensor_names.emplace_back(tag_to_tensor_map_[tag_pair.first]);
+        output_name_in_signature.emplace_back(tag_pair.first);
+      }
+    }
+    std::vector<tf::Tensor> outputs;
 
-  SimpleSemaphore* session_run_throttle = nullptr;
-  if (options_.max_concurrent_session_runs() > 0) {
-    session_run_throttle =
-        get_session_run_throttle(options_.max_concurrent_session_runs());
-    session_run_throttle->Acquire(1);
-  }
-  const int64 run_start_time = absl::ToUnixMicros(clock_->TimeNow());
-  tf::Status tf_status;
-  {
+    SimpleSemaphore* session_run_throttle = nullptr;
+    if (options_.max_concurrent_session_runs() > 0) {
+      session_run_throttle =
+          get_session_run_throttle(options_.max_concurrent_session_runs());
+      session_run_throttle->Acquire(1);
+    }
+    const int64 run_start_time = absl::ToUnixMicros(clock_->TimeNow());
+    tf::Status tf_status;
+    {
 #if !defined(MEDIAPIPE_MOBILE) && !defined(__APPLE__)
-    tensorflow::profiler::TraceMe trace(absl::string_view(cc->NodeName()));
+      tensorflow::profiler::TraceMe trace(absl::string_view(cc->NodeName()));
 #endif
-    tf_status = session_->Run(input_tensors, output_tensor_names,
-                              {} /* target_node_names */, &outputs);
-  }
+      tf_status = session_->Run(input_tensors, output_tensor_names,
+                                {} /* target_node_names */, &outputs);
+    }
 
-  if (session_run_throttle != nullptr) {
-    session_run_throttle->Release(1);
-  }
+    if (session_run_throttle != nullptr) {
+      session_run_throttle->Release(1);
+    }
 
-  // RET_CHECK on the tf::Status object itself in order to print an
-  // informative error message.
-  RET_CHECK(tf_status.ok()) << "Run failed: " << tf_status.ToString();
+    // RET_CHECK on the tf::Status object itself in order to print an
+    // informative error message.
+    RET_CHECK(tf_status.ok()) << "Run failed: " << tf_status.ToString();
 
-  const int64 run_end_time = absl::ToUnixMicros(clock_->TimeNow());
-  cc->GetCounter(kTotalSessionRunsTimeUsecsCounterSuffix)
-      ->IncrementBy(run_end_time - run_start_time);
-  cc->GetCounter(kTotalNumSessionRunsCounterSuffix)->Increment();
+    const int64 run_end_time = absl::ToUnixMicros(clock_->TimeNow());
+    cc->GetCounter(kTotalSessionRunsTimeUsecsCounterSuffix)
+        ->IncrementBy(run_end_time - run_start_time);
+    cc->GetCounter(kTotalNumSessionRunsCounterSuffix)->Increment();
 
-  // Feed back the recurrent state.
-  for (const auto& tag_pair : recurrent_fetch_tags_to_feed_tags_) {
-    int pos = std::find(output_name_in_signature.begin(),
-                        output_name_in_signature.end(), tag_pair.first) -
-              output_name_in_signature.begin();
-    inference_state->input_tensor_batches_[tag_pair.second].emplace_back(
-        outputs[pos]);
-  }
+    // Feed back the recurrent state.
+    for (const auto& tag_pair : recurrent_fetch_tags_to_feed_tags_) {
+      int pos = std::find(output_name_in_signature.begin(),
+                          output_name_in_signature.end(), tag_pair.first) -
+                output_name_in_signature.begin();
+      inference_state->input_tensor_batches_[tag_pair.second].emplace_back(
+          outputs[pos]);
+    }
 
-  absl::WriterMutexLock l(&mutex_);
-  // Set that we want to split on each index of the 0th dimension.
-  std::vector<tf::int64> split_vector(options_.batch_size(), 1);
-  for (int i = 0; i < output_tensor_names.size(); ++i) {
-    if (options_.batch_size() == 1) {
-      if (cc->Outputs().HasTag(output_name_in_signature[i])) {
-        tf::Tensor output_tensor(outputs[i]);
-        RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
-        cc->Outputs()
-            .Tag(output_name_in_signature[i])
-            .Add(new tf::Tensor(output_tensor),
-                 inference_state->batch_timestamps_[0]);
-      }
-    } else {
-      std::vector<tf::Tensor> split_tensors;
-      const tf::Status split_status =
-          tf::tensor::Split(outputs[i], split_vector, &split_tensors);
-      CHECK(split_status.ok()) << split_status.ToString();
-      // Loop over timestamps so that we don't copy the padding.
-      for (int j = 0; j < inference_state->batch_timestamps_.size(); ++j) {
-        tf::Tensor output_tensor(split_tensors[j]);
-        RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
-        cc->Outputs()
-            .Tag(output_name_in_signature[i])
-            .Add(new tf::Tensor(output_tensor),
-                 inference_state->batch_timestamps_[j]);
+    absl::WriterMutexLock l(&mutex_);
+    // Set that we want to split on each index of the 0th dimension.
+    std::vector<tf::int64> split_vector(options_.batch_size(), 1);
+    for (int i = 0; i < output_tensor_names.size(); ++i) {
+      if (options_.batch_size() == 1) {
+        if (cc->Outputs().HasTag(output_name_in_signature[i])) {
+          tf::Tensor output_tensor(outputs[i]);
+          RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
+          cc->Outputs()
+              .Tag(output_name_in_signature[i])
+              .Add(new tf::Tensor(output_tensor),
+                   inference_state->batch_timestamps_[0]);
+        }
+      } else {
+        std::vector<tf::Tensor> split_tensors;
+        const tf::Status split_status =
+            tf::tensor::Split(outputs[i], split_vector, &split_tensors);
+        CHECK(split_status.ok()) << split_status.ToString();
+        // Loop over timestamps so that we don't copy the padding.
+        for (int j = 0; j < inference_state->batch_timestamps_.size(); ++j) {
+          tf::Tensor output_tensor(split_tensors[j]);
+          RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
+          cc->Outputs()
+              .Tag(output_name_in_signature[i])
+              .Add(new tf::Tensor(output_tensor),
+                   inference_state->batch_timestamps_[j]);
+        }
       }
     }
+
+    // Get end time and report.
+    const int64 end_time = absl::ToUnixMicros(clock_->TimeNow());
+    cc->GetCounter(kTotalUsecsCounterSuffix)
+        ->IncrementBy(end_time - start_time);
+    cc->GetCounter(kTotalProcessedTimestampsCounterSuffix)
+        ->IncrementBy(inference_state->batch_timestamps_.size());
+
+    // Make sure we hold on to the recursive state.
+    if (!options_.recurrent_tag_pair().empty()) {
+      inference_state_ = std::move(inference_state);
+      inference_state_->batch_timestamps_.clear();
+    }
+
+    return absl::OkStatus();
   }
 
-  // Get end time and report.
-  const int64 end_time = absl::ToUnixMicros(clock_->TimeNow());
-  cc->GetCounter(kTotalUsecsCounterSuffix)->IncrementBy(end_time - start_time);
-  cc->GetCounter(kTotalProcessedTimestampsCounterSuffix)
-      ->IncrementBy(inference_state->batch_timestamps_.size());
+ private:
+  // The Session object is provided by a packet factory and is owned by the
+  // MediaPipe framework. Individual calls are thread-safe, but session state
+  // may be shared across threads.
+  tf::Session* session_;
 
-  // Make sure we hold on to the recursive state.
-  if (!options_.recurrent_tag_pair().empty()) {
-    inference_state_ = std::move(inference_state);
-    inference_state_->batch_timestamps_.clear();
+  // A mapping between stream tags and the tensor names they are bound to.
+  std::map<std::string, std::string> tag_to_tensor_map_;
+
+  absl::Mutex mutex_;
+  std::unique_ptr<InferenceState> inference_state_ ABSL_GUARDED_BY(mutex_);
+
+  // The options for the calculator.
+  TensorFlowInferenceCalculatorOptions options_;
+
+  // Store the feed and fetch tags for feed/fetch recurrent networks.
+  std::set<std::string> recurrent_feed_tags_;
+  std::map<std::string, std::string> recurrent_fetch_tags_to_feed_tags_;
+
+  // Clock used to measure the computation time in OutputBatch().
+  std::unique_ptr<mediapipe::Clock> clock_;
+
+  // The static singleton semaphore to throttle concurrent session runs.
+  static SimpleSemaphore* get_session_run_throttle(
+      int32 max_concurrent_session_runs) {
+    static SimpleSemaphore* session_run_throttle =
+        new SimpleSemaphore(max_concurrent_session_runs);
+    return session_run_throttle;
   }
-
-  return absl::OkStatus();
-}
-
-private:
-// The Session object is provided by a packet factory and is owned by the
-// MediaPipe framework. Individual calls are thread-safe, but session state may
-// be shared across threads.
-tf::Session* session_;
-
-// A mapping between stream tags and the tensor names they are bound to.
-std::map<std::string, std::string> tag_to_tensor_map_;
-
-absl::Mutex mutex_;
-std::unique_ptr<InferenceState> inference_state_ ABSL_GUARDED_BY(mutex_);
-
-// The options for the calculator.
-TensorFlowInferenceCalculatorOptions options_;
-
-// Store the feed and fetch tags for feed/fetch recurrent networks.
-std::set<std::string> recurrent_feed_tags_;
-std::map<std::string, std::string> recurrent_fetch_tags_to_feed_tags_;
-
-// Clock used to measure the computation time in OutputBatch().
-std::unique_ptr<mediapipe::Clock> clock_;
-
-// The static singleton semaphore to throttle concurrent session runs.
-static SimpleSemaphore* get_session_run_throttle(
-    int32 max_concurrent_session_runs) {
-  static SimpleSemaphore* session_run_throttle =
-      new SimpleSemaphore(max_concurrent_session_runs);
-  return session_run_throttle;
-}
-}
-;
+};
 REGISTER_CALCULATOR(TensorFlowInferenceCalculator);
 
 constexpr char TensorFlowInferenceCalculator::kTotalUsecsCounterSuffix[];

mediapipe/calculators/tensorflow/tensorflow_session_from_saved_model_calculator.cc

@@ -80,6 +80,7 @@ const std::string MaybeConvertSignatureToTag(
 // which in turn contains a TensorFlow Session ready for execution and a map
 // between tags and tensor names.
 //
+//
 // Example usage:
 // node {
 //   calculator: "TensorFlowSessionFromSavedModelCalculator"

mediapipe/calculators/tensorflow/unpack_media_sequence_calculator.cc

@@ -217,38 +217,41 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
             first_timestamp_seen_ = recent_timestamp;
           }
         }
-        if (recent_timestamp > last_timestamp_seen) {
+        if (recent_timestamp > last_timestamp_seen &&
+            recent_timestamp < Timestamp::PostStream().Value()) {
           last_timestamp_key_ = map_kv.first;
           last_timestamp_seen = recent_timestamp;
         }
       }
     }
     if (!timestamps_.empty()) {
-      RET_CHECK(!last_timestamp_key_.empty())
-          << "Something went wrong because the timestamp key is unset. "
-             "Example: "
-          << sequence_->DebugString();
-      RET_CHECK_GT(last_timestamp_seen, Timestamp::PreStream().Value())
-          << "Something went wrong because the last timestamp is unset. "
-             "Example: "
-          << sequence_->DebugString();
-      RET_CHECK_LT(first_timestamp_seen_,
-                   Timestamp::OneOverPostStream().Value())
-          << "Something went wrong because the first timestamp is unset. "
-             "Example: "
-          << sequence_->DebugString();
+      for (const auto& kv : timestamps_) {
+        if (!kv.second.empty() &&
+            kv.second[0] < Timestamp::PostStream().Value()) {
+          // These checks only make sense if any values are not PostStream, but
+          // only need to be made once.
+          RET_CHECK(!last_timestamp_key_.empty())
+              << "Something went wrong because the timestamp key is unset. "
+              << "Example: " << sequence_->DebugString();
+          RET_CHECK_GT(last_timestamp_seen, Timestamp::PreStream().Value())
+              << "Something went wrong because the last timestamp is unset. "
+              << "Example: " << sequence_->DebugString();
+          RET_CHECK_LT(first_timestamp_seen_,
+                       Timestamp::OneOverPostStream().Value())
+              << "Something went wrong because the first timestamp is unset. "
+              << "Example: " << sequence_->DebugString();
+          break;
+        }
+      }
     }
     current_timestamp_index_ = 0;
+    process_poststream_ = false;
 
     // Determine the data path and output it.
     const auto& options = cc->Options<UnpackMediaSequenceCalculatorOptions>();
     const auto& sequence = cc->InputSidePackets()
                                .Tag(kSequenceExampleTag)
                                .Get<tensorflow::SequenceExample>();
-    if (cc->Outputs().HasTag(kKeypointsTag)) {
-      keypoint_names_ = absl::StrSplit(options.keypoint_names(), ',');
-      default_keypoint_location_ = options.default_keypoint_location();
-    }
     if (cc->OutputSidePackets().HasTag(kDataPath)) {
       std::string root_directory = "";
       if (cc->InputSidePackets().HasTag(kDatasetRootDirTag)) {
@@ -349,19 +352,30 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
     // all packets on all streams that have a timestamp between the current
     // reference timestep and the previous reference timestep. This ensures that
     // we emit all timestamps in order, but also only emit a limited number in
-    // any particular call to Process().
-    int64 start_timestamp =
-        timestamps_[last_timestamp_key_][current_timestamp_index_];
-    if (current_timestamp_index_ == 0) {
-      start_timestamp = first_timestamp_seen_;
+    // any particular call to Process(). At the every end, we output the
+    // poststream packets. If we only have poststream packets,
+    // last_timestamp_key_ will be empty.
+    int64 start_timestamp = 0;
+    int64 end_timestamp = 0;
+    if (last_timestamp_key_.empty() || process_poststream_) {
+      process_poststream_ = true;
+      start_timestamp = Timestamp::PostStream().Value();
+      end_timestamp = Timestamp::OneOverPostStream().Value();
+    } else {
+      start_timestamp =
+          timestamps_[last_timestamp_key_][current_timestamp_index_];
+      if (current_timestamp_index_ == 0) {
+        start_timestamp = first_timestamp_seen_;
+      }
+
+      end_timestamp = start_timestamp + 1;  // Base case at end of sequence.
+      if (current_timestamp_index_ <
+          timestamps_[last_timestamp_key_].size() - 1) {
+        end_timestamp =
+            timestamps_[last_timestamp_key_][current_timestamp_index_ + 1];
+      }
     }
 
-    int64 end_timestamp = start_timestamp + 1;  // Base case at end of sequence.
-    if (current_timestamp_index_ <
-        timestamps_[last_timestamp_key_].size() - 1) {
-      end_timestamp =
-          timestamps_[last_timestamp_key_][current_timestamp_index_ + 1];
-    }
     for (const auto& map_kv : timestamps_) {
       for (int i = 0; i < map_kv.second.size(); ++i) {
         if (map_kv.second[i] >= start_timestamp &&
@@ -438,7 +452,14 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
     if (current_timestamp_index_ < timestamps_[last_timestamp_key_].size()) {
       return absl::OkStatus();
     } else {
-      return tool::StatusStop();
+      if (process_poststream_) {
+        // Once we've processed the PostStream timestamp we can stop.
+        return tool::StatusStop();
+      } else {
+        // Otherwise, we still need to do one more pass to process it.
+        process_poststream_ = true;
+        return absl::OkStatus();
+      }
     }
   }
 
@@ -462,6 +483,7 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
   std::vector<std::string> keypoint_names_;
   // Default keypoint location when missing.
   float default_keypoint_location_;
+  bool process_poststream_;
 };
 REGISTER_CALCULATOR(UnpackMediaSequenceCalculator);
 }  // namespace mediapipe

mediapipe/calculators/tensorflow/unpack_media_sequence_calculator_test.cc

@@ -412,6 +412,72 @@ TEST_F(UnpackMediaSequenceCalculatorTest, UnpacksTwoPostStreamFloatLists) {
               ::testing::Eq(Timestamp::PostStream()));
 }
 
+TEST_F(UnpackMediaSequenceCalculatorTest, UnpacksImageWithPostStreamFloatList) {
+  SetUpCalculator({"IMAGE:images"}, {});
+  auto input_sequence = absl::make_unique<tf::SequenceExample>();
+  std::string test_video_id = "test_video_id";
+  mpms::SetClipMediaId(test_video_id, input_sequence.get());
+
+  std::string test_image_string = "test_image_string";
+
+  int num_images = 1;
+  for (int i = 0; i < num_images; ++i) {
+    mpms::AddImageTimestamp(i, input_sequence.get());
+    mpms::AddImageEncoded(test_image_string, input_sequence.get());
+  }
+
+  mpms::AddFeatureFloats("FDENSE_MAX", {3.0f, 4.0f}, input_sequence.get());
+  mpms::AddFeatureTimestamp("FDENSE_MAX", Timestamp::PostStream().Value(),
+                            input_sequence.get());
+
+  runner_->MutableSidePackets()->Tag("SEQUENCE_EXAMPLE") =
+      Adopt(input_sequence.release());
+
+  MP_ASSERT_OK(runner_->Run());
+
+  const std::vector<Packet>& output_packets =
+      runner_->Outputs().Tag("IMAGE").packets;
+  ASSERT_EQ(num_images, output_packets.size());
+
+  for (int i = 0; i < num_images; ++i) {
+    const std::string& output_image = output_packets[i].Get<std::string>();
+    ASSERT_EQ(output_image, test_image_string);
+  }
+}
+
+TEST_F(UnpackMediaSequenceCalculatorTest, UnpacksPostStreamFloatListWithImage) {
+  SetUpCalculator({"FLOAT_FEATURE_FDENSE_MAX:max"}, {});
+  auto input_sequence = absl::make_unique<tf::SequenceExample>();
+  std::string test_video_id = "test_video_id";
+  mpms::SetClipMediaId(test_video_id, input_sequence.get());
+
+  std::string test_image_string = "test_image_string";
+
+  int num_images = 1;
+  for (int i = 0; i < num_images; ++i) {
+    mpms::AddImageTimestamp(i, input_sequence.get());
+    mpms::AddImageEncoded(test_image_string, input_sequence.get());
+  }
+
+  mpms::AddFeatureFloats("FDENSE_MAX", {3.0f, 4.0f}, input_sequence.get());
+  mpms::AddFeatureTimestamp("FDENSE_MAX", Timestamp::PostStream().Value(),
+                            input_sequence.get());
+
+  runner_->MutableSidePackets()->Tag("SEQUENCE_EXAMPLE") =
+      Adopt(input_sequence.release());
+
+  MP_ASSERT_OK(runner_->Run());
+
+  const std::vector<Packet>& fdense_max_packets =
+      runner_->Outputs().Tag("FLOAT_FEATURE_FDENSE_MAX").packets;
+  ASSERT_EQ(fdense_max_packets.size(), 1);
+  const auto& fdense_max_vector =
+      fdense_max_packets[0].Get<std::vector<float>>();
+  ASSERT_THAT(fdense_max_vector, ::testing::ElementsAreArray({3.0f, 4.0f}));
+  ASSERT_THAT(fdense_max_packets[0].Timestamp(),
+              ::testing::Eq(Timestamp::PostStream()));
+}
+
 TEST_F(UnpackMediaSequenceCalculatorTest, GetDatasetFromPacket) {
   SetUpCalculator({}, {"DATA_PATH:data_path"}, {"DATASET_ROOT:root"});
 

mediapipe/calculators/tflite/tflite_inference_calculator.cc

@@ -904,7 +904,8 @@ absl::Status TfLiteInferenceCalculator::LoadDelegate(CalculatorContext* cc) {
 #if MEDIAPIPE_TFLITE_GL_INFERENCE
   // Configure and create the delegate.
   TfLiteGpuDelegateOptions options = TfLiteGpuDelegateOptionsDefault();
-  options.compile_options.precision_loss_allowed = 1;
+  options.compile_options.precision_loss_allowed =
+      allow_precision_loss_ ? 1 : 0;
   options.compile_options.preferred_gl_object_type =
       TFLITE_GL_OBJECT_TYPE_FASTEST;
   options.compile_options.dynamic_batch_enabled = 0;
@@ -968,7 +969,7 @@ absl::Status TfLiteInferenceCalculator::LoadDelegate(CalculatorContext* cc) {
   const int kHalfSize = 2;  // sizeof(half)
   // Configure and create the delegate.
   TFLGpuDelegateOptions options;
-  options.allow_precision_loss = true;
+  options.allow_precision_loss = allow_precision_loss_;
   options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeActive;
   if (!delegate_)
     delegate_ = TfLiteDelegatePtr(TFLGpuDelegateCreate(&options),
@@ -1080,9 +1081,10 @@ absl::Status TfLiteInferenceCalculator::LoadDelegate(CalculatorContext* cc) {
     }
 
     // Create converter for GPU output.
-    converter_from_BPHWC4_ = [[TFLBufferConvert alloc] initWithDevice:device
-                                                            isFloat16:true
-                                                      convertToPBHWC4:false];
+    converter_from_BPHWC4_ =
+        [[TFLBufferConvert alloc] initWithDevice:device
+                                       isFloat16:allow_precision_loss_
+                                 convertToPBHWC4:false];
     if (converter_from_BPHWC4_ == nil) {
       return absl::InternalError(
           "Error initializating output buffer converter");

mediapipe/calculators/tflite/tflite_tensors_to_segmentation_calculator.cc

@@ -439,7 +439,7 @@ absl::Status TfLiteTensorsToSegmentationCalculator::ProcessGpu(
 
   // Run shader, upsample result.
   {
-    gpu_helper_.BindFramebuffer(output_texture);  // GL_TEXTURE0
+    gpu_helper_.BindFramebuffer(output_texture);
     glActiveTexture(GL_TEXTURE1);
     glBindTexture(GL_TEXTURE_2D, small_mask_texture.id());
     GlRender();

mediapipe/calculators/util/BUILD

@@ -821,6 +821,25 @@ cc_library(
     alwayslink = 1,
 )
 
+cc_test(
+    name = "landmark_projection_calculator_test",
+    srcs = ["landmark_projection_calculator_test.cc"],
+    deps = [
+        ":landmark_projection_calculator",
+        "//mediapipe/calculators/tensor:image_to_tensor_utils",
+        "//mediapipe/framework:calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:calculator_runner",
+        "//mediapipe/framework/deps:message_matchers",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/formats:rect_cc_proto",
+        "//mediapipe/framework/port:gtest_main",
+        "//mediapipe/framework/port:parse_text_proto",
+        "@com_google_absl//absl/memory",
+        "@com_google_googletest//:gtest_main",
+    ],
+)
+
 mediapipe_proto_library(
     name = "landmarks_smoothing_calculator_proto",
     srcs = ["landmarks_smoothing_calculator.proto"],
@@ -840,6 +859,7 @@ cc_library(
         "//mediapipe/framework:calculator_framework",
         "//mediapipe/framework:timestamp",
         "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/formats:rect_cc_proto",
         "//mediapipe/framework/port:ret_check",
         "//mediapipe/util/filtering:one_euro_filter",
         "//mediapipe/util/filtering:relative_velocity_filter",
@@ -1252,3 +1272,45 @@ cc_test(
         "//mediapipe/framework/port:parse_text_proto",
     ],
 )
+
+mediapipe_proto_library(
+    name = "refine_landmarks_from_heatmap_calculator_proto",
+    srcs = ["refine_landmarks_from_heatmap_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
+cc_library(
+    name = "refine_landmarks_from_heatmap_calculator",
+    srcs = ["refine_landmarks_from_heatmap_calculator.cc"],
+    hdrs = ["refine_landmarks_from_heatmap_calculator.h"],
+    copts = select({
+        "//mediapipe:apple": [
+            "-x objective-c++",
+            "-fobjc-arc",  # enable reference-counting
+        ],
+        "//conditions:default": [],
+    }),
+    visibility = ["//visibility:public"],
+    deps = [
+        ":refine_landmarks_from_heatmap_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/api2:node",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/formats:tensor",
+        "//mediapipe/framework/port:statusor",
+    ],
+    alwayslink = 1,
+)
+
+cc_test(
+    name = "refine_landmarks_from_heatmap_calculator_test",
+    srcs = ["refine_landmarks_from_heatmap_calculator_test.cc"],
+    deps = [
+        ":refine_landmarks_from_heatmap_calculator",
+        "//mediapipe/framework/port:gtest_main",
+    ],
+)

mediapipe/calculators/util/annotation_overlay_calculator.cc

@@ -402,7 +402,7 @@ absl::Status AnnotationOverlayCalculator::RenderToGpu(CalculatorContext* cc,
 
   // Blend overlay image in GPU shader.
   {
-    gpu_helper_.BindFramebuffer(output_texture);  // GL_TEXTURE0
+    gpu_helper_.BindFramebuffer(output_texture);
 
     glActiveTexture(GL_TEXTURE1);
     glBindTexture(GL_TEXTURE_2D, input_texture.name());

mediapipe/calculators/util/detection_label_id_to_text_calculator.cc

@@ -54,6 +54,7 @@ class DetectionLabelIdToTextCalculator : public CalculatorBase {
 
  private:
   absl::node_hash_map<int, std::string> label_map_;
+  ::mediapipe::DetectionLabelIdToTextCalculatorOptions options_;
 };
 REGISTER_CALCULATOR(DetectionLabelIdToTextCalculator);
 
@@ -68,13 +69,13 @@ absl::Status DetectionLabelIdToTextCalculator::GetContract(
 absl::Status DetectionLabelIdToTextCalculator::Open(CalculatorContext* cc) {
   cc->SetOffset(TimestampDiff(0));
 
-  const auto& options =
+  options_ =
       cc->Options<::mediapipe::DetectionLabelIdToTextCalculatorOptions>();
 
-  if (options.has_label_map_path()) {
+  if (options_.has_label_map_path()) {
     std::string string_path;
     ASSIGN_OR_RETURN(string_path,
-                     PathToResourceAsFile(options.label_map_path()));
+                     PathToResourceAsFile(options_.label_map_path()));
     std::string label_map_string;
     MP_RETURN_IF_ERROR(file::GetContents(string_path, &label_map_string));
 
@@ -85,8 +86,8 @@ absl::Status DetectionLabelIdToTextCalculator::Open(CalculatorContext* cc) {
       label_map_[i++] = line;
     }
   } else {
-    for (int i = 0; i < options.label_size(); ++i) {
-      label_map_[i] = options.label(i);
+    for (int i = 0; i < options_.label_size(); ++i) {
+      label_map_[i] = options_.label(i);
     }
   }
   return absl::OkStatus();
@@ -106,7 +107,7 @@ absl::Status DetectionLabelIdToTextCalculator::Process(CalculatorContext* cc) {
       }
     }
     // Remove label_id field if text labels exist.
-    if (has_text_label) {
+    if (has_text_label && !options_.keep_label_id()) {
       output_detection.clear_label_id();
     }
   }

mediapipe/calculators/util/detection_label_id_to_text_calculator.proto

@@ -31,4 +31,9 @@ message DetectionLabelIdToTextCalculatorOptions {
   // label: "label for id 1"
   // ...
   repeated string label = 2;
+
+  // By default, the `label_id` field from the input is stripped if a text label
+  // could be found. By setting this field to true, it is always copied to the
+  // output detections.
+  optional bool keep_label_id = 3;
 }

mediapipe/calculators/util/detections_to_rects_calculator.cc

@@ -323,7 +323,7 @@ absl::Status DetectionsToRectsCalculator::ComputeRotation(
 DetectionSpec DetectionsToRectsCalculator::GetDetectionSpec(
     const CalculatorContext* cc) {
   absl::optional<std::pair<int, int>> image_size;
-  if (cc->Inputs().HasTag(kImageSizeTag)) {
+  if (HasTagValue(cc->Inputs(), kImageSizeTag)) {
     image_size = cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
   }
 

mediapipe/calculators/util/detections_to_rects_calculator_test.cc

@@ -157,6 +157,12 @@ TEST(DetectionsToRectsCalculatorTest, DetectionKeyPointsToRect) {
       /*image_size=*/{640, 480});
   MP_ASSERT_OK(status_or_value);
   EXPECT_THAT(status_or_value.value(), RectEq(480, 360, 320, 240));
+
+  status_or_value = RunDetectionKeyPointsToRectCalculation(
+      /*detection=*/DetectionWithKeyPoints({{0.25f, 0.25f}, {0.75f, 0.75f}}),
+      /*image_size=*/{0, 0});
+  MP_ASSERT_OK(status_or_value);
+  EXPECT_THAT(status_or_value.value(), RectEq(0, 0, 0, 0));
 }
 
 TEST(DetectionsToRectsCalculatorTest, DetectionToNormalizedRect) {

mediapipe/calculators/util/labels_to_render_data_calculator.cc

@@ -120,7 +120,11 @@ absl::Status LabelsToRenderDataCalculator::Process(CalculatorContext* cc) {
     labels.resize(classifications.classification_size());
     scores.resize(classifications.classification_size());
     for (int i = 0; i < classifications.classification_size(); ++i) {
-      labels[i] = classifications.classification(i).label();
+      if (options_.use_display_name()) {
+        labels[i] = classifications.classification(i).display_name();
+      } else {
+        labels[i] = classifications.classification(i).label();
+      }
       scores[i] = classifications.classification(i).score();
     }
   } else {

mediapipe/calculators/util/labels_to_render_data_calculator.proto

@@ -59,4 +59,7 @@ message LabelsToRenderDataCalculatorOptions {
     BOTTOM_LEFT = 1;
   }
   optional Location location = 6 [default = TOP_LEFT];
+
+  // Uses Classification.display_name field instead of Classification.label.
+  optional bool use_display_name = 9 [default = false];
 }

mediapipe/calculators/util/landmark_projection_calculator.cc

@@ -13,6 +13,7 @@
 // limitations under the License.
 
 #include <cmath>
+#include <functional>
 #include <vector>
 
 #include "mediapipe/calculators/util/landmark_projection_calculator.pb.h"
@@ -27,20 +28,32 @@ namespace {
 
 constexpr char kLandmarksTag[] = "NORM_LANDMARKS";
 constexpr char kRectTag[] = "NORM_RECT";
+constexpr char kProjectionMatrix[] = "PROJECTION_MATRIX";
 
 }  // namespace
 
-// Projects normalized landmarks in a rectangle to its original coordinates. The
-// rectangle must also be in normalized coordinates.
+// Projects normalized landmarks to its original coordinates.
 // Input:
-//   NORM_LANDMARKS: A NormalizedLandmarkList representing landmarks
-//                   in a normalized rectangle.
-//   NORM_RECT: An NormalizedRect representing a normalized rectangle in image
-//              coordinates.
+//   NORM_LANDMARKS - NormalizedLandmarkList
+//     Represents landmarks in a normalized rectangle if NORM_RECT is specified
+//     or landmarks that should be projected using PROJECTION_MATRIX if
+//     specified. (Prefer using PROJECTION_MATRIX as it eliminates need of
+//     letterbox removal step.)
+//   NORM_RECT - NormalizedRect
+//     Represents a normalized rectangle in image coordinates and results in
+//     landmarks with their locations adjusted to the image.
+//   PROJECTION_MATRIX - std::array<float, 16>
+//     A 4x4 row-major-order matrix that maps landmarks' locations from one
+//     coordinate system to another. In this case from the coordinate system of
+//     the normalized region of interest to the coordinate system of the image.
+//
+//   Note: either NORM_RECT or PROJECTION_MATRIX has to be specified.
+//   Note: landmark's Z is projected in a custom way - it's scaled by width of
+//     the normalized region of interest used during landmarks detection.
 //
 // Output:
-//   NORM_LANDMARKS: A NormalizedLandmarkList representing landmarks
-//                   with their locations adjusted to the image.
+//   NORM_LANDMARKS - NormalizedLandmarkList
+//     Landmarks with their locations adjusted according to the inputs.
 //
 // Usage example:
 // node {
@@ -58,12 +71,27 @@ constexpr char kRectTag[] = "NORM_RECT";
 //   output_stream: "NORM_LANDMARKS:0:projected_landmarks_0"
 //   output_stream: "NORM_LANDMARKS:1:projected_landmarks_1"
 // }
+//
+// node {
+//   calculator: "LandmarkProjectionCalculator"
+//   input_stream: "NORM_LANDMARKS:landmarks"
+//   input_stream: "PROECTION_MATRIX:matrix"
+//   output_stream: "NORM_LANDMARKS:projected_landmarks"
+// }
+//
+// node {
+//   calculator: "LandmarkProjectionCalculator"
+//   input_stream: "NORM_LANDMARKS:0:landmarks_0"
+//   input_stream: "NORM_LANDMARKS:1:landmarks_1"
+//   input_stream: "PROECTION_MATRIX:matrix"
+//   output_stream: "NORM_LANDMARKS:0:projected_landmarks_0"
+//   output_stream: "NORM_LANDMARKS:1:projected_landmarks_1"
+// }
 class LandmarkProjectionCalculator : public CalculatorBase {
  public:
   static absl::Status GetContract(CalculatorContract* cc) {
-    RET_CHECK(cc->Inputs().HasTag(kLandmarksTag) &&
-              cc->Inputs().HasTag(kRectTag))
-        << "Missing one or more input streams.";
+    RET_CHECK(cc->Inputs().HasTag(kLandmarksTag))
+        << "Missing NORM_LANDMARKS input.";
 
     RET_CHECK_EQ(cc->Inputs().NumEntries(kLandmarksTag),
                  cc->Outputs().NumEntries(kLandmarksTag))
@@ -73,7 +101,14 @@ class LandmarkProjectionCalculator : public CalculatorBase {
          id != cc->Inputs().EndId(kLandmarksTag); ++id) {
       cc->Inputs().Get(id).Set<NormalizedLandmarkList>();
     }
-    cc->Inputs().Tag(kRectTag).Set<NormalizedRect>();
+    RET_CHECK(cc->Inputs().HasTag(kRectTag) ^
+              cc->Inputs().HasTag(kProjectionMatrix))
+        << "Either NORM_RECT or PROJECTION_MATRIX must be specified.";
+    if (cc->Inputs().HasTag(kRectTag)) {
+      cc->Inputs().Tag(kRectTag).Set<NormalizedRect>();
+    } else {
+      cc->Inputs().Tag(kProjectionMatrix).Set<std::array<float, 16>>();
+    }
 
     for (CollectionItemId id = cc->Outputs().BeginId(kLandmarksTag);
          id != cc->Outputs().EndId(kLandmarksTag); ++id) {
@@ -89,31 +124,50 @@ class LandmarkProjectionCalculator : public CalculatorBase {
     return absl::OkStatus();
   }
 
+  static void ProjectXY(const NormalizedLandmark& lm,
+                        const std::array<float, 16>& matrix,
+                        NormalizedLandmark* out) {
+    out->set_x(lm.x() * matrix[0] + lm.y() * matrix[1] + lm.z() * matrix[2] +
+               matrix[3]);
+    out->set_y(lm.x() * matrix[4] + lm.y() * matrix[5] + lm.z() * matrix[6] +
+               matrix[7]);
+  }
+
+  /**
+   * Landmark's Z scale is equal to a relative (to image) width of region of
+   * interest used during detection. To calculate based on matrix:
+   * 1. Project (0,0) --- (1,0) segment using matrix.
+   * 2. Calculate length of the projected segment.
+   */
+  static float CalculateZScale(const std::array<float, 16>& matrix) {
+    NormalizedLandmark a;
+    a.set_x(0.0f);
+    a.set_y(0.0f);
+    NormalizedLandmark b;
+    b.set_x(1.0f);
+    b.set_y(0.0f);
+    NormalizedLandmark a_projected;
+    ProjectXY(a, matrix, &a_projected);
+    NormalizedLandmark b_projected;
+    ProjectXY(b, matrix, &b_projected);
+    return std::sqrt(std::pow(b_projected.x() - a_projected.x(), 2) +
+                     std::pow(b_projected.y() - a_projected.y(), 2));
+  }
+
   absl::Status Process(CalculatorContext* cc) override {
-    if (cc->Inputs().Tag(kRectTag).IsEmpty()) {
-      return absl::OkStatus();
-    }
-    const auto& input_rect = cc->Inputs().Tag(kRectTag).Get<NormalizedRect>();
-
-    const auto& options =
-        cc->Options<::mediapipe::LandmarkProjectionCalculatorOptions>();
-
-    CollectionItemId input_id = cc->Inputs().BeginId(kLandmarksTag);
-    CollectionItemId output_id = cc->Outputs().BeginId(kLandmarksTag);
-    // Number of inputs and outpus is the same according to the contract.
-    for (; input_id != cc->Inputs().EndId(kLandmarksTag);
-         ++input_id, ++output_id) {
-      const auto& input_packet = cc->Inputs().Get(input_id);
-      if (input_packet.IsEmpty()) {
-        continue;
+    std::function<void(const NormalizedLandmark&, NormalizedLandmark*)>
+        project_fn;
+    if (cc->Inputs().HasTag(kRectTag)) {
+      if (cc->Inputs().Tag(kRectTag).IsEmpty()) {
+        return absl::OkStatus();
       }
-
-      const auto& input_landmarks = input_packet.Get<NormalizedLandmarkList>();
-      NormalizedLandmarkList output_landmarks;
-      for (int i = 0; i < input_landmarks.landmark_size(); ++i) {
-        const NormalizedLandmark& landmark = input_landmarks.landmark(i);
-        NormalizedLandmark* new_landmark = output_landmarks.add_landmark();
-
+      const auto& input_rect = cc->Inputs().Tag(kRectTag).Get<NormalizedRect>();
+      const auto& options =
+          cc->Options<mediapipe::LandmarkProjectionCalculatorOptions>();
+      project_fn = [&input_rect, &options](const NormalizedLandmark& landmark,
+                                           NormalizedLandmark* new_landmark) {
+        // TODO: fix projection or deprecate (current projection
+        // calculations are incorrect for general case).
         const float x = landmark.x() - 0.5f;
         const float y = landmark.y() - 0.5f;
         const float angle =
@@ -130,10 +184,44 @@ class LandmarkProjectionCalculator : public CalculatorBase {
         new_landmark->set_x(new_x);
         new_landmark->set_y(new_y);
         new_landmark->set_z(new_z);
+      };
+    } else if (cc->Inputs().HasTag(kProjectionMatrix)) {
+      if (cc->Inputs().Tag(kProjectionMatrix).IsEmpty()) {
+        return absl::OkStatus();
+      }
+      const auto& project_mat =
+          cc->Inputs().Tag(kProjectionMatrix).Get<std::array<float, 16>>();
+      const float z_scale = CalculateZScale(project_mat);
+      project_fn = [&project_mat, z_scale](const NormalizedLandmark& lm,
+                                           NormalizedLandmark* new_landmark) {
+        *new_landmark = lm;
+        ProjectXY(lm, project_mat, new_landmark);
+        new_landmark->set_z(z_scale * lm.z());
+      };
+    } else {
+      return absl::InternalError("Either rect or matrix must be specified.");
+    }
+
+    CollectionItemId input_id = cc->Inputs().BeginId(kLandmarksTag);
+    CollectionItemId output_id = cc->Outputs().BeginId(kLandmarksTag);
+    // Number of inputs and outpus is the same according to the contract.
+    for (; input_id != cc->Inputs().EndId(kLandmarksTag);
+         ++input_id, ++output_id) {
+      const auto& input_packet = cc->Inputs().Get(input_id);
+      if (input_packet.IsEmpty()) {
+        continue;
+      }
+
+      const auto& input_landmarks = input_packet.Get<NormalizedLandmarkList>();
+      NormalizedLandmarkList output_landmarks;
+      for (int i = 0; i < input_landmarks.landmark_size(); ++i) {
+        const NormalizedLandmark& landmark = input_landmarks.landmark(i);
+        NormalizedLandmark* new_landmark = output_landmarks.add_landmark();
+        project_fn(landmark, new_landmark);
       }
 
       cc->Outputs().Get(output_id).AddPacket(
-          MakePacket<NormalizedLandmarkList>(output_landmarks)
+          MakePacket<NormalizedLandmarkList>(std::move(output_landmarks))
               .At(cc->InputTimestamp()));
     }
     return absl::OkStatus();

mediapipe/calculators/util/landmark_projection_calculator_test.cc

@@ -0,0 +1,240 @@
+#include <array>
+#include <vector>
+
+#include "absl/memory/memory.h"
+#include "mediapipe/calculators/tensor/image_to_tensor_utils.h"
+#include "mediapipe/framework/calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_runner.h"
+#include "mediapipe/framework/deps/message_matchers.h"
+#include "mediapipe/framework/formats/landmark.pb.h"
+#include "mediapipe/framework/formats/rect.pb.h"
+#include "mediapipe/framework/port/gtest.h"
+#include "mediapipe/framework/port/parse_text_proto.h"
+#include "mediapipe/framework/port/status_matchers.h"
+
+namespace mediapipe {
+namespace {
+
+absl::StatusOr<mediapipe::NormalizedLandmarkList> RunCalculator(
+    mediapipe::NormalizedLandmarkList input, mediapipe::NormalizedRect rect) {
+  mediapipe::CalculatorRunner runner(
+      ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig::Node>(R"pb(
+        calculator: "LandmarkProjectionCalculator"
+        input_stream: "NORM_LANDMARKS:landmarks"
+        input_stream: "NORM_RECT:rect"
+        output_stream: "NORM_LANDMARKS:projected_landmarks"
+      )pb"));
+  runner.MutableInputs()
+      ->Tag("NORM_LANDMARKS")
+      .packets.push_back(
+          MakePacket<mediapipe::NormalizedLandmarkList>(std::move(input))
+              .At(Timestamp(1)));
+  runner.MutableInputs()
+      ->Tag("NORM_RECT")
+      .packets.push_back(MakePacket<mediapipe::NormalizedRect>(std::move(rect))
+                             .At(Timestamp(1)));
+
+  MP_RETURN_IF_ERROR(runner.Run());
+  const auto& output_packets = runner.Outputs().Tag("NORM_LANDMARKS").packets;
+  RET_CHECK_EQ(output_packets.size(), 1);
+  return output_packets[0].Get<mediapipe::NormalizedLandmarkList>();
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithDefaultRect) {
+  mediapipe::NormalizedLandmarkList landmarks =
+      ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 10, y: 20, z: -0.5 }
+      )pb");
+  mediapipe::NormalizedRect rect =
+      ParseTextProtoOrDie<mediapipe::NormalizedRect>(
+          R"pb(
+            x_center: 0.5,
+            y_center: 0.5,
+            width: 1.0,
+            height: 1.0,
+            rotation: 0.0
+          )pb");
+
+  auto status_or_result = RunCalculator(std::move(landmarks), std::move(rect));
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(
+      status_or_result.value(),
+      EqualsProto(ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 10, y: 20, z: -0.5 }
+      )pb")));
+}
+
+mediapipe::NormalizedRect GetCroppedRect() {
+  return ParseTextProtoOrDie<mediapipe::NormalizedRect>(
+      R"pb(
+        x_center: 0.5, y_center: 0.5, width: 0.5, height: 2, rotation: 0.0
+      )pb");
+}
+
+mediapipe::NormalizedLandmarkList GetCroppedRectTestInput() {
+  return ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+    landmark { x: 1.0, y: 1.0, z: -0.5 }
+  )pb");
+}
+
+mediapipe::NormalizedLandmarkList GetCroppedRectTestExpectedResult() {
+  return ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+    landmark { x: 0.75, y: 1.5, z: -0.25 }
+  )pb");
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithCroppedRect) {
+  auto status_or_result =
+      RunCalculator(GetCroppedRectTestInput(), GetCroppedRect());
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(status_or_result.value(),
+              EqualsProto(GetCroppedRectTestExpectedResult()));
+}
+
+absl::StatusOr<mediapipe::NormalizedLandmarkList> RunCalculator(
+    mediapipe::NormalizedLandmarkList input, std::array<float, 16> matrix) {
+  mediapipe::CalculatorRunner runner(
+      ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig::Node>(R"pb(
+        calculator: "LandmarkProjectionCalculator"
+        input_stream: "NORM_LANDMARKS:landmarks"
+        input_stream: "PROJECTION_MATRIX:matrix"
+        output_stream: "NORM_LANDMARKS:projected_landmarks"
+      )pb"));
+  runner.MutableInputs()
+      ->Tag("NORM_LANDMARKS")
+      .packets.push_back(
+          MakePacket<mediapipe::NormalizedLandmarkList>(std::move(input))
+              .At(Timestamp(1)));
+  runner.MutableInputs()
+      ->Tag("PROJECTION_MATRIX")
+      .packets.push_back(MakePacket<std::array<float, 16>>(std::move(matrix))
+                             .At(Timestamp(1)));
+
+  MP_RETURN_IF_ERROR(runner.Run());
+  const auto& output_packets = runner.Outputs().Tag("NORM_LANDMARKS").packets;
+  RET_CHECK_EQ(output_packets.size(), 1);
+  return output_packets[0].Get<mediapipe::NormalizedLandmarkList>();
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithIdentityMatrix) {
+  mediapipe::NormalizedLandmarkList landmarks =
+      ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 10, y: 20, z: -0.5 }
+      )pb");
+  // clang-format off
+  std::array<float, 16> matrix = {
+    1.0f, 0.0f, 0.0f, 0.0f,
+    0.0f, 1.0f, 0.0f, 0.0f,
+    0.0f, 0.0f, 1.0f, 0.0f,
+    0.0f, 0.0f, 0.0f, 1.0f,
+  };
+  // clang-format on
+
+  auto status_or_result =
+      RunCalculator(std::move(landmarks), std::move(matrix));
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(
+      status_or_result.value(),
+      EqualsProto(ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 10, y: 20, z: -0.5 }
+      )pb")));
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithCroppedRectMatrix) {
+  constexpr int kRectWidth = 1280;
+  constexpr int kRectHeight = 720;
+  auto roi = GetRoi(kRectWidth, kRectHeight, GetCroppedRect());
+  std::array<float, 16> matrix;
+  GetRotatedSubRectToRectTransformMatrix(roi, kRectWidth, kRectHeight,
+                                         /*flip_horizontaly=*/false, &matrix);
+  auto status_or_result = RunCalculator(GetCroppedRectTestInput(), matrix);
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(status_or_result.value(),
+              EqualsProto(GetCroppedRectTestExpectedResult()));
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithScaleMatrix) {
+  mediapipe::NormalizedLandmarkList landmarks =
+      ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 10, y: 20, z: -0.5 }
+        landmark { x: 5, y: 6, z: 7 }
+      )pb");
+  // clang-format off
+  std::array<float, 16> matrix = {
+    10.0f, 0.0f,   0.0f, 0.0f,
+    0.0f,  100.0f, 0.0f, 0.0f,
+    0.0f,  0.0f,   1.0f, 0.0f,
+    0.0f,  0.0f,   0.0f, 1.0f,
+  };
+  // clang-format on
+
+  auto status_or_result =
+      RunCalculator(std::move(landmarks), std::move(matrix));
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(
+      status_or_result.value(),
+      EqualsProto(ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 100, y: 2000, z: -5 }
+        landmark { x: 50, y: 600, z: 70 }
+      )pb")));
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithTranslateMatrix) {
+  mediapipe::NormalizedLandmarkList landmarks =
+      ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 10, y: 20, z: -0.5 }
+      )pb");
+  // clang-format off
+  std::array<float, 16> matrix = {
+    1.0f, 0.0f, 0.0f, 1.0f,
+    0.0f, 1.0f, 0.0f, 2.0f,
+    0.0f, 0.0f, 1.0f, 0.0f,
+    0.0f, 0.0f, 0.0f, 1.0f,
+  };
+  // clang-format on
+
+  auto status_or_result =
+      RunCalculator(std::move(landmarks), std::move(matrix));
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(
+      status_or_result.value(),
+      EqualsProto(ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 11, y: 22, z: -0.5 }
+      )pb")));
+}
+
+TEST(LandmarkProjectionCalculatorTest, ProjectingWithRotationMatrix) {
+  mediapipe::NormalizedLandmarkList landmarks =
+      ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 4, y: 0, z: -0.5 }
+      )pb");
+  // clang-format off
+  // 90 degrees rotation matrix
+  std::array<float, 16> matrix = {
+    0.0f, -1.0f, 0.0f, 0.0f,
+    1.0f,  0.0f, 0.0f, 0.0f,
+    0.0f,  0.0f, 1.0f, 0.0f,
+    0.0f,  0.0f, 0.0f, 1.0f,
+  };
+  // clang-format on
+
+  auto status_or_result =
+      RunCalculator(std::move(landmarks), std::move(matrix));
+  MP_ASSERT_OK(status_or_result);
+
+  EXPECT_THAT(
+      status_or_result.value(),
+      EqualsProto(ParseTextProtoOrDie<mediapipe::NormalizedLandmarkList>(R"pb(
+        landmark { x: 0, y: 4, z: -0.5 }
+      )pb")));
+}
+
+}  // namespace
+}  // namespace mediapipe

mediapipe/calculators/util/landmarks_smoothing_calculator.cc

@@ -18,6 +18,7 @@
 #include "mediapipe/calculators/util/landmarks_smoothing_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/formats/landmark.pb.h"
+#include "mediapipe/framework/formats/rect.pb.h"
 #include "mediapipe/framework/port/ret_check.h"
 #include "mediapipe/framework/timestamp.h"
 #include "mediapipe/util/filtering/one_euro_filter.h"
@@ -30,6 +31,7 @@ namespace {
 constexpr char kNormalizedLandmarksTag[] = "NORM_LANDMARKS";
 constexpr char kLandmarksTag[] = "LANDMARKS";
 constexpr char kImageSizeTag[] = "IMAGE_SIZE";
+constexpr char kObjectScaleRoiTag[] = "OBJECT_SCALE_ROI";
 constexpr char kNormalizedFilteredLandmarksTag[] = "NORM_FILTERED_LANDMARKS";
 constexpr char kFilteredLandmarksTag[] = "FILTERED_LANDMARKS";
 
@@ -94,6 +96,18 @@ float GetObjectScale(const LandmarkList& landmarks) {
   return (object_width + object_height) / 2.0f;
 }
 
+float GetObjectScale(const NormalizedRect& roi, const int image_width,
+                     const int image_height) {
+  const float object_width = roi.width() * image_width;
+  const float object_height = roi.height() * image_height;
+
+  return (object_width + object_height) / 2.0f;
+}
+
+float GetObjectScale(const Rect& roi) {
+  return (roi.width() + roi.height()) / 2.0f;
+}
+
 // Abstract class for various landmarks filters.
 class LandmarksFilter {
  public:
@@ -103,6 +117,7 @@ class LandmarksFilter {
 
   virtual absl::Status Apply(const LandmarkList& in_landmarks,
                              const absl::Duration& timestamp,
+                             const absl::optional<float> object_scale_opt,
                              LandmarkList* out_landmarks) = 0;
 };
 
@@ -111,6 +126,7 @@ class NoFilter : public LandmarksFilter {
  public:
   absl::Status Apply(const LandmarkList& in_landmarks,
                      const absl::Duration& timestamp,
+                     const absl::optional<float> object_scale_opt,
                      LandmarkList* out_landmarks) override {
     *out_landmarks = in_landmarks;
     return absl::OkStatus();
@@ -136,13 +152,15 @@ class VelocityFilter : public LandmarksFilter {
 
   absl::Status Apply(const LandmarkList& in_landmarks,
                      const absl::Duration& timestamp,
+                     const absl::optional<float> object_scale_opt,
                      LandmarkList* out_landmarks) override {
     // Get value scale as inverse value of the object scale.
     // If value is too small smoothing will be disabled and landmarks will be
     // returned as is.
     float value_scale = 1.0f;
     if (!disable_value_scaling_) {
-      const float object_scale = GetObjectScale(in_landmarks);
+      const float object_scale =
+          object_scale_opt ? *object_scale_opt : GetObjectScale(in_landmarks);
       if (object_scale < min_allowed_object_scale_) {
         *out_landmarks = in_landmarks;
         return absl::OkStatus();
@@ -205,11 +223,14 @@ class VelocityFilter : public LandmarksFilter {
 class OneEuroFilterImpl : public LandmarksFilter {
  public:
   OneEuroFilterImpl(double frequency, double min_cutoff, double beta,
-                    double derivate_cutoff)
+                    double derivate_cutoff, float min_allowed_object_scale,
+                    bool disable_value_scaling)
       : frequency_(frequency),
         min_cutoff_(min_cutoff),
         beta_(beta),
-        derivate_cutoff_(derivate_cutoff) {}
+        derivate_cutoff_(derivate_cutoff),
+        min_allowed_object_scale_(min_allowed_object_scale),
+        disable_value_scaling_(disable_value_scaling) {}
 
   absl::Status Reset() override {
     x_filters_.clear();
@@ -220,19 +241,37 @@ class OneEuroFilterImpl : public LandmarksFilter {
 
   absl::Status Apply(const LandmarkList& in_landmarks,
                      const absl::Duration& timestamp,
+                     const absl::optional<float> object_scale_opt,
                      LandmarkList* out_landmarks) override {
     // Initialize filters once.
     MP_RETURN_IF_ERROR(InitializeFiltersIfEmpty(in_landmarks.landmark_size()));
 
+    // Get value scale as inverse value of the object scale.
+    // If value is too small smoothing will be disabled and landmarks will be
+    // returned as is.
+    float value_scale = 1.0f;
+    if (!disable_value_scaling_) {
+      const float object_scale =
+          object_scale_opt ? *object_scale_opt : GetObjectScale(in_landmarks);
+      if (object_scale < min_allowed_object_scale_) {
+        *out_landmarks = in_landmarks;
+        return absl::OkStatus();
+      }
+      value_scale = 1.0f / object_scale;
+    }
+
     // Filter landmarks. Every axis of every landmark is filtered separately.
     for (int i = 0; i < in_landmarks.landmark_size(); ++i) {
       const auto& in_landmark = in_landmarks.landmark(i);
 
       auto* out_landmark = out_landmarks->add_landmark();
       *out_landmark = in_landmark;
-      out_landmark->set_x(x_filters_[i].Apply(timestamp, in_landmark.x()));
-      out_landmark->set_y(y_filters_[i].Apply(timestamp, in_landmark.y()));
-      out_landmark->set_z(z_filters_[i].Apply(timestamp, in_landmark.z()));
+      out_landmark->set_x(
+          x_filters_[i].Apply(timestamp, value_scale, in_landmark.x()));
+      out_landmark->set_y(
+          y_filters_[i].Apply(timestamp, value_scale, in_landmark.y()));
+      out_landmark->set_z(
+          z_filters_[i].Apply(timestamp, value_scale, in_landmark.z()));
     }
 
     return absl::OkStatus();
@@ -265,6 +304,8 @@ class OneEuroFilterImpl : public LandmarksFilter {
   double min_cutoff_;
   double beta_;
   double derivate_cutoff_;
+  double min_allowed_object_scale_;
+  bool disable_value_scaling_;
 
   std::vector<OneEuroFilter> x_filters_;
   std::vector<OneEuroFilter> y_filters_;
@@ -280,6 +321,10 @@ class OneEuroFilterImpl : public LandmarksFilter {
 //   IMAGE_SIZE: A std::pair<int, int> represention of image width and height.
 //     Required to perform all computations in absolute coordinates to avoid any
 //     influence of normalized values.
+//   OBJECT_SCALE_ROI (optional): A NormRect or Rect (depending on the format of
+//     input landmarks) used to determine the object scale for some of the
+//     filters. If not provided - object scale will be calculated from
+//     landmarks.
 //
 // Outputs:
 //   NORM_FILTERED_LANDMARKS: A NormalizedLandmarkList of smoothed landmarks.
@@ -289,6 +334,7 @@ class OneEuroFilterImpl : public LandmarksFilter {
 //     calculator: "LandmarksSmoothingCalculator"
 //     input_stream: "NORM_LANDMARKS:pose_landmarks"
 //     input_stream: "IMAGE_SIZE:image_size"
+//     input_stream: "OBJECT_SCALE_ROI:roi"
 //     output_stream: "NORM_FILTERED_LANDMARKS:pose_landmarks_filtered"
 //     options: {
 //       [mediapipe.LandmarksSmoothingCalculatorOptions.ext] {
@@ -318,9 +364,17 @@ absl::Status LandmarksSmoothingCalculator::GetContract(CalculatorContract* cc) {
     cc->Outputs()
         .Tag(kNormalizedFilteredLandmarksTag)
         .Set<NormalizedLandmarkList>();
+
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag)) {
+      cc->Inputs().Tag(kObjectScaleRoiTag).Set<NormalizedRect>();
+    }
   } else {
     cc->Inputs().Tag(kLandmarksTag).Set<LandmarkList>();
     cc->Outputs().Tag(kFilteredLandmarksTag).Set<LandmarkList>();
+
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag)) {
+      cc->Inputs().Tag(kObjectScaleRoiTag).Set<Rect>();
+    }
   }
 
   return absl::OkStatus();
@@ -344,7 +398,9 @@ absl::Status LandmarksSmoothingCalculator::Open(CalculatorContext* cc) {
         options.one_euro_filter().frequency(),
         options.one_euro_filter().min_cutoff(),
         options.one_euro_filter().beta(),
-        options.one_euro_filter().derivate_cutoff());
+        options.one_euro_filter().derivate_cutoff(),
+        options.one_euro_filter().min_allowed_object_scale(),
+        options.one_euro_filter().disable_value_scaling());
   } else {
     RET_CHECK_FAIL()
         << "Landmarks filter is either not specified or not supported";
@@ -376,13 +432,20 @@ absl::Status LandmarksSmoothingCalculator::Process(CalculatorContext* cc) {
     std::tie(image_width, image_height) =
         cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
 
+    absl::optional<float> object_scale;
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag) &&
+        !cc->Inputs().Tag(kObjectScaleRoiTag).IsEmpty()) {
+      auto& roi = cc->Inputs().Tag(kObjectScaleRoiTag).Get<NormalizedRect>();
+      object_scale = GetObjectScale(roi, image_width, image_height);
+    }
+
     auto in_landmarks = absl::make_unique<LandmarkList>();
     NormalizedLandmarksToLandmarks(in_norm_landmarks, image_width, image_height,
                                    in_landmarks.get());
 
     auto out_landmarks = absl::make_unique<LandmarkList>();
-    MP_RETURN_IF_ERROR(landmarks_filter_->Apply(*in_landmarks, timestamp,
-                                                out_landmarks.get()));
+    MP_RETURN_IF_ERROR(landmarks_filter_->Apply(
+        *in_landmarks, timestamp, object_scale, out_landmarks.get()));
 
     auto out_norm_landmarks = absl::make_unique<NormalizedLandmarkList>();
     LandmarksToNormalizedLandmarks(*out_landmarks, image_width, image_height,
@@ -395,9 +458,16 @@ absl::Status LandmarksSmoothingCalculator::Process(CalculatorContext* cc) {
     const auto& in_landmarks =
         cc->Inputs().Tag(kLandmarksTag).Get<LandmarkList>();
 
+    absl::optional<float> object_scale;
+    if (cc->Inputs().HasTag(kObjectScaleRoiTag) &&
+        !cc->Inputs().Tag(kObjectScaleRoiTag).IsEmpty()) {
+      auto& roi = cc->Inputs().Tag(kObjectScaleRoiTag).Get<Rect>();
+      object_scale = GetObjectScale(roi);
+    }
+
     auto out_landmarks = absl::make_unique<LandmarkList>();
-    MP_RETURN_IF_ERROR(
-        landmarks_filter_->Apply(in_landmarks, timestamp, out_landmarks.get()));
+    MP_RETURN_IF_ERROR(landmarks_filter_->Apply(
+        in_landmarks, timestamp, object_scale, out_landmarks.get()));
 
     cc->Outputs()
         .Tag(kFilteredLandmarksTag)

mediapipe/calculators/util/landmarks_smoothing_calculator.proto

@@ -41,18 +41,18 @@ message LandmarksSmoothingCalculatorOptions {
     optional float min_allowed_object_scale = 3 [default = 1e-6];
 
     // Disable value scaling based on object size and use `1.0` instead.
-    // Value scale is calculated as inverse value of object size. Object size is
-    // calculated as maximum side of rectangular bounding box of the object in
-    // XY plane.
+    // If not disabled, value scale is calculated as inverse value of object
+    // size. Object size is calculated as maximum side of rectangular bounding
+    // box of the object in XY plane.
     optional bool disable_value_scaling = 4 [default = false];
   }
 
   // For the details of the filter implementation and the procedure of its
   // configuration please check http://cristal.univ-lille.fr/~casiez/1euro/
   message OneEuroFilter {
-    // Frequency of incomming frames defined in seconds. Used only if can't be
-    // calculated from provided events (e.g. on the very first frame).
-    optional float frequency = 1 [default = 0.033];
+    // Frequency of incomming frames defined in frames per seconds. Used only if
+    // can't be calculated from provided events (e.g. on the very first frame).
+    optional float frequency = 1 [default = 30.0];
 
     // Minimum cutoff frequency. Start by tuning this parameter while keeping
     // `beta = 0` to reduce jittering to the desired level. 1Hz (the default
@@ -68,6 +68,16 @@ message LandmarksSmoothingCalculatorOptions {
     // algorithm, but can be tuned to further smooth the speed (i.e. derivate)
     // on the object.
     optional float derivate_cutoff = 4 [default = 1.0];
+
+    // If calculated object scale is less than given value smoothing will be
+    // disabled and landmarks will be returned as is.
+    optional float min_allowed_object_scale = 5 [default = 1e-6];
+
+    // Disable value scaling based on object size and use `1.0` instead.
+    // If not disabled, value scale is calculated as inverse value of object
+    // size. Object size is calculated as maximum side of rectangular bounding
+    // box of the object in XY plane.
+    optional bool disable_value_scaling = 6 [default = false];
   }
 
   oneof filter_options {

mediapipe/calculators/util/rect_to_render_scale_calculator.cc

@@ -1,3 +1,17 @@
+// Copyright 2020 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 #include "mediapipe/calculators/util/rect_to_render_scale_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/formats/rect.pb.h"

mediapipe/calculators/util/rect_to_render_scale_calculator.proto

@@ -1,3 +1,17 @@
+// Copyright 2020 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 syntax = "proto2";
 
 package mediapipe;

mediapipe/calculators/util/refine_landmarks_from_heatmap_calculator.cc

@@ -0,0 +1,187 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "mediapipe/calculators/util/refine_landmarks_from_heatmap_calculator.h"
+
+#include "mediapipe/calculators/util/refine_landmarks_from_heatmap_calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+
+namespace mediapipe {
+
+namespace {
+
+inline float Sigmoid(float value) { return 1.0f / (1.0f + std::exp(-value)); }
+
+absl::StatusOr<std::tuple<int, int, int>> GetHwcFromDims(
+    const std::vector<int>& dims) {
+  if (dims.size() == 3) {
+    return std::make_tuple(dims[0], dims[1], dims[2]);
+  } else if (dims.size() == 4) {
+    // BHWC format check B == 1
+    RET_CHECK_EQ(1, dims[0]) << "Expected batch to be 1 for BHWC heatmap";
+    return std::make_tuple(dims[1], dims[2], dims[3]);
+  } else {
+    RET_CHECK(false) << "Invalid shape size for heatmap tensor" << dims.size();
+  }
+}
+
+}  // namespace
+
+namespace api2 {
+
+// Refines landmarks using correspond heatmap area.
+//
+// Input:
+//   NORM_LANDMARKS - Required. Input normalized landmarks to update.
+//   TENSORS - Required. Vector of input tensors. 0th element should be heatmap.
+//             The rest is unused.
+// Output:
+//   NORM_LANDMARKS - Required. Updated normalized landmarks.
+class RefineLandmarksFromHeatmapCalculatorImpl
+    : public NodeImpl<RefineLandmarksFromHeatmapCalculator,
+                      RefineLandmarksFromHeatmapCalculatorImpl> {
+ public:
+  absl::Status Open(CalculatorContext* cc) override { return absl::OkStatus(); }
+
+  absl::Status Process(CalculatorContext* cc) override {
+    // Make sure we bypass landmarks if there is no detection.
+    if (kInLandmarks(cc).IsEmpty()) {
+      return absl::OkStatus();
+    }
+    // If for some reason heatmap is missing, just return original landmarks.
+    if (kInTensors(cc).IsEmpty()) {
+      kOutLandmarks(cc).Send(*kInLandmarks(cc));
+      return absl::OkStatus();
+    }
+
+    // Check basic prerequisites.
+    const auto& input_tensors = *kInTensors(cc);
+    RET_CHECK(!input_tensors.empty()) << "Empty input tensors list. First "
+                                         "element is expeced to be a heatmap";
+
+    const auto& hm_tensor = input_tensors[0];
+    const auto& in_lms = *kInLandmarks(cc);
+    auto hm_view = hm_tensor.GetCpuReadView();
+    auto hm_raw = hm_view.buffer<float>();
+    const auto& options =
+        cc->Options<mediapipe::RefineLandmarksFromHeatmapCalculatorOptions>();
+
+    ASSIGN_OR_RETURN(
+        auto out_lms,
+        RefineLandmarksFromHeatMap(
+            in_lms, hm_raw, hm_tensor.shape().dims, options.kernel_size(),
+            options.min_confidence_to_refine(), options.refine_presence(),
+            options.refine_visibility()));
+
+    kOutLandmarks(cc).Send(std::move(out_lms));
+    return absl::OkStatus();
+  }
+};
+
+}  // namespace api2
+
+// Runs actual refinement
+// High level algorithm:
+//
+// Heatmap is accepted as tensor in HWC layout where i-th channel is a heatmap
+// for the i-th landmark.
+//
+// For each landmark we replace original value with a value calculated from the
+// area in heatmap close to original landmark position (in particular are
+// covered with kernel of size options.kernel_size). To calculate new coordinate
+// from heatmap we calculate an weighted average inside the kernel. We update
+// the landmark iff heatmap is confident in it's prediction i.e. max(heatmap) in
+// kernel is at least options.min_confidence_to_refine big.
+absl::StatusOr<mediapipe::NormalizedLandmarkList> RefineLandmarksFromHeatMap(
+    const mediapipe::NormalizedLandmarkList& in_lms,
+    const float* heatmap_raw_data, const std::vector<int>& heatmap_dims,
+    int kernel_size, float min_confidence_to_refine, bool refine_presence,
+    bool refine_visibility) {
+  ASSIGN_OR_RETURN(auto hm_dims, GetHwcFromDims(heatmap_dims));
+  auto [hm_height, hm_width, hm_channels] = hm_dims;
+
+  RET_CHECK_EQ(in_lms.landmark_size(), hm_channels)
+      << "Expected heatmap to have number of layers == to number of "
+         "landmarks";
+
+  int hm_row_size = hm_width * hm_channels;
+  int hm_pixel_size = hm_channels;
+
+  mediapipe::NormalizedLandmarkList out_lms = in_lms;
+  for (int lm_index = 0; lm_index < out_lms.landmark_size(); ++lm_index) {
+    int center_col = out_lms.landmark(lm_index).x() * hm_width;
+    int center_row = out_lms.landmark(lm_index).y() * hm_height;
+    // Point is outside of the image let's keep it intact.
+    if (center_col < 0 || center_col >= hm_width || center_row < 0 ||
+        center_col >= hm_height) {
+      continue;
+    }
+
+    int offset = (kernel_size - 1) / 2;
+    // Calculate area to iterate over. Note that we decrease the kernel on
+    // the edges of the heatmap. Equivalent to zero border.
+    int begin_col = std::max(0, center_col - offset);
+    int end_col = std::min(hm_width, center_col + offset + 1);
+    int begin_row = std::max(0, center_row - offset);
+    int end_row = std::min(hm_height, center_row + offset + 1);
+
+    float sum = 0;
+    float weighted_col = 0;
+    float weighted_row = 0;
+    float max_confidence_value = 0;
+
+    // Main loop. Go over kernel and calculate weighted sum of coordinates,
+    // sum of weights and max weights.
+    for (int row = begin_row; row < end_row; ++row) {
+      for (int col = begin_col; col < end_col; ++col) {
+        // We expect memory to be in HWC layout without padding.
+        int idx = hm_row_size * row + hm_pixel_size * col + lm_index;
+        // Right now we hardcode sigmoid activation as it will be wasteful to
+        // calculate sigmoid for each value of heatmap in the model itself.  If
+        // we ever have other activations it should be trivial to expand via
+        // options.
+        float confidence = Sigmoid(heatmap_raw_data[idx]);
+        sum += confidence;
+        max_confidence_value = std::max(max_confidence_value, confidence);
+        weighted_col += col * confidence;
+        weighted_row += row * confidence;
+      }
+    }
+    if (max_confidence_value >= min_confidence_to_refine && sum > 0) {
+      out_lms.mutable_landmark(lm_index)->set_x(weighted_col / hm_width / sum);
+      out_lms.mutable_landmark(lm_index)->set_y(weighted_row / hm_height / sum);
+    }
+    if (refine_presence && sum > 0 &&
+        out_lms.landmark(lm_index).has_presence()) {
+      // We assume confidence in heatmaps describes landmark presence.
+      // If landmark is not confident in heatmaps, probably it is not present.
+      const float presence = out_lms.landmark(lm_index).presence();
+      const float new_presence = std::min(presence, max_confidence_value);
+      out_lms.mutable_landmark(lm_index)->set_presence(new_presence);
+    }
+    if (refine_visibility && sum > 0 &&
+        out_lms.landmark(lm_index).has_visibility()) {
+      // We assume confidence in heatmaps describes landmark presence.
+      // As visibility = (not occluded but still present) -> that mean that if
+      // landmark is not present, it is not visible as well.
+      // I.e. visibility confidence cannot be bigger than presence confidence.
+      const float visibility = out_lms.landmark(lm_index).visibility();
+      const float new_visibility = std::min(visibility, max_confidence_value);
+      out_lms.mutable_landmark(lm_index)->set_visibility(new_visibility);
+    }
+  }
+  return out_lms;
+}
+
+}  // namespace mediapipe

mediapipe/calculators/util/refine_landmarks_from_heatmap_calculator.h

@@ -0,0 +1,51 @@
+// Copyright 2021 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef MEDIAPIPE_CALCULATORS_UTIL_REFINE_LANDMARKS_FROM_HEATMAP_CALCULATOR_H_
+#define MEDIAPIPE_CALCULATORS_UTIL_REFINE_LANDMARKS_FROM_HEATMAP_CALCULATOR_H_
+
+#include <vector>
+
+#include "mediapipe/framework/api2/node.h"
+#include "mediapipe/framework/formats/landmark.pb.h"
+#include "mediapipe/framework/formats/tensor.h"
+#include "mediapipe/framework/port/statusor.h"
+
+namespace mediapipe {
+namespace api2 {
+
+class RefineLandmarksFromHeatmapCalculator : public NodeIntf {
+ public:
+  static constexpr Input<mediapipe::NormalizedLandmarkList> kInLandmarks{
+      "NORM_LANDMARKS"};
+  static constexpr Input<std::vector<Tensor>> kInTensors{"TENSORS"};
+  static constexpr Output<mediapipe::NormalizedLandmarkList> kOutLandmarks{
+      "NORM_LANDMARKS"};
+
+  MEDIAPIPE_NODE_INTERFACE(RefineLandmarksFromHeatmapCalculator, kInLandmarks,
+                           kInTensors, kOutLandmarks);
+};
+
+}  // namespace api2
+
+// Exposed for testing.
+absl::StatusOr<mediapipe::NormalizedLandmarkList> RefineLandmarksFromHeatMap(
+    const mediapipe::NormalizedLandmarkList& in_lms,
+    const float* heatmap_raw_data, const std::vector<int>& heatmap_dims,
+    int kernel_size, float min_confidence_to_refine, bool refine_presence,
+    bool refine_visibility);
+
+}  // namespace mediapipe
+
+#endif  // MEDIAPIPE_CALCULATORS_UTIL_REFINE_LANDMARKS_FROM_HEATMAP_CALCULATOR_H_