Project import generated by Copybara.

GitOrigin-RevId: f9a66589eaf652bb93f8e37ed9e4da26e59ef214
2020-10-19 13:23:25 -04:00 · 2020-09-28 01:22:05 -04:00 · 2020-09-17 11:52:51 -04:00 · 2020-09-17 11:09:17 -04:00 · 2020-08-31 16:03:43 -07:00 · 2020-08-30 19:52:55 -04:00
404 changed files with 47488 additions and 3515 deletions
@@ -54,7 +54,7 @@ RUN pip3 install tf_slim
 RUN ln -s /usr/bin/python3 /usr/bin/python

 # Install bazel
-ARG BAZEL_VERSION=2.0.0
+ARG BAZEL_VERSION=3.4.1
 RUN mkdir /bazel && \
    wget --no-check-certificate -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/b\
 azel-${BAZEL_VERSION}-installer-linux-x86_64.sh" && \
@@ -1,6 +1,11 @@
 global-exclude .git*
 global-exclude *_test.py

-recursive-include mediapipe/models *.tflite *.txt
+include CONTRIBUTING.md
+include LICENSE
+include MANIFEST.in
+include README.md
+include requirements.txt
+
 recursive-include mediapipe/modules *.tflite *.txt
 recursive-include mediapipe/graphs *.binarypb
@@ -8,46 +8,50 @@ nav_order: 1

 --------------------------------------------------------------------------------

-## Cross-platform ML solutions made simple
+## Live ML anywhere

-[MediaPipe](https://google.github.io/mediapipe/) is the simplest way for researchers
-and developers to build world-class ML solutions and applications for mobile,
-desktop/cloud, web and IoT devices.
+[MediaPipe](https://google.github.io/mediapipe/) offers cross-platform, customizable
+ML solutions for live and streaming media.

 ![accelerated.png](docs/images/accelerated_small.png)                                                               | ![cross_platform.png](docs/images/cross_platform_small.png)
 :------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------:
-***End-to-End acceleration***: *built-in fast ML inference and processing accelerated even on common hardware* | ***Build one, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT*
+***End-to-End acceleration***: *Built-in fast ML inference and processing accelerated even on common hardware* | ***Build once, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT*
 ![ready_to_use.png](docs/images/ready_to_use_small.png)                                                             | ![open_source.png](docs/images/open_source_small.png)
 ***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework*            | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*

 ## ML solutions in MediaPipe

-Face Detection                                                                                                                 | Face Mesh                                                                                                       | Iris 🆕                                                                                                    | Hands                                                                                                      | Pose 🆕
-:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :----:
-[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](docs/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose)
+Face Detection                                                                                                                 | Face Mesh                                                                                                       | Iris                                                                                                      | Hands                                                                                                      | Pose                                                                                                      | Hair Segmentation
+:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------:
+[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](docs/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose) | [![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation)

-Hair Segmentation                                                                                                                       | Object Detection                                                                                                                     | Box Tracking                                                                                                                | Objectron                                                                                                             | KNIFT
-:-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
-[![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) | [![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
+Object Detection                                                                                                                     | Box Tracking                                                                                                                | Instant Motion Tracking                                                                                                                               | Objectron                                                                                                             | KNIFT
+:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
+[![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![instant_motion_tracking](docs/images/mobile/instant_motion_tracking_android_small.gif)](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)

 <!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
 <!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->

-[]()                                                                          | Android | iOS | Desktop | Python | Web | Coral
-:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
-[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)       | ✅       | ✅   | ✅       |        | ✅   | ✅
-[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                 | ✅       | ✅   | ✅       |        |     |
-[Iris](https://google.github.io/mediapipe/solutions/iris) 🆕                         | ✅       | ✅   | ✅       |        | ✅   |
-[Hands](https://google.github.io/mediapipe/solutions/hands)                         | ✅       | ✅   | ✅       |        | ✅   |
-[Pose](https://google.github.io/mediapipe/solutions/pose) 🆕                         | ✅       | ✅   | ✅       | ✅      | ✅   |
-[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅       |     | ✅       |        | ✅   |
-[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)   | ✅       | ✅   | ✅       |        |     | ✅
-[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)           | ✅       | ✅   | ✅       |        |     |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                 | ✅       |     |         |        |     |
-[KNIFT](https://google.github.io/mediapipe/solutions/knift)                         | ✅       |     |         |        |     |
-[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                   |         |     | ✅       |        |     |
-[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)        |         |     | ✅       |        |     |
-[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)               |         |     | ✅       |        |     |
+[]()                                                                                      | Android | iOS | Desktop | Python | Web | Coral
+:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
+[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)                   | ✅       | ✅   | ✅       |        | ✅   | ✅
+[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                             | ✅       | ✅   | ✅       |        |     |
+[Iris](https://google.github.io/mediapipe/solutions/iris)                                       | ✅       | ✅   | ✅       |        | ✅   |
+[Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅       | ✅   | ✅       |        | ✅   |
+[Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅       | ✅   | ✅       | ✅      | ✅   |
+[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅       |     | ✅       |        | ✅   |
+[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅       | ✅   | ✅       |        |     | ✅
+[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅       | ✅   | ✅       |        |     |
+[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅       |     |         |        |     |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅       |     |         |        |     |
+[KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅       |     |         |        |     |
+[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |         |     | ✅       |        |     |
+[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |         |     | ✅       |        |     |
+[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)                           |         |     | ✅       |        |     |
+
+See also
+[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
+for ML models released in MediaPipe.

 ## MediaPipe on the Web

@@ -88,7 +92,11 @@ run code search using

 ## Publications

-*   [BlazePose - On-device Real-time Body Pose Tracking](https://mediapipe.page.link/blazepose-blog)
+*   [MediaPipe 3D Face Transform](https://developers.googleblog.com/2020/09/mediapipe-3d-face-transform.html)
+    in Google Developers Blog
+*   [Instant Motion Tracking With MediaPipe](https://developers.googleblog.com/2020/08/instant-motion-tracking-with-mediapipe.html)
+    in Google Developers Blog
+*   [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
    in Google AI Blog
 *   [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
    in Google AI Blog
@@ -10,7 +10,7 @@ http_archive(
    sha256 = "1dde365491125a3db70731e25658dfdd3bc5dbdfd11b840b3e987ecf043c7ca0",
 )
 load("@bazel_skylib//lib:versions.bzl", "versions")
-versions.check(minimum_bazel_version = "2.0.0")
+versions.check(minimum_bazel_version = "3.4.0")


 # ABSL cpp library lts_2020_02_25
@@ -36,6 +36,19 @@ http_archive(
    urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
 )

+http_archive(
+   name = "rules_foreign_cc",
+   strip_prefix = "rules_foreign_cc-master",
+   url = "https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip",
+)
+
+load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
+
+rules_foreign_cc_dependencies()
+
+# This is used to select all contents of the archives for CMake-based packages to give CMake access to them.
+all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
+
 # GoogleTest/GoogleMock framework. Used by most unit-tests.
 # Last updated 2020-06-30.
 http_archive(
@@ -68,14 +81,23 @@ http_archive(
    url = "https://github.com/gflags/gflags/archive/v2.2.2.zip",
 )

-# glog v0.3.5
-# TODO: Migrate MediaPipe to use com_github_glog_glog on all platforms.
+# 2020-08-21
 http_archive(
-    name = "com_github_glog_glog_v_0_3_5",
-    url = "https://github.com/google/glog/archive/v0.3.5.zip",
-    sha256 = "267103f8a1e9578978aa1dc256001e6529ef593e5aea38193d31c2872ee025e8",
-    strip_prefix = "glog-0.3.5",
-    build_file = "@//third_party:glog.BUILD",
+    name = "com_github_glog_glog",
+    strip_prefix = "glog-0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6",
+    sha256 = "58c9b3b6aaa4dd8b836c0fd8f65d0f941441fb95e27212c5eeb9979cfd3592ab",
+    urls = [
+        "https://github.com/google/glog/archive/0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6.zip",
+    ],
+)
+http_archive(
+    name = "com_github_glog_glog_no_gflags",
+    strip_prefix = "glog-0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6",
+    sha256 = "58c9b3b6aaa4dd8b836c0fd8f65d0f941441fb95e27212c5eeb9979cfd3592ab",
+    build_file = "@//third_party:glog_no_gflags.BUILD",
+    urls = [
+        "https://github.com/google/glog/archive/0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6.zip",
+    ],
    patches = [
        "@//third_party:com_github_glog_glog_9779e5ea6ef59562b030248947f787d1256132ae.diff"
    ],
@@ -84,16 +106,6 @@ http_archive(
    ],
 )

-# 2020-02-16
-http_archive(
-    name = "com_github_glog_glog",
-    strip_prefix = "glog-3ba8976592274bc1f907c402ce22558011d6fc5e",
-    sha256 = "feca3c7e29a693cab7887409756d89d342d4a992d54d7c5599bebeae8f7b50be",
-    urls = [
-        "https://github.com/google/glog/archive/3ba8976592274bc1f907c402ce22558011d6fc5e.zip",
-    ],
-)
-
 # easyexif
 http_archive(
    name = "easyexif",
@@ -169,6 +181,13 @@ http_archive(
    sha256 = "5ba6d0db4e784621fda44a50c58bb23b0892684692f0c623e2063f9c19f192f1"
 )

+http_archive(
+    name = "opencv",
+    build_file_content = all_content,
+    strip_prefix = "opencv-3.4.10",
+    urls = ["https://github.com/opencv/opencv/archive/3.4.10.tar.gz"],
+)
+
 new_local_repository(
    name = "linux_opencv",
    build_file = "@//third_party:opencv_linux.BUILD",
@@ -184,13 +203,13 @@ new_local_repository(
 new_local_repository(
    name = "macos_opencv",
    build_file = "@//third_party:opencv_macos.BUILD",
-    path = "/usr",
+    path = "/usr/local/opt/opencv@3",
 )

 new_local_repository(
    name = "macos_ffmpeg",
    build_file = "@//third_party:ffmpeg_macos.BUILD",
-    path = "/usr",
+    path = "/usr/local/opt/ffmpeg",
 )

 new_local_repository(
@@ -301,24 +320,27 @@ load("@rules_jvm_external//:defs.bzl", "maven_install")
 maven_install(
    name = "maven",
    artifacts = [
-        "junit:junit:4.12",
-        "androidx.test.espresso:espresso-core:3.1.1",
-        "org.hamcrest:hamcrest-library:1.3",
        "androidx.concurrent:concurrent-futures:1.0.0-alpha03",
        "androidx.lifecycle:lifecycle-common:2.2.0",
        "androidx.annotation:annotation:aar:1.1.0",
        "androidx.appcompat:appcompat:aar:1.1.0-rc01",
-        "androidx.camera:camera-core:aar:1.0.0-alpha06",
-        "androidx.camera:camera-camera2:aar:1.0.0-alpha06",
+        "androidx.camera:camera-core:1.0.0-beta10",
+        "androidx.camera:camera-camera2:1.0.0-beta10",
+        "androidx.camera:camera-lifecycle:1.0.0-beta10",
        "androidx.constraintlayout:constraintlayout:aar:1.1.3",
        "androidx.core:core:aar:1.1.0-rc03",
        "androidx.legacy:legacy-support-v4:aar:1.0.0",
        "androidx.recyclerview:recyclerview:aar:1.1.0-beta02",
+        "androidx.test.espresso:espresso-core:3.1.1",
+        "com.github.bumptech.glide:glide:4.11.0",
        "com.google.android.material:material:aar:1.0.0-rc01",
        "com.google.code.findbugs:jsr305:3.0.2",
        "com.google.flogger:flogger-system-backend:0.3.1",
        "com.google.flogger:flogger:0.3.1",
        "com.google.guava:guava:27.0.1-android",
+        "com.google.guava:listenablefuture:1.0",
+        "junit:junit:4.12",
+        "org.hamcrest:hamcrest-library:1.3",
    ],
    repositories = [
        "https://jcenter.bintray.com",
@@ -342,13 +364,12 @@ http_archive(
 )

 #Tensorflow repo should always go after the other external dependencies.
-# 2020-05-11
-_TENSORFLOW_GIT_COMMIT = "7c09d15f9fcc14343343c247ebf5b8e0afe3e4aa"
-_TENSORFLOW_SHA256= "673d00cbd2676ae43df1993e0d28c10b5ffbe96d9e2ab29f88a77b43c0211299"
+# 2020-08-30
+_TENSORFLOW_GIT_COMMIT = "57b009e31e59bd1a7ae85ef8c0232ed86c9b71db"
+_TENSORFLOW_SHA256= "de7f5f06204e057383028c7e53f3b352cdf85b3a40981b1a770c9a415a792c0e"
 http_archive(
    name = "org_tensorflow",
    urls = [
-      "https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
      "https://github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
    ],
    patches = [
@@ -0,0 +1,108 @@
+#!/bin/bash
+# Copyright 2020 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =========================================================================
+#
+# Script to build/run all MediaPipe desktop example apps (with webcam input).
+#
+# To build and run all apps and store them in out_dir:
+#   $ ./build_ios_examples.sh -d out_dir
+#   Omitting -d and the associated directory saves all generated apps in the
+#   current directory.
+# To build all apps and store them in out_dir:
+#   $ ./build_ios_examples.sh -d out_dir -b
+#   Omitting -d and the associated directory saves all generated apps in the
+#   current directory.
+# To run all apps already stored in out_dir:
+#   $ ./build_ios_examples.sh -d out_dir -r
+#   Omitting -d and the associated directory assumes all apps are in the current
+#   directory.
+
+set -e
+
+out_dir="."
+build_only=false
+run_only=false
+app_dir="mediapipe/examples/desktop"
+bin_dir="bazel-bin"
+declare -a default_bazel_flags=(build -c opt --define MEDIAPIPE_DISABLE_GPU=1)
+
+while [[ -n $1 ]]; do
+  case $1 in
+    -d)
+      shift
+      out_dir=$1
+      ;;
+    -b)
+      build_only=true
+      ;;
+    -r)
+      run_only=true
+      ;;
+    *)
+      echo "Unsupported input argument $1."
+      exit 1
+      ;;
+  esac
+  shift
+done
+
+echo "app_dir: $app_dir"
+echo "out_dir: $out_dir"
+
+declare -a bazel_flags
+
+apps="${app_dir}/*"
+for app in ${apps}; do
+  if [[ -d "${app}" ]]; then
+    target_name=${app##*/}
+    if [[ "${target_name}" == "autoflip" ||
+          "${target_name}" == "hello_world" ||
+          "${target_name}" == "media_sequence" ||
+          "${target_name}" == "template_matching" ||
+          "${target_name}" == "youtube8m" ]]; then
+      continue
+    fi
+    target="${app}:${target_name}_cpu"
+
+    echo "=== Target: ${target}"
+
+    if [[ $run_only == false ]]; then
+      bazel_flags=("${default_bazel_flags[@]}")
+      bazel_flags+=(${target})
+
+      bazel "${bazel_flags[@]}"
+      cp -f "${bin_dir}/${app}/"*"_cpu" "${out_dir}"
+    fi
+    if [[ $build_only == false ]]; then
+      if  [[ ${target_name} == "multi_hand_tracking" ]]; then
+        graph_name="hand_tracking/multi_hand_tracking"
+      elif  [[ ${target_name} == "object_tracking" ]]; then
+        graph_name="tracking/object_detection_tracking"
+      elif [[ ${target_name} == "upper_body_pose_tracking" ]]; then
+        graph_name="pose_tracking/upper_body_pose_tracking"
+      else
+        graph_name="${target_name}/${target_name}"
+      fi
+      if [[ ${target_name} == "iris_tracking" ||
+            ${target_name} == "upper_body_pose_tracking" ]]; then
+        graph_suffix="cpu"
+      else
+        graph_suffix="desktop_live"
+      fi
+      GLOG_logtostderr=1 "${out_dir}/${target_name}_cpu" \
+        --calculator_graph_config_file=mediapipe/graphs/"${graph_name}_${graph_suffix}.pbtxt"
+    fi
+  fi
+done
@@ -22,8 +22,8 @@ aux_links:
 # Footer content appears at the bottom of every page's main content
 footer_content: "&copy; 2020 GOOGLE LLC | <a href=\"https://policies.google.com/privacy\">PRIVACY POLICY</a> | <a href=\"https://policies.google.com/terms\">TERMS OF SERVICE</a>"

-# Color scheme currently only supports "dark" or nil (default)
-color_scheme: nil
+# Color scheme currently only supports "dark", "light"/nil (default), or a custom scheme that you define
+color_scheme: mediapipe

 # Google Analytics Tracking (optional)
 ga_tracking: UA-140696581-2
@@ -0,0 +1 @@
+$link-color: #0097A7;
@@ -425,7 +425,47 @@ Note: This currently works only on Linux, and please first follow

 ## Python

-### Prerequisite
+MediaPipe Python package is available on
+[PyPI](https://pypi.org/project/mediapipe/), and can be installed simply by `pip
+install mediapipe` on Linux and macOS, as described below in
+[Run in python interpreter](#run-in-python-interpreter) and in this
+[colab](https://mediapipe.page.link/mp-py-colab).
+
+### Run in Python interpreter
+
+Using [MediaPipe Pose](../solutions/pose.md) as an example:
+
+```bash
+# Activate a Python virtual environment.
+$ python3 -m venv mp_env && source mp_env/bin/activate
+
+# Install MediaPipe Python package
+(mp_env)$ pip install mediapipe
+
+# Run in Python interpreter
+(mp_env)$ python3
+>>> import mediapipe as mp
+>>> pose_tracker = mp.examples.UpperBodyPoseTracker()
+
+# For image input
+>>> pose_landmarks, _ = pose_tracker.run(input_file='/path/to/input/file', output_file='/path/to/output/file')
+>>> pose_landmarks, annotated_image = pose_tracker.run(input_file='/path/to/file')
+
+# For live camera input
+# (Press Esc within the output image window to stop the run or let it self terminate after 30 seconds.)
+>>> pose_tracker.run_live()
+
+# Close the tracker.
+>>> pose_tracker.close()
+```
+
+Tip: Use command `deactivate` to exit the Python virtual environment.
+
+### Building Python package from source
+
+Follow these steps only if you have local changes and need to build the Python
+package from source. Otherwise, we strongly encourage our users to simply run
+`pip install mediapipe`, more convenient and much faster.

 1.  Make sure that Bazel and OpenCV are correctly installed and configured for
    MediaPipe. Please see [Installation](./install.md) for how to setup Bazel
@@ -445,50 +485,23 @@ Note: This currently works only on Linux, and please first follow
    $ brew install protobuf
    ```

-### Set up Python virtual environment.
-
-1.  Activate a Python virtual environment.
+3.  Activate a Python virtual environment.

    ```bash
    $ python3 -m venv mp_env && source mp_env/bin/activate
    ```

-2.  In the virtual environment, go to the MediaPipe repo directory.
+4.  In the virtual environment, go to the MediaPipe repo directory.

-3.  Install the required Python packages.
+5.  Install the required Python packages.

    ```bash
    (mp_env)mediapipe$ pip3 install -r requirements.txt
    ```

-4.  Generate and install MediaPipe package.
+6.  Generate and install MediaPipe package.

    ```bash
    (mp_env)mediapipe$ python3 setup.py gen_protos
-    (mp_env)mediapipe$ python3 setup.py install
+    (mp_env)mediapipe$ python3 setup.py install --link-opencv
    ```
-
-### Run in Python interpreter
-
-Make sure you are not in the MediaPipe repo directory.
-
-Using [MediaPipe Pose](../solutions/pose.md) as an example:
-
-```bash
-(mp_env)$ python3
->>> import mediapipe as mp
->>> pose_tracker = mp.examples.UpperBodyPoseTracker()
-
-# For image input
->>> pose_landmarks, _ = pose_tracker.run(input_file='/path/to/input/file', output_file='/path/to/output/file')
->>> pose_landmarks, annotated_image = pose_tracker.run(input_file='/path/to/file')
-
-# For live camera input
-# (Press Esc within the output image window to stop the run or let it self terminate after 30 seconds.)
->>> pose_tracker.run_live()
-
-# Close the tracker.
->>> pose_tracker.close()
-```
-
-Tip: Use command `deactivate` to exit the Python virtual environment.
@@ -446,8 +446,8 @@ visible so that we can start seeing frames from the `previewFrameTexture`.
 However, before starting the camera, we need to decide which camera we want to
 use. [`CameraXPreviewHelper`] inherits from [`CameraHelper`] which provides two
 options, `FRONT` and `BACK`. We can pass in the decision from the `BUILD` file
-as metadata such that no code change is required to build a another version of
-the app using a different camera.
+as metadata such that no code change is required to build another version of the
+app using a different camera.

 Assuming we want to use `BACK` camera to perform edge detection on a live scene
 that we view from the camera, add the metadata into `AndroidManifest.xml`:
@@ -42,16 +42,16 @@ apps, see these [instructions](./building_examples.md#ios).

    Follow the official
    [Bazel documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
-    to install Bazel 2.0 or higher.
+    to install Bazel 3.4 or higher.

    For Nvidia Jetson and Raspberry Pi devices with ARM Ubuntu, Bazel needs to
    be built from source.

    ```bash
-    # For Bazel 3.0.0
-    wget https://github.com/bazelbuild/bazel/releases/download/3.0.0/bazel-3.0.0-dist.zip
+    # For Bazel 3.4.0
+    wget https://github.com/bazelbuild/bazel/releases/download/3.4.0/bazel-3.4.0-dist.zip
    sudo apt-get install build-essential openjdk-8-jdk python zip unzip
-    unzip bazel-3.0.0-dist.zip
+    unzip bazel-3.4.0-dist.zip
    env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
    sudo cp output/bazel /usr/local/bin/
    ```
@@ -221,7 +221,7 @@ build issues.

    Follow the official
    [Bazel documentation](https://docs.bazel.build/versions/master/install-redhat.html)
-    to install Bazel 2.0 or higher.
+    to install Bazel 3.4 or higher.

 3.  Install OpenCV.

@@ -356,7 +356,7 @@ build issues.

    Option 2. Follow the official
    [Bazel documentation](https://docs.bazel.build/versions/master/install-os-x.html#install-with-installer-mac-os-x)
-    to install Bazel 2.0 or higher.
+    to install Bazel 3.4 or higher.

 4.  Install OpenCV and FFmpeg.

@@ -427,7 +427,6 @@ build issues.
        linkstatic = 1,
        visibility = ["//visibility:public"],
    )
-
    ```

 5.  Make sure that Python 3 and the Python "six" library are installed.
@@ -506,7 +505,7 @@ next section.

    Follow the official
    [Bazel documentation](https://docs.bazel.build/versions/master/install-windows.html)
-    to install Bazel 2.0 or higher.
+    to install Bazel 3.4 or higher.

 6.  Set Bazel variables.

@@ -567,7 +566,6 @@ next section.
    # I20200514 20:43:12.279618  1200 hello_world.cc:56] Hello World!
    # I20200514 20:43:12.279618  1200 hello_world.cc:56] Hello World!
    # I20200514 20:43:12.280613  1200 hello_world.cc:56] Hello World!
-
    ```

 If you run into a build error, please read
@@ -607,14 +605,14 @@ cameras. Alternatively, you use a video file as input.

    ```bash
    username@DESKTOP-TMVLBJ1:~$ curl -sLO --retry 5 --retry-max-time 10 \
-    https://storage.googleapis.com/bazel/3.0.0/release/bazel-3.0.0-installer-linux-x86_64.sh && \
-    sudo mkdir -p /usr/local/bazel/3.0.0 && \
-    chmod 755 bazel-3.0.0-installer-linux-x86_64.sh && \
-    sudo ./bazel-3.0.0-installer-linux-x86_64.sh --prefix=/usr/local/bazel/3.0.0 && \
-    source /usr/local/bazel/3.0.0/lib/bazel/bin/bazel-complete.bash
+    https://storage.googleapis.com/bazel/3.4.0/release/bazel-3.4.0-installer-linux-x86_64.sh && \
+    sudo mkdir -p /usr/local/bazel/3.4.0 && \
+    chmod 755 bazel-3.4.0-installer-linux-x86_64.sh && \
+    sudo ./bazel-3.4.0-installer-linux-x86_64.sh --prefix=/usr/local/bazel/3.4.0 && \
+    source /usr/local/bazel/3.4.0/lib/bazel/bin/bazel-complete.bash

-    username@DESKTOP-TMVLBJ1:~$ /usr/local/bazel/3.0.0/lib/bazel/bin/bazel version && \
-    alias bazel='/usr/local/bazel/3.0.0/lib/bazel/bin/bazel'
+    username@DESKTOP-TMVLBJ1:~$ /usr/local/bazel/3.4.0/lib/bazel/bin/bazel version && \
+    alias bazel='/usr/local/bazel/3.4.0/lib/bazel/bin/bazel'
    ```

 6.  Checkout MediaPipe repository.
@@ -8,46 +8,50 @@ nav_order: 1

 --------------------------------------------------------------------------------

-## Cross-platform ML solutions made simple
+## Live ML anywhere

-[MediaPipe](https://google.github.io/mediapipe/) is the simplest way for researchers
-and developers to build world-class ML solutions and applications for mobile,
-desktop/cloud, web and IoT devices.
+[MediaPipe](https://google.github.io/mediapipe/) offers cross-platform, customizable
+ML solutions for live and streaming media.

 ![accelerated.png](images/accelerated_small.png)                                                               | ![cross_platform.png](images/cross_platform_small.png)
 :------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------:
-***End-to-End acceleration***: *built-in fast ML inference and processing accelerated even on common hardware* | ***Build one, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT*
+***End-to-End acceleration***: *Built-in fast ML inference and processing accelerated even on common hardware* | ***Build once, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT*
 ![ready_to_use.png](images/ready_to_use_small.png)                                                             | ![open_source.png](images/open_source_small.png)
 ***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework*            | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*

 ## ML solutions in MediaPipe

-Face Detection                                                                                                                 | Face Mesh                                                                                                       | Iris 🆕                                                                                                    | Hands                                                                                                      | Pose 🆕
-:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :----:
-[![face_detection](images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose)
+Face Detection                                                                                                                 | Face Mesh                                                                                                       | Iris                                                                                                      | Hands                                                                                                      | Pose                                                                                                      | Hair Segmentation
+:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------:
+[![face_detection](images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose) | [![hair_segmentation](images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation)

-Hair Segmentation                                                                                                                       | Object Detection                                                                                                                     | Box Tracking                                                                                                                | Objectron                                                                                                             | KNIFT
-:-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
-[![hair_segmentation](images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) | [![object_detection](images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
+Object Detection                                                                                                                     | Box Tracking                                                                                                                | Instant Motion Tracking                                                                                                                               | Objectron                                                                                                             | KNIFT
+:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
+[![object_detection](images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![instant_motion_tracking](images/mobile/instant_motion_tracking_android_small.gif)](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [![objectron](images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)

 <!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
 <!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->

-[]()                                                                          | Android | iOS | Desktop | Python | Web | Coral
-:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
-[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)       | ✅       | ✅   | ✅       |        | ✅   | ✅
-[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                 | ✅       | ✅   | ✅       |        |     |
-[Iris](https://google.github.io/mediapipe/solutions/iris) 🆕                         | ✅       | ✅   | ✅       |        | ✅   |
-[Hands](https://google.github.io/mediapipe/solutions/hands)                         | ✅       | ✅   | ✅       |        | ✅   |
-[Pose](https://google.github.io/mediapipe/solutions/pose) 🆕                         | ✅       | ✅   | ✅       | ✅      | ✅   |
-[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅       |     | ✅       |        | ✅   |
-[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)   | ✅       | ✅   | ✅       |        |     | ✅
-[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)           | ✅       | ✅   | ✅       |        |     |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                 | ✅       |     |         |        |     |
-[KNIFT](https://google.github.io/mediapipe/solutions/knift)                         | ✅       |     |         |        |     |
-[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                   |         |     | ✅       |        |     |
-[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)        |         |     | ✅       |        |     |
-[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)               |         |     | ✅       |        |     |
+[]()                                                                                      | Android | iOS | Desktop | Python | Web | Coral
+:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
+[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)                   | ✅       | ✅   | ✅       |        | ✅   | ✅
+[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                             | ✅       | ✅   | ✅       |        |     |
+[Iris](https://google.github.io/mediapipe/solutions/iris)                                       | ✅       | ✅   | ✅       |        | ✅   |
+[Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅       | ✅   | ✅       |        | ✅   |
+[Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅       | ✅   | ✅       | ✅      | ✅   |
+[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅       |     | ✅       |        | ✅   |
+[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅       | ✅   | ✅       |        |     | ✅
+[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅       | ✅   | ✅       |        |     |
+[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅       |     |         |        |     |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅       |     |         |        |     |
+[KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅       |     |         |        |     |
+[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |         |     | ✅       |        |     |
+[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |         |     | ✅       |        |     |
+[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)                           |         |     | ✅       |        |     |
+
+See also
+[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
+for ML models released in MediaPipe.

 ## MediaPipe on the Web

@@ -88,7 +92,11 @@ run code search using

 ## Publications

-*   [BlazePose - On-device Real-time Body Pose Tracking](https://mediapipe.page.link/blazepose-blog)
+*   [MediaPipe 3D Face Transform](https://developers.googleblog.com/2020/09/mediapipe-3d-face-transform.html)
+    in Google Developers Blog
+*   [Instant Motion Tracking With MediaPipe](https://developers.googleblog.com/2020/08/instant-motion-tracking-with-mediapipe.html)
+    in Google Developers Blog
+*   [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
    in Google AI Blog
 *   [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
    in Google AI Blog
@@ -2,7 +2,7 @@
 layout: default
 title: AutoFlip (Saliency-aware Video Cropping)
 parent: Solutions
-nav_order: 11
+nav_order: 12
 ---

 # AutoFlip: Saliency-aware Video Cropping
@@ -102,9 +102,4 @@ to cross-compile and run MediaPipe examples on the
    [BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs](https://arxiv.org/abs/1907.05047)
    ([presentation](https://docs.google.com/presentation/d/1YCtASfnYyZtH-41QvnW5iZxELFnf0MF-pPWSLGj8yjQ/present?slide=id.g5bc8aeffdd_1_0))
    ([poster](https://drive.google.com/file/d/1u6aB6wxDY7X2TmeUUKgFydulNtXkb3pu/view))
-*   For front-facing/selfie camera:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite),
-    [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
-*   For back-facing camera:
-    [TFLite model ](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_back.tflite)
-*   [Model card](https://mediapipe.page.link/blazeface-mc)
+*   [Models and model cards](./models.md#face_detection)
@@ -19,13 +19,18 @@ landmarks in real-time even on mobile devices. It employs machine learning (ML)
 to infer the 3D surface geometry, requiring only a single camera input without
 the need for a dedicated depth sensor. Utilizing lightweight model architectures
 together with GPU acceleration throughout the pipeline, the solution delivers
-real-time performance critical for live experiences. The core of the solution is
-the same as what powers
-[YouTube Stories](https://youtube-creators.googleblog.com/2018/11/introducing-more-ways-to-share-your.html)'
-creator effects, the
-[Augmented Faces API in ARCore](https://developers.google.com/ar/develop/java/augmented-faces/)
-and the
-[ML Kit Face Contour Detection API](https://firebase.google.com/docs/ml-kit/face-detection-concepts#contours).
+real-time performance critical for live experiences.
+
+Additionally, the solution is bundled with the Face Geometry module that bridges
+the gap between the face landmark estimation and useful real-time augmented
+reality (AR) applications. It establishes a metric 3D space and uses the face
+landmark screen positions to estimate face geometry within that space. The face
+geometry data consists of common 3D geometry primitives, including a face pose
+transformation matrix and a triangular face mesh. Under the hood, a lightweight
+statistical analysis method called
+[Procrustes Analysis](https://en.wikipedia.org/wiki/Procrustes_analysis) is
+employed to drive a robust, performant and portable logic. The analysis runs on
+CPU and has a minimal speed/memory footprint on top of the ML model inference.

 ![face_mesh_ar_effects.gif](../images/face_mesh_ar_effects.gif) |
 :-------------------------------------------------------------: |
@@ -67,15 +72,15 @@ Note: To visualize a graph, copy the graph and paste it into
 to visualize its associated subgraphs, please see
 [visualizer documentation](../tools/visualizer.md).

-## Models
+### Models

-### Face Detection Model
+#### Face Detection Model

 The face detector is the same [BlazeFace](https://arxiv.org/abs/1907.05047)
 model used in [MediaPipe Face Detection](./face_detection.md). Please refer to
 [MediaPipe Face Detection](./face_detection.md) for details.

-### Face Landmark Model
+#### Face Landmark Model

 For 3D face landmarks we employed transfer learning and trained a network with
 several objectives: the network simultaneously predicts 3D landmark coordinates
@@ -98,7 +103,108 @@ You can find more information about the face landmark model in this

 ![face_mesh_android_gpu.gif](../images/mobile/face_mesh_android_gpu.gif)   |
 :------------------------------------------------------------------------: |
-*Fig 2. Output of MediaPipe Face Mesh: the red box indicates the cropped area as input to the landmark model, the red dots represent the 468 landmarks in 3D, and the green lines connecting landmarks illustrate the contours around the eyes, eyebrows, lips and the entire face.* |
+*Fig 2. Face landmarks: the red box indicates the cropped area as input to the landmark model, the red dots represent the 468 landmarks in 3D, and the green lines connecting landmarks illustrate the contours around the eyes, eyebrows, lips and the entire face.* |
+
+## Face Geometry Module
+
+The [Face Landmark Model](#face-landmark-model) performs a single-camera face landmark
+detection in the screen coordinate space: the X- and Y- coordinates are
+normalized screen coordinates, while the Z coordinate is relative and is scaled
+as the X coodinate under the
+[weak perspective projection camera model](https://en.wikipedia.org/wiki/3D_projection#Weak_perspective_projection).
+This format is well-suited for some applications, however it does not directly
+enable the full spectrum of augmented reality (AR) features like aligning a
+virtual 3D object with a detected face.
+
+The
+[Face Geometry module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry)
+moves away from the screen coordinate space towards a metric 3D space and
+provides necessary primitives to handle a detected face as a regular 3D object.
+By design, you'll be able to use a perspective camera to project the final 3D
+scene back into the screen coordinate space with a guarantee that the face
+landmark positions are not changed.
+
+### Key Concepts
+
+#### Metric 3D Space
+
+The **Metric 3D space** established within the Face Geometry module is a
+right-handed orthonormal metric 3D coordinate space. Within the space, there is
+a **virtual perspective camera** located at the space origin and pointed in the
+negative direction of the Z-axis. In the current pipeline, it is assumed that
+the input camera frames are observed by exactly this virtual camera and
+therefore its parameters are later used to convert the screen landmark
+coordinates back into the Metric 3D space. The *virtual camera parameters* can
+be set freely, however for better results it is advised to set them as close to
+the *real physical camera parameters* as possible.
+
+![face_geometry_metric_3d_space.gif](../images/face_geometry_metric_3d_space.gif) |
+:----------------------------------------------------------------------------: |
+*Fig 3. A visualization of multiple key elements in the Metric 3D space.*      |
+
+#### Canonical Face Model
+
+The **Canonical Face Model** is a static 3D model of a human face, which follows
+the 468 3D face landmark topology of the
+[Face Landmark Model](#face-landmark-model). The model bears two important
+functions:
+
+-   **Defines metric units**: the scale of the canonical face model defines the
+    metric units of the Metric 3D space. A metric unit used by the
+    [default canonical face model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/data/canonical_face_model.fbx)
+    is a centimeter;
+-   **Bridges static and runtime spaces**: the face pose transformation matrix
+    is - in fact - a linear map from the canonical face model into the runtime
+    face landmark set estimated on each frame. This way, virtual 3D assets
+    modeled around the canonical face model can be aligned with a tracked face
+    by applying the face pose transformation matrix to them.
+
+### Components
+
+#### Geometry Pipeline
+
+The **Geometry Pipeline** is a key component, which is responsible for
+estimating face geometry objects within the Metric 3D space. On each frame, the
+following steps are executed in the given order:
+
+-   Face landmark screen coordinates are converted into the Metric 3D space
+    coordinates;
+-   Face pose transformation matrix is estimated as a rigid linear mapping from
+    the canonical face metric landmark set into the runtime face metric landmark
+    set in a way that minimizes a difference between the two;
+-   A face mesh is created using the runtime face metric landmarks as the vertex
+    positions (XYZ), while both the vertex texture coordinates (UV) and the
+    triangular topology are inherited from the canonical face model.
+
+The geometry pipeline is implemented as a MediaPipe
+[calculator](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/geometry_pipeline_calculator.cc).
+For your convenience, the face geometry pipeline calculator is bundled together
+with the face landmark module into a unified MediaPipe
+[subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/face_geometry_front_gpu.pbtxt).
+The face geometry format is defined as a Protocol Buffer
+[message](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/protos/face_geometry.proto).
+
+#### Effect Renderer
+
+The **Effect Renderer** is a component, which serves as a working example of a
+face effect renderer. It targets the *OpenGL ES 2.0* API to enable a real-time
+performance on mobile devices and supports the following rendering modes:
+
+-   **3D object rendering mode**: a virtual object is aligned with a detected
+    face to emulate an object attached to the face (example: glasses);
+-   **Face mesh rendering mode**: a texture is stretched on top of the face mesh
+    surface to emulate a face painting technique.
+
+In both rendering modes, the face mesh is first rendered as an occluder straight
+into the depth buffer. This step helps to create a more believable effect via
+hiding invisible elements behind the face surface.
+
+The effect renderer is implemented as a MediaPipe
+[calculator](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/effect_renderer_calculator.cc).
+
+| ![face_geometry_renderer.gif](../images/face_geometry_renderer.gif)     |
+| :---------------------------------------------------------------------: |
+| *Fig 4. An example of face effects rendered by the Face Geometry Effect Renderer.* |

 ## Example Apps

@@ -111,7 +217,12 @@ Note: To visualize a graph, copy the graph and paste it into
 to visualize its associated subgraphs, please see
 [visualizer documentation](../tools/visualizer.md).

-### Mobile
+### Face Landmark Example
+
+Face landmark example showcases real-time, cross-platform face landmark
+detection. For visual reference, please refer to *Fig. 2*.
+
+#### Mobile

 *   Graph:
    [`mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt)
@@ -127,7 +238,7 @@ it, for Android modify `NUM_FACES` in
 and for iOS modify `kNumFaces` in
 [FaceMeshGpuViewController.mm](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facemeshgpu/FaceMeshGpuViewController.mm).

-### Desktop
+#### Desktop

 *   Running on CPU
    *   Graph:
@@ -143,18 +254,35 @@ and for iOS modify `kNumFaces` in
 Tip: Maximum number of faces to detect/process is set to 1 by default. To change
 it, in the graph file modify the option of `ConstantSidePacketCalculator`.

+### Face Effect Example
+
+Face effect example showcases real-time mobile face effect application use case
+for the Face Mesh solution. To enable a better user experience, this example
+only works for a single face. For visual reference, please refer to *Fig. 4*.
+
+#### Mobile
+
+*   Graph:
+    [`mediapipe/graphs/face_effect/face_effect_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_effect/face_effect_gpu.pbtxt)
+*   Android target:
+    [(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1ccnaDnffEuIXriBZr2SK_Eu4FpO7K44s)
+    [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/faceeffect`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/faceeffect/BUILD)
+*   iOS target:
+    [`mediapipe/examples/ios/faceeffect`](http:/mediapipe/examples/ios/faceeffect/BUILD)
+
 ## Resources

 *   Google AI Blog:
    [Real-Time AR Self-Expression with Machine Learning](https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html)
 *   TensorFlow Blog:
    [Face and hand tracking in the browser with MediaPipe and TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html)
+*   Google Developers Blog:
+    [MediaPipe 3D Face Transform](https://developers.googleblog.com/2020/09/mediapipe-3d-face-transform.html)
 *   Paper:
    [Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://arxiv.org/abs/1907.06724)
    ([poster](https://docs.google.com/presentation/d/1-LWwOMO9TzEVdrZ1CS1ndJzciRHfYDJfbSxH_ke_JRg/present?slide=id.g5986dd4b4c_4_212))
-*   Face detection model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite)
-*   Face landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
-    [TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
-*   [Model card](https://mediapipe.page.link/facemesh-mc)
+*   Canonical face model:
+    [FBX](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/data/canonical_face_model.fbx),
+    [OBJ](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/data/canonical_face_model.obj),
+    [UV visualization](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png)
+*   [Models and model cards](./models.md#face_mesh)
@@ -54,5 +54,4 @@ Please refer to [these instructions](../index.md#mediapipe-on-the-web).
    [Real-time Hair segmentation and recoloring on Mobile GPUs](https://arxiv.org/abs/1907.06740)
    ([presentation](https://drive.google.com/file/d/1C8WYlWdDRNtU1_pYBvkkG5Z5wqYqf0yj/view))
    ([supplementary video](https://drive.google.com/file/d/1LPtM99Ch2ogyXYbDNpEqnUfhFq0TfLuf/view))
-*   [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite)
-*   [Model card](https://mediapipe.page.link/hairsegmentation-mc)
+*   [Models and model cards](./models.md#hair_segmentation)
@@ -226,10 +226,4 @@ Please refer to [these instructions](../index.md#mediapipe-on-the-web).
 *   Paper:
    [MediaPipe Hands: On-device Real-time Hand Tracking](https://arxiv.org/abs/2006.10214)
    ([presentation](https://www.youtube.com/watch?v=I-UOrvxxXEk))
-*   Palm detection model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/palm_detection.tflite),
-    [TF.js model](https://tfhub.dev/mediapipe/handdetector/1)
-*   Hand landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark.tflite),
-    [TF.js model](https://tfhub.dev/mediapipe/handskeleton/1)
-*   [Model card](https://mediapipe.page.link/handmc)
+*   [Models and model cards](./models.md#hands)
@@ -0,0 +1,122 @@
+---
+layout: default
+title: Instant Motion Tracking
+parent: Solutions
+nav_order: 9
+---
+
+# MediaPipe Instant Motion Tracking
+{: .no_toc }
+
+1. TOC
+{:toc}
+---
+
+## Overview
+
+Augmented Reality (AR) technology creates fun, engaging, and immersive user
+experiences. The ability to perform AR tracking across devices and platforms,
+without initialization, remains important to power AR applications at scale.
+
+MediaPipe Instant Motion Tracking provides AR tracking across devices and
+platforms without initialization or calibration. It is built upon the
+[MediaPipe Box Tracking](./box_tracking.md) solution. With Instant Motion
+Tracking, you can easily place virtual 2D and 3D content on static or moving
+surfaces, allowing them to seamlessly interact with the real-world environment.
+
+![instant_motion_tracking_android_small](../images/mobile/instant_motion_tracking_android_small.gif) |
+:-----------------------------------------------------------------------: |
+*Fig 1. Instant Motion Tracking is used to augment the world with a 3D sticker.* |
+
+## Pipeline
+
+The Instant Motion Tracking pipeline is implemented as a MediaPipe
+[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt),
+which internally utilizes a
+[RegionTrackingSubgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/subgraphs/region_tracking.pbtxt)
+in order to perform anchor tracking for each individual 3D sticker.
+
+We first use a
+[StickerManagerCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/calculators/sticker_manager_calculator.cc)
+to prepare the individual sticker data for the rest of the application. This
+information is then sent to the
+[RegionTrackingSubgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/subgraphs/region_tracking.pbtxt)
+that performs 3D region tracking for sticker placement and rendering. Once
+acquired, our tracked sticker regions are sent with user transformations (i.e.
+gestures from the user to rotate and zoom the sticker) and IMU data to the
+[MatricesManagerCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/calculators/matrices_manager_calculator.cc),
+which turns all our sticker transformation data into a set of model matrices.
+This data is handled directly by our
+[GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
+as an input stream, which will render the provided texture and object file using
+our matrix specifications. The output of
+[GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
+is a video stream depicting the virtual 3D content rendered on top of the real
+world, creating immersive AR experiences for users.
+
+## Using Instant Motion Tracking
+
+With the Instant Motion Tracking MediaPipe [graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt),
+an application can create an interactive and realistic AR experience by
+specifying the required input streams, side packets, and output streams.
+The input streams are the following:
+
+*    Input Video (GpuBuffer): Video frames to render augmented stickers onto.
+*    Rotation Matrix (9-element Float Array): The 3x3 row-major rotation
+matrix from the device IMU to determine proper orientation of the device.
+*    Sticker Proto String (String): A string representing the
+serialized [sticker buffer protobuf message](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/calculators/sticker_buffer.proto),
+containing a list of all stickers and their attributes.
+  *    Each sticker in the Protobuffer has a unique ID to find associated
+  anchors and transforms, an initial anchor placement in a normalized [0.0, 1.0]
+  3D space, a user rotation and user scaling transform on the sticker,
+  and an integer indicating which type of objects to render for the
+  sticker (e.g. 3D asset or GIF).
+*    Sticker Sentinel (Integer): When an anchor must be initially placed or
+repositioned, this value must be changed to the ID of the anchor to reset from
+the sticker buffer protobuf message. If no valid ID is provided, the system
+will simply maintain tracking.
+
+Side packets are also an integral part of the Instant Motion Tracking solution
+to provide device-specific information for the rendering system:
+
+*   Field of View (Float): The field of view of the camera in radians.
+*   Aspect Ratio (Float): The aspect ratio (width / height) of the camera frames
+    (this ratio corresponds to the image frames themselves, not necessarily the
+    screen bounds).
+*   Object Asset (String): The
+    [GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
+    must be provided with an associated asset file name pointing to the 3D model
+    to render in the viewfinder.
+*   (Optional) Texture (ImageFrame on Android, GpuBuffer on iOS): Textures for
+    the
+    [GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
+    can be provided either via an input stream (dynamic texturing) or as a side
+    packet (unchanging texture).
+
+The rendering system for the Instant Motion Tracking is powered by OpenGL. For
+more information regarding the structure of model matrices and OpenGL rendering,
+please visit [OpenGL Wiki](https://www.khronos.org/opengl/wiki/). With the
+specifications above, the Instant Motion Tracking capabilities can be adapted to
+any device that is able to run the MediaPipe framework with a working IMU system
+and connected camera.
+
+## Example Apps
+
+Please first see general instructions for
+[Android](../getting_started/building_examples.md#android) on how to build
+MediaPipe examples.
+
+* Graph: [mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt)
+
+* Android target (or download prebuilt [ARM64 APK](https://drive.google.com/file/d/1KnaBBoKpCHR73nOBJ4fL_YdWVTAcwe6L/view?usp=sharing)):
+[`mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking:instantmotiontracking`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/BUILD)
+
+## Resources
+
+*   Google Developers Blog:
+    [Instant Motion Tracking With MediaPipe](https://developers.googleblog.com/2020/08/instant-motion-tracking-with-mediapipe.html)
+*   Google AI Blog:
+    [The Instant Motion Tracking Behind Motion Stills AR](https://ai.googleblog.com/2018/02/the-instant-motion-tracking-behind.html)
+*   Paper:
+    [Instant Motion Tracking and Its Applications to Augmented Reality](https://arxiv.org/abs/1907.06796)
@@ -55,7 +55,7 @@ that uses a
 from the
 [face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark),
 an
-[iris landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_tracking/iris_landmark_left_and_right_gpu.pbtxt)
+[iris landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark_left_and_right_gpu.pbtxt)
 from the
 [iris landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark),
 and renders using a dedicated
@@ -72,6 +72,11 @@ Note: To visualize a graph, copy the graph and paste it into
 to visualize its associated subgraphs, please see
 [visualizer documentation](../tools/visualizer.md).

+The output of the pipeline is a set of 478 3D landmarks, including 468 face
+landmarks from [MediaPipe Face Mesh](./face_mesh.md), with those around the eyes
+further refined (see Fig 2), and 10 additional iris landmarks appended at the
+end (5 for each eye, and see Fig 2 also).
+
 ## Models

 ### Face Detection Model
@@ -194,11 +199,4 @@ Please refer to [these instructions](../index.md#mediapipe-on-the-web).
 *   Paper:
    [Real-time Pupil Tracking from Monocular Video for Digital Puppetry](https://arxiv.org/abs/2006.11341)
    ([presentation](https://youtu.be/cIhXkiiapQI))
-*   Face detection model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite)
-*   Face landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
-    [TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
-*   Iris landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark.tflite)
-*   [Model card](https://mediapipe.page.link/iris-mc)
+*   [Models and model cards](./models.md#iris)
@@ -2,7 +2,7 @@
 layout: default
 title: KNIFT (Template-based Feature Matching)
 parent: Solutions
-nav_order: 10
+nav_order: 11
 ---

 # MediaPipe KNIFT
@@ -139,7 +139,4 @@ to run regular TFLite inference.

 *   Google Developers Blog:
    [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
-*   [TFLite model for up to 200 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float.tflite)
-*   [TFLite model for up to 400 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float_400.tflite)
-*   [TFLite model for up to 1000 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float_1k.tflite)
-*   [Model card](https://mediapipe.page.link/knift-mc)
+*   [Models and model cards](./models.md#knift)
@@ -2,7 +2,7 @@
 layout: default
 title: Dataset Preparation with MediaSequence
 parent: Solutions
-nav_order: 12
+nav_order: 13
 ---

 # Dataset Preparation with MediaSequence
@@ -0,0 +1,77 @@
+---
+layout: default
+title: Models and Model Cards
+parent: Solutions
+nav_order: 30
+---
+
+# MediaPipe Models and Model Cards
+{: .no_toc }
+
+1. TOC
+{:toc}
+---
+
+### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
+
+*   Face detection model for front-facing/selfie camera:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite),
+    [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
+*   Face detection model for back-facing camera:
+    [TFLite model ](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_back.tflite)
+*   [Model card](https://mediapipe.page.link/blazeface-mc)
+
+### [Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)
+
+*   Face landmark model:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
+    [TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
+*   [Model card](https://mediapipe.page.link/facemesh-mc)
+
+### [Iris](https://google.github.io/mediapipe/solutions/iris)
+
+*   Iris landmark model:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark.tflite)
+*   [Model card](https://mediapipe.page.link/iris-mc)
+
+### [Hands](https://google.github.io/mediapipe/solutions/hands)
+
+*   Palm detection model:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/palm_detection.tflite),
+    [TF.js model](https://tfhub.dev/mediapipe/handdetector/1)
+*   Hand landmark model:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark.tflite),
+    [TF.js model](https://tfhub.dev/mediapipe/handskeleton/1)
+*   [Model card](https://mediapipe.page.link/handmc)
+
+### [Pose](https://google.github.io/mediapipe/solutions/pose)
+
+*   Pose detection model:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection/pose_detection.tflite)
+*   Upper-body pose landmark model:
+    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_upper_body.tflite)
+*   [Model card](https://mediapipe.page.link/blazepose-mc)
+
+### [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)
+
+*   [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite)
+*   [Model card](https://mediapipe.page.link/hairsegmentation-mc)
+
+### [Object Detection](https://google.github.io/mediapipe/solutions/object_detection)
+
+*   [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/ssdlite_object_detection.tflite)
+*   [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/object-detector-quantized_edgetpu.tflite)
+*   [TensorFlow model](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model)
+*   [Model information](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md)
+
+### [Objectron](https://google.github.io/mediapipe/solutions/objectron)
+
+*   [TFLite model for shoes](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_3d_sneakers.tflite)
+*   [TFLite model for chairs](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_3d_chair.tflite)
+
+### [KNIFT](https://google.github.io/mediapipe/solutions/knift)
+
+*   [TFLite model for up to 200 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float.tflite)
+*   [TFLite model for up to 400 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float_400.tflite)
+*   [TFLite model for up to 1000 keypoints](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_float_1k.tflite)
+*   [Model card](https://mediapipe.page.link/knift-mc)
@@ -144,7 +144,4 @@ to cross-compile and run MediaPipe examples on the

 ## Resources

-*   [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/ssdlite_object_detection.tflite)
-*   [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/object-detector-quantized_edgetpu.tflite)
-*   [TensorFlow model](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model)
-*   [Model information](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md)
+*   [Models and model cards](./models.md#object_detection)
@@ -2,7 +2,7 @@
 layout: default
 title: Objectron (3D Object Detection)
 parent: Solutions
-nav_order: 9
+nav_order: 10
 ---

 # MediaPipe Objectron
@@ -161,7 +161,7 @@ to visualize its associated subgraphs, please see
 ### Objectron for Shoes

 *   Graph:
-    [`mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/shoe_classic_occlusion_tracking.pbtxt)
+    [`mediapipe/graphs/object_detection_3d/shoe_classic_occlusion_tracking.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/shoe_classic_occlusion_tracking.pbtxt)
 *   Android target:
    [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1S0K4hbWt3o31FfQ4QU3Rz7IHrvOUMx1d)
    [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetection3d:objectdetection3d`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetection3d/BUILD)
@@ -191,5 +191,4 @@ to visualize its associated subgraphs, please see
 *   Paper:
    [Instant 3D Object Tracking with Applications in Augmented Reality](https://drive.google.com/open?id=1O_zHmlgXIzAdKljp20U_JUkEHOGG52R8)
    ([presentation](https://www.youtube.com/watch?v=9ndF1AIo7h0))
-*   [TFLite model for shoes](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_3d_sneakers.tflite)
-*   [TFLite model for chairs](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_3d_chair.tflite)
+*   [Models and model cards](./models.md#objectron)
@@ -5,7 +5,7 @@ parent: Solutions
 nav_order: 5
 ---

-# MediaPipe Pose
+# MediaPipe BlazePose
 {: .no_toc }

 1. TOC
@@ -22,12 +22,13 @@ on top of the physical world in augmented reality.

 MediaPipe Pose is a ML solution for high-fidelity upper-body pose tracking,
 inferring 25 2D upper-body landmarks from RGB video frames utilizing our
-[BlazePose](https://mediapipe.page.link/blazepose-blog) research. Current
-state-of-the-art approaches rely primarily on powerful desktop environments for
-inference, whereas our method achieves real-time performance on most modern
-[mobile phones](#mobile), [desktops/laptops](#desktop), in [python](#python) and
-even on the [web](#web). A variant of MediaPipe Pose that performs full-body
-pose tracking on mobile phones will be included in an upcoming release of
+[BlazePose](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
+research. Current state-of-the-art approaches rely primarily on powerful desktop
+environments for inference, whereas our method achieves real-time performance on
+most modern [mobile phones](#mobile), [desktops/laptops](#desktop), in
+[python](#python) and even on the [web](#web). A variant of MediaPipe Pose that
+performs full-body pose tracking on mobile phones will be included in an
+upcoming release of
 [ML Kit](https://developers.google.com/ml-kit/early-access/pose-detection).

 ![pose_tracking_upper_body_example.gif](../images/mobile/pose_tracking_upper_body_example.gif) |
@@ -87,12 +88,14 @@ hip midpoints.
 ### Pose Landmark Model (BlazePose Tracker)

 The landmark model currently included in MediaPipe Pose predicts the location of
-25 upper-body landmarks (see figure below), with three degrees of freedom each
-(x, y location and visibility), plus two virtual alignment keypoints. It shares
-the same architecture as the full-body version that predicts 33 landmarks,
-described in more detail in the
-[BlazePose Google AI Blog](https://mediapipe.page.link/blazepose-blog) and in
-this [paper](https://arxiv.org/abs/2006.10204).
+25 upper-body landmarks (see figure below), each with `(x, y, z, visibility)`,
+plus two virtual alignment keypoints. Note that the `z` value should be
+discarded as the model is currently not fully trained to predict depth, but this
+is something we have on the roadmap. The model shares the same architecture as
+the full-body version that predicts 33 landmarks, described in more detail in
+the
+[BlazePose Google AI Blog](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
+and in this [paper](https://arxiv.org/abs/2006.10204).

 ![pose_tracking_upper_body_landmarks.png](../images/mobile/pose_tracking_upper_body_landmarks.png) |
 :------------------------------------------------------------------------------------------------: |
@@ -141,10 +144,21 @@ MediaPipe examples.

 ### Python

-Please first see general instructions for
-[Python](../getting_started/building_examples.md#python) examples.
+MediaPipe Python package is available on
+[PyPI](https://pypi.org/project/mediapipe/), and can be installed simply by `pip
+install mediapipe` on Linux and macOS, as described below and in this
+[colab](https://mediapipe.page.link/mp-py-colab). If you do need to build the
+Python package from source, see
+[additional instructions](../getting_started/building_examples.md#python).

 ```bash
+# Activate a Python virtual environment.
+$ python3 -m venv mp_env && source mp_env/bin/activate
+
+# Install MediaPipe Python package
+(mp_env)$ pip install mediapipe
+
+# Run in Python interpreter
 (mp_env)$ python3
 >>> import mediapipe as mp
 >>> pose_tracker = mp.examples.UpperBodyPoseTracker()
@@ -152,6 +166,9 @@ Please first see general instructions for
 # For image input
 >>> pose_landmarks, _ = pose_tracker.run(input_file='/path/to/input/file', output_file='/path/to/output/file')
 >>> pose_landmarks, annotated_image = pose_tracker.run(input_file='/path/to/file')
+# To print out the pose landmarks, you can simply do "print(pose_landmarks)".
+# However, the data points can be more accessible with the following approach.
+>>> [print('x is', data_point.x, 'y is', data_point.y, 'z is', data_point.z, 'visibility is', data_point.visibility) for data_point in pose_landmarks.landmark]

 # For live camera input
 # (Press Esc within the output image window to stop the run or let it self terminate after 30 seconds.)
@@ -161,6 +178,8 @@ Please first see general instructions for
 >>> pose_tracker.close()
 ```

+Tip: Use command `deactivate` to exit the Python virtual environment.
+
 ### Web

 Please refer to [these instructions](../index.md#mediapipe-on-the-web).
@@ -168,12 +187,8 @@ Please refer to [these instructions](../index.md#mediapipe-on-the-web).
 ## Resources

 *   Google AI Blog:
-    [BlazePose - On-device Real-time Body Pose Tracking](https://mediapipe.page.link/blazepose-blog)
+    [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
 *   Paper:
    [BlazePose: On-device Real-time Body Pose Tracking](https://arxiv.org/abs/2006.10204)
    ([presentation](https://youtu.be/YPpUOTRn5tA))
-*   Pose detection model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection/pose_detection.tflite)
-*   Upper-body pose landmark model:
-    [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_upper_body.tflite)
-*   [Model card](https://mediapipe.page.link/blazepose-mc)
+*   [Models and model cards](./models.md#pose)
@@ -16,18 +16,23 @@ has_toc: false
 <!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
 <!-- Whenever this table is updated, paste a copy to ../external_index.md. -->

-[]()                                                                          | Android | iOS | Desktop | Python | Web | Coral
-:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
-[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)       | ✅       | ✅   | ✅       |        | ✅   | ✅
-[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                 | ✅       | ✅   | ✅       |        |     |
-[Iris](https://google.github.io/mediapipe/solutions/iris) 🆕                         | ✅       | ✅   | ✅       |        | ✅   |
-[Hands](https://google.github.io/mediapipe/solutions/hands)                         | ✅       | ✅   | ✅       |        | ✅   |
-[Pose](https://google.github.io/mediapipe/solutions/pose) 🆕                         | ✅       | ✅   | ✅       | ✅      | ✅   |
-[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅       |     | ✅       |        | ✅   |
-[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)   | ✅       | ✅   | ✅       |        |     | ✅
-[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)           | ✅       | ✅   | ✅       |        |     |
-[Objectron](https://google.github.io/mediapipe/solutions/objectron)                 | ✅       |     |         |        |     |
-[KNIFT](https://google.github.io/mediapipe/solutions/knift)                         | ✅       |     |         |        |     |
-[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                   |         |     | ✅       |        |     |
-[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)        |         |     | ✅       |        |     |
-[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)               |         |     | ✅       |        |     |
+[]()                                                                                      | Android | iOS | Desktop | Python | Web | Coral
+:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
+[Face Detection](https://google.github.io/mediapipe/solutions/face_detection)                   | ✅       | ✅   | ✅       |        | ✅   | ✅
+[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)                             | ✅       | ✅   | ✅       |        |     |
+[Iris](https://google.github.io/mediapipe/solutions/iris)                                       | ✅       | ✅   | ✅       |        | ✅   |
+[Hands](https://google.github.io/mediapipe/solutions/hands)                                     | ✅       | ✅   | ✅       |        | ✅   |
+[Pose](https://google.github.io/mediapipe/solutions/pose)                                       | ✅       | ✅   | ✅       | ✅      | ✅   |
+[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation)             | ✅       |     | ✅       |        | ✅   |
+[Object Detection](https://google.github.io/mediapipe/solutions/object_detection)               | ✅       | ✅   | ✅       |        |     | ✅
+[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking)                       | ✅       | ✅   | ✅       |        |     |
+[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅       |     |         |        |     |
+[Objectron](https://google.github.io/mediapipe/solutions/objectron)                             | ✅       |     |         |        |     |
+[KNIFT](https://google.github.io/mediapipe/solutions/knift)                                     | ✅       |     |         |        |     |
+[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip)                               |         |     | ✅       |        |     |
+[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence)                    |         |     | ✅       |        |     |
+[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m)                           |         |     | ✅       |        |     |
+
+See also
+[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
+for ML models released in MediaPipe.
@@ -2,7 +2,7 @@
 layout: default
 title: YouTube-8M Feature Extraction and Model Inference
 parent: Solutions
-nav_order: 13
+nav_order: 14
 ---

 # YouTube-8M Feature Extraction and Model Inference
@@ -26,9 +26,10 @@ To enable tracing and profiling of a mediapipe graph:
   1. The profiling library must be linked to the framework.
   2. Tracing and profiling must be enabled in the graph configuration.

-The profiling library is linked to the framework by default.  If needed,
-the profiling library can be omitted from the framework using the bazel
-command line option: `--define MEDIAPIPE_PROFILING=0`.
+The profiling library is linked to the framework by default for Desktop.
+If needed, it can be omitted from the framework using the bazel command line
+option: `--define MEDIAPIPE_PROFILING=0`.  For other platforms, you can use the
+bazel command line option `--define MEDIAPIPE_PROFILING=1` to link it.

 To enable tracing and profiling, the `CalculatorGraphConfig` (in
 [calculator.proto](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto))
@@ -38,6 +39,7 @@ is a simple setup that turns on tracing and keeps 100 seconds of timing events:
 ```
 profiler_config {
  trace_enabled: true
+  enable_profiler: true
  trace_log_interval_count: 200
 }
 ```
@@ -71,6 +73,9 @@ MediaPipe will emit data into a pre-specified directory:

    You can open the Download Container. Logs will be located in `application
    container/.xcappdata/AppData/Documents/`
+    If XCode shows empty content for the downloaded container file, you can
+    right click and select 'Show Package Contents' in Finder. Logs
+    will be located in 'AppData/Documents/'

    ![Windows Download Container](../images/visualizer/ios_download_container.png)

@@ -144,10 +149,14 @@ we record ten intervals of half a second each. This can be overridden by adding
    ```bash
    profiler_config {
      trace_enabled: true
-      trace_log_path: "/sdcard/profiles"
+      enable_profiler: true
+      trace_log_path: "/sdcard/profiles/"
    }
    ```

+    Note: The forward slash at the end of the `trace_log_path` is necessary for
+    indicating that `profiles` is a directory (that *should* exist).
+
 *   Download the trace files from the device.

    ```bash
@@ -10,6 +10,7 @@
    "mediapipe/examples/ios/helloworld/BUILD",
    "mediapipe/examples/ios/facedetectioncpu/BUILD",
    "mediapipe/examples/ios/facedetectiongpu/BUILD",
+    "mediapipe/examples/ios/faceeffect/BUILD",
    "mediapipe/examples/ios/facemeshgpu/BUILD",
    "mediapipe/examples/ios/handdetectiongpu/BUILD",
    "mediapipe/examples/ios/handtrackinggpu/BUILD",
@@ -23,6 +24,7 @@
    "//mediapipe/examples/ios/helloworld:HelloWorldApp",
    "//mediapipe/examples/ios/facedetectioncpu:FaceDetectionCpuApp",
    "//mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp",
+    "//mediapipe/examples/ios/faceeffect:FaceEffectApp",
    "//mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp",
    "//mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp",
    "//mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp",
@@ -90,6 +92,8 @@
    "mediapipe/examples/ios/helloworld",
    "mediapipe/examples/ios/facedetectioncpu",
    "mediapipe/examples/ios/facedetectiongpu",
+    "mediapipe/examples/ios/faceeffect",
+    "mediapipe/examples/ios/faceeffect/Base.lproj",
    "mediapipe/examples/ios/handdetectiongpu",
    "mediapipe/examples/ios/handtrackinggpu",
    "mediapipe/examples/ios/iristrackinggpu",
@@ -110,6 +114,7 @@
    "mediapipe/graphs",
    "mediapipe/graphs/edge_detection",
    "mediapipe/graphs/face_detection",
+    "mediapipe/graphs/face_geometry",
    "mediapipe/graphs/hand_tracking",
    "mediapipe/graphs/object_detection",
    "mediapipe/graphs/pose_tracking",
@@ -13,6 +13,7 @@
    "mediapipe/examples/ios",
    "mediapipe/examples/ios/facedetectioncpu",
    "mediapipe/examples/ios/facedetectiongpu",
+    "mediapipe/examples/ios/faceeffect",
    "mediapipe/examples/ios/facemeshgpu",
    "mediapipe/examples/ios/handdetectiongpu",
    "mediapipe/examples/ios/handtrackinggpu",
@@ -12,7 +12,3 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 """
-
-import mediapipe.examples.python as examples
-from mediapipe.python import *
-import mediapipe.util as util
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -13,7 +13,7 @@
 # limitations under the License.
 #

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 filegroup(
    name = "test_audios",
@@ -13,181 +13,120 @@
 # limitations under the License.
 #

-load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
+load("//mediapipe/framework/port:build_config.bzl", "mediapipe_proto_library")

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

-proto_library(
+mediapipe_proto_library(
    name = "concatenate_vector_calculator_proto",
    srcs = ["concatenate_vector_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "dequantize_byte_array_calculator_proto",
    srcs = ["dequantize_byte_array_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "packet_cloner_calculator_proto",
    srcs = ["packet_cloner_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "packet_resampler_calculator_proto",
    srcs = ["packet_resampler_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "packet_thinner_calculator_proto",
    srcs = ["packet_thinner_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "split_vector_calculator_proto",
    srcs = ["split_vector_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "quantize_float_vector_calculator_proto",
    srcs = ["quantize_float_vector_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "sequence_shift_calculator_proto",
    srcs = ["sequence_shift_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "gate_calculator_proto",
    srcs = ["gate_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "constant_side_packet_calculator_proto",
    srcs = ["constant_side_packet_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "clip_vector_size_calculator_proto",
    srcs = ["clip_vector_size_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "packet_cloner_calculator_cc_proto",
-    srcs = ["packet_cloner_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":packet_cloner_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "packet_resampler_calculator_cc_proto",
-    srcs = ["packet_resampler_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":packet_resampler_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "packet_thinner_calculator_cc_proto",
-    srcs = ["packet_thinner_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":packet_thinner_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "split_vector_calculator_cc_proto",
-    srcs = ["split_vector_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":split_vector_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "concatenate_vector_calculator_cc_proto",
-    srcs = ["concatenate_vector_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":concatenate_vector_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "clip_vector_size_calculator_cc_proto",
-    srcs = ["clip_vector_size_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":clip_vector_size_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "dequantize_byte_array_calculator_cc_proto",
-    srcs = ["dequantize_byte_array_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":dequantize_byte_array_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "quantize_float_vector_calculator_cc_proto",
-    srcs = ["quantize_float_vector_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":quantize_float_vector_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "sequence_shift_calculator_cc_proto",
-    srcs = ["sequence_shift_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":sequence_shift_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "gate_calculator_cc_proto",
-    srcs = ["gate_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":gate_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "constant_side_packet_calculator_cc_proto",
-    srcs = ["constant_side_packet_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":constant_side_packet_calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

 cc_library(
@@ -286,11 +225,22 @@ cc_library(
    name = "concatenate_vector_calculator",
    srcs = ["concatenate_vector_calculator.cc"],
    hdrs = ["concatenate_vector_calculator.h"],
+    copts = select({
+        # Needed for "//mediapipe/framework/formats:tensor" compatibility on Apple
+        # platforms for Metal pulled in via the tensor.h header.
+        "//mediapipe:apple": [
+            "-x objective-c++",
+            "-fobjc-arc",  # enable reference-counting
+        ],
+        "//conditions:default": [],
+    }),
    visibility = ["//visibility:public"],
    deps = [
        ":concatenate_vector_calculator_cc_proto",
        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework/formats:classification_cc_proto",
        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/port:integral_types",
        "//mediapipe/framework/port:ret_check",
        "//mediapipe/framework/port:status",
        "@org_tensorflow//tensorflow/lite:framework",
@@ -1119,6 +1069,7 @@ cc_library(
        ":constant_side_packet_calculator_cc_proto",
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework:collection_item_id",
+        "//mediapipe/framework/port:integral_types",
        "//mediapipe/framework/port:ret_check",
        "//mediapipe/framework/port:status",
    ],
@@ -16,7 +16,9 @@

 #include <vector>

+#include "mediapipe/framework/formats/classification.pb.h"
 #include "mediapipe/framework/formats/landmark.pb.h"
+#include "mediapipe/framework/port/integral_types.h"
 #include "tensorflow/lite/interpreter.h"

 #if !defined(MEDIAPIPE_DISABLE_GL_COMPUTE)
@@ -45,6 +47,12 @@ REGISTER_CALCULATOR(ConcatenateFloatVectorCalculator);
 typedef ConcatenateVectorCalculator<int32> ConcatenateInt32VectorCalculator;
 REGISTER_CALCULATOR(ConcatenateInt32VectorCalculator);

+typedef ConcatenateVectorCalculator<uint64> ConcatenateUInt64VectorCalculator;
+REGISTER_CALCULATOR(ConcatenateUInt64VectorCalculator);
+
+typedef ConcatenateVectorCalculator<bool> ConcatenateBoolVectorCalculator;
+REGISTER_CALCULATOR(ConcatenateBoolVectorCalculator);
+
 // Example config:
 // node {
 //   calculator: "ConcatenateTfLiteTensorVectorCalculator"
@@ -60,6 +68,14 @@ typedef ConcatenateVectorCalculator<::mediapipe::NormalizedLandmark>
    ConcatenateLandmarkVectorCalculator;
 REGISTER_CALCULATOR(ConcatenateLandmarkVectorCalculator);

+typedef ConcatenateVectorCalculator<::mediapipe::NormalizedLandmarkList>
+    ConcatenateLandmarListVectorCalculator;
+REGISTER_CALCULATOR(ConcatenateLandmarListVectorCalculator);
+
+typedef ConcatenateVectorCalculator<mediapipe::ClassificationList>
+    ConcatenateClassificationListVectorCalculator;
+REGISTER_CALCULATOR(ConcatenateClassificationListVectorCalculator);
+
 #if !defined(MEDIAPIPE_DISABLE_GL_COMPUTE)
 typedef ConcatenateVectorCalculator<::tflite::gpu::gl::GlBuffer>
    ConcatenateGlBufferVectorCalculator;
@@ -15,6 +15,7 @@
 #ifndef MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_VECTOR_CALCULATOR_H_
 #define MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_VECTOR_CALCULATOR_H_

+#include <string>
 #include <type_traits>
 #include <vector>

@@ -26,10 +27,10 @@

 namespace mediapipe {

-// Concatenates several std::vector<T> following stream index order. This class
-// assumes that every input stream contains the vector<T> type. To use this
-// class for a particular type T, regisiter a calculator using
-// ConcatenateVectorCalculator<T>.
+// Concatenates several objects of type T or std::vector<T> following stream
+// index order. This class assumes that every input stream contains either T or
+// vector<T> type. To use this class for a particular type T, regisiter a
+// calculator using ConcatenateVectorCalculator<T>.
 template <typename T>
 class ConcatenateVectorCalculator : public CalculatorBase {
 public:
@@ -38,7 +39,8 @@ class ConcatenateVectorCalculator : public CalculatorBase {
    RET_CHECK(cc->Outputs().NumEntries() == 1);

    for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
-      cc->Inputs().Index(i).Set<std::vector<T>>();
+      // Actual type T or vector<T> will be validated in Process().
+      cc->Inputs().Index(i).SetAny();
    }

    cc->Outputs().Index(0).Set<std::vector<T>>();
@@ -69,9 +71,19 @@ class ConcatenateVectorCalculator : public CalculatorBase {
                                         CalculatorContext* cc) {
    auto output = absl::make_unique<std::vector<U>>();
    for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
-      if (cc->Inputs().Index(i).IsEmpty()) continue;
-      const std::vector<U>& input = cc->Inputs().Index(i).Get<std::vector<U>>();
-      output->insert(output->end(), input.begin(), input.end());
+      auto& input = cc->Inputs().Index(i);
+
+      if (input.IsEmpty()) continue;
+
+      if (input.Value().ValidateAsType<U>().ok()) {
+        const U& value = input.Get<U>();
+        output->push_back(value);
+      } else if (input.Value().ValidateAsType<std::vector<U>>().ok()) {
+        const std::vector<U>& value = input.Get<std::vector<U>>();
+        output->insert(output->end(), value.begin(), value.end());
+      } else {
+        return ::mediapipe::InvalidArgumentError("Invalid input stream type.");
+      }
    }
    cc->Outputs().Index(0).Add(output.release(), cc->InputTimestamp());
    return ::mediapipe::OkStatus();
@@ -88,17 +100,32 @@ class ConcatenateVectorCalculator : public CalculatorBase {
                                                   CalculatorContext* cc) {
    auto output = absl::make_unique<std::vector<U>>();
    for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
-      if (cc->Inputs().Index(i).IsEmpty()) continue;
-      ::mediapipe::StatusOr<std::unique_ptr<std::vector<U>>> input_status =
-          cc->Inputs().Index(i).Value().Consume<std::vector<U>>();
-      if (input_status.ok()) {
-        std::unique_ptr<std::vector<U>> input_vector =
-            std::move(input_status).ValueOrDie();
-        output->insert(output->end(),
-                       std::make_move_iterator(input_vector->begin()),
-                       std::make_move_iterator(input_vector->end()));
+      auto& input = cc->Inputs().Index(i);
+
+      if (input.IsEmpty()) continue;
+
+      if (input.Value().ValidateAsType<U>().ok()) {
+        ::mediapipe::StatusOr<std::unique_ptr<U>> value_status =
+            input.Value().Consume<U>();
+        if (value_status.ok()) {
+          std::unique_ptr<U> value = std::move(value_status).ValueOrDie();
+          output->push_back(std::move(*value));
+        } else {
+          return value_status.status();
+        }
+      } else if (input.Value().ValidateAsType<std::vector<U>>().ok()) {
+        ::mediapipe::StatusOr<std::unique_ptr<std::vector<U>>> value_status =
+            input.Value().Consume<std::vector<U>>();
+        if (value_status.ok()) {
+          std::unique_ptr<std::vector<U>> value =
+              std::move(value_status).ValueOrDie();
+          output->insert(output->end(), std::make_move_iterator(value->begin()),
+                         std::make_move_iterator(value->end()));
+        } else {
+          return value_status.status();
+        }
      } else {
-        return input_status.status();
+        return ::mediapipe::InvalidArgumentError("Invalid input stream type.");
      }
    }
    cc->Outputs().Index(0).Add(output.release(), cc->InputTimestamp());
@@ -109,7 +136,7 @@ class ConcatenateVectorCalculator : public CalculatorBase {
  ::mediapipe::Status ConsumeAndConcatenateVectors(std::false_type,
                                                   CalculatorContext* cc) {
    return ::mediapipe::InternalError(
-        "Cannot copy or move input vectors to concatenate them");
+        "Cannot copy or move inputs to concatenate them");
  }

 private:
@@ -30,11 +30,29 @@ namespace mediapipe {
 typedef ConcatenateVectorCalculator<int> TestConcatenateIntVectorCalculator;
 REGISTER_CALCULATOR(TestConcatenateIntVectorCalculator);

+void AddInputVector(int index, const std::vector<int>& input, int64 timestamp,
+                    CalculatorRunner* runner) {
+  runner->MutableInputs()->Index(index).packets.push_back(
+      MakePacket<std::vector<int>>(input).At(Timestamp(timestamp)));
+}
+
 void AddInputVectors(const std::vector<std::vector<int>>& inputs,
                     int64 timestamp, CalculatorRunner* runner) {
  for (int i = 0; i < inputs.size(); ++i) {
-    runner->MutableInputs()->Index(i).packets.push_back(
-        MakePacket<std::vector<int>>(inputs[i]).At(Timestamp(timestamp)));
+    AddInputVector(i, inputs[i], timestamp, runner);
+  }
+}
+
+void AddInputItem(int index, int input, int64 timestamp,
+                  CalculatorRunner* runner) {
+  runner->MutableInputs()->Index(index).packets.push_back(
+      MakePacket<int>(input).At(Timestamp(timestamp)));
+}
+
+void AddInputItems(const std::vector<int>& inputs, int64 timestamp,
+                   CalculatorRunner* runner) {
+  for (int i = 0; i < inputs.size(); ++i) {
+    AddInputItem(i, inputs[i], timestamp, runner);
  }
 }

@@ -131,6 +149,135 @@ TEST(TestConcatenateIntVectorCalculatorTest, OneEmptyStreamNoOutput) {
  EXPECT_EQ(0, outputs.size());
 }

+TEST(TestConcatenateIntVectorCalculatorTest, ItemsOneTimestamp) {
+  CalculatorRunner runner("TestConcatenateIntVectorCalculator",
+                          /*options_string=*/"", /*num_inputs=*/3,
+                          /*num_outputs=*/1, /*num_side_packets=*/0);
+
+  std::vector<int> inputs = {1, 2, 3};
+  AddInputItems(inputs, /*timestamp=*/1, &runner);
+  MP_ASSERT_OK(runner.Run());
+
+  const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
+  EXPECT_EQ(1, outputs.size());
+  EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
+  std::vector<int> expected_vector = {1, 2, 3};
+  EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
+}
+
+TEST(TestConcatenateIntVectorCalculatorTest, ItemsTwoInputsAtTwoTimestamps) {
+  CalculatorRunner runner("TestConcatenateIntVectorCalculator",
+                          /*options_string=*/"", /*num_inputs=*/3,
+                          /*num_outputs=*/1, /*num_side_packets=*/0);
+
+  {
+    std::vector<int> inputs = {1, 2, 3};
+    AddInputItems(inputs, /*timestamp=*/1, &runner);
+  }
+  {
+    std::vector<int> inputs = {4, 5, 6};
+    AddInputItems(inputs, /*timestamp=*/2, &runner);
+  }
+  MP_ASSERT_OK(runner.Run());
+
+  const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
+  EXPECT_EQ(2, outputs.size());
+  {
+    EXPECT_EQ(3, outputs[0].Get<std::vector<int>>().size());
+    EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
+    std::vector<int> expected_vector = {1, 2, 3};
+    EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
+  }
+  {
+    EXPECT_EQ(3, outputs[1].Get<std::vector<int>>().size());
+    EXPECT_EQ(Timestamp(2), outputs[1].Timestamp());
+    std::vector<int> expected_vector = {4, 5, 6};
+    EXPECT_EQ(expected_vector, outputs[1].Get<std::vector<int>>());
+  }
+}
+
+TEST(TestConcatenateIntVectorCalculatorTest, ItemsOneEmptyStreamStillOutput) {
+  CalculatorRunner runner("TestConcatenateIntVectorCalculator",
+                          /*options_string=*/"", /*num_inputs=*/3,
+                          /*num_outputs=*/1, /*num_side_packets=*/0);
+
+  // No third input item.
+  std::vector<int> inputs = {1, 2};
+  AddInputItems(inputs, /*timestamp=*/1, &runner);
+  MP_ASSERT_OK(runner.Run());
+
+  const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
+  EXPECT_EQ(1, outputs.size());
+  EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
+  std::vector<int> expected_vector = {1, 2};
+  EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
+}
+
+TEST(TestConcatenateIntVectorCalculatorTest, ItemsOneEmptyStreamNoOutput) {
+  CalculatorRunner runner("TestConcatenateIntVectorCalculator",
+                          /*options_string=*/
+                          "[mediapipe.ConcatenateVectorCalculatorOptions.ext]: "
+                          "{only_emit_if_all_present: true}",
+                          /*num_inputs=*/3,
+                          /*num_outputs=*/1, /*num_side_packets=*/0);
+
+  // No third input item.
+  std::vector<int> inputs = {1, 2};
+  AddInputItems(inputs, /*timestamp=*/1, &runner);
+  MP_ASSERT_OK(runner.Run());
+
+  const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
+  EXPECT_EQ(0, outputs.size());
+}
+
+TEST(TestConcatenateIntVectorCalculatorTest, MixedVectorsAndItems) {
+  CalculatorRunner runner("TestConcatenateIntVectorCalculator",
+                          /*options_string=*/"", /*num_inputs=*/4,
+                          /*num_outputs=*/1, /*num_side_packets=*/0);
+
+  std::vector<int> vector_0 = {1, 2};
+  std::vector<int> vector_1 = {3, 4, 5};
+  int item_0 = 6;
+  int item_1 = 7;
+
+  AddInputVector(/*index*/ 0, vector_0, /*timestamp=*/1, &runner);
+  AddInputVector(/*index*/ 1, vector_1, /*timestamp=*/1, &runner);
+  AddInputItem(/*index*/ 2, item_0, /*timestamp=*/1, &runner);
+  AddInputItem(/*index*/ 3, item_1, /*timestamp=*/1, &runner);
+
+  MP_ASSERT_OK(runner.Run());
+
+  const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
+  EXPECT_EQ(1, outputs.size());
+  EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
+  std::vector<int> expected_vector = {1, 2, 3, 4, 5, 6, 7};
+  EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
+}
+
+TEST(TestConcatenateIntVectorCalculatorTest, MixedVectorsAndItemsAnother) {
+  CalculatorRunner runner("TestConcatenateIntVectorCalculator",
+                          /*options_string=*/"", /*num_inputs=*/4,
+                          /*num_outputs=*/1, /*num_side_packets=*/0);
+
+  int item_0 = 1;
+  std::vector<int> vector_0 = {2, 3};
+  std::vector<int> vector_1 = {4, 5, 6};
+  int item_1 = 7;
+
+  AddInputItem(/*index*/ 0, item_0, /*timestamp=*/1, &runner);
+  AddInputVector(/*index*/ 1, vector_0, /*timestamp=*/1, &runner);
+  AddInputVector(/*index*/ 2, vector_1, /*timestamp=*/1, &runner);
+  AddInputItem(/*index*/ 3, item_1, /*timestamp=*/1, &runner);
+
+  MP_ASSERT_OK(runner.Run());
+
+  const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
+  EXPECT_EQ(1, outputs.size());
+  EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
+  std::vector<int> expected_vector = {1, 2, 3, 4, 5, 6, 7};
+  EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
+}
+
 void AddInputVectors(const std::vector<std::vector<float>>& inputs,
                     int64 timestamp, CalculatorRunner* runner) {
  for (int i = 0; i < inputs.size(); ++i) {
@@ -18,6 +18,7 @@
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/collection_item_id.h"
 #include "mediapipe/framework/port/canonical_errors.h"
+#include "mediapipe/framework/port/integral_types.h"
 #include "mediapipe/framework/port/ret_check.h"
 #include "mediapipe/framework/port/status.h"

@@ -71,6 +72,8 @@ class ConstantSidePacketCalculator : public CalculatorBase {
        packet.Set<bool>();
      } else if (packet_options.has_string_value()) {
        packet.Set<std::string>();
+      } else if (packet_options.has_uint64_value()) {
+        packet.Set<uint64>();
      } else {
        return ::mediapipe::InvalidArgumentError(
            "None of supported values were specified in options.");
@@ -95,6 +98,8 @@ class ConstantSidePacketCalculator : public CalculatorBase {
        packet.Set(MakePacket<bool>(packet_options.bool_value()));
      } else if (packet_options.has_string_value()) {
        packet.Set(MakePacket<std::string>(packet_options.string_value()));
+      } else if (packet_options.has_uint64_value()) {
+        packet.Set(MakePacket<uint64>(packet_options.uint64_value()));
      } else {
        return ::mediapipe::InvalidArgumentError(
            "None of supported values were specified in options.");
@@ -29,6 +29,7 @@ message ConstantSidePacketCalculatorOptions {
      float float_value = 2;
      bool bool_value = 3;
      string string_value = 4;
+      uint64 uint64_value = 5;
    }
  }

@@ -1,4 +1,4 @@
-// Copyright 2019 The MediaPipe Authors.
+// Copyright 2019-2020 The MediaPipe Authors.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -134,7 +134,13 @@ class GateCalculator : public CalculatorBase {
  }

  ::mediapipe::Status Open(CalculatorContext* cc) final {
-    use_side_packet_for_allow_disallow_ = false;
+    const auto& options = cc->Options<::mediapipe::GateCalculatorOptions>();
+    use_calculator_option_for_allow_disallow_ =
+        options.has_allowance_override();
+    if (use_calculator_option_for_allow_disallow_) {
+      allow_by_calculator_option_ = options.allowance_override();
+    }
+
    if (cc->InputSidePackets().HasTag("ALLOW")) {
      use_side_packet_for_allow_disallow_ = true;
      allow_by_side_packet_decision_ =
@@ -150,24 +156,27 @@ class GateCalculator : public CalculatorBase {
    last_gate_state_ = GATE_UNINITIALIZED;
    RET_CHECK_OK(CopyInputHeadersToOutputs(cc->Inputs(), &cc->Outputs()));

-    const auto& options = cc->Options<::mediapipe::GateCalculatorOptions>();
    empty_packets_as_allow_ = options.empty_packets_as_allow();
-
    return ::mediapipe::OkStatus();
  }

  ::mediapipe::Status Process(CalculatorContext* cc) final {
-    bool allow = empty_packets_as_allow_;
-    if (use_side_packet_for_allow_disallow_) {
-      allow = allow_by_side_packet_decision_;
-    } else {
-      if (cc->Inputs().HasTag("ALLOW") &&
-          !cc->Inputs().Tag("ALLOW").IsEmpty()) {
-        allow = cc->Inputs().Tag("ALLOW").Get<bool>();
-      }
-      if (cc->Inputs().HasTag("DISALLOW") &&
-          !cc->Inputs().Tag("DISALLOW").IsEmpty()) {
-        allow = !cc->Inputs().Tag("DISALLOW").Get<bool>();
+    // The allow/disallow signal in the calculator option has the highest
+    // priority. If it's not set, use the stream/side packet signal.
+    bool allow = allow_by_calculator_option_;
+    if (!use_calculator_option_for_allow_disallow_) {
+      allow = empty_packets_as_allow_;
+      if (use_side_packet_for_allow_disallow_) {
+        allow = allow_by_side_packet_decision_;
+      } else {
+        if (cc->Inputs().HasTag("ALLOW") &&
+            !cc->Inputs().Tag("ALLOW").IsEmpty()) {
+          allow = cc->Inputs().Tag("ALLOW").Get<bool>();
+        }
+        if (cc->Inputs().HasTag("DISALLOW") &&
+            !cc->Inputs().Tag("DISALLOW").IsEmpty()) {
+          allow = !cc->Inputs().Tag("DISALLOW").Get<bool>();
+        }
      }
    }
    const GateState new_gate_state = allow ? GATE_ALLOW : GATE_DISALLOW;
@@ -203,9 +212,11 @@ class GateCalculator : public CalculatorBase {
 private:
  GateState last_gate_state_ = GATE_UNINITIALIZED;
  int num_data_streams_;
-  bool empty_packets_as_allow_;
-  bool use_side_packet_for_allow_disallow_;
-  bool allow_by_side_packet_decision_;
+  bool empty_packets_as_allow_ = false;
+  bool use_side_packet_for_allow_disallow_ = false;
+  bool allow_by_side_packet_decision_ = false;
+  bool use_calculator_option_for_allow_disallow_ = false;
+  bool allow_by_calculator_option_ = false;
 };
 REGISTER_CALCULATOR(GateCalculator);

@@ -1,4 +1,4 @@
-// Copyright 2019 The MediaPipe Authors.
+// Copyright 2019-2020 The MediaPipe Authors.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -27,4 +27,9 @@ message GateCalculatorOptions {
  // disallowing the corresponding packets in the data input streams. Setting
  // this option to true inverts that, allowing the data packets to go through.
  optional bool empty_packets_as_allow = 1;
+
+  // If set, the calculator will always allow (if set to yes) or disallow (if
+  // set to no) the input streams to pass through, and ignore the ALLOW or
+  // DISALLOW input stream or side input packets.
+  optional bool allowance_override = 2;
 }
@@ -1,4 +1,4 @@
-// Copyright 2019 The MediaPipe Authors.
+// Copyright 2019-2020 The MediaPipe Authors.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
@@ -330,5 +330,52 @@ TEST_F(GateCalculatorTest, AllowInitialNoStateTransition) {
  ASSERT_EQ(0, output.size());
 }

+TEST_F(GateCalculatorTest,
+       TestCalculatorOptionDecisionOverrideOverStreamSingal) {
+  SetRunner(R"(
+        calculator: "GateCalculator"
+        input_stream: "test_input"
+        input_stream: "ALLOW:gating_stream"
+        output_stream: "test_output"
+        options: {
+          [mediapipe.GateCalculatorOptions.ext] {
+            allowance_override: false
+          }
+        }
+  )");
+
+  constexpr int64 kTimestampValue0 = 42;
+  // The CalculatorOptions says disallow and the stream says allow. Should
+  // follow the CalculatorOptions' decision to disallow outputting anything.
+  RunTimeStep(kTimestampValue0, "ALLOW", true);
+
+  const std::vector<Packet>& output = runner()->Outputs().Get("", 0).packets;
+  ASSERT_EQ(0, output.size());
+}
+
+TEST_F(GateCalculatorTest,
+       TestCalculatorOptionDecisionOverrideOverSidePacketSingal) {
+  SetRunner(R"(
+        calculator: "GateCalculator"
+        input_stream: "test_input"
+        input_side_packet: "ALLOW:gating_packet"
+        output_stream: "test_output"
+        options: {
+          [mediapipe.GateCalculatorOptions.ext] {
+            allowance_override: true
+          }
+        }
+  )");
+
+  constexpr int64 kTimestampValue0 = 42;
+  // The CalculatorOptions says allow and the side packet says disallow. Should
+  // follow the CalculatorOptions' decision to allow outputting a packet.
+  runner()->MutableSidePackets()->Tag("ALLOW") = Adopt(new bool(false));
+  RunTimeStep(kTimestampValue0, true);
+
+  const std::vector<Packet>& output = runner()->Outputs().Get("", 0).packets;
+  ASSERT_EQ(1, output.size());
+}
+
 }  // namespace
 }  // namespace mediapipe
@@ -12,6 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+#include <memory>
+
 #include "mediapipe/calculators/core/split_vector_calculator.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/calculator_runner.h"
@@ -233,5 +235,71 @@ TEST(MuxCalculatorTest, InputStreamSelector_MuxInputStreamHandler) {
           kOutputName, output_fn);
  EXPECT_EQ(output, input_packets);
 }
+
+constexpr char kDualInputGraphConfig[] = R"proto(
+  input_stream: "input_0"
+  input_stream: "input_1"
+  input_stream: "input_select"
+  output_stream: "test_output"
+  node {
+    calculator: "MuxCalculator"
+    input_stream: "INPUT:0:input_0"
+    input_stream: "INPUT:1:input_1"
+    input_stream: "SELECT:input_select"
+    output_stream: "OUTPUT:test_output"
+  }
+)proto";
+
+TEST(MuxCalculatorTest, DiscardSkippedInputs_MuxInputStreamHandler) {
+  CalculatorGraphConfig config =
+      ::mediapipe::ParseTextProtoOrDie<CalculatorGraphConfig>(
+          kDualInputGraphConfig);
+  CalculatorGraph graph;
+  MP_ASSERT_OK(graph.Initialize(config));
+
+  std::shared_ptr<int> output;
+  MP_ASSERT_OK(
+      graph.ObserveOutputStream("test_output", [&output](const Packet& p) {
+        output = p.Get<std::shared_ptr<int>>();
+        return ::mediapipe::OkStatus();
+      }));
+
+  MP_ASSERT_OK(graph.StartRun({}));
+
+  auto one = std::make_shared<int>(1);
+  auto two = std::make_shared<int>(2);
+  auto three = std::make_shared<int>(3);
+  std::weak_ptr<int> one_weak = one;
+  std::weak_ptr<int> two_weak = two;
+
+  MP_ASSERT_OK(graph.AddPacketToInputStream(
+      "input_0",
+      MakePacket<std::shared_ptr<int>>(std::move(one)).At(Timestamp(0))));
+  MP_ASSERT_OK(graph.AddPacketToInputStream(
+      "input_1",
+      MakePacket<std::shared_ptr<int>>(std::move(two)).At(Timestamp(0))));
+  MP_ASSERT_OK(graph.AddPacketToInputStream(
+      "input_1",
+      MakePacket<std::shared_ptr<int>>(std::move(three)).At(Timestamp(1))));
+  EXPECT_EQ(one, nullptr);
+  EXPECT_EQ(two, nullptr);
+  EXPECT_EQ(three, nullptr);
+
+  MP_ASSERT_OK(graph.AddPacketToInputStream(
+      "input_select", MakePacket<int>(0).At(Timestamp(0))));
+  MP_ASSERT_OK(graph.WaitUntilIdle());
+  EXPECT_EQ(*output, 1);
+  EXPECT_NE(one_weak.lock(), nullptr);
+  EXPECT_EQ(two_weak.lock(), nullptr);
+
+  MP_ASSERT_OK(graph.AddPacketToInputStream(
+      "input_select", MakePacket<int>(1).At(Timestamp(1))));
+  MP_ASSERT_OK(graph.WaitUntilIdle());
+  EXPECT_EQ(*output, 3);
+
+  MP_ASSERT_OK(graph.CloseAllInputStreams());
+  MP_ASSERT_OK(graph.WaitUntilDone());
+}
+
 }  // namespace
 }  // namespace mediapipe
@@ -12,148 +12,78 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
+load("//mediapipe/framework/port:build_config.bzl", "mediapipe_proto_library")

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

-exports_files(["LICENSE"])
-
-proto_library(
+mediapipe_proto_library(
    name = "opencv_image_encoder_calculator_proto",
    srcs = ["opencv_image_encoder_calculator.proto"],
-    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    visibility = [
+        "//visibility:public",
+    ],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "scale_image_calculator_proto",
    srcs = ["scale_image_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/framework/formats:image_format_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "set_alpha_calculator_proto",
    srcs = ["set_alpha_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "image_cropping_calculator_proto",
    srcs = ["image_cropping_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "bilateral_filter_calculator_proto",
    srcs = ["bilateral_filter_calculator.proto"],
    visibility = [
        "//visibility:public",
    ],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "recolor_calculator_proto",
    srcs = ["recolor_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
    ],
 )

-mediapipe_cc_proto_library(
-    name = "opencv_image_encoder_calculator_cc_proto",
-    srcs = ["opencv_image_encoder_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = [
-        "//visibility:public",
-    ],
-    deps = [":opencv_image_encoder_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "opencv_encoded_image_to_image_frame_calculator_cc_proto",
-    srcs = ["opencv_encoded_image_to_image_frame_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":opencv_encoded_image_to_image_frame_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "mask_overlay_calculator_cc_proto",
-    srcs = ["mask_overlay_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":mask_overlay_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "scale_image_calculator_cc_proto",
-    srcs = ["scale_image_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/framework/formats:image_format_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":scale_image_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "set_alpha_calculator_cc_proto",
-    srcs = ["set_alpha_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":set_alpha_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "image_cropping_calculator_cc_proto",
-    srcs = ["image_cropping_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":image_cropping_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "bilateral_filter_calculator_cc_proto",
-    srcs = ["bilateral_filter_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = [
-        "//visibility:public",
-    ],
-    deps = [":bilateral_filter_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "recolor_calculator_cc_proto",
-    srcs = ["recolor_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":recolor_calculator_proto"],
-)
-
 cc_library(
    name = "color_convert_calculator",
    srcs = ["color_convert_calculator.cc"],
@@ -279,27 +209,17 @@ cc_library(
    alwayslink = 1,
 )

-proto_library(
+mediapipe_proto_library(
    name = "image_transformation_calculator_proto",
    srcs = ["image_transformation_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/gpu:scale_mode_proto",
    ],
 )

-mediapipe_cc_proto_library(
-    name = "image_transformation_calculator_cc_proto",
-    srcs = ["image_transformation_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/gpu:scale_mode_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":image_transformation_calculator_proto"],
-)
-
 cc_library(
    name = "image_transformation_calculator",
    srcs = ["image_transformation_calculator.cc"],
@@ -560,32 +480,33 @@ cc_test(
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "mask_overlay_calculator_proto",
    srcs = ["mask_overlay_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "opencv_encoded_image_to_image_frame_calculator_proto",
    srcs = ["opencv_encoded_image_to_image_frame_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "feature_detector_calculator_proto",
    srcs = ["feature_detector_calculator.proto"],
-    deps = ["//mediapipe/framework:calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "feature_detector_calculator_cc_proto",
-    srcs = ["feature_detector_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":feature_detector_calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

 cc_library(
@@ -13,7 +13,7 @@
 # limitations under the License.
 #

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 filegroup(
    name = "test_images",
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")

@@ -15,7 +15,7 @@

 load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -311,7 +311,6 @@ cc_library(
            "@org_tensorflow//tensorflow/core:framework",
        ],
        "//mediapipe:android": [
-            "@org_tensorflow//tensorflow/core:portable_tensorflow_lib_lite",
        ],
    }),
    alwayslink = 1,
@@ -427,6 +426,10 @@ cc_library(
    deps = [
        ":tensorflow_session",
        ":tensorflow_inference_calculator_cc_proto",
+        "//mediapipe/framework:timestamp",
+        "@com_google_absl//absl/base:core_headers",
+        "@com_google_absl//absl/memory",
+        "//mediapipe/framework:calculator_context",
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/tool:status_util",
        "@com_google_absl//absl/strings",
@@ -434,6 +437,8 @@ cc_library(
        "//mediapipe/framework/deps:clock",
        "//mediapipe/framework/port:status",
        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/framework/port:map_util",
+        "//mediapipe/framework:packet",
    ] + select({
        "//conditions:default": [
            "@org_tensorflow//tensorflow/core:framework",
@@ -42,16 +42,16 @@ namespace tf = tensorflow;
 // a flag controls whether a new first dimension is inserted before
 // concatenation.
 //
-// Currently, the number of tensors output will be buffer_size less than the
-// number of input tensors because no padding is implemented and only full
-// buffers are output.
+// The number of tensors output will be buffer_size less than the
+// number of input tensors unless padding is set to a non-zero value in the
+// options proto.
 //
 // The timestamp of the output batch will match the timestamp of the first
 // tensor in that batch by default. (e.g. when buffer_size frames are added, the
 // output tensor will have the timestamp of the first input.). This behavior can
 // be adjusted by the timestamp_offset option.
 //
-// Example config:
+// Example config without padding:
 // node {
 //   calculator: "LappedTensorBufferCalculator"
 //   input_stream: "input_tensor"
@@ -64,26 +64,50 @@ namespace tf = tensorflow;
 //     }
 //   }
 // }
+//
+// Example config with padding and timestamp output:
+// node {
+//   calculator: "LappedTensorBufferCalculator"
+//   input_stream: "input_tensor"
+//   output_stream: "output_tensor"
+//   output_stream: "output_timestamp"
+//   options {
+//     [mediapipe.LappedTensorBufferCalculatorOptions.ext] {
+//       buffer_size: 100
+//       overlap: 50
+//       add_batch_dim_to_tensors: true
+//       timestamp_offset: 25
+//       padding: 25
+//     }
+//   }
+// }
+
 class LappedTensorBufferCalculator : public CalculatorBase {
 public:
  static ::mediapipe::Status GetContract(CalculatorContract* cc);

  ::mediapipe::Status Open(CalculatorContext* cc) override;
  ::mediapipe::Status Process(CalculatorContext* cc) override;
+  ::mediapipe::Status Close(CalculatorContext* cc) override;

 private:
-  // Adds a batch dimension to the input tensor if specified in the calculator
-  // options.
+  // Adds a batch dimension to the input tensor if specified in the
+  // calculator options.
  ::mediapipe::Status AddBatchDimension(tf::Tensor* input_tensor);
+  // Sends the current buffer downstream.
+  ::mediapipe::Status ProcessBuffer(CalculatorContext* cc);

  int steps_until_output_;
  int buffer_size_;
  int overlap_;
  int timestamp_offset_;
+  int initialized_;
+
  std::unique_ptr<CircularBuffer<Timestamp>> timestamp_buffer_;
  std::unique_ptr<CircularBuffer<tf::Tensor>> buffer_;
  LappedTensorBufferCalculatorOptions options_;
 };
+
 REGISTER_CALCULATOR(LappedTensorBufferCalculator);

 ::mediapipe::Status LappedTensorBufferCalculator::GetContract(
@@ -93,8 +117,8 @@ REGISTER_CALCULATOR(LappedTensorBufferCalculator);
  cc->Inputs().Index(0).Set<tf::Tensor>(
      // tensorflow::Tensor stream.
  );
-  RET_CHECK_EQ(cc->Inputs().NumEntries(), 1)
-      << "Only one output stream is supported.";
+  RET_CHECK_LE(cc->Outputs().NumEntries(), 2)
+      << "Only one or two output stream(s) is/are supported.";

  if (cc->InputSidePackets().HasTag(kBufferSize)) {
    cc->InputSidePackets().Tag(kBufferSize).Set<int>();
@@ -108,11 +132,15 @@ REGISTER_CALCULATOR(LappedTensorBufferCalculator);
  if (cc->InputSidePackets().HasTag(kCalculatorOptions)) {
    cc->InputSidePackets()
        .Tag(kCalculatorOptions)
-        .Set<LappedTensorBufferCalculatorOptions>();
+        .Set<LappedTensorBufferCalculator>();
  }
  cc->Outputs().Index(0).Set<tf::Tensor>(
      // Output tensorflow::Tensor stream with possibly overlapping steps.
  );
+  // Output timestamp stream with possibly overlapping steps.
+  if (cc->Outputs().NumEntries() > 1) {
+    cc->Outputs().Index(1).Set<std::vector<Timestamp>>();
+  }
  return ::mediapipe::OkStatus();
 }

@@ -141,10 +169,13 @@ REGISTER_CALCULATOR(LappedTensorBufferCalculator);
      << "Negative timestamp_offset is not allowed.";
  RET_CHECK_LT(timestamp_offset_, buffer_size_)
      << "output_frame_num_offset has to be less than buffer_size.";
+  RET_CHECK_LT(options_.padding(), buffer_size_)
+      << "padding option must be smaller than buffer size.";
  timestamp_buffer_ =
      absl::make_unique<CircularBuffer<Timestamp>>(buffer_size_);
  buffer_ = absl::make_unique<CircularBuffer<tf::Tensor>>(buffer_size_);
-  steps_until_output_ = buffer_size_;
+  steps_until_output_ = buffer_size_ - options_.padding();
+  initialized_ = false;
  return ::mediapipe::OkStatus();
 }

@@ -156,23 +187,36 @@ REGISTER_CALCULATOR(LappedTensorBufferCalculator);
  if (options_.add_batch_dim_to_tensors()) {
    RET_CHECK_OK(AddBatchDimension(&input_tensor));
  }
+  // Pad frames at the beginning with the first frame.
+  if (!initialized_) {
+    for (int i = 0; i < options_.padding(); ++i) {
+      buffer_->push_back(input_tensor);
+      timestamp_buffer_->push_back(cc->InputTimestamp());
+    }
+    initialized_ = true;
+  }
  buffer_->push_back(input_tensor);
  timestamp_buffer_->push_back(cc->InputTimestamp());
  --steps_until_output_;
-
  if (steps_until_output_ <= 0) {
-    auto concatenated = ::absl::make_unique<tf::Tensor>();
-
-    const tf::Status concat_status = tf::tensor::Concat(
-        std::vector<tf::Tensor>(buffer_->begin(), buffer_->end()),
-        concatenated.get());
-    RET_CHECK(concat_status.ok()) << concat_status.ToString();
-
-    cc->Outputs().Index(0).Add(concatenated.release(),
-                               timestamp_buffer_->Get(timestamp_offset_));
-
-    steps_until_output_ = buffer_size_ - overlap_;
+    MP_RETURN_IF_ERROR(ProcessBuffer(cc));
  }
+
+  return ::mediapipe::OkStatus();
+}
+
+::mediapipe::Status LappedTensorBufferCalculator::Close(CalculatorContext* cc) {
+  if (!initialized_ || options_.padding() == 0) {
+    return ::mediapipe::OkStatus();
+  }
+  int last_frame = buffer_size_ - steps_until_output_ - 1;
+  const auto& pad_frame = buffer_->Get(last_frame);
+  for (int i = 0; i < steps_until_output_ + options_.padding(); ++i) {
+    buffer_->push_back(pad_frame);
+    timestamp_buffer_->push_back(cc->InputTimestamp());
+  }
+  MP_RETURN_IF_ERROR(ProcessBuffer(cc));
+
  return ::mediapipe::OkStatus();
 }

@@ -190,4 +234,29 @@ REGISTER_CALCULATOR(LappedTensorBufferCalculator);
  return ::mediapipe::OkStatus();
 }

+// Process buffer
+::mediapipe::Status LappedTensorBufferCalculator::ProcessBuffer(
+    CalculatorContext* cc) {
+  auto concatenated = ::absl::make_unique<tf::Tensor>();
+  const tf::Status concat_status = tf::tensor::Concat(
+      std::vector<tf::Tensor>(buffer_->begin(), buffer_->end()),
+      concatenated.get());
+  RET_CHECK(concat_status.ok()) << concat_status.ToString();
+  // Output cancatenated tensor.
+  cc->Outputs().Index(0).Add(concatenated.release(),
+                             timestamp_buffer_->Get(timestamp_offset_));
+  if (cc->Outputs().NumEntries() > 1) {
+    auto output_timestamp = ::absl::make_unique<std::vector<Timestamp>>();
+    // Output timestamp vector.
+    *output_timestamp = std::vector<Timestamp>(timestamp_buffer_->begin(),
+                                               timestamp_buffer_->end());
+    RET_CHECK_EQ(output_timestamp->size(), buffer_size_)
+        << "Output timestamp size is not correct.";
+    cc->Outputs().Index(1).Add(output_timestamp.release(),
+                               timestamp_buffer_->Get(timestamp_offset_));
+  }
+  steps_until_output_ = buffer_size_ - overlap_;
+  return ::mediapipe::OkStatus();
+}
+
 }  // namespace mediapipe
@@ -45,4 +45,8 @@ message LappedTensorBufferCalculatorOptions {
  // This is useful for aligning the timestamp to be centered on the input
  // range.
  optional int32 timestamp_offset = 4 [default = 0];
+
+  // Amount of padding (repeating of first/last value) to add to the beginning
+  // and end of the input stream.
+  optional int32 padding = 5;
 }
@@ -31,11 +31,15 @@ namespace tf = ::tensorflow;
 class LappedTensorBufferCalculatorTest : public ::testing::Test {
 protected:
  void SetUpCalculator(int buffer_size, int overlap, bool add_dim,
-                       int timestamp_offset) {
+                       int timestamp_offset, int padding,
+                       bool timestamp_output) {
    CalculatorGraphConfig::Node config;
    config.set_calculator("LappedTensorBufferCalculator");
    config.add_input_stream("input_tensor");
    config.add_output_stream("output_tensor");
+    if (timestamp_output) {
+      config.add_output_stream("output_timestamp");
+    }
    auto options = config.mutable_options()->MutableExtension(
        LappedTensorBufferCalculatorOptions::ext);
    options->set_buffer_size(buffer_size);
@@ -44,13 +48,14 @@ class LappedTensorBufferCalculatorTest : public ::testing::Test {
      options->set_add_batch_dim_to_tensors(true);
    }
    options->set_timestamp_offset(timestamp_offset);
+    options->set_padding(padding);
    runner_ = ::absl::make_unique<CalculatorRunner>(config);
  }
  std::unique_ptr<CalculatorRunner> runner_;
 };

 TEST_F(LappedTensorBufferCalculatorTest, OneToOne) {
-  SetUpCalculator(1, 0, false, 0);
+  SetUpCalculator(1, 0, false, 0, 0, false);
  int num_timesteps = 3;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -74,7 +79,7 @@ TEST_F(LappedTensorBufferCalculatorTest, OneToTwo) {
  int buffer_size = 2;
  int overlap = 1;
  bool add_dim = false;
-  SetUpCalculator(buffer_size, overlap, add_dim, 0);
+  SetUpCalculator(buffer_size, overlap, add_dim, 0, 0, false);
  int num_timesteps = 3;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -100,7 +105,7 @@ TEST_F(LappedTensorBufferCalculatorTest, OneToThree) {
  int buffer_size = 3;
  int overlap = 2;
  bool add_dim = false;
-  SetUpCalculator(buffer_size, overlap, add_dim, 0);
+  SetUpCalculator(buffer_size, overlap, add_dim, 0, 0, false);
  int num_timesteps = 3;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -126,7 +131,7 @@ TEST_F(LappedTensorBufferCalculatorTest, OneToThreeSkip) {
  int buffer_size = 3;
  int overlap = 1;
  bool add_dim = false;
-  SetUpCalculator(buffer_size, overlap, add_dim, 0);
+  SetUpCalculator(buffer_size, overlap, add_dim, 0, 0, false);
  int num_timesteps = 3;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -152,7 +157,7 @@ TEST_F(LappedTensorBufferCalculatorTest, OneToThreeBatch) {
  int buffer_size = 3;
  int overlap = 2;
  bool add_dim = true;
-  SetUpCalculator(buffer_size, overlap, add_dim, 0);
+  SetUpCalculator(buffer_size, overlap, add_dim, 0, 0, false);
  int num_timesteps = 3;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -180,7 +185,7 @@ TEST_F(LappedTensorBufferCalculatorTest, NegativeTimestampOffsetFails) {
  int overlap = 15;
  bool add_dim = true;
  int timestamp_offset = -7;
-  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset);
+  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset, 0, false);
  int num_timesteps = 20;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -197,7 +202,7 @@ TEST_F(LappedTensorBufferCalculatorTest, OutOfRangeTimestampOffsetFails) {
  int overlap = 15;
  bool add_dim = true;
  int timestamp_offset = buffer_size;
-  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset);
+  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset, 0, false);
  int num_timesteps = 20;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -214,7 +219,7 @@ TEST_F(LappedTensorBufferCalculatorTest, OneToThreeBatchTimestampOffset) {
  int overlap = 15;
  bool add_dim = true;
  int timestamp_offset = 7;
-  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset);
+  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset, 0, false);
  int num_timesteps = 20;
  for (int i = 0; i < num_timesteps; ++i) {
    auto input = ::absl::make_unique<tensorflow::Tensor>(
@@ -236,5 +241,37 @@ TEST_F(LappedTensorBufferCalculatorTest, OneToThreeBatchTimestampOffset) {
  }
 }

+TEST_F(LappedTensorBufferCalculatorTest,
+       OneToThreeBatchTimestampOffsetPadding) {
+  int buffer_size = 12;
+  int overlap = 6;
+  bool add_dim = true;
+  int timestamp_offset = 3;
+  int padding = 0;
+  SetUpCalculator(buffer_size, overlap, add_dim, timestamp_offset, padding,
+                  true);
+  int num_timesteps = 20;
+  for (int i = 0; i < num_timesteps; ++i) {
+    auto input = ::absl::make_unique<tensorflow::Tensor>(
+        tensorflow::DT_FLOAT, tensorflow::TensorShape({1}));
+    input->tensor<float, 1>()(0) = i;
+    runner_->MutableInputs()->Index(0).packets.push_back(
+        Adopt(input.release()).At(Timestamp(i)));
+  }
+  ASSERT_TRUE(runner_->Run().ok());
+
+  const int output_size = num_timesteps / buffer_size + 1;
+  const std::vector<Packet>& output_packets =
+      runner_->Outputs().Index(0).packets;
+  ASSERT_EQ(output_size, output_packets.size());
+  for (int i = 0; i < output_size; ++i) {
+    int64 value = output_packets[i].Timestamp().Value();
+    ASSERT_EQ(i * overlap + timestamp_offset, value);
+  }
+  const std::vector<Packet>& output_timestamps =
+      runner_->Outputs().Index(1).packets;
+  ASSERT_EQ(output_size, output_timestamps.size());
+}
+
 }  // namespace
 }  // namespace mediapipe
@@ -184,6 +184,7 @@ class PackMediaSequenceCalculator : public CalculatorBase {
      features_present_[tag] = false;
    }

+    replace_keypoints_ = false;
    if (cc->Options<PackMediaSequenceCalculatorOptions>()
            .replace_data_instead_of_append()) {
      for (const auto& tag : cc->Inputs().GetTags()) {
@@ -212,6 +213,15 @@ class PackMediaSequenceCalculator : public CalculatorBase {
          }
          mpms::ClearBBox(key, sequence_.get());
          mpms::ClearBBoxTimestamp(key, sequence_.get());
+          mpms::ClearBBoxIsAnnotated(key, sequence_.get());
+          mpms::ClearBBoxNumRegions(key, sequence_.get());
+          mpms::ClearBBoxLabelString(key, sequence_.get());
+          mpms::ClearBBoxLabelIndex(key, sequence_.get());
+          mpms::ClearBBoxClassString(key, sequence_.get());
+          mpms::ClearBBoxClassIndex(key, sequence_.get());
+          mpms::ClearBBoxTrackString(key, sequence_.get());
+          mpms::ClearBBoxTrackIndex(key, sequence_.get());
+          mpms::ClearUnmodifiedBBoxTimestamp(key, sequence_.get());
        }
        if (absl::StartsWith(tag, kFloatFeaturePrefixTag)) {
          std::string key = tag.substr(sizeof(kFloatFeaturePrefixTag) /
@@ -223,8 +233,7 @@ class PackMediaSequenceCalculator : public CalculatorBase {
        if (absl::StartsWith(tag, kKeypointsTag)) {
          std::string key =
              tag.substr(sizeof(kKeypointsTag) / sizeof(*kKeypointsTag) - 1);
-          mpms::ClearBBoxPoint(key, sequence_.get());
-          mpms::ClearBBoxTimestamp(key, sequence_.get());
+          replace_keypoints_ = true;
        }
      }
      if (cc->Inputs().HasTag(kForwardFlowEncodedTag)) {
@@ -342,11 +351,25 @@ class PackMediaSequenceCalculator : public CalculatorBase {
                .Get<std::unordered_map<
                    std::string, std::vector<std::pair<float, float>>>>();
        for (const auto& pair : keypoints) {
-          mpms::AddBBoxTimestamp(mpms::merge_prefix(key, pair.first),
-                                 cc->InputTimestamp().Value(), sequence_.get());
-          mpms::AddBBoxPoint(mpms::merge_prefix(key, pair.first), pair.second,
-                             sequence_.get());
+          std::string prefix = mpms::merge_prefix(key, pair.first);
+          if (replace_keypoints_) {
+            mpms::ClearBBoxPoint(prefix, sequence_.get());
+            mpms::ClearBBoxTimestamp(prefix, sequence_.get());
+            mpms::ClearBBoxIsAnnotated(prefix, sequence_.get());
+            mpms::ClearBBoxNumRegions(prefix, sequence_.get());
+            mpms::ClearBBoxLabelString(prefix, sequence_.get());
+            mpms::ClearBBoxLabelIndex(prefix, sequence_.get());
+            mpms::ClearBBoxClassString(prefix, sequence_.get());
+            mpms::ClearBBoxClassIndex(prefix, sequence_.get());
+            mpms::ClearBBoxTrackString(prefix, sequence_.get());
+            mpms::ClearBBoxTrackIndex(prefix, sequence_.get());
+            mpms::ClearUnmodifiedBBoxTimestamp(prefix, sequence_.get());
+          }
+          mpms::AddBBoxTimestamp(prefix, cc->InputTimestamp().Value(),
+                                 sequence_.get());
+          mpms::AddBBoxPoint(prefix, pair.second, sequence_.get());
        }
+        replace_keypoints_ = false;
      }
      if (absl::StartsWith(tag, kFloatContextFeaturePrefixTag) &&
          !cc->Inputs().Tag(tag).IsEmpty()) {
@@ -475,6 +498,7 @@ class PackMediaSequenceCalculator : public CalculatorBase {

  std::unique_ptr<tf::SequenceExample> sequence_;
  std::map<std::string, bool> features_present_;
+  bool replace_keypoints_;
 };
 REGISTER_CALCULATOR(PackMediaSequenceCalculator);

@@ -839,5 +839,59 @@ TEST_F(PackMediaSequenceCalculatorTest, TestReconcilingAnnotations) {
  ASSERT_EQ(mpms::GetBBoxTimestampAt("PREFIX", output_sequence, 4), 50);
 }

+TEST_F(PackMediaSequenceCalculatorTest, TestOverwritingAndReconciling) {
+  SetUpCalculator({"IMAGE:images", "BBOX:bbox"}, {}, false, true);
+  auto input_sequence = ::absl::make_unique<tf::SequenceExample>();
+  cv::Mat image(2, 3, CV_8UC3, cv::Scalar(0, 0, 255));
+  std::vector<uchar> bytes;
+  ASSERT_TRUE(cv::imencode(".jpg", image, bytes, {80}));
+  std::string test_image_string(bytes.begin(), bytes.end());
+  OpenCvImageEncoderCalculatorResults encoded_image;
+  encoded_image.set_encoded_image(test_image_string);
+  int height = 2;
+  int width = 2;
+  encoded_image.set_width(width);
+  encoded_image.set_height(height);
+
+  int num_images = 5;  // Timestamps: 10, 20, 30, 40, 50
+  for (int i = 0; i < num_images; ++i) {
+    auto image_ptr =
+        ::absl::make_unique<OpenCvImageEncoderCalculatorResults>(encoded_image);
+    runner_->MutableInputs()->Tag("IMAGE").packets.push_back(
+        Adopt(image_ptr.release()).At(Timestamp(i)));
+  }
+
+  for (int i = 0; i < num_images; ++i) {
+    auto detections = ::absl::make_unique<::std::vector<Detection>>();
+    Detection detection;
+    detection = Detection();
+    detection.add_label("relative bbox");
+    detection.add_label_id(1);
+    detection.add_score(0.75);
+    Location::CreateRelativeBBoxLocation(0, 0.5, 0.5, 0.5)
+        .ConvertToProto(detection.mutable_location_data());
+    detections->push_back(detection);
+    runner_->MutableInputs()->Tag("BBOX").packets.push_back(
+        Adopt(detections.release()).At(Timestamp(i)));
+  }
+
+  for (int i = 0; i < 10; ++i) {
+    mpms::AddBBoxTimestamp(-1, input_sequence.get());
+    mpms::AddBBoxIsAnnotated(-1, input_sequence.get());
+    mpms::AddBBoxNumRegions(-1, input_sequence.get());
+    mpms::AddBBoxLabelString({"anything"}, input_sequence.get());
+    mpms::AddBBoxLabelIndex({-1}, input_sequence.get());
+    mpms::AddBBoxClassString({"anything"}, input_sequence.get());
+    mpms::AddBBoxClassIndex({-1}, input_sequence.get());
+    mpms::AddBBoxTrackString({"anything"}, input_sequence.get());
+    mpms::AddBBoxTrackIndex({-1}, input_sequence.get());
+  }
+
+  runner_->MutableSidePackets()->Tag("SEQUENCE_EXAMPLE") =
+      Adopt(input_sequence.release());
+  // If the all the previous values aren't cleared, this assert will fail.
+  MP_ASSERT_OK(runner_->Run());
+}
+
 }  // namespace
 }  // namespace mediapipe
@@ -19,16 +19,22 @@
 #include <unordered_set>
 #include <vector>

+#include "absl/base/thread_annotations.h"
+#include "absl/memory/memory.h"
 #include "absl/strings/str_split.h"
 #include "absl/synchronization/mutex.h"
 #include "mediapipe/calculators/tensorflow/tensorflow_inference_calculator.pb.h"
 #include "mediapipe/calculators/tensorflow/tensorflow_session.h"
+#include "mediapipe/framework/calculator_context.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/deps/clock.h"
 #include "mediapipe/framework/deps/monotonic_clock.h"
+#include "mediapipe/framework/packet.h"
+#include "mediapipe/framework/port/map_util.h"
 #include "mediapipe/framework/port/ret_check.h"
 #include "mediapipe/framework/port/status.h"
 #include "mediapipe/framework/port/status_macros.h"
+#include "mediapipe/framework/timestamp.h"
 #include "mediapipe/framework/tool/status_util.h"
 #include "tensorflow/core/framework/tensor.h"
 #include "tensorflow/core/framework/tensor_shape.h"
@@ -77,6 +83,17 @@ class SimpleSemaphore {
  absl::Mutex mutex_;
  absl::CondVar cond_;
 };
+
+class InferenceState {
+ public:
+  InferenceState() : input_tensor_batches_(), batch_timestamps_() {}
+  // A mapping between stream tags and the tensors we are collecting as a
+  // batch.
+  std::map<std::string, std::vector<tf::Tensor>> input_tensor_batches_;
+  // The timestamps that go into a batch.
+  std::vector<Timestamp> batch_timestamps_;
+};
+
 }  // namespace

 // This calculator performs inference on a trained TensorFlow model.
@@ -218,11 +235,16 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
  }

  static ::mediapipe::Status GetContract(CalculatorContract* cc) {
+    const auto& options = cc->Options<TensorFlowInferenceCalculatorOptions>();
    RET_CHECK(!cc->Inputs().GetTags().empty());
    for (const std::string& tag : cc->Inputs().GetTags()) {
      // The tensorflow::Tensor with the tag equal to the graph node. May
      // have a TimeSeriesHeader if all present TimeSeriesHeaders match.
-      cc->Inputs().Tag(tag).Set<tf::Tensor>();
+      if (!options.batched_input()) {
+        cc->Inputs().Tag(tag).Set<tf::Tensor>();
+      } else {
+        cc->Inputs().Tag(tag).Set<std::vector<mediapipe::Packet>>();
+      }
    }
    RET_CHECK(!cc->Outputs().GetTags().empty());
    for (const std::string& tag : cc->Outputs().GetTags()) {
@@ -242,6 +264,22 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
    return ::mediapipe::OkStatus();
  }

+  std::unique_ptr<InferenceState> CreateInferenceState(CalculatorContext* cc)
+      ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
+    std::unique_ptr<InferenceState> inference_state =
+        absl::make_unique<InferenceState>();
+    if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS") &&
+        !cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS").IsEmpty()) {
+      std::map<std::string, tf::Tensor>* init_tensor_map;
+      init_tensor_map = GetFromUniquePtr<std::map<std::string, tf::Tensor>>(
+          cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS"));
+      for (const auto& p : *init_tensor_map) {
+        inference_state->input_tensor_batches_[p.first].emplace_back(p.second);
+      }
+    }
+    return inference_state;
+  }
+
  ::mediapipe::Status Open(CalculatorContext* cc) override {
    options_ = cc->Options<TensorFlowInferenceCalculatorOptions>();

@@ -275,15 +313,6 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
      recurrent_feed_tags_.insert(tags[0]);
      recurrent_fetch_tags_to_feed_tags_[tags[1]] = tags[0];
    }
-    if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS") &&
-        !cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS").IsEmpty()) {
-      std::map<std::string, tf::Tensor>* init_tensor_map;
-      init_tensor_map = GetFromUniquePtr<std::map<std::string, tf::Tensor>>(
-          cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS"));
-      for (const auto& p : *init_tensor_map) {
-        input_tensor_batches_[p.first].emplace_back(p.second);
-      }
-    }

    // Check that all tags are present in this signature bound to tensors.
    for (const std::string& tag : cc->Inputs().GetTags()) {
@@ -297,9 +326,15 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
          << options_.signature_name();
    }

-    if (options_.batch_size() == 1) {
+    {
+      absl::WriterMutexLock l(&mutex_);
+      inference_state_ = std::unique_ptr<InferenceState>();
+    }
+
+    if (options_.batch_size() == 1 || options_.batched_input()) {
      cc->SetOffset(0);
    }
+
    return ::mediapipe::OkStatus();
  }

@@ -316,6 +351,24 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
    return ::mediapipe::OkStatus();
  }

+  ::mediapipe::Status AggregateTensorPacket(
+      const std::string& tag_name, const Packet& packet,
+      std::map<Timestamp, std::map<std::string, tf::Tensor>>*
+          input_tensors_by_tag_by_timestamp,
+      InferenceState* inference_state) ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
+    tf::Tensor input_tensor(packet.Get<tf::Tensor>());
+    RET_CHECK_OK(AddBatchDimension(&input_tensor));
+    if (::mediapipe::ContainsKey(recurrent_feed_tags_, tag_name)) {
+      // If we receive an input on a recurrent tag, override the state.
+      // It's OK to override the global state because there is just one
+      // input stream allowed for recurrent tensors.
+      inference_state_->input_tensor_batches_[tag_name].clear();
+    }
+    (*input_tensors_by_tag_by_timestamp)[packet.Timestamp()].insert(
+        std::make_pair(tag_name, input_tensor));
+    return ::mediapipe::OkStatus();
+  }
+
  // Removes the batch dimension of the output tensor if specified in the
  // calculator options.
  ::mediapipe::Status RemoveBatchDimension(tf::Tensor* output_tensor) {
@@ -331,48 +384,85 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
  }

  ::mediapipe::Status Process(CalculatorContext* cc) override {
-    std::map<std::string, tf::Tensor> input_tensors_by_tag;
-    for (const std::string& tag_as_node_name : cc->Inputs().GetTags()) {
-      if (cc->Inputs().Tag(tag_as_node_name).IsEmpty()) {
-        // Recurrent tensors can be empty.
-        if (!::mediapipe::ContainsKey(recurrent_feed_tags_, tag_as_node_name)) {
-          if (options_.skip_on_missing_features()) {
-            return ::mediapipe::OkStatus();
-          } else {
-            return ::mediapipe::InvalidArgumentError(absl::StrCat(
-                "Tag ", tag_as_node_name,
-                " not present at timestamp: ", cc->InputTimestamp().Value()));
+    std::unique_ptr<InferenceState> inference_state_to_process;
+    {
+      absl::WriterMutexLock l(&mutex_);
+      if (inference_state_ == nullptr) {
+        inference_state_ = CreateInferenceState(cc);
+      }
+      std::map<Timestamp, std::map<std::string, tf::Tensor>>
+          input_tensors_by_tag_by_timestamp;
+      for (const std::string& tag_as_node_name : cc->Inputs().GetTags()) {
+        if (cc->Inputs().Tag(tag_as_node_name).IsEmpty()) {
+          // Recurrent tensors can be empty.
+          if (!::mediapipe::ContainsKey(recurrent_feed_tags_,
+                                        tag_as_node_name)) {
+            if (options_.skip_on_missing_features()) {
+              return ::mediapipe::OkStatus();
+            } else {
+              return ::mediapipe::InvalidArgumentError(absl::StrCat(
+                  "Tag ", tag_as_node_name,
+                  " not present at timestamp: ", cc->InputTimestamp().Value()));
+            }
          }
+        } else if (options_.batched_input()) {
+          const auto& tensor_packets =
+              cc->Inputs().Tag(tag_as_node_name).Get<std::vector<Packet>>();
+          if (tensor_packets.size() > options_.batch_size()) {
+            return ::mediapipe::InvalidArgumentError(absl::StrCat(
+                "Batch for tag ", tag_as_node_name,
+                " has more packets than batch capacity. batch_size: ",
+                options_.batch_size(), " packets: ", tensor_packets.size()));
+          }
+          for (const auto& packet : tensor_packets) {
+            RET_CHECK_OK(AggregateTensorPacket(
+                tag_as_node_name, packet, &input_tensors_by_tag_by_timestamp,
+                inference_state_.get()));
+          }
+        } else {
+          RET_CHECK_OK(AggregateTensorPacket(
+              tag_as_node_name, cc->Inputs().Tag(tag_as_node_name).Value(),
+              &input_tensors_by_tag_by_timestamp, inference_state_.get()));
        }
-      } else {
-        tf::Tensor input_tensor(
-            cc->Inputs().Tag(tag_as_node_name).Get<tf::Tensor>());
-        RET_CHECK_OK(AddBatchDimension(&input_tensor));
-        if (::mediapipe::ContainsKey(recurrent_feed_tags_, tag_as_node_name)) {
-          // If we receive an input on a recurrent tag, override the state.
-          // It's OK to override the global state because there is just one
-          // input stream allowed for recurrent tensors.
-          input_tensor_batches_[tag_as_node_name].clear();
+      }
+      for (const auto& timestamp_and_input_tensors_by_tag :
+           input_tensors_by_tag_by_timestamp) {
+        inference_state_->batch_timestamps_.emplace_back(
+            timestamp_and_input_tensors_by_tag.first);
+        for (const auto& input_tensor_and_tag :
+             timestamp_and_input_tensors_by_tag.second) {
+          inference_state_->input_tensor_batches_[input_tensor_and_tag.first]
+              .emplace_back(input_tensor_and_tag.second);
        }
-        input_tensors_by_tag.insert(
-            std::make_pair(tag_as_node_name, input_tensor));
+      }
+      if (inference_state_->batch_timestamps_.size() == options_.batch_size() ||
+          options_.batched_input()) {
+        inference_state_to_process = std::move(inference_state_);
+        inference_state_ = std::unique_ptr<InferenceState>();
      }
    }
-    batch_timestamps_.emplace_back(cc->InputTimestamp());
-    for (const auto& input_tensor_and_tag : input_tensors_by_tag) {
-      input_tensor_batches_[input_tensor_and_tag.first].emplace_back(
-          input_tensor_and_tag.second);
+
+    if (inference_state_to_process) {
+      MP_RETURN_IF_ERROR(
+          OutputBatch(cc, std::move(inference_state_to_process)));
    }

-    if (batch_timestamps_.size() == options_.batch_size()) {
-      MP_RETURN_IF_ERROR(OutputBatch(cc));
-    }
    return ::mediapipe::OkStatus();
  }

  ::mediapipe::Status Close(CalculatorContext* cc) override {
-    if (!batch_timestamps_.empty()) {
-      MP_RETURN_IF_ERROR(OutputBatch(cc));
+    std::unique_ptr<InferenceState> inference_state_to_process = nullptr;
+    {
+      absl::WriterMutexLock l(&mutex_);
+      if (cc->GraphStatus().ok() && inference_state_ != nullptr &&
+          !inference_state_->batch_timestamps_.empty()) {
+        inference_state_to_process = std::move(inference_state_);
+        inference_state_ = std::unique_ptr<InferenceState>();
+      }
+    }
+    if (inference_state_to_process) {
+      MP_RETURN_IF_ERROR(
+          OutputBatch(cc, std::move(inference_state_to_process)));
    }
    return ::mediapipe::OkStatus();
  }
@@ -385,10 +475,12 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
  // memory buffer. Therefore, copies are cheap and should not cause the memory
  // buffer to fall out of scope. In contrast, concat is only used where
  // necessary.
-  ::mediapipe::Status OutputBatch(CalculatorContext* cc) {
+  ::mediapipe::Status OutputBatch(
+      CalculatorContext* cc, std::unique_ptr<InferenceState> inference_state) {
    const int64 start_time = absl::ToUnixMicros(clock_->TimeNow());
    std::vector<std::pair<mediapipe::ProtoString, tf::Tensor>> input_tensors;
-    for (auto& keyed_tensors : input_tensor_batches_) {
+
+    for (auto& keyed_tensors : inference_state->input_tensor_batches_) {
      if (options_.batch_size() == 1) {
        // Short circuit to avoid the cost of deep copying tensors in concat.
        if (!keyed_tensors.second.empty()) {
@@ -404,7 +496,8 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
      } else {
        // Pad by replicating the first tens  or, then ignore the values.
        keyed_tensors.second.resize(options_.batch_size());
-        std::fill(keyed_tensors.second.begin() + batch_timestamps_.size(),
+        std::fill(keyed_tensors.second.begin() +
+                      inference_state->batch_timestamps_.size(),
                  keyed_tensors.second.end(), keyed_tensors.second[0]);
        tf::Tensor concated;
        const tf::Status concat_status =
@@ -414,7 +507,7 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
                                   concated);
      }
    }
-    input_tensor_batches_.clear();
+    inference_state->input_tensor_batches_.clear();
    std::vector<mediapipe::ProtoString> output_tensor_names;
    std::vector<std::string> output_name_in_signature;
    for (const std::string& tag : cc->Outputs().GetTags()) {
@@ -466,9 +559,11 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
      int pos = std::find(output_name_in_signature.begin(),
                          output_name_in_signature.end(), tag_pair.first) -
                output_name_in_signature.begin();
-      input_tensor_batches_[tag_pair.second].emplace_back(outputs[pos]);
+      inference_state->input_tensor_batches_[tag_pair.second].emplace_back(
+          outputs[pos]);
    }

+    absl::WriterMutexLock l(&mutex_);
    // Set that we want to split on each index of the 0th dimension.
    std::vector<tf::int64> split_vector(options_.batch_size(), 1);
    for (int i = 0; i < output_tensor_names.size(); ++i) {
@@ -478,7 +573,8 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
          RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
          cc->Outputs()
              .Tag(output_name_in_signature[i])
-              .Add(new tf::Tensor(output_tensor), batch_timestamps_[0]);
+              .Add(new tf::Tensor(output_tensor),
+                   inference_state->batch_timestamps_[0]);
        }
      } else {
        std::vector<tf::Tensor> split_tensors;
@@ -486,22 +582,30 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
            tf::tensor::Split(outputs[i], split_vector, &split_tensors);
        CHECK(split_status.ok()) << split_status.ToString();
        // Loop over timestamps so that we don't copy the padding.
-        for (int j = 0; j < batch_timestamps_.size(); ++j) {
+        for (int j = 0; j < inference_state->batch_timestamps_.size(); ++j) {
          tf::Tensor output_tensor(split_tensors[j]);
          RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
          cc->Outputs()
              .Tag(output_name_in_signature[i])
-              .Add(new tf::Tensor(output_tensor), batch_timestamps_[j]);
+              .Add(new tf::Tensor(output_tensor),
+                   inference_state->batch_timestamps_[j]);
        }
      }
    }
+
    // Get end time and report.
    const int64 end_time = absl::ToUnixMicros(clock_->TimeNow());
    cc->GetCounter(kTotalUsecsCounterSuffix)
        ->IncrementBy(end_time - start_time);
    cc->GetCounter(kTotalProcessedTimestampsCounterSuffix)
-        ->IncrementBy(batch_timestamps_.size());
-    batch_timestamps_.clear();
+        ->IncrementBy(inference_state->batch_timestamps_.size());
+
+    // Make sure we hold on to the recursive state.
+    if (!options_.recurrent_tag_pair().empty()) {
+      inference_state_ = std::move(inference_state);
+      inference_state_->batch_timestamps_.clear();
+    }
+
    return ::mediapipe::OkStatus();
  }

@@ -514,11 +618,8 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
  // A mapping between stream tags and the tensor names they are bound to.
  std::map<std::string, std::string> tag_to_tensor_map_;

-  // A mapping between stream tags and the tensors we are collecting as a batch.
-  std::map<std::string, std::vector<tf::Tensor>> input_tensor_batches_;
-
-  // The timestamps that go into a batch.
-  std::vector<Timestamp> batch_timestamps_;
+  absl::Mutex mutex_;
+  std::unique_ptr<InferenceState> inference_state_ ABSL_GUARDED_BY(mutex_);

  // The options for the calculator.
  TensorFlowInferenceCalculatorOptions options_;
@@ -76,4 +76,13 @@ message TensorFlowInferenceCalculatorOptions {
  // only works in the local process, not "globally" across multiple processes
  // or replicas (if any). Default to 0, i.e. no limit.
  optional int32 max_concurrent_session_runs = 6 [default = 0];
+
+  // If turned on, the Calculator expects a vector of batched packages as input.
+  // This will make sure that you can turn on max_in_flight for batch_size
+  // greater than 1. Otherwise it results in problems of none-monotonically
+  // increasing timestamps.
+  // Use BatchSequentialCalculator to create the batches. The batch_size
+  // should agree for both calculators. All the data in a batch is processed
+  // together. The BatchSequentialCalculator can't run with max_in_flight.
+  optional bool batched_input = 7;
 }
@@ -89,17 +89,31 @@ class TensorflowInferenceCalculatorTest : public ::testing::Test {
        output_side_packets.Tag("SESSION");
  }

-  // Create tensor from Vector and add as a Packet to the provided tag as input.
-  void AddVectorToInputsAsTensor(const std::vector<int32>& input,
-                                 const std::string& tag, int64 time) {
+  Packet CreateTensorPacket(const std::vector<int32>& input, int64 time) {
    tf::TensorShape tensor_shape;
    tensor_shape.AddDim(input.size());
    auto tensor = absl::make_unique<tf::Tensor>(tf::DT_INT32, tensor_shape);
    for (int i = 0; i < input.size(); ++i) {
      tensor->vec<int32>()(i) = input[i];
    }
+    return Adopt(tensor.release()).At(Timestamp(time));
+  }
+
+  // Create tensor from Vector and add as a Packet to the provided tag as input.
+  void AddVectorToInputsAsTensor(const std::vector<int32>& input,
+                                 const std::string& tag, int64 time) {
    runner_->MutableInputs()->Tag(tag).packets.push_back(
-        Adopt(tensor.release()).At(Timestamp(time)));
+        CreateTensorPacket(input, time));
+  }
+
+  // Create tensor from Vector and add as a Packet to the provided tag as input.
+  void AddVectorToInputsAsPacket(const std::vector<Packet>& packets,
+                                 const std::string& tag) {
+    CHECK(!packets.empty())
+        << "Please specify at least some data in the packet";
+    auto packets_ptr = absl::make_unique<std::vector<Packet>>(packets);
+    runner_->MutableInputs()->Tag(tag).packets.push_back(
+        Adopt(packets_ptr.release()).At(packets.begin()->Timestamp()));
  }

  std::unique_ptr<CalculatorRunner> runner_;
@@ -183,6 +197,45 @@ TEST_F(TensorflowInferenceCalculatorTest, GetComputed) {
  EXPECT_THAT(run_status.ToString(), testing::HasSubstr("Tag B"));
 }

+TEST_F(TensorflowInferenceCalculatorTest, GetComputed_MaxInFlight) {
+  CalculatorGraphConfig::Node config;
+  config.set_calculator("TensorFlowInferenceCalculator");
+  config.add_input_stream("A:tensor_a");
+  config.add_input_stream("B:tensor_b");
+  config.add_output_stream("MULTIPLIED:tensor_o1");
+  config.add_input_side_packet("SESSION:session");
+  config.set_max_in_flight(2);
+  CalculatorOptions options;
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batch_size(1);
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_add_batch_dim_to_tensors(false);
+  *config.mutable_options() = options;
+
+  runner_ = absl::make_unique<CalculatorRunner>(config);
+  AddSessionInputSidePacket();
+  AddVectorToInputsAsTensor({2, 2, 2}, "A", 0);
+  AddVectorToInputsAsTensor({3, 4, 5}, "B", 0);
+  MP_ASSERT_OK(runner_->Run());
+
+  const std::vector<Packet>& output_packets_mult =
+      runner_->Outputs().Tag("MULTIPLIED").packets;
+  ASSERT_EQ(1, output_packets_mult.size());
+  const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
+  tf::TensorShape expected_shape({3});
+  auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10}, expected_shape);
+  tf::test::ExpectTensorEqual<int32>(expected_tensor, tensor_mult);
+
+  // Add only one of the two expected tensors at the next timestamp, expect
+  // useful failure message.
+  AddVectorToInputsAsTensor({1, 2, 3}, "A", 1);
+  auto run_status = runner_->Run();
+  ASSERT_FALSE(run_status.ok());
+  EXPECT_THAT(run_status.ToString(),
+              testing::HasSubstr("TensorFlowInferenceCalculator"));
+  EXPECT_THAT(run_status.ToString(), testing::HasSubstr("Tag B"));
+}
+
 TEST_F(TensorflowInferenceCalculatorTest, BadTag) {
  CalculatorGraphConfig::Node config;
  config.set_calculator("TensorFlowInferenceCalculator");
@@ -235,6 +288,86 @@ TEST_F(TensorflowInferenceCalculatorTest, GetMultiBatchComputed) {
                   ->Get());
 }

+TEST_F(TensorflowInferenceCalculatorTest, GetMultiBatchComputed_MaxInFlight) {
+  CalculatorGraphConfig::Node config;
+  config.set_calculator("TensorFlowInferenceCalculator");
+  config.add_input_stream("A:tensor_a");
+  config.add_input_stream("B:tensor_b");
+  config.add_output_stream("MULTIPLIED:tensor_o1");
+  config.add_input_side_packet("SESSION:session");
+  config.set_max_in_flight(2);
+  CalculatorOptions options;
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batch_size(1);
+  *config.mutable_options() = options;
+
+  runner_ = absl::make_unique<CalculatorRunner>(config);
+  AddSessionInputSidePacket();
+  AddVectorToInputsAsTensor({2, 2, 2}, "A", 0);
+  AddVectorToInputsAsTensor({3, 4, 5}, "B", 0);
+  AddVectorToInputsAsTensor({3, 3, 3}, "A", 1);
+  AddVectorToInputsAsTensor({3, 4, 5}, "B", 1);
+  MP_ASSERT_OK(runner_->Run());
+
+  const std::vector<Packet>& output_packets_mult =
+      runner_->Outputs().Tag("MULTIPLIED").packets;
+  ASSERT_EQ(2, output_packets_mult.size());
+  const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
+  auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult, expected_tensor);
+  const tf::Tensor& tensor_mult1 = output_packets_mult[1].Get<tf::Tensor>();
+  auto expected_tensor1 = tf::test::AsTensor<int32>({9, 12, 15});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult1, expected_tensor1);
+
+  EXPECT_EQ(2, runner_
+                   ->GetCounter(
+                       "TensorFlowInferenceCalculator-TotalProcessedTimestamps")
+                   ->Get());
+}
+
+TEST_F(TensorflowInferenceCalculatorTest,
+       GetMultiBatchComputed_MoreThanMaxInFlight) {
+  CalculatorGraphConfig::Node config;
+  config.set_calculator("TensorFlowInferenceCalculator");
+  config.add_input_stream("A:tensor_a");
+  config.add_input_stream("B:tensor_b");
+  config.add_output_stream("MULTIPLIED:tensor_o1");
+  config.add_input_side_packet("SESSION:session");
+  config.set_max_in_flight(2);
+  CalculatorOptions options;
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batch_size(1);
+  *config.mutable_options() = options;
+
+  runner_ = absl::make_unique<CalculatorRunner>(config);
+  AddSessionInputSidePacket();
+  AddVectorToInputsAsTensor({2, 2, 2}, "A", 0);
+  AddVectorToInputsAsTensor({3, 4, 5}, "B", 0);
+  AddVectorToInputsAsTensor({3, 3, 3}, "A", 1);
+  AddVectorToInputsAsTensor({3, 4, 5}, "B", 1);
+  AddVectorToInputsAsTensor({4, 4, 4}, "A", 2);
+  AddVectorToInputsAsTensor({3, 4, 5}, "B", 2);
+  MP_ASSERT_OK(runner_->Run());
+
+  const std::vector<Packet>& output_packets_mult =
+      runner_->Outputs().Tag("MULTIPLIED").packets;
+  ASSERT_EQ(3, output_packets_mult.size());
+  const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
+  auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult, expected_tensor);
+  const tf::Tensor& tensor_mult1 = output_packets_mult[1].Get<tf::Tensor>();
+  auto expected_tensor1 = tf::test::AsTensor<int32>({9, 12, 15});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult1, expected_tensor1);
+  const tf::Tensor& tensor_mult2 = output_packets_mult[2].Get<tf::Tensor>();
+  auto expected_tensor2 = tf::test::AsTensor<int32>({12, 16, 20});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult2, expected_tensor2);
+
+  EXPECT_EQ(3, runner_
+                   ->GetCounter(
+                       "TensorFlowInferenceCalculator-TotalProcessedTimestamps")
+                   ->Get());
+}
+
 TEST_F(TensorflowInferenceCalculatorTest, GetSingleBatchComputed) {
  CalculatorGraphConfig::Node config;
  config.set_calculator("TensorFlowInferenceCalculator");
@@ -311,6 +444,66 @@ TEST_F(TensorflowInferenceCalculatorTest, GetCloseBatchComputed) {
                   ->Get());
 }

+TEST_F(TensorflowInferenceCalculatorTest, GetBatchComputed_MaxInFlight) {
+  CalculatorGraphConfig::Node config;
+  config.set_calculator("TensorFlowInferenceCalculator");
+  config.add_input_stream("A:tensor_a");
+  config.add_input_stream("B:tensor_b");
+  config.add_output_stream("MULTIPLIED:tensor_o1");
+  config.add_input_side_packet("SESSION:session");
+  config.set_max_in_flight(2);
+  CalculatorOptions options;
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batch_size(2);
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_add_batch_dim_to_tensors(true);
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batched_input(true);
+  *config.mutable_options() = options;
+
+  runner_ = absl::make_unique<CalculatorRunner>(config);
+  AddSessionInputSidePacket();
+  AddVectorToInputsAsPacket(
+      {CreateTensorPacket({2, 2, 2}, 0), CreateTensorPacket({3, 3, 3}, 1)},
+      "A");
+  AddVectorToInputsAsPacket(
+      {CreateTensorPacket({3, 4, 5}, 0), CreateTensorPacket({3, 4, 5}, 1)},
+      "B");
+  AddVectorToInputsAsPacket(
+      {CreateTensorPacket({4, 4, 4}, 2), CreateTensorPacket({5, 5, 5}, 3)},
+      "A");
+  AddVectorToInputsAsPacket(
+      {CreateTensorPacket({3, 4, 5}, 2), CreateTensorPacket({3, 4, 5}, 3)},
+      "B");
+  AddVectorToInputsAsPacket({CreateTensorPacket({6, 6, 6}, 4)}, "A");
+  AddVectorToInputsAsPacket({CreateTensorPacket({3, 4, 5}, 4)}, "B");
+  MP_ASSERT_OK(runner_->Run());
+
+  const std::vector<Packet>& output_packets_mult =
+      runner_->Outputs().Tag("MULTIPLIED").packets;
+  ASSERT_EQ(5, output_packets_mult.size());
+  const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
+  auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult, expected_tensor);
+  const tf::Tensor& tensor_mult1 = output_packets_mult[1].Get<tf::Tensor>();
+  auto expected_tensor1 = tf::test::AsTensor<int32>({9, 12, 15});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult1, expected_tensor1);
+  const tf::Tensor& tensor_mult2 = output_packets_mult[2].Get<tf::Tensor>();
+  auto expected_tensor2 = tf::test::AsTensor<int32>({12, 16, 20});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult2, expected_tensor2);
+  const tf::Tensor& tensor_mult3 = output_packets_mult[3].Get<tf::Tensor>();
+  auto expected_tensor3 = tf::test::AsTensor<int32>({15, 20, 25});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult3, expected_tensor3);
+  const tf::Tensor& tensor_mult4 = output_packets_mult[4].Get<tf::Tensor>();
+  auto expected_tensor4 = tf::test::AsTensor<int32>({18, 24, 30});
+  tf::test::ExpectTensorEqual<int32>(tensor_mult4, expected_tensor4);
+
+  EXPECT_EQ(5, runner_
+                   ->GetCounter(
+                       "TensorFlowInferenceCalculator-TotalProcessedTimestamps")
+                   ->Get());
+}
+
 TEST_F(TensorflowInferenceCalculatorTest, TestRecurrentStates) {
  CalculatorGraphConfig::Node config;
  config.set_calculator("TensorFlowInferenceCalculator");
@@ -509,4 +702,40 @@ TEST_F(TensorflowInferenceCalculatorTest,
                   ->Get());
 }

+TEST_F(TensorflowInferenceCalculatorTest, BatchedInputTooBigBatch) {
+  CalculatorGraphConfig::Node config;
+  config.set_calculator("TensorFlowInferenceCalculator");
+  config.add_input_stream("A:tensor_a");
+  config.add_input_stream("B:tensor_b");
+  config.add_output_stream("MULTIPLIED:tensor_o1");
+  config.add_input_side_packet("SESSION:session");
+  config.set_max_in_flight(2);
+  CalculatorOptions options;
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batch_size(2);
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_add_batch_dim_to_tensors(true);
+  options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
+      ->set_batched_input(true);
+  *config.mutable_options() = options;
+
+  runner_ = absl::make_unique<CalculatorRunner>(config);
+  AddSessionInputSidePacket();
+  AddVectorToInputsAsPacket(
+      {CreateTensorPacket({2, 2, 2}, 0), CreateTensorPacket({3, 3, 3}, 1),
+       CreateTensorPacket({4, 4, 4}, 2)},
+      "A");
+  AddVectorToInputsAsPacket(
+      {CreateTensorPacket({3, 4, 5}, 0), CreateTensorPacket({3, 4, 5}, 1),
+       CreateTensorPacket({3, 4, 5}, 2)},
+      "B");
+
+  auto status = runner_->Run();
+  ASSERT_FALSE(status.ok());
+  EXPECT_THAT(
+      status.message(),
+      ::testing::HasSubstr(
+          "has more packets than batch capacity. batch_size: 2 packets: 3"));
+}
+
 }  // namespace mediapipe
@@ -51,7 +51,7 @@ namespace {
 void SetPreferredDevice(tf::GraphDef* graph_def, absl::string_view device_id) {
  for (auto& node : *graph_def->mutable_node()) {
    if (node.device().empty()) {
-      node.set_device(device_id);
+      node.set_device(std::string(device_id));
    }
  }
 }
@@ -47,7 +47,7 @@ namespace {
 void SetPreferredDevice(tf::GraphDef* graph_def, absl::string_view device_id) {
  for (auto& node : *graph_def->mutable_node()) {
    if (node.device().empty()) {
-      node.set_device(device_id);
+      node.set_device(std::string(device_id));
    }
  }
 }
@@ -29,6 +29,7 @@ namespace mediapipe {
 // Streams:
 const char kBBoxTag[] = "BBOX";
 const char kImageTag[] = "IMAGE";
+const char kKeypointsTag[] = "KEYPOINTS";
 const char kFloatFeaturePrefixTag[] = "FLOAT_FEATURE_";
 const char kForwardFlowImageTag[] = "FORWARD_FLOW_ENCODED";

@@ -150,7 +151,6 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
             << "or" << kAudioDecoderOptions;
    }

-    // Optional streams.
    if (cc->Outputs().HasTag(kForwardFlowImageTag)) {
      cc->Outputs().Tag(kForwardFlowImageTag).Set<std::string>();
    }
@@ -244,6 +244,10 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
    const auto& sequence = cc->InputSidePackets()
                               .Tag(kSequenceExampleTag)
                               .Get<tensorflow::SequenceExample>();
+    if (cc->Outputs().HasTag(kKeypointsTag)) {
+      keypoint_names_ = absl::StrSplit(options.keypoint_names(), ',');
+      default_keypoint_location_ = options.default_keypoint_location();
+    }
    if (cc->OutputSidePackets().HasTag(kDataPath)) {
      std::string root_directory = "";
      if (cc->InputSidePackets().HasTag(kDatasetRootDirTag)) {
@@ -357,7 +361,6 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
      end_timestamp =
          timestamps_[last_timestamp_key_][current_timestamp_index_ + 1];
    }
-
    for (const auto& map_kv : timestamps_) {
      for (int i = 0; i < map_kv.second.size(); ++i) {
        if (map_kv.second[i] >= start_timestamp &&
@@ -454,6 +457,10 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
  int current_timestamp_index_;
  // Store the very first timestamp, so we output everything on the first frame.
  int64 first_timestamp_seen_;
+  // List of keypoint names.
+  std::vector<std::string> keypoint_names_;
+  // Default keypoint location when missing.
+  float default_keypoint_location_;
 };
 REGISTER_CALCULATOR(UnpackMediaSequenceCalculator);
 }  // namespace mediapipe
@@ -56,4 +56,11 @@ message UnpackMediaSequenceCalculatorOptions {
  // the clip start and end times and outputs these for the
  // AudioDecoderCalculator to consume.
  optional AudioDecoderOptions base_audio_decoder_options = 9;
+
+  optional string keypoint_names = 10 [
+    default =
+        "NOSE,LEFT_EAR,RIGHT_EAR,LEFT_SHOULDER,RIGHT_SHOULDER,LEFT_FORE_PAW,RIGHT_FORE_PAW,LEFT_HIP,RIGHT_HIP,LEFT_HIND_PAW,RIGHT_HIND_PAW,ROOT_TAIL"
+  ];
+  // When the keypoint doesn't exists, output this default value.
+  optional float default_keypoint_location = 11 [default = -1.0];
 }
@@ -13,131 +13,91 @@
 # limitations under the License.
 #

-load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
+load("//mediapipe/framework/port:build_config.bzl", "mediapipe_proto_library")
 load("@bazel_skylib//lib:selects.bzl", "selects")

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

-proto_library(
+mediapipe_proto_library(
    name = "ssd_anchors_calculator_proto",
    srcs = ["ssd_anchors_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_custom_op_resolver_calculator_proto",
    srcs = ["tflite_custom_op_resolver_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_inference_calculator_proto",
    srcs = ["tflite_inference_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_converter_calculator_proto",
    srcs = ["tflite_converter_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_tensors_to_segmentation_calculator_proto",
    srcs = ["tflite_tensors_to_segmentation_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_tensors_to_detections_calculator_proto",
    srcs = ["tflite_tensors_to_detections_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_tensors_to_classification_calculator_proto",
    srcs = ["tflite_tensors_to_classification_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "tflite_tensors_to_landmarks_calculator_proto",
    srcs = ["tflite_tensors_to_landmarks_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "ssd_anchors_calculator_cc_proto",
-    srcs = ["ssd_anchors_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":ssd_anchors_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_custom_op_resolver_calculator_cc_proto",
-    srcs = ["tflite_custom_op_resolver_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_custom_op_resolver_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_converter_calculator_cc_proto",
-    srcs = ["tflite_converter_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_converter_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_tensors_to_segmentation_calculator_cc_proto",
-    srcs = ["tflite_tensors_to_segmentation_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_tensors_to_segmentation_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_inference_calculator_cc_proto",
-    srcs = ["tflite_inference_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_inference_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_tensors_to_detections_calculator_cc_proto",
-    srcs = ["tflite_tensors_to_detections_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_tensors_to_detections_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_tensors_to_classification_calculator_cc_proto",
-    srcs = ["tflite_tensors_to_classification_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_tensors_to_classification_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "tflite_tensors_to_landmarks_calculator_cc_proto",
-    srcs = ["tflite_tensors_to_landmarks_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":tflite_tensors_to_landmarks_calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

 cc_library(
@@ -257,6 +217,7 @@ cc_library(
    }) + select({
        "//conditions:default": [],
        "//mediapipe:android": [
+            "//mediapipe/util/android/file/base",
            "@org_tensorflow//tensorflow/lite/delegates/nnapi:nnapi_delegate",
        ],
    }) + select({
@@ -33,6 +33,12 @@
 #include "tensorflow/lite/kernels/register.h"
 #include "tensorflow/lite/model.h"

+#if defined(MEDIAPIPE_ANDROID)
+#include "mediapipe/util/android/file/base/file.h"
+#include "mediapipe/util/android/file/base/filesystem.h"
+#include "mediapipe/util/android/file/base/helpers.h"
+#endif  // ANDROID
+
 #if MEDIAPIPE_TFLITE_GL_INFERENCE
 #include "mediapipe/gpu/gl_calculator_helper.h"
 #include "mediapipe/gpu/gpu_buffer.h"
@@ -219,6 +225,8 @@ class TfLiteInferenceCalculator : public CalculatorBase {
  ::mediapipe::Status Close(CalculatorContext* cc) override;

 private:
+  ::mediapipe::Status ReadKernelsFromFile();
+  ::mediapipe::Status WriteKernelsToFile();
  ::mediapipe::Status LoadModel(CalculatorContext* cc);
  ::mediapipe::StatusOr<Packet> GetModelAsPacket(const CalculatorContext& cc);
  ::mediapipe::Status LoadDelegate(CalculatorContext* cc);
@@ -273,6 +281,12 @@ class TfLiteInferenceCalculator : public CalculatorBase {
  bool use_quantized_tensors_ = false;

  bool use_advanced_gpu_api_ = false;
+  bool allow_precision_loss_ = false;
+  ::mediapipe::TfLiteInferenceCalculatorOptions_Delegate_Gpu_API
+      tflite_gpu_runner_api_;
+
+  bool use_kernel_caching_ = false;
+  std::string cached_kernel_filename_;
 };
 REGISTER_CALCULATOR(TfLiteInferenceCalculator);

@@ -354,6 +368,19 @@ bool ShouldUseGpu(CC* cc) {
                          options.has_delegate() &&
                          options.delegate().has_gpu() &&
                          options.delegate().gpu().use_advanced_gpu_api();
+  allow_precision_loss_ = options.delegate().gpu().allow_precision_loss();
+  tflite_gpu_runner_api_ = options.delegate().gpu().api();
+
+  use_kernel_caching_ =
+      use_advanced_gpu_api_ && options.delegate().gpu().use_kernel_caching();
+
+  if (use_kernel_caching_) {
+#if MEDIAPIPE_TFLITE_GL_INFERENCE && defined(MEDIAPIPE_ANDROID)
+    cached_kernel_filename_ =
+        "/sdcard/" + mediapipe::File::Basename(options.model_path()) + ".ker";
+#endif  // MEDIAPIPE_TFLITE_GL_INFERENCE && MEDIAPIPE_ANDROID
+  }
+
  if (use_advanced_gpu_api_ && !gpu_input_) {
    LOG(WARNING) << "Cannot use advanced GPU APIs, input must be GPU buffers."
                    "Falling back to the default TFLite API.";
@@ -423,7 +450,23 @@ bool ShouldUseGpu(CC* cc) {
  });
 }

+::mediapipe::Status TfLiteInferenceCalculator::WriteKernelsToFile() {
+#if MEDIAPIPE_TFLITE_GL_INFERENCE && defined(MEDIAPIPE_ANDROID)
+  if (use_kernel_caching_) {
+    // Save kernel file.
+    auto kernel_cache = absl::make_unique<std::vector<uint8_t>>(
+        tflite_gpu_runner_->GetSerializedBinaryCache());
+    std::string cache_str(kernel_cache->begin(), kernel_cache->end());
+    MP_RETURN_IF_ERROR(
+        mediapipe::file::SetContents(cached_kernel_filename_, cache_str));
+  }
+#endif  // MEDIAPIPE_TFLITE_GL_INFERENCE && MEDIAPIPE_ANDROID
+  return ::mediapipe::OkStatus();
+}
+
 ::mediapipe::Status TfLiteInferenceCalculator::Close(CalculatorContext* cc) {
+  MP_RETURN_IF_ERROR(WriteKernelsToFile());
+
  return RunInContextIfNeeded([this]() -> ::mediapipe::Status {
    if (delegate_) {
      interpreter_ = nullptr;
@@ -635,6 +678,22 @@ bool ShouldUseGpu(CC* cc) {
  return ::mediapipe::OkStatus();
 }

+::mediapipe::Status TfLiteInferenceCalculator::ReadKernelsFromFile() {
+#if MEDIAPIPE_TFLITE_GL_INFERENCE && defined(MEDIAPIPE_ANDROID)
+  if (use_kernel_caching_) {
+    // Load pre-compiled kernel file.
+    if (mediapipe::File::Exists(cached_kernel_filename_)) {
+      std::string cache_str;
+      MP_RETURN_IF_ERROR(
+          mediapipe::file::GetContents(cached_kernel_filename_, &cache_str));
+      std::vector<uint8_t> cache_vec(cache_str.begin(), cache_str.end());
+      tflite_gpu_runner_->SetSerializedBinaryCache(std::move(cache_vec));
+    }
+  }
+#endif  // MEDIAPIPE_TFLITE_GL_INFERENCE && MEDIAPIPE_ANDROID
+  return ::mediapipe::OkStatus();
+}
+
 ::mediapipe::Status TfLiteInferenceCalculator::InitTFLiteGPURunner(
    CalculatorContext* cc) {
 #if MEDIAPIPE_TFLITE_GL_INFERENCE
@@ -649,11 +708,23 @@ bool ShouldUseGpu(CC* cc) {

  // Create runner
  tflite::gpu::InferenceOptions options;
-  options.priority1 = tflite::gpu::InferencePriority::MIN_LATENCY;
+  options.priority1 = allow_precision_loss_
+                          ? tflite::gpu::InferencePriority::MIN_LATENCY
+                          : tflite::gpu::InferencePriority::MAX_PRECISION;
  options.priority2 = tflite::gpu::InferencePriority::AUTO;
  options.priority3 = tflite::gpu::InferencePriority::AUTO;
  options.usage = tflite::gpu::InferenceUsage::SUSTAINED_SPEED;
  tflite_gpu_runner_ = std::make_unique<tflite::gpu::TFLiteGPURunner>(options);
+  if (tflite_gpu_runner_api_ ==
+      ::mediapipe::TfLiteInferenceCalculatorOptions_Delegate_Gpu_API::
+          TfLiteInferenceCalculatorOptions_Delegate_Gpu_API_OPENGL) {
+    tflite_gpu_runner_->ForceOpenGL();
+  }
+  if (tflite_gpu_runner_api_ ==
+      ::mediapipe::TfLiteInferenceCalculatorOptions_Delegate_Gpu_API::
+          TfLiteInferenceCalculatorOptions_Delegate_Gpu_API_OPENCL) {
+    tflite_gpu_runner_->ForceOpenCL();
+  }
  MP_RETURN_IF_ERROR(
      tflite_gpu_runner_->InitializeWithModel(model, op_resolver));

@@ -692,6 +763,9 @@ bool ShouldUseGpu(CC* cc) {
        ::tflite::gpu::gl::CreateReadWriteShaderStorageBuffer<float>(
            gpu_data_out_[i]->elements, &gpu_data_out_[i]->buffer));
  }
+
+  MP_RETURN_IF_ERROR(ReadKernelsFromFile());
+
  MP_RETURN_IF_ERROR(tflite_gpu_runner_->Build());
 #endif  // MEDIAPIPE_TFLITE_GL_INFERENCE

@@ -48,6 +48,24 @@ message TfLiteInferenceCalculatorOptions {
      // example:
      //   delegate: { gpu { use_advanced_gpu_api: true } }
      optional bool use_advanced_gpu_api = 1 [default = false];
+
+      // This option is valid for TFLite GPU delegate API2 only,
+      // Choose any of available APIs to force running inference using it.
+      enum API {
+        ANY = 0;
+        OPENGL = 1;
+        OPENCL = 2;
+      }
+      optional API api = 4 [default = ANY];
+
+      // This option is valid for TFLite GPU delegate API2 only,
+      // Set to true to use 16-bit float precision. If max precision is needed,
+      // set to false for 32-bit float calculations only.
+      optional bool allow_precision_loss = 3 [default = true];
+
+      // Load pre-compiled serialized binary cache to accelerate init process.
+      // Only available for OpenCL delegate on Android.
+      optional bool use_kernel_caching = 2 [default = false];
    }
    // Android only.
    message Nnapi {}
@@ -21,9 +21,9 @@
 namespace mediapipe {

 // A calculator for converting TFLite tensors from regression models into
-// landmarks. Note that if the landmarks in the tensor has more than 4
-// dimensions, only the first 4 dimensions will be converted to
-// [x,y,z, visibility].
+// landmarks. Note that if the landmarks in the tensor has more than 5
+// dimensions, only the first 5 dimensions will be converted to
+// [x,y,z, visibility, presence].
 //
 // Input:
 //  TENSORS - Vector of TfLiteTensor of type kTfLiteFloat32. Only the first
@@ -209,6 +209,9 @@ REGISTER_CALCULATOR(TfLiteTensorsToLandmarksCalculator);
    if (num_dimensions > 3) {
      landmark->set_visibility(raw_landmarks[offset + 3]);
    }
+    if (num_dimensions > 4) {
+      landmark->set_presence(raw_landmarks[offset + 4]);
+    }
  }

  // Output normalized landmarks if required.
@@ -223,6 +226,7 @@ REGISTER_CALCULATOR(TfLiteTensorsToLandmarksCalculator);
      norm_landmark->set_z(landmark.z() / options_.input_image_width() /
                           options_.normalize_z());
      norm_landmark->set_visibility(landmark.visibility());
+      norm_landmark->set_presence(landmark.presence());
    }
    cc->Outputs()
        .Tag("NORM_LANDMARKS")
@@ -12,14 +12,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
+load("//mediapipe/framework/port:build_config.bzl", "mediapipe_proto_library")

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:public"])

-exports_files(["LICENSE"])
-
 cc_library(
    name = "alignment_points_to_rects_calculator",
    srcs = ["alignment_points_to_rects_calculator.cc"],
@@ -38,186 +36,95 @@ cc_library(
    alwayslink = 1,
 )

-proto_library(
+mediapipe_proto_library(
    name = "annotation_overlay_calculator_proto",
    srcs = ["annotation_overlay_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "detection_label_id_to_text_calculator_proto",
    srcs = ["detection_label_id_to_text_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "timed_box_list_id_to_label_calculator_proto",
    srcs = ["timed_box_list_id_to_label_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "latency_proto",
    srcs = ["latency.proto"],
 )

-proto_library(
+mediapipe_proto_library(
    name = "non_max_suppression_calculator_proto",
    srcs = ["non_max_suppression_calculator.proto"],
    visibility = ["//visibility:public"],
-    deps = ["//mediapipe/framework:calculator_proto"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "packet_frequency_proto",
    srcs = ["packet_frequency.proto"],
 )

-proto_library(
+mediapipe_proto_library(
    name = "packet_frequency_calculator_proto",
    srcs = ["packet_frequency_calculator.proto"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "packet_latency_calculator_proto",
    srcs = ["packet_latency_calculator.proto"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "collection_has_min_size_calculator_proto",
    srcs = ["collection_has_min_size_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "association_calculator_proto",
    srcs = ["association_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-mediapipe_cc_proto_library(
-    name = "annotation_overlay_calculator_cc_proto",
-    srcs = ["annotation_overlay_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":annotation_overlay_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "detection_label_id_to_text_calculator_cc_proto",
-    srcs = ["detection_label_id_to_text_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [
-        ":detection_label_id_to_text_calculator_proto",
-    ],
-)
-
-mediapipe_cc_proto_library(
-    name = "timed_box_list_id_to_label_calculator_cc_proto",
-    srcs = ["timed_box_list_id_to_label_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [
-        ":timed_box_list_id_to_label_calculator_proto",
-    ],
-)
-
-mediapipe_cc_proto_library(
-    name = "latency_cc_proto",
-    srcs = ["latency.proto"],
-    visibility = ["//mediapipe:__subpackages__"],
-    deps = [":latency_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "non_max_suppression_calculator_cc_proto",
-    srcs = ["non_max_suppression_calculator.proto"],
-    cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
-    visibility = ["//visibility:public"],
-    deps = [":non_max_suppression_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "packet_frequency_cc_proto",
-    srcs = ["packet_frequency.proto"],
-    visibility = ["//mediapipe:__subpackages__"],
-    deps = [":packet_frequency_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "packet_frequency_calculator_cc_proto",
-    srcs = ["packet_frequency_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//mediapipe:__subpackages__"],
-    deps = [
-        ":packet_frequency_calculator_proto",
-    ],
-)
-
-mediapipe_cc_proto_library(
-    name = "packet_latency_calculator_cc_proto",
-    srcs = ["packet_latency_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//mediapipe:__subpackages__"],
-    deps = [
-        ":packet_latency_calculator_proto",
-    ],
-)
-
-mediapipe_cc_proto_library(
-    name = "collection_has_min_size_calculator_cc_proto",
-    srcs = ["collection_has_min_size_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//mediapipe:__subpackages__"],
-    deps = [":collection_has_min_size_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "association_calculator_cc_proto",
-    srcs = ["association_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//mediapipe:__subpackages__"],
-    deps = [":association_calculator_proto"],
-)
-
 cc_library(
    name = "packet_frequency_calculator",
    srcs = ["packet_frequency_calculator.cc"],
@@ -341,9 +248,11 @@ cc_library(
        "@com_google_absl//absl/strings",
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/formats:image_frame",
+        "//mediapipe/framework/formats:image_frame_opencv",
        "//mediapipe/framework/formats:video_stream_header",
        "//mediapipe/framework/port:logging",
        "//mediapipe/framework/port:opencv_core",
+        "//mediapipe/framework/port:opencv_imgproc",
        "//mediapipe/framework/port:status",
        "//mediapipe/framework/port:vector",
        "//mediapipe/util:annotation_renderer",
@@ -367,6 +276,7 @@ cc_library(
    deps = [
        ":detection_label_id_to_text_calculator_cc_proto",
        "//mediapipe/framework/formats:detection_cc_proto",
+        "@com_google_absl//absl/container:node_hash_map",
        "//mediapipe/framework/port:status",
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework:packet",
@@ -434,16 +344,6 @@ cc_library(
    alwayslink = 1,
 )

-mediapipe_cc_proto_library(
-    name = "thresholding_calculator_cc_proto",
-    srcs = ["thresholding_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":thresholding_calculator_proto"],
-)
-
 cc_library(
    name = "thresholding_calculator",
    srcs = ["thresholding_calculator.cc"],
@@ -457,16 +357,6 @@ cc_library(
    alwayslink = 1,
 )

-mediapipe_cc_proto_library(
-    name = "landmarks_to_detection_calculator_cc_proto",
-    srcs = ["landmarks_to_detection_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":landmarks_to_detection_calculator_proto"],
-)
-
 cc_library(
    name = "landmarks_to_detection_calculator",
    srcs = ["landmarks_to_detection_calculator.cc"],
@@ -483,46 +373,6 @@ cc_library(
    alwayslink = 1,
 )

-mediapipe_cc_proto_library(
-    name = "detections_to_rects_calculator_cc_proto",
-    srcs = ["detections_to_rects_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":detections_to_rects_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "landmark_projection_calculator_cc_proto",
-    srcs = ["landmark_projection_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":landmark_projection_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "landmarks_to_floats_calculator_cc_proto",
-    srcs = ["landmarks_to_floats_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":landmarks_to_floats_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "rect_transformation_calculator_cc_proto",
-    srcs = ["rect_transformation_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":rect_transformation_calculator_proto"],
-)
-
 cc_library(
    name = "detections_to_rects_calculator",
    srcs = [
@@ -593,162 +443,140 @@ cc_test(
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "rect_to_render_data_calculator_proto",
    srcs = ["rect_to_render_data_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "rect_to_render_scale_calculator_proto",
    srcs = ["rect_to_render_scale_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "detections_to_render_data_calculator_proto",
    srcs = ["detections_to_render_data_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "landmarks_to_render_data_calculator_proto",
    srcs = ["landmarks_to_render_data_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "timed_box_list_to_render_data_calculator_proto",
    srcs = ["timed_box_list_to_render_data_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "labels_to_render_data_calculator_proto",
    srcs = ["labels_to_render_data_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "thresholding_calculator_proto",
    srcs = ["thresholding_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "detections_to_rects_calculator_proto",
    srcs = ["detections_to_rects_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "landmark_projection_calculator_proto",
    srcs = ["landmark_projection_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "landmarks_to_floats_calculator_proto",
    srcs = ["landmarks_to_floats_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "rect_transformation_calculator_proto",
    srcs = ["rect_transformation_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "landmarks_to_detection_calculator_proto",
    srcs = ["landmarks_to_detection_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
        "//mediapipe/util:color_proto",
        "//mediapipe/util:render_data_proto",
    ],
 )

-mediapipe_cc_proto_library(
-    name = "rect_to_render_data_calculator_cc_proto",
-    srcs = ["rect_to_render_data_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-        "//mediapipe/util:render_data_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":rect_to_render_data_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "rect_to_render_scale_calculator_cc_proto",
-    srcs = ["rect_to_render_scale_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":rect_to_render_scale_calculator_proto"],
-)
-
-mediapipe_cc_proto_library(
-    name = "detections_to_render_data_calculator_cc_proto",
-    srcs = ["detections_to_render_data_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-        "//mediapipe/util:render_data_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":detections_to_render_data_calculator_proto"],
-)
-
 cc_library(
    name = "detections_to_render_data_calculator",
    srcs = ["detections_to_render_data_calculator.cc"],
@@ -768,21 +596,10 @@ cc_library(
    alwayslink = 1,
 )

-mediapipe_cc_proto_library(
-    name = "landmarks_to_render_data_calculator_cc_proto",
-    srcs = ["landmarks_to_render_data_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-        "//mediapipe/util:render_data_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":landmarks_to_render_data_calculator_proto"],
-)
-
 cc_library(
    name = "landmarks_to_render_data_calculator",
    srcs = ["landmarks_to_render_data_calculator.cc"],
+    hdrs = ["landmarks_to_render_data_calculator.h"],
    visibility = ["//visibility:public"],
    deps = [
        ":landmarks_to_render_data_calculator_cc_proto",
@@ -799,18 +616,6 @@ cc_library(
    alwayslink = 1,
 )

-mediapipe_cc_proto_library(
-    name = "timed_box_list_to_render_data_calculator_cc_proto",
-    srcs = ["timed_box_list_to_render_data_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-        "//mediapipe/util:render_data_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":timed_box_list_to_render_data_calculator_proto"],
-)
-
 cc_library(
    name = "timed_box_list_to_render_data_calculator",
    srcs = ["timed_box_list_to_render_data_calculator.cc"],
@@ -944,6 +749,32 @@ cc_library(
    alwayslink = 1,
 )

+mediapipe_proto_library(
+    name = "landmarks_smoothing_calculator_proto",
+    srcs = ["landmarks_smoothing_calculator.proto"],
+    visibility = ["//visibility:public"],
+    deps = [
+        "//mediapipe/framework:calculator_options_proto",
+        "//mediapipe/framework:calculator_proto",
+    ],
+)
+
+cc_library(
+    name = "landmarks_smoothing_calculator",
+    srcs = ["landmarks_smoothing_calculator.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":landmarks_smoothing_calculator_cc_proto",
+        "//mediapipe/framework:calculator_framework",
+        "//mediapipe/framework:timestamp",
+        "//mediapipe/framework/formats:landmark_cc_proto",
+        "//mediapipe/framework/port:ret_check",
+        "//mediapipe/util/filtering:relative_velocity_filter",
+        "@com_google_absl//absl/algorithm:container",
+    ],
+    alwayslink = 1,
+)
+
 cc_library(
    name = "landmarks_to_floats_calculator",
    srcs = ["landmarks_to_floats_calculator.cc"],
@@ -991,25 +822,16 @@ cc_test(
    ],
 )

-proto_library(
+mediapipe_proto_library(
    name = "top_k_scores_calculator_proto",
    srcs = ["top_k_scores_calculator.proto"],
    visibility = ["//visibility:public"],
    deps = [
+        "//mediapipe/framework:calculator_options_proto",
        "//mediapipe/framework:calculator_proto",
    ],
 )

-mediapipe_cc_proto_library(
-    name = "top_k_scores_calculator_cc_proto",
-    srcs = ["top_k_scores_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":top_k_scores_calculator_proto"],
-)
-
 cc_library(
    name = "top_k_scores_calculator",
    srcs = ["top_k_scores_calculator.cc"],
@@ -1055,17 +877,6 @@ cc_test(
    ],
 )

-mediapipe_cc_proto_library(
-    name = "labels_to_render_data_calculator_cc_proto",
-    srcs = ["labels_to_render_data_calculator.proto"],
-    cc_deps = [
-        "//mediapipe/framework:calculator_cc_proto",
-        "//mediapipe/util:color_cc_proto",
-    ],
-    visibility = ["//visibility:public"],
-    deps = [":labels_to_render_data_calculator_proto"],
-)
-
 cc_library(
    name = "local_file_contents_calculator",
    srcs = ["local_file_contents_calculator.cc"],
@@ -1102,6 +913,7 @@ cc_library(
        "//mediapipe/framework/formats:classification_cc_proto",
        "//mediapipe/framework/formats:landmark_cc_proto",
        "//mediapipe/framework/formats:rect_cc_proto",
+        "//mediapipe/framework/port:integral_types",
        "//mediapipe/framework/port:ret_check",
        "//mediapipe/framework/port:status",
        "@com_google_absl//absl/strings",
@@ -20,9 +20,11 @@
 #include "mediapipe/framework/calculator_options.pb.h"
 #include "mediapipe/framework/formats/image_format.pb.h"
 #include "mediapipe/framework/formats/image_frame.h"
+#include "mediapipe/framework/formats/image_frame_opencv.h"
 #include "mediapipe/framework/formats/video_stream_header.h"
 #include "mediapipe/framework/port/logging.h"
 #include "mediapipe/framework/port/opencv_core_inc.h"
+#include "mediapipe/framework/port/opencv_imgproc_inc.h"
 #include "mediapipe/framework/port/status.h"
 #include "mediapipe/framework/port/vector.h"
 #include "mediapipe/util/annotation_renderer.h"
@@ -40,13 +42,9 @@ namespace mediapipe {

 namespace {

-constexpr char kInputFrameTag[] = "IMAGE";
-constexpr char kOutputFrameTag[] = "IMAGE";
-
-constexpr char kInputVectorTag[] = "VECTOR";
-
-constexpr char kInputFrameTagGpu[] = "IMAGE_GPU";
-constexpr char kOutputFrameTagGpu[] = "IMAGE_GPU";
+constexpr char kVectorTag[] = "VECTOR";
+constexpr char kGpuBufferTag[] = "IMAGE_GPU";
+constexpr char kImageFrameTag[] = "IMAGE";

 enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };

@@ -57,12 +55,15 @@ size_t RoundUp(size_t n, size_t m) { return ((n + m - 1) / m) * m; }  // NOLINT
 // merges the annotation overlay with the image frame. As a result, drawing in
 // this color is not supported and it should be set to something unlikely used.
 constexpr uchar kAnnotationBackgroundColor = 2;  // Grayscale value.
+
+// Future Image type.
+inline bool HasImageTag(mediapipe::CalculatorContext* cc) { return false; }
 }  // namespace

 // A calculator for rendering data on images.
 //
 // Inputs:
-//  1. IMAGE or IMAGE_GPU (optional): An ImageFrame (or GpuBuffer)
+//  1. IMAGE or IMAGE_GPU (optional): An ImageFrame (or GpuBuffer),
 //     containing the input image.
 //     If output is CPU, and input isn't provided, the renderer creates a
 //     blank canvas with the width, height and color provided in the options.
@@ -74,7 +75,8 @@ constexpr uchar kAnnotationBackgroundColor = 2;  // Grayscale value.
 //     input vector items. These input streams are tagged with "VECTOR".
 //
 // Output:
-//  1. IMAGE or IMAGE_GPU: A rendered ImageFrame (or GpuBuffer).
+//  1. IMAGE or IMAGE_GPU: A rendered ImageFrame (or GpuBuffer),
+//  Note: Output types should match their corresponding input stream type.
 //
 // For CPU input frames, only SRGBA, SRGB and GRAY8 format are supported. The
 // output format is the same as input except for GRAY8 where the output is in
@@ -133,14 +135,17 @@ class AnnotationOverlayCalculator : public CalculatorBase {
  ::mediapipe::Status CreateRenderTargetCpu(CalculatorContext* cc,
                                            std::unique_ptr<cv::Mat>& image_mat,
                                            ImageFormat::Format* target_format);
+  template <typename Type, const char* Tag>
  ::mediapipe::Status CreateRenderTargetGpu(
      CalculatorContext* cc, std::unique_ptr<cv::Mat>& image_mat);
+  template <typename Type, const char* Tag>
  ::mediapipe::Status RenderToGpu(CalculatorContext* cc, uchar* overlay_image);
  ::mediapipe::Status RenderToCpu(CalculatorContext* cc,
                                  const ImageFormat::Format& target_format,
                                  uchar* data_image);

  ::mediapipe::Status GlRender(CalculatorContext* cc);
+  template <typename Type, const char* Tag>
  ::mediapipe::Status GlSetup(CalculatorContext* cc);

  // Options for the calculator.
@@ -172,24 +177,26 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);

  bool use_gpu = false;

-  if (cc->Inputs().HasTag(kInputFrameTag) &&
-      cc->Inputs().HasTag(kInputFrameTagGpu)) {
+  if (cc->Inputs().HasTag(kImageFrameTag) &&
+      cc->Inputs().HasTag(kGpuBufferTag)) {
    return ::mediapipe::InternalError("Cannot have multiple input images.");
  }
-  if (cc->Inputs().HasTag(kInputFrameTagGpu) !=
-      cc->Outputs().HasTag(kOutputFrameTagGpu)) {
+  if (cc->Inputs().HasTag(kGpuBufferTag) !=
+      cc->Outputs().HasTag(kGpuBufferTag)) {
    return ::mediapipe::InternalError("GPU output must have GPU input.");
  }

-  // Input image to render onto copy of.
+  // Input image to render onto copy of. Should be same type as output.
 #if !defined(MEDIAPIPE_DISABLE_GPU)
-  if (cc->Inputs().HasTag(kInputFrameTagGpu)) {
-    cc->Inputs().Tag(kInputFrameTagGpu).Set<mediapipe::GpuBuffer>();
-    use_gpu |= true;
+  if (cc->Inputs().HasTag(kGpuBufferTag)) {
+    cc->Inputs().Tag(kGpuBufferTag).Set<mediapipe::GpuBuffer>();
+    CHECK(cc->Outputs().HasTag(kGpuBufferTag));
+    use_gpu = true;
  }
 #endif  //  !MEDIAPIPE_DISABLE_GPU
-  if (cc->Inputs().HasTag(kInputFrameTag)) {
-    cc->Inputs().Tag(kInputFrameTag).Set<ImageFrame>();
+  if (cc->Inputs().HasTag(kImageFrameTag)) {
+    cc->Inputs().Tag(kImageFrameTag).Set<ImageFrame>();
+    CHECK(cc->Outputs().HasTag(kImageFrameTag));
  }

  // Data streams to render.
@@ -197,7 +204,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
       ++id) {
    auto tag_and_index = cc->Inputs().TagAndIndexFromId(id);
    std::string tag = tag_and_index.first;
-    if (tag == kInputVectorTag) {
+    if (tag == kVectorTag) {
      cc->Inputs().Get(id).Set<std::vector<RenderData>>();
    } else if (tag.empty()) {
      // Empty tag defaults to accepting a single object of RenderData type.
@@ -205,15 +212,14 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
    }
  }

-  // Rendered image.
+  // Rendered image. Should be same type as input.
 #if !defined(MEDIAPIPE_DISABLE_GPU)
-  if (cc->Outputs().HasTag(kOutputFrameTagGpu)) {
-    cc->Outputs().Tag(kOutputFrameTagGpu).Set<mediapipe::GpuBuffer>();
-    use_gpu |= true;
+  if (cc->Outputs().HasTag(kGpuBufferTag)) {
+    cc->Outputs().Tag(kGpuBufferTag).Set<mediapipe::GpuBuffer>();
  }
 #endif  //  !MEDIAPIPE_DISABLE_GPU
-  if (cc->Outputs().HasTag(kOutputFrameTag)) {
-    cc->Outputs().Tag(kOutputFrameTag).Set<ImageFrame>();
+  if (cc->Outputs().HasTag(kImageFrameTag)) {
+    cc->Outputs().Tag(kImageFrameTag).Set<ImageFrame>();
  }

  if (use_gpu) {
@@ -229,20 +235,16 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
  cc->SetOffset(TimestampDiff(0));

  options_ = cc->Options<AnnotationOverlayCalculatorOptions>();
-  if (cc->Inputs().HasTag(kInputFrameTagGpu) &&
-      cc->Outputs().HasTag(kOutputFrameTagGpu)) {
+  if (cc->Inputs().HasTag(kGpuBufferTag) || HasImageTag(cc)) {
 #if !defined(MEDIAPIPE_DISABLE_GPU)
    use_gpu_ = true;
-#else
-    RET_CHECK_FAIL() << "GPU processing not enabled.";
 #endif  //  !MEDIAPIPE_DISABLE_GPU
  }

-  if (cc->Inputs().HasTag(kInputFrameTagGpu) ||
-      cc->Inputs().HasTag(kInputFrameTag)) {
+  if (cc->Inputs().HasTag(kGpuBufferTag) ||
+      cc->Inputs().HasTag(kImageFrameTag) || HasImageTag(cc)) {
    image_frame_available_ = true;
  } else {
-    image_frame_available_ = false;
    RET_CHECK(options_.has_canvas_width_px());
    RET_CHECK(options_.has_canvas_height_px());
  }
@@ -253,14 +255,12 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
  if (use_gpu_) renderer_->SetScaleFactor(options_.gpu_scale_factor());

  // Set the output header based on the input header (if present).
-  const char* input_tag = use_gpu_ ? kInputFrameTagGpu : kInputFrameTag;
-  const char* output_tag = use_gpu_ ? kOutputFrameTagGpu : kOutputFrameTag;
-  if (image_frame_available_ &&
-      !cc->Inputs().Tag(input_tag).Header().IsEmpty()) {
+  const char* tag = use_gpu_ ? kGpuBufferTag : kImageFrameTag;
+  if (image_frame_available_ && !cc->Inputs().Tag(tag).Header().IsEmpty()) {
    const auto& input_header =
-        cc->Inputs().Tag(input_tag).Header().Get<VideoHeader>();
+        cc->Inputs().Tag(tag).Header().Get<VideoHeader>();
    auto* output_video_header = new VideoHeader(input_header);
-    cc->Outputs().Tag(output_tag).SetHeader(Adopt(output_video_header));
+    cc->Outputs().Tag(tag).SetHeader(Adopt(output_video_header));
  }

  if (use_gpu_) {
@@ -282,15 +282,20 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
    if (!gpu_initialized_) {
      MP_RETURN_IF_ERROR(
          gpu_helper_.RunInGlContext([this, cc]() -> ::mediapipe::Status {
-            MP_RETURN_IF_ERROR(GlSetup(cc));
-            return ::mediapipe::OkStatus();
+            return GlSetup<mediapipe::GpuBuffer, kGpuBufferTag>(cc);
          }));
      gpu_initialized_ = true;
    }
+    if (cc->Inputs().HasTag(kGpuBufferTag)) {
+      MP_RETURN_IF_ERROR(
+          (CreateRenderTargetGpu<mediapipe::GpuBuffer, kGpuBufferTag>(
+              cc, image_mat)));
+    }
 #endif  //  !MEDIAPIPE_DISABLE_GPU
-    MP_RETURN_IF_ERROR(CreateRenderTargetGpu(cc, image_mat));
  } else {
-    MP_RETURN_IF_ERROR(CreateRenderTargetCpu(cc, image_mat, &target_format));
+    if (cc->Inputs().HasTag(kImageFrameTag)) {
+      MP_RETURN_IF_ERROR(CreateRenderTargetCpu(cc, image_mat, &target_format));
+    }
  }

  // Reset the renderer with the image_mat. No copy here.
@@ -301,7 +306,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
       ++id) {
    auto tag_and_index = cc->Inputs().TagAndIndexFromId(id);
    std::string tag = tag_and_index.first;
-    if (!tag.empty() && tag != kInputVectorTag) {
+    if (!tag.empty() && tag != kVectorTag) {
      continue;
    }
    if (cc->Inputs().Get(id).IsEmpty()) {
@@ -312,7 +317,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
      const RenderData& render_data = cc->Inputs().Get(id).Get<RenderData>();
      renderer_->RenderDataOnImage(render_data);
    } else {
-      RET_CHECK_EQ(kInputVectorTag, tag);
+      RET_CHECK_EQ(kVectorTag, tag);
      const std::vector<RenderData>& render_data_vec =
          cc->Inputs().Get(id).Get<std::vector<RenderData>>();
      for (const RenderData& render_data : render_data_vec) {
@@ -327,8 +332,8 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
    uchar* image_mat_ptr = image_mat->data;
    MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext(
        [this, cc, image_mat_ptr]() -> ::mediapipe::Status {
-          MP_RETURN_IF_ERROR(RenderToGpu(cc, image_mat_ptr));
-          return ::mediapipe::OkStatus();
+          return RenderToGpu<mediapipe::GpuBuffer, kGpuBufferTag>(
+              cc, image_mat_ptr);
        }));
 #endif  //  !MEDIAPIPE_DISABLE_GPU
  } else {
@@ -369,19 +374,21 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
                              ImageFrame::kDefaultAlignmentBoundary);
 #endif  //  !MEDIAPIPE_DISABLE_GPU

-  cc->Outputs()
-      .Tag(kOutputFrameTag)
-      .Add(output_frame.release(), cc->InputTimestamp());
+  if (cc->Outputs().HasTag(kImageFrameTag)) {
+    cc->Outputs()
+        .Tag(kImageFrameTag)
+        .Add(output_frame.release(), cc->InputTimestamp());
+  }

  return ::mediapipe::OkStatus();
 }

+template <typename Type, const char* Tag>
 ::mediapipe::Status AnnotationOverlayCalculator::RenderToGpu(
    CalculatorContext* cc, uchar* overlay_image) {
 #if !defined(MEDIAPIPE_DISABLE_GPU)
  // Source and destination textures.
-  const auto& input_frame =
-      cc->Inputs().Tag(kInputFrameTagGpu).Get<mediapipe::GpuBuffer>();
+  const auto& input_frame = cc->Inputs().Tag(Tag).Get<Type>();
  auto input_texture = gpu_helper_.CreateSourceTexture(input_frame);

  auto output_texture = gpu_helper_.CreateDestinationTexture(
@@ -389,8 +396,6 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);

  // Upload render target to GPU.
  {
-    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
-
    glBindTexture(GL_TEXTURE_2D, image_mat_tex_);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, width_canvas_, height_canvas_,
                    GL_RGB, GL_UNSIGNED_BYTE, overlay_image);
@@ -416,10 +421,8 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
  }

  // Send out blended image as GPU packet.
-  auto output_frame = output_texture.GetFrame<mediapipe::GpuBuffer>();
-  cc->Outputs()
-      .Tag(kOutputFrameTagGpu)
-      .Add(output_frame.release(), cc->InputTimestamp());
+  auto output_frame = output_texture.GetFrame<Type>();
+  cc->Outputs().Tag(Tag).Add(output_frame.release(), cc->InputTimestamp());

  // Cleanup
  input_texture.Release();
@@ -434,7 +437,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
    ImageFormat::Format* target_format) {
  if (image_frame_available_) {
    const auto& input_frame =
-        cc->Inputs().Tag(kInputFrameTag).Get<ImageFrame>();
+        cc->Inputs().Tag(kImageFrameTag).Get<ImageFrame>();

    int target_mat_type;
    switch (input_frame.Format()) {
@@ -457,39 +460,32 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);

    image_mat = absl::make_unique<cv::Mat>(
        input_frame.Height(), input_frame.Width(), target_mat_type);
+
+    auto input_mat = formats::MatView(&input_frame);
    if (input_frame.Format() == ImageFormat::GRAY8) {
-      const int target_num_channels =
-          ImageFrame::NumberOfChannelsForFormat(*target_format);
-      for (int i = 0; i < input_frame.PixelDataSize(); i++) {
-        const auto& pix = input_frame.PixelData()[i];
-        for (int c = 0; c < target_num_channels; c++) {
-          image_mat->data[i * target_num_channels + c] = pix;
-        }
-      }
+      cv::Mat rgb_mat;
+      cv::cvtColor(input_mat, rgb_mat, CV_GRAY2RGB);
+      rgb_mat.copyTo(*image_mat);
    } else {
-      // Make of a copy since the input frame may be consumed by other nodes.
-      const int buffer_size =
-          input_frame.Height() * input_frame.Width() *
-          ImageFrame::NumberOfChannelsForFormat(*target_format);
-      input_frame.CopyToBuffer(image_mat->data, buffer_size);
+      input_mat.copyTo(*image_mat);
    }
  } else {
    image_mat = absl::make_unique<cv::Mat>(
        options_.canvas_height_px(), options_.canvas_width_px(), CV_8UC3,
        cv::Scalar(options_.canvas_color().r(), options_.canvas_color().g(),
                   options_.canvas_color().b()));
+    *target_format = ImageFormat::SRGB;
  }

  return ::mediapipe::OkStatus();
 }

+template <typename Type, const char* Tag>
 ::mediapipe::Status AnnotationOverlayCalculator::CreateRenderTargetGpu(
    CalculatorContext* cc, std::unique_ptr<cv::Mat>& image_mat) {
 #if !defined(MEDIAPIPE_DISABLE_GPU)
  if (image_frame_available_) {
-    const auto& input_frame =
-        cc->Inputs().Tag(kInputFrameTagGpu).Get<mediapipe::GpuBuffer>();
-
+    const auto& input_frame = cc->Inputs().Tag(Tag).Get<Type>();
    const mediapipe::ImageFormat::Format format =
        mediapipe::ImageFormatForGpuBufferFormat(input_frame.format());
    if (format != mediapipe::ImageFormat::SRGBA &&
@@ -565,6 +561,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
  return ::mediapipe::OkStatus();
 }

+template <typename Type, const char* Tag>
 ::mediapipe::Status AnnotationOverlayCalculator::GlSetup(
    CalculatorContext* cc) {
 #if !defined(MEDIAPIPE_DISABLE_GPU)
@@ -640,8 +637,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
  const float alignment = ImageFrame::kGlDefaultAlignmentBoundary;
  const float scale_factor = options_.gpu_scale_factor();
  if (image_frame_available_) {
-    const auto& input_frame =
-        cc->Inputs().Tag(kInputFrameTagGpu).Get<mediapipe::GpuBuffer>();
+    const auto& input_frame = cc->Inputs().Tag(Tag).Get<Type>();
    width_ = RoundUp(input_frame.width(), alignment);
    height_ = RoundUp(input_frame.height(), alignment);
  } else {
@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+#include "absl/container/node_hash_map.h"
 #include "mediapipe/calculators/util/detection_label_id_to_text_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/formats/detection.pb.h"
@@ -52,7 +53,7 @@ class DetectionLabelIdToTextCalculator : public CalculatorBase {
  ::mediapipe::Status Process(CalculatorContext* cc) override;

 private:
-  std::unordered_map<int, std::string> label_map_;
+  absl::node_hash_map<int, std::string> label_map_;
 };
 REGISTER_CALCULATOR(DetectionLabelIdToTextCalculator);

@@ -20,9 +20,14 @@
 #include "mediapipe/framework/formats/classification.pb.h"
 #include "mediapipe/framework/formats/landmark.pb.h"
 #include "mediapipe/framework/formats/rect.pb.h"
+#include "mediapipe/framework/port/integral_types.h"

 namespace mediapipe {

+typedef FilterCollectionCalculator<std::vector<uint64>>
+    FilterUInt64CollectionCalculator;
+REGISTER_CALCULATOR(FilterUInt64CollectionCalculator);
+
 typedef FilterCollectionCalculator<std::vector<::mediapipe::NormalizedRect>>
    FilterNormalizedRectCollectionCalculator;
 REGISTER_CALCULATOR(FilterNormalizedRectCollectionCalculator);
@@ -131,6 +131,8 @@ class LandmarkLetterboxRemovalCalculator : public CalculatorBase {
        new_landmark->set_z(new_z);
        // Keep visibility as is.
        new_landmark->set_visibility(landmark.visibility());
+        // Keep presence as is.
+        new_landmark->set_presence(landmark.presence());
      }

      cc->Outputs().Get(output_id).AddPacket(
@@ -131,6 +131,8 @@ class LandmarkProjectionCalculator : public CalculatorBase {
        new_landmark->set_z(new_z);
        // Keep visibility as is.
        new_landmark->set_visibility(landmark.visibility());
+        // Keep presence as is.
+        new_landmark->set_presence(landmark.presence());
      }

      cc->Outputs().Get(output_id).AddPacket(
@@ -13,12 +13,12 @@
 // limitations under the License.

 #include "absl/algorithm/container.h"
+#include "mediapipe/calculators/util/landmarks_smoothing_calculator.pb.h"
 #include "mediapipe/framework/calculator_framework.h"
 #include "mediapipe/framework/formats/landmark.pb.h"
 #include "mediapipe/framework/port/ret_check.h"
 #include "mediapipe/framework/timestamp.h"
-#include "mediapipe/graphs/pose_tracking/calculators/landmarks_smoothing_calculator.pb.h"
-#include "mediapipe/graphs/pose_tracking/calculators/relative_velocity_filter.h"
+#include "mediapipe/util/filtering/relative_velocity_filter.h"

 namespace mediapipe {

@@ -38,17 +38,17 @@ using ::mediapipe::RelativeVelocityFilter;
 // with sides parallel to axis.
 float GetObjectScale(const NormalizedLandmarkList& landmarks, int image_width,
                     int image_height) {
-  const auto& [lm_min_x, lm_max_x] = absl::c_minmax_element(
+  const auto& lm_minmax_x = absl::c_minmax_element(
      landmarks.landmark(),
      [](const auto& a, const auto& b) { return a.x() < b.x(); });
-  const float x_min = lm_min_x->x();
-  const float x_max = lm_max_x->x();
+  const float x_min = lm_minmax_x.first->x();
+  const float x_max = lm_minmax_x.second->x();

-  const auto& [lm_min_y, lm_max_y] = absl::c_minmax_element(
+  const auto& lm_minmax_y = absl::c_minmax_element(
      landmarks.landmark(),
      [](const auto& a, const auto& b) { return a.y() < b.y(); });
-  const float y_min = lm_min_y->y();
-  const float y_max = lm_max_y->y();
+  const float y_min = lm_minmax_y.first->y();
+  const float y_max = lm_minmax_y.second->y();

  const float object_width = (x_max - x_min) * image_width;
  const float object_height = (y_max - y_min) * image_height;
@@ -137,6 +137,8 @@ class VelocityFilter : public LandmarksFilter {
                          image_width);
      // Keep visibility as is.
      out_landmark->set_visibility(in_landmark.visibility());
+      // Keep presence as is.
+      out_landmark->set_presence(in_landmark.presence());
    }

    return ::mediapipe::OkStatus();
@@ -191,8 +193,8 @@ class VelocityFilter : public LandmarksFilter {
 //     input_stream: "NORM_LANDMARKS:pose_landmarks"
 //     input_stream: "IMAGE_SIZE:image_size"
 //     output_stream: "NORM_FILTERED_LANDMARKS:pose_landmarks_filtered"
-//     node_options: {
-//       [type.googleapis.com/mediapipe.LandmarksSmoothingCalculatorOptions] {
+//     options: {
+//       [mediapipe.LandmarksSmoothingCalculatorOptions.ext] {
 //         velocity_filter: {
 //           window_size: 5
 //           velocity_scale: 10.0
@@ -16,7 +16,7 @@ syntax = "proto2";

 package mediapipe;

-import "mediapipe/framework/calculator.proto";
+import "mediapipe/framework/calculator_options.proto";

 message LandmarksSmoothingCalculatorOptions {
  extend CalculatorOptions {
@@ -11,6 +11,7 @@
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
+#include "mediapipe/calculators/util/landmarks_to_render_data_calculator.h"

 #include "absl/memory/memory.h"
 #include "absl/strings/str_cat.h"
@@ -32,9 +33,6 @@ constexpr char kNormLandmarksTag[] = "NORM_LANDMARKS";
 constexpr char kRenderScaleTag[] = "RENDER_SCALE";
 constexpr char kRenderDataTag[] = "RENDER_DATA";
 constexpr char kLandmarkLabel[] = "KEYPOINT";
-constexpr int kMaxLandmarkThickness = 18;
-
-using ::mediapipe::RenderAnnotation_Point;

 inline void SetColor(RenderAnnotation* annotation, const Color& color) {
  annotation->mutable_color()->set_r(color.r());
@@ -60,15 +58,18 @@ inline void GetMinMaxZ(const LandmarkListType& landmarks, float* z_min,
 }

 void SetColorSizeValueFromZ(float z, float z_min, float z_max,
-                            RenderAnnotation* render_annotation) {
+                            RenderAnnotation* render_annotation,
+                            float min_depth_circle_thickness,
+                            float max_depth_circle_thickness) {
  const int color_value = 255 - static_cast<int>(Remap(z, z_min, z_max, 255));
  ::mediapipe::Color color;
  color.set_r(color_value);
  color.set_g(color_value);
  color.set_b(color_value);
  SetColor(render_annotation, color);
-  const int thickness = static_cast<int>((1.f - Remap(z, z_min, z_max, 1)) *
-                                         kMaxLandmarkThickness);
+  const float scale = max_depth_circle_thickness - min_depth_circle_thickness;
+  const int thickness = static_cast<int>(
+      min_depth_circle_thickness + (1.f - Remap(z, z_min, z_max, 1)) * scale);
  render_annotation->set_thickness(thickness);
 }

@@ -98,14 +99,19 @@ template <class LandmarkListType, class LandmarkType>
 void AddConnectionsWithDepth(const LandmarkListType& landmarks,
                             const std::vector<int>& landmark_connections,
                             bool utilize_visibility,
-                             float visibility_threshold, float thickness,
+                             float visibility_threshold, bool utilize_presence,
+                             float presence_threshold, float thickness,
                             bool normalized, float min_z, float max_z,
                             RenderData* render_data) {
  for (int i = 0; i < landmark_connections.size(); i += 2) {
    const auto& ld0 = landmarks.landmark(landmark_connections[i]);
    const auto& ld1 = landmarks.landmark(landmark_connections[i + 1]);
-    if (visibility_threshold && (ld0.visibility() < visibility_threshold ||
-                                 ld1.visibility() < visibility_threshold)) {
+    if (utilize_visibility && (ld0.visibility() < visibility_threshold ||
+                               ld1.visibility() < visibility_threshold)) {
+      continue;
+    }
+    if (utilize_presence && (ld0.presence() < presence_threshold ||
+                             ld1.presence() < presence_threshold)) {
      continue;
    }
    const int gray_val1 =
@@ -137,13 +143,18 @@ template <class LandmarkListType, class LandmarkType>
 void AddConnections(const LandmarkListType& landmarks,
                    const std::vector<int>& landmark_connections,
                    bool utilize_visibility, float visibility_threshold,
+                    bool utilize_presence, float presence_threshold,
                    const Color& connection_color, float thickness,
                    bool normalized, RenderData* render_data) {
  for (int i = 0; i < landmark_connections.size(); i += 2) {
    const auto& ld0 = landmarks.landmark(landmark_connections[i]);
    const auto& ld1 = landmarks.landmark(landmark_connections[i + 1]);
-    if (visibility_threshold && (ld0.visibility() < visibility_threshold ||
-                                 ld1.visibility() < visibility_threshold)) {
+    if (utilize_visibility && (ld0.visibility() < visibility_threshold ||
+                               ld1.visibility() < visibility_threshold)) {
+      continue;
+    }
+    if (utilize_presence && (ld0.presence() < presence_threshold ||
+                             ld1.presence() < presence_threshold)) {
      continue;
    }
    AddConnectionToRenderData<LandmarkType>(ld0, ld1, connection_color,
@@ -162,45 +173,6 @@ RenderAnnotation* AddPointRenderData(const Color& landmark_color,

 }  // namespace

-// A calculator that converts Landmark proto to RenderData proto for
-// visualization. The input should be LandmarkList proto. It is also possible
-// to specify the connections between landmarks.
-//
-// Example config:
-// node {
-//   calculator: "LandmarksToRenderDataCalculator"
-//   input_stream: "NORM_LANDMARKS:landmarks"
-//   output_stream: "RENDER_DATA:render_data"
-//   options {
-//     [LandmarksToRenderDataCalculatorOptions.ext] {
-//       landmark_connections: [0, 1, 1, 2]
-//       landmark_color { r: 0 g: 255 b: 0 }
-//       connection_color { r: 0 g: 255 b: 0 }
-//       thickness: 4.0
-//     }
-//   }
-// }
-class LandmarksToRenderDataCalculator : public CalculatorBase {
- public:
-  LandmarksToRenderDataCalculator() {}
-  ~LandmarksToRenderDataCalculator() override {}
-  LandmarksToRenderDataCalculator(const LandmarksToRenderDataCalculator&) =
-      delete;
-  LandmarksToRenderDataCalculator& operator=(
-      const LandmarksToRenderDataCalculator&) = delete;
-
-  static ::mediapipe::Status GetContract(CalculatorContract* cc);
-
-  ::mediapipe::Status Open(CalculatorContext* cc) override;
-
-  ::mediapipe::Status Process(CalculatorContext* cc) override;
-
- private:
-  LandmarksToRenderDataCalculatorOptions options_;
-  std::vector<int> landmark_connections_;
-};
-REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
-
 ::mediapipe::Status LandmarksToRenderDataCalculator::GetContract(
    CalculatorContract* cc) {
  RET_CHECK(cc->Inputs().HasTag(kLandmarksTag) ||
@@ -278,13 +250,15 @@ REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
    if (visualize_depth) {
      AddConnectionsWithDepth<LandmarkList, Landmark>(
          landmarks, landmark_connections_, options_.utilize_visibility(),
-          options_.visibility_threshold(), thickness, /*normalized=*/false,
-          z_min, z_max, render_data.get());
+          options_.visibility_threshold(), options_.utilize_presence(),
+          options_.presence_threshold(), thickness, /*normalized=*/false, z_min,
+          z_max, render_data.get());
    } else {
      AddConnections<LandmarkList, Landmark>(
          landmarks, landmark_connections_, options_.utilize_visibility(),
-          options_.visibility_threshold(), options_.connection_color(),
-          thickness, /*normalized=*/false, render_data.get());
+          options_.visibility_threshold(), options_.utilize_presence(),
+          options_.presence_threshold(), options_.connection_color(), thickness,
+          /*normalized=*/false, render_data.get());
    }
    for (int i = 0; i < landmarks.landmark_size(); ++i) {
      const Landmark& landmark = landmarks.landmark(i);
@@ -294,11 +268,17 @@ REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
        continue;
      }

+      if (options_.utilize_presence() &&
+          landmark.presence() < options_.presence_threshold()) {
+        continue;
+      }
+
      auto* landmark_data_render = AddPointRenderData(
          options_.landmark_color(), thickness, render_data.get());
      if (visualize_depth) {
-        SetColorSizeValueFromZ(landmark.z(), z_min, z_max,
-                               landmark_data_render);
+        SetColorSizeValueFromZ(landmark.z(), z_min, z_max, landmark_data_render,
+                               options_.min_depth_circle_thickness(),
+                               options_.max_depth_circle_thickness());
      }
      auto* landmark_data = landmark_data_render->mutable_point();
      landmark_data->set_normalized(false);
@@ -319,13 +299,15 @@ REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
    if (visualize_depth) {
      AddConnectionsWithDepth<NormalizedLandmarkList, NormalizedLandmark>(
          landmarks, landmark_connections_, options_.utilize_visibility(),
-          options_.visibility_threshold(), thickness, /*normalized=*/true,
-          z_min, z_max, render_data.get());
+          options_.visibility_threshold(), options_.utilize_presence(),
+          options_.presence_threshold(), thickness, /*normalized=*/true, z_min,
+          z_max, render_data.get());
    } else {
      AddConnections<NormalizedLandmarkList, NormalizedLandmark>(
          landmarks, landmark_connections_, options_.utilize_visibility(),
-          options_.visibility_threshold(), options_.connection_color(),
-          thickness, /*normalized=*/true, render_data.get());
+          options_.visibility_threshold(), options_.utilize_presence(),
+          options_.presence_threshold(), options_.connection_color(), thickness,
+          /*normalized=*/true, render_data.get());
    }
    for (int i = 0; i < landmarks.landmark_size(); ++i) {
      const NormalizedLandmark& landmark = landmarks.landmark(i);
@@ -334,12 +316,17 @@ REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
          landmark.visibility() < options_.visibility_threshold()) {
        continue;
      }
+      if (options_.utilize_presence() &&
+          landmark.presence() < options_.presence_threshold()) {
+        continue;
+      }

      auto* landmark_data_render = AddPointRenderData(
          options_.landmark_color(), thickness, render_data.get());
      if (visualize_depth) {
-        SetColorSizeValueFromZ(landmark.z(), z_min, z_max,
-                               landmark_data_render);
+        SetColorSizeValueFromZ(landmark.z(), z_min, z_max, landmark_data_render,
+                               options_.min_depth_circle_thickness(),
+                               options_.max_depth_circle_thickness());
      }
      auto* landmark_data = landmark_data_render->mutable_point();
      landmark_data->set_normalized(true);
@@ -354,4 +341,5 @@ REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
  return ::mediapipe::OkStatus();
 }

+REGISTER_CALCULATOR(LandmarksToRenderDataCalculator);
 }  // namespace mediapipe
@@ -0,0 +1,69 @@
+// Copyright 2020 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+#ifndef MEDIAPIPE_CALCULATORS_UTIL_LANDMARKS_TO_RENDER_DATA_CALCULATOR_H_
+#define MEDIAPIPE_CALCULATORS_UTIL_LANDMARKS_TO_RENDER_DATA_CALCULATOR_H_
+
+#include "absl/memory/memory.h"
+#include "absl/strings/str_cat.h"
+#include "absl/strings/str_join.h"
+#include "mediapipe/calculators/util/landmarks_to_render_data_calculator.pb.h"
+#include "mediapipe/framework/calculator_framework.h"
+#include "mediapipe/framework/calculator_options.pb.h"
+#include "mediapipe/framework/formats/landmark.pb.h"
+#include "mediapipe/framework/formats/location_data.pb.h"
+#include "mediapipe/framework/port/ret_check.h"
+#include "mediapipe/util/color.pb.h"
+#include "mediapipe/util/render_data.pb.h"
+namespace mediapipe {
+
+// A calculator that converts Landmark proto to RenderData proto for
+// visualization. The input should be LandmarkList proto. It is also possible
+// to specify the connections between landmarks.
+//
+// Example config:
+// node {
+//   calculator: "LandmarksToRenderDataCalculator"
+//   input_stream: "NORM_LANDMARKS:landmarks"
+//   output_stream: "RENDER_DATA:render_data"
+//   options {
+//     [LandmarksToRenderDataCalculatorOptions.ext] {
+//       landmark_connections: [0, 1, 1, 2]
+//       landmark_color { r: 0 g: 255 b: 0 }
+//       connection_color { r: 0 g: 255 b: 0 }
+//       thickness: 4.0
+//     }
+//   }
+// }
+class LandmarksToRenderDataCalculator : public CalculatorBase {
+ public:
+  LandmarksToRenderDataCalculator() {}
+  ~LandmarksToRenderDataCalculator() override {}
+  LandmarksToRenderDataCalculator(const LandmarksToRenderDataCalculator&) =
+      delete;
+  LandmarksToRenderDataCalculator& operator=(
+      const LandmarksToRenderDataCalculator&) = delete;
+
+  static ::mediapipe::Status GetContract(CalculatorContract* cc);
+
+  ::mediapipe::Status Open(CalculatorContext* cc) override;
+
+  ::mediapipe::Status Process(CalculatorContext* cc) override;
+
+ protected:
+  ::mediapipe::LandmarksToRenderDataCalculatorOptions options_;
+  std::vector<int> landmark_connections_;
+};
+
+}  // namespace mediapipe
+#endif  // MEDIAPIPE_CALCULATORS_UTIL_LANDMARKS_TO_RENDER_DATA_CALCULATOR_H_
@@ -49,4 +49,19 @@ message LandmarksToRenderDataCalculatorOptions {
  // Threshold to determine visibility of the landmark. Landmark with visibility
  // greater or equal than threshold is considered visible.
  optional double visibility_threshold = 7 [default = 0.0];
+
+  // Use landmarks presence while rendering landmarks and connections. If
+  // landmark is not present, neither it nor adjacent connections will be
+  // rendered.
+  optional bool utilize_presence = 8 [default = false];
+
+  // Threshold to determine presence of the landmark. Landmark with presence
+  // greater or equal than threshold is considered present.
+  optional double presence_threshold = 9 [default = 0.0];
+
+  // Min thickness of the drawing for landmark circle.
+  optional double min_depth_circle_thickness = 10 [default = 0.0];
+
+  // Max thickness of the drawing for landmark circle.
+  optional double max_depth_circle_thickness = 11 [default = 18.0];
 }
@@ -19,7 +19,7 @@ load(
    "mediapipe_binary_graph",
 )

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -317,6 +317,7 @@ cc_library(
        "//mediapipe/util/tracking:box_tracker",
        "//mediapipe/util/tracking:tracking_visualization_utilities",
        "@com_google_absl//absl/container:flat_hash_set",
+        "@com_google_absl//absl/container:node_hash_map",
        "@com_google_absl//absl/container:node_hash_set",
        "@com_google_absl//absl/strings",
    ],
@@ -19,6 +19,7 @@
 #include <unordered_set>

 #include "absl/container/flat_hash_set.h"
+#include "absl/container/node_hash_map.h"
 #include "absl/container/node_hash_set.h"
 #include "absl/strings/numbers.h"
 #include "mediapipe/calculators/video/box_tracker_calculator.pb.h"
@@ -207,7 +208,7 @@ class BoxTrackerCalculator : public CalculatorBase {
  // Boxes that are tracked in streaming mode.
  MotionBoxMap streaming_motion_boxes_;

-  std::unordered_map<int, std::pair<TimedBox, TimedBox>> last_tracked_boxes_;
+  absl::node_hash_map<int, std::pair<TimedBox, TimedBox>> last_tracked_boxes_;
  int frame_num_since_reset_ = 0;

  // Cache used during streaming mode for fast forward tracking.
@@ -15,12 +15,10 @@

 load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//mediapipe/calculators/video:__subpackages__"])

-exports_files(["LICENSE"])
-
 proto_library(
    name = "flow_quantizer_model_proto",
    srcs = ["flow_quantizer_model.proto"],
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 # Basic library common across example apps.
 android_library(
@@ -80,7 +80,7 @@ public class MainActivity extends AppCompatActivity {
  @Override
  protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
-    setContentView(R.layout.activity_main);
+    setContentView(getContentViewLayoutResId());

    try {
      applicationInfo =
@@ -112,6 +112,12 @@ public class MainActivity extends AppCompatActivity {
    PermissionHelper.checkAndRequestCameraPermissions(this);
  }

+  // Used to obtain the content view for this application. If you are extending this class, and
+  // have a custom layout, override this method and return the custom layout.
+  protected int getContentViewLayoutResId() {
+    return R.layout.activity_main;
+  }
+
  @Override
  protected void onResume() {
    super.onResume();
@@ -128,6 +134,9 @@ public class MainActivity extends AppCompatActivity {
  protected void onPause() {
    super.onPause();
    converter.close();
+
+    // Hide preview display until we re-open the camera again.
+    previewDisplayView.setVisibility(View.GONE);
  }

  @Override
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -0,0 +1,67 @@
+# Copyright 2020 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:private"])
+
+cc_binary(
+    name = "libmediapipe_jni.so",
+    linkshared = 1,
+    linkstatic = 1,
+    deps = [
+        "//mediapipe/graphs/face_effect:face_effect_gpu_deps",
+        "//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
+    ],
+)
+
+cc_library(
+    name = "mediapipe_jni_lib",
+    srcs = [":libmediapipe_jni.so"],
+    alwayslink = 1,
+)
+
+android_binary(
+    name = "faceeffect",
+    srcs = glob(["*.java"]),
+    assets = [
+        "//mediapipe/graphs/face_effect/data:facepaint.pngblob",
+        "//mediapipe/graphs/face_effect/data:glasses.binarypb",
+        "//mediapipe/graphs/face_effect/data:glasses.pngblob",
+        "//mediapipe/graphs/face_effect:face_effect_gpu.binarypb",
+        "//mediapipe/modules/face_detection:face_detection_front.tflite",
+        "//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
+        "//mediapipe/modules/face_landmark:face_landmark.tflite",
+    ],
+    assets_dir = "",
+    manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
+    manifest_values = {
+        "applicationId": "com.google.mediapipe.apps.faceeffect",
+        "appName": "Face Effect",
+        "mainActivity": ".MainActivity",
+        "cameraFacingFront": "True",
+        "binaryGraphName": "face_effect_gpu.binarypb",
+        "inputVideoStreamName": "input_video",
+        "outputVideoStreamName": "output_video",
+        "flipFramesVertically": "True",
+    },
+    multidex = "native",
+    deps = [
+        ":mediapipe_jni_lib",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
+        "//mediapipe/framework/formats:matrix_data_java_proto_lite",
+        "//mediapipe/java/com/google/mediapipe/framework:android_framework",
+        "//mediapipe/modules/face_geometry/protos:face_geometry_java_proto_lite",
+    ],
+)
@@ -0,0 +1,176 @@
+// Copyright 2020 The MediaPipe Authors.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.mediapipe.apps.faceeffect;
+
+import android.graphics.Color;
+import android.os.Bundle;
+import android.util.Log;
+import android.view.GestureDetector;
+import android.view.Gravity;
+import android.view.MotionEvent;
+import android.view.View;
+import android.view.ViewGroup;
+import android.view.ViewGroup.LayoutParams;
+import android.widget.RelativeLayout;
+import android.widget.TextView;
+import com.google.mediapipe.framework.Packet;
+import com.google.mediapipe.framework.PacketGetter;
+import com.google.mediapipe.modules.facegeometry.FaceGeometryProto.FaceGeometry;
+import com.google.mediapipe.formats.proto.MatrixDataProto.MatrixData;
+import java.util.List;
+
+/** Main activity of MediaPipe face mesh app. */
+public class MainActivity extends com.google.mediapipe.apps.basic.MainActivity {
+  private static final String TAG = "MainActivity";
+
+  // Stream names.
+  private static final String IS_FACEPAINT_EFFECT_SELECTED_INPUT_STREAM_NAME =
+      "is_facepaint_effect_selected";
+  private static final String OUTPUT_FACE_GEOMETRY_STREAM_NAME = "multi_face_geometry";
+
+  private static final String EFFECT_SWITCHING_HINT_TEXT = "Tap to switch between effects!";
+
+  private static final int MATRIX_TRANSLATION_Z_INDEX = 14;
+
+  private final Object isFacepaintEffectSelectedLock = new Object();
+  private boolean isFacepaintEffectSelected;
+
+  private View effectSwitchingHintView;
+  private GestureDetector tapGestureDetector;
+
+  @Override
+  protected void onCreate(Bundle savedInstanceState) {
+    super.onCreate(savedInstanceState);
+
+    // Add an effect switching hint view to the preview layout.
+    effectSwitchingHintView = createEffectSwitchingHintView();
+    effectSwitchingHintView.setVisibility(View.INVISIBLE);
+    ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
+    viewGroup.addView(effectSwitchingHintView);
+
+    // By default, render the glasses effect.
+    isFacepaintEffectSelected = false;
+
+    // This callback demonstrates how the output face geometry packet can be obtained and used
+    // in an Android app. As an example, the Z-translation component of the face pose transform
+    // matrix is logged for each face being equal to the approximate distance away from the camera
+    // in centimeters.
+    processor.addPacketCallback(
+        OUTPUT_FACE_GEOMETRY_STREAM_NAME,
+        (packet) -> {
+          effectSwitchingHintView.post(
+              new Runnable() {
+                @Override
+                public void run() {
+                  effectSwitchingHintView.setVisibility(View.VISIBLE);
+                }
+              });
+
+          Log.d(TAG, "Received a multi face geometry packet.");
+          List<FaceGeometry> multiFaceGeometry =
+              PacketGetter.getProtoVector(packet, FaceGeometry.parser());
+
+          StringBuilder approxDistanceAwayFromCameraLogMessage = new StringBuilder();
+          for (FaceGeometry faceGeometry : multiFaceGeometry) {
+            if (approxDistanceAwayFromCameraLogMessage.length() > 0) {
+              approxDistanceAwayFromCameraLogMessage.append(' ');
+            }
+            MatrixData poseTransformMatrix = faceGeometry.getPoseTransformMatrix();
+            approxDistanceAwayFromCameraLogMessage.append(
+                -poseTransformMatrix.getPackedData(MATRIX_TRANSLATION_Z_INDEX));
+          }
+
+          Log.d(
+              TAG,
+              "[TS:"
+                  + packet.getTimestamp()
+                  + "] size = "
+                  + multiFaceGeometry.size()
+                  + "; approx. distance away from camera in cm for faces = ["
+                  + approxDistanceAwayFromCameraLogMessage
+                  + "]");
+        });
+
+    // Alongside the input camera frame, we also send the `is_facepaint_effect_selected` boolean
+    // packet to indicate which effect should be rendered on this frame.
+    processor.setOnWillAddFrameListener(
+        (timestamp) -> {
+          Packet isFacepaintEffectSelectedPacket = null;
+          try {
+            synchronized (isFacepaintEffectSelectedLock) {
+              isFacepaintEffectSelectedPacket =
+                  processor.getPacketCreator().createBool(isFacepaintEffectSelected);
+            }
+
+            processor
+                .getGraph()
+                .addPacketToInputStream(
+                    IS_FACEPAINT_EFFECT_SELECTED_INPUT_STREAM_NAME,
+                    isFacepaintEffectSelectedPacket,
+                    timestamp);
+          } catch (RuntimeException e) {
+            Log.e(
+                TAG,
+                "Exception while adding packet to input stream while switching effects: " + e);
+          } finally {
+            if (isFacepaintEffectSelectedPacket != null) {
+              isFacepaintEffectSelectedPacket.release();
+            }
+          }
+        });
+
+    // We use the tap gesture detector to switch between face effects. This allows users to try
+    // multiple pre-bundled face effects without a need to recompile the app.
+    tapGestureDetector =
+        new GestureDetector(
+            this,
+            new GestureDetector.SimpleOnGestureListener() {
+              @Override
+              public void onLongPress(MotionEvent event) {
+                switchEffect();
+              }
+
+              @Override
+              public boolean onSingleTapUp(MotionEvent event) {
+                switchEffect();
+                return true;
+              }
+
+              private void switchEffect() {
+                synchronized (isFacepaintEffectSelectedLock) {
+                  isFacepaintEffectSelected = !isFacepaintEffectSelected;
+                }
+              }
+            });
+  }
+
+  @Override
+  public boolean onTouchEvent(MotionEvent event) {
+    return tapGestureDetector.onTouchEvent(event);
+  }
+
+  private View createEffectSwitchingHintView() {
+    TextView effectSwitchingHintView = new TextView(getApplicationContext());
+    effectSwitchingHintView.setLayoutParams(
+        new RelativeLayout.LayoutParams(LayoutParams.FILL_PARENT, LayoutParams.FILL_PARENT));
+    effectSwitchingHintView.setText(EFFECT_SWITCHING_HINT_TEXT);
+    effectSwitchingHintView.setGravity(Gravity.CENTER_HORIZONTAL | Gravity.BOTTOM);
+    effectSwitchingHintView.setPadding(0, 0, 0, 480);
+    effectSwitchingHintView.setTextColor(Color.parseColor("#ffffff"));
+    effectSwitchingHintView.setTextSize((float) 24);
+
+    return effectSwitchingHintView;
+  }
+}
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-licenses(["notice"])  # Apache 2.0
+licenses(["notice"])

 package(default_visibility = ["//visibility:private"])

@@ -0,0 +1,99 @@
+# Copyright 2020 The MediaPipe Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:private"])
+
+java_lite_proto_library(
+    name = "sticker_buffer_java_proto_lite",
+    deps = ["//mediapipe/graphs/instant_motion_tracking/calculators:sticker_buffer_proto"],
+)
+
+android_library(
+    name = "instantmotiontracking_lib",
+    srcs = glob(["*.java"]),
+    manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
+    resource_files = glob([
+        "res/layout/**",
+        "res/drawable/**",
+    ]),
+    visibility = ["//visibility:public"],
+    deps = [
+        ":sticker_buffer_java_proto_lite",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
+        "//mediapipe/java/com/google/mediapipe/components:android_components",
+        "//mediapipe/java/com/google/mediapipe/framework:android_framework",
+        "//third_party:androidx_appcompat",
+        "//third_party:androidx_core",
+        "//third_party:opencv",
+        "@maven//:androidx_concurrent_concurrent_futures",
+        "@maven//:com_github_bumptech_glide_glide",
+        "@maven//:com_google_guava_guava",
+    ],
+)
+
+# Include all calculators specific to this project defined by BUILD in graphs
+cc_binary(
+    name = "libmediapipe_jni.so",
+    linkshared = 1,
+    linkstatic = 1,
+    deps = [
+        "//mediapipe/graphs/instant_motion_tracking:instant_motion_tracking_deps",
+        "//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
+    ],
+)
+
+# Converts the .so cc_binary into a cc_library, to be consumed in an android_binary.
+cc_library(
+    name = "mediapipe_jni_lib",
+    srcs = [":libmediapipe_jni.so"],
+    alwayslink = 1,
+)
+
+genrule(
+    name = "asset3d",
+    srcs = ["//mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/assets:robot/robot.obj.uuu.zip"],
+    outs = ["robot/robot.obj.uuu"],
+    cmd = "unzip -p $< > $@",
+)
+
+android_binary(
+    name = "instantmotiontracking",
+    assets = [
+        ":asset3d",
+        "//mediapipe/graphs/instant_motion_tracking:instant_motion_tracking.binarypb",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/assets:gif/gif.obj.uuu",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/assets:gif/default_gif_texture.jpg",
+        "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/assets:robot/robot_texture.jpg",
+    ],
+    assets_dir = "",
+    manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
+    manifest_values = {
+        "applicationId": "com.google.mediapipe.apps.instantmotiontracking",
+        "appName": "Instant Motion Tracking",
+        "mainActivity": ".MainActivity",
+        "cameraFacingFront": "False",
+        "binaryGraphName": "instant_motion_tracking.binarypb",
+        "inputVideoStreamName": "input_video",
+        "outputVideoStreamName": "output_video",
+        "flipFramesVertically": "True",
+    },
+    multidex = "native",
+    deps = [
+        ":instantmotiontracking_lib",
+        ":mediapipe_jni_lib",
+        "//mediapipe/java/com/google/mediapipe/framework:android_framework",
+    ],
+)
@@ -0,0 +1,103 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.mediapipe.apps.instantmotiontracking;
+
+import android.content.ClipDescription;
+import android.content.Context;
+import android.net.Uri;
+import android.os.Bundle;
+import androidx.appcompat.widget.AppCompatEditText;
+import android.util.AttributeSet;
+import android.util.Log;
+import android.view.inputmethod.EditorInfo;
+import android.view.inputmethod.InputConnection;
+import androidx.core.view.inputmethod.EditorInfoCompat;
+import androidx.core.view.inputmethod.InputConnectionCompat;
+import androidx.core.view.inputmethod.InputContentInfoCompat;
+
+// import android.support.v13.view.inputmethod.EditorInfoCompat;
+// import android.support.v13.view.inputmethod.InputConnectionCompat;
+// import android.support.v13.view.inputmethod.InputContentInfoCompat;
+
+/**
+ * This custom EditText implementation uses the existing EditText framework in
+ * order to develop a GIFEditText input box which is capable of accepting GIF
+ * animations from the Android system keyboard and return the GIF location with
+ * a content URI.
+ */
+public class GIFEditText extends AppCompatEditText {
+
+  private GIFCommitListener gifCommitListener;
+
+  public GIFEditText(Context context) {
+    super(context);
+  }
+
+  public GIFEditText(Context context, AttributeSet attrs) {
+    super(context, attrs);
+  }
+
+  /**
+   * onGIFCommit is called once content is pushed to the EditText via the
+   * Android keyboard.
+   */
+  public interface GIFCommitListener {
+    void onGIFCommit(Uri contentUri, ClipDescription description);
+  }
+
+  /**
+   * Used to set the gifCommitListener for this GIFEditText.
+   *
+   * @param gifCommitListener handles response to new content pushed to EditText
+   */
+  public void setGIFCommitListener(GIFCommitListener gifCommitListener) {
+    this.gifCommitListener = gifCommitListener;
+  }
+
+  @Override
+  public InputConnection onCreateInputConnection(EditorInfo editorInfo) {
+    final InputConnection inputConnection = super.onCreateInputConnection(editorInfo);
+    EditorInfoCompat.setContentMimeTypes(editorInfo, new String[] {"image/gif"});
+    return InputConnectionCompat.createWrapper(
+        inputConnection,
+        editorInfo,
+        new InputConnectionCompat.OnCommitContentListener() {
+          @Override
+          public boolean onCommitContent(
+              final InputContentInfoCompat inputContentInfo, int flags, Bundle opts) {
+            try {
+              if (gifCommitListener != null) {
+                Runnable runnable =
+                    new Runnable() {
+                      @Override
+                      public void run() {
+                        inputContentInfo.requestPermission();
+                        gifCommitListener.onGIFCommit(
+                            inputContentInfo.getContentUri(), inputContentInfo.getDescription());
+                        inputContentInfo.releasePermission();
+                      }
+                    };
+                new Thread(runnable).start();
+              }
+            } catch (RuntimeException e) {
+              Log.e("GIFEditText", "Input connection to GIF selection failed");
+              e.printStackTrace();
+              return false;
+            }
+            return true;
+          }
+        });
+  }
+}
@@ -0,0 +1,633 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.mediapipe.apps.instantmotiontracking;
+
+import static java.lang.Math.max;
+
+import android.content.ClipDescription;
+import android.content.Context;
+import android.content.Intent;
+import android.graphics.Bitmap;
+import android.graphics.BitmapFactory;
+import android.graphics.Color;
+import android.graphics.Matrix;
+import android.graphics.drawable.Drawable;
+import android.hardware.Sensor;
+import android.hardware.SensorEvent;
+import android.hardware.SensorEventListener;
+import android.hardware.SensorManager;
+import android.net.Uri;
+import android.os.Bundle;
+import android.util.Log;
+import android.util.Size;
+import android.view.MotionEvent;
+import android.view.SurfaceHolder;
+import android.view.View;
+import android.view.ViewGroup;
+import android.view.inputmethod.InputMethodManager;
+import android.widget.ImageButton;
+import android.widget.ImageView;
+import android.widget.LinearLayout;
+import com.bumptech.glide.Glide;
+import com.bumptech.glide.load.resource.gif.GifDrawable;
+import com.bumptech.glide.request.target.CustomTarget;
+import com.bumptech.glide.request.transition.Transition;
+import com.google.mediapipe.components.FrameProcessor;
+import com.google.mediapipe.framework.AndroidPacketCreator;
+import com.google.mediapipe.framework.Packet;
+import java.io.InputStream;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * This is the MainActivity that handles camera input, IMU sensor data acquisition
+ * and sticker management for the InstantMotionTracking MediaPipe project.
+ */
+public class MainActivity extends com.google.mediapipe.apps.basic.MainActivity {
+  private static final String TAG = "InstantMotionTrackingMainActivity";
+
+  // Allows for automated packet transmission to graph
+  private MediaPipePacketManager mediaPipePacketManager;
+
+  private static final int TARGET_CAMERA_WIDTH = 960;
+  private static final int TARGET_CAMERA_HEIGHT = 1280;
+  private static final float TARGET_CAMERA_ASPECT_RATIO =
+      (float) TARGET_CAMERA_WIDTH / (float) TARGET_CAMERA_HEIGHT;
+
+  // Bounds for a single click (sticker anchor reset)
+  private static final long CLICK_DURATION = 300; // ms
+  private long clickStartMillis = 0;
+  private ViewGroup viewGroup;
+  // Contains dynamic layout of sticker data controller
+  private LinearLayout buttonLayout;
+
+  private ArrayList<StickerManager> stickerArrayList;
+  // Current sticker being edited by user
+  private StickerManager currentSticker;
+  // Trip value used to determine sticker re-anchoring
+  private static final String STICKER_SENTINEL_TAG = "sticker_sentinel";
+  private int stickerSentinel = -1;
+
+  // Define parameters for 'reactivity' of object
+  private static final float ROTATION_SPEED = 5.0f;
+  private static final float SCALING_FACTOR = 0.025f;
+
+  // Parameters of device visual field for rendering system
+  // (68 degrees, 4:3 for Pixel 4)
+  // TODO : Make acquisition of this information automated
+  private static final float VERTICAL_FOV_RADIANS = (float) Math.toRadians(68.0);
+  private static final String FOV_SIDE_PACKET_TAG = "vertical_fov_radians";
+  private static final String ASPECT_RATIO_SIDE_PACKET_TAG = "aspect_ratio";
+
+  private static final String IMU_MATRIX_TAG = "imu_rotation_matrix";
+  private static final int SENSOR_SAMPLE_DELAY = SensorManager.SENSOR_DELAY_FASTEST;
+  private final float[] rotationMatrix = new float[9];
+
+  private static final String STICKER_PROTO_TAG = "sticker_proto_string";
+  // Assets for object rendering
+  // All animation assets and tags for the first asset (1)
+  private Bitmap asset3dTexture = null;
+  private static final String ASSET_3D_TEXTURE = "robot/robot_texture.jpg";
+  private static final String ASSET_3D_FILE = "robot/robot.obj.uuu";
+  private static final String ASSET_3D_TEXTURE_TAG = "texture_3d";
+  private static final String ASSET_3D_TAG = "asset_3d";
+  // All GIF animation assets and tags
+  private GIFEditText editText;
+  private ArrayList<Bitmap> gifBitmaps = new ArrayList<>();
+  private int gifCurrentIndex = 0;
+  private Bitmap defaultGIFTexture = null;  // Texture sent if no gif available
+  // last time the GIF was updated
+  private long gifLastFrameUpdateMS = System.currentTimeMillis();
+  private static final int GIF_FRAME_RATE = 20; // 20 FPS
+  private static final String GIF_ASPECT_RATIO_TAG = "gif_aspect_ratio";
+  private static final String DEFAULT_GIF_TEXTURE = "gif/default_gif_texture.jpg";
+  private static final String GIF_FILE = "gif/gif.obj.uuu";
+  private static final String GIF_TEXTURE_TAG = "gif_texture";
+  private static final String GIF_ASSET_TAG = "gif_asset_name";
+
+  private int cameraWidth = TARGET_CAMERA_WIDTH;
+  private int cameraHeight = TARGET_CAMERA_HEIGHT;
+
+  @Override
+  protected Size cameraTargetResolution() {
+    // Camera size is in landscape, so here we have (height, width)
+    return new Size(TARGET_CAMERA_HEIGHT, TARGET_CAMERA_WIDTH);
+  }
+
+  @Override
+  protected Size computeViewSize(int width, int height) {
+    // Try to force aspect ratio of view size to match our target aspect ratio
+    return new Size(height, (int) (height * TARGET_CAMERA_ASPECT_RATIO));
+  }
+
+  @Override
+  protected void onPreviewDisplaySurfaceChanged(
+      SurfaceHolder holder, int format, int width, int height) {
+    super.onPreviewDisplaySurfaceChanged(holder, format, width, height);
+    boolean isCameraRotated = cameraHelper.isCameraRotated();
+
+    // cameraImageSize computation logic duplicated from base MainActivity
+    Size viewSize = computeViewSize(width, height);
+    Size cameraImageSize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
+    cameraWidth =
+        isCameraRotated ? cameraImageSize.getHeight() : cameraImageSize.getWidth();
+    cameraHeight =
+        isCameraRotated ? cameraImageSize.getWidth() : cameraImageSize.getHeight();
+  }
+
+  @Override
+  protected void onCreate(Bundle savedInstanceState) {
+
+    super.onCreate(savedInstanceState);
+
+    editText = findViewById(R.id.gif_edit_text);
+    editText.setGIFCommitListener(
+        new GIFEditText.GIFCommitListener() {
+          @Override
+          public void onGIFCommit(Uri contentUri, ClipDescription description) {
+            // The application must have permission to access the GIF content
+            grantUriPermission(
+                "com.google.mediapipe.apps.instantmotiontracking",
+                contentUri,
+                Intent.FLAG_GRANT_READ_URI_PERMISSION);
+            // Set GIF frames from content URI
+            setGIFBitmaps(contentUri.toString());
+            // Close the keyboard upon GIF acquisition
+            closeKeyboard();
+          }
+        });
+
+    // Send loaded 3d render assets as side packets to graph
+    prepareDemoAssets();
+    AndroidPacketCreator packetCreator = processor.getPacketCreator();
+
+    Map<String, Packet> inputSidePackets = new HashMap<>();
+    inputSidePackets.put(ASSET_3D_TEXTURE_TAG,
+      packetCreator.createRgbaImageFrame(asset3dTexture));
+    inputSidePackets.put(ASSET_3D_TAG,
+      packetCreator.createString(ASSET_3D_FILE));
+    inputSidePackets.put(GIF_ASSET_TAG,
+      packetCreator.createString(GIF_FILE));
+    processor.setInputSidePackets(inputSidePackets);
+
+    // Add frame listener to PacketManagement system
+    mediaPipePacketManager = new MediaPipePacketManager();
+    processor.setOnWillAddFrameListener(mediaPipePacketManager);
+
+    // Send device properties to render objects via OpenGL
+    Map<String, Packet> devicePropertiesSidePackets = new HashMap<>();
+    // TODO: Note that if our actual camera stream resolution does not match the
+    // requested aspect ratio, then we will need to update the value used for
+    // this packet, or else tracking results will be off.
+    devicePropertiesSidePackets.put(
+        ASPECT_RATIO_SIDE_PACKET_TAG, packetCreator.createFloat32(TARGET_CAMERA_ASPECT_RATIO));
+    devicePropertiesSidePackets.put(
+        FOV_SIDE_PACKET_TAG, packetCreator.createFloat32(VERTICAL_FOV_RADIANS));
+    processor.setInputSidePackets(devicePropertiesSidePackets);
+
+    // Begin with 0 stickers in dataset
+    stickerArrayList = new ArrayList<>();
+    currentSticker = null;
+
+    SensorManager sensorManager = (SensorManager) getSystemService(SENSOR_SERVICE);
+    List<Sensor> sensorList = sensorManager.getSensorList(Sensor.TYPE_ROTATION_VECTOR);
+    sensorManager.registerListener(
+        new SensorEventListener() {
+          private final float[] rotMatFromVec = new float[9];
+
+          @Override
+          public void onAccuracyChanged(Sensor sensor, int accuracy) {}
+          // Update procedure on sensor adjustment (phone changes orientation)
+
+          @Override
+          public void onSensorChanged(SensorEvent event) {
+            // Get the Rotation Matrix from the Rotation Vector
+            SensorManager.getRotationMatrixFromVector(rotMatFromVec, event.values);
+            // AXIS_MINUS_X is used to remap the rotation matrix for left hand
+            // rules in the MediaPipe graph
+            SensorManager.remapCoordinateSystem(
+                rotMatFromVec, SensorManager.AXIS_MINUS_X, SensorManager.AXIS_Y, rotationMatrix);
+          }
+        },
+        (Sensor) sensorList.get(0),
+        SENSOR_SAMPLE_DELAY);
+
+    // Mechanisms for zoom, pinch, rotation, tap gestures
+    buttonLayout = (LinearLayout) findViewById(R.id.button_layout);
+    viewGroup = findViewById(R.id.preview_display_layout);
+    viewGroup.setOnTouchListener(
+        new View.OnTouchListener() {
+          @Override
+          public boolean onTouch(View v, MotionEvent event) {
+            return manageUiTouch(event);
+          }
+        });
+    refreshUi();
+  }
+
+  // Obtain our custom activity_main layout for InstantMotionTracking
+  @Override
+  protected int getContentViewLayoutResId() {
+    return R.layout.instant_motion_tracking_activity_main;
+  }
+
+  // Manages a touch event in order to perform placement/rotation/scaling gestures
+  // on virtual sticker objects.
+  private boolean manageUiTouch(MotionEvent event) {
+    if (currentSticker != null) {
+      switch (event.getAction()) {
+          // Detecting a single click for object re-anchoring
+        case (MotionEvent.ACTION_DOWN):
+          clickStartMillis = System.currentTimeMillis();
+          break;
+        case (MotionEvent.ACTION_UP):
+          if (System.currentTimeMillis() - clickStartMillis <= CLICK_DURATION) {
+            recordClick(event);
+          }
+          break;
+        case (MotionEvent.ACTION_MOVE):
+          // Rotation and Scaling are independent events and can occur simulataneously
+          if (event.getPointerCount() == 2) {
+            if (event.getHistorySize() > 1) {
+              // Calculate user scaling of sticker
+              float newScaleFactor = getNewScaleFactor(event, currentSticker.getScaleFactor());
+              currentSticker.setScaleFactor(newScaleFactor);
+              // calculate rotation (radians) for dynamic y-axis rotations
+              float rotationIncrement = calculateRotationRadians(event);
+              currentSticker.setRotation(currentSticker.getRotation() + rotationIncrement);
+            }
+          }
+          break;
+        default:
+          // fall out
+      }
+    }
+    return true;
+  }
+
+  // Returns a float value that is equal to the radians of rotation from a two-finger
+  // MotionEvent recorded by the OnTouchListener.
+  private static float calculateRotationRadians(MotionEvent event) {
+    float tangentA =
+        (float) Math.atan2(event.getY(1) - event.getY(0), event.getX(1) - event.getX(0));
+    float tangentB =
+        (float)
+            Math.atan2(
+                event.getHistoricalY(1, 0) - event.getHistoricalY(0, 0),
+                event.getHistoricalX(1, 0) - event.getHistoricalX(0, 0));
+    float angle = ((float) Math.toDegrees(tangentA - tangentB)) % 360f;
+    angle += ((angle < -180f) ? +360f : ((angle > 180f) ? -360f : 0.0f));
+    float rotationIncrement = (float) (Math.PI * ((angle * ROTATION_SPEED) / 180));
+    return rotationIncrement;
+  }
+
+  // Returns a float value that is equal to the translation distance between
+  // two-fingers that move in a pinch/spreading direction.
+  private static float getNewScaleFactor(MotionEvent event, float currentScaleFactor) {
+    double newDistance = getDistance(event.getX(0), event.getY(0), event.getX(1), event.getY(1));
+    double oldDistance =
+        getDistance(
+            event.getHistoricalX(0, 0),
+            event.getHistoricalY(0, 0),
+            event.getHistoricalX(1, 0),
+            event.getHistoricalY(1, 0));
+    float signFloat =
+        (newDistance < oldDistance)
+            ? -SCALING_FACTOR
+            : SCALING_FACTOR; // Are they moving towards each other?
+    currentScaleFactor *= (1f + signFloat);
+    return currentScaleFactor;
+  }
+
+  // Called if a single touch event is recorded on the screen and used to set the
+  // new anchor position for the current sticker in focus.
+  private void recordClick(MotionEvent event) {
+    // First normalize our click position w.r.t. to the view display
+    float x = (event.getX() / viewGroup.getWidth());
+    float y = (event.getY() / viewGroup.getHeight());
+
+    // MediaPipe can automatically crop our camera stream when displaying it to
+    // our surface, which can throw off our touch point calulations. So we need
+    // to replicate that logic here. See FrameScaleMode::kFillAndCrop usage in
+    // gl_quad_renderer.cc for more details.
+    float widthRatio = (float) viewGroup.getWidth() / (float) cameraWidth;
+    float heightRatio = (float) viewGroup.getHeight() / (float) cameraHeight;
+
+    float maxRatio = max(widthRatio, heightRatio);
+    widthRatio /= maxRatio;
+    heightRatio /= maxRatio;
+
+    // Now we scale by the scale factors, and then reposition (since cropping
+    // is always centered)
+    x *= widthRatio;
+    x += 0.5f * (1.0f - widthRatio);
+    y *= heightRatio;
+    y += 0.5f * (1.0f - heightRatio);
+
+    // Finally, we can pass our adjusted x and y points to the StickerManager
+    currentSticker.setAnchorCoordinate(x, y);
+    stickerSentinel = currentSticker.getstickerId();
+  }
+
+  // Provided the X and Y coordinates of two points, the distance between them
+  // will be returned.
+  private static double getDistance(double x1, double y1, double x2, double y2) {
+    return Math.hypot((y2 - y1), (x2 - x1));
+  }
+
+  // Called upon each button click, and used to populate the buttonLayout with the
+  // current sticker data in addition to sticker controls (delete, remove, back).
+  private void refreshUi() {
+    if (currentSticker != null) { // No sticker in view
+      buttonLayout.removeAllViews();
+      ImageButton deleteSticker = new ImageButton(this);
+      setControlButtonDesign(deleteSticker, R.drawable.baseline_clear_24);
+      deleteSticker.setOnClickListener(
+          new View.OnClickListener() {
+            @Override
+            public void onClick(View v) {
+              if (currentSticker != null) {
+                stickerArrayList.remove(currentSticker);
+                currentSticker = null;
+                refreshUi();
+              }
+            }
+          });
+      // Go to home sticker menu
+      ImageButton goBack = new ImageButton(this);
+      setControlButtonDesign(goBack, R.drawable.baseline_arrow_back_24);
+      goBack.setOnClickListener(
+          new View.OnClickListener() {
+            @Override
+            public void onClick(View v) {
+              currentSticker = null;
+              refreshUi();
+            }
+          });
+      // Change sticker to next possible render
+      ImageButton loopRender = new ImageButton(this);
+      setControlButtonDesign(loopRender, R.drawable.baseline_loop_24);
+      loopRender.setOnClickListener(
+          new View.OnClickListener() {
+            @Override
+            public void onClick(View v) {
+              currentSticker.setRender(currentSticker.getRender().iterate());
+              refreshUi();
+            }
+          });
+      buttonLayout.addView(deleteSticker);
+      buttonLayout.addView(goBack);
+      buttonLayout.addView(loopRender);
+
+      // Add the GIF search option if current sticker is GIF
+      if (currentSticker.getRender() == StickerManager.Render.GIF) {
+        ImageButton gifSearch = new ImageButton(this);
+        setControlButtonDesign(gifSearch, R.drawable.baseline_search_24);
+        gifSearch.setOnClickListener(
+            new View.OnClickListener() {
+              @Override
+              public void onClick(View v) {
+                // Clear the text field to prevent text artifacts in GIF selection
+                editText.setText("");
+                // Open the Keyboard to allow user input
+                openKeyboard();
+              }
+            });
+        buttonLayout.addView(gifSearch);
+      }
+    } else {
+      buttonLayout.removeAllViews();
+      // Display stickers
+      for (final StickerManager sticker : stickerArrayList) {
+        final ImageButton stickerButton = new ImageButton(this);
+        stickerButton.setOnClickListener(
+            new View.OnClickListener() {
+              @Override
+              public void onClick(View v) {
+                currentSticker = sticker;
+                refreshUi();
+              }
+            });
+        if (sticker.getRender() == StickerManager.Render.GIF) {
+          setControlButtonDesign(stickerButton, R.drawable.asset_gif_preview);
+        } else if (sticker.getRender() == StickerManager.Render.ASSET_3D) {
+          setStickerButtonDesign(stickerButton, R.drawable.asset_3d_preview);
+        }
+
+        buttonLayout.addView(stickerButton);
+      }
+      ImageButton addSticker = new ImageButton(this);
+      setControlButtonDesign(addSticker, R.drawable.baseline_add_24);
+      addSticker.setOnClickListener(
+          new View.OnClickListener() {
+            @Override
+            public void onClick(View v) {
+              StickerManager newSticker = new StickerManager();
+              stickerArrayList.add(newSticker);
+              currentSticker = newSticker;
+              refreshUi();
+            }
+          });
+      ImageButton clearStickers = new ImageButton(this);
+      setControlButtonDesign(clearStickers, R.drawable.baseline_clear_all_24);
+      clearStickers.setOnClickListener(
+          new View.OnClickListener() {
+            @Override
+            public void onClick(View v) {
+              stickerArrayList.clear();
+              refreshUi();
+            }
+          });
+
+      buttonLayout.addView(addSticker);
+      buttonLayout.addView(clearStickers);
+    }
+  }
+
+  // Sets ImageButton UI for Control Buttons.
+  private void setControlButtonDesign(ImageButton btn, int imageDrawable) {
+    // btn.setImageDrawable(getResources().getDrawable(imageDrawable));
+    btn.setImageDrawable(getDrawable(imageDrawable));
+    btn.setBackgroundColor(Color.parseColor("#00ffffff"));
+    btn.setColorFilter(Color.parseColor("#0494a4"));
+    btn.setLayoutParams(new LinearLayout.LayoutParams(200, 200));
+    btn.setPadding(25, 25, 25, 25);
+    btn.setScaleType(ImageView.ScaleType.FIT_XY);
+  }
+
+  // Sets ImageButton UI for Sticker Buttons.
+  private void setStickerButtonDesign(ImageButton btn, int imageDrawable) {
+    btn.setImageDrawable(getDrawable(imageDrawable));
+    btn.setBackground(getDrawable(R.drawable.circle_button));
+    btn.setLayoutParams(new LinearLayout.LayoutParams(250, 250));
+    btn.setPadding(25, 25, 25, 25);
+    btn.setScaleType(ImageView.ScaleType.CENTER_INSIDE);
+  }
+
+  // Used to set ArrayList of Bitmap frames
+  private void setGIFBitmaps(String gifUrl) {
+    gifBitmaps = new ArrayList<>(); // Empty the bitmap array
+    Glide.with(this)
+        .asGif()
+        .load(gifUrl)
+        .into(
+            new CustomTarget<GifDrawable>() {
+              @Override
+              public void onLoadCleared(Drawable placeholder) {}
+
+              @Override
+              public void onResourceReady(
+                  GifDrawable resource, Transition<? super GifDrawable> transition) {
+                try {
+                  Object startConstant = resource.getConstantState();
+                  Field frameManager = startConstant.getClass().getDeclaredField("frameLoader");
+                  frameManager.setAccessible(true);
+                  Object frameLoader = frameManager.get(startConstant);
+                  Field decoder = frameLoader.getClass().getDeclaredField("gifDecoder");
+                  decoder.setAccessible(true);
+
+                  Object frameObject = (decoder.get(frameLoader));
+                  for (int i = 0; i < resource.getFrameCount(); i++) {
+                    frameObject.getClass().getMethod("advance").invoke(frameObject);
+                    Bitmap bmp =
+                        (Bitmap)
+                            frameObject.getClass().getMethod("getNextFrame").invoke(frameObject);
+                    gifBitmaps.add(flipHorizontal(bmp));
+                  }
+                } catch (Exception e) {
+                  Log.e(TAG, "", e);
+                }
+              }
+            });
+  }
+
+  // Bitmaps must be flipped due to native acquisition of frames from Android OS
+  private static Bitmap flipHorizontal(Bitmap bmp) {
+    Matrix matrix = new Matrix();
+    // Flip Bitmap frames horizontally
+    matrix.preScale(-1.0f, 1.0f);
+    return Bitmap.createBitmap(bmp, 0, 0, bmp.getWidth(), bmp.getHeight(), matrix, true);
+  }
+
+  // Function that is continuously called in order to time GIF frame updates
+  private void updateGIFFrame() {
+    long millisPerFrame = 1000 / GIF_FRAME_RATE;
+    if (System.currentTimeMillis() - gifLastFrameUpdateMS >= millisPerFrame) {
+      // Update GIF timestamp
+      gifLastFrameUpdateMS = System.currentTimeMillis();
+      // Cycle through every possible frame and avoid a divide by 0
+      gifCurrentIndex = gifBitmaps.isEmpty() ? 1 : (gifCurrentIndex + 1) % gifBitmaps.size();
+    }
+  }
+
+  // Called once to popup the Keyboard via Android OS with focus set to editText
+  private void openKeyboard() {
+    editText.requestFocus();
+    InputMethodManager imm = (InputMethodManager) getSystemService(Context.INPUT_METHOD_SERVICE);
+    imm.showSoftInput(editText, InputMethodManager.SHOW_IMPLICIT);
+  }
+
+  // Called once to close the Keyboard via Android OS
+  private void closeKeyboard() {
+    View view = this.getCurrentFocus();
+    if (view != null) {
+      InputMethodManager imm = (InputMethodManager) getSystemService(Context.INPUT_METHOD_SERVICE);
+      imm.hideSoftInputFromWindow(view.getWindowToken(), 0);
+    }
+  }
+
+  private void prepareDemoAssets() {
+    // We render from raw data with openGL, so disable decoding preprocessing
+    BitmapFactory.Options decodeOptions = new BitmapFactory.Options();
+    decodeOptions.inScaled = false;
+    decodeOptions.inDither = false;
+    decodeOptions.inPremultiplied = false;
+
+    try {
+      InputStream inputStream = getAssets().open(DEFAULT_GIF_TEXTURE);
+      defaultGIFTexture =
+          flipHorizontal(
+              BitmapFactory.decodeStream(inputStream, null /*outPadding*/, decodeOptions));
+      inputStream.close();
+    } catch (Exception e) {
+      Log.e(TAG, "Error parsing object texture; error: ", e);
+      throw new IllegalStateException(e);
+    }
+
+    try {
+      InputStream inputStream = getAssets().open(ASSET_3D_TEXTURE);
+      asset3dTexture = BitmapFactory.decodeStream(inputStream, null /*outPadding*/, decodeOptions);
+      inputStream.close();
+    } catch (Exception e) {
+      Log.e(TAG, "Error parsing object texture; error: ", e);
+      throw new IllegalStateException(e);
+    }
+  }
+
+  private class MediaPipePacketManager implements FrameProcessor.OnWillAddFrameListener {
+    @Override
+    public void onWillAddFrame(long timestamp) {
+      // set current GIF bitmap as default texture
+      Bitmap currentGIFBitmap = defaultGIFTexture;
+      // If current index is in bounds, display current frame
+      if (gifCurrentIndex <= gifBitmaps.size() - 1) {
+        currentGIFBitmap = gifBitmaps.get(gifCurrentIndex);
+      }
+      // Update to next GIF frame based on timing and frame rate
+      updateGIFFrame();
+
+      // Calculate and set the aspect ratio of the GIF
+      float gifAspectRatio =
+          (float) currentGIFBitmap.getWidth() / (float) currentGIFBitmap.getHeight();
+
+      Packet stickerSentinelPacket = processor.getPacketCreator().createInt32(stickerSentinel);
+      // Sticker sentinel value must be reset for next graph iteration
+      stickerSentinel = -1;
+      // Initialize sticker data protobufferpacket information
+      Packet stickerProtoDataPacket =
+          processor
+              .getPacketCreator()
+              .createSerializedProto(StickerManager.getMessageLiteData(stickerArrayList));
+      // Define and set the IMU sensory information float array
+      Packet imuDataPacket = processor.getPacketCreator().createFloat32Array(rotationMatrix);
+      // Communicate GIF textures (dynamic texturing) to graph
+      Packet gifTexturePacket = processor.getPacketCreator().createRgbaImageFrame(currentGIFBitmap);
+      Packet gifAspectRatioPacket = processor.getPacketCreator().createFloat32(gifAspectRatio);
+      processor
+          .getGraph()
+          .addConsumablePacketToInputStream(STICKER_SENTINEL_TAG, stickerSentinelPacket, timestamp);
+      processor
+          .getGraph()
+          .addConsumablePacketToInputStream(STICKER_PROTO_TAG, stickerProtoDataPacket, timestamp);
+      processor
+          .getGraph()
+          .addConsumablePacketToInputStream(IMU_MATRIX_TAG, imuDataPacket, timestamp);
+      processor
+          .getGraph()
+          .addConsumablePacketToInputStream(GIF_TEXTURE_TAG, gifTexturePacket, timestamp);
+      processor
+          .getGraph()
+          .addConsumablePacketToInputStream(GIF_ASPECT_RATIO_TAG, gifAspectRatioPacket, timestamp);
+      stickerSentinelPacket.release();
+      stickerProtoDataPacket.release();
+      imuDataPacket.release();
+      gifTexturePacket.release();
+      gifAspectRatioPacket.release();
+    }
+  }
+}
@@ -0,0 +1,191 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.mediapipe.apps.instantmotiontracking;
+
+import com.google.mediapipe.graphs.instantmotiontracking.StickerBufferProto.Sticker;
+import com.google.mediapipe.graphs.instantmotiontracking.StickerBufferProto.StickerRoll;
+import java.util.ArrayList;
+
+/**
+ * This class represents a single sticker object placed in the
+ * instantmotiontracking system. StickerManagers represent a unique object to render
+ * and manipulate in an AR scene.
+ * <p>A sticker has a sticker_id (a unique integer identifying a sticker object
+ * to render), x and y normalized anchor coordinates [0.0-1.0], user inputs for
+ * rotation in radians, scaling, and a renderID (another unique integer which
+ * determines what object model to render for this unique sticker).
+ */
+public class StickerManager {
+
+  /** All types of possible objects to render for our application. */
+  public enum Render {
+    // Every possible render for a sticker object
+    GIF,
+    ASSET_3D;
+
+    /**
+     * Once called, will set the value of the current render to the next
+     * possible Render available. If all possible Renders have been iterated
+     * through, the function will loop and set to the first available Render.
+     */
+    public Render iterate() {
+      int newEnumIdx = (this.ordinal() + 1) % Render.values().length;
+      return Render.values()[newEnumIdx];
+   }
+  }
+
+  // Current render of the sticker object
+  private Render currentRender;
+
+  // Normalized X and Y coordinates of anchor
+  // (0,0) lies at top-left corner of screen
+  // (1.0,1.0) lies at bottom-right corner of screen
+  private float anchorX;
+  private float anchorY;
+
+  // Rotation in radians from user
+  private float userRotation = 0f;
+  // Scaling factor as defined by user (defaults to 1.0)
+  private float userScalingFactor = 1f;
+
+  // Unique sticker integer ID
+  private final int stickerId;
+
+  // Used to determine next stickerId
+  private static int globalIDLimit = 1;
+
+  /**
+   * Used to create a StickerManager object with a newly generated stickerId and a
+   * default Render of the first possible render in our Render enum.
+   */
+  public StickerManager() {
+    // Every sticker will have a default render of the first 3D asset
+    this.currentRender = Render.values()[1];
+    // StickerManager will render out of view by default
+    this.setAnchorCoordinate(2.0f, 2.0f);
+    // Set the global sticker ID limit for the next sticker
+    stickerId = StickerManager.globalIDLimit++;
+  }
+
+  /**
+   * Used to create a StickerManager object with a newly generated stickerId.
+   *
+   * @param render initial Render of the new StickerManager object
+   */
+  public StickerManager(Render render) {
+    this.currentRender = render;
+    // StickerManager will render out of view by default
+    this.setAnchorCoordinate(2.0f, 2.0f);
+    // Set the global sticker ID limit for the next sticker
+    stickerId = StickerManager.globalIDLimit++;
+  }
+
+  /**
+   * Used to get the sticker ID of the object.
+   *
+   * @return integer of the unique sticker ID
+   */
+  public int getstickerId() {
+    return this.stickerId;
+  }
+
+  /**
+   * Used to update or reset the anchor positions in normalized [0.0-1.0]
+   * coordinate space for the sticker object.
+   *
+   * @param normalizedX normalized X coordinate for the new anchor position
+   * @param normalizedY normalized Y coordinate for the new anchor position
+   */
+  public void setAnchorCoordinate(float normalizedX, float normalizedY) {
+    this.anchorX = normalizedX;
+    this.anchorY = normalizedY;
+  }
+
+  /** Returns the normalized X anchor coordinate of the sticker object. */
+  public float getAnchorX() {
+    return anchorX;
+  }
+
+  /** Returns the normalized Y anchor coordinate of the sticker object. */
+  public float getAnchorY() {
+    return anchorY;
+  }
+
+  /** Returns current asset to be rendered for this sticker object. */
+  public Render getRender() {
+    return currentRender;
+  }
+
+  /** Set render for this sticker object */
+  public void setRender(Render render) {
+    this.currentRender = render;
+  }
+
+  /**
+   * Sets new user value of rotation radians. This rotation is not cumulative,
+   * and must be set to an absolute value of rotation applied to the object.
+   *
+   * @param radians specified radians to rotate the sticker object by
+   */
+  public void setRotation(float radians) {
+    this.userRotation = radians;
+  }
+
+  /** Returns current user radian rotation setting. */
+  public float getRotation() {
+    return this.userRotation;
+  }
+
+  /**
+   * Sets new user scale factor. This factor will be proportional to the scale
+   * of the sticker object.
+   *
+   * @param scaling scale factor to be applied
+   */
+  public void setScaleFactor(float scaling) {
+    this.userScalingFactor = scaling;
+  }
+
+  /** Returns current user scale factor setting. */
+  public float getScaleFactor() {
+    return this.userScalingFactor;
+  }
+
+  /**
+   * This method converts an ArrayList of stickers to a MessageLite object
+   * which can be passed directly to the MediaPipe graph.
+   *
+   * @param stickerArrayList ArrayList of StickerManager objects to convert to data string
+   * @return MessageLite protobuffer of all sticker data
+   */
+  public static StickerRoll getMessageLiteData(
+    ArrayList<StickerManager> stickerArrayList) {
+    StickerRoll.Builder stickerRollBuilder
+      = StickerRoll.newBuilder();
+    for (final StickerManager sticker : stickerArrayList) {
+      Sticker protoSticker =
+          Sticker.newBuilder()
+              .setId(sticker.getstickerId())
+              .setX(sticker.getAnchorX())
+              .setY(sticker.getAnchorY())
+              .setRotation(sticker.getRotation())
+              .setScale(sticker.getScaleFactor())
+              .setRenderId(sticker.getRender().ordinal())
+              .build();
+      stickerRollBuilder.addSticker(protoSticker);
+    }
+    return stickerRollBuilder.build();
+  }
+}
@@ -0,0 +1,21 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+licenses(["notice"])
+
+package(default_visibility = ["//visibility:public"])
+
+exports_files(
+    srcs = glob(["**"]),
+)
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
MediaPipe Teamandchuoling	c828392681	Project import generated by Copybara. GitOrigin-RevId: f9a66589eaf652bb93f8e37ed9e4da26e59ef214	2020-10-19 13:23:25 -04:00
MediaPipe Teamandchuoling	cccf6244d3	Project import generated by Copybara. GitOrigin-RevId: a67c2c19ade0ba122816a250ecee3aab490ef16b	2020-09-28 01:22:05 -04:00
MediaPipe Teamandchuoling	8f69af91fe	Project import generated by Copybara. GitOrigin-RevId: 947096e4fc99d6b974e9f50d360d7c0a75072c5d	2020-09-17 11:52:51 -04:00
MediaPipe Teamandchuoling	a908d668c7	Project import generated by Copybara. GitOrigin-RevId: 4cee4a2c2317fb190680c17e31ebbb03bb73b71c	2020-09-17 11:09:17 -04:00
MediaPipe Teamandjqtang	1db91b550a	Project import generated by Copybara. GitOrigin-RevId: c27bb068d24890a4e3b7d2d536267e08d7e4abbd	2020-08-31 16:03:43 -07:00
MediaPipe Teamandchuoling	c0124fb83c	Project import generated by Copybara. GitOrigin-RevId: 612e50bb8db2ec3dc1c30049372d87a80c3848db	2020-08-30 19:52:55 -04:00
MediaPipe Teamandchuoling	a7225b938a	Project import generated by Copybara. GitOrigin-RevId: aaca5c37abcf8b7a6c3c28804739afdbad46e704	2020-08-13 15:10:54 -04:00