Project import generated by Copybara.

PiperOrigin-RevId: 253489161

mediapipe/docs/Makefile

@@ -0,0 +1,21 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	rm -rf ./_build
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

mediapipe/docs/README.md

@@ -0,0 +1,2 @@
+This directory contains the source markdown files presented on
+the [MediaPipe Read-the-Docs](https://mediapipe.readthedocs.io) documentation site.

mediapipe/docs/calculator.md

@@ -0,0 +1,165 @@
+## Building MediaPipe Calculators
+
+-   [Example calculator](#example-calculator)
+
+
+### Example calculator
+
+This section discusses the implementation of `PacketClonerCalculator`, which
+does a relatively simple job, and is used in many calculator graphs.
+`PacketClonerCalculator` simply produces a copy of its most recent input
+packets on demand.
+
+`PacketClonerCalculator` is useful when the timestamps of arriving data packets
+are not aligned perfectly. Suppose we have a room with a microphone, light
+sensor and a video camera that is collecting sensory data. Each of the sensors
+operates independently and collects data intermittently. Suppose that the output
+of each sensor is:
+
+*   microphone = loudness in decibels of sound in the room (Integer)
+*   light sensor = brightness of room (Integer)
+*   video camera = RGB image frame of room (ImageFrame)
+
+Our simple perception pipeline is designed to process sensory data from these 3
+sensors such that at any time when we have image frame data from the camera that
+is synchronized with the last collected microphone loudness data and light
+sensor brightness data. To do this with MediaPipe, our perception pipeline has 3
+input streams:
+
+*   room_mic_signal - Each packet of data in this input stream is integer data
+    representing how loud audio is in a room with timestamp.
+*   room_lightening_sensor - Each packet of data in this input stream is integer
+    data representing how bright is the room illuminated with timestamp.
+*   room_video_tick_signal - Each packet of data in this input stream is
+    imageframe of video data representing video collected from camera in the
+    room with timestamp.
+
+Below is the implementation of the `PacketClonerCalculator`.  You can see
+the `GetContract()`, `Open()`, and `Process()` methods as well as the instance
+variable `current_` which holds the most recent input packets.
+
+```c++
+// This takes packets from N+1 streams, A_1, A_2, ..., A_N, B.
+// For every packet that appears in B, outputs the most recent packet from each
+// of the A_i on a separate stream.
+
+#include <vector>
+
+#include "absl/strings/str_cat.h"
+#include "mediapipe/framework/calculator_framework.h"
+
+namespace mediapipe {
+
+// For every packet received on the last stream, output the latest packet
+// obtained on all other streams. Therefore, if the last stream outputs at a
+// higher rate than the others, this effectively clones the packets from the
+// other streams to match the last.
+//
+// Example config:
+// node {
+//   calculator: "PacketClonerCalculator"
+//   input_stream: "first_base_signal"
+//   input_stream: "second_base_signal"
+//   input_stream: "tick_signal"
+//   output_stream: "cloned_first_base_signal"
+//   output_stream: "cloned_second_base_signal"
+// }
+//
+class PacketClonerCalculator : public CalculatorBase {
+ public:
+  static ::mediapipe::Status GetContract(CalculatorContract* cc) {
+    const int tick_signal_index = cc->Inputs().NumEntries() - 1;
+    // cc->Inputs().NumEntries() returns the number of input streams
+    // for the PacketClonerCalculator
+    for (int i = 0; i < tick_signal_index; ++i) {
+      cc->Inputs().Index(i).SetAny();
+      // cc->Inputs().Index(i) returns the input stream pointer by index
+      cc->Outputs().Index(i).SetSameAs(&cc->Inputs().Index(i));
+    }
+    cc->Inputs().Index(tick_signal_index).SetAny();
+    return ::mediapipe::OkStatus();
+  }
+
+  ::mediapipe::Status Open(CalculatorContext* cc) final {
+    tick_signal_index_ = cc->Inputs().NumEntries() - 1;
+    current_.resize(tick_signal_index_);
+    // Pass along the header for each stream if present.
+    for (int i = 0; i < tick_signal_index_; ++i) {
+      if (!cc->Inputs().Index(i).Header().IsEmpty()) {
+        cc->Outputs().Index(i).SetHeader(cc->Inputs().Index(i).Header());
+        // Sets the output stream of index i header to be the same as
+        // the header for the input stream of index i
+      }
+    }
+    return ::mediapipe::OkStatus();
+  }
+
+  ::mediapipe::Status Process(CalculatorContext* cc) final {
+    // Store input signals.
+    for (int i = 0; i < tick_signal_index_; ++i) {
+      if (!cc->Inputs().Index(i).Value().IsEmpty()) {
+        current_[i] = cc->Inputs().Index(i).Value();
+      }
+    }
+
+    // Output if the tick signal is non-empty.
+    if (!cc->Inputs().Index(tick_signal_index_).Value().IsEmpty()) {
+      for (int i = 0; i < tick_signal_index_; ++i) {
+        if (!current_[i].IsEmpty()) {
+          cc->Outputs().Index(i).AddPacket(
+              current_[i].At(cc->InputTimestamp()));
+          // Add a packet to output stream of index i a packet from inputstream i
+          // with timestamp common to all present inputs
+          //
+        } else {
+          cc->Outputs().Index(i).SetNextTimestampBound(
+              cc->InputTimestamp().NextAllowedInStream());
+          // if current_[i], 1 packet buffer for input stream i is empty, we will set
+          // next allowed timestamp for input stream i to be current timestamp + 1
+        }
+      }
+    }
+    return ::mediapipe::OkStatus();
+  }
+
+ private:
+  std::vector<Packet> current_;
+  int tick_signal_index_;
+};
+
+REGISTER_CALCULATOR(PacketClonerCalculator);
+}  // namespace mediapipe
+```
+
+Typically, a calculator has only a .cc file. No .h is required, because
+mediapipe uses registration to make calculators known to it. After you have
+defined your calculator class, register it with a macro invocation
+REGISTER_CALCULATOR(calculator_class_name).
+
+Below is a trivial MediaPipe graph that has 3 input streams, 1 node
+(PacketClonerCalculator) and 3 output streams.
+
+```proto
+input_stream: "room_mic_signal"
+input_stream: "room_lighting_sensor"
+input_stream: "room_video_tick_signal"
+
+node {
+   calculator: "PacketClonerCalculator"
+   input_stream: "room_mic_signal"
+   input_stream: "room_lighting_sensor"
+   input_stream: "room_video_tick_signal"
+   output_stream: "cloned_room_mic_signal"
+   output_stream: "cloned_lighting_sensor"
+   output_stream: "cloned_video_tick_signal"
+ }
+```
+
+The diagram below shows how the `PacketClonerCalculator` defines its output
+packets based on its series of input packets.
+
+| ![Graph using PacketClonerCalculator](images/packet_cloner_calculator.png) |
+|:--:|
+| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams.  The sequence of output packets is determined by the sequene of input packets and their timestamps. The timestamps are shows along the right side of the diagram.* |
+
+

mediapipe/docs/concepts.md

@@ -0,0 +1,92 @@
+# MediaPipe Concepts
+
+## The basics
+
+### Packet
+
+The basic data flow unit. A packet consists of a numeric timestamp and a shared pointer to an **immutable** payload. The payload can be of any C++ type, and the payload's type is also referred to as the type of the packet. Packets are value classes and can be copied cheaply. Each copy shares ownership of the payload, with reference-counting semantics. Each copy has its own timestamp. [Details](packets.md).
+
+### Graph
+
+MediaPipe processing takes place inside a graph, which defines packet flow paths
+between **nodes**. A graph can have any number of inputs and outputs, and data
+flow can branch and merge. Generally data flows forward, but
+[backward loops](cycles.md) are possible.
+
+### Nodes
+
+Nodes produce and/or consume packets, and they are where the bulk of the graph’s
+work takes place. They are also known as “calculators”, for historical reasons.
+Each node’s interface defines a number of input and output **ports**, identified by
+a tag and/or an index.
+
+### Streams
+
+A stream is a connection between two nodes that carries a sequence of packets,
+whose timestamps must be monotonically increasing.
+
+### Side packets
+
+A side packet connection between nodes carries a single packet (with unspecified
+timestamp). It can be used to provide some data that will remain constant,
+whereas a stream represents a flow of data that changes over time.
+
+### Packet Ports
+
+A port has an associated type; packets transiting through the port must be of
+that type. An output stream port can be connected to any number of
+input stream ports of the same type; each consumer receives a separate copy of
+the output packets, and has its own queue, so it can consume them at its own
+pace. Similarly, a side packet output port can be connected to as many side
+packet input ports as desired.
+
+A port can be required, meaning that a connection must be made for the graph to
+be valid, or optional, meaning it may remain unconnected.
+
+Note: even if a stream connection is required, the stream may not carry a packet for all timestamps.
+
+## Input and output
+
+Data flow can originate from **source nodes**, which have no input streams and
+produce packets spontaneously (e.g. by reading from a file); or from **graph input streams**, which let an application feed packets into a graph.
+
+Similarly, there are **sink nodes** that receive data and write it to various
+destinations (e.g. a file, a memory buffer, etc.), and an application can also
+receive output from the graph using **callbacks**.
+
+## Runtime behavior
+
+### Graph lifetime
+
+Once a graph has been initialized, it can be **started** to begin processing
+data, and can process a stream of packets until each stream is closed or the
+graph is **canceled**. Then the graph can be destroyed or **started** again.
+
+### Node lifetime
+
+There are three main lifetime methods the framework will call on a node:
+
+-   Open: called once, before the other methods. When it is called, all input
+    side packets required by the node will be available.
+-   Process: called multiple times, when a new set of inputs is available,
+    according to the node’s input policy.
+-   Close: called once, at the end.
+
+In addition, each calculator can define constructor and destructor, which are
+useful for creating and deallocating resources that are independent of the
+processed data.
+
+### Input policies
+
+The default input policy is deterministic collation of packets by timestamp. A node receives
+all inputs for the same timestamp at the same time, in an invocation of its
+Process method; and successive input sets are received in their timestamp order. This can
+require delaying the processing of some packets until a packet with the same
+timestamp is received on all input streams, or until it can be guaranteed that a
+packet with that timestamp will not be arriving on the streams that have not
+received it.
+
+Other policies are also available, implemented using a separate kind of
+component known as an InputStreamHandler.
+
+See [scheduling](scheduling_sync.md) for more details.

mediapipe/docs/conf.py

@@ -0,0 +1,57 @@
+"""Configuration file for the Sphinx documentation builder.
+
+This file only contains a selection of the most common options.
+For a full list see the documentation:
+http://www.sphinx-doc.org/en/master/config
+-- Path setup --------------------------------------------------------------
+If extensions (or modules to document with autodoc) are in another directory,
+add these directories to sys.path here.
+If the directory is relative to the documentation root,
+use os.path.abspath to make it absolute, like shown here.
+
+"""
+import sphinx_rtd_theme
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'MediaPipe'
+author = 'Google LLC'
+
+# The full version, including alpha/beta/rc tags
+release = 'v0.5'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'recommonmark'
+]
+
+master_doc = 'index'
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']

mediapipe/docs/cycles.md

@@ -0,0 +1,128 @@
+# Cycles in MediaPipe Graphs
+
+<!-- TODO: add discussion of PreviousLoopbackCalculator -->
+
+[TOC]
+
+By default, MediaPipe requires calculator graphs to be acyclic and treats cycles
+in a graph as errors. If a graph is intended to have cycles, the cycles need to
+be annotated in the graph config. This page describes how to do that.
+
+NOTE: The current approach is experimental and subject to change. We welcome
+your feedback.
+
+Please use the `CalculatorGraphTest.Cycle` unit test in
+`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown
+below is the cyclic graph in the test. The `sum` output of the adder is the sum
+of the integers generated by the integer source calculator.
+
+![a cyclic graph that adds a stream of integers](images/cyclic_integer_sum_graph.svg "A cyclic graph")
+
+This simple graph illustrates all the issues in supporting cyclic graphs.
+
+## Back Edge Annotation
+
+We require that an edge in each cycle be annotated as a back edge. This allows
+MediaPipe’s topological sort to work, after removing all the back edges.
+
+There are usually multiple ways to select the back edges. Which edges are marked
+as back edges affects which nodes are considered as upstream and which nodes are
+considered as downstream, which in turn affects the priorities MediaPipe assigns
+to the nodes.
+
+For example, the `CalculatorGraphTest.Cycle` test marks the `old_sum` edge as a
+back edge, so the Delay node is considered as a downstream node of the adder
+node and is given a higher priority. Alternatively, we could mark the `sum`
+input to the delay node as the back edge, in which case the delay node would be
+considered as an upstream node of the adder node and is given a lower priority.
+
+## Initial Packet
+
+For the adder calculator to be runnable when the first integer from the integer
+source arrives, we need an initial packet, with value 0 and with the same
+timestamp, on the `old_sum` input stream to the adder. This initial packet
+should be output by the delay calculator in the `Open()` method.
+
+## Delay in a Loop
+
+Each loop should incur a delay to align the previous `sum` output with the next
+integer input. This is also done by the delay node. So the delay node needs to
+know the following about the timestamps of the integer source calculator:
+
+*   The timestamp of the first output.
+
+*   The timestamp delta between successive outputs.
+
+We plan to add an alternative scheduling policy that only cares about packet
+ordering and ignores packet timestamps, which will eliminate this inconvenience.
+
+## Early Termination of a Calculator When One Input Stream is Done
+
+By default, MediaPipe calls the `Close()` method of a non-source calculator when
+all of its input streams are done. In the example graph, we want to stop the
+adder node as soon as the integer source is done. This is accomplished by
+configuring the adder node with an alternative input stream hander,
+`EarlyCloseInputStreamHandler`.
+
+## Relevant Source Code
+
+### Delay Calculator
+
+Note the code in `Open()` that outputs the initial packet and the code in
+`Process()` that adds a (unit) delay to input packets. As noted above, this
+delay node assumes that its output stream is used alongside an input stream with
+packet timestamps 0, 1, 2, 3, ...
+
+```c++
+class UnitDelayCalculator : public Calculator {
+ public:
+  static ::util::Status FillExpectations(
+      const CalculatorOptions& extendable_options, PacketTypeSet* inputs,
+      PacketTypeSet* outputs, PacketTypeSet* input_side_packets) {
+    inputs->Index(0)->Set<int>("An integer.");
+    outputs->Index(0)->Set<int>("The input delayed by one time unit.");
+    return ::mediapipe::OkStatus();
+  }
+
+  ::util::Status Open() final {
+    Output()->Add(new int(0), Timestamp(0));
+    return ::mediapipe::OkStatus();
+  }
+
+  ::util::Status Process() final {
+    const Packet& packet = Input()->Value();
+    Output()->AddPacket(packet.At(packet.Timestamp().NextAllowedInStream()));
+    return ::mediapipe::OkStatus();
+  }
+};
+```
+
+### Graph Config
+
+Note the `back_edge` annotation and the alternative `input_stream_handler`.
+
+```proto
+node {
+  calculator: 'GlobalCountSourceCalculator'
+  input_side_packet: 'global_counter'
+  output_stream: 'integers'
+}
+node {
+  calculator: 'IntAdderCalculator'
+  input_stream: 'integers'
+  input_stream: 'old_sum'
+  input_stream_info: {
+    tag_index: ':1'  # 'old_sum'
+    back_edge: true
+  }
+  output_stream: 'sum'
+  input_stream_handler {
+    input_stream_handler: 'EarlyCloseInputStreamHandler'
+  }
+}
+node {
+  calculator: 'UnitDelayCalculator'
+  input_stream: 'sum'
+  output_stream: 'old_sum'
+}
+```

mediapipe/docs/examples.md

@@ -0,0 +1,73 @@
+# Examples
+
+Below are code samples on how to run MediaPipe on both mobile and desktop. We
+currently support MediaPipe APIs on mobile for Android only but will add support
+for Objective-C shortly.
+
+## Mobile
+
+### Hello World! on Android
+
+[Hello World! on Android](./hello_world_android.md) should be the first mobile
+example users go through in detail. It teaches the following:
+
+*   Introduction of a simple MediaPipe graph running on mobile GPUs for
+    [Sobel edge detection].
+*   Building a simple baseline Android application that displays "Hello World!".
+*   Adding camera preview support into the baseline application using the
+    Android [CameraX] API.
+*   Incorporating the Sobel edge detection graph to process the live camera
+    preview and display the processed video in real-time.
+
+### Object Detection with GPU on Android
+
+[Object Detection on GPU on Android](./object_detection_android_gpu.md)
+illustrates how to use MediaPipe with a TFLite model for object detection in a
+GPU-accelerated pipeline.
+
+### Object Detection with CPU on Android
+
+[Object Detection on CPU on Android](./object_detection_android_cpu.md)
+illustrates using the same TFLite model in a CPU-based pipeline. This example
+highlights how graphs can be easily adapted to run on CPU v.s. GPU.
+
+### Face Detection on Android
+
+[Face Detection on Android](./face_detection_android_gpu.md) illustrates how to
+use MediaPipe with a TFLite model for face detection in a GPU-accelerated
+pipeline.
+
+*   The selfie face detection TFLite model is based on
+    ["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface).
+*   [Model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
+
+### Hair Segmentation on Android
+
+[Hair Segmentation on Android](./hair_segmentation_android_gpu.md) illustrates
+how to use MediaPipe with a TFLite model for hair segmentation in a
+GPU-accelerated pipeline.
+
+*   The selfie hair segmentation TFLite model is based on
+    ["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation).
+*   [Model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
+
+## Desktop
+
+### Hello World for C++
+
+[Hello World for C++](./hello_world_desktop.md) shows how to run a simple graph
+using the MediaPipe C++ APIs.
+
+### Preparing Data Sets with MediaSequence
+
+[Preparing Data Sets with MediaSequence](./media_sequence.md) shows how to use
+MediaPipe for media processing to prepare video data sets for training a
+TensorFlow model.
+
+### Object Detection on Desktop
+
+[Object Detection on Desktop](./object_detection_desktop.md) shows how to run
+object detection models (TensorFlow and TFLite) using the MediaPipe C++ APIs.
+
+[Sobel edge detection]:https://en.wikipedia.org/wiki/Sobel_operator
+[CameraX]:https://developer.android.com/training/camerax

mediapipe/docs/face_detection_android_gpu.md

@@ -0,0 +1,231 @@
+# Face Detection on Android
+
+Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
+general instructions to develop an Android application that uses MediaPipe. This
+doc focuses on the
+[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_android_gpu.pbtxt)
+that performs face detection with TensorFlow Lite on GPU.
+
+![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif){width="300"}
+
+## App
+
+The graph is used in the
+[Face Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu)
+example app. To build the app, run:
+
+```bash
+bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu
+```
+
+To further install the app on android device, run:
+
+```bash
+adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.apk
+```
+
+## Graph
+
+![face_detection_android_gpu_graph](images/mobile/face_detection_android_gpu.png){width="400"}
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into [MediaPipe Visualizer](https://mediapipe-viz.appspot.com/).
+
+```bash
+# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
+# Used in the example in
+# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectiongpu.
+
+# Images on GPU coming into and out of the graph.
+input_stream: "input_video"
+output_stream: "output_video"
+
+# Throttles the images flowing downstream for flow control. It passes through
+# the very first incoming image unaltered, and waits for
+# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
+# generating the corresponding detections before it passes through another
+# image. All images that come in while waiting are dropped, limiting the number
+# of in-flight images between this calculator and
+# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
+# from queuing up incoming images and data excessively, which leads to increased
+# latency and memory usage, unwanted in real-time mobile applications. It also
+# eliminates unnecessarily computation, e.g., a transformed image produced by
+# ImageTransformationCalculator may get dropped downstream if the subsequent
+# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
+# processing previous inputs.
+node {
+  calculator: "RealTimeFlowLimiterCalculator"
+  input_stream: "input_video"
+  input_stream: "FINISHED:detections"
+  input_stream_info: {
+    tag_index: "FINISHED"
+    back_edge: true
+  }
+  output_stream: "throttled_input_video"
+}
+
+# Transforms the input image on GPU to a 320x320 image. To scale the image, by
+# default it uses the STRETCH scale mode that maps the entire input image to the
+# entire transformed image. As a result, image aspect ratio may be changed and
+# objects in the image may be deformed (stretched or squeezed), but the object
+# detection model used in this graph is agnostic to that deformation.
+node: {
+  calculator: "ImageTransformationCalculator"
+  input_stream: "IMAGE_GPU:throttled_input_video"
+  output_stream: "IMAGE_GPU:transformed_input_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
+      output_width: 320
+      output_height: 320
+    }
+  }
+}
+
+# Converts the transformed input image on GPU into an image tensor stored in
+# tflite::gpu::GlBuffer. The zero_center option is set to true to normalize the
+# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f]. The flip_vertically
+# option is set to true to account for the descrepancy between the
+# representation of the input image (origin at the bottom-left corner, the
+# OpenGL convention) and what the model used in this graph is expecting (origin
+# at the top-left corner).
+node {
+  calculator: "TfLiteConverterCalculator"
+  input_stream: "IMAGE_GPU:transformed_input_video"
+  output_stream: "TENSORS_GPU:image_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
+      zero_center: true
+      flip_vertically: true
+    }
+  }
+}
+
+# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
+# vector of tensors representing, for instance, detection boxes/keypoints and
+# scores.
+node {
+  calculator: "TfLiteInferenceCalculator"
+  input_stream: "TENSORS_GPU:image_tensor"
+  output_stream: "TENSORS_GPU:detection_tensors"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+      model_path: "ssdlite_object_detection.tflite"
+    }
+  }
+}
+
+# Generates a single side packet containing a vector of SSD anchors based on
+# the specification in the options.
+node {
+  calculator: "SsdAnchorsCalculator"
+  output_side_packet: "anchors"
+  node_options: {
+    [type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
+      num_layers: 6
+      min_scale: 0.2
+      max_scale: 0.95
+      input_size_height: 320
+      input_size_width: 320
+      anchor_offset_x: 0.5
+      anchor_offset_y: 0.5
+      strides: 16
+      strides: 32
+      strides: 64
+      strides: 128
+      strides: 256
+      strides: 512
+      aspect_ratios: 1.0
+      aspect_ratios: 2.0
+      aspect_ratios: 0.5
+      aspect_ratios: 3.0
+      aspect_ratios: 0.3333
+      reduce_boxes_in_lowest_layer: true
+    }
+  }
+}
+
+# Decodes the detection tensors generated by the TensorFlow Lite model, based on
+# the SSD anchors and the specification in the options, into a vector of
+# detections. Each detection describes a detected object.
+node {
+  calculator: "TfLiteTensorsToDetectionsCalculator"
+  input_stream: "TENSORS_GPU:detection_tensors"
+  input_side_packet: "ANCHORS:anchors"
+  output_stream: "DETECTIONS:detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
+      num_classes: 91
+      num_boxes: 2034
+      num_coords: 4
+      ignore_classes: 0
+      sigmoid_score: true
+      apply_exponential_on_box_size: true
+      x_scale: 10.0
+      y_scale: 10.0
+      h_scale: 5.0
+      w_scale: 5.0
+      flip_vertically: true
+    }
+  }
+}
+
+# Performs non-max suppression to remove excessive detections.
+node {
+  calculator: "NonMaxSuppressionCalculator"
+  input_stream: "detections"
+  output_stream: "filtered_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
+      min_suppression_threshold: 0.4
+      min_score_threshold: 0.6
+      max_num_detections: 3
+      overlap_type: INTERSECTION_OVER_UNION
+    }
+  }
+}
+
+# Maps detection label IDs to the corresponding label text. The label map is
+# provided in the label_map_path option.
+node {
+  calculator: "DetectionLabelIdToTextCalculator"
+  input_stream: "filtered_detections"
+  output_stream: "output_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
+      label_map_path: "ssdlite_object_detection_labelmap.txt"
+    }
+  }
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTION_VECTOR:output_detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the original image coming into
+# the graph. Annotation drawing is performed on CPU, and the result is
+# transferred to GPU and overlaid on the input image. The calculator assumes
+# that image origin is always at the top-left corner and renders text
+# accordingly. However, the input image has its origin at the bottom-left corner
+# (OpenGL convention) and the flip_text_vertically option is set to true to
+# compensate that.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "INPUT_FRAME_GPU:throttled_input_video"
+  input_stream: "render_data"
+  output_stream: "OUTPUT_FRAME_GPU:output_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
+      flip_text_vertically: true
+    }
+  }
+}
+```

mediapipe/docs/framework_concepts.md

@@ -0,0 +1,436 @@
+## Framework Concepts
+
+- [CalculatorBase](#calculatorbase)
+- [Life of a Calculator](#life-of-a-calculator)
+- [Identifying inputs and outputs](#identifying-inputs-and-outputs)
+- [Processing](#processing)
+- [GraphConfig](#graphconfig)
+- [Subgraph](#subgraph)
+
+Each calculator is a node of a graph. We describe how to create a new
+calculator, how to initialize a calculator, how to perform its calculations,
+input and output streams, timestamps, and options. Each node in the graph is
+implemented as a `Calculator`. The bulk of graph execution happens inside its
+calculators. A calculator may receive zero or more input streams and/or side
+packets and produces zero or more output streams and/or side packets.
+
+### CalculatorBase
+
+A calculator is created by defining a new sub-class of the
+[`CalculatorBase`](http://github.com/google/mediapipe/mediapipe/framework/calculator_base.cc)
+class, implementing a number of methods, and registering the new sub-class with
+Mediapipe. At a minimum, a new calculator must implement the below four methods
+
+* `GetContract()`
+  * Calculator authors can specify the expected types of inputs and outputs of a calculator in GetContract(). When a graph is initialized, the framework calls a static method to verify if the packet types of the connected inputs and outputs match the information in this specification.
+* `Open()`
+  * After a graph starts, the framework calls `Open()`. The input side packets are available to the calculator at this point. `Open()` interprets the node configuration (see Section \ref{graph_config}) operations and prepares the calculator's per-graph-run state. This function may also write packets to calculator outputs. An error during `Open()` can terminate the graph run.
+* `Process()`
+  * For a calculator with inputs, the framework calls `Process()` repeatedly whenever at least one input stream has a packet available. The framework by default guarantees that all inputs have the same timestamp (see Section \ref{scheduling} for more information). Multiple `Process()` calls can be invoked simultaneously when parallel execution is enabled. If an error occurs during `Process()`, the framework calls `Close()` and the graph run terminates.
+* `Close()`
+  * After all calls to `Process()` finish or when all input streams close, the framework calls `Close()`. This function is always called if `Open()` was called and succeeded and even if the graph run terminated because of an error. No inputs are available via any input streams during `Close()`, but it still has access to input side packets and therefore may write outputs. After `Close()` returns, the calculator should be considered a dead node. The calculator object is destroyed as soon as the graph finishes running.
+
+The following are code snippets from
+[CalculatorBase.h](http://github.com/google/mediapipe/mediapipe/framework/calculator_base.h).
+
+```c++
+class CalculatorBase {
+ public:
+  ...
+
+  // The subclasses of CalculatorBase must implement GetContract.
+  // ...
+  static ::MediaPipe::Status GetContract(CalculatorContract* cc);
+
+  // Open is called before any Process() calls, on a freshly constructed
+  // calculator.  Subclasses may override this method to perform necessary
+  // setup, and possibly output Packets and/or set output streams' headers.
+  // ...
+  virtual ::MediaPipe::Status Open(CalculatorContext* cc) {
+    return ::MediaPipe::OkStatus();
+  }
+
+  // Processes the incoming inputs. May call the methods on cc to access
+  // inputs and produce outputs.
+  // ...
+  virtual ::MediaPipe::Status Process(CalculatorContext* cc) = 0;
+
+  // Is called if Open() was called and succeeded.  Is called either
+  // immediately after processing is complete or after a graph run has ended
+  // (if an error occurred in the graph).  ...
+  virtual ::MediaPipe::Status Close(CalculatorContext* cc) {
+    return ::MediaPipe::OkStatus();
+  }
+
+  ...
+};
+```
+### Life of a calculator
+
+During initialization of a MediaPipe graph, the framework calls a
+`GetContract()` static method to determine what kinds of packets are expected.
+
+The framework constructs and destroys the entire calculator for each graph run (e.g. once per video or once per image). Expensive or large objects that remain constant across graph runs should be supplied as input side packets so the calculations are not repeated on subsequent runs.
+
+After initialization, for each run of the graph, the following sequence occurs:
+
+* `Open()`
+* `Process()` (repeatedly)
+* `Close()`
+
+The framework calls `Open()` to initialize the calculator. `Open()` should interpret any options and set up the calculator's per-graph-run state. `Open()` may obtain input side packets and write packets to calculator outputs. If appropriate, it should call `SetOffset()` to reduce potential packet buffering of input streams.
+
+If an error occurs during `Open()` or `Process()` (as indicated by one of them returning a non-`Ok ` status), the graph run is terminated with no further calls to the calculator's methods, and the calculator is destroyed.
+
+For a calculator with inputs, the framework calls `Process()` whenever at least one input has a packet available. The framework guarantees that inputs all have the same timestamp, that timestamps increase with each call to `Process()` and that all packets are delivered. As a consequence, some inputs may not have any packets when `Process()` is called. An input whose packet is missing appears to produce an empty packet (with no timestamp).
+
+The framework calls `Close()` after all calls to `Process()`. All inputs will have been exhausted, but `Close()` has access to input side packets and may write outputs. After Close returns, the calculator is destroyed.
+
+Calculators with no inputs are referred to as sources. A source calculator continues to have `Process()` called as long as it returns an `Ok` status. A source calculator indicates that it is exhausted by returning a stop status (i.e. MediaPipe::tool::StatusStop).
+
+### Identifying inputs and outputs
+
+The public interface to a calculator consists of a set of input streams and
+output streams. In a CalculatorGraphConfiguration, the outputs from some
+calculators are connected to the inputs of other calculators using named
+streams. Stream names are normally lowercase, while input and output tags are
+normally UPPERCASE. In the example below, the output with tag name `VIDEO` is
+connected to the input with tag name `VIDEO_IN` using the stream named
+`video_stream`.
+
+```proto
+# Graph describing calculator SomeAudioVideoCalculator
+node {
+  calculator: "SomeAudioVideoCalculator"
+  input_stream: "INPUT:combined_input"
+  output_stream: "VIDEO:video_stream"
+}
+node {
+  calculator: "SomeVideoCalculator"
+  input_stream: "VIDEO_IN:video_stream"
+  output_stream: "VIDEO_OUT:processed_video"
+}
+```
+
+Input and output streams can be identified by index number, by tag name, or by a
+combination of tag name and index number. You can see some examples of input and
+output identifiers in the example below. `SomeAudioVideoCalculator` identifies
+its video output by tag and its audio outputs by the combination of tag and
+index. The input with tag `VIDEO` is connected to the stream named
+`video_stream`. The inputs with tag `AUDIO` and indices `0` and `1` are
+connected to the streams named `audio_left` and `audio_right`.
+`SomeAudioCalculator` identifies its audio inputs by index only (no tag needed).
+
+```proto
+# Graph describing calculator SomeAudioVideoCalculator
+node {
+  calculator: "SomeAudioVideoCalculator"
+  input_stream: "combined_input"
+  output_stream: "VIDEO:video_stream"
+  output_stream: "AUDIO:0:audio_left"
+  output_stream: "AUDIO:1:audio_right"
+}
+
+node {
+  calculator: "SomeAudioCalculator"
+  input_stream: "audio_left"
+  input_stream: "audio_right"
+  output_stream: "audio_energy"
+}
+```
+
+In the calculator implementation, inputs and outputs are also identified by tag
+name and index number.  In the function below input are output are identified:
+
+*   By index number: The combined input stream is identified simply by index
+    `0`.
+*   By tag name: The video output stream is identified by tag name "VIDEO".
+*   By tag name and index number: The output audio streams are identified by the
+    combination of the tag name `AUDIO` and the index numbers `0` and `1`.
+
+```c++
+// c++ Code snippet describing the SomeAudioVideoCalculator GetContract() method
+class SomeAudioVideoCalculator : public CalculatorBase {
+ public:
+  static ::mediapipe::Status GetContract(CalculatorContract* cc) {
+    cc->Inputs().Index(0).SetAny();
+    // SetAny() is used to specify that whatever the type of the
+    // stream is, it's acceptable.  This does not mean that any
+    // packet is acceptable.  Packets in the stream still have a
+    // particular type.  SetAny() has the same effect as explicitly
+    // setting the type to be the stream's type.
+    cc->Outputs().Tag("VIDEO").Set<ImageFrame>();
+    cc->Outputs().Get("AUDIO", 0).Set<Matrix>;
+    cc->Outputs().Get("AUDIO", 1).Set<Matrix>;
+    return ::mediapipe::OkStatus();
+  }
+```
+
+### Processing
+
+`Process()` called on a non-source node must return `::mediapipe::OkStatus()` to
+indicate that all went well, or any other status code to signal an error
+
+If a non-source calculator returns `tool::StatusStop()`, then this signals the
+graph is being cancelled early. In this case, all source calculators and graph
+input streams will be closed (and remaining Packets will propagate through the
+graph).
+
+A source node in a graph will continue to have `Process()` called on it as long
+as it returns `::mediapipe::OkStatus(`). To indicate that there is no more data
+to be generated return `tool::StatusStop()`. Any other status indicates an error
+has occurred.
+
+`Close()` returns `::mediapipe::OkStatus()` to indicate success. Any other
+status indicates a failure.
+
+Here is the basic `Process()` function. It uses the `Input()` method (which can
+be used only if the calculator has a single input) to request its input data. It
+then uses `std::unique_ptr` to allocate the memory needed for the output packet,
+and does the calculations. When done it releases the pointer when adding it to
+the output stream.
+
+```c++
+::util::Status MyCalculator::Process() {
+  const Matrix& input = Input()->Get<Matrix>();
+  std::unique_ptr<Matrix> output(new Matrix(input.rows(), input.cols()));
+  // do your magic here....
+  //    output->row(n) =  ...
+  Output()->Add(output.release(), InputTimestamp());
+  return ::mediapipe::OkStatus();
+}
+```
+
+### GraphConfig
+
+A `GraphConfig` is a specification that describes the topology and functionality
+of a MediaPipe graph. In the specification, a node in the graph represents an
+instance of a particular calculator. All the necessary configurations of the
+node, such its type, inputs and outputs must be described in the specification.
+Description of the node can also include several optional fields, such as
+node-specific options, input policy and executor, discussed in Section
+[Framework Concepts > Scheduling mechanics](scheduling_sync.md#scheduling-mechanics).
+
+`GraphConfig` has several other fields to configure the global graph-level
+settings, eg, graph executor configs, number of threads, and maximum queue size
+of input streams. Several graph-level settings are useful for tuning the
+performance of the graph on different platforms (eg, desktop v.s. mobile). For
+instance, on mobile, attaching a heavy model-inference calculator to a separate
+executor can improve the performance of a real-time application since this
+enables thread locality.
+
+Below is a trivial `GraphConfig` example where we have series of passthrough
+calculators :
+
+```proto
+# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
+# passthrough calculators.
+input_stream: "in"
+node {
+    calculator: "PassThroughCalculator"
+    input_stream: "in"
+    output_stream: "out1"
+}
+node {
+    calculator: "PassThroughCalculator"
+    input_stream: "out1"
+    output_stream: "out2"
+}
+node {
+    calculator: "PassThroughCalculator"
+    input_stream: "out2"
+    output_stream: "out3"
+}
+node {
+    calculator: "PassThroughCalculator"
+    input_stream: "out3"
+    output_stream: "out4"
+}
+```
+
+### Subgraph
+
+To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
+of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
+public interface to a subgraph consists of a set of input and output streams
+similar to the public interface of a calculator. The subgraph can then be
+included in an `CalculatorGraphConfig` as if it were a calculator. When a
+MediaPipe graph is loaded from a `CalculatorGraphConfig`, each subgraph node is
+replaced by the corresponding graph of calculators. As a result, the semantics
+and performance of the subgraph is identical to the corresponding graph of
+calculators.
+
+Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`
+
+1. Defining the subgraph.
+
+    ```proto
+    # This subgraph is defined in two_pass_through_subgraph.pbtxt
+    # that is registered in the BUILD file as "TwoPassThroughSubgraph"
+    input_stream: "out1"
+    output_stream: "out3"
+
+    node {
+        calculator: "PassThroughculator"
+        input_stream: "out1"
+        output_stream: "out2"
+    }
+    node {
+        calculator: "PassThroughculator"
+        input_stream: "out2"
+        output_stream: "out3"
+    }
+    ```
+
+The public interface to the graph that consist of:
+   * Graph input streams
+   * Graph output streams
+   * Graph input side packets
+   * Graph output side packets
+
+2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`
+   * The parameter `register_as` defines the component name for the new subgraph
+
+    ```proto
+    # Small section of BUILD file for registering the "TwoPassThroughSubgraph"
+    # subgraph for use by main graph main_pass_throughcals.pbtxt
+    #
+    mediapipe_simple_subgraph(
+        name = "twopassthrough_subgraph",
+        graph = "twopassthrough_subgraph.pbtxt",
+        register_as = "TwoPassThroughSubgraph",
+        deps = [
+                "//mediapipe/calculators/core:pass_through_calculator",
+                "//mediapipe/framework:calculator_graph",
+        ],
+    )
+    ```
+
+3. Use the subgraph in the main graph
+
+    ```proto
+    # This main graph is defined in main_pass_throughcals.pbtxt
+    # using subgraph called "TwoPassThroughSubgraph"
+    #
+    input_stream: "in"
+    node {
+        calculator: "PassThroughCalculator"
+        input_stream: "in"
+        output_stream: "out1"
+    }
+    node {
+        calculator: "TwoPassThroughSubgraph"
+        input_stream: "out1"
+        output_stream: "out3"
+    }
+    node {
+        calculator: "PassThroughCalculator"
+        input_stream: "out3"
+        output_stream: "out4"
+    }
+    ```
+
+<!---
+### Graph Templates
+
+A MediaPipe graph template looks exactly like a calculator graph .pbtxt file with some embedded parameters like `%num_detectors%`. When the template parameters are replaced by argument values, the template defines a complete `CalculatorGraphConfig`.
+
+1. Defining and using a graph template by writing a `CalculatorGraphConfig` text protobuf file containing template parameters. The file extension .textpbt stands for "text protobuf template".
+    ```proto
+    # Test graph with an iteration template directive
+    node: {
+    name: %name_1%
+    calculator: "IntervalFilterCalculator"
+    options: {
+        [mediapipe.IntervalFilterCalculatorOptions.ext] {
+        intervals {
+            % for (interval : intervals_1) %
+            interval {
+            start_us: %interval.begin%
+            end_us: %interval.end%
+            }
+            %end%
+        }
+        }
+    }
+    }
+    ```
+2. Specify values for the template parameters as name-value pairs in a [`TemplateDict protobuf`](http://github.com/mediapipe/framework/tool/calculator_graph_template.proto)
+    ```proto
+    # Some test template arguments for iteration_test_template.textpbt
+    arg: {key: "name_1" value: {str: "hooloo"}}
+    arg: {key: "intervals_1" value: {
+    element: { dict: {
+        arg: {key: "begin" value: {num:33}}
+        arg: {key: "end" value: {num:44}}
+    }}
+    element: { dict: {
+        arg: {key: "begin" value: {num:55}}
+        arg: {key: "end" value: {num:66}}
+    }}
+    element: { dict: {
+        arg: {key: "begin" value: {num:77}}
+        arg: {key: "end" value: {num:88}}
+    }}
+    }}
+    ```
+3. Register the subgraph using the build rule: `mediapipe_template_subgraph`.
+    ```proto
+    mediapipe_template_graph(
+        name = "iteration_test_subgraph",
+        register_as = "IterationTestTemplateSubgraph",
+        template = "iteration_test_template.textpbt",
+        deps = [
+            "//mediapipe/core:interval_filter_calculator",
+        ],
+    )
+    ```
+    Alternatively, the template and the parameter values can be combined using the build rule: `mediapipe_template_graph`.
+    ```proto
+    mediapipe_template_graph(
+        name = "iteration_test_2_graph",
+        arg_file = "iteration_test_arg.pbtxt",
+        template = "iteration_test_template.textpbt",
+    )
+    ```
+4. The result is a complete CalculatorGraphConfig protobuf, such as the following:
+    ```proto
+    node {
+    name: "hooloo"
+    calculator: "IntervalFilterCalculator"
+    options {
+        [mediapipe.IntervalFilterCalculatorOptions.ext] {
+        intervals {
+            interval {
+            start_us: 33
+            end_us: 44
+            }
+            interval {
+            start_us: 55
+            end_us: 66
+            }
+            interval {
+            start_us: 77
+            end_us: 88
+            }
+        }
+        }
+    }
+    }
+    ```
+Graph template parameters
+* For a template parameter, you can specify a simple parameter name such as `%end_time%`
+```proto
+interval {
+  start_us: 10
+  end_us: %end_time%
+}
+```
+or a more complex expression, such as `% begin_time + duration %`
+```proto
+interval {
+  start_us: %begin_time%
+  end_us: % begin_time + duration %
+}
+```
+--->

mediapipe/docs/gpu.md

@@ -0,0 +1,130 @@
+## Running on GPUs
+
+-   [Overview](#overview)
+-   [OpenGL Support](#graphconfig)
+-   [Life of a GPU calculator](#life-of-a-gpu-calculator)
+-   [GpuBuffer to ImageFrame converters](#gpubuffer-to-imageframe-converters)
+
+
+### Overview
+MediaPipe supports calculator nodes for GPU compute and rendering, and allows combining multiple GPU nodes, as well as mixing them with CPU based calculator nodes. There exist several GPU APIs on mobile platforms (eg, OpenGL ES, Metal and Vulkan). MediaPipe does not attempt to offer a single cross-API GPU abstraction. Individual nodes can be written using different APIs, allowing them to take advantage of platform specific features when needed.
+
+GPU support is essential for good performance on mobile platforms, especially for real-time video. MediaPipe enables developers to write GPU compatible calculators that support the use of GPU for:
+
+   * On-device real-time processing, not just batch processing
+   * Video rendering and effects, not just analysis
+
+Below are the design principles for GPU support in MediaPipe
+
+   * GPU-based calculators should be able to occur anywhere in the graph, and not necessarily be used for on-screen rendering.
+   * Transfer of frame data from one GPU-based calculator to another should be fast, and not incur expensive copy operations.
+   * Transfer of frame data between CPU and GPU should be as efficient as the platform allows.
+   * Because different platforms may require different techniques for best performance, the API should allow flexibility in the way things are implemented behind the scenes.
+   * A calculator should be allowed maximum flexibility in using the GPU for all or part of its operation, combining it with the CPU if necessary.
+
+### OpenGL support
+MediaPipe supports OpenGL ES up to version 3.2 on Android and up to ES 3.0 on iOS. In addition, MediaPipe also supports Metal on iOS.
+
+   * MediaPipe allows graphs to run OpenGL in multiple GL contexts. For example, this can be very useful in graphs that combine a slower GPU inference path (eg, at 10 FPS) with a faster GPU rendering path (eg, at 30 FPS): since one GL context corresponds to one sequential command queue, using the same context for both tasks would reduce the rendering frame rate. One challenge MediaPipe's use of multiple contexts solves is the ability to communicate across them. An example scenario is one with an input video that is sent to both the rendering and inferences paths, and rendering needs to have access to the latest output from inference.
+
+   * An OpenGL context cannot be accessed by multiple threads at the same time. Furthermore, switching the active GL context on the same thread can be slow on some Android devices. Therefore, our approach is to have one dedicated thread per context. Each thread issues GL commands, building up a serial command queue on its context, which is then executed by the GPU asynchronously.
+
+### Life of a GPU calculator
+
+This section presents the basic structure of the Process method of a GPU
+calculator derived from base class GlSimpleCalculator. The GPU calculator
+`LuminanceCalculator` is shown as an example. The method
+`LuminanceCalculator::GlRender` is called from `GlSimpleCalculator::Process`.
+
+```
+// Converts RGB images into luminance images, still stored in RGB format.
+// See GlSimpleCalculator for inputs, outputs and input side packets.
+class LuminanceCalculator : public GlSimpleCalculator {
+ public:
+  ::mediapipe::Status GlSetup() override;
+  ::mediapipe::Status GlRender(const GlTexture& src,
+                               const GlTexture& dst) override;
+  ::mediapipe::Status GlTeardown() override;
+
+ private:
+  GLuint program_ = 0;
+  GLint frame_;
+};
+REGISTER_CALCULATOR(LuminanceCalculator);
+
+::mediapipe::Status LuminanceCalculator::GlRender(const GlTexture& src,
+                                                  const GlTexture& dst) {
+  static const GLfloat square_vertices[] = {
+      -1.0f, -1.0f,  // bottom left
+      1.0f,  -1.0f,  // bottom right
+      -1.0f, 1.0f,   // top left
+      1.0f,  1.0f,   // top right
+  };
+  static const GLfloat texture_vertices[] = {
+      0.0f, 0.0f,  // bottom left
+      1.0f, 0.0f,  // bottom right
+      0.0f, 1.0f,  // top left
+      1.0f, 1.0f,  // top right
+  };
+
+  // program
+  glUseProgram(program_);
+  glUniform1i(frame_, 1);
+
+  // vertex storage
+  GLuint vbo[2];
+  glGenBuffers(2, vbo);
+  GLuint vao;
+  glGenVertexArrays(1, &vao);
+  glBindVertexArray(vao);
+
+  // vbo 0
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_VERTEX);
+  glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // vbo 1
+  glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
+  glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
+               GL_STATIC_DRAW);
+  glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
+
+  // draw
+  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
+
+  // cleanup
+  glDisableVertexAttribArray(ATTRIB_VERTEX);
+  glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
+  glBindBuffer(GL_ARRAY_BUFFER, 0);
+  glBindVertexArray(0);
+  glDeleteVertexArrays(1, &vao);
+  glDeleteBuffers(2, vbo);
+
+  return ::mediapipe::OkStatus();
+}
+```
+
+The design principles mentioned above have resulted in the following design
+choices for MediaPipe GPU support:
+
+   * We have a GPU data type, called `GpuBuffer`, for representing image data, optimized for GPU usage. The exact contents of this data type are opaque and platform-specific.
+   * A low-level API based on composition, where any calculator that wants to make use of the GPU creates and owns an instance of the `GlCalculatorHelper` class. This class offers a platform-agnostic API for managing the OpenGL context, setting up textures for inputs and outputs, etc.
+   * A high-level API based on subclassing, where simple calculators implementing image filters subclass from `GlSimpleCalculator` and only need to override a couple of virtual methods with their specific OpenGL code, while the superclass takes care of all the plumbing.
+   * Data that needs to be shared between all GPU-based calculators is provided as a external input that is implemented as a graph service and is managed by the `GlCalculatorHelper` class.
+   * The combination of calculator-specific helpers and a shared graph service allows us great flexibility in managing the GPU resource: we can have a separate context per calculator, share a single context, share a lock or other synchronization primitives, etc. -- and all of this is managed by the helper and hidden from the individual calculators.
+
+### GpuBuffer to ImageFrame converters
+
+We provide two calculators called `GpuBufferToImageFrameCalculator` and `ImageFrameToGpuBufferCalculator`. These calculators convert between `ImageFrame` and `GpuBuffer`, allowing the construction of graphs that combine GPU and CPU calculators. They are supported on both iOS and Android
+
+When possible, these calculators use platform-specific functionality to share data between the CPU and the GPU without copying.
+
+The below diagram shows the data flow in a mobile application that captures video from the camera, runs it through a MediaPipe graph, and renders the output on the screen in real time. The dashed line indicates which parts are inside the MediaPipe graph proper. This application runs a Canny edge-detection filter on the CPU using OpenCV, and overlays it on top of the original video using the GPU.
+
+| ![How GPU calculators interact](images/gpu_example_graph.png) |
+|:--:|
+| *Video frames from the camera are fed into the graph as `GpuBuffer` packets. The input stream is accessed by two calculators in parallel. `GpuBufferToImageFrameCalculator` converts the buffer into an `ImageFrame`, which is then sent through a grayscale converter and a canny filter (both based on OpenCV and running on the CPU), whose output is then converted into a `GpuBuffer` again. A multi-input GPU calculator, GlOverlayCalculator, takes as input both the original `GpuBuffer` and the one coming out of the edge detector, and overlays them using a shader. The output is then sent back to the application using a callback calculator, and the application renders the image to the screen using OpenGL.* |
+

mediapipe/docs/hair_segmentation_android_gpu.md

@@ -0,0 +1,194 @@
+# Hair Segmentation on Android
+
+Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
+general instructions to develop an Android application that uses MediaPipe. This
+doc focuses on the
+[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_android_gpu.pbtxt)
+that performs hair segmentation with TensorFlow Lite on GPU.
+
+![hair_segmentation_android_gpu_gif](images/mobile/hair_segmentation_android_gpu.gif){width="300"}
+
+## App
+
+The graph is used in the
+[Hair Segmentation GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu)
+example app. To build the app, run:
+
+```bash
+bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu
+```
+
+To further install the app on android device, run:
+
+```bash
+adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu/hairsegmentationgpu.apk
+```
+
+## Graph
+
+![hair_segmentation_android_gpu_graph](images/mobile/hair_segmentation_android_gpu.png){width="600"}
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into [MediaPipe Visualizer](https://mediapipe-viz.appspot.com/).
+
+```bash
+# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
+# Used in the example in
+# mediapipie/examples/android/src/java/com/mediapipe/apps/hairsegmentationgpu.
+
+# Images on GPU coming into and out of the graph.
+input_stream: "input_video"
+output_stream: "output_video"
+
+# Throttles the images flowing downstream for flow control. It passes through
+# the very first incoming image unaltered, and waits for
+# TfLiteTensorsToSegmentationCalculator downstream in the graph to finish
+# generating the corresponding hair mask before it passes through another
+# image. All images that come in while waiting are dropped, limiting the number
+# of in-flight images between this calculator and
+# TfLiteTensorsToSegmentationCalculator to 1. This prevents the nodes in between
+# from queuing up incoming images and data excessively, which leads to increased
+# latency and memory usage, unwanted in real-time mobile applications. It also
+# eliminates unnecessarily computation, e.g., a transformed image produced by
+# ImageTransformationCalculator may get dropped downstream if the subsequent
+# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
+# processing previous inputs.
+node {
+  calculator: "RealTimeFlowLimiterCalculator"
+  input_stream: "input_video"
+  input_stream: "FINISHED:hair_mask"
+  input_stream_info: {
+    tag_index: "FINISHED"
+    back_edge: true
+  }
+  output_stream: "throttled_input_video"
+}
+
+# Transforms the input image on GPU to a 512x512 image. To scale the image, by
+# default it uses the STRETCH scale mode that maps the entire input image to the
+# entire transformed image. As a result, image aspect ratio may be changed and
+# objects in the image may be deformed (stretched or squeezed), but the hair
+# segmentation model used in this graph is agnostic to that deformation.
+node: {
+  calculator: "ImageTransformationCalculator"
+  input_stream: "IMAGE_GPU:throttled_input_video"
+  output_stream: "IMAGE_GPU:transformed_input_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
+      output_width: 512
+      output_height: 512
+    }
+  }
+}
+
+# Waits for a mask from the previous round of hair segmentation to be fed back
+# as an input, and caches it. Upon the arrival of an input image, it checks if
+# there is a mask cached, and sends out the mask with the timestamp replaced by
+# that of the input image. This is needed so that the "current image" and the
+# "previous mask" share the same timestamp, and as a result can be synchronized
+# and combined in the subsequent calculator. Note that upon the arrival of the
+# very first input frame, an empty packet is sent out to jump start the feedback
+# loop.
+node {
+  calculator: "PreviousLoopbackCalculator"
+  input_stream: "MAIN:throttled_input_video"
+  input_stream: "LOOP:hair_mask"
+  input_stream_info: {
+    tag_index: "LOOP"
+    back_edge: true
+  }
+  output_stream: "PREV_LOOP:previous_hair_mask"
+}
+
+# Embeds the hair mask generated from the previous round of hair segmentation
+# as the alpha channel of the current input image.
+node {
+  calculator: "SetAlphaCalculator"
+  input_stream: "IMAGE_GPU:transformed_input_video"
+  input_stream: "ALPHA_GPU:previous_hair_mask"
+  output_stream: "IMAGE_GPU:mask_embedded_input_video"
+}
+
+# Converts the transformed input image on GPU into an image tensor stored in
+# tflite::gpu::GlBuffer. The zero_center option is set to false to normalize the
+# pixel values to [0.f, 1.f] as opposed to [-1.f, 1.f]. The flip_vertically
+# option is set to true to account for the descrepancy between the
+# representation of the input image (origin at the bottom-left corner, the
+# OpenGL convention) and what the model used in this graph is expecting (origin
+# at the top-left corner). With the max_num_channels option set to 4, all 4 RGBA
+# channels are contained in the image tensor.
+node {
+  calculator: "TfLiteConverterCalculator"
+  input_stream: "IMAGE_GPU:mask_embedded_input_video"
+  output_stream: "TENSORS_GPU:image_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
+      zero_center: false
+      flip_vertically: true
+      max_num_channels: 4
+    }
+  }
+}
+
+# Generates a single side packet containing a TensorFlow Lite op resolver that
+# supports custom ops needed by the model used in this graph.
+node {
+  calculator: "TfLiteCustomOpResolverCalculator"
+  output_side_packet: "op_resolver"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteCustomOpResolverCalculatorOptions] {
+      use_gpu: true
+    }
+  }
+}
+
+# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
+# tensor representing the hair segmentation, which has the same width and height
+# as the input image tensor.
+node {
+  calculator: "TfLiteInferenceCalculator"
+  input_stream: "TENSORS_GPU:image_tensor"
+  output_stream: "TENSORS_GPU:segmentation_tensor"
+  input_side_packet: "CUSTOM_OP_RESOLVER:op_resolver"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+      model_path: "hair_segmentation.tflite"
+      use_gpu: true
+    }
+  }
+}
+
+# Decodes the segmentation tensor generated by the TensorFlow Lite model into a
+# mask of values in [0.f, 1.f], stored in the R channel of a GPU buffer. It also
+# takes the mask generated previously as another input to improve the temporal
+# consistency.
+node {
+  calculator: "TfLiteTensorsToSegmentationCalculator"
+  input_stream: "TENSORS_GPU:segmentation_tensor"
+  input_stream: "PREV_MASK_GPU:previous_hair_mask"
+  output_stream: "MASK_GPU:hair_mask"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteTensorsToSegmentationCalculatorOptions] {
+      tensor_width: 512
+      tensor_height: 512
+      tensor_channels: 2
+      combine_with_previous_ratio: 0.9
+      output_layer_index: 1
+    }
+  }
+}
+
+# Colors the hair segmentation with the color specified in the option.
+node {
+  calculator: "RecolorCalculator"
+  input_stream: "IMAGE_GPU:throttled_input_video"
+  input_stream: "MASK_GPU:hair_mask"
+  output_stream: "IMAGE_GPU:output_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.RecolorCalculatorOptions] {
+      color { r: 0 g: 0 b: 255 }
+      mask_channel: RED
+    }
+  }
+}
+```

mediapipe/docs/hello_world_android.md

@@ -0,0 +1,728 @@
+# Hello World! in MediaPipe on Android
+
+## Introduction
+
+This codelab uses MediaPipe on an Android device.
+
+### What you will learn
+
+How to develop an Android application that uses MediaPipe and run a MediaPipe
+graph on Android.
+
+### What you will build
+
+A simple camera app for real-time Sobel edge detection applied to a live video
+stream on an Android device.
+
+![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif){width="300"}
+
+## Setup
+
+1.  Install MediaPipe on your system, see [MediaPipe installation guide] for
+    details.
+2.  Install Android Development SDK and Android NDK. See how to do so in
+    [Setting up Android SDK and NDK].
+3.  Enable [developer options] on your Android device.
+4.  Setup [Bazel] on your system to build and deploy the Android app.
+
+## Graph for edge detection
+
+We will be using the following graph, [`edge_detection_android_gpu.pbtxt`]:
+
+```
+input_stream: "input_video"
+output_stream: "output_video"
+
+node: {
+  calculator: "LuminanceCalculator"
+  input_stream: "input_video"
+  output_stream: "luma_video"
+}
+
+node: {
+  calculator: "SobelEdgesCalculator"
+  input_stream: "luma_video"
+  output_stream: "output_video"
+}
+```
+
+A visualization of the graph is shown below:
+
+![edge_detection_android_gpu_graph](images/mobile/edge_detection_android_graph_gpu.png){width="200"}
+
+This graph has a single input stream named `input_video` for all incoming frames
+that will be provided by your device's camera.
+
+The first node in the graph, `LuminanceCalculator`, takes a single packet (image
+frame) and applies a change in luminance using an OpenGL shader. The resulting
+image frame is sent to the `luma_video` output stream.
+
+The second node, `SobelEdgesCalculator` applies edge detection to incoming
+packets in the `luma_video` stream and outputs results in `output_video` output
+stream.
+
+Our Android application will display the output image frames of the
+`sobel_video` stream.
+
+## Initial minimal application setup
+
+We first start with an simple Android application that displays "Hello World!"
+on the screen. You may skip this step if you are familiar with building Android
+applications using `bazel`.
+
+Create a new directory where you will create your Android application. For
+example, the complete code of this tutorial can be found at
+`mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu`.
+We will refer to this path as `$APPLICATION_PATH` throughout the codelab.
+
+Note that in the path to the application:
+
+*   The application is named `edgedetectiongpu`.
+*   The `$PACKAGE_PATH` of the application is
+    `com.google.mediapipe.apps.edgdetectiongpu`. This is used in code snippets in
+    this tutorial, so please remember to use your own `$PACKAGE_PATH` when you
+    copy/use the code snippets.
+
+Add a file `activity_main.xml` to `$APPLICATION_PATH/res/layout`. This displays
+a [`TextView`] on the full screen of the application with the string `Hello
+World!`:
+
+```
+<?xml version="1.0" encoding="utf-8"?>
+<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
+    xmlns:app="http://schemas.android.com/apk/res-auto"
+    xmlns:tools="http://schemas.android.com/tools"
+    android:layout_width="match_parent"
+    android:layout_height="match_parent">
+
+  <TextView
+    android:layout_width="wrap_content"
+    android:layout_height="wrap_content"
+    android:text="Hello World!"
+    app:layout_constraintBottom_toBottomOf="parent"
+    app:layout_constraintLeft_toLeftOf="parent"
+    app:layout_constraintRight_toRightOf="parent"
+    app:layout_constraintTop_toTopOf="parent" />
+
+</android.support.constraint.ConstraintLayout>
+```
+
+Add a simple `MainActivity.java` to `$APPLICATION_PATH` which loads the content
+of the `activity_main.xml` layout as shown below:
+
+```
+package com.google.mediapipe.apps.edgedetectiongpu;
+
+import android.os.Bundle;
+import androidx.appcompat.app.AppCompatActivity;
+
+/** Bare-bones main activity. */
+public class MainActivity extends AppCompatActivity {
+
+  @Override
+  protected void onCreate(Bundle savedInstanceState) {
+    super.onCreate(savedInstanceState);
+    setContentView(R.layout.activity_main);
+  }
+}
+```
+
+Add a manifest file, `AndroidManifest.xml` to `$APPLICATION_PATH`, which
+launches `MainActivity` on application start:
+
+```
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="com.google.mediapipe.apps.edgedetectiongpu">
+
+  <uses-sdk
+      android:minSdkVersion="19"
+      android:targetSdkVersion="19" />
+
+  <application
+      android:allowBackup="true"
+      android:label="@string/app_name"
+      android:supportsRtl="true"
+      android:theme="@style/AppTheme">
+      <activity
+          android:name=".MainActivity"
+          android:exported="true"
+          android:screenOrientation="portrait">
+          <intent-filter>
+              <action android:name="android.intent.action.MAIN" />
+              <category android:name="android.intent.category.LAUNCHER" />
+          </intent-filter>
+      </activity>
+  </application>
+
+</manifest>
+```
+
+To get `@string/app_name`, we need to add a file `strings.xml` to
+`$APPLICATION_PATH/res/values/`:
+
+```
+<resources>
+    <string name="app_name" translatable="false">Edge Detection GPU</string>
+</resources>
+```
+
+Also, in our application we are using a `Theme.AppCompat` theme in the app, so
+we need appropriate theme references. Add `colors.xml` to
+`$APPLICATION_PATH/res/values/`:
+
+```
+<?xml version="1.0" encoding="utf-8"?>
+<resources>
+    <color name="colorPrimary">#008577</color>
+    <color name="colorPrimaryDark">#00574B</color>
+    <color name="colorAccent">#D81B60</color>
+</resources>
+```
+
+Add `styles.xml` to `$APPLICATION_PATH/res/values/`:
+
+```
+<resources>
+
+    <!-- Base application theme. -->
+    <style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
+        <!-- Customize your theme here. -->
+        <item name="colorPrimary">@color/colorPrimary</item>
+        <item name="colorPrimaryDark">@color/colorPrimaryDark</item>
+        <item name="colorAccent">@color/colorAccent</item>
+    </style>
+
+</resources>
+```
+
+To build the application, add a `BUILD` file to `$APPLICATION_PATH`:
+
+```
+android_library(
+    name = "mediapipe_lib",
+    srcs = glob(["*.java"]),
+    manifest = "AndroidManifest.xml",
+    resource_files = glob(["res/**"]),
+    deps = [
+        "//third_party:android_constraint_layout",
+        "//third_party:androidx_appcompat",
+    ],
+)
+
+android_binary(
+    name = "edgedetectiongpu",
+    aapt_version = "aapt2",
+    manifest = "AndroidManifest.xml",
+    manifest_values = {"applicationId": "com.google.mediapipe.apps.edgedetectiongpu"},
+    multidex = "native",
+    deps = [
+        ":mediapipe_lib",
+    ],
+)
+
+```
+
+The `android_library` rule adds dependencies for `MainActivity`, resource files
+and `AndroidManifest.xml`.
+
+The `android_binary` rule, uses the `mediapipe_lib` Android library generated to
+build a binary APK for installation on your Android device.
+
+To build the app, use the following command:
+
+```
+bazel build -c opt --config=android_arm64 $APPLICATION_PATH
+```
+
+Install the generated APK file using `adb install`. For example:
+
+```
+adb install bazel-bin/$APPLICATION_PATH/edgedetectiongpu.apk
+```
+
+Open the application on your device. It should display a screen with the text
+`Hello World!`.
+
+![bazel_hello_world_android](images/mobile/bazel_hello_world_android.png){width="300"}
+
+## Using the camera via `CameraX`
+
+### Camera Permissions
+
+To use the camera in our application, we need to request the user to provide
+access to the camera. To request camera permissions, add the following to
+`AndroidManifest.xml`:
+
+```
+<!-- For using the camera -->
+<uses-permission android:name="android.permission.CAMERA" />
+<uses-feature android:name="android.hardware.camera" />
+```
+
+Change the minimum SDK version to `21` and target SDK version to `27` in the
+same file:
+
+```
+<uses-sdk
+    android:minSdkVersion="21"
+    android:targetSdkVersion="27" />
+```
+
+This ensures that the user is prompted to request camera permission and enables
+us to use the [CameraX] library for camera access.
+
+To request camera permissions, we can use a utility provided by MediaPipe
+components, namely [`PermissionHelper`]. To use it, add a dependency
+`"//mediapipe/java/com/google/mediapipe/components:android_components"` in the
+`mediapipe_lib` rule in `BUILD`.
+
+To use the `PermissionHelper` in `MainActivity`, add the following line to the
+`onCreate` function:
+
+```
+PermissionHelper.checkAndRequestCameraPermissions(this);
+```
+
+This prompts the user with a dialog on the screen to request for permissions to
+use the camera in this application.
+
+Add the following code to handle the user response:
+
+```
+@Override
+public void onRequestPermissionsResult(
+    int requestCode, String[] permissions, int[] grantResults) {
+  super.onRequestPermissionsResult(requestCode, permissions, grantResults);
+  PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
+}
+
+@Override
+protected void onResume() {
+  super.onResume();
+  if (PermissionHelper.cameraPermissionsGranted(this)) {
+    startCamera();
+  }
+}
+
+public void startCamera() {}
+```
+
+We will leave the `startCamera()` method empty for now. When the user responds
+to the prompt, the `MainActivity` will resume and `onResume()` will be called.
+The code will confirm that permissions for using the camera have been granted,
+and then will start the camera.
+
+Rebuild and install the application. You should now see a prompt requesting
+access to the camera for the application.
+
+Note: If the there is no dialog prompt, uninstall and reinstall the application.
+This may also happen if you haven't changed the `minSdkVersion` and
+`targetSdkVersion` in the `AndroidManifest.xml` file.
+
+### Camera Access
+
+With camera permissions available, we can start and fetch frames from the
+camera.
+
+To view the frames from the camera we will use a [`SurfaceView`]. Each frame
+from the camera will be stored in a [`SurfaceTexture`] object. To use these, we
+first need to change the layout of our application.
+
+Remove the entire [`TextView`] code block from
+`$APPLICATION_PATH/res/layout/activity_main.xml` and add the following code
+instead:
+
+```
+<FrameLayout
+    android:id="@+id/preview_display_layout"
+    android:layout_width="fill_parent"
+    android:layout_height="fill_parent"
+    android:layout_weight="1">
+    <TextView
+        android:id="@+id/no_camera_access_view"
+        android:layout_height="fill_parent"
+        android:layout_width="fill_parent"
+        android:gravity="center"
+        android:text="@string/no_camera_access" />
+</FrameLayout>
+```
+
+This code block has a new [`FrameLayout`] named `preview_display_layout` and a
+[`TextView`] nested inside it, named `no_camera_access_preview`. When camera
+access permissions are not granted, our application will display the
+[`TextView`] with a string message, stored in the variable `no_camera_access`.
+Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file:
+
+```
+<string name="no_camera_access" translatable="false">Please grant camera permissions.</string>
+```
+
+When the user doesn't grant camera permission, the screen will now look like
+this:
+
+![missing_camera_permission_android](images/mobile/missing_camera_permission_android.png){width="300"}
+
+Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
+`MainActivity`:
+
+```
+private SurfaceTexture previewFrameTexture;
+private SurfaceView previewDisplayView;
+```
+
+In the `onCreate(Bundle)` function, add the following two lines _before_
+requesting camera permissions:
+
+```
+previewDisplayView = new SurfaceView(this);
+setupPreviewDisplayView();
+```
+
+And now add the code defining `setupPreviewDisplayView()`:
+
+```
+private void setupPreviewDisplayView() {
+  previewDisplayView.setVisibility(View.GONE);
+  ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
+  viewGroup.addView(previewDisplayView);
+}
+```
+
+We define a new [`SurfaceView`] object and add it to the
+`preview_display_layout` [`FrameLayout`] object so that we can use it to display
+the camera frames using a [`SurfaceTexture`] object named `previewFrameTexture`.
+
+To use `previewFrameTexture` for getting camera frames, we will use [CameraX].
+MediaPipe provides a utility named [`CameraXPreviewHelper`] to use [CameraX].
+This class updates a listener when camera is started via
+`onCameraStarted(@Nullable SurfaceTexture)`.
+
+To use this utility, modify the `BUILD` file to add a dependency on
+`"//mediapipe/java/com/google/mediapipe/components:android_camerax_helper"`.
+
+Now import [`CameraXPreviewHelper`] and add the following line to
+`MainActivity`:
+
+```
+private CameraXPreviewHelper cameraHelper;
+```
+
+Now, we can add our implementation to `startCamera()`:
+
+```
+public void startCamera() {
+  cameraHelper = new CameraXPreviewHelper();
+  cameraHelper.setOnCameraStartedListener(
+    surfaceTexture -> {
+      previewFrameTexture = surfaceTexture;
+      // Make the display view visible to start showing the preview.
+      previewDisplayView.setVisibility(View.VISIBLE);
+    });
+}
+```
+
+This creates a new [`CameraXPreviewHelper`] object and adds an anonymous
+listener on the object. When `cameraHelper` signals that the camera has started
+and a `surfaceTexture` to grab frames is available, we save that
+`surfaceTexture` as `previewFrameTexture`, and make the `previewDisplayView`
+visible so that we can start seeing frames from the `previewFrameTexture`.
+
+However, before starting the camera, we need to decide which camera we want to
+use. [`CameraXPreviewHelper`] inherits from [`CameraHelper`] which provides two
+options, `FRONT` and `BACK`. We will use `BACK` camera for this application to
+perform edge detection on a live scene that we view from the camera.
+
+Add the following line to define `CAMERA_FACING` for our application,
+
+```
+private static final CameraHelper.CameraFacing CAMERA_FACING = CameraHelper.CameraFacing.BACK;
+```
+
+`CAMERA_FACING` is a static variable as we will use the same camera throughout
+the application from start to finish.
+
+Now add the following line at the end of the `startCamera()` function:
+
+```
+cameraHelper.startCamera(this, CAMERA_FACING, /*surfaceTexture=*/ null);
+```
+
+At this point, the application should build successfully. However, when you run
+the application on your device, you will see a black screen (even though camera
+permissions have been granted). This is because even though we save the
+`surfaceTexture` variable provided by the [`CameraXPreviewHelper`], the
+`previewSurfaceView` doesn't use its output and display it on screen yet.
+
+Since we want to use the frames in a MediaPipe graph, we will not add code to
+view the camera output directly in this tutorial. Instead, we skip ahead to how
+we can send camera frames for processing to a MediaPipe graph and display the
+output of the graph on the screen.
+
+## `ExternalTextureConverter` setup
+
+A [`SurfaceTexture`] captures image frames from a stream as an OpenGL ES
+texture. To use a MediaPipe graph, frames captured from the camera should be
+stored in a regular Open GL texture object. MediaPipe provides a class,
+[`ExternalTextureConverter`] to convert the image stored in a [`SurfaceTexture`]
+object to a regular OpenGL texture object.
+
+To use [`ExternalTextureConverter`], we also need an `EGLContext`, which is
+created and managed by an [`EglManager`] object. Add a dependency to the `BUILD`
+file to use [`EglManager`], `"//mediapipe/java/com/google/mediapipe/glutil"`.
+
+In `MainActivity`, add the following declarations:
+
+```
+private EglManager eglManager;
+private ExternalTextureConverter converter;
+```
+
+In the `onCreate(Bundle)` function, add a statement to initialize the
+`eglManager` object before requesting camera permissions:
+
+```
+eglManager = new EglManager(null);
+```
+
+Recall that we defined the `onResume()` function in `MainActivity` to confirm
+camera permissions have been granted and call `startCamera()`. Before this
+check, add the following line in `onResume()` to initialize the `converter`
+object:
+
+```
+converter = new ExternalTextureConverter(eglManager.getContext());
+```
+
+This `converter` now uses the `GLContext` managed by `eglManager`.
+
+We also need to override the `onPause()` function in the `MainActivity` so that
+if the application goes into a paused state, we close the `converter` properly:
+
+```
+@Override
+protected void onPause() {
+  super.onPause();
+  converter.close();
+}
+```
+
+To pipe the output of `previewFrameTexture` to the `converter`, add the
+following block of code to `setupPreviewDisplayView()`:
+
+```
+previewDisplayView
+ .getHolder()
+ .addCallback(
+     new SurfaceHolder.Callback() {
+       @Override
+       public void surfaceCreated(SurfaceHolder holder) {}
+
+       @Override
+       public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
+         // (Re-)Compute the ideal size of the camera-preview display (the area that the
+         // camera-preview frames get rendered onto, potentially with scaling and rotation)
+         // based on the size of the SurfaceView that contains the display.
+         Size viewSize = new Size(width, height);
+         Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
+
+         // Connect the converter to the camera-preview frames as its input (via
+         // previewFrameTexture), and configure the output width and height as the computed
+         // display size.
+         converter.setSurfaceTextureAndAttachToGLContext(
+             previewFrameTexture, displaySize.getWidth(), displaySize.getHeight());
+       }
+
+       @Override
+       public void surfaceDestroyed(SurfaceHolder holder) {}
+     });
+```
+
+In this code block, we add a custom [`SurfaceHolder.Callback`] to
+`previewDisplayView` and implement the `surfaceChanged(SurfaceHolder holder, int
+format, int width, int height)` function to compute an appropriate display size
+of the camera frames on the device screen and to tie the `previewFrameTexture`
+object and send frames of the computed `displaySize` to the `converter`.
+
+We are now ready to use camera frames in a MediaPipe graph.
+
+## Using a MediaPipe graph in Android
+
+### Add relevant dependencies
+
+To use a MediaPipe graph, we need to add dependencies to the MediaPipe framework
+on Android. We will first add a build rule to build a `cc_binary` using JNI code
+of the MediaPipe framework and then build a `cc_library` rule to use this binary
+in our application. Add the following code block to your `BUILD` file:
+
+```
+cc_binary(
+    name = "libmediapipe_jni.so",
+    linkshared = 1,
+    linkstatic = 1,
+    deps = [
+        "//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
+    ],
+)
+
+cc_library(
+    name = "mediapipe_jni_lib",
+    srcs = [":libmediapipe_jni.so"],
+    alwayslink = 1,
+)
+```
+
+Add the dependency `":mediapipe_jni_lib"` to the `mediapipe_lib` build rule in
+the `BUILD` file.
+
+Next, we need to add dependencies specific to the MediaPipe graph we want to use
+in the application.
+
+First, add dependencies to all calculator code in the `libmediapipe_jni.so`
+build rule:
+
+```
+"//mediapipe/graphs/edge_detection:android_calculators",
+```
+
+MediaPipe graphs are `.pbtxt` files, but to use them in the application, we need
+to use the `mediapipe_binary_graph` build rule to generate a `.binarypb` file.
+We can then use an application specific alias for the graph via the `genrule`
+build rule. Add the following `genrule` to use an alias for the edge detection
+graph:
+
+```
+genrule(
+    name = "binary_graph",
+    srcs = ["//mediapipe/graphs/edge_detection:android_gpu_binary_graph"],
+    outs = ["edgedetectiongpu.binarypb"],
+    cmd = "cp $< $@",
+)
+```
+
+Then in the `mediapipe_lib` build rule, add assets:
+
+```
+assets = [
+  ":binary_graph",
+],
+assets_dir = "",
+```
+
+In the `assets` build rule, you can also add other assets such as TensorFlowLite
+models used in your graph.
+
+Now, the `MainActivity` needs to load the MediaPipe framework. Also, the
+framework uses OpenCV, so `MainActvity` should also load `OpenCV`. Use the
+following code in `MainActivity` (inside the class, but not inside any function)
+to load both dependencies:
+
+```
+static {
+  // Load all native libraries needed by the app.
+  System.loadLibrary("mediapipe_jni");
+  System.loadLibrary("opencv_java4");
+}
+```
+
+### Use the graph in `MainActivity`
+
+First, we need to load the asset which contains the `.binarypb` compiled from
+the `.pbtxt` file of the graph. To do this, we can use a MediaPipe utility,
+[`AndroidAssetUtil`].
+
+Initialize the asset manager in `onCreate(Bundle)` before initializing
+`eglManager`:
+
+```
+// Initilize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
+// binary graphs.
+AndroidAssetUtil.initializeNativeAssetManager(this);
+```
+
+Declare a static variable with the graph name, the name of the input stream and
+the name of the output stream:
+
+```
+private static final String BINARY_GRAPH_NAME = "edgedetectiongpu.binarypb";
+private static final String INPUT_VIDEO_STREAM_NAME = "input_video";
+private static final String OUTPUT_VIDEO_STREAM_NAME = "output_video";
+```
+
+Now, we need to setup a [`FrameProcessor`] object that sends camera frames
+prepared by the `converter` to the MediaPipe graph and runs the graph, prepares
+the output and then updates the `previewDisplayView` to display the output. Add
+the following code to declare the `FrameProcessor`:
+
+```
+private FrameProcessor processor;
+```
+
+and initialize it in `onCreate(Bundle)` after initializing `eglManager`:
+
+```
+processor =
+    new FrameProcessor(
+        this,
+        eglManager.getNativeContext(),
+        BINARY_GRAPH_NAME,
+        INPUT_VIDEO_STREAM_NAME,
+        OUTPUT_VIDEO_STREAM_NAME);
+```
+
+The `processor` needs to consume the converted frames from the `converter` for
+processing. Add the following line to `onResume()` after initializing the
+`converter`:
+
+```
+converter.setConsumer(processor);
+```
+
+The `processor` should send its output to `previewDisplayView` To do this, add
+the following function definitions to our custom [`SurfaceHolder.Callback`]:
+
+```
+@Override
+public void surfaceCreated(SurfaceHolder holder) {
+  processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
+}
+
+@Override
+public void surfaceDestroyed(SurfaceHolder holder) {
+  processor.getVideoSurfaceOutput().setSurface(null);
+}
+```
+
+When the `SurfaceHolder` is created, we had the `Surface` to the
+`VideoSurfaceOutput` of the `processor`. When it is destroyed, we remove it from
+the `VideoSurfaceOutput` of the `processor`.
+
+And that's it! You should now be able to successfully build and run the
+application on the device and see Sobel edge detection running on a live camera
+feed! Congrats!
+
+![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif){width="300"}
+
+If you ran into any issues, please see the full code of the tutorial
+[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu).
+
+[`AndroidAssetUtil`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/framework/AndroidAssetUtil.java
+[Bazel]:https://bazel.build/
+[`CameraHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraHelper.java
+[CameraX]:https://developer.android.com/training/camerax
+[`CameraXPreviewHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraXPreviewHelper.java
+[developer options]:https://developer.android.com/studio/debug/dev-options
+[`edge_detection_android_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_gpu.pbtxt
+[`EdgeDetectionGPU` example]:https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu/
+[`EglManager`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/glutil/EglManager.java
+[`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java
+[`FrameLayout`]:https://developer.android.com/reference/android/widget/FrameLayout
+[`FrameProcessor`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/FrameProcessor.java
+[MediaPipe installation guide]:./install.md
+[`PermissionHelper`]: https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/PermissionHelper.java
+[Setting up Android SDK and NDK]:./install.md#setting-up-android-sdk-and-ndk
+[`SurfaceHolder.Callback`]:https://developer.android.com/reference/android/view/SurfaceHolder.Callback.html
+[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
+[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
+[`SurfaceTexture`]:https://developer.android.com/reference/android/graphics/SurfaceTexture
+[`TextView`]:https://developer.android.com/reference/android/widget/TextView

mediapipe/docs/hello_world_desktop.md

@@ -0,0 +1,115 @@
+## Hello World for C++
+
+1.  Ensure you have a working version of MediaPipe. See
+    [installation instructions](./install.md).
+
+2.  To run the [`hello world`] example:
+
+    ```bash
+    $ git clone https://github.com/google/mediapipe/mediapipe.git
+    $ cd mediapipe
+
+    # Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is not supported currently.
+    $ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
+        mediapipe/examples/desktop/hello_world:hello_world
+
+    # It should print 10 rows of Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    ```
+
+3.  The [`hello world`] example uses a simple MediaPipe graph in the
+    `PrintHelloWorld()` function, defined in a [`CalculatorGraphConfig`] proto.
+
+    ```C++
+    ::mediapipe::Status PrintHelloWorld() {
+      // Configures a simple graph, which concatenates 2 PassThroughCalculators.
+      CalculatorGraphConfig config = ParseTextProtoOrDie<CalculatorGraphConfig>(R"(
+        input_stream: "in"
+        output_stream: "out"
+        node {
+          calculator: "PassThroughCalculator"
+          input_stream: "in"
+          output_stream: "out1"
+        }
+        node {
+          calculator: "PassThroughCalculator"
+          input_stream: "out1"
+          output_stream: "out"
+        }
+      )");
+    ```
+
+    You can visualize this graph using
+    [MediaPipe Visualizer](https://mediapipe-viz.appspot.com) by pasting the
+    CalculatorGraphConfig content below into the visualizer. See
+    [here](./visualizer.md) for help on the visualizer.
+
+    ```bash
+        input_stream: "in"
+        output_stream: "out"
+        node {
+          calculator: "PassThroughCalculator"
+          input_stream: "in"
+          output_stream: "out1"
+        }
+        node {
+          calculator: "PassThroughCalculator"
+          input_stream: "out1"
+          output_stream: "out"
+        }
+    ```
+
+    This graph consists of 1 graph input stream (`in`) and 1 graph output stream
+    (`out`), and 2 [`PassThroughCalculator`]s connected serially.
+
+    ![hello_world.cc graph](./images/hello_world_graph.png){width="200"}
+
+4.  Before running the graph, an `OutputStreamPoller` object is connected to the
+    output stream in order to later retrieve the graph output, and a graph run
+    is started with [`StartRun`].
+
+    ```c++
+    CalculatorGraph graph;
+    RETURN_IF_ERROR(graph.Initialize(config));
+    ASSIGN_OR_RETURN(OutputStreamPoller poller,
+                     graph.AddOutputStreamPoller("out"));
+    RETURN_IF_ERROR(graph.StartRun({}));
+    ```
+
+5.  The example then creates 10 packets (each packet contains a string "Hello
+    World!" with Timestamp values ranging from 0, 1, ... 9) using the
+    [`MakePacket`] function, adds each packet into the graph through the `in`
+    input stream, and finally closes the input stream to finish the graph run.
+
+    ```c++
+    for (int i = 0; i < 10; ++i) {
+      RETURN_IF_ERROR(graph.AddPacketToInputStream("in", MakePacket<std::string>("Hello World!").At(Timestamp(i))));
+    }
+    RETURN_IF_ERROR(graph.CloseInputStream("in"));
+    ```
+
+6.  Through the `OutputStreamPoller` object the example then retrieves all 10
+    packets from the output stream, gets the string content out of each packet
+    and prints it to the output log.
+
+    ```c++
+    mediapipe::Packet packet;
+    while (poller.Next(&packet)) {
+      LOG(INFO) << packet.Get<string>();
+    }
+    ```
+
+[`hello world`]: https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hello_world/hello_world.cc
+[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`PassThroughCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/pass_through_calculator.cc
+[`MakePacket`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/packet.h
+[`StartRun`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h

mediapipe/docs/help.md

@@ -0,0 +1,19 @@
+## Getting help
+
+-   [Technical questions](#technical-questions)
+-   [Bugs and Feature requests](#bugs-and-feature-requests)
+
+Below are the various ways to get help
+
+### Technical questions
+
+For help with technical or algorithmic questions, visit
+[Stack Overflow](https://stackoverflow.com/questions/tagged/mediapipe) to find
+answers and support from the MediaPipe community.
+
+### Bugs and Feature requests
+
+To report bugs or make feature requests,
+[file an issue on Github](https://github.com/google/mediapipe/mediapipe/issues).
+Please choose the appropriate repository for the project from the
+[MediaPipe repo](https://github.com/google/mediapipe/mediapipe)

mediapipe/docs/how_to_questions.md

@@ -0,0 +1,143 @@
+## Questions and Answers
+
+-   [How to convert ImageFrames and GpuBuffers](#how-to-convert-imageframes-and-gpubuffers)
+-   [How to visualize perceived results](#how-to-visualize-perception-results)
+-   [How to run calculators in parallel](#how-to-run-calculators-in-parallel)
+-   [Output timestamps when using ImmediateInputStreamHandler](#output-timestamps-when-using-immediateinputstreamhandler)
+-   [How to change settings at runtime](#how-to-change-settings-at-runtime)
+-   [How to process real-time input streams](#how-to-process-real-time-input-streams)
+-   [Can I run MediaPipe on MS Windows?](#can-i-run-mediapipe-on-ms-windows)
+
+### How to convert ImageFrames and GpuBuffers
+
+The Calculators [`ImageFrameToGpuBufferCalculator`] and
+[`GpuBufferToImageFrameCalculator`] convert back and forth between packets of
+type [`ImageFrame`] and [`GpuBuffer`]. [`ImageFrame`] refers to image data in
+CPU memory in any of a number of bitmap image formats. [`GpuBuffer`] refers to
+image data in GPU memory. You can find more detail in the Framework Concepts
+section
+[GpuBuffer to ImageFrame converters](./gpu.md).
+You can see an example in:
+
+   * [`object_detection_android_cpu.pbtxt`]
+
+### How to visualize perception results
+
+The [`AnnotationOverlayCalculator`] allows perception results, such as boudning
+boxes, arrows, and ovals, to be superimposed on the video frames aligned with
+the recognized objects. The results can be displayed in a diagnostic window when
+running on a workstation, or in a texture frame when running on device. You can
+see an example use of [`AnnotationOverlayCalculator`] in:
+
+   * [`face_detection_android_gpu.pbtxt`].
+
+### How to run calculators in parallel
+
+Within a calculator graph, MediaPipe routinely runs separate calculator nodes
+in parallel.  MediaPipe maintains a pool of threads, and runs each calculator
+as soon as a thread is available and all of it's inputs are ready.  Each
+calculator instance is only run for one set of inputs at a time, so most
+calculators need only to be *thread-compatible* and not *thread-safe*.
+
+In order to enable one calculator to process multiple inputs in parallel, there
+are two possible approaches:
+
+1.  Define multiple calulator nodes and dispatch input packets to all nodes.
+2.  Make the calculator thread-safe and configure its [`max_in_flight`] setting.
+
+The first approach can be followed using the calculators designed to distribute
+packets across other calculators, such as [`RoundRobinDemuxCalculator`]. A
+single [`RoundRobinDemuxCalculator`] can distribute successive packets across
+several identically configured [`ScaleImageCalculator`] nodes.
+
+The second approach allows up to [`max_in_flight`] invocations of the
+[`CalculatorBase::Process`] method on the same calculator node. The output
+packets from [`CalculatorBase::Process`] are automatically ordered by timestamp
+before they are passed along to downstream calculators.
+
+With either aproach, you must be aware that the calculator running in parallel
+cannot maintain internal state in the same way as a normal sequential
+calculator.
+
+### Output timestamps when using ImmediateInputStreamHandler
+
+The [`ImmediateInputStreamHandler`] delivers each packet as soon as it arrives
+at an input stream. As a result, it can deliver a packet
+with a higher timestamp from one input stream before delivering a packet with a
+lower timestamp from a different input stream. If these input timestamps are
+both used for packets sent to one output stream, that output stream will
+complain that the timestamps are not monotonically increasing. In order to
+remedy this, the calculator must take care to output a packet only after
+processing is complete for its timestamp. This could be accomplished by waiting
+until input packets have been received from all inputstreams for that timestamp,
+or by ignoring a packet that arrives with a timestamp that has already been
+processed.
+
+### How to change settings at runtime
+
+There are two main approaches to changing the settings of a calculator graph
+while the application is running:
+
+1. Restart the calculator graph with modified [`CalculatorGraphConfig`].
+2. Send new calculator options through packets on graph input-streams.
+
+The first approach has the advantage of leveraging [`CalculatorGraphConfig`]
+processing tools such as "subgraphs". The second approach has the advantage of
+allowing active calculators and packets to remain in-flight while settings
+change. Mediapipe contributors are currently investigating alternative approaches
+to achieve both of these adantages.
+
+### How to process realtime input streams
+
+The mediapipe framework can be used to process data streams either online or
+offline. For offline processing, packets are pushed into the graph as soon as
+calculators are ready to process those packets. For online processing, one
+packet for each frame is pushed into the graph as that frame is recorded.
+
+The MediaPipe framework requires only that successive packets be assigned
+monotonically increasing timestamps. By convention, realtime calculators and
+graphs use the recording time or the presentation time as the timestamp for each
+packet, with each timestamp representing microseconds since
+`Jan/1/1970:00:00:00`. This allows packets from various sources to be processed
+in a gloablly consistent order.
+
+Normally for offline processing, every input packet is processed and processing
+continues as long as necessary. For online processing, it is often necessary to
+drop input packets in order to keep pace with the arrival of input data frames.
+When inputs arrive too frequently, the recommended technique for dropping
+packets is to use the MediaPipe calculators designed specifically for this
+purpose such as [`RealTimeFlowLimiterCalculator`] and [`PacketClonerCalculator`].
+
+For online processing, it is also necessary to promptly determine when processing
+can proceed. MediaPipe supports this by propagating timestamp bounds between
+calculators. Timestamp bounds indicate timestamp intervals that will contain no
+input packets, and they allow calculators to begin processing for those
+timestamps immediately. Calculators designed for realtime processing should
+carefully calculate timestamp bounds in order to begin processing as promptly as
+possible. For example, the [`MakePairCalculator`] uses the `SetOffset` API to
+propagate timestamp bounds from input streams to output streams.
+
+### Can I run MediaPipe on MS Windows?
+
+Currently MediaPipe portability supports Debian Linux, Ubuntu Linux,
+MacOS, Android, and iOS.  The core of MediaPipe framework is a C++ library
+conforming to the C++11 standard, so it is relatively easy to port to
+additional platforms.
+
+[`object_detection_android_cpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_cpu.pbtxt
+
+[`ImageFrame`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/formats/image_frame.h
+[`GpuBuffer`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer.h
+[`GpuBufferToImageFrameCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer_to_image_frame_calculator.cc
+[`ImageFrameToGpuBufferCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/image_frame_to_gpu_buffer_calculator.cc
+[`AnnotationOverlayCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/util/annotation_overlay_calculator.cc
+[`face_detection_android_gpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_android_gpu.pbtxt
+[`CalculatorBase::Process`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
+[`max_in_flight`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`RoundRobinDemuxCalculator`]: https://github.com/google/mediapipe/tree/master//mediapipe/calculators/core/round_robin_demux_calculator.cc
+[`ScaleImageCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/image/scale_image_calculator.cc
+[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.cc
+[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`RealTimeFlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/real_time_flow_limiter_calculator.cc
+[`PacketClonerCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/packet_cloner_calculator.cc
+[`MakePairCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/make_pair_calculator.cc

mediapipe/docs/index.rst

@@ -0,0 +1,60 @@
+MediaPipe
+=====================================
+`MediaPipe <http://github.com/google/mediapipe>`_ is a graph-based framework for
+building multimodal (video, audio, and sensor) applied machine learning pipelines.
+MediaPipe is cross-platform running on mobile devices, workstations and servers,
+and supports mobile GPU acceleration. With MediaPipe, an applied
+machine learning pipeline can be built as a graph of modular components,
+including, for instance, inference models and media processing functions. Sensory
+data such as audio and video streams enter the graph, and perceived descriptions
+such as object-localization and face-landmark streams exit the graph. An example
+graph that performs real-time face detection on mobile GPU is shown below.
+
+.. image:: images/mobile/face_detection_android_gpu.png
+   :width: 400
+   :alt: Example MediaPipe graph
+
+MediaPipe is designed for machine learning (ML) practitioners, including
+researchers, students, and software developers, who implement production-ready
+ML applications, publish code accompanying research work, and build technology
+prototypes. The main use case for MediaPipe is rapid prototyping of applied
+machine learning pipelines with inference models and other reusable components.
+MediaPipe also facilitates the deployment of machine learning technology into
+demos and applications on a wide variety of different hardware platforms
+(e.g., Android, iOS, workstations).
+
+APIs for MediaPipe
+    * Calculator API in C++
+    * Graph Construction API in ProtoBuf
+    * (Coming Soon) Graph Construction API in C++
+    * Graph Execution API in C++
+    * Graph Execution API in Java (Android)
+    * (Coming Soon) Graph Execution API in Objective-C (iOS)
+
+User Documentation
+==================
+
+.. toctree::
+   :maxdepth: 3
+
+   install
+   concepts
+   calculator
+   Examples <examples>
+   visualizer
+   measure_performance
+   how_to_questions
+   troubleshooting
+   help
+   framework_concepts
+   gpu
+   scheduling_sync
+   license
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

mediapipe/docs/install.md

@@ -0,0 +1,384 @@
+## Installing MediaPipe
+
+Choose your operating system:
+
+-   [Dependences](#dependences)
+-   [Installing on Debian and Ubuntu](#installing-on-debian-and-ubuntu)
+-   [Installing on CentOS](#installing-on-centos)
+-   [Installing on macOS](#installing-on-macos)
+-   [Installing using Docker](#installing-using-docker)
+-   [Setting up Android SDK and NDK](#setting-up-android-sdk-and-ndk)
+
+### Dependences
+
+Required libraries
+
+*   Prefer OpenCV 3.x and above but can work with OpenCV 2.x (deprecation in the
+    future)
+
+*   Bazel 0.23 and above
+
+*   gcc and g++ version other than 6.3 and 7.3 (if you need TensorFlow
+    calculators/demos)
+
+*   Android SDK release 28.0.3 and above
+
+*   Android NDK r18b and above
+
+### Installing on Debian and Ubuntu
+
+1.  Checkout mediapipe repository
+
+    ```bash
+    $ git clone https://github.com/google/mediapipe/mediapipe.git
+
+    # Change directory into mediapipe root directory
+    $ cd mediapipe
+    ```
+
+2.  Install Bazel
+
+    Option 1. Use package manager tool to install the latest version of Bazel.
+
+    ```bash
+    $ sudo apt-get install bazel
+
+    # Run 'bazel version' to check version of bazel installed
+    ```
+
+    Option 2. Follow Bazel's
+    [documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
+    to install any version of Bazel manually.
+
+3.  Install OpenCV
+
+    Option 1. Use package manager tool to install the pre-compiled OpenCV
+    libraries.
+
+    Note that Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to
+    take option 2 or 3 to install OpenCV 3 or above.
+
+    ```bash
+    $ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \
+                           libopencv-imgproc-dev libopencv-video-dev
+    ```
+
+    Option 2. Run [`setup_opencv.sh`] to automatically build OpenCV from source
+    and modify MediaPipe's OpenCV config.
+
+    Option 3. Follow OpenCV's
+    [documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
+    to manually build OpenCV from source code.
+
+    You may need to modify [`WORKSAPCE`] and [`opencv_linux.BUILD`] to point
+    MediaPipe to your own OpenCV libraries. For example, if OpenCV 4 is
+    installed in "/usr/local/", you need to update the "linux_opencv"
+    new_local_repository rule in [`WORKSAPCE`] and "opencv" cc_library rule in
+    [`opencv_linux.BUILD`] to be:
+
+    ```bash
+    new_local_repository(
+        name = "linux_opencv",
+        build_file = "@//third_party:opencv_linux.BUILD",
+        path = "/usr/local",
+    )
+
+    cc_library(
+      name = "opencv",
+      srcs = glob(
+          [
+              "lib/libopencv_core.so*",
+              "lib/libopencv_highgui.so*",
+              "lib/libopencv_imgcodecs.so*",
+              "lib/libopencv_imgproc.so*",
+              "lib/libopencv_video.so*",
+              "lib/libopencv_videoio.so*",
+
+          ],
+      ),
+      hdrs = glob(["include/opencv4/**/*.h*"]),
+      includes = ["include/opencv4/"],
+      linkstatic = 1,
+      visibility = ["//visibility:public"],
+    )
+
+    ```
+
+4.  Run the hello world desktop example
+
+    ```bash
+    # Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
+    $ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
+        mediapipe/examples/desktop/hello_world:hello_world
+
+    # Should print:
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    ```
+
+### Installing on CentOS
+
+1.  Checkout mediapipe repository
+
+    ```bash
+    $ git clone https://github.com/google/mediapipe/mediapipe.git
+
+    # Change directory into mediapipe root directory
+    $ cd mediapipe
+    ```
+
+2.  Install Bazel
+
+    Follow Bazel's
+    [documentation](https://docs.bazel.build/versions/master/install-redhat.html)
+    to install Bazel manually.
+
+3.  Install OpenCV
+
+    Option 1. Use package manager tool to install the pre-compiled version.
+
+    Note that yum installs OpenCV 2.4.5, which may have an opencv/gstreamer
+    [issue](https://github.com/opencv/opencv/issues/4592).
+
+    ```bash
+    $ sudo yum install opencv-devel
+    ```
+
+    Option 2. Build OpenCV from source code.
+
+    You may need to modify [`WORKSAPCE`] and [`opencv_linux.BUILD`] to point
+    MediaPipe to your own OpenCV libraries. For example, if OpenCV 4 is
+    installed in "/usr/local/", you need to update the "linux_opencv"
+    new_local_repository rule in [`WORKSAPCE`] and "opencv" cc_library rule in
+    [`opencv_linux.BUILD`] to be:
+
+    ```bash
+    new_local_repository(
+        name = "linux_opencv",
+        build_file = "@//third_party:opencv_linux.BUILD",
+        path = "/usr/local",
+    )
+
+    cc_library(
+      name = "opencv",
+      srcs = glob(
+          [
+              "lib/libopencv_core.so*",
+              "lib/libopencv_highgui.so*",
+              "lib/libopencv_imgcodecs.so*",
+              "lib/libopencv_imgproc.so*",
+              "lib/libopencv_video.so*",
+              "lib/libopencv_videoio.so*",
+
+          ],
+      ),
+      hdrs = glob(["include/opencv4/**/*.h*"]),
+      includes = ["include/opencv4/"],
+      linkstatic = 1,
+      visibility = ["//visibility:public"],
+    )
+
+    ```
+
+4.  Run the hello world desktop example
+
+    ```bash
+    # Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
+    $ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
+        mediapipe/examples/desktop/hello_world:hello_world
+
+    # Should print:
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    ```
+
+### Installing on macOS
+
+1.  Checkout mediapipe repository
+
+    ```bash
+    $ git clone https://github.com/google/mediapipe/mediapipe.git
+
+    $ cd mediapipe
+    ```
+
+2.  Install Bazel
+
+    Option 1. Use package manager tool to install the latest version of Bazel.
+
+    ```bash
+    $ brew install bazel
+
+    # Run 'bazel version' to check version of bazel installed
+    ```
+
+    Option 2. Follow Bazel's
+    [documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
+    to install any version of Bazel manually.
+
+3.  Install OpenCV
+
+    Use package manager tool to install the pre-compiled OpenCV libraries.
+
+    ```bash
+    $ brew install opencv
+    ```
+
+4.  Run the hello world desktop example
+
+    ```bash
+    # Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
+    $ bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' \
+        mediapipe/examples/desktop/hello_world:hello_world
+
+    # Should print:
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    ```
+
+### Installing using Docker
+
+This will use a Docker image that will isolate mediapipe's installation from the rest of the system.
+
+1.  [Install Docker](https://docs.docker.com/install/#supported-platforms) on
+    your host sytem
+
+2.  Build a docker image with tag "mediapipe"
+
+    ```bash
+    $ git clone https://github.com/google/mediapipe/mediapipe.git
+    $ cd mediapipe
+    $ docker build --tag=mediapipe .
+
+    # Should print:
+    # Sending build context to Docker daemon  147.8MB
+    # Step 1/9 : FROM ubuntu:latest
+    # latest: Pulling from library/ubuntu
+    # 6abc03819f3e: Pull complete
+    # 05731e63f211: Pull complete
+    # ........
+    # See http://bazel.build/docs/getting-started.html to start a new project!
+    # Removing intermediate container 82901b5e79fa
+    # ---> f5d5f402071b
+    # Step 9/9 : COPY . /mediapipe/
+    # ---> a95c212089c5
+    # Successfully built a95c212089c5
+    # Successfully tagged mediapipe:latest
+    ```
+
+3.  Run the hello world desktop example in docker
+
+    ```bash
+    $ docker run -it --name mediapipe mediapipe:latest
+
+    root@bca08b91ff63:/mediapipe# bazel run --define 'MEDIAPIPE_DISABLE_GPU=1' mediapipe/examples/desktop/hello_world:hello_world
+
+    # Should print:
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    # Hello World!
+    ```
+
+<!-- 4.  Uncomment the last line of the Dockerfile
+
+    ```bash
+    RUN bazel build -c opt --define 'MEDIAPIPE_DISABLE_GPU=1' mediapipe/examples/desktop/demo:object_detection_tensorflow_demo
+    ```
+
+    and rebuild the image and then run the docker image
+
+    ```bash
+    docker build --tag=mediapipe .
+    docker run -i -t mediapipe:latest
+    ``` -->
+
+
+### Setting up Android Studio with MediaPipe
+
+The steps below use Android Studio to build and install a MediaPipe demo app.
+
+1.  Install and launch android studio.
+
+2.  Select `Configure` | `SDK Manager` | `SDK Platforms`
+
+    *   verify that an Android SDK is installed
+    *   note the Android SDK Location such as `/usr/local/home/Android/Sdk`
+
+3.  Select `Configure` | `SDK Manager` | `SDK Tools`
+
+    *   verify that an Android NDK is installed
+    *   note the Android NDK Location such as `/usr/local/home/Android/Sdk/ndk-bundle`
+
+4.  Set environment variables `$ANDROID_HOME` and `$ANDROID_NDK_HOME` to point to
+    the installed SDK and NDK.
+
+    ```bash
+    export ANDROID_HOME=/usr/local/home/Android/Sdk
+    export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk-bundle
+    ```
+
+5.  Select `Configure` | `Plugins` install `Bazel`.
+
+6.  Select `Import Bazel Project`
+
+    *   select `Workspace`: `/path/to/mediapipe`
+    *   select `Generate from BUILD file`: `/path/to/mediapipe/BUILD`
+    *   select `Finish`
+
+7.  Connect an android device to the workstation.
+
+8.  Select `Run...` | `Edit Configurations...`
+
+    *   enter Target Expression:
+        `//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu`
+    *   enter Bazel command: `mobile-install`
+    *   enter Bazel flags: `-c opt --config=android_arm64` select `Run`
+
+### Setting up Android SDK and NDK
+
+If Android SDK and NDK are installed (likely by Android Studio), please set
+$ANDROID_HOME and $ANDROID_NDK_HOME to point to the installed SDK and NDK.
+
+```bash
+export ANDROID_HOME=<path to the Android SDK>
+export ANDROID_NDK_HOME=<path to the Android NDK>
+```
+
+Otherwise, please run [`setup_android_sdk_and_ndk.sh`] to download and setup
+Android SDK and NDK for MediaPipe before building any Android demos.
+
+[`WORKSAPCE`]: https://github.com/google/mediapipe/tree/master/WORKSPACE
+[`opencv_linux.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_linux.BUILD
+[`setup_opencv.sh`]: https://github.com/google/mediapipe/tree/master/setup_opencv.sh
+[`setup_android_sdk_and_ndk.sh`]: https://github.com/google/mediapipe/tree/master/setup_android_sdk_and_ndk.sh

mediapipe/docs/license.md

@@ -0,0 +1,205 @@
+License
+===============
+Copyright 2019 The MediaPipe Authors.  All rights reserved.
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2017, The MediaPipe Authors.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.\n

mediapipe/docs/measure_performance.md

@@ -0,0 +1,18 @@
+# Measuring Performance
+
+*Coming soon.*
+
+MediaPipe includes APIs for gathering aggregate performance data and
+event timing data for CPU and GPU operations.  These API's can be found at:
+
+<!-- TODO: Update the source code URL's to local or public URL's -->
+
+   * [`GraphProfiler`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/profiler/graph_profiler.h):
+     Accumulates for each running calculator a histogram of latencies for
+     Process calls.
+   * [`GraphTracer`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/profiler/graph_tracer.h):
+     Records for each running calculator and each processed packet a series
+     of timed events including the start and finish of each Process call.
+
+Future mediapipe releases will include tools for visualizing and analysing
+the latency histograms and timed events captured by these API's.

mediapipe/docs/media_sequence.md

@@ -0,0 +1,198 @@
+## Preparing Data Sets with MediaSequence
+
+MediaPipe is useful and general framework for media processing that can
+assist with research, development, and deployment of ML models. This example
+focuses on development by demonstrating how to prepare video data for training
+a TensorFlow model.
+
+The MediaSequence library provides an extensive set of tools for storing data in
+TensorFlow.SequenceExamples. SequenceExamples provide matched semantics to most
+video tasks and are efficient to use with TensorFlow. The sequence semantics
+allow for a variable number of annotations per frame, which is necessary for
+tasks like video object detection, but very difficult to encode in
+TensorFlow.Examples. The goal of MediaSequence is to simplify working with
+SequenceExamples and to automate common preparation tasks. Much more information
+is available about the MediaSequence pipeline, including how to use it to
+process new data sets, in the [documentation](https://github.com/google/mediapipe/tree/master/mediapipe/util/sequence/README.md).
+
+### Preparing an example data set
+
+1.  Checkout mediapipe repository
+
+    ```bash
+    git clone https://github.com/google/mediapipe/mediapipe
+    cd mediapipe
+    ```
+
+1.  Compile the MediaSequence demo C++ binary
+
+    ```bash
+    bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo --define 'MEDIAPIPE_DISABLE_GPU=1'
+    ```
+
+    MediaSequence uses C++ binaries to improve multimedia processing speed and
+    encourage a strong separation between annotations and the image data or
+    other features. The binary code is very general in that it reads from files
+    into input side packets and writes output side packets to files when
+    completed, but it also links in all of the calculators for necessary for
+    the MediaPipe graphs preparing the Charades data set.
+
+1.  Download and prepare the data set through Python
+
+    To run this step, you must have Python 2.7 or 3.5+ installed with the
+    TensorFlow 1.19+ package installed.
+
+    ```bash
+    python -m mediapipe.examples.desktop.media_sequence.demo_dataset \
+      --path_to_demo_data=/tmp/demo_data/ \
+      --path_to_mediapipe_binary=bazel-bin/mediapipe/examples/desktop/media_sequence/media_sequence_demo \
+      --path_to_graph_directory=mediapipe/graphs/media_sequence/
+    ```
+
+    The arguments define where data is stored. `--path_to_demo_data` defines
+    where the data will be downloaded to and where prepared data will be
+    generated. `--path_to_mediapipe_binary` is the path to the binary built in
+    the previous step. `--path_to_graph_directory` defines where to look for
+    MediaPipe graphs during processing.
+
+    Running this module
+    1. Downloads videos from the internet.
+    1. For each annotation in a CSV, creates a structured metadata file.
+    1. Runs MediaPipe to extract images as defined by the metadata.
+    1. Stores the results in numbered set of TFRecords files.
+
+    MediaSequence uses SequenceExamples as the format of both inputs and
+    outputs. Annotations are encoded as inputs in a SequenceExample of metadata
+    that defines the labels and the path to the cooresponding video file. This
+    metadata is passed as input to the C++ `media_sequence_demo` binary, and the
+    output is a SequenceExample filled with images and annotations ready for
+    model training.
+
+1.  Reading the data in TensorFlow
+
+    To read the data in tensorflow, first add the repo to your PYTHONPATH
+
+    ```bash
+    PYTHONPATH="${PYTHONPATH};"+`pwd`
+    ```
+
+    and then you can import the data set in Python.
+
+    ```python
+    import tensorflow as tf
+    from mediapipe.examples.desktop.media_sequence.demo_dataset import DemoDataset
+    demo_data_path = '/tmp/demo_data/'
+    with tf.Graph().as_default():
+       d = DemoDataset(demo_data_path)
+       dataset = d.as_dataset("test")
+       # implement additional processing and batching here
+       output = dataset.make_one_shot_iterator().get_next()
+
+       with tf.Session() as sess:
+         output_ = sess.run(output)
+    ```
+
+### Preparing a practical data set
+As an example of processing a practical data set, a similar set of commands will
+prepare the [Charades data set](https://allenai.org/plato/charades/). The
+Charades data set is a data set of human action recognition collected with and
+maintained by the Allen Institute for Artificial Intelligence. To follow this
+code lab, you must abide by the [license](https://allenai.org/plato/charades/license.txt)
+for the Charades data set provided by the Allen Institute.
+
+The Charades data set is large (~150 GB), and will take considerable time to
+download and process (4-8 hours).
+
+```bash
+bazel build -c opt mediapipe/examples/desktop/media_sequence:media_sequence_demo --define 'MEDIAPIPE_DISABLE_GPU=1'
+
+python -m mediapipe.examples.desktop.media_sequence.demo_dataset \
+  --alsologtostderr \
+  --path_to_charades_data=/tmp/demo_data/ \
+  --path_to_mediapipe_binary=bazel-bin/mediapipe/examples/desktop/media_sequence/media_sequence_demo \
+  --path_to_graph_directory=mediapipe/graphs/media_sequence/
+```
+
+### Preparing your own data set
+The process for preparing your own data set is described in the [MediaSequence
+documentation](https://github.com/google/mediapipe/tree/master/mediapipe/util/sequence/README.md).
+The Python code for Charades can easily be modified to process most annotations,
+but the MediaPipe processing warrants further discussion. MediaSequence uses
+MediaPipe graphs to extract features related to the metadata or previously
+extracted data. Each graph can focus on extracting a single type of feature, and
+graphs can be chained together to extract derived features in a composable way.
+For example, one graph may extract images from a video at 10 fps and another
+graph extract images at 24 fps. A subsequent graph can extract ResNet-50
+features from the output of either preceding graph. MediaPipe enables a
+composable interface of data process for machine learning at multiple levels.
+
+The MediaPipe graph with brief annotations for adding images to a data set is as
+follows. Common changes would be to change the frame_rate or encoding quality of
+frames.
+
+```
+# Convert the string input into a decoded SequenceExample.
+node {
+  calculator: "StringToSequenceExampleCalculator"
+  input_side_packet: "STRING:input_sequence_example"
+  output_side_packet: "SEQUENCE_EXAMPLE:parsed_sequence_example"
+}
+
+# Unpack the data path and clip timing from the SequenceExample.
+node {
+  calculator: "UnpackMediaSequenceCalculator"
+  input_side_packet: "SEQUENCE_EXAMPLE:parsed_sequence_example"
+  output_side_packet: "DATA_PATH:input_video_path"
+  output_side_packet: "RESAMPLER_OPTIONS:packet_resampler_options"
+  options {
+    [mediapipe.UnpackMediaSequenceCalculatorOptions.ext]: {
+      base_packet_resampler_options {
+        frame_rate: 24.0
+        base_timestamp: 0
+      }
+    }
+  }
+}
+
+# Decode the entire video.
+node {
+  calculator: "OpenCvVideoDecoderCalculator"
+  input_side_packet: "INPUT_FILE_PATH:input_video_path"
+  output_stream: "VIDEO:decoded_frames"
+}
+
+# Extract the subset of frames we want to keep.
+node {
+  calculator: "PacketResamplerCalculator"
+  input_stream: "decoded_frames"
+  output_stream: "sampled_frames"
+  input_side_packet: "OPTIONS:packet_resampler_options"
+}
+
+# Encode the images to store in the SequenceExample.
+node {
+  calculator: "OpenCvImageEncoderCalculator"
+  input_stream: "sampled_frames"
+  output_stream: "encoded_frames"
+  node_options {
+    [type.googleapis.com/mediapipe.OpenCvImageEncoderCalculatorOptions]: {
+      quality: 80
+    }
+  }
+}
+
+# Store the images in the SequenceExample.
+node {
+  calculator: "PackMediaSequenceCalculator"
+  input_side_packet: "SEQUENCE_EXAMPLE:parsed_sequence_example"
+  output_side_packet: "SEQUENCE_EXAMPLE:sequence_example_to_serialize"
+  input_stream: "IMAGE:encoded_frames"
+}
+
+# Serialize the SequenceExample to a string for storage.
+node {
+  calculator: "StringToSequenceExampleCalculator"
+  input_side_packet: "SEQUENCE_EXAMPLE:sequence_example_to_serialize"
+  output_side_packet: "STRING:output_sequence_example"
+}
+```

mediapipe/docs/object_detection_android_cpu.md

@@ -0,0 +1,254 @@
+# Object Detection on CPU on Android
+
+Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
+general instructions to develop an Android application that uses MediaPipe. This
+doc focuses on the
+[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_cpu.pbtxt)
+that performs object detection with TensorFlow Lite on CPU.
+
+This is very similar to the
+[Object Detection on GPU on Android](object_detection_android_gpu.md) example
+except that at the beginning and the end of the graph it performs GPU-to-CPU and
+CPU-to-GPU image transfer respectively. As a result, the rest of graph, which
+shares the same configuration as the
+[GPU graph](images/mobile/object_detection_android_gpu.png), runs entirely on
+CPU.
+
+![object_detection_android_cpu_gif](images/mobile/object_detection_android_cpu.gif){width="300"}
+
+## App
+
+The graph is used in the
+[Object Detection CPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu)
+example app. To build the app, run:
+
+```bash
+bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu
+```
+
+To further install the app on android device, run:
+
+```bash
+adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu/objectdetectioncpu.apk
+```
+
+## Graph
+
+![object_detection_android_cpu_graph](images/mobile/object_detection_android_cpu.png){width="400"}
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into [MediaPipe Visualizer](https://mediapipe-viz.appspot.com/).
+
+```bash
+# MediaPipe graph that performs object detection with TensorFlow Lite on CPU.
+# Used in the example in
+# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectioncpu.
+
+# Images on GPU coming into and out of the graph.
+input_stream: "input_video"
+output_stream: "output_video"
+
+# Transfers the input image from GPU to CPU memory for the purpose of
+# demonstrating a CPU-based pipeline. Note that the input image on GPU has the
+# origin defined at the bottom-left corner (OpenGL convention). As a result,
+# the transferred image on CPU also shares the same representation.
+node: {
+  calculator: "GpuBufferToImageFrameCalculator"
+  input_stream: "input_video"
+  output_stream: "input_video_cpu"
+}
+
+# Throttles the images flowing downstream for flow control. It passes through
+# the very first incoming image unaltered, and waits for
+# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
+# generating the corresponding detections before it passes through another
+# image. All images that come in while waiting are dropped, limiting the number
+# of in-flight images between this calculator and
+# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
+# from queuing up incoming images and data excessively, which leads to increased
+# latency and memory usage, unwanted in real-time mobile applications. It also
+# eliminates unnecessarily computation, e.g., a transformed image produced by
+# ImageTransformationCalculator may get dropped downstream if the subsequent
+# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
+# processing previous inputs.
+node {
+  calculator: "RealTimeFlowLimiterCalculator"
+  input_stream: "input_video_cpu"
+  input_stream: "FINISHED:detections"
+  input_stream_info: {
+    tag_index: "FINISHED"
+    back_edge: true
+  }
+  output_stream: "throttled_input_video_cpu"
+}
+
+# Transforms the input image on CPU to a 320x320 image. To scale the image, by
+# default it uses the STRETCH scale mode that maps the entire input image to the
+# entire transformed image. As a result, image aspect ratio may be changed and
+# objects in the image may be deformed (stretched or squeezed), but the object
+# detection model used in this graph is agnostic to that deformation.
+node: {
+  calculator: "ImageTransformationCalculator"
+  input_stream: "IMAGE:throttled_input_video_cpu"
+  output_stream: "IMAGE:transformed_input_video_cpu"
+  node_options: {
+    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
+      output_width: 320
+      output_height: 320
+    }
+  }
+}
+
+# Converts the transformed input image on CPU into an image tensor as a
+# TfLiteTensor. The zero_center option is set to true to normalize the
+# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f]. The flip_vertically
+# option is set to true to account for the descrepancy between the
+# representation of the input image (origin at the bottom-left corner) and what
+# the model used in this graph is expecting (origin at the top-left corner).
+node {
+  calculator: "TfLiteConverterCalculator"
+  input_stream: "IMAGE:transformed_input_video_cpu"
+  output_stream: "TENSORS:image_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
+      zero_center: true
+      flip_vertically: true
+    }
+  }
+}
+
+# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
+# vector of tensors representing, for instance, detection boxes/keypoints and
+# scores.
+node {
+  calculator: "TfLiteInferenceCalculator"
+  input_stream: "TENSORS:image_tensor"
+  output_stream: "TENSORS:detection_tensors"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+      model_path: "ssdlite_object_detection.tflite"
+    }
+  }
+}
+
+# Generates a single side packet containing a vector of SSD anchors based on
+# the specification in the options.
+node {
+  calculator: "SsdAnchorsCalculator"
+  output_side_packet: "anchors"
+  node_options: {
+    [type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
+      num_layers: 6
+      min_scale: 0.2
+      max_scale: 0.95
+      input_size_height: 320
+      input_size_width: 320
+      anchor_offset_x: 0.5
+      anchor_offset_y: 0.5
+      strides: 16
+      strides: 32
+      strides: 64
+      strides: 128
+      strides: 256
+      strides: 512
+      aspect_ratios: 1.0
+      aspect_ratios: 2.0
+      aspect_ratios: 0.5
+      aspect_ratios: 3.0
+      aspect_ratios: 0.3333
+      reduce_boxes_in_lowest_layer: true
+    }
+  }
+}
+
+# Decodes the detection tensors generated by the TensorFlow Lite model, based on
+# the SSD anchors and the specification in the options, into a vector of
+# detections. Each detection describes a detected object.
+node {
+  calculator: "TfLiteTensorsToDetectionsCalculator"
+  input_stream: "TENSORS:detection_tensors"
+  input_side_packet: "ANCHORS:anchors"
+  output_stream: "DETECTIONS:detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
+      num_classes: 91
+      num_boxes: 2034
+      num_coords: 4
+      ignore_classes: 0
+      sigmoid_score: true
+      apply_exponential_on_box_size: true
+      x_scale: 10.0
+      y_scale: 10.0
+      h_scale: 5.0
+      w_scale: 5.0
+      flip_vertically: true
+    }
+  }
+}
+
+# Performs non-max suppression to remove excessive detections.
+node {
+  calculator: "NonMaxSuppressionCalculator"
+  input_stream: "detections"
+  output_stream: "filtered_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
+      min_suppression_threshold: 0.4
+      min_score_threshold: 0.6
+      max_num_detections: 3
+      overlap_type: INTERSECTION_OVER_UNION
+    }
+  }
+}
+
+# Maps detection label IDs to the corresponding label text. The label map is
+# provided in the label_map_path option.
+node {
+  calculator: "DetectionLabelIdToTextCalculator"
+  input_stream: "filtered_detections"
+  output_stream: "output_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
+      label_map_path: "ssdlite_object_detection_labelmap.txt"
+    }
+  }
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTION_VECTOR:output_detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the CPU copy of the original
+# image coming into the graph. The calculator assumes that image origin is
+# always at the top-left corner and renders text accordingly. However, the input
+# image has its origin at the bottom-left corner (OpenGL convention) and the
+# flip_text_vertically option is set to true to compensate that.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "INPUT_FRAME:throttled_input_video_cpu"
+  input_stream: "render_data"
+  output_stream: "OUTPUT_FRAME:output_video_cpu"
+  node_options: {
+    [type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
+      flip_text_vertically: true
+    }
+  }
+}
+
+# Transfers the annotated image from CPU back to GPU memory, to be sent out of
+# the graph.
+node: {
+  calculator: "ImageFrameToGpuBufferCalculator"
+  input_stream: "output_video_cpu"
+  output_stream: "output_video"
+}
+```

mediapipe/docs/object_detection_android_gpu.md

@@ -0,0 +1,231 @@
+# Object Detection on GPU on Android
+
+Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
+general instructions to develop an Android application that uses MediaPipe. This
+doc focuses on the
+[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_android_gpu.pbtxt)
+that performs object detection with TensorFlow Lite on GPU.
+
+![object_detection_android_gpu_gif](images/mobile/object_detection_android_gpu.gif){width="300"}
+
+## App
+
+The graph is used in the
+[Object Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu)
+example app. To build the app, run:
+
+```bash
+bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu
+```
+
+To further install the app on android device, run:
+
+```bash
+adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
+```
+
+## Graph
+
+![object_detection_android_gpu_graph](images/mobile/object_detection_android_gpu.png){width="400"}
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into [MediaPipe Visualizer](https://mediapipe-viz.appspot.com/).
+
+```bash
+# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
+# Used in the example in
+# mediapipie/examples/android/src/java/com/mediapipe/apps/objectdetectiongpu.
+
+# Images on GPU coming into and out of the graph.
+input_stream: "input_video"
+output_stream: "output_video"
+
+# Throttles the images flowing downstream for flow control. It passes through
+# the very first incoming image unaltered, and waits for
+# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
+# generating the corresponding detections before it passes through another
+# image. All images that come in while waiting are dropped, limiting the number
+# of in-flight images between this calculator and
+# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
+# from queuing up incoming images and data excessively, which leads to increased
+# latency and memory usage, unwanted in real-time mobile applications. It also
+# eliminates unnecessarily computation, e.g., a transformed image produced by
+# ImageTransformationCalculator may get dropped downstream if the subsequent
+# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
+# processing previous inputs.
+node {
+  calculator: "RealTimeFlowLimiterCalculator"
+  input_stream: "input_video"
+  input_stream: "FINISHED:detections"
+  input_stream_info: {
+    tag_index: "FINISHED"
+    back_edge: true
+  }
+  output_stream: "throttled_input_video"
+}
+
+# Transforms the input image on GPU to a 320x320 image. To scale the image, by
+# default it uses the STRETCH scale mode that maps the entire input image to the
+# entire transformed image. As a result, image aspect ratio may be changed and
+# objects in the image may be deformed (stretched or squeezed), but the object
+# detection model used in this graph is agnostic to that deformation.
+node: {
+  calculator: "ImageTransformationCalculator"
+  input_stream: "IMAGE_GPU:throttled_input_video"
+  output_stream: "IMAGE_GPU:transformed_input_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
+      output_width: 320
+      output_height: 320
+    }
+  }
+}
+
+# Converts the transformed input image on GPU into an image tensor stored in
+# tflite::gpu::GlBuffer. The zero_center option is set to true to normalize the
+# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f]. The flip_vertically
+# option is set to true to account for the descrepancy between the
+# representation of the input image (origin at the bottom-left corner, the
+# OpenGL convention) and what the model used in this graph is expecting (origin
+# at the top-left corner).
+node {
+  calculator: "TfLiteConverterCalculator"
+  input_stream: "IMAGE_GPU:transformed_input_video"
+  output_stream: "TENSORS_GPU:image_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
+      zero_center: true
+      flip_vertically: true
+    }
+  }
+}
+
+# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
+# vector of tensors representing, for instance, detection boxes/keypoints and
+# scores.
+node {
+  calculator: "TfLiteInferenceCalculator"
+  input_stream: "TENSORS_GPU:image_tensor"
+  output_stream: "TENSORS_GPU:detection_tensors"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+      model_path: "ssdlite_object_detection.tflite"
+    }
+  }
+}
+
+# Generates a single side packet containing a vector of SSD anchors based on
+# the specification in the options.
+node {
+  calculator: "SsdAnchorsCalculator"
+  output_side_packet: "anchors"
+  node_options: {
+    [type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
+      num_layers: 6
+      min_scale: 0.2
+      max_scale: 0.95
+      input_size_height: 320
+      input_size_width: 320
+      anchor_offset_x: 0.5
+      anchor_offset_y: 0.5
+      strides: 16
+      strides: 32
+      strides: 64
+      strides: 128
+      strides: 256
+      strides: 512
+      aspect_ratios: 1.0
+      aspect_ratios: 2.0
+      aspect_ratios: 0.5
+      aspect_ratios: 3.0
+      aspect_ratios: 0.3333
+      reduce_boxes_in_lowest_layer: true
+    }
+  }
+}
+
+# Decodes the detection tensors generated by the TensorFlow Lite model, based on
+# the SSD anchors and the specification in the options, into a vector of
+# detections. Each detection describes a detected object.
+node {
+  calculator: "TfLiteTensorsToDetectionsCalculator"
+  input_stream: "TENSORS_GPU:detection_tensors"
+  input_side_packet: "ANCHORS:anchors"
+  output_stream: "DETECTIONS:detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
+      num_classes: 91
+      num_boxes: 2034
+      num_coords: 4
+      ignore_classes: 0
+      sigmoid_score: true
+      apply_exponential_on_box_size: true
+      x_scale: 10.0
+      y_scale: 10.0
+      h_scale: 5.0
+      w_scale: 5.0
+      flip_vertically: true
+    }
+  }
+}
+
+# Performs non-max suppression to remove excessive detections.
+node {
+  calculator: "NonMaxSuppressionCalculator"
+  input_stream: "detections"
+  output_stream: "filtered_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
+      min_suppression_threshold: 0.4
+      min_score_threshold: 0.6
+      max_num_detections: 3
+      overlap_type: INTERSECTION_OVER_UNION
+    }
+  }
+}
+
+# Maps detection label IDs to the corresponding label text. The label map is
+# provided in the label_map_path option.
+node {
+  calculator: "DetectionLabelIdToTextCalculator"
+  input_stream: "filtered_detections"
+  output_stream: "output_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
+      label_map_path: "ssdlite_object_detection_labelmap.txt"
+    }
+  }
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTION_VECTOR:output_detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the original image coming into
+# the graph. Annotation drawing is performed on CPU, and the result is
+# transferred to GPU and overlaid on the input image. The calculator assumes
+# that image origin is always at the top-left corner and renders text
+# accordingly. However, the input image has its origin at the bottom-left corner
+# (OpenGL convention) and the flip_text_vertically option is set to true to
+# compensate that.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "INPUT_FRAME_GPU:throttled_input_video"
+  input_stream: "render_data"
+  output_stream: "OUTPUT_FRAME_GPU:output_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
+      flip_text_vertically: true
+    }
+  }
+}
+```

mediapipe/docs/object_detection_desktop.md

@@ -0,0 +1,428 @@
+## Object Detection on Desktop
+
+This is an example of using MediaPipe to run object detection models (TensorFlow
+and TensorFlow Lite) and render bounding boxes on the detected objects. To know
+more about the object detection models and TensorFlow-to-TFLite model
+conversion, please refer to the model [`README file`]. Moreover, if you are
+interested in running the same TensorfFlow Lite model on Android, please see the
+[Object Detection on GPU on Android](object_detection_android_gpu.md) and
+[Object Detection on CPU on Android](object_detection_android_cpu.md) examples.
+
+### TensorFlow Model
+
+To build and run the TensorFlow example on desktop, run:
+
+```bash
+# Note that this command also builds TensorFlow targets from scratch, it may
+# take a long time (e.g., up to 30 mins) to build for the first time.
+$ bazel build -c opt \
+    --define 'MEDIAPIPE_DISABLE_GPU=1' \
+    --define 'no_aws_support=true' \
+    mediapipe/examples/desktop/object_detection:object_detection_tensorflow
+
+# It should print:
+# Target //mediapipe/examples/desktop/object_detection:object_detection_tensorflow up-to-date:
+#   bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tensorflow
+# INFO: Elapsed time: 172.262s, Critical Path: 125.68s
+# INFO: 2675 processes: 2673 linux-sandbox, 2 local.
+# INFO: Build completed successfully, 2807 total actions
+
+# Replace <input video path> and <output video path>.
+# You can find a test video in mediapipe/examples/desktop/object_detection.
+$ bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tensorflow \
+  --calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tensorflow_graph.pbtxt \
+  --input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
+```
+
+#### Graph
+
+![graph visualization](images/object_detection_desktop_tensorflow.png){width="800"}
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into
+[MediaPipe Visualizer](https://mediapipe-viz.appspot.com).
+
+```bash
+# MediaPipe graph that performs object detection on desktop with TensorFlow
+# on CPU.
+# Used in the example in
+# mediapipie/examples/desktop/object_detection:object_detection_tensorflow.
+
+# Decodes an input video file into images and a video header.
+node {
+  calculator: "OpenCvVideoDecoderCalculator"
+  input_side_packet: "INPUT_FILE_PATH:input_video_path"
+  output_stream: "VIDEO:input_video"
+  output_stream: "VIDEO_PRESTREAM:input_video_header"
+}
+
+# Converts the input image into an image tensor as a tensorflow::Tensor.
+node {
+  calculator: "ImageFrameToTensorCalculator"
+  input_stream: "input_video"
+  output_stream: "image_tensor"
+}
+
+# Generates a single side packet containing a TensorFlow session from a saved
+# model. The directory path that contains the saved model is specified in the
+# saved_model_path option, and the name of the saved model file has to be
+# "saved_model.pb".
+node {
+  calculator: "TensorFlowSessionFromSavedModelCalculator"
+  output_side_packet: "SESSION:object_detection_session"
+  node_options: {
+    [type.googleapis.com/mediapipe.TensorFlowSessionFromSavedModelCalculatorOptions]: {
+      saved_model_path: "mediapipe/models/object_detection_saved_model"
+    }
+  }
+}
+
+# Runs a TensorFlow session (specified as an input side packet) that takes an
+# image tensor and outputs multiple tensors that describe the objects detected
+# in the image. The batch_size option is set to 1 to disable batching entirely.
+# Note that the particular TensorFlow model used in this session handles image
+# scaling internally before the object-detection inference, and therefore no
+# additional calculator for image transformation is needed in this MediaPipe
+# graph.
+node: {
+  calculator: "TensorFlowInferenceCalculator"
+  input_side_packet: "SESSION:object_detection_session"
+  input_stream: "INPUTS:image_tensor"
+  output_stream: "DETECTION_BOXES:detection_boxes_tensor"
+  output_stream: "DETECTION_CLASSES:detection_classes_tensor"
+  output_stream: "DETECTION_SCORES:detection_scores_tensor"
+  output_stream: "NUM_DETECTIONS:num_detections_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.TensorFlowInferenceCalculatorOptions]: {
+      batch_size: 1
+    }
+  }
+}
+
+# Decodes the detection tensors from the TensorFlow model into a vector of
+# detections. Each detection describes a detected object.
+node {
+  calculator: "ObjectDetectionTensorsToDetectionsCalculator"
+  input_stream: "BOXES:detection_boxes_tensor"
+  input_stream: "SCORES:detection_scores_tensor"
+  input_stream: "CLASSES:detection_classes_tensor"
+  input_stream: "NUM_DETECTIONS:num_detections_tensor"
+  output_stream: "DETECTIONS:detections"
+}
+
+# Performs non-max suppression to remove excessive detections.
+node {
+  calculator: "NonMaxSuppressionCalculator"
+  input_stream: "detections"
+  output_stream: "filtered_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
+      min_suppression_threshold: 0.4
+      min_score_threshold: 0.6
+      max_num_detections: 10
+      overlap_type: INTERSECTION_OVER_UNION
+    }
+  }
+}
+
+# Maps detection label IDs to the corresponding label text. The label map is
+# provided in the label_map_path option.
+node {
+  calculator: "DetectionLabelIdToTextCalculator"
+  input_stream: "filtered_detections"
+  output_stream: "output_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
+      label_map_path: "mediapipe/models/ssdlite_object_detection_labelmap.txt"
+    }
+  }
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTION_VECTOR:output_detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the original image coming into
+# the graph.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "INPUT_FRAME:input_video"
+  input_stream: "render_data"
+  output_stream: "OUTPUT_FRAME:output_video"
+}
+
+# Encodes the annotated images into a video file, adopting properties specified
+# in the input video header, e.g., video framerate.
+node {
+  calculator: "OpenCvVideoEncoderCalculator"
+  input_stream: "VIDEO:output_video"
+  input_stream: "VIDEO_PRESTREAM:input_video_header"
+  input_side_packet: "OUTPUT_FILE_PATH:output_video_path"
+  node_options: {
+    [type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: {
+      codec: "avc1"
+      video_format: "mp4"
+    }
+  }
+}
+```
+
+### TensorFlow Lite Model
+
+To build and run the TensorFlow Lite example on desktop, run:
+
+```bash
+$ bazel build -c opt --define 'MEDIAPIPE_DISABLE_GPU=1' \
+    mediapipe/examples/desktop/object_detection:object_detection_tflite
+
+# It should print:
+# Target //mediapipe/examples/desktop/object_detection:object_detection_tflite up-to-date:
+#   bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite
+# INFO: Elapsed time: 36.417s, Critical Path: 23.22s
+# INFO: 711 processes: 710 linux-sandbox, 1 local.
+# INFO: Build completed successfully, 734 total actions
+
+# Replace <input video path> and <output video path>.
+# You can find a test video in mediapipe/examples/desktop/object_detection.
+$ bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite \
+    --calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tflite_graph.pbtxt \
+    --input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
+```
+
+#### Graph
+
+![graph visualization](images/object_detection_desktop_tflite.png){width="400"}
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into
+[MediaPipe Visualizer](https://mediapipe-viz.appspot.com).
+
+```bash
+# MediaPipe graph that performs object detection on desktop with TensorFlow Lite
+# on CPU.
+# Used in the example in
+# mediapipie/examples/desktop/object_detection:object_detection_tflite.
+
+# max_queue_size limits the number of packets enqueued on any input stream
+# by throttling inputs to the graph. This makes the graph only process one
+# frame per time.
+max_queue_size: 1
+
+# Decodes an input video file into images and a video header.
+node {
+  calculator: "OpenCvVideoDecoderCalculator"
+  input_side_packet: "INPUT_FILE_PATH:input_video_path"
+  output_stream: "VIDEO:input_video"
+  output_stream: "VIDEO_PRESTREAM:input_video_header"
+}
+
+# Transforms the input image on CPU to a 320x320 image. To scale the image, by
+# default it uses the STRETCH scale mode that maps the entire input image to the
+# entire transformed image. As a result, image aspect ratio may be changed and
+# objects in the image may be deformed (stretched or squeezed), but the object
+# detection model used in this graph is agnostic to that deformation.
+node: {
+  calculator: "ImageTransformationCalculator"
+  input_stream: "IMAGE:input_video"
+  output_stream: "IMAGE:transformed_input_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
+      output_width: 320
+      output_height: 320
+    }
+  }
+}
+
+# Converts the transformed input image on CPU into an image tensor as a
+# TfLiteTensor. The zero_center option is set to true to normalize the
+# pixel values to [-1.f, 1.f] as opposed to [0.f, 1.f].
+node {
+  calculator: "TfLiteConverterCalculator"
+  input_stream: "IMAGE:transformed_input_video"
+  output_stream: "TENSORS:image_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
+      zero_center: true
+    }
+  }
+}
+
+# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
+# vector of tensors representing, for instance, detection boxes/keypoints and
+# scores.
+node {
+  calculator: "TfLiteInferenceCalculator"
+  input_stream: "TENSORS:image_tensor"
+  output_stream: "TENSORS:detection_tensors"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
+      model_path: "mediapipe/models/ssdlite_object_detection.tflite"
+    }
+  }
+}
+
+# Generates a single side packet containing a vector of SSD anchors based on
+# the specification in the options.
+node {
+  calculator: "SsdAnchorsCalculator"
+  output_side_packet: "anchors"
+  node_options: {
+    [type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
+      num_layers: 6
+      min_scale: 0.2
+      max_scale: 0.95
+      input_size_height: 320
+      input_size_width: 320
+      anchor_offset_x: 0.5
+      anchor_offset_y: 0.5
+      strides: 16
+      strides: 32
+      strides: 64
+      strides: 128
+      strides: 256
+      strides: 512
+      aspect_ratios: 1.0
+      aspect_ratios: 2.0
+      aspect_ratios: 0.5
+      aspect_ratios: 3.0
+      aspect_ratios: 0.3333
+      reduce_boxes_in_lowest_layer: true
+    }
+  }
+}
+
+# Decodes the detection tensors generated by the TensorFlow Lite model, based on
+# the SSD anchors and the specification in the options, into a vector of
+# detections. Each detection describes a detected object.
+node {
+  calculator: "TfLiteTensorsToDetectionsCalculator"
+  input_stream: "TENSORS:detection_tensors"
+  input_side_packet: "ANCHORS:anchors"
+  output_stream: "DETECTIONS:detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
+      num_classes: 91
+      num_boxes: 2034
+      num_coords: 4
+      ignore_classes: 0
+      apply_exponential_on_box_size: true
+
+      x_scale: 10.0
+      y_scale: 10.0
+      h_scale: 5.0
+      w_scale: 5.0
+    }
+  }
+}
+
+# Performs non-max suppression to remove excessive detections.
+node {
+  calculator: "NonMaxSuppressionCalculator"
+  input_stream: "detections"
+  output_stream: "filtered_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
+      min_suppression_threshold: 0.4
+      min_score_threshold: 0.6
+      max_num_detections: 5
+      overlap_type: INTERSECTION_OVER_UNION
+    }
+  }
+}
+
+# Maps detection label IDs to the corresponding label text. The label map is
+# provided in the label_map_path option.
+node {
+  calculator: "DetectionLabelIdToTextCalculator"
+  input_stream: "filtered_detections"
+  output_stream: "output_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
+      label_map_path: "mediapipe/models/ssdlite_object_detection_labelmap.txt"
+    }
+  }
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionsToRenderDataCalculator"
+  input_stream: "DETECTION_VECTOR:output_detections"
+  output_stream: "RENDER_DATA:render_data"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
+      thickness: 4.0
+      color { r: 255 g: 0 b: 0 }
+    }
+  }
+}
+
+# Draws annotations and overlays them on top of the original image coming into
+# the graph.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "INPUT_FRAME:input_video"
+  input_stream: "render_data"
+  output_stream: "OUTPUT_FRAME:output_video"
+}
+
+# Encodes the annotated images into a video file, adopting properties specified
+# in the input video header, e.g., video framerate.
+node {
+  calculator: "OpenCvVideoEncoderCalculator"
+  input_stream: "VIDEO:output_video"
+  input_stream: "VIDEO_PRESTREAM:input_video_header"
+  input_side_packet: "OUTPUT_FILE_PATH:output_video_path"
+  node_options: {
+    [type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: {
+      codec: "avc1"
+      video_format: "mp4"
+    }
+  }
+}
+```
+
+### Known issues with OpenCV 2
+
+Note that OpenCV 2 may not be able to render an mp4 file and returns the
+following error message:
+
+```
+[libx264 @ 0x7fe6eadf49a0] broken ffmpeg default settings detected
+[libx264 @ 0x7fe6eadf49a0] use an encoding preset (e.g. -vpre medium)
+[libx264 @ 0x7fe6eadf49a0] preset usage: -vpre <speed> -vpre <profile>
+[libx264 @ 0x7fe6eadf49a0] speed presets are listed in x264 --help
+[libx264 @ 0x7fe6eadf49a0] profile is optional; x264 defaults to high
+Could not open codec 'libx264': Unspecified errorE0612 19:40:09.067003  2089 simple_run_graph_main.cc:64] Fail to run the graph: CalculatorGraph::Run() failed in Run:
+Calculator::Process() for node "[OpenCvVideoEncoderCalculator, OpenCvVideoEncoderCalculator with node ID: 7 and input streams: <decorated_frames,video_prestream>]" failed: ; Fail to open file at ...
+```
+
+In that case, please change the OpenCvVideoEncoderCalculator option in either
+the [`TensorFlow graph`] or the [`TensorFlow Lite graph`] to the following and
+in the command line specify the output video to be a .mkv file.
+
+```bash
+node {
+  calculator: "OpenCvVideoEncoderCalculator"
+  input_stream: "VIDEO:output_video"
+  input_stream: "VIDEO_PRESTREAM:input_video_header"
+  input_side_packet: "OUTPUT_FILE_PATH:output_video_path"
+  node_options {
+    [type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: {
+      codec: "MPEG"
+     video_format: "mkv"
+  }
+}
+```
+
+[`README file`]:https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md
+[`TensorFlow graph`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_desktop_tensorflow_graph.pbtxt
+[`TensorFlow Lite graph`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_desktop_tflite_graph.pbtxt

mediapipe/docs/packets.md

@@ -0,0 +1,19 @@
+### Packets
+
+- [Creating a packet](#creating-a-packet)
+
+Each calculator is a node of of a graph. We describe how to create a new calculator, how to initialize a calculator, how to perform its calculations, input and output streams, timestamps, and options
+
+#### Creating a packet
+Packets are generally created with `MediaPipe::Adopt()` (from packet.h).
+
+```c++
+// Create some data.
+auto data = gtl::MakeUnique<MyDataClass>("constructor_argument");
+// Create a packet to own the data.
+Packet p = Adopt(data.release());
+// Make a new packet with the same data and a different timestamp.
+Packet p2 = p.At(Timestamp::PostStream());
+```
+
+Data within a packet is accessed with `Packet::Get<T>()`

mediapipe/docs/scheduling_sync.md

@@ -0,0 +1,164 @@
+# Framework Architecture
+
+## Scheduling mechanics
+
+Data processing in a MediaPipe graph occurs inside processing nodes defined as
+[`CalculatorBase`] subclasses. The scheduling system decides when each
+calculator should run.
+
+Each graph has at least one **scheduler queue**. Each scheduler queue has
+exactly one **executor**. Nodes are statically assigned to a queue (and
+therefore to an executor). By default there is one queue, whose executor is a
+thread pool with a number of threads based on the system’s capabilities.
+
+Each node has a scheduling state, which can be *not ready*, *ready*, or
+*running*. A readiness function determines whether a node is ready to run. This
+function is invoked at graph initialization, whenever a node finishes running,
+and whenever the state of a node’s inputs changes.
+
+The readiness function used depends on the type of node. A node with no stream
+inputs is known as a **source node**; source nodes are always ready to run,
+until they tell the framework they have no more data to output, at which point
+they are closed.
+
+Non-source nodes are ready if they have inputs to process, and if those inputs
+form a valid input set according to the conditions set by the node’s **input
+policy** (discussed below). Most nodes use the default input policy, but some
+nodes specify a different one.
+
+Note: Because changing the input policy changes the guarantees the calculator’s
+code can expect from its inputs, it is not generally possible to mix and match
+calculators with arbitrary input policies. Thus a calculator that uses a special
+input policy should be written for it, and declare it in its contract.
+
+When a node becomes ready, a task is added to the corresponding scheduler queue,
+which is a priority queue. The priority function is currently fixed, and takes
+into account static properties of the nodes and their topological sorting within
+the graph. For example, nodes closer to the output side of the graph have higher
+priority, while source nodes have the lowest priority.
+
+Each queue is served by an executor, which is responsible for actually running
+the task by invoking the calculator’s code. Different executors can be provided
+and configured; this can be used to customize the use of execution resources,
+e.g. by running certain nodes on lower-priority threads.
+
+## Timestamp Synchronization
+
+MediaPipe graph execution is decentralized: there is no global clock, and
+different nodes can process data from different timestamps at the same time.
+This allows higher throughput via pipelining.
+
+However, time information is very important for many perception workflows. Nodes
+that receive multiple input streams generally need to coordinate them in some
+way. For example, an object detector may output a list of boundary rectangles
+from a frame, and this information may be fed into a rendering node, which
+should process it together with the original frame.
+
+Therefore, one of the key responsibilities of the MediaPipe framework is to
+provide input synchronization for nodes. In terms of framework mechanics, the
+primary role of a timestamp is to serve as a **synchronization key**.
+
+Furthermore, MediaPipe is designed to support deterministic operations, which is
+important in many scenarios (testing, simulation, batch processing, etc.), while
+allowing graph authors to relax determinism where needed to meet real-time
+constraints.
+
+The two objectives of synchronization and determinism underlie several design
+choices. Notably, the packets pushed into a given stream must have monotonically
+increasing timestamps: this is not just a useful assumption for many nodes, but
+it is also relied upon by the synchronization logic. Each stream has a
+**timestamp bound**, which is the lowest possible timestamp allowed for a new
+packet on the stream. When a packet with timestamp `T` arrives, the bound
+automatically advances to `T+1`, reflecting the monotonic requirement. This
+allows the framework to know for certain that no more packets with timestamp
+lower than `T` will arrive.
+
+## Input policies
+
+Synchronization is handled locally on each node, using the input policy
+specified by the node.
+
+The default input policy, defined by [`DefaultInputStreamHandler`], provides
+deterministic synchronization of inputs, with the following guarantees:
+
+*   If packets with the same timestamp are provided on multiple input streams,
+    they will always be processed together regardless of their arrival order in
+    real time.
+
+*   Input sets are processed in strictly ascending timestamp order.
+
+*   No packets are dropped, and the processing is fully deterministic.
+
+*   The node becomes ready to process data as soon as possible given the
+    guarantees above.
+
+Note: An important consequence of this is that if the calculator always uses the
+current input timestamp when outputting packets, the output will inherently obey
+the monotonically increasing timestamp requirement.
+
+Warning: On the other hand, it is not guaranteed that an input packet will
+always be available for all streams.
+
+To explain how it works, we need to introduce the definition of a settled
+timestamp. We say that a timestamp in a stream is *settled* if it lower than the
+timestamp bound. In other words, a timestamp is settled for a stream once the
+state of the input at that timestamp is irrevocably known: either there is a
+packet, or there is the certainty that a packet with that timestamp will not
+arrive.
+
+Note: For this reason, MediaPipe also allows a stream producer to explicitly
+advance the timestamp bound farther that what the last packet implies, i.e. to
+provide a tighter bound. This can allow the downstream nodes to settle their
+inputs sooner.
+
+A timestamp is settled across multiple streams if it is settled on each of those
+streams. Furthermore, if a timestamp is settled it implies that all previous
+timestamps are also settled. Thus settled timestamps can be processed
+deterministically in ascending order.
+
+Given this definition, a calculator with the default input policy is ready if
+there is a timestamp which is settled across all input streams and contains a
+packet on at least one input stream. The input policy provides all available
+packets for a settled timestamp as a single *input set* to the calculator.
+
+One consequence of this deterministic behavior is that, for nodes with multiple
+input streams, there can be a theoretically unbounded wait for a timestamp to be
+settled, and an unbounded number of packets can be buffered in the meantime.
+(Consider a node with two input streams, one of which keeps sending packets
+while the other sends nothing and does not advance the bound.)
+
+Therefore, we also provide for custom input policies: for example, splitting the
+inputs in different synchronization sets defined by
+[`SyncSetInputStreamHandler`], or avoiding synchronization altogether and
+processing inputs immediately as they arrive defined by
+[`ImmediateInputStreamHandler`].
+
+## Flow control
+
+There are two main flow control mechanisms. A backpressure mechanism throttles
+the execution of upstream nodes when the packets buffered on a stream reach a
+(configurable) limit defined by [`CalculatorGraphConfig::max_queue_size`]. This
+mechanism maintains deterministic behavior, and includes a deadlock avoidance
+system that relaxes configured limits when needed.
+
+The second system consists of inserting special nodes which can drop packets
+according to real-time constraints (typically using custom input policies)
+defined by [`RealTimeFlowLimiterCalculator`]. For example, a common pattern
+places a flow-control node at the input of a subgraph, with a loopback
+connection from the final output to the flow-control node. The flow-control node
+is thus able to keep track of how many timestamps are being processed in the
+downstream graph, and drop packets if this count hits a (configurable) limit;
+and since packets are dropped upstream, we avoid the wasted work that would
+result from partially processing a timestamp and then dropping packets between
+intermediate stages.
+
+This calculator-based approach gives the graph author control of where packets
+can be dropped, and allows flexibility in adapting and customizing the graph’s
+behavior depending on resource constraints.
+
+[`CalculatorBase`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
+[`DefaultInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/default_input_stream_handler.h
+[`SyncSetInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/sync_set_input_stream_handler.h
+[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.h
+[`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`RealTimeFlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/real_time_flow_limiter_calculator.cc

mediapipe/docs/troubleshooting.md

@@ -0,0 +1,144 @@
+# Troubleshooting
+
+-   [Native method not found](#native-method-not-found)
+-   [No registered calculator found](#no-registered-calculator-found)
+-   [Out Of Memory error](#out-of-memory-error)
+-   [Graph hangs](#graph-hangs)
+-   [Calculator is scheduled infrequently](#calculator-is-scheduled-infrequently)
+-   [Output timing is uneven](#output-timing-is-uneven)
+-   [CalculatorGraph lags behind inputs](#calculatorgraph-lags-behind-inputs)
+
+## Native method not found
+
+The error message:
+
+```
+java.lang.UnsatisfiedLinkError: No implementation found for void com.google.wick.Wick.nativeWick
+```
+
+usually indicates that a needed native library, such as `/libwickjni.so` has not
+been loaded or has not been included in the dependencies of the app or cannot be
+found for some reason. Note that Java requires every native library to be
+explicitly loaded using the function `System.loadLibrary`.
+
+## No registered calculator found
+
+The error message:
+
+```
+No registered object with name: OurNewCalculator; Unable to find Calculator "OurNewCalculator"
+```
+
+usually indicates that `OurNewCalculator` is referenced by name in a
+[`CalculatorGraphConfig`] but that the library target for OurNewCalculator has
+not been linked to the application binary. When a new calculator is added to a
+calculator graph, that calculator must also be added as a build dependency of
+the applications using the calculator graph.
+
+This error is caught at runtime because calculator graphs reference their
+calculators by name through the field `CalculatorGraphConfig::Node:calculator`.
+When the library for a calculator is linked into an application binary, the
+calculator is automatically registered by name through the
+[`REGISTER_CALCULATOR`] macro using the [`registration.h`] library. Note that
+[`REGISTER_CALCULATOR`] can register a calculator with a namespace prefix,
+identical to its C++ namespace. In this case, the calcultor graph must also use
+the same namespace prefix.
+
+## Out Of Memory error
+
+Exhausting memory can be a symptom of too many packets accumulating inside a
+running MediaPipe graph. This can occur for a number of reasons, such as:
+
+1.  Some calculators in the graph simply can't keep pace with the arrival of
+    packets from a realtime input stream such as a video camera.
+2.  Some calculators are waiting for packets that will never arrive.
+
+For problem (1), it may be necessary to drop some old packets in older to
+process the more recent packets. For some hints, see:
+[How to process realtime input streams](how_to_questions.md#how-to-process-realtime-input-streams)
+
+For problem (2), it could be that one input stream is lacking packets for some
+reason. A device or a calculator may be misconfigured or may produce packets
+only sporadically. This can cause downstream calculators to wait for many
+packets that will never arrive, which in turn causes packets to accumulate on
+some of their input streams. MediaPipe addresses this sort of problem using
+"timestamp bounds". For some hints see:
+[How to process realtime input streams](how_to_questions.md#how-to-process-realtime-input-streams)
+
+The MediaPipe setting [`CalculatorGraphConfig::max_queue_size`] limits the
+number of packets enqueued on any input stream by throttling inputs to the
+graph. For realtime input streams, the number of packets queued at an input
+stream should almost always be zero or one. If this is not the case, you may see
+the following warning message:
+
+```
+Resolved a deadlock by increasing max_queue_size of input stream
+```
+
+Also, the setting [`CalculatorGraphConfig::report_deadlock`] can be set to cause
+graph run to fail and surface the deadlock as an error, such that max_queue_size
+to acts as a memory usage limit.
+
+## Graph hangs
+
+Many applications will call [`CalculatorGraph::CloseAllPacketSources`] and
+[`CalculatorGraph::WaitUntilDone`] to finish or suspend execution of a MediaPipe
+graph. The objective here is to allow any pending calculators or packets to
+complete processing, and then to shutdown the graph. If all goes well, every
+stream in the graph will reach [`Timestamp::Done`], and every calculator will
+reach [`CalculatorBase::Close`], and then [`CalculatorGraph::WaitUntilDone`]
+will complete successfully.
+
+If some calculators or streams cannot reach state [`Timestamp::Done`] or
+[`CalculatorBase::Close`], then the method [`CalculatorGraph::Cancel`] can be
+called to terminate the graph run without waiting for all pending calculators
+and packets to complete.
+
+## Output timing is uneven
+
+Some realtime MediaPipe graphs produce a series of video frames for viewing as a
+video effect or as a video diagnostic. Sometimes, a MediaPipe graph will produce
+these frames in clusters, for example when several output frames are
+extrapolated from the same cluster of input frames. If the outputs are presented
+as they are produced, some output frames are immediately replaced by later
+frames in the same cluster, which makes the results hard to see and evaluate
+visually. In cases like this, the output visualization can be improved by
+presenting the frames at even intervals in real time.
+
+MediaPipe addresses this use case by mapping timestamps to points in real time.
+Each timestamp indicates a time in microseconds, and a calculator such as
+`LiveClockSyncCalculator` can delay the output of packets to match their
+timestamps. This sort of calculator adjusts the timing of outputs such that:
+
+1.  The time between outputs corresponds to the time between timestamps as
+    closely as possible.
+2.  Outputs are produced with the smallest delay possible.
+
+## CalculatorGraph lags behind inputs
+
+For many realtime MediaPipe graphs, low latency is an objective. MediaPipe
+supports "pipelined" style parallel processing in order to begin processing of
+each packet as early as possible. Normally the lowest possible latency is the
+total time required by each calculator along a "critical path" of successive
+calculators. The latency of the a MediaPipe graph could be worse than the ideal
+due to delays introduced to display frames a even intervals as described in
+[Output timing is uneven](troubleshooting.md?cl=252235797#output-timing-is-uneven).
+
+If some of the calculators in the graph cannot keep pace with the realtime input
+streams, then latency will continue to increase, and it becomes necessary to
+drop some input packets. The recommended technique is to use the MediaPipe
+calculators designed specifically for this purpose such as
+[`RealTimeFlowLimiterCalculator`] as described in
+[How to process realtime input streams](how_to_questions.md#how-to-process-realtime-input-streams).
+
+[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`CalculatorGraphConfig::report_deadlock`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
+[`REGISTER_CALCULATOR`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_registry.h
+[`registration.h`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/deps/registration.h
+[`CalculatorGraph::CloseAllPacketSources`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
+[`CalculatorGraph::Cancel`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
+[`CalculatorGraph::WaitUntilDone`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
+[`Timestamp::Done`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/timestamp.h
+[`CalculatorBase::Close`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
+[`RealTimeFlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/real_time_flow_limiter_calculator.cc

mediapipe/docs/visualizer.md

@@ -0,0 +1,61 @@
+## Visualizing MediaPipe Graphs
+
+-   [Working within the editor](#working-within-the-editor)
+-   [Understanding the Graph](#understanding-the-graph)
+
+To help users understand the structure of their calculator graphs and to
+understand the overall behavior of their machine learning inference pipelines,
+we have built the [MediaPipe Visualizer](https://mediapipe-viz.appspot.com/) that is available online.
+
+*   A graph view allows users to see a connected calculator graph as expressed
+    through a graph configuration that is pasted into the graph editor or
+    uploaded. The user can visualize and troubleshoot a graph they have created.
+
+    ![Startup screen](./images/startup_screen.png){width="800"}
+
+### Working within the editor
+
+Getting Started:
+
+The graph can be modified by adding and editing code in the Editor view.
+
+![Editor UI](./images/editor_view.png){width="600"}
+
+*   Pressing the "New" button in the upper right corner will clear any existing
+    code in the Editor window.
+
+    ![New Button](./images/upload_button.png){width="300"}
+
+*   Pressing the "Upload" button will prompt the user to select a local PBTXT
+    file, which will everwrite the current code within the editor.
+
+*   Alternatively, code can be pasted directly into the editor window.
+
+*   Errors and informational messages will appear in the Feedback window.
+
+    ![Error Msg](./images/console_error.png){width="400"}
+
+### Understanding the Graph
+
+The visualizer graph shows the connections between calculator nodes.
+
+*   Streams exit from the bottom of the calculator producing the stream and
+    enter the top of any calculator receiving the stream. (Notice the use of the
+    key, "input_stream" and "output_stream").
+
+    ![Stream UI](./images/stream_ui.png){width="350"}
+    ![Stream_code](./images/stream_code.png){width="350"}
+
+*   Sidepackets work the same, except that they exit a node on the right and
+    enter on the left. (Notice the use of the key, "input_side_packet" and
+    "output_side_packet").
+
+    ![Sidepacket UI](./images/side_packet.png){width="350"}
+    ![Sidepacket_code](./images/side_packet_code.png){width="350"}
+
+*   There are special nodes that represent inputs and outputs to the graph and
+    can supply either side packets or streams.
+
+    ![Special nodes](./images/special_nodes.png){width="350"}
+    ![Special nodes](./images/special_nodes_code.png){width="350"}
+