talksik/mediapipe
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
GitOrigin-RevId: f72a0f86c2c2acdb1920973c718a9e26ed3ec4b6
This commit is contained in:
MediaPipe Team
2020-06-08 12:08:33 -04:00
committed by chuoling
parent 5d028d923b
commit cd2b69d58c
315 changed files with 8162 additions and 7502 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Makefile
+21
View File
@@ -0,0 +1,21 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
rm -rf ./_build
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
docs/_config.yml
+29
View File
@@ -0,0 +1,29 @@
# Configuration for GitHub Pages
remote_theme: pmarsceill/just-the-docs
# Set a path/url to a logo that will be displayed instead of the title
logo: "images/logo_horizontal_color.png"
# Enable or disable the site search
search_enabled: true
# Set the search token separator for hyphenated-word search:
search_tokenizer_separator: /[\s/]+/
# Enable or disable heading anchors
heading_anchors: true
# Aux links for the upper right navigation
aux_links:
"MediaPipe on GitHub":
- "//github.com/google/mediapipe"
# Footer content appears at the bottom of every page's main content
footer_content: "&copy; 2020 GOOGLE LLC | <a href=\"https://policies.google.com/privacy\">PRIVACY POLICY</a> | <a href=\"https://policies.google.com/terms\">TERMS OF SERVICE</a>"
# Color scheme currently only supports "dark" or nil (default)
color_scheme: nil
# Google Analytics Tracking (optional)
ga_tracking: UA-140696581-2
docs/conf.py
+57
View File
@@ -0,0 +1,57 @@
"""Configuration file for the Sphinx documentation builder.
This file only contains a selection of the most common options.
For a full list see the documentation:
http://www.sphinx-doc.org/en/master/config
-- Path setup --------------------------------------------------------------
If extensions (or modules to document with autodoc) are in another directory,
add these directories to sys.path here.
If the directory is relative to the documentation root,
use os.path.abspath to make it absolute, like shown here.
"""
import sphinx_rtd_theme
# -- Project information -----------------------------------------------------
project = 'MediaPipe'
author = 'Google LLC'
# The full version, including alpha/beta/rc tags
release = 'v0.5'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'recommonmark'
]
master_doc = 'index'
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
docs/data/visualizer/sample_trace.binarypb
BIN
View File
Binary file not shown.
docs/examples.md
+96
View File
@@ -0,0 +1,96 @@
---
nav_exclude: true
---
# Examples
Below are code samples on how to run MediaPipe on both mobile and desktop. We
currently support MediaPipe APIs on mobile for Android only but will add support
for Objective-C shortly.
## Mobile
### [Hello World! on Android](./getting_started/hello_world_android.md)
This should be the first mobile Android example users go through in detail. It
teaches the following:
* Introduction of a simple MediaPipe graph running on mobile GPUs for
[Sobel edge detection](https://en.wikipedia.org/wiki/Sobel_operator).
* Building a simple baseline Android application that displays "Hello World!".
* Adding camera preview support into the baseline application using the
Android [CameraX] API.
* Incorporating the Sobel edge detection graph to process the live camera
preview and display the processed video in real-time.
[Sobel edge detection]:https://en.wikipedia.org/wiki/Sobel_operator
[CameraX]:https://developer.android.com/training/camerax
### [Hello World! on iOS](./getting_started/hello_world_ios.md)
This is the iOS version of Sobel edge detection example.
### [Face Detection](./solutions/face_detection.md)
### [Face Mesh](./solutions/face_mesh.md)
### [Hand](./solutions/hand.md)
### [Hair Segmentation](./solutions/hair_segmentation.md)
### [Object Detection](./solutions/object_detection.md)
### [Box Tracking](./solutions/box_tracking.md)
### [Objectron: 3D Object Detection](./solutions/objectron.md)
### [KNIFT: Template-based Feature Matching](./solutions/knift.md)
## Desktop
### [Hello World for C++](./getting_started/hello_world_desktop.md)
This shows how to run a simple graph using the MediaPipe C++ APIs.
### [Face Detection](./solutions/face_detection.md)
### [Face Mesh](./solutions/face_mesh.md)
### [Hand](./solutions/hand.md)
### [Hair Segmentation](./solutions/hair_segmentation.md)
### [Object Detection](./solutions/object_detection.md)
### [Box Tracking](./solutions/box_tracking.md)
### [AutoFlip - Semantic-aware Video Cropping](./solutions/autoflip.md)
### [Preparing Data Sets with MediaSequence](./solutions/media_sequence.md)
This shows how to use MediaPipe for media processing to prepare video data sets
for training a TensorFlow model.
### [Feature Extraction and Model Inference for YouTube-8M Challenge](./solutions/youtube_8m.md)
This shows how to use MediaPipe to prepare training data for the YouTube-8M
Challenge and do the model inference with the baseline model.
## Google Coral (ML acceleration with Google EdgeTPU)
### [Face Detection](./solutions/face_detection.md)
### [Object Detection](./solutions/object_detection.md)
## Web Browser
See more details [here](./getting_started/web.md) and
[Google Developer blog post](https://mediapipe.page.link/webdevblog).
### [Face Detection in Browser](https://viz.mediapipe.dev/demo/face_detection)
### [Hand Detection in Browser](https://viz.mediapipe.dev/demo/hand_detection)
### [Hand Tracking in Browser](https://viz.mediapipe.dev/demo/hand_tracking)
### [Hair Segmentation in Browser](https://viz.mediapipe.dev/demo/hair_segmentation)
docs/framework_concepts/calculators.md
+412
View File
@@ -0,0 +1,412 @@
---
layout: default
title: Calculators
parent: Framework Concepts
nav_order: 1
---
# Calculators
{: .no_toc }
1. TOC
{:toc}
---
Each calculator is a node of a graph. We describe how to create a new
calculator, how to initialize a calculator, how to perform its calculations,
input and output streams, timestamps, and options. Each node in the graph is
implemented as a `Calculator`. The bulk of graph execution happens inside its
calculators. A calculator may receive zero or more input streams and/or side
packets and produces zero or more output streams and/or side packets.
## CalculatorBase
A calculator is created by defining a new sub-class of the
[`CalculatorBase`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.cc)
class, implementing a number of methods, and registering the new sub-class with
Mediapipe. At a minimum, a new calculator must implement the below four methods
* `GetContract()`
* Calculator authors can specify the expected types of inputs and outputs
of a calculator in GetContract(). When a graph is initialized, the
framework calls a static method to verify if the packet types of the
connected inputs and outputs match the information in this
specification.
* `Open()`
* After a graph starts, the framework calls `Open()`. The input side
packets are available to the calculator at this point. `Open()`
interprets the node configuration operations (see [Graphs](graphs.md))
and prepares the calculator's per-graph-run state. This function may
also write packets to calculator outputs. An error during `Open()` can
terminate the graph run.
* `Process()`
* For a calculator with inputs, the framework calls `Process()` repeatedly
whenever at least one input stream has a packet available. The framework
by default guarantees that all inputs have the same timestamp (see
[Synchronization](synchronization.md) for more information). Multiple
`Process()` calls can be invoked simultaneously when parallel execution
is enabled. If an error occurs during `Process()`, the framework calls
`Close()` and the graph run terminates.
* `Close()`
* After all calls to `Process()` finish or when all input streams close,
the framework calls `Close()`. This function is always called if
`Open()` was called and succeeded and even if the graph run terminated
because of an error. No inputs are available via any input streams
during `Close()`, but it still has access to input side packets and
therefore may write outputs. After `Close()` returns, the calculator
should be considered a dead node. The calculator object is destroyed as
soon as the graph finishes running.
The following are code snippets from
[CalculatorBase.h](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h).
```c++
class CalculatorBase {
public:
...
// The subclasses of CalculatorBase must implement GetContract.
// ...
static ::MediaPipe::Status GetContract(CalculatorContract* cc);
// Open is called before any Process() calls, on a freshly constructed
// calculator. Subclasses may override this method to perform necessary
// setup, and possibly output Packets and/or set output streams' headers.
// ...
virtual ::MediaPipe::Status Open(CalculatorContext* cc) {
return ::MediaPipe::OkStatus();
}
// Processes the incoming inputs. May call the methods on cc to access
// inputs and produce outputs.
// ...
virtual ::MediaPipe::Status Process(CalculatorContext* cc) = 0;
// Is called if Open() was called and succeeded. Is called either
// immediately after processing is complete or after a graph run has ended
// (if an error occurred in the graph). ...
virtual ::MediaPipe::Status Close(CalculatorContext* cc) {
return ::MediaPipe::OkStatus();
}
...
};
```
## Life of a calculator
During initialization of a MediaPipe graph, the framework calls a
`GetContract()` static method to determine what kinds of packets are expected.
The framework constructs and destroys the entire calculator for each graph run
(e.g. once per video or once per image). Expensive or large objects that remain
constant across graph runs should be supplied as input side packets so the
calculations are not repeated on subsequent runs.
After initialization, for each run of the graph, the following sequence occurs:
* `Open()`
* `Process()` (repeatedly)
* `Close()`
The framework calls `Open()` to initialize the calculator. `Open()` should
interpret any options and set up the calculator's per-graph-run state. `Open()`
may obtain input side packets and write packets to calculator outputs. If
appropriate, it should call `SetOffset()` to reduce potential packet buffering
of input streams.
If an error occurs during `Open()` or `Process()` (as indicated by one of them
returning a non-`Ok` status), the graph run is terminated with no further calls
to the calculator's methods, and the calculator is destroyed.
For a calculator with inputs, the framework calls `Process()` whenever at least
one input has a packet available. The framework guarantees that inputs all have
the same timestamp, that timestamps increase with each call to `Process()` and
that all packets are delivered. As a consequence, some inputs may not have any
packets when `Process()` is called. An input whose packet is missing appears to
produce an empty packet (with no timestamp).
The framework calls `Close()` after all calls to `Process()`. All inputs will
have been exhausted, but `Close()` has access to input side packets and may
write outputs. After Close returns, the calculator is destroyed.
Calculators with no inputs are referred to as sources. A source calculator
continues to have `Process()` called as long as it returns an `Ok` status. A
source calculator indicates that it is exhausted by returning a stop status
(i.e. MediaPipe::tool::StatusStop).
## Identifying inputs and outputs
The public interface to a calculator consists of a set of input streams and
output streams. In a CalculatorGraphConfiguration, the outputs from some
calculators are connected to the inputs of other calculators using named
streams. Stream names are normally lowercase, while input and output tags are
normally UPPERCASE. In the example below, the output with tag name `VIDEO` is
connected to the input with tag name `VIDEO_IN` using the stream named
`video_stream`.
```proto
# Graph describing calculator SomeAudioVideoCalculator
node {
calculator: "SomeAudioVideoCalculator"
input_stream: "INPUT:combined_input"
output_stream: "VIDEO:video_stream"
}
node {
calculator: "SomeVideoCalculator"
input_stream: "VIDEO_IN:video_stream"
output_stream: "VIDEO_OUT:processed_video"
}
```
Input and output streams can be identified by index number, by tag name, or by a
combination of tag name and index number. You can see some examples of input and
output identifiers in the example below. `SomeAudioVideoCalculator` identifies
its video output by tag and its audio outputs by the combination of tag and
index. The input with tag `VIDEO` is connected to the stream named
`video_stream`. The outputs with tag `AUDIO` and indices `0` and `1` are
connected to the streams named `audio_left` and `audio_right`.
`SomeAudioCalculator` identifies its audio inputs by index only (no tag needed).
```proto
# Graph describing calculator SomeAudioVideoCalculator
node {
calculator: "SomeAudioVideoCalculator"
input_stream: "combined_input"
output_stream: "VIDEO:video_stream"
output_stream: "AUDIO:0:audio_left"
output_stream: "AUDIO:1:audio_right"
}
node {
calculator: "SomeAudioCalculator"
input_stream: "audio_left"
input_stream: "audio_right"
output_stream: "audio_energy"
}
```
In the calculator implementation, inputs and outputs are also identified by tag
name and index number. In the function below input are output are identified:
* By index number: The combined input stream is identified simply by index
`0`.
* By tag name: The video output stream is identified by tag name "VIDEO".
* By tag name and index number: The output audio streams are identified by the
combination of the tag name `AUDIO` and the index numbers `0` and `1`.
```c++
// c++ Code snippet describing the SomeAudioVideoCalculator GetContract() method
class SomeAudioVideoCalculator : public CalculatorBase {
public:
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
cc->Inputs().Index(0).SetAny();
// SetAny() is used to specify that whatever the type of the
// stream is, it's acceptable. This does not mean that any
// packet is acceptable. Packets in the stream still have a
// particular type. SetAny() has the same effect as explicitly
// setting the type to be the stream's type.
cc->Outputs().Tag("VIDEO").Set<ImageFrame>();
cc->Outputs().Get("AUDIO", 0).Set<Matrix>;
cc->Outputs().Get("AUDIO", 1).Set<Matrix>;
return ::mediapipe::OkStatus();
}
```
## Processing
`Process()` called on a non-source node must return `::mediapipe::OkStatus()` to
indicate that all went well, or any other status code to signal an error
If a non-source calculator returns `tool::StatusStop()`, then this signals the
graph is being cancelled early. In this case, all source calculators and graph
input streams will be closed (and remaining Packets will propagate through the
graph).
A source node in a graph will continue to have `Process()` called on it as long
as it returns `::mediapipe::OkStatus(`). To indicate that there is no more data
to be generated return `tool::StatusStop()`. Any other status indicates an error
has occurred.
`Close()` returns `::mediapipe::OkStatus()` to indicate success. Any other
status indicates a failure.
Here is the basic `Process()` function. It uses the `Input()` method (which can
be used only if the calculator has a single input) to request its input data. It
then uses `std::unique_ptr` to allocate the memory needed for the output packet,
and does the calculations. When done it releases the pointer when adding it to
the output stream.
```c++
::util::Status MyCalculator::Process() {
const Matrix& input = Input()->Get<Matrix>();
std::unique_ptr<Matrix> output(new Matrix(input.rows(), input.cols()));
// do your magic here....
// output->row(n) = ...
Output()->Add(output.release(), InputTimestamp());
return ::mediapipe::OkStatus();
}
```
## Example calculator
This section discusses the implementation of `PacketClonerCalculator`, which
does a relatively simple job, and is used in many calculator graphs.
`PacketClonerCalculator` simply produces a copy of its most recent input
packets on demand.
`PacketClonerCalculator` is useful when the timestamps of arriving data packets
are not aligned perfectly. Suppose we have a room with a microphone, light
sensor and a video camera that is collecting sensory data. Each of the sensors
operates independently and collects data intermittently. Suppose that the output
of each sensor is:
* microphone = loudness in decibels of sound in the room (Integer)
* light sensor = brightness of room (Integer)
* video camera = RGB image frame of room (ImageFrame)
Our simple perception pipeline is designed to process sensory data from these 3
sensors such that at any time when we have image frame data from the camera that
is synchronized with the last collected microphone loudness data and light
sensor brightness data. To do this with MediaPipe, our perception pipeline has 3
input streams:
* room_mic_signal - Each packet of data in this input stream is integer data
representing how loud audio is in a room with timestamp.
* room_lightening_sensor - Each packet of data in this input stream is integer
data representing how bright is the room illuminated with timestamp.
* room_video_tick_signal - Each packet of data in this input stream is
imageframe of video data representing video collected from camera in the
room with timestamp.
Below is the implementation of the `PacketClonerCalculator`. You can see
the `GetContract()`, `Open()`, and `Process()` methods as well as the instance
variable `current_` which holds the most recent input packets.
```c++
// This takes packets from N+1 streams, A_1, A_2, ..., A_N, B.
// For every packet that appears in B, outputs the most recent packet from each
// of the A_i on a separate stream.
#include <vector>
#include "absl/strings/str_cat.h"
#include "mediapipe/framework/calculator_framework.h"
namespace mediapipe {
// For every packet received on the last stream, output the latest packet
// obtained on all other streams. Therefore, if the last stream outputs at a
// higher rate than the others, this effectively clones the packets from the
// other streams to match the last.
//
// Example config:
// node {
// calculator: "PacketClonerCalculator"
// input_stream: "first_base_signal"
// input_stream: "second_base_signal"
// input_stream: "tick_signal"
// output_stream: "cloned_first_base_signal"
// output_stream: "cloned_second_base_signal"
// }
//
class PacketClonerCalculator : public CalculatorBase {
public:
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
const int tick_signal_index = cc->Inputs().NumEntries() - 1;
// cc->Inputs().NumEntries() returns the number of input streams
// for the PacketClonerCalculator
for (int i = 0; i < tick_signal_index; ++i) {
cc->Inputs().Index(i).SetAny();
// cc->Inputs().Index(i) returns the input stream pointer by index
cc->Outputs().Index(i).SetSameAs(&cc->Inputs().Index(i));
}
cc->Inputs().Index(tick_signal_index).SetAny();
return ::mediapipe::OkStatus();
}
::mediapipe::Status Open(CalculatorContext* cc) final {
tick_signal_index_ = cc->Inputs().NumEntries() - 1;
current_.resize(tick_signal_index_);
// Pass along the header for each stream if present.
for (int i = 0; i < tick_signal_index_; ++i) {
if (!cc->Inputs().Index(i).Header().IsEmpty()) {
cc->Outputs().Index(i).SetHeader(cc->Inputs().Index(i).Header());
// Sets the output stream of index i header to be the same as
// the header for the input stream of index i
}
}
return ::mediapipe::OkStatus();
}
::mediapipe::Status Process(CalculatorContext* cc) final {
// Store input signals.
for (int i = 0; i < tick_signal_index_; ++i) {
if (!cc->Inputs().Index(i).Value().IsEmpty()) {
current_[i] = cc->Inputs().Index(i).Value();
}
}
// Output if the tick signal is non-empty.
if (!cc->Inputs().Index(tick_signal_index_).Value().IsEmpty()) {
for (int i = 0; i < tick_signal_index_; ++i) {
if (!current_[i].IsEmpty()) {
cc->Outputs().Index(i).AddPacket(
current_[i].At(cc->InputTimestamp()));
// Add a packet to output stream of index i a packet from inputstream i
// with timestamp common to all present inputs
//
} else {
cc->Outputs().Index(i).SetNextTimestampBound(
cc->InputTimestamp().NextAllowedInStream());
// if current_[i], 1 packet buffer for input stream i is empty, we will set
// next allowed timestamp for input stream i to be current timestamp + 1
}
}
}
return ::mediapipe::OkStatus();
}
private:
std::vector<Packet> current_;
int tick_signal_index_;
};
REGISTER_CALCULATOR(PacketClonerCalculator);
} // namespace mediapipe
```
Typically, a calculator has only a .cc file. No .h is required, because
mediapipe uses registration to make calculators known to it. After you have
defined your calculator class, register it with a macro invocation
REGISTER_CALCULATOR(calculator_class_name).
Below is a trivial MediaPipe graph that has 3 input streams, 1 node
(PacketClonerCalculator) and 3 output streams.
```proto
input_stream: "room_mic_signal"
input_stream: "room_lighting_sensor"
input_stream: "room_video_tick_signal"
node {
calculator: "PacketClonerCalculator"
input_stream: "room_mic_signal"
input_stream: "room_lighting_sensor"
input_stream: "room_video_tick_signal"
output_stream: "cloned_room_mic_signal"
output_stream: "cloned_lighting_sensor"
}
```
The diagram below shows how the `PacketClonerCalculator` defines its output
packets based on its series of input packets.
| ![Graph using |
: PacketClonerCalculator](../images/packet_cloner_calculator.png) :
| :--------------------------------------------------------------------------: |
| *Each time it receives a packet on its TICK input stream, the |
: PacketClonerCalculator outputs the most recent packet from each of its input :
: streams. The sequence of output packets is determined by the sequene of :
: input packets and their timestamps. The timestamps are shows along the right :
: side of the diagram.* :
docs/framework_concepts/framework_concepts.md
+112
View File
@@ -0,0 +1,112 @@
---
layout: default
title: Framework Concepts
nav_order: 5
has_children: true
has_toc: false
---
# Framework Concepts
{: .no_toc }
1. TOC
{:toc}
---
## The basics
### Packet
The basic data flow unit. A packet consists of a numeric timestamp and a shared
pointer to an **immutable** payload. The payload can be of any C++ type, and the
payload's type is also referred to as the type of the packet. Packets are value
classes and can be copied cheaply. Each copy shares ownership of the payload,
with reference-counting semantics. Each copy has its own timestamp. See also
[Packet](packets.md).
### Graph
MediaPipe processing takes place inside a graph, which defines packet flow paths
between **nodes**. A graph can have any number of inputs and outputs, and data
flow can branch and merge. Generally data flows forward, but backward loops are
possible. See [Graphs](graphs.md) for details.
### Nodes
Nodes produce and/or consume packets, and they are where the bulk of the graph’s
work takes place. They are also known as “calculators”, for historical reasons.
Each node’s interface defines a number of input and output **ports**, identified
by a tag and/or an index. See [Calculators](calculators.md) for details.
### Streams
A stream is a connection between two nodes that carries a sequence of packets,
whose timestamps must be monotonically increasing.
### Side packets
A side packet connection between nodes carries a single packet (with unspecified
timestamp). It can be used to provide some data that will remain constant,
whereas a stream represents a flow of data that changes over time.
### Packet Ports
A port has an associated type; packets transiting through the port must be of
that type. An output stream port can be connected to any number of input stream
ports of the same type; each consumer receives a separate copy of the output
packets, and has its own queue, so it can consume them at its own pace.
Similarly, a side packet output port can be connected to as many side packet
input ports as desired.
A port can be required, meaning that a connection must be made for the graph to
be valid, or optional, meaning it may remain unconnected.
Note: even if a stream connection is required, the stream may not carry a packet
for all timestamps.
## Input and output
Data flow can originate from **source nodes**, which have no input streams and
produce packets spontaneously (e.g. by reading from a file); or from **graph
input streams**, which let an application feed packets into a graph.
Similarly, there are **sink nodes** that receive data and write it to various
destinations (e.g. a file, a memory buffer, etc.), and an application can also
receive output from the graph using **callbacks**.
## Runtime behavior
### Graph lifetime
Once a graph has been initialized, it can be **started** to begin processing
data, and can process a stream of packets until each stream is closed or the
graph is **canceled**. Then the graph can be destroyed or **started** again.
### Node lifetime
There are three main lifetime methods the framework will call on a node:
- Open: called once, before the other methods. When it is called, all input
side packets required by the node will be available.
- Process: called multiple times, when a new set of inputs is available,
according to the node’s input policy.
- Close: called once, at the end.
In addition, each calculator can define constructor and destructor, which are
useful for creating and deallocating resources that are independent of the
processed data.
### Input policies
The default input policy is deterministic collation of packets by timestamp. A
node receives all inputs for the same timestamp at the same time, in an
invocation of its Process method; and successive input sets are received in
their timestamp order. This can require delaying the processing of some packets
until a packet with the same timestamp is received on all input streams, or
until it can be guaranteed that a packet with that timestamp will not be
arriving on the streams that have not received it.
Other policies are also available, implemented using a separate kind of
component known as an InputStreamHandler.
See [Synchronization](synchronization.md) for more details.
docs/framework_concepts/gpu.md
+163
View File
@@ -0,0 +1,163 @@
---
layout: default
title: GPU
parent: Framework Concepts
nav_order: 5
---
# GPU
{: .no_toc }
1. TOC
{:toc}
---
## Overview
MediaPipe supports calculator nodes for GPU compute and rendering, and allows combining multiple GPU nodes, as well as mixing them with CPU based calculator nodes. There exist several GPU APIs on mobile platforms (eg, OpenGL ES, Metal and Vulkan). MediaPipe does not attempt to offer a single cross-API GPU abstraction. Individual nodes can be written using different APIs, allowing them to take advantage of platform specific features when needed.
GPU support is essential for good performance on mobile platforms, especially for real-time video. MediaPipe enables developers to write GPU compatible calculators that support the use of GPU for:
* On-device real-time processing, not just batch processing
* Video rendering and effects, not just analysis
Below are the design principles for GPU support in MediaPipe
* GPU-based calculators should be able to occur anywhere in the graph, and not necessarily be used for on-screen rendering.
* Transfer of frame data from one GPU-based calculator to another should be fast, and not incur expensive copy operations.
* Transfer of frame data between CPU and GPU should be as efficient as the platform allows.
* Because different platforms may require different techniques for best performance, the API should allow flexibility in the way things are implemented behind the scenes.
* A calculator should be allowed maximum flexibility in using the GPU for all or part of its operation, combining it with the CPU if necessary.
## OpenGL ES Support
MediaPipe supports OpenGL ES up to version 3.2 on Android/Linux and up to ES 3.0
on iOS. In addition, MediaPipe also supports Metal on iOS.
OpenGL ES 3.1 or greater is required (on Android/Linux systems) for running
machine learning inference calculators and graphs.
MediaPipe allows graphs to run OpenGL in multiple GL contexts. For example, this
can be very useful in graphs that combine a slower GPU inference path (eg, at 10
FPS) with a faster GPU rendering path (eg, at 30 FPS): since one GL context
corresponds to one sequential command queue, using the same context for both
tasks would reduce the rendering frame rate.
One challenge MediaPipe's use of multiple contexts solves is the ability to
communicate across them. An example scenario is one with an input video that is
sent to both the rendering and inferences paths, and rendering needs to have
access to the latest output from inference.
An OpenGL context cannot be accessed by multiple threads at the same time.
Furthermore, switching the active GL context on the same thread can be slow on
some Android devices. Therefore, our approach is to have one dedicated thread
per context. Each thread issues GL commands, building up a serial command queue
on its context, which is then executed by the GPU asynchronously.
## Life of a GPU Calculator
This section presents the basic structure of the Process method of a GPU
calculator derived from base class GlSimpleCalculator. The GPU calculator
`LuminanceCalculator` is shown as an example. The method
`LuminanceCalculator::GlRender` is called from `GlSimpleCalculator::Process`.
```c++
// Converts RGB images into luminance images, still stored in RGB format.
// See GlSimpleCalculator for inputs, outputs and input side packets.
class LuminanceCalculator : public GlSimpleCalculator {
public:
::mediapipe::Status GlSetup() override;
::mediapipe::Status GlRender(const GlTexture& src,
const GlTexture& dst) override;
::mediapipe::Status GlTeardown() override;
private:
GLuint program_ = 0;
GLint frame_;
};
REGISTER_CALCULATOR(LuminanceCalculator);
::mediapipe::Status LuminanceCalculator::GlRender(const GlTexture& src,
const GlTexture& dst) {
static const GLfloat square_vertices[] = {
-1.0f, -1.0f, // bottom left
1.0f, -1.0f, // bottom right
-1.0f, 1.0f, // top left
1.0f, 1.0f, // top right
};
static const GLfloat texture_vertices[] = {
0.0f, 0.0f, // bottom left
1.0f, 0.0f, // bottom right
0.0f, 1.0f, // top left
1.0f, 1.0f, // top right
};
// program
glUseProgram(program_);
glUniform1i(frame_, 1);
// vertex storage
GLuint vbo[2];
glGenBuffers(2, vbo);
GLuint vao;
glGenVertexArrays(1, &vao);
glBindVertexArray(vao);
// vbo 0
glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
GL_STATIC_DRAW);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
// vbo 1
glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
GL_STATIC_DRAW);
glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
// draw
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
// cleanup
glDisableVertexAttribArray(ATTRIB_VERTEX);
glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
glBindBuffer(GL_ARRAY_BUFFER, 0);
glBindVertexArray(0);
glDeleteVertexArrays(1, &vao);
glDeleteBuffers(2, vbo);
return ::mediapipe::OkStatus();
}
```
The design principles mentioned above have resulted in the following design
choices for MediaPipe GPU support:
* We have a GPU data type, called `GpuBuffer`, for representing image data, optimized for GPU usage. The exact contents of this data type are opaque and platform-specific.
* A low-level API based on composition, where any calculator that wants to make use of the GPU creates and owns an instance of the `GlCalculatorHelper` class. This class offers a platform-agnostic API for managing the OpenGL context, setting up textures for inputs and outputs, etc.
* A high-level API based on subclassing, where simple calculators implementing image filters subclass from `GlSimpleCalculator` and only need to override a couple of virtual methods with their specific OpenGL code, while the superclass takes care of all the plumbing.
* Data that needs to be shared between all GPU-based calculators is provided as a external input that is implemented as a graph service and is managed by the `GlCalculatorHelper` class.
* The combination of calculator-specific helpers and a shared graph service allows us great flexibility in managing the GPU resource: we can have a separate context per calculator, share a single context, share a lock or other synchronization primitives, etc. -- and all of this is managed by the helper and hidden from the individual calculators.
## GpuBuffer to ImageFrame Converters
We provide two calculators called `GpuBufferToImageFrameCalculator` and `ImageFrameToGpuBufferCalculator`. These calculators convert between `ImageFrame` and `GpuBuffer`, allowing the construction of graphs that combine GPU and CPU calculators. They are supported on both iOS and Android
When possible, these calculators use platform-specific functionality to share data between the CPU and the GPU without copying.
The below diagram shows the data flow in a mobile application that captures video from the camera, runs it through a MediaPipe graph, and renders the output on the screen in real time. The dashed line indicates which parts are inside the MediaPipe graph proper. This application runs a Canny edge-detection filter on the CPU using OpenCV, and overlays it on top of the original video using the GPU.
| ![How GPU calculators interact](../images/gpu_example_graph.png) |
| :--------------------------------------------------------------------------: |
| *Video frames from the camera are fed into the graph as `GpuBuffer` packets. |
: The input stream is accessed by two calculators in parallel. :
: `GpuBufferToImageFrameCalculator` converts the buffer into an `ImageFrame`, :
: which is then sent through a grayscale converter and a canny filter (both :
: based on OpenCV and running on the CPU), whose output is then converted into :
: a `GpuBuffer` again. A multi-input GPU calculator, GlOverlayCalculator, :
: takes as input both the original `GpuBuffer` and the one coming out of the :
: edge detector, and overlays them using a shader. The output is then sent :
: back to the application using a callback calculator, and the application :
: renders the image to the screen using OpenGL.* :
docs/framework_concepts/graphs.md
+271
View File
@@ -0,0 +1,271 @@
---
layout: default
title: Graphs
parent: Framework Concepts
nav_order: 2
---
# Graphs
{: .no_toc }
1. TOC
{:toc}
---
## GraphConfig
A `GraphConfig` is a specification that describes the topology and functionality
of a MediaPipe graph. In the specification, a node in the graph represents an
instance of a particular calculator. All the necessary configurations of the
node, such its type, inputs and outputs must be described in the specification.
Description of the node can also include several optional fields, such as
node-specific options, input policy and executor, discussed in
[Synchronization](synchronization.md).
`GraphConfig` has several other fields to configure the global graph-level
settings, eg, graph executor configs, number of threads, and maximum queue size
of input streams. Several graph-level settings are useful for tuning the
performance of the graph on different platforms (eg, desktop v.s. mobile). For
instance, on mobile, attaching a heavy model-inference calculator to a separate
executor can improve the performance of a real-time application since this
enables thread locality.
Below is a trivial `GraphConfig` example where we have series of passthrough
calculators :
```proto
# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
# passthrough calculators.
input_stream: "in"
node {
calculator: "PassThroughCalculator"
input_stream: "in"
output_stream: "out1"
}
node {
calculator: "PassThroughCalculator"
input_stream: "out1"
output_stream: "out2"
}
node {
calculator: "PassThroughCalculator"
input_stream: "out2"
output_stream: "out3"
}
node {
calculator: "PassThroughCalculator"
input_stream: "out3"
output_stream: "out4"
}
```
## Subgraph
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
public interface of a subgraph consists of a set of input and output streams
similar to a calculator's public interface. The subgraph can then be included in
an `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is
loaded from a `CalculatorGraphConfig`, each subgraph node is replaced by the
corresponding graph of calculators. As a result, the semantics and performance
of the subgraph is identical to the corresponding graph of calculators.
Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`.
1. Defining the subgraph.
```proto
# This subgraph is defined in two_pass_through_subgraph.pbtxt
# and is registered as "TwoPassThroughSubgraph"
type: "TwoPassThroughSubgraph"
input_stream: "out1"
output_stream: "out3"
node {
calculator: "PassThroughculator"
input_stream: "out1"
output_stream: "out2"
}
node {
calculator: "PassThroughculator"
input_stream: "out2"
output_stream: "out3"
}
```
The public interface to the subgraph consists of:
* Graph input streams
* Graph output streams
* Graph input side packets
* Graph output side packets
2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`. The
parameter `register_as` defines the component name for the new subgraph.
```proto
# Small section of BUILD file for registering the "TwoPassThroughSubgraph"
# subgraph for use by main graph main_pass_throughcals.pbtxt
mediapipe_simple_subgraph(
name = "twopassthrough_subgraph",
graph = "twopassthrough_subgraph.pbtxt",
register_as = "TwoPassThroughSubgraph",
deps = [
"//mediapipe/calculators/core:pass_through_calculator",
"//mediapipe/framework:calculator_graph",
],
)
```
3. Use the subgraph in the main graph.
```proto
# This main graph is defined in main_pass_throughcals.pbtxt
# using subgraph called "TwoPassThroughSubgraph"
input_stream: "in"
node {
calculator: "PassThroughCalculator"
input_stream: "in"
output_stream: "out1"
}
node {
calculator: "TwoPassThroughSubgraph"
input_stream: "out1"
output_stream: "out3"
}
node {
calculator: "PassThroughCalculator"
input_stream: "out3"
output_stream: "out4"
}
```
## Cycles
<!-- TODO: add discussion of PreviousLoopbackCalculator -->
By default, MediaPipe requires calculator graphs to be acyclic and treats cycles
in a graph as errors. If a graph is intended to have cycles, the cycles need to
be annotated in the graph config. This page describes how to do that.
NOTE: The current approach is experimental and subject to change. We welcome
your feedback.
Please use the `CalculatorGraphTest.Cycle` unit test in
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown
below is the cyclic graph in the test. The `sum` output of the adder is the sum
of the integers generated by the integer source calculator.
![a cyclic graph that adds a stream of integers](../images/cyclic_integer_sum_graph.svg "A cyclic graph")
This simple graph illustrates all the issues in supporting cyclic graphs.
### Back Edge Annotation
We require that an edge in each cycle be annotated as a back edge. This allows
MediaPipe’s topological sort to work, after removing all the back edges.
There are usually multiple ways to select the back edges. Which edges are marked
as back edges affects which nodes are considered as upstream and which nodes are
considered as downstream, which in turn affects the priorities MediaPipe assigns
to the nodes.
For example, the `CalculatorGraphTest.Cycle` test marks the `old_sum` edge as a
back edge, so the Delay node is considered as a downstream node of the adder
node and is given a higher priority. Alternatively, we could mark the `sum`
input to the delay node as the back edge, in which case the delay node would be
considered as an upstream node of the adder node and is given a lower priority.
### Initial Packet
For the adder calculator to be runnable when the first integer from the integer
source arrives, we need an initial packet, with value 0 and with the same
timestamp, on the `old_sum` input stream to the adder. This initial packet
should be output by the delay calculator in the `Open()` method.
### Delay in a Loop
Each loop should incur a delay to align the previous `sum` output with the next
integer input. This is also done by the delay node. So the delay node needs to
know the following about the timestamps of the integer source calculator:
* The timestamp of the first output.
* The timestamp delta between successive outputs.
We plan to add an alternative scheduling policy that only cares about packet
ordering and ignores packet timestamps, which will eliminate this inconvenience.
### Early Termination of a Calculator When One Input Stream is Done
By default, MediaPipe calls the `Close()` method of a non-source calculator when
all of its input streams are done. In the example graph, we want to stop the
adder node as soon as the integer source is done. This is accomplished by
configuring the adder node with an alternative input stream handler,
`EarlyCloseInputStreamHandler`.
### Relevant Source Code
#### Delay Calculator
Note the code in `Open()` that outputs the initial packet and the code in
`Process()` that adds a (unit) delay to input packets. As noted above, this
delay node assumes that its output stream is used alongside an input stream with
packet timestamps 0, 1, 2, 3, ...
```c++
class UnitDelayCalculator : public Calculator {
public:
 static ::util::Status FillExpectations(
     const CalculatorOptions& extendable_options, PacketTypeSet* inputs,
     PacketTypeSet* outputs, PacketTypeSet* input_side_packets) {
   inputs->Index(0)->Set<int>("An integer.");
   outputs->Index(0)->Set<int>("The input delayed by one time unit.");
   return ::mediapipe::OkStatus();
 }
 ::util::Status Open() final {
   Output()->Add(new int(0), Timestamp(0));
   return ::mediapipe::OkStatus();
 }
 ::util::Status Process() final {
   const Packet& packet = Input()->Value();
   Output()->AddPacket(packet.At(packet.Timestamp().NextAllowedInStream()));
   return ::mediapipe::OkStatus();
 }
};
```
#### Graph Config
Note the `back_edge` annotation and the alternative `input_stream_handler`.
```proto
node {
  calculator: 'GlobalCountSourceCalculator'
  input_side_packet: 'global_counter'
  output_stream: 'integers'
}
node {
  calculator: 'IntAdderCalculator'
  input_stream: 'integers'
  input_stream: 'old_sum'
  input_stream_info: {
    tag_index: ':1' # 'old_sum'
    back_edge: true
  }
  output_stream: 'sum'
  input_stream_handler {
    input_stream_handler: 'EarlyCloseInputStreamHandler'
  }
}
node {
  calculator: 'UnitDelayCalculator'
  input_stream: 'sum'
  output_stream: 'old_sum'
}
```
docs/framework_concepts/packets.md
+30
View File
@@ -0,0 +1,30 @@
---
layout: default
title: Packets
parent: Framework Concepts
nav_order: 3
---
# Packets
{: .no_toc }
1. TOC
{:toc}
---
Each calculator is a node of of a graph. We describe how to create a new calculator, how to initialize a calculator, how to perform its calculations, input and output streams, timestamps, and options
## Creating a packet
Packets are generally created with `MediaPipe::Adopt()` (from packet.h).
```c++
// Create some data.
auto data = gtl::MakeUnique<MyDataClass>("constructor_argument");
// Create a packet to own the data.
Packet p = Adopt(data.release());
// Make a new packet with the same data and a different timestamp.
Packet p2 = p.At(Timestamp::PostStream());
```
Data within a packet is accessed with `Packet::Get<T>()`
docs/framework_concepts/synchronization.md
+175
View File
@@ -0,0 +1,175 @@
---
layout: default
title: Synchronization
parent: Framework Concepts
nav_order: 4
---
# Synchronization
{: .no_toc }
1. TOC
{:toc}
---
## Scheduling mechanics
Data processing in a MediaPipe graph occurs inside processing nodes defined as
[`CalculatorBase`] subclasses. The scheduling system decides when each
calculator should run.
Each graph has at least one **scheduler queue**. Each scheduler queue has
exactly one **executor**. Nodes are statically assigned to a queue (and
therefore to an executor). By default there is one queue, whose executor is a
thread pool with a number of threads based on the system’s capabilities.
Each node has a scheduling state, which can be *not ready*, *ready*, or
*running*. A readiness function determines whether a node is ready to run. This
function is invoked at graph initialization, whenever a node finishes running,
and whenever the state of a node’s inputs changes.
The readiness function used depends on the type of node. A node with no stream
inputs is known as a **source node**; source nodes are always ready to run,
until they tell the framework they have no more data to output, at which point
they are closed.
Non-source nodes are ready if they have inputs to process, and if those inputs
form a valid input set according to the conditions set by the node’s **input
policy** (discussed below). Most nodes use the default input policy, but some
nodes specify a different one.
Note: Because changing the input policy changes the guarantees the calculator’s
code can expect from its inputs, it is not generally possible to mix and match
calculators with arbitrary input policies. Thus a calculator that uses a special
input policy should be written for it, and declare it in its contract.
When a node becomes ready, a task is added to the corresponding scheduler queue,
which is a priority queue. The priority function is currently fixed, and takes
into account static properties of the nodes and their topological sorting within
the graph. For example, nodes closer to the output side of the graph have higher
priority, while source nodes have the lowest priority.
Each queue is served by an executor, which is responsible for actually running
the task by invoking the calculator’s code. Different executors can be provided
and configured; this can be used to customize the use of execution resources,
e.g. by running certain nodes on lower-priority threads.
## Timestamp Synchronization
MediaPipe graph execution is decentralized: there is no global clock, and
different nodes can process data from different timestamps at the same time.
This allows higher throughput via pipelining.
However, time information is very important for many perception workflows. Nodes
that receive multiple input streams generally need to coordinate them in some
way. For example, an object detector may output a list of boundary rectangles
from a frame, and this information may be fed into a rendering node, which
should process it together with the original frame.
Therefore, one of the key responsibilities of the MediaPipe framework is to
provide input synchronization for nodes. In terms of framework mechanics, the
primary role of a timestamp is to serve as a **synchronization key**.
Furthermore, MediaPipe is designed to support deterministic operations, which is
important in many scenarios (testing, simulation, batch processing, etc.), while
allowing graph authors to relax determinism where needed to meet real-time
constraints.
The two objectives of synchronization and determinism underlie several design
choices. Notably, the packets pushed into a given stream must have monotonically
increasing timestamps: this is not just a useful assumption for many nodes, but
it is also relied upon by the synchronization logic. Each stream has a
**timestamp bound**, which is the lowest possible timestamp allowed for a new
packet on the stream. When a packet with timestamp `T` arrives, the bound
automatically advances to `T+1`, reflecting the monotonic requirement. This
allows the framework to know for certain that no more packets with timestamp
lower than `T` will arrive.
## Input policies
Synchronization is handled locally on each node, using the input policy
specified by the node.
The default input policy, defined by [`DefaultInputStreamHandler`], provides
deterministic synchronization of inputs, with the following guarantees:
* If packets with the same timestamp are provided on multiple input streams,
they will always be processed together regardless of their arrival order in
real time.
* Input sets are processed in strictly ascending timestamp order.
* No packets are dropped, and the processing is fully deterministic.
* The node becomes ready to process data as soon as possible given the
guarantees above.
Note: An important consequence of this is that if the calculator always uses the
current input timestamp when outputting packets, the output will inherently obey
the monotonically increasing timestamp requirement.
Warning: On the other hand, it is not guaranteed that an input packet will
always be available for all streams.
To explain how it works, we need to introduce the definition of a settled
timestamp. We say that a timestamp in a stream is *settled* if it lower than the
timestamp bound. In other words, a timestamp is settled for a stream once the
state of the input at that timestamp is irrevocably known: either there is a
packet, or there is the certainty that a packet with that timestamp will not
arrive.
Note: For this reason, MediaPipe also allows a stream producer to explicitly
advance the timestamp bound farther that what the last packet implies, i.e. to
provide a tighter bound. This can allow the downstream nodes to settle their
inputs sooner.
A timestamp is settled across multiple streams if it is settled on each of those
streams. Furthermore, if a timestamp is settled it implies that all previous
timestamps are also settled. Thus settled timestamps can be processed
deterministically in ascending order.
Given this definition, a calculator with the default input policy is ready if
there is a timestamp which is settled across all input streams and contains a
packet on at least one input stream. The input policy provides all available
packets for a settled timestamp as a single *input set* to the calculator.
One consequence of this deterministic behavior is that, for nodes with multiple
input streams, there can be a theoretically unbounded wait for a timestamp to be
settled, and an unbounded number of packets can be buffered in the meantime.
(Consider a node with two input streams, one of which keeps sending packets
while the other sends nothing and does not advance the bound.)
Therefore, we also provide for custom input policies: for example, splitting the
inputs in different synchronization sets defined by
[`SyncSetInputStreamHandler`], or avoiding synchronization altogether and
processing inputs immediately as they arrive defined by
[`ImmediateInputStreamHandler`].
## Flow control
There are two main flow control mechanisms. A backpressure mechanism throttles
the execution of upstream nodes when the packets buffered on a stream reach a
(configurable) limit defined by [`CalculatorGraphConfig::max_queue_size`]. This
mechanism maintains deterministic behavior, and includes a deadlock avoidance
system that relaxes configured limits when needed.
The second system consists of inserting special nodes which can drop packets
according to real-time constraints (typically using custom input policies)
defined by [`FlowLimiterCalculator`]. For example, a common pattern places a
flow-control node at the input of a subgraph, with a loopback connection from
the final output to the flow-control node. The flow-control node is thus able to
keep track of how many timestamps are being processed in the downstream graph,
and drop packets if this count hits a (configurable) limit; and since packets
are dropped upstream, we avoid the wasted work that would result from partially
processing a timestamp and then dropping packets between intermediate stages.
This calculator-based approach gives the graph author control of where packets
can be dropped, and allows flexibility in adapting and customizing the graph’s
behavior depending on resource constraints.
[`CalculatorBase`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
[`DefaultInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/default_input_stream_handler.h
[`SyncSetInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/sync_set_input_stream_handler.h
[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.h
[`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
docs/getting_started/android_archive_library.md
+146
View File
@@ -0,0 +1,146 @@
---
layout: default
title: MediaPipe Android Archive
parent: Getting Started
nav_order: 7
---
# MediaPipe Android Archive
{: .no_toc }
1. TOC
{:toc}
---
***Experimental Only***
The MediaPipe Android Archive (AAR) library is a convenient way to use MediaPipe
with Android Studio and Gradle. MediaPipe doesn't publish a general AAR that can
be used by all projects. Instead, developers need to add a mediapipe_aar()
target to generate a custom AAR file for their own projects. This is necessary
in order to include specific resources such as MediaPipe calculators needed for
each project.
## Steps to build a MediaPipe AAR
1. Create a mediapipe_aar() target.
In the MediaPipe directory, create a new mediapipe_aar() target in a BUILD
file. You need to figure out what calculators are used in the graph and
provide the calculator dependencies to the mediapipe_aar(). For example, to
build an AAR for [MediaPipe Face Detection](../solutions/face_detection.md),
you can put the following code into
mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/BUILD.
```
load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")
mediapipe_aar(
name = "mp_face_detection_aar",
calculators = ["//mediapipe/graphs/face_detection:mobile_calculators"],
)
```
2. Run the Bazel build command to generate the AAR.
```bash
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \
//path/to/the/aar/build/file:aar_name
```
For the face detection AAR target we made in the step 1, run:
```bash
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \
//mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar
# It should print:
# Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar up-to-date:
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
```
3. (Optional) Save the AAR to your preferred location.
```bash
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
/absolute/path/to/your/preferred/location
```
## Steps to use a MediaPipe AAR in Android Studio with Gradle
1. Start Android Studio and go to your project.
2. Copy the AAR into app/libs.
```bash
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar
/path/to/your/app/libs/
```
![Screenshot](../images/mobile/aar_location.png)
3. Make app/src/main/assets and copy assets (graph, model, and etc) into
app/src/main/assets.
Build the MediaPipe binary graph and copy the assets into
app/src/main/assets, e.g., for the face detection graph, you need to build
and copy
[the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41),
[the tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite),
and
[the label map](https://github.com/google/mediapipe/blob/master/mediapipe/models/face_detection_front_labelmap.txt).
```bash
bazel build -c opt mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu:binary_graph
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.binarypb /path/to/your/app/src/main/assets/
cp mediapipe/models/face_detection_front.tflite /path/to/your/app/src/main/assets/
cp mediapipe/models/face_detection_front_labelmap.txt /path/to/your/app/src/main/assets/
```
![Screenshot](../images/mobile/assets_location.png)
4. Make app/src/main/jniLibs and copy OpenCV JNI libraries into
app/src/main/jniLibs.
MediaPipe depends on OpenCV, you will need to copy the precompiled OpenCV so
files into app/src/main/jniLibs. You can download the official OpenCV
Android SDK from
[here](https://github.com/opencv/opencv/releases/download/3.4.3/opencv-3.4.3-android-sdk.zip)
and run:
```bash
cp -R ~/Downloads/OpenCV-android-sdk/sdk/native/libs/arm* /path/to/your/app/src/main/jniLibs/
```
![Screenshot](../images/mobile/android_studio_opencv_location.png)
5. Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR.
```
dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar', '*.aar'])
implementation 'androidx.appcompat:appcompat:1.0.2'
implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'androidx.test.ext:junit:1.1.0'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.1.1'
// MediaPipe deps
implementation 'com.google.flogger:flogger:0.3.1'
implementation 'com.google.flogger:flogger-system-backend:0.3.1'
implementation 'com.google.code.findbugs:jsr305:3.0.2'
implementation 'com.google.guava:guava:27.0.1-android'
implementation 'com.google.guava:guava:27.0.1-android'
implementation 'com.google.protobuf:protobuf-java:3.11.4''
// CameraX core library
def camerax_version = "1.0.0-alpha06"
implementation "androidx.camera:camera-core:$camerax_version"
implementation "androidx.camera:camera-camera2:$camerax_version"
}
```
6. Follow our Android app examples to use MediaPipe in Android Studio for your
use case. If you are looking for an example, a face detection example can be
found
[here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and
a multi-hand tracking example can be found
[here](https://github.com/jiuqiant/mediapipe_multi_hands_tracking_aar_example).
docs/getting_started/building_examples.md
+338
View File
@@ -0,0 +1,338 @@
---
layout: default
title: Building MediaPipe Examples
parent: Getting Started
nav_order: 2
---
# Building MediaPipe Examples
{: .no_toc }
1. TOC
{:toc}
---
## Android
### Prerequisite
* Java Runtime.
* Android SDK release 28.0.3 and above.
* Android NDK r18b and above.
MediaPipe recommends setting up Android SDK and NDK via Android Studio (and see
below for Android Studio setup). However, if you prefer using MediaPipe without
Android Studio, please run
[`setup_android_sdk_and_ndk.sh`](https://github.com/google/mediapipe/tree/master/setup_android_sdk_and_ndk.sh)
to download and setup Android SDK and NDK before building any Android example
apps.
If Android SDK and NDK are already installed (e.g., by Android Studio), set
$ANDROID_HOME and $ANDROID_NDK_HOME to point to the installed SDK and NDK.
```bash
export ANDROID_HOME=<path to the Android SDK>
export ANDROID_NDK_HOME=<path to the Android NDK>
```
In order to use MediaPipe on earlier Android versions, MediaPipe needs to switch
to a lower Android API level. You can achieve this by specifying `api_level =
$YOUR_INTENDED_API_LEVEL` in android_ndk_repository() and/or
android_sdk_repository() in the
[`WORKSPACE`](https://github.com/google/mediapipe/tree/master/WORKSPACE) file.
Please verify all the necessary packages are installed.
* Android SDK Platform API Level 28 or 29
* Android SDK Build-Tools 28 or 29
* Android SDK Platform-Tools 28 or 29
* Android SDK Tools 26.1.1
* Android NDK 17c or above
### Option 1: Build with Bazel in Command Line
1. To build an Android example app, build against the corresponding
`android_binary` build target. For instance, for
[MediaPipe Hand](../solutions/hand.md) the target is `handtrackinggpu` in
the
[BUILD](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/BUILD)
file:
Note: To reduce the binary size, consider appending `--linkopt="-s"` to the
command below to strip symbols.
```bash
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu
```
1. Install it on a device with:
```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
```
### Option 2: Build with Bazel in Android Studio
The MediaPipe project can be imported into Android Studio using the Bazel
plugins. This allows the MediaPipe examples to be built and modified in Android
Studio.
To incorporate MediaPipe into an existing Android Studio project, see these
[instructions](./android_archive_library.md) that use Android Archive (AAR) and
Gradle.
The steps below use Android Studio 3.5 to build and install a MediaPipe example
app:
1. Install and launch Android Studio 3.5.
2. Select `Configure` -> `SDK Manager` -> `SDK Platforms`.
* Verify that Android SDK Platform API Level 28 or 29 is installed.
* Take note of the Android SDK Location, e.g.,
`/usr/local/home/Android/Sdk`.
3. Select `Configure` -> `SDK Manager` -> `SDK Tools`.
* Verify that Android SDK Build-Tools 28 or 29 is installed.
* Verify that Android SDK Platform-Tools 28 or 29 is installed.
* Verify that Android SDK Tools 26.1.1 is installed.
* Verify that Android NDK 17c or above is installed.
* Take note of the Android NDK Location, e.g.,
`/usr/local/home/Android/Sdk/ndk-bundle` or
`/usr/local/home/Android/Sdk/ndk/20.0.5594570`.
4. Set environment variables `$ANDROID_HOME` and `$ANDROID_NDK_HOME` to point
to the installed SDK and NDK.
```bash
export ANDROID_HOME=/usr/local/home/Android/Sdk
# If the NDK libraries are installed by a previous version of Android Studio, do
export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk-bundle
# If the NDK libraries are installed by Android Studio 3.5, do
export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk/<version number>
```
5. Select `Configure` -> `Plugins` to install `Bazel`.
6. On Linux, select `File` -> `Settings` -> `Bazel settings`. On macos, select
`Android Studio` -> `Preferences` -> `Bazel settings`. Then, modify `Bazel
binary location` to be the same as the output of `$ which bazel`.
7. Select `Import Bazel Project`.
* Select `Workspace`: `/path/to/mediapipe` and select `Next`.
* Select `Generate from BUILD file`: `/path/to/mediapipe/BUILD` and select
`Next`.
* Modify `Project View` to be the following and select `Finish`.
```
directories:
# read project settings, e.g., .bazelrc
.
-mediapipe/objc
-mediapipe/examples/ios
targets:
//mediapipe/examples/android/...:all
//mediapipe/java/...:all
android_sdk_platform: android-29
sync_flags:
--host_crosstool_top=@bazel_tools//tools/cpp:toolchain
```
8. Select `Bazel` -> `Sync` -> `Sync project with Build files`.
Note: Even after doing step 4, if you still see the error: `"no such package
'@androidsdk//': Either the path attribute of android_sdk_repository or the
ANDROID_HOME environment variable must be set."`, please modify the
[`WORKSPACE`](https://github.com/google/mediapipe/tree/master/WORKSPACE) file to point to your
SDK and NDK library locations, as below:
```
android_sdk_repository(
name = "androidsdk",
path = "/path/to/android/sdk"
)
android_ndk_repository(
name = "androidndk",
path = "/path/to/android/ndk"
)
```
9. Connect an Android device to the workstation.
10. Select `Run...` -> `Edit Configurations...`.
* Select `Templates` -> `Bazel Command`.
* Enter Target Expression:
`//mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu`
* Enter Bazel command: `mobile-install`.
* Enter Bazel flags: `-c opt --config=android_arm64`.
* Press the `[+]` button to add the new configuration.
* Select `Run` to run the example app on the connected Android device.
## iOS
### Prerequisite
1. Install [Xcode](https://developer.apple.com/xcode/) and the Command Line
Tools.
Follow Apple's instructions to obtain the required development certificates
and provisioning profiles for your iOS device. Install the Command Line
Tools by
```bash
xcode-select --install
```
2. Install [Bazel](https://bazel.build/).
We recommend using [Homebrew](https://brew.sh/) to get the latest version.
3. Set Python 3.7 as the default Python version and install the Python "six"
library.
To make Mediapipe work with TensorFlow, please set Python 3.7 as the default
Python version and install the Python "six" library.
```bash
pip3 install --user six
```
4. Clone the MediaPipe repository.
```bash
git clone https://github.com/google/mediapipe.git
```
5. Symlink or copy your provisioning profile to
`mediapipe/mediapipe/provisioning_profile.mobileprovision`.
```bash
cd mediapipe
ln -s ~/Downloads/MyProvisioningProfile.mobileprovision mediapipe/provisioning_profile.mobileprovision
```
Tip: You can use this command to see the provisioning profiles you have
previously downloaded using Xcode: `open
~/Library/MobileDevice/"Provisioning Profiles"`. If there are none, generate
and download a profile on
[Apple's developer site](https://developer.apple.com/account/resources/).
### Option 1: Build with Bazel in Command Line
1. Modify the `bundle_id` field of the app's `ios_application` build target to
use your own identifier. For instance, for
[MediaPipe Hand](../solutions/hand.md), the `bundle_id` is in the
`HandTrackingGpuApp` target in the
[BUILD](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu/BUILD)
file.
2. Again using [MediaPipe Hand](../solutions/hand.md) for example, run:
```bash
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
```
You may see a permission request from `codesign` in order to sign the app.
3. In Xcode, open the `Devices and Simulators` window (command-shift-2).
4. Make sure your device is connected. You will see a list of installed apps.
Press the "+" button under the list, and select the `.ipa` file built by
Bazel.
5. You can now run the app on your device.
### Option 2: Build in Xcode
Note: This workflow requires a separate tool in addition to Bazel. If it fails
to work for some reason, please resort to the command-line build instructions in
the previous section.
1. We will use a tool called [Tulsi](https://tulsi.bazel.build/) for generating
Xcode projects from Bazel build configurations.
```bash
# cd out of the mediapipe directory, then:
git clone https://github.com/bazelbuild/tulsi.git
cd tulsi
# remove Xcode version from Tulsi's .bazelrc (see http://github.com/bazelbuild/tulsi#building-and-installing):
sed -i .orig '/xcode_version/d' .bazelrc
# build and run Tulsi:
sh build_and_run.sh
```
This will install `Tulsi.app` inside the `Applications` directory in your
home directory.
2. Open `mediapipe/Mediapipe.tulsiproj` using the Tulsi app.
Important: If Tulsi displays an error saying "Bazel could not be found",
press the "Bazel..." button in the Packages tab and select the `bazel`
executable in your homebrew `/bin/` directory.
3. Select the MediaPipe config in the Configs tab, then press the Generate
button below. You will be asked for a location to save the Xcode project.
Once the project is generated, it will be opened in Xcode.
4. You can now select any of the MediaPipe demos in the target menu, and build
and run them as normal.
Note: When you ask Xcode to run an app, by default it will use the Debug
configuration. Some of our demos are computationally heavy; you may want to
use the Release configuration for better performance.
Tip: To switch build configuration in Xcode, click on the target menu,
choose "Edit Scheme...", select the Run action, and switch the Build
Configuration from Debug to Release. Note that this is set independently for
each target.
## Desktop
### Option 1: Running on CPU
1. To build, for example, [MediaPipe Hand](../solutions/hand.md), run:
```bash
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible.
2. To run the application:
```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
```
### Option 2: Running on GPU
Note: This currently works only on Linux, and please first follow
[OpenGL ES Setup on Linux Desktop](./gpu_support.md#opengl-es-setup-on-linux-desktop).
1. To build, for example, [MediaPipe Hand](../solutions/hand.md), run:
```bash
bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible, or GPU drivers not setup
properly.
2. To run the application:
```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
```
docs/getting_started/faq.md
+146
View File
@@ -0,0 +1,146 @@
---
layout: default
title: FAQ
parent: Getting Started
nav_order: 9
---
# FAQ
{: .no_toc }
1. TOC
{:toc}
---
### How to convert ImageFrames and GpuBuffers
The Calculators [`ImageFrameToGpuBufferCalculator`] and
[`GpuBufferToImageFrameCalculator`] convert back and forth between packets of
type [`ImageFrame`] and [`GpuBuffer`]. [`ImageFrame`] refers to image data in
CPU memory in any of a number of bitmap image formats. [`GpuBuffer`] refers to
image data in GPU memory. You can find more detail in the Framework Concepts
section
[GpuBuffer to ImageFrame Converters](./gpu.md#gpubuffer-to-imageframe-converters).
You can see an example in:
* [`object_detection_mobile_cpu.pbtxt`]
### How to visualize perception results
The [`AnnotationOverlayCalculator`] allows perception results, such as bounding
boxes, arrows, and ovals, to be superimposed on the video frames aligned with
the recognized objects. The results can be displayed in a diagnostic window when
running on a workstation, or in a texture frame when running on device. You can
see an example use of [`AnnotationOverlayCalculator`] in:
* [`face_detection_mobile_gpu.pbtxt`].
### How to run calculators in parallel
Within a calculator graph, MediaPipe routinely runs separate calculator nodes
in parallel. MediaPipe maintains a pool of threads, and runs each calculator
as soon as a thread is available and all of it's inputs are ready. Each
calculator instance is only run for one set of inputs at a time, so most
calculators need only to be *thread-compatible* and not *thread-safe*.
In order to enable one calculator to process multiple inputs in parallel, there
are two possible approaches:
1. Define multiple calulator nodes and dispatch input packets to all nodes.
2. Make the calculator thread-safe and configure its [`max_in_flight`] setting.
The first approach can be followed using the calculators designed to distribute
packets across other calculators, such as [`RoundRobinDemuxCalculator`]. A
single [`RoundRobinDemuxCalculator`] can distribute successive packets across
several identically configured [`ScaleImageCalculator`] nodes.
The second approach allows up to [`max_in_flight`] invocations of the
[`CalculatorBase::Process`] method on the same calculator node. The output
packets from [`CalculatorBase::Process`] are automatically ordered by timestamp
before they are passed along to downstream calculators.
With either aproach, you must be aware that the calculator running in parallel
cannot maintain internal state in the same way as a normal sequential
calculator.
### Output timestamps when using ImmediateInputStreamHandler
The [`ImmediateInputStreamHandler`] delivers each packet as soon as it arrives
at an input stream. As a result, it can deliver a packet
with a higher timestamp from one input stream before delivering a packet with a
lower timestamp from a different input stream. If these input timestamps are
both used for packets sent to one output stream, that output stream will
complain that the timestamps are not monotonically increasing. In order to
remedy this, the calculator must take care to output a packet only after
processing is complete for its timestamp. This could be accomplished by waiting
until input packets have been received from all inputstreams for that timestamp,
or by ignoring a packet that arrives with a timestamp that has already been
processed.
### How to change settings at runtime
There are two main approaches to changing the settings of a calculator graph
while the application is running:
1. Restart the calculator graph with modified [`CalculatorGraphConfig`].
2. Send new calculator options through packets on graph input-streams.
The first approach has the advantage of leveraging [`CalculatorGraphConfig`]
processing tools such as "subgraphs". The second approach has the advantage of
allowing active calculators and packets to remain in-flight while settings
change. Mediapipe contributors are currently investigating alternative approaches
to achieve both of these advantages.
### How to process realtime input streams
The mediapipe framework can be used to process data streams either online or
offline. For offline processing, packets are pushed into the graph as soon as
calculators are ready to process those packets. For online processing, one
packet for each frame is pushed into the graph as that frame is recorded.
The MediaPipe framework requires only that successive packets be assigned
monotonically increasing timestamps. By convention, realtime calculators and
graphs use the recording time or the presentation time as the timestamp for each
packet, with each timestamp representing microseconds since
`Jan/1/1970:00:00:00`. This allows packets from various sources to be processed
in a gloablly consistent order.
Normally for offline processing, every input packet is processed and processing
continues as long as necessary. For online processing, it is often necessary to
drop input packets in order to keep pace with the arrival of input data frames.
When inputs arrive too frequently, the recommended technique for dropping
packets is to use the MediaPipe calculators designed specifically for this
purpose such as [`FlowLimiterCalculator`] and [`PacketClonerCalculator`].
For online processing, it is also necessary to promptly determine when processing
can proceed. MediaPipe supports this by propagating timestamp bounds between
calculators. Timestamp bounds indicate timestamp intervals that will contain no
input packets, and they allow calculators to begin processing for those
timestamps immediately. Calculators designed for realtime processing should
carefully calculate timestamp bounds in order to begin processing as promptly as
possible. For example, the [`MakePairCalculator`] uses the `SetOffset` API to
propagate timestamp bounds from input streams to output streams.
### Can I run MediaPipe on MS Windows?
Currently MediaPipe portability supports Debian Linux, Ubuntu Linux,
MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library
conforming to the C++11 standard, so it is relatively easy to port to
additional platforms.
[`object_detection_mobile_cpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt
[`ImageFrame`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/formats/image_frame.h
[`GpuBuffer`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer.h
[`GpuBufferToImageFrameCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/gpu_buffer_to_image_frame_calculator.cc
[`ImageFrameToGpuBufferCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/gpu/image_frame_to_gpu_buffer_calculator.cc
[`AnnotationOverlayCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/util/annotation_overlay_calculator.cc
[`face_detection_mobile_gpu.pbtxt`]: https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt
[`CalculatorBase::Process`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
[`max_in_flight`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`RoundRobinDemuxCalculator`]: https://github.com/google/mediapipe/tree/master//mediapipe/calculators/core/round_robin_demux_calculator.cc
[`ScaleImageCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/image/scale_image_calculator.cc
[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.cc
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
[`PacketClonerCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/packet_cloner_calculator.cc
[`MakePairCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/make_pair_calculator.cc
docs/getting_started/getting_started.md
+13
View File
@@ -0,0 +1,13 @@
---
layout: default
title: Getting Started
nav_order: 2
has_children: true
---
# Getting Started
{: .no_toc }
1. TOC
{:toc}
---
docs/getting_started/gpu_support.md
+186
View File
@@ -0,0 +1,186 @@
---
layout: default
title: GPU Support
parent: Getting Started
nav_order: 6
---
# GPU Support
{: .no_toc }
1. TOC
{:toc}
---
## OpenGL ES Support
MediaPipe supports OpenGL ES up to version 3.2 on Android/Linux and up to ES 3.0
on iOS. In addition, MediaPipe also supports Metal on iOS.
OpenGL ES 3.1 or greater is required (on Android/Linux systems) for running
machine learning inference calculators and graphs.
## Disable OpenGL ES Support
By default, building MediaPipe (with no special bazel flags) attempts to compile
and link against OpenGL ES (and for iOS also Metal) libraries.
On platforms where OpenGL ES is not available (see also
[OpenGL ES Setup on Linux Desktop](#opengl-es-setup-on-linux-desktop)), you
should disable OpenGL ES support with:
```
$ bazel build --define MEDIAPIPE_DISABLE_GPU=1 <my-target>
```
Note: On Android and iOS, OpenGL ES is required by MediaPipe framework and the
support should never be disabled.
## OpenGL ES Setup on Linux Desktop
On Linux desktop with video cards that support OpenGL ES 3.1+, MediaPipe can run
GPU compute and rendering and perform TFLite inference on GPU.
To check if your Linux desktop GPU can run MediaPipe with OpenGL ES:
```bash
$ sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
$ sudo apt-get install mesa-utils
$ glxinfo | grep -i opengl
```
For example, it may print:
```bash
$ glxinfo | grep -i opengl
...
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 430.50
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:
```
*Notice the ES 3.20 text above.*
You need to see ES 3.1 or greater printed in order to perform TFLite inference
on GPU in MediaPipe. With this setup, build with:
```
$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 <my-target>
```
If only ES 3.0 or below is supported, you can still build MediaPipe targets that
don't require TFLite inference on GPU with:
```
$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 --copt -DMEDIAPIPE_DISABLE_GL_COMPUTE <my-target>
```
Note: MEDIAPIPE_DISABLE_GL_COMPUTE is already defined automatically on all Apple
systems (Apple doesn't support OpenGL ES 3.1+).
## TensorFlow CUDA Support and Setup on Linux Desktop
MediaPipe framework doesn't require CUDA for GPU compute and rendering. However,
MediaPipe can work with TensorFlow to perform GPU inference on video cards that
support CUDA.
To enable TensorFlow GPU inference with MediaPipe, the first step is to follow
the
[TensorFlow GPU documentation](https://www.tensorflow.org/install/gpu#software_requirements)
to install the required NVIDIA software on your Linux desktop.
After installation, update `$PATH` and `$LD_LIBRARY_PATH` and run `ldconfig`
with:
```
$ export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}}
$ export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64,/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
$ sudo ldconfig
```
It's recommended to verify the installation of CUPTI, CUDA, CuDNN, and NVCC:
```
$ ls /usr/local/cuda/extras/CUPTI
/lib64
libcupti.so libcupti.so.10.1.208 libnvperf_host.so libnvperf_target.so
libcupti.so.10.1 libcupti_static.a libnvperf_host_static.a
$ ls /usr/local/cuda-10.1
LICENSE bin extras lib64 libnvvp nvml samples src tools
README doc include libnsight nsightee_plugins nvvm share targets version.txt
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
$ ls /usr/lib/x86_64-linux-gnu/ | grep libcudnn.so
libcudnn.so
libcudnn.so.7
libcudnn.so.7.6.4
```
Setting `$TF_CUDA_PATHS` is the way to declare where the CUDA library is. Note
that the following code snippet also adds `/usr/lib/x86_64-linux-gnu` and
`/usr/include` into `$TF_CUDA_PATHS` for cudablas and libcudnn.
```
$ export TF_CUDA_PATHS=/usr/local/cuda-10.1,/usr/lib/x86_64-linux-gnu,/usr/include
```
To make MediaPipe get TensorFlow's CUDA settings, find TensorFlow's
[.bazelrc](https://github.com/tensorflow/tensorflow/blob/master/.bazelrc) and
copy the `build:using_cuda` and `build:cuda` section into MediaPipe's .bazelrc
file. For example, as of April 23, 2020, TensorFlow's CUDA setting is the
following:
```
# This config refers to building with CUDA available. It does not necessarily
# mean that we build CUDA op kernels.
build:using_cuda --define=using_cuda=true
build:using_cuda --action_env TF_NEED_CUDA=1
build:using_cuda --crosstool_top=@local_config_cuda//crosstool:toolchain
# This config refers to building CUDA op kernels with nvcc.
build:cuda --config=using_cuda
build:cuda --define=using_cuda_nvcc=true
```
Finally, build MediaPipe with TensorFlow GPU with two more flags `--config=cuda`
and `--spawn_strategy=local`. For example:
```
$ bazel build -c opt --config=cuda --spawn_strategy=local \
--define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS \
mediapipe/examples/desktop/object_detection:object_detection_tensorflow
```
While the binary is running, it prints out the GPU device info:
```
I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5 coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
```
You can monitor the GPU usage to verify whether the GPU is used for model
inference.
```
$ nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1
0 %
0 %
4 %
5 %
83 %
21 %
22 %
27 %
29 %
100 %
0 %
0%
```
docs/getting_started/hello_world_android.md
+778
View File
@@ -0,0 +1,778 @@
---
layout: default
title: Hello World! on Android
parent: Getting Started
nav_order: 3
---
# Hello World! on Android
{: .no_toc }
1. TOC
{:toc}
---
## Introduction
This codelab uses MediaPipe on an Android device.
### What you will learn
How to develop an Android application that uses MediaPipe and run a MediaPipe
graph on Android.
### What you will build
A simple camera app for real-time Sobel edge detection applied to a live video
stream on an Android device.
![edge_detection_android_gpu_gif](../images/mobile/edge_detection_android_gpu.gif)
## Setup
1. Install MediaPipe on your system, see [MediaPipe installation guide] for
details.
2. Install Android Development SDK and Android NDK. See how to do so also in
[MediaPipe installation guide].
3. Enable [developer options] on your Android device.
4. Setup [Bazel] on your system to build and deploy the Android app.
## Graph for edge detection
We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
```
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
# Used in the examples
# mediapipe/examples/android/src/java/com/mediapipe/apps/basic.
# mediapipe/examples/ios/edgedetectiongpu.
# Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"
# Converts RGB images into luminance images, still stored in RGB format.
node: {
calculator: "LuminanceCalculator"
input_stream: "input_video"
output_stream: "luma_video"
}
# Applies the Sobel filter to luminance images sotred in RGB format.
node: {
calculator: "SobelEdgesCalculator"
input_stream: "luma_video"
output_stream: "output_video"
}
```
A visualization of the graph is shown below:
![edge_detection_mobile_gpu](../images/mobile/edge_detection_mobile_gpu.png)
This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera.
The first node in the graph, `LuminanceCalculator`, takes a single packet (image
frame) and applies a change in luminance using an OpenGL shader. The resulting
image frame is sent to the `luma_video` output stream.
The second node, `SobelEdgesCalculator` applies edge detection to incoming
packets in the `luma_video` stream and outputs results in `output_video` output
stream.
Our Android application will display the output image frames of the
`output_video` stream.
## Initial minimal application setup
We first start with an simple Android application that displays "Hello World!"
on the screen. You may skip this step if you are familiar with building Android
applications using `bazel`.
Create a new directory where you will create your Android application. For
example, the complete code of this tutorial can be found at
`mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic`. We
will refer to this path as `$APPLICATION_PATH` throughout the codelab.
Note that in the path to the application:
* The application is named `helloworld`.
* The `$PACKAGE_PATH` of the application is
`com.google.mediapipe.apps.basic`. This is used in code snippets in this
tutorial, so please remember to use your own `$PACKAGE_PATH` when you
copy/use the code snippets.
Add a file `activity_main.xml` to `$APPLICATION_PATH/res/layout`. This displays
a [`TextView`] on the full screen of the application with the string `Hello
World!`:
```
<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent">
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Hello World!"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintLeft_toLeftOf="parent"
app:layout_constraintRight_toRightOf="parent"
app:layout_constraintTop_toTopOf="parent" />
</android.support.constraint.ConstraintLayout>
```
Add a simple `MainActivity.java` to `$APPLICATION_PATH` which loads the content
of the `activity_main.xml` layout as shown below:
```
package com.google.mediapipe.apps.basic;
import android.os.Bundle;
import androidx.appcompat.app.AppCompatActivity;
/** Bare-bones main activity. */
public class MainActivity extends AppCompatActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
}
}
```
Add a manifest file, `AndroidManifest.xml` to `$APPLICATION_PATH`, which
launches `MainActivity` on application start:
```
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.google.mediapipe.apps.basic">
<uses-sdk
android:minSdkVersion="19"
android:targetSdkVersion="19" />
<application
android:allowBackup="true"
android:label="${appName}"
android:supportsRtl="true"
android:theme="@style/AppTheme">
<activity
android:name="${mainActivity}"
android:exported="true"
android:screenOrientation="portrait">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>
```
In our application we are using a `Theme.AppCompat` theme in the app, so we need
appropriate theme references. Add `colors.xml` to
`$APPLICATION_PATH/res/values/`:
```
<?xml version="1.0" encoding="utf-8"?>
<resources>
<color name="colorPrimary">#008577</color>
<color name="colorPrimaryDark">#00574B</color>
<color name="colorAccent">#D81B60</color>
</resources>
```
Add `styles.xml` to `$APPLICATION_PATH/res/values/`:
```
<resources>
<!-- Base application theme. -->
<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
<!-- Customize your theme here. -->
<item name="colorPrimary">@color/colorPrimary</item>
<item name="colorPrimaryDark">@color/colorPrimaryDark</item>
<item name="colorAccent">@color/colorAccent</item>
</style>
</resources>
```
To build the application, add a `BUILD` file to `$APPLICATION_PATH`, and
`${appName}` and `${mainActivity}` in the manifest will be replaced by strings
specified in `BUILD` as shown below.
```
android_library(
name = "basic_lib",
srcs = glob(["*.java"]),
manifest = "AndroidManifest.xml",
resource_files = glob(["res/**"]),
deps = [
"//third_party:android_constraint_layout",
"//third_party:androidx_appcompat",
],
)
android_binary(
name = "helloworld",
manifest = "AndroidManifest.xml",
manifest_values = {
"applicationId": "com.google.mediapipe.apps.basic",
"appName": "Hello World",
"mainActivity": ".MainActivity",
},
multidex = "native",
deps = [
":basic_lib",
],
)
```
The `android_library` rule adds dependencies for `MainActivity`, resource files
and `AndroidManifest.xml`.
The `android_binary` rule, uses the `basic_lib` Android library generated to
build a binary APK for installation on your Android device.
To build the app, use the following command:
```
bazel build -c opt --config=android_arm64 $APPLICATION_PATH:helloworld
```
Install the generated APK file using `adb install`. For example:
```
adb install bazel-bin/$APPLICATION_PATH/helloworld.apk
```
Open the application on your device. It should display a screen with the text
`Hello World!`.
![bazel_hello_world_android](../images/mobile/bazel_hello_world_android.png)
## Using the camera via `CameraX`
### Camera Permissions
To use the camera in our application, we need to request the user to provide
access to the camera. To request camera permissions, add the following to
`AndroidManifest.xml`:
```
<!-- For using the camera -->
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />
```
Change the minimum SDK version to `21` and target SDK version to `27` in the
same file:
```
<uses-sdk
android:minSdkVersion="21"
android:targetSdkVersion="27" />
```
This ensures that the user is prompted to request camera permission and enables
us to use the [CameraX] library for camera access.
To request camera permissions, we can use a utility provided by MediaPipe
components, namely [`PermissionHelper`]. To use it, add a dependency
`"//mediapipe/java/com/google/mediapipe/components:android_components"` in the
`mediapipe_lib` rule in `BUILD`.
To use the `PermissionHelper` in `MainActivity`, add the following line to the
`onCreate` function:
```
PermissionHelper.checkAndRequestCameraPermissions(this);
```
This prompts the user with a dialog on the screen to request for permissions to
use the camera in this application.
Add the following code to handle the user response:
```
@Override
public void onRequestPermissionsResult(
int requestCode, String[] permissions, int[] grantResults) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
}
@Override
protected void onResume() {
super.onResume();
if (PermissionHelper.cameraPermissionsGranted(this)) {
startCamera();
}
}
public void startCamera() {}
```
We will leave the `startCamera()` method empty for now. When the user responds
to the prompt, the `MainActivity` will resume and `onResume()` will be called.
The code will confirm that permissions for using the camera have been granted,
and then will start the camera.
Rebuild and install the application. You should now see a prompt requesting
access to the camera for the application.
Note: If the there is no dialog prompt, uninstall and reinstall the application.
This may also happen if you haven't changed the `minSdkVersion` and
`targetSdkVersion` in the `AndroidManifest.xml` file.
### Camera Access
With camera permissions available, we can start and fetch frames from the
camera.
To view the frames from the camera we will use a [`SurfaceView`]. Each frame
from the camera will be stored in a [`SurfaceTexture`] object. To use these, we
first need to change the layout of our application.
Remove the entire [`TextView`] code block from
`$APPLICATION_PATH/res/layout/activity_main.xml` and add the following code
instead:
```
<FrameLayout
android:id="@+id/preview_display_layout"
android:layout_width="fill_parent"
android:layout_height="fill_parent"
android:layout_weight="1">
<TextView
android:id="@+id/no_camera_access_view"
android:layout_height="fill_parent"
android:layout_width="fill_parent"
android:gravity="center"
android:text="@string/no_camera_access" />
</FrameLayout>
```
This code block has a new [`FrameLayout`] named `preview_display_layout` and a
[`TextView`] nested inside it, named `no_camera_access_preview`. When camera
access permissions are not granted, our application will display the
[`TextView`] with a string message, stored in the variable `no_camera_access`.
Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file:
```
<string name="no_camera_access" translatable="false">Please grant camera permissions.</string>
```
When the user doesn't grant camera permission, the screen will now look like
this:
![missing_camera_permission_android](../images/mobile/missing_camera_permission_android.png)
Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
`MainActivity`:
```
private SurfaceTexture previewFrameTexture;
private SurfaceView previewDisplayView;
```
In the `onCreate(Bundle)` function, add the following two lines _before_
requesting camera permissions:
```
previewDisplayView = new SurfaceView(this);
setupPreviewDisplayView();
```
And now add the code defining `setupPreviewDisplayView()`:
```
private void setupPreviewDisplayView() {
previewDisplayView.setVisibility(View.GONE);
ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
viewGroup.addView(previewDisplayView);
}
```
We define a new [`SurfaceView`] object and add it to the
`preview_display_layout` [`FrameLayout`] object so that we can use it to display
the camera frames using a [`SurfaceTexture`] object named `previewFrameTexture`.
To use `previewFrameTexture` for getting camera frames, we will use [CameraX].
MediaPipe provides a utility named [`CameraXPreviewHelper`] to use [CameraX].
This class updates a listener when camera is started via
`onCameraStarted(@Nullable SurfaceTexture)`.
To use this utility, modify the `BUILD` file to add a dependency on
`"//mediapipe/java/com/google/mediapipe/components:android_camerax_helper"`.
Now import [`CameraXPreviewHelper`] and add the following line to
`MainActivity`:
```
private CameraXPreviewHelper cameraHelper;
```
Now, we can add our implementation to `startCamera()`:
```
public void startCamera() {
cameraHelper = new CameraXPreviewHelper();
cameraHelper.setOnCameraStartedListener(
surfaceTexture -> {
previewFrameTexture = surfaceTexture;
// Make the display view visible to start showing the preview.
previewDisplayView.setVisibility(View.VISIBLE);
});
}
```
This creates a new [`CameraXPreviewHelper`] object and adds an anonymous
listener on the object. When `cameraHelper` signals that the camera has started
and a `surfaceTexture` to grab frames is available, we save that
`surfaceTexture` as `previewFrameTexture`, and make the `previewDisplayView`
visible so that we can start seeing frames from the `previewFrameTexture`.
However, before starting the camera, we need to decide which camera we want to
use. [`CameraXPreviewHelper`] inherits from [`CameraHelper`] which provides two
options, `FRONT` and `BACK`. We can pass in the decision from the `BUILD` file
as metadata such that no code change is required to build a another version of
the app using a different camera.
Assuming we want to use `BACK` camera to perform edge detection on a live scene
that we view from the camera, add the metadata into `AndroidManifest.xml`:
```
...
<meta-data android:name="cameraFacingFront" android:value="${cameraFacingFront}"/>
</application>
</manifest>
```
and specify the selection in `BUILD` in the `helloworld` android binary rule
with a new entry in `manifest_values`:
```
manifest_values = {
"applicationId": "com.google.mediapipe.apps.basic",
"appName": "Hello World",
"mainActivity": ".MainActivity",
"cameraFacingFront": "False",
},
```
Now, in `MainActivity` to retrieve the metadata specified in `manifest_values`,
add an [`ApplicationInfo`] object:
```
private ApplicationInfo applicationInfo;
```
In the `onCreate()` function, add:
```
try {
applicationInfo =
getPackageManager().getApplicationInfo(getPackageName(), PackageManager.GET_META_DATA);
} catch (NameNotFoundException e) {
Log.e(TAG, "Cannot find application info: " + e);
}
```
Now add the following line at the end of the `startCamera()` function:
```
CameraHelper.CameraFacing cameraFacing =
applicationInfo.metaData.getBoolean("cameraFacingFront", false)
? CameraHelper.CameraFacing.FRONT
: CameraHelper.CameraFacing.BACK;
cameraHelper.startCamera(this, cameraFacing, /*surfaceTexture=*/ null);
```
At this point, the application should build successfully. However, when you run
the application on your device, you will see a black screen (even though camera
permissions have been granted). This is because even though we save the
`surfaceTexture` variable provided by the [`CameraXPreviewHelper`], the
`previewSurfaceView` doesn't use its output and display it on screen yet.
Since we want to use the frames in a MediaPipe graph, we will not add code to
view the camera output directly in this tutorial. Instead, we skip ahead to how
we can send camera frames for processing to a MediaPipe graph and display the
output of the graph on the screen.
## `ExternalTextureConverter` setup
A [`SurfaceTexture`] captures image frames from a stream as an OpenGL ES
texture. To use a MediaPipe graph, frames captured from the camera should be
stored in a regular Open GL texture object. MediaPipe provides a class,
[`ExternalTextureConverter`] to convert the image stored in a [`SurfaceTexture`]
object to a regular OpenGL texture object.
To use [`ExternalTextureConverter`], we also need an `EGLContext`, which is
created and managed by an [`EglManager`] object. Add a dependency to the `BUILD`
file to use [`EglManager`], `"//mediapipe/java/com/google/mediapipe/glutil"`.
In `MainActivity`, add the following declarations:
```
private EglManager eglManager;
private ExternalTextureConverter converter;
```
In the `onCreate(Bundle)` function, add a statement to initialize the
`eglManager` object before requesting camera permissions:
```
eglManager = new EglManager(null);
```
Recall that we defined the `onResume()` function in `MainActivity` to confirm
camera permissions have been granted and call `startCamera()`. Before this
check, add the following line in `onResume()` to initialize the `converter`
object:
```
converter = new ExternalTextureConverter(eglManager.getContext());
```
This `converter` now uses the `GLContext` managed by `eglManager`.
We also need to override the `onPause()` function in the `MainActivity` so that
if the application goes into a paused state, we close the `converter` properly:
```
@Override
protected void onPause() {
super.onPause();
converter.close();
}
```
To pipe the output of `previewFrameTexture` to the `converter`, add the
following block of code to `setupPreviewDisplayView()`:
```
previewDisplayView
.getHolder()
.addCallback(
new SurfaceHolder.Callback() {
@Override
public void surfaceCreated(SurfaceHolder holder) {}
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
// (Re-)Compute the ideal size of the camera-preview display (the area that the
// camera-preview frames get rendered onto, potentially with scaling and rotation)
// based on the size of the SurfaceView that contains the display.
Size viewSize = new Size(width, height);
Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
// Connect the converter to the camera-preview frames as its input (via
// previewFrameTexture), and configure the output width and height as the computed
// display size.
converter.setSurfaceTextureAndAttachToGLContext(
previewFrameTexture, displaySize.getWidth(), displaySize.getHeight());
}
@Override
public void surfaceDestroyed(SurfaceHolder holder) {}
});
```
In this code block, we add a custom [`SurfaceHolder.Callback`] to
`previewDisplayView` and implement the `surfaceChanged(SurfaceHolder holder, int
format, int width, int height)` function to compute an appropriate display size
of the camera frames on the device screen and to tie the `previewFrameTexture`
object and send frames of the computed `displaySize` to the `converter`.
We are now ready to use camera frames in a MediaPipe graph.
## Using a MediaPipe graph in Android
### Add relevant dependencies
To use a MediaPipe graph, we need to add dependencies to the MediaPipe framework
on Android. We will first add a build rule to build a `cc_binary` using JNI code
of the MediaPipe framework and then build a `cc_library` rule to use this binary
in our application. Add the following code block to your `BUILD` file:
```
cc_binary(
name = "libmediapipe_jni.so",
linkshared = 1,
linkstatic = 1,
deps = [
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
],
)
cc_library(
name = "mediapipe_jni_lib",
srcs = [":libmediapipe_jni.so"],
alwayslink = 1,
)
```
Add the dependency `":mediapipe_jni_lib"` to the `mediapipe_lib` build rule in
the `BUILD` file.
Next, we need to add dependencies specific to the MediaPipe graph we want to use
in the application.
First, add dependencies to all calculator code in the `libmediapipe_jni.so`
build rule:
```
"//mediapipe/graphs/edge_detection:mobile_calculators",
```
MediaPipe graphs are `.pbtxt` files, but to use them in the application, we need
to use the `mediapipe_binary_graph` build rule to generate a `.binarypb` file.
In the `helloworld` android binary build rule, add the `mediapipe_binary_graph`
target specific to the graph as an asset:
```
assets = [
"//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph",
],
assets_dir = "",
```
In the `assets` build rule, you can also add other assets such as TensorFlowLite
models used in your graph.
In addition, add additional `manifest_values` for properties specific to the
graph, to be later retrieved in `MainActivity`:
```
manifest_values = {
"applicationId": "com.google.mediapipe.apps.basic",
"appName": "Hello World",
"mainActivity": ".MainActivity",
"cameraFacingFront": "False",
"binaryGraphName": "mobile_gpu.binarypb",
"inputVideoStreamName": "input_video",
"outputVideoStreamName": "output_video",
},
```
Note that `binaryGraphName` indicates the filename of the binary graph,
determined by the `output_name` field in the `mediapipe_binary_graph` target.
`inputVideoStreamName` and `outputVideoStreamName` are the input and output
video stream name specified in the graph respectively.
Now, the `MainActivity` needs to load the MediaPipe framework. Also, the
framework uses OpenCV, so `MainActvity` should also load `OpenCV`. Use the
following code in `MainActivity` (inside the class, but not inside any function)
to load both dependencies:
```
static {
// Load all native libraries needed by the app.
System.loadLibrary("mediapipe_jni");
System.loadLibrary("opencv_java3");
}
```
### Use the graph in `MainActivity`
First, we need to load the asset which contains the `.binarypb` compiled from
the `.pbtxt` file of the graph. To do this, we can use a MediaPipe utility,
[`AndroidAssetUtil`].
Initialize the asset manager in `onCreate(Bundle)` before initializing
`eglManager`:
```
// Initialize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
// binary graphs.
AndroidAssetUtil.initializeNativeAssetManager(this);
```
Now, we need to setup a [`FrameProcessor`] object that sends camera frames
prepared by the `converter` to the MediaPipe graph and runs the graph, prepares
the output and then updates the `previewDisplayView` to display the output. Add
the following code to declare the `FrameProcessor`:
```
private FrameProcessor processor;
```
and initialize it in `onCreate(Bundle)` after initializing `eglManager`:
```
processor =
new FrameProcessor(
this,
eglManager.getNativeContext(),
applicationInfo.metaData.getString("binaryGraphName"),
applicationInfo.metaData.getString("inputVideoStreamName"),
applicationInfo.metaData.getString("outputVideoStreamName"));
```
The `processor` needs to consume the converted frames from the `converter` for
processing. Add the following line to `onResume()` after initializing the
`converter`:
```
converter.setConsumer(processor);
```
The `processor` should send its output to `previewDisplayView` To do this, add
the following function definitions to our custom [`SurfaceHolder.Callback`]:
```
@Override
public void surfaceCreated(SurfaceHolder holder) {
processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
}
@Override
public void surfaceDestroyed(SurfaceHolder holder) {
processor.getVideoSurfaceOutput().setSurface(null);
}
```
When the `SurfaceHolder` is created, we had the `Surface` to the
`VideoSurfaceOutput` of the `processor`. When it is destroyed, we remove it from
the `VideoSurfaceOutput` of the `processor`.
And that's it! You should now be able to successfully build and run the
application on the device and see Sobel edge detection running on a live camera
feed! Congrats!
![edge_detection_android_gpu_gif](../images/mobile/edge_detection_android_gpu.gif)
If you ran into any issues, please see the full code of the tutorial
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic).
[`ApplicationInfo`]:https://developer.android.com/reference/android/content/pm/ApplicationInfo
[`AndroidAssetUtil`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/framework/AndroidAssetUtil.java
[Bazel]:https://bazel.build/
[`CameraHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraHelper.java
[CameraX]:https://developer.android.com/training/camerax
[`CameraXPreviewHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraXPreviewHelper.java
[developer options]:https://developer.android.com/studio/debug/dev-options
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
[`EglManager`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/glutil/EglManager.java
[`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java
[`FrameLayout`]:https://developer.android.com/reference/android/widget/FrameLayout
[`FrameProcessor`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/FrameProcessor.java
[MediaPipe installation guide]:./install.md
[`PermissionHelper`]: https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/PermissionHelper.java
[`SurfaceHolder.Callback`]:https://developer.android.com/reference/android/view/SurfaceHolder.Callback.html
[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
[`SurfaceTexture`]:https://developer.android.com/reference/android/graphics/SurfaceTexture
[`TextView`]:https://developer.android.com/reference/android/widget/TextView
docs/getting_started/hello_world_desktop.md
+128
View File
@@ -0,0 +1,128 @@
---
layout: default
title: Hello World! on Desktop (C++)
parent: Getting Started
nav_order: 5
---
# Hello World! on Desktop (C++)
{: .no_toc }
1. TOC
{:toc}
---
1. Ensure you have a working version of MediaPipe. See
[installation instructions](./install.md).
2. To run the [`hello world`] example:
```bash
$ git clone https://github.com/google/mediapipe/mediapipe.git
$ cd mediapipe
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is not supported currently.
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# It should print 10 rows of Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
3. The [`hello world`] example uses a simple MediaPipe graph in the
`PrintHelloWorld()` function, defined in a [`CalculatorGraphConfig`] proto.
```C++
::mediapipe::Status PrintHelloWorld() {
// Configures a simple graph, which concatenates 2 PassThroughCalculators.
CalculatorGraphConfig config = ParseTextProtoOrDie<CalculatorGraphConfig>(R"(
input_stream: "in"
output_stream: "out"
node {
calculator: "PassThroughCalculator"
input_stream: "in"
output_stream: "out1"
}
node {
calculator: "PassThroughCalculator"
input_stream: "out1"
output_stream: "out"
}
)");
```
You can visualize this graph using
[MediaPipe Visualizer](https://viz.mediapipe.dev) by pasting the
CalculatorGraphConfig content below into the visualizer. See
[here](../tools/visualizer.md) for help on the visualizer.
```bash
input_stream: "in"
output_stream: "out"
node {
calculator: "PassThroughCalculator"
input_stream: "in"
output_stream: "out1"
}
node {
calculator: "PassThroughCalculator"
input_stream: "out1"
output_stream: "out"
}
```
This graph consists of 1 graph input stream (`in`) and 1 graph output stream
(`out`), and 2 [`PassThroughCalculator`]s connected serially.
![hello_world graph](../images/hello_world.png)
4. Before running the graph, an `OutputStreamPoller` object is connected to the
output stream in order to later retrieve the graph output, and a graph run
is started with [`StartRun`].
```c++
CalculatorGraph graph;
RETURN_IF_ERROR(graph.Initialize(config));
ASSIGN_OR_RETURN(OutputStreamPoller poller,
graph.AddOutputStreamPoller("out"));
RETURN_IF_ERROR(graph.StartRun({}));
```
5. The example then creates 10 packets (each packet contains a string "Hello
World!" with Timestamp values ranging from 0, 1, ... 9) using the
[`MakePacket`] function, adds each packet into the graph through the `in`
input stream, and finally closes the input stream to finish the graph run.
```c++
for (int i = 0; i < 10; ++i) {
RETURN_IF_ERROR(graph.AddPacketToInputStream("in", MakePacket<std::string>("Hello World!").At(Timestamp(i))));
}
RETURN_IF_ERROR(graph.CloseInputStream("in"));
```
6. Through the `OutputStreamPoller` object the example then retrieves all 10
packets from the output stream, gets the string content out of each packet
and prints it to the output log.
```c++
mediapipe::Packet packet;
while (poller.Next(&packet)) {
LOG(INFO) << packet.Get<string>();
}
```
[`hello world`]: https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hello_world/hello_world.cc
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`PassThroughCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/pass_through_calculator.cc
[`MakePacket`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/packet.h
[`StartRun`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
docs/getting_started/hello_world_ios.md
+560
View File
@@ -0,0 +1,560 @@
---
layout: default
title: Hello World! on iOS
parent: Getting Started
nav_order: 4
---
# Hello World! on iOS
{: .no_toc }
1. TOC
{:toc}
---
## Introduction
This codelab uses MediaPipe on an iOS device.
### What you will learn
How to develop an iOS application that uses MediaPipe and run a MediaPipe
graph on iOS.
### What you will build
A simple camera app for real-time Sobel edge detection applied to a live video
stream on an iOS device.
![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif)
## Setup
1. Install MediaPipe on your system, see [MediaPipe installation guide] for
details.
2. Setup your iOS device for development.
3. Setup [Bazel] on your system to build and deploy the iOS app.
## Graph for edge detection
We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
```
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
# Used in the examples
# mediapipe/examples/android/src/java/com/mediapipe/apps/edgedetectiongpu.
# mediapipe/examples/ios/edgedetectiongpu.
# Images coming into and out of the graph.
input_stream: "input_video"
output_stream: "output_video"
# Converts RGB images into luminance images, still stored in RGB format.
node: {
calculator: "LuminanceCalculator"
input_stream: "input_video"
output_stream: "luma_video"
}
# Applies the Sobel filter to luminance images sotred in RGB format.
node: {
calculator: "SobelEdgesCalculator"
input_stream: "luma_video"
output_stream: "output_video"
}
```
A visualization of the graph is shown below:
![edge_detection_mobile_gpu](../images/mobile/edge_detection_mobile_gpu.png)
This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera.
The first node in the graph, `LuminanceCalculator`, takes a single packet (image
frame) and applies a change in luminance using an OpenGL shader. The resulting
image frame is sent to the `luma_video` output stream.
The second node, `SobelEdgesCalculator` applies edge detection to incoming
packets in the `luma_video` stream and outputs results in `output_video` output
stream.
Our iOS application will display the output image frames of the `output_video`
stream.
## Initial minimal application setup
We first start with a simple iOS application and demonstrate how to use `bazel`
to build it.
First, create an XCode project via File > New > Single View App.
Set the product name to "EdgeDetectionGpu", and use an appropriate organization
identifier, such as `com.google.mediapipe`. The organization identifier
alongwith the product name will be the `bundle_id` for the application, such as
`com.google.mediapipe.EdgeDetectionGpu`.
Set the language to Objective-C.
Save the project to an appropriate location. Let's call this
`$PROJECT_TEMPLATE_LOC`. So your project will be in the
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu` directory. This directory will contain
another directory named `EdgeDetectionGpu` and an `EdgeDetectionGpu.xcodeproj` file.
The `EdgeDetectionGpu.xcodeproj` will not be useful for this tutorial, as we will
use bazel to build the iOS application. The content of the
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu/EdgeDetectionGpu` directory is listed below:
1. `AppDelegate.h` and `AppDelegate.m`
2. `ViewController.h` and `ViewController.m`
3. `main.m`
4. `Info.plist`
5. `Main.storyboard` and `Launch.storyboard`
6. `Assets.xcassets` directory.
Copy these files to a directory named `EdgeDetectionGpu` to a location that can
access the MediaPipe source code. For example, the source code of the
application that we will build in this tutorial is located in
`mediapipe/examples/ios/EdgeDetectionGpu`. We will refer to this path as the
`$APPLICATION_PATH` throughout the codelab.
Note: MediaPipe provides Objective-C bindings for iOS. The edge detection
application in this tutorial and all iOS examples using MediaPipe use
Objective-C with C++ in `.mm` files.
Create a `BUILD` file in the `$APPLICATION_PATH` and add the following build
rules:
```
MIN_IOS_VERSION = "10.0"
load(
"@build_bazel_rules_apple//apple:ios.bzl",
"ios_application",
)
ios_application(
name = "EdgeDetectionGpuApp",
bundle_id = "com.google.mediapipe.EdgeDetectionGpu",
families = [
"iphone",
"ipad",
],
infoplists = ["Info.plist"],
minimum_os_version = MIN_IOS_VERSION,
provisioning_profile = "//mediapipe/examples/ios:developer_provisioning_profile",
deps = [":EdgeDetectionGpuAppLibrary"],
)
objc_library(
name = "EdgeDetectionGpuAppLibrary",
srcs = [
"AppDelegate.m",
"ViewController.m",
"main.m",
],
hdrs = [
"AppDelegate.h",
"ViewController.h",
],
data = [
"Base.lproj/LaunchScreen.storyboard",
"Base.lproj/Main.storyboard",
],
sdk_frameworks = [
"UIKit",
],
deps = [],
)
```
The `objc_library` rule adds dependencies for the `AppDelegate` and
`ViewController` classes, `main.m` and the application storyboards. The
templated app depends only on the `UIKit` SDK.
The `ios_application` rule uses the `EdgeDetectionGpuAppLibrary` Objective-C
library generated to build an iOS application for installation on your iOS
device.
Note: You need to point to your own iOS developer provisioning profile to be
able to run the application on your iOS device.
To build the app, use the following command in a terminal:
```
bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
```
For example, to build the `EdgeDetectionGpuApp` application in
`mediapipe/examples/ios/edgedetectiongpu`, use the following
command:
```
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp
```
Then, go back to XCode, open Window > Devices and Simulators, select your
device, and add the `.ipa` file generated by the command above to your device.
Here is the document on [setting up and compiling](./building_examples.md#ios) iOS
MediaPipe apps.
Open the application on your device. Since it is empty, it should display a
blank white screen.
## Use the camera for the live view feed
In this tutorial, we will use the `MPPCameraInputSource` class to access and
grab frames from the camera. This class uses the `AVCaptureSession` API to get
the frames from the camera.
But before using this class, change the `Info.plist` file to support camera
usage in the app.
In `ViewController.m`, add the following import line:
```
#import "mediapipe/objc/MPPCameraInputSource.h"
```
Add the following to its implementation block to create an object
`_cameraSource`:
```
@implementation ViewController {
// Handles camera access via AVCaptureSession library.
MPPCameraInputSource* _cameraSource;
}
```
Add the following code to `viewDidLoad()`:
```
-(void)viewDidLoad {
[super viewDidLoad];
_cameraSource = [[MPPCameraInputSource alloc] init];
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
}
```
The code initializes `_cameraSource`, sets the capture session preset, and which
camera to use.
We need to get frames from the `_cameraSource` into our application
`ViewController` to display them. `MPPCameraInputSource` is a subclass of
`MPPInputSource`, which provides a protocol for its delegates, namely the
`MPPInputSourceDelegate`. So our application `ViewController` can be a delegate
of `_cameraSource`.
To handle camera setup and process incoming frames, we should use a queue
different from the main queue. Add the following to the implementation block of
the `ViewController`:
```
// Process camera frames on this queue.
dispatch_queue_t _videoQueue;
```
In `viewDidLoad()`, add the following line after initializing the
`_cameraSource` object:
```
[_cameraSource setDelegate:self queue:_videoQueue];
```
And add the following code to initialize the queue before setting up the
`_cameraSource` object:
```
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
_videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
```
We will use a serial queue with the priority `QOS_CLASS_USER_INTERACTIVE` for
processing camera frames.
Add the following line after the header imports at the top of the file, before
the interface/implementation of the `ViewController`:
```
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
```
Before implementing any method from `MPPInputSourceDelegate` protocol, we must
first set up a way to display the camera frames. MediaPipe provides another
utility called `MPPLayerRenderer` to display images on the screen. This utility
can be used to display `CVPixelBufferRef` objects, which is the type of the
images provided by `MPPCameraInputSource` to its delegates.
To display images of the screen, we need to add a new `UIView` object called
`_liveView` to the `ViewController`.
Add the following lines to the implementation block of the `ViewController`:
```
// Display the camera preview frames.
IBOutlet UIView* _liveView;
// Render frames in a layer.
MPPLayerRenderer* _renderer;
```
Go to `Main.storyboard`, add a `UIView` object from the object library to the
`View` of the `ViewController` class. Add a referencing outlet from this view to
the `_liveView` object you just added to the `ViewController` class. Resize the
view so that it is centered and covers the entire application screen.
Go back to `ViewController.m` and add the following code to `viewDidLoad()` to
initialize the `_renderer` object:
```
_renderer = [[MPPLayerRenderer alloc] init];
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
```
To get frames from the camera, we will implement the following method:
```
// Must be invoked on _videoQueue.
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MPPInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
}
// Display the captured image on the screen.
CFRetain(imageBuffer);
dispatch_async(dispatch_get_main_queue(), ^{
[_renderer renderPixelBuffer:imageBuffer];
CFRelease(imageBuffer);
});
}
```
This is a delegate method of `MPPInputSource`. We first check that we are
getting frames from the right source, i.e. the `_cameraSource`. Then we display
the frame received from the camera via `_renderer` on the main queue.
Now, we need to start the camera as soon as the view to display the frames is
about to appear. To do this, we will implement the
`viewWillAppear:(BOOL)animated` function:
```
-(void)viewWillAppear:(BOOL)animated {
[super viewWillAppear:animated];
}
```
Before we start running the camera, we need the user's permission to access it.
`MPPCameraInputSource` provides a function
`requestCameraAccessWithCompletionHandler:(void (^_Nullable)(BOOL
granted))handler` to request camera access and do some work when the user has
responded. Add the following code to `viewWillAppear:animated`:
```
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
if (granted) {
dispatch_async(_videoQueue, ^{
[_cameraSource start];
});
}
}];
```
Before building the application, add the following dependencies to your `BUILD`
file:
```
sdk_frameworks = [
"AVFoundation",
"CoreGraphics",
"CoreMedia",
],
deps = [
"//mediapipe/objc:mediapipe_framework_ios",
"//mediapipe/objc:mediapipe_input_sources_ios",
"//mediapipe/objc:mediapipe_layer_renderer",
],
```
Now build and run the application on your iOS device. You should see a live
camera view feed after accepting camera permissions.
We are now ready to use camera frames in a MediaPipe graph.
## Using a MediaPipe graph in iOS
### Add relevant dependencies
We already added the dependencies of the MediaPipe framework code which contains
the iOS API to use a MediaPipe graph. To use a MediaPipe graph, we need to add a
dependency on the graph we intend to use in our application. Add the following
line to the `data` list in your `BUILD` file:
```
"//mediapipe/graphs/edge_detection:mobile_gpu_binary_graph",
```
Now add the dependency to the calculators used in this graph in the `deps` field
in the `BUILD` file:
```
"//mediapipe/graphs/edge_detection:mobile_calculators",
```
Finally, rename the file `ViewController.m` to `ViewController.mm` to support
Objective-C++.
### Use the graph in `ViewController`
Declare a static constant with the name of the graph, the input stream and the
output stream:
```
static NSString* const kGraphName = @"mobile_gpu";
static const char* kInputStream = "input_video";
static const char* kOutputStream = "output_video";
```
Add the following property to the interface of the `ViewController`:
```
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
// sent video frames on _videoQueue.
@property(nonatomic) MPPGraph* mediapipeGraph;
```
As explained in the comment above, we will initialize this graph in
`viewDidLoad` first. To do so, we need to load the graph from the `.pbtxt` file
using the following function:
```
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
// Load the graph config resource.
NSError* configLoadError = nil;
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
if (!resource || resource.length == 0) {
return nil;
}
NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
if (!data) {
NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
return nil;
}
// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
mediapipe::CalculatorGraphConfig config;
config.ParseFromArray(data.bytes, data.length);
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MPPPacketTypePixelBuffer];
return newGraph;
}
```
Use this function to initialize the graph in `viewDidLoad` as follows:
```
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
```
The graph should send the results of processing camera frames back to the
`ViewController`. Add the following line after initializing the graph to set the
`ViewController` as a delegate of the `mediapipeGraph` object:
```
self.mediapipeGraph.delegate = self;
```
To avoid memory contention while processing frames from the live video feed, add
the following line:
```
// Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
self.mediapipeGraph.maxFramesInFlight = 2;
```
Now, start the graph when the user has granted the permission to use the camera
in our app:
```
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
if (granted) {
// Start running self.mediapipeGraph.
NSError* error;
if (![self.mediapipeGraph startWithError:&error]) {
NSLog(@"Failed to start graph: %@", error);
}
dispatch_async(_videoQueue, ^{
[_cameraSource start];
});
}
}];
```
Note: It is important to start the graph before starting the camera, so that
the graph is ready to process frames as soon as the camera starts sending them.
Earlier, when we received frames from the camera in the `processVideoFrame`
function, we displayed them in the `_liveView` using the `_renderer`. Now, we
need to send those frames to the graph and render the results instead. Modify
this function's implementation to do the following:
```
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MPPInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
}
[self.mediapipeGraph sendPixelBuffer:imageBuffer
intoStream:kInputStream
packetType:MPPPacketTypePixelBuffer];
}
```
We send the `imageBuffer` to `self.mediapipeGraph` as a packet of type
`MPPPacketTypePixelBuffer` into the input stream `kInputStream`, i.e.
"input_video".
The graph will run with this input packet and output a result in
`kOutputStream`, i.e. "output_video". We can implement the following delegate
method to receive packets on this output stream and display them on the screen:
```
- (void)mediapipeGraph:(MPPGraph*)graph
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
fromStream:(const std::string&)streamName {
if (streamName == kOutputStream) {
// Display the captured image on the screen.
CVPixelBufferRetain(pixelBuffer);
dispatch_async(dispatch_get_main_queue(), ^{
[_renderer renderPixelBuffer:pixelBuffer];
CVPixelBufferRelease(pixelBuffer);
});
}
}
```
And that is all! Build and run the app on your iOS device. You should see the
results of running the edge detection graph on a live video feed. Congrats!
![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif)
If you ran into any issues, please see the full code of the tutorial
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
[Bazel]:https://bazel.build/
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
[MediaPipe installation guide]:./install.md
docs/getting_started/help.md
+48
View File
@@ -0,0 +1,48 @@
---
layout: default
title: Getting Help
parent: Getting Started
nav_order: 8
---
# Getting Help
{: .no_toc }
1. TOC
{:toc}
---
## Technical questions
For help with technical or algorithmic questions, visit
[Stack Overflow](https://stackoverflow.com/questions/tagged/mediapipe) to find
answers and support from the MediaPipe community.
## Bugs and feature requests
To report bugs or make feature requests,
[file an issue on GitHub](https://github.com/google/mediapipe/issues).
If you open a GitHub issue, here is our policy:
1. It must be a bug, a feature request, or a significant problem with documentation (for small doc fixes please send a PR instead).
2. The form below must be filled out.
**Here's why we have that policy**: MediaPipe developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.
------------------------
### System information
- **Have I written custom code**:
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
- **Bazel version**:
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev enviroment)**:
- **Xcode & Tulsi version (if issue is related to building in mobile dev enviroment)**:
- **Exact steps to reproduce**:
### Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in MediaPipe or a feature request.
### Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached instead of being pasted into the issue as text.
docs/getting_started/install.md
+675
View File
@@ -0,0 +1,675 @@
---
layout: default
title: Installation
parent: Getting Started
nav_order: 1
---
# Installation
{: .no_toc }
1. TOC
{:toc}
---
Note: To interoperate with OpenCV, OpenCV 3.x and above are preferred. OpenCV
2.x currently works but interoperability support may be deprecated in the
future.
Note: If you plan to use TensorFlow calculators and example apps, there is a
known issue with gcc and g++ version 6.3 and 7.3. Please use other versions.
Note: To make Mediapipe work with TensorFlow, please set Python 3.7 as the
default Python version and install the Python "six" library by running `pip3
install --user six`.
Note: To build and run Android example apps, see these
[instructions](./building_examples.md#android). To build and run iOS example
apps, see these [instructions](./building_examples.md#ios).
## Installing on Debian and Ubuntu
1. Checkout MediaPipe repository.
```bash
$ git clone https://github.com/google/mediapipe.git
# Change directory into MediaPipe root directory
$ cd mediapipe
```
2. Install Bazel.
Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
to install Bazel 2.0 or higher.
3. Install OpenCV and FFmpeg.
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries. FFmpeg will be installed via libopencv-video-dev.
Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to take
option 2 or 3 to install OpenCV 3 or above.
```bash
$ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \
libopencv-calib3d-dev libopencv-features2d-dev \
libopencv-imgproc-dev libopencv-video-dev
```
Option 2. Run [`setup_opencv.sh`] to automatically build OpenCV from source
and modify MediaPipe's OpenCV config.
Option 3. Follow OpenCV's
[documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
to manually build OpenCV from source code.
Note: You may need to modify [`WORKSPACE`] and [`opencv_linux.BUILD`] to
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
rule in [`WORKSPACE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
like the following:
```bash
new_local_repository(
name = "linux_opencv",
build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr/local",
)
cc_library(
name = "opencv",
srcs = glob(
[
"lib/libopencv_core.so",
"lib/libopencv_highgui.so",
"lib/libopencv_imgcodecs.so",
"lib/libopencv_imgproc.so",
"lib/libopencv_video.so",
"lib/libopencv_videoio.so",
],
),
hdrs = glob(["include/opencv4/**/*.h*"]),
includes = ["include/opencv4/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
```
4. For running desktop examples on Linux only (not on OS X) with GPU
acceleration.
```bash
# Requires a GPU with EGL driver support.
# Can use mesa GPU libraries for desktop, (or Nvidia/AMD equivalent).
sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
# To compile with GPU support, replace
--define MEDIAPIPE_DISABLE_GPU=1
# with
--copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11
# when building GPU examples.
```
5. Run the [Hello World desktop example](./hello_world_desktop.md).
```bash
$ export GLOG_logtostderr=1
# if you are running on Linux desktop with CPU only
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# If you are running on Linux desktop with GPU support enabled (via mesa drivers)
$ bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
## Installing on CentOS
1. Checkout MediaPipe repository.
```bash
$ git clone https://github.com/google/mediapipe.git
# Change directory into MediaPipe root directory
$ cd mediapipe
```
2. Install Bazel.
Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-redhat.html)
to install Bazel 2.0 or higher.
3. Install OpenCV.
Option 1. Use package manager tool to install the pre-compiled version.
Note: yum installs OpenCV 2.4.5, which may have an opencv/gstreamer
[issue](https://github.com/opencv/opencv/issues/4592).
```bash
$ sudo yum install opencv-devel
```
Option 2. Build OpenCV from source code.
Note: You may need to modify [`WORKSPACE`] and [`opencv_linux.BUILD`] to
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
rule in [`WORKSPACE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
like the following:
```bash
new_local_repository(
name = "linux_opencv",
build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr/local",
)
cc_library(
name = "opencv",
srcs = glob(
[
"lib/libopencv_core.so",
"lib/libopencv_highgui.so",
"lib/libopencv_imgcodecs.so",
"lib/libopencv_imgproc.so",
"lib/libopencv_video.so",
"lib/libopencv_videoio.so",
],
),
hdrs = glob(["include/opencv4/**/*.h*"]),
includes = ["include/opencv4/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
```
4. Run the [Hello World desktop example](./hello_world_desktop.md).
```bash
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' if you are running on Linux desktop with CPU only
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
## Installing on macOS
1. Prework:
* Install [Homebrew](https://brew.sh).
* Install [Xcode](https://developer.apple.com/xcode/) and its Command Line
Tools by `xcode-select --install`.
2. Checkout MediaPipe repository.
```bash
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
```
3. Install Bazel.
Option 1. Use package manager tool to install Bazel
```bash
$ brew install bazel
# Run 'bazel version' to check version of bazel
```
Option 2. Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-os-x.html#install-with-installer-mac-os-x)
to install Bazel 2.0 or higher.
4. Install OpenCV and FFmpeg.
Option 1. Use HomeBrew package manager tool to install the pre-compiled
OpenCV 3.4.5 libraries. FFmpeg will be installed via OpenCV.
```bash
$ brew install opencv@3
# There is a known issue caused by the glog dependency. Uninstall glog.
$ brew uninstall --ignore-dependencies glog
```
Option 2. Use MacPorts package manager tool to install the OpenCV libraries.
```bash
$ port install opencv
```
Note: when using MacPorts, please edit the [`WORKSPACE`],
[`opencv_macos.BUILD`], and [`ffmpeg_macos.BUILD`] files like the following:
```bash
new_local_repository(
name = "macos_opencv",
build_file = "@//third_party:opencv_macos.BUILD",
path = "/opt",
)
new_local_repository(
name = "macos_ffmpeg",
build_file = "@//third_party:ffmpeg_macos.BUILD",
path = "/opt",
)
cc_library(
name = "opencv",
srcs = glob(
[
"local/lib/libopencv_core.dylib",
"local/lib/libopencv_highgui.dylib",
"local/lib/libopencv_imgcodecs.dylib",
"local/lib/libopencv_imgproc.dylib",
"local/lib/libopencv_video.dylib",
"local/lib/libopencv_videoio.dylib",
],
),
hdrs = glob(["local/include/opencv2/**/*.h*"]),
includes = ["local/include/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
cc_library(
name = "libffmpeg",
srcs = glob(
[
"local/lib/libav*.dylib",
],
),
hdrs = glob(["local/include/libav*/*.h"]),
includes = ["local/include/"],
linkopts = [
"-lavcodec",
"-lavformat",
"-lavutil",
],
linkstatic = 1,
visibility = ["//visibility:public"],
)
```
5. Make sure that Python 3 and the Python "six" library are installed.
```
$ brew install python
$ sudo ln -s -f /usr/local/bin/python3.7 /usr/local/bin/python
$ python --version
Python 3.7.4
$ pip3 install --user six
```
6. Run the [Hello World desktop example](./hello_world_desktop.md).
```bash
$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
## Installing on Windows
**Disclaimer**: Running MediaPipe on Windows is experimental.
Note: building MediaPipe Android apps is still not possible on native
Windows. Please do this in WSL instead and see the WSL setup instruction in the
next section.
1. Install [MSYS2](https://www.msys2.org/) and edit the `%PATH%` environment
variable.
If MSYS2 is installed to `C:\msys64`, add `C:\msys64\usr\bin` to your
`%PATH%` environment variable.
2. Install necessary packages.
```
C:\> pacman -S git patch unzip
```
3. Install Python and allow the executable to edit the `%PATH%` environment
variable.
Download Python Windows executable from
https://www.python.org/downloads/windows/ and install.
4. Install Visual C++ Build Tools 2019 and WinSDK
Go to https://visualstudio.microsoft.com/visual-cpp-build-tools, download
build tools, and install Microsoft Visual C++ 2019 Redistributable and
Microsoft Build Tools 2019.
Download the WinSDK from
https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/ and
install.
5. Install Bazel and add the location of the Bazel executable to the `%PATH%`
environment variable.
Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-windows.html)
to install Bazel 2.0 or higher.
6. Set Bazel variables.
```
# Find the exact paths and version numbers from your local version.
C:\> set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools
C:\> set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC
C:\> set BAZEL_VC_FULL_VERSION=14.25.28610
C:\> set BAZEL_WINSDK_FULL_VERSION=10.1.18362.1
```
7. Checkout MediaPipe repository.
```
C:\Users\Username\mediapipe_repo> git clone https://github.com/google/mediapipe.git
# Change directory into MediaPipe root directory
C:\Users\Username\mediapipe_repo> cd mediapipe
```
8. Install OpenCV.
Download the Windows executable from https://opencv.org/releases/ and
install. We currently use OpenCV 3.4.10. Remember to edit the [`WORKSPACE`]
file if OpenCV is not installed at `C:\opencv`.
```
new_local_repository(
name = "windows_opencv",
build_file = "@//third_party:opencv_windows.BUILD",
path = "C:\\<path to opencv>\\build",
)
```
9. Run the [Hello World desktop example](./hello_world_desktop.md).
Note: For building MediaPipe on Windows, please add `--action_env
PYTHON_BIN_PATH="C:/path/to/python.exe"` to the build command.
Alternatively, you can follow
[issue 724](https://github.com/google/mediapipe/issues/724) to fix the
python configuration manually.
```
C:\Users\Username\mediapipe_repo>bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH="C:/python_36/python.exe" mediapipe/examples/desktop/hello_world
C:\Users\Username\mediapipe_repo>set GLOG_logtostderr=1
C:\Users\Username\mediapipe_repo>bazel-bin\mediapipe\examples\desktop\hello_world\hello_world.exe
# should print:
# I20200514 20:43:12.277598 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.278597 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.279618 1200 hello_world.cc:56] Hello World!
# I20200514 20:43:12.280613 1200 hello_world.cc:56] Hello World!
```
## Installing on Windows Subsystem for Linux (WSL)
Note: The pre-built OpenCV packages don't support cameras in WSL. Unless you
[compile](https://funvision.blogspot.com/2019/12/opencv-web-camera-and-video-streams-in.html)
OpenCV with FFMPEG and GStreamer in WSL, the live demos won't work with any
cameras. Alternatively, you use a video file as input.
1. Follow the
[instruction](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to
install Windows Sysystem for Linux (Ubuntu).
2. Install Windows ADB and start the ADB server in Windows.
Note: Windows' and WSL’s adb versions must be the same version, e.g., if WSL
has ADB 1.0.39, you need to download the corresponding Windows ADB from
[here](https://dl.google.com/android/repository/platform-tools_r26.0.1-windows.zip).
3. Launch WSL.
Note: All the following steps will be executed in WSL. The Windows directory
of the Linux Subsystem can be found in
C:\Users\YourUsername\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_SomeID\LocalState\rootfs\home
4. Install the needed packages.
```bash
username@DESKTOP-TMVLBJ1:~$ sudo apt-get update && sudo apt-get install -y build-essential git python zip adb openjdk-8-jdk
```
5. Install Bazel.
```bash
username@DESKTOP-TMVLBJ1:~$ curl -sLO --retry 5 --retry-max-time 10 \
https://storage.googleapis.com/bazel/2.0.0/release/bazel-2.0.0-installer-linux-x86_64.sh && \
sudo mkdir -p /usr/local/bazel/2.0.0 && \
chmod 755 bazel-2.0.0-installer-linux-x86_64.sh && \
sudo ./bazel-2.0.0-installer-linux-x86_64.sh --prefix=/usr/local/bazel/2.0.0 && \
source /usr/local/bazel/2.0.0/lib/bazel/bin/bazel-complete.bash
username@DESKTOP-TMVLBJ1:~$ /usr/local/bazel/2.0.0/lib/bazel/bin/bazel version && \
alias bazel='/usr/local/bazel/2.0.0/lib/bazel/bin/bazel'
```
6. Checkout MediaPipe repository.
```bash
username@DESKTOP-TMVLBJ1:~$ git clone https://github.com/google/mediapipe.git
username@DESKTOP-TMVLBJ1:~$ cd mediapipe
```
7. Install OpenCV and FFmpeg.
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries. FFmpeg will be installed via libopencv-video-dev.
```bash
username@DESKTOP-TMVLBJ1:~/mediapipe$ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \
libopencv-calib3d-dev libopencv-features2d-dev \
libopencv-imgproc-dev libopencv-video-dev
```
Option 2. Run [`setup_opencv.sh`] to automatically build OpenCV from source
and modify MediaPipe's OpenCV config.
Option 3. Follow OpenCV's
[documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
to manually build OpenCV from source code.
Note: You may need to modify [`WORKSPACE`] and [`opencv_linux.BUILD`] to
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
rule in [`WORKSPACE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
like the following:
```bash
new_local_repository(
name = "linux_opencv",
build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr/local",
)
cc_library(
name = "opencv",
srcs = glob(
[
"lib/libopencv_core.so",
"lib/libopencv_highgui.so",
"lib/libopencv_imgcodecs.so",
"lib/libopencv_imgproc.so",
"lib/libopencv_video.so",
"lib/libopencv_videoio.so",
],
),
hdrs = glob(["include/opencv4/**/*.h*"]),
includes = ["include/opencv4/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
```
8. Run the [Hello World desktop example](./hello_world_desktop.md).
```bash
username@DESKTOP-TMVLBJ1:~/mediapipe$ export GLOG_logtostderr=1
# Need bazel flag 'MEDIAPIPE_DISABLE_GPU=1' as desktop GPU is currently not supported
username@DESKTOP-TMVLBJ1:~/mediapipe$ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
# Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
## Installing using Docker
This will use a Docker image that will isolate mediapipe's installation from the rest of the system.
1. [Install Docker](https://docs.docker.com/install/#supported-platforms) on
your host system.
2. Build a docker image with tag "mediapipe".
```bash
$ git clone https://github.com/google/mediapipe.git
$ cd mediapipe
$ docker build --tag=mediapipe .
# Should print:
# Sending build context to Docker daemon 147.8MB
# Step 1/9 : FROM ubuntu:latest
# latest: Pulling from library/ubuntu
# 6abc03819f3e: Pull complete
# 05731e63f211: Pull complete
# ........
# See http://bazel.build/docs/getting-started.html to start a new project!
# Removing intermediate container 82901b5e79fa
# ---> f5d5f402071b
# Step 9/9 : COPY . /mediapipe/
# ---> a95c212089c5
# Successfully built a95c212089c5
# Successfully tagged mediapipe:latest
```
3. Run the [Hello World desktop example](./hello_world_desktop.md).
```bash
$ docker run -it --name mediapipe mediapipe:latest
root@bca08b91ff63:/mediapipe# GLOG_logtostderr=1 bazel run --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world:hello_world
# Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
4. Build a MediaPipe Android example.
```bash
$ docker run -it --name mediapipe mediapipe:latest
root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk.sh
# Should print:
# Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r18b
# Set android_ndk_repository and android_sdk_repository in WORKSPACE
# Done
root@bca08b91ff63:/mediapipe# bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu
# Should print:
# Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu up-to-date:
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu_deploy.jar
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu_unsigned.apk
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
# INFO: Elapsed time: 144.462s, Critical Path: 79.47s
# INFO: 1958 processes: 1 local, 1863 processwrapper-sandbox, 94 worker.
# INFO: Build completed successfully, 2028 total actions
```
<!-- 5. Uncomment the last line of the Dockerfile
```bash
RUN bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/demo:object_detection_tensorflow_demo
```
and rebuild the image and then run the docker image
```bash
docker build --tag=mediapipe .
docker run -i -t mediapipe:latest
``` -->
[`WORKSPACE`]: https://github.com/google/mediapipe/tree/master/WORKSPACE
[`opencv_linux.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_linux.BUILD
[`opencv_macos.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_macos.BUILD
[`ffmpeg_macos.BUILD`]:https://github.com/google/mediapipe/tree/master/third_party/ffmpeg_macos.BUILD
[`setup_opencv.sh`]: https://github.com/google/mediapipe/tree/master/setup_opencv.sh
docs/getting_started/troubleshooting.md
+149
View File
@@ -0,0 +1,149 @@
---
layout: default
title: Troubleshooting
parent: Getting Started
nav_order: 10
---
# Troubleshooting
{: .no_toc }
1. TOC
{:toc}
---
## Native method not found
The error message:
```
java.lang.UnsatisfiedLinkError: No implementation found for void com.google.wick.Wick.nativeWick
```
usually indicates that a needed native library, such as `/libwickjni.so` has not
been loaded or has not been included in the dependencies of the app or cannot be
found for some reason. Note that Java requires every native library to be
explicitly loaded using the function `System.loadLibrary`.
## No registered calculator found
The error message:
```
No registered object with name: OurNewCalculator; Unable to find Calculator "OurNewCalculator"
```
usually indicates that `OurNewCalculator` is referenced by name in a
[`CalculatorGraphConfig`] but that the library target for OurNewCalculator has
not been linked to the application binary. When a new calculator is added to a
calculator graph, that calculator must also be added as a build dependency of
the applications using the calculator graph.
This error is caught at runtime because calculator graphs reference their
calculators by name through the field `CalculatorGraphConfig::Node:calculator`.
When the library for a calculator is linked into an application binary, the
calculator is automatically registered by name through the
[`REGISTER_CALCULATOR`] macro using the [`registration.h`] library. Note that
[`REGISTER_CALCULATOR`] can register a calculator with a namespace prefix,
identical to its C++ namespace. In this case, the calculator graph must also use
the same namespace prefix.
## Out Of Memory error
Exhausting memory can be a symptom of too many packets accumulating inside a
running MediaPipe graph. This can occur for a number of reasons, such as:
1. Some calculators in the graph simply can't keep pace with the arrival of
packets from a realtime input stream such as a video camera.
2. Some calculators are waiting for packets that will never arrive.
For problem (1), it may be necessary to drop some old packets in older to
process the more recent packets. For some hints, see:
[`How to process realtime input streams`].
For problem (2), it could be that one input stream is lacking packets for some
reason. A device or a calculator may be misconfigured or may produce packets
only sporadically. This can cause downstream calculators to wait for many
packets that will never arrive, which in turn causes packets to accumulate on
some of their input streams. MediaPipe addresses this sort of problem using
"timestamp bounds". For some hints see:
[`How to process realtime input streams`].
The MediaPipe setting [`CalculatorGraphConfig::max_queue_size`] limits the
number of packets enqueued on any input stream by throttling inputs to the
graph. For realtime input streams, the number of packets queued at an input
stream should almost always be zero or one. If this is not the case, you may see
the following warning message:
```
Resolved a deadlock by increasing max_queue_size of input stream
```
Also, the setting [`CalculatorGraphConfig::report_deadlock`] can be set to cause
graph run to fail and surface the deadlock as an error, such that max_queue_size
to acts as a memory usage limit.
## Graph hangs
Many applications will call [`CalculatorGraph::CloseAllPacketSources`] and
[`CalculatorGraph::WaitUntilDone`] to finish or suspend execution of a MediaPipe
graph. The objective here is to allow any pending calculators or packets to
complete processing, and then to shutdown the graph. If all goes well, every
stream in the graph will reach [`Timestamp::Done`], and every calculator will
reach [`CalculatorBase::Close`], and then [`CalculatorGraph::WaitUntilDone`]
will complete successfully.
If some calculators or streams cannot reach state [`Timestamp::Done`] or
[`CalculatorBase::Close`], then the method [`CalculatorGraph::Cancel`] can be
called to terminate the graph run without waiting for all pending calculators
and packets to complete.
## Output timing is uneven
Some realtime MediaPipe graphs produce a series of video frames for viewing as a
video effect or as a video diagnostic. Sometimes, a MediaPipe graph will produce
these frames in clusters, for example when several output frames are
extrapolated from the same cluster of input frames. If the outputs are presented
as they are produced, some output frames are immediately replaced by later
frames in the same cluster, which makes the results hard to see and evaluate
visually. In cases like this, the output visualization can be improved by
presenting the frames at even intervals in real time.
MediaPipe addresses this use case by mapping timestamps to points in real time.
Each timestamp indicates a time in microseconds, and a calculator such as
`LiveClockSyncCalculator` can delay the output of packets to match their
timestamps. This sort of calculator adjusts the timing of outputs such that:
1. The time between outputs corresponds to the time between timestamps as
closely as possible.
2. Outputs are produced with the smallest delay possible.
## CalculatorGraph lags behind inputs
For many realtime MediaPipe graphs, low latency is an objective. MediaPipe
supports "pipelined" style parallel processing in order to begin processing of
each packet as early as possible. Normally the lowest possible latency is the
total time required by each calculator along a "critical path" of successive
calculators. The latency of the a MediaPipe graph could be worse than the ideal
due to delays introduced to display frames a even intervals as described in
[Output timing is uneven](#output-timing-is-uneven).
If some of the calculators in the graph cannot keep pace with the realtime input
streams, then latency will continue to increase, and it becomes necessary to
drop some input packets. The recommended technique is to use the MediaPipe
calculators designed specifically for this purpose such as
[`FlowLimiterCalculator`] as described in
[`How to process realtime input streams`].
[`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`CalculatorGraphConfig::report_deadlock`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`REGISTER_CALCULATOR`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_registry.h
[`registration.h`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/deps/registration.h
[`CalculatorGraph::CloseAllPacketSources`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
[`CalculatorGraph::Cancel`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
[`CalculatorGraph::WaitUntilDone`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_graph.h
[`Timestamp::Done`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/timestamp.h
[`CalculatorBase::Close`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
[`How to process realtime input streams`]: faq.md#how-to-process-realtime-input-streams
docs/images/accelerated.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

docs/images/accelerated_small.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

docs/images/add_ipa.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 51 KiB

docs/images/app_ipa.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 885 KiB

docs/images/app_ipa_added.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

docs/images/autoflip_edited_example.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.2 MiB

docs/images/autoflip_graph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 170 KiB

docs/images/autoflip_is_required.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.5 MiB

docs/images/bazel_permission.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 39 KiB

docs/images/click_subgraph_handdetection.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 361 KiB

docs/images/console_error.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

docs/images/cross_platform.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.8 KiB

docs/images/cross_platform_small.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

docs/images/cyclic_integer_sum_graph.svg
+4
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

docs/images/device.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 195 KiB

docs/images/editor_view.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

docs/images/face_detection_desktop.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 118 KiB

docs/images/face_mesh_ar_effects.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.1 MiB

docs/images/favicon.ico
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

docs/images/faviconv2.ico
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 923 B

docs/images/gpu_example_graph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

docs/images/graph_visual.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 217 KiB

docs/images/hand_tracking_desktop.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 163 KiB

docs/images/hello_world.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.6 KiB

docs/images/iconv2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.6 KiB

docs/images/knift_stop_sign.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

docs/images/logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.8 KiB

docs/images/logo_horizontal_black.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

docs/images/logo_horizontal_color.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 31 KiB

docs/images/logo_horizontal_white.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

docs/images/logov2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.6 KiB

docs/images/maingraph_visualizer.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 302 KiB

docs/images/mediapipe_small.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.4 KiB

docs/images/mobile/aar_location.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 35 KiB

docs/images/mobile/android_studio_opencv_location.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

docs/images/mobile/assets_location.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

docs/images/mobile/bazel_hello_world_android.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

docs/images/mobile/box_tracking_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

docs/images/mobile/edge_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 666 KiB

docs/images/mobile/edge_detection_ios_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 529 KiB

docs/images/mobile/edge_detection_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/images/mobile/face_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.3 MiB

docs/images/mobile/face_detection_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 808 KiB

docs/images/mobile/face_detection_mobile_cpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

docs/images/mobile/face_detection_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 72 KiB

docs/images/mobile/face_landmark_front_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

docs/images/mobile/face_mesh_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.3 MiB

docs/images/mobile/face_mesh_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.0 MiB

docs/images/mobile/face_mesh_mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

docs/images/mobile/face_renderer_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 59 KiB

docs/images/mobile/hair_segmentation_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

docs/images/mobile/hair_segmentation_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 460 KiB

docs/images/mobile/hair_segmentation_mobile_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

docs/images/mobile/hand_crops.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 299 KiB

docs/images/mobile/hand_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.1 MiB

docs/images/mobile/hand_detection_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 383 KiB

docs/images/mobile/hand_detection_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 84 KiB

docs/images/mobile/hand_detection_mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

docs/images/mobile/hand_landmark_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 293 KiB

docs/images/mobile/hand_renderer_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 93 KiB

docs/images/mobile/hand_tracking_3d_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.6 MiB

docs/images/mobile/hand_tracking_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.7 MiB

docs/images/mobile/hand_tracking_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 448 KiB

docs/images/mobile/hand_tracking_mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 150 KiB

docs/images/mobile/missing_camera_permission_android.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 20 KiB

docs/images/mobile/multi_hand_detection_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 149 KiB

docs/images/mobile/multi_hand_landmark_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 193 KiB

docs/images/mobile/multi_hand_renderer_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 213 KiB

docs/images/mobile/multi_hand_tracking_3d_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

docs/images/mobile/multi_hand_tracking_3d_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 475 KiB

docs/images/mobile/multi_hand_tracking_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

docs/images/mobile/multi_hand_tracking_mobile.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 112 KiB

docs/images/mobile/object_detection_3d_android_gpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 100 KiB

docs/images/mobile/object_detection_android_cpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1004 KiB

docs/images/mobile/object_detection_android_gpu.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 945 KiB

docs/images/mobile/object_detection_android_gpu_small.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 336 KiB

docs/images/mobile/object_detection_gpu_subgraph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

docs/images/mobile/object_detection_mobile_cpu.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 68 KiB

Some files were not shown because too many files have changed in this diff Show More