talksik/mediapipe
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
GitOrigin-RevId: e3566e5029af25b0fc4b1071a49e49ae20aa5df6
This commit is contained in:
MediaPipe Team
2019-12-02 17:54:10 -08:00
committed by jqtang
parent 446d7cf6b6
commit 137867d088
74 changed files with 909 additions and 504 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mediapipe/docs/android_archive_library.md
+2 -2
View File
@@ -126,6 +126,6 @@ project.
```
6. Follow our Android app examples to use MediaPipe in Android Studio for your
use case. If you are looking for an example, a working face detection
use case. If you are looking for an example, a face detection
example can be found
[here](https://github.com/jiuqiant/mediapipe_aar_example).
[here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and a multi-hand tracking example can be found [here](https://github.com/jiuqiant/mediapipe_multi_hands_tracking_aar_example).
mediapipe/docs/gpu.md
+26 -2
View File
@@ -1,10 +1,10 @@
## Running on GPUs
- [Overview](#overview)
- [OpenGL Support](#graphconfig)
- [OpenGL Support](#opengl-support)
- [Life of a GPU calculator](#life-of-a-gpu-calculator)
- [GpuBuffer to ImageFrame converters](#gpubuffer-to-imageframe-converters)
- [Disable GPU support](#disable-gpu-support)
### Overview
MediaPipe supports calculator nodes for GPU compute and rendering, and allows combining multiple GPU nodes, as well as mixing them with CPU based calculator nodes. There exist several GPU APIs on mobile platforms (eg, OpenGL ES, Metal and Vulkan). MediaPipe does not attempt to offer a single cross-API GPU abstraction. Individual nodes can be written using different APIs, allowing them to take advantage of platform specific features when needed.
@@ -23,6 +23,7 @@ Below are the design principles for GPU support in MediaPipe
* A calculator should be allowed maximum flexibility in using the GPU for all or part of its operation, combining it with the CPU if necessary.
### OpenGL support
MediaPipe supports OpenGL ES up to version 3.2 on Android and up to ES 3.0 on iOS. In addition, MediaPipe also supports Metal on iOS.
* MediaPipe allows graphs to run OpenGL in multiple GL contexts. For example, this can be very useful in graphs that combine a slower GPU inference path (eg, at 10 FPS) with a faster GPU rendering path (eg, at 30 FPS): since one GL context corresponds to one sequential command queue, using the same context for both tasks would reduce the rendering frame rate. One challenge MediaPipe's use of multiple contexts solves is the ability to communicate across them. An example scenario is one with an input video that is sent to both the rendering and inferences paths, and rendering needs to have access to the latest output from inference.
@@ -128,3 +129,26 @@ The below diagram shows the data flow in a mobile application that captures vide
|:--:|
| *Video frames from the camera are fed into the graph as `GpuBuffer` packets. The input stream is accessed by two calculators in parallel. `GpuBufferToImageFrameCalculator` converts the buffer into an `ImageFrame`, which is then sent through a grayscale converter and a canny filter (both based on OpenCV and running on the CPU), whose output is then converted into a `GpuBuffer` again. A multi-input GPU calculator, GlOverlayCalculator, takes as input both the original `GpuBuffer` and the one coming out of the edge detector, and overlays them using a shader. The output is then sent back to the application using a callback calculator, and the application renders the image to the screen using OpenGL.* |
### Disable GPU Support
By default, building MediaPipe (with no special bazel flags) attempts to compile
and link against OpenGL/Metal libraries.
There are some command line build flags available to disable/enable GPU support
within the MediaPipe framework:
```
# To disable *all* gpu support
bazel build --define MEDIAPIPE_DISABLE_GPU=1 <my-target>
# to enable full GPU support (OpenGL ES 3.1+ & Metal)
bazel build --copt -DMESA_EGL_NO_X11_HEADERS <my-target>
# to enable only OpenGL ES 3.0 and below (no GLES 3.1+ features)
bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DMEDIAPIPE_DISABLE_GL_COMPUTE <my-target>
```
Note *MEDIAPIPE_DISABLE_GL_COMPUTE* is automatically defined on all Apple
systems (Apple doesn't support OpenGL ES 3.1+).
Note on iOS and Android, it is assumed that GPU support will be enabled.
mediapipe/docs/install.md
+11 -7
View File
@@ -245,19 +245,23 @@ To build and run iOS apps:
$ cd mediapipe
```
3. Install Bazel (0.24.1 and above required).
3. Install Bazel (version between 0.24.1 and 1.1.0).
Option 1. Use package manager tool to install the latest version of Bazel.
Option 1. Use package manager tool to install Bazel 1.1.0
```bash
$ brew install bazel
# Run 'bazel version' to check version of bazel installed
# If Bazel 1.1.0+ was installed.
$ brew uninstall bazel
# Install Bazel 1.1.0
$ brew install https://raw.githubusercontent.com/bazelbuild/homebrew-tap/f8a0fa981bcb1784a0d0823e14867b844e94fb3d/Formula/bazel.rb
$ brew link bazel
# Run 'bazel version' to check version of bazel
```
Option 2. Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-os-x.html#install-with-installer-mac-os-x)
to install any version of Bazel manually.
to install any version of Bazel manually. Note that MediaPipe doesn't
support Bazel 1.1.0+ on macOS yet.
4. Install OpenCV and FFmpeg.
@@ -526,7 +530,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
```bash
$ docker run -it --name mediapipe mediapipe:latest
root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk
root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk.sh
# Should print:
# Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r18b
mediapipe/docs/mediapipe_ios_setup.md
+14 -3
View File
@@ -7,7 +7,8 @@
2. Install [Bazel](https://bazel.build/).
See their [instructions](https://docs.bazel.build/versions/master/install-os-x.html).
See their
[instructions](https://docs.bazel.build/versions/master/install-os-x.html).
We recommend using [Homebrew](https://brew.sh/):
```bash
@@ -15,13 +16,23 @@
brew install bazelbuild/tap/bazel
```
3. Clone the MediaPipe repository.
3. Install python "future" and "six".
To make Mediapipe work with TensorFlow, please install the python "future"
library and the python "six" library:
```bash
pip install --user future six
```
4. Clone the MediaPipe repository.
```bash
git clone https://github.com/google/mediapipe.git
```
4. Symlink or copy your provisioning profile to `mediapipe/mediapipe/provisioning_profile.mobileprovision`.
5. Symlink or copy your provisioning profile to
`mediapipe/mediapipe/provisioning_profile.mobileprovision`.
```bash
cd mediapipe
mediapipe/docs/multi_hand_tracking_mobile_gpu.md
+3 -3
View File
@@ -560,7 +560,7 @@ node {
# BATCH_END timestamp, outputs the vector of landmarks at the BATCH_END
# timestamp.
node {
calculator: "EndLoopNormalizedLandmarksVectorCalculator"
calculator: "EndLoopNormalizedLandmarkListVectorCalculator"
input_stream: "ITEM:single_hand_landmarks"
input_stream: "BATCH_END:single_hand_rect_timestamp"
output_stream: "ITERABLE:multi_hand_landmarks"
@@ -580,7 +580,7 @@ node {
# hand. If the hand presence for hand #i is false, the set of landmarks
# corresponding to that hand are dropped from the vector.
node {
calculator: "FilterLandmarksCollectionCalculator"
calculator: "FilterLandmarkListCollectionCalculator"
input_stream: "ITERABLE:multi_hand_landmarks"
input_stream: "CONDITION:multi_hand_presence"
output_stream: "ITERABLE:filtered_multi_hand_landmarks"
@@ -669,7 +669,7 @@ node {
# timestamp for downstream calculators to inform them that all elements in the
# vector have been processed.
node {
calculator: "BeginLoopNormalizedLandmarksVectorCalculator"
calculator: "BeginLoopNormalizedLandmarkListVectorCalculator"
input_stream: "ITERABLE:multi_hand_landmarks"
output_stream: "ITEM:single_hand_landmarks"
output_stream: "BATCH_END:landmark_timestamp"