Compare commits

...
33 Commits
Author SHA1 Message Date
MediaPipe TeamandSebastian Schmidt 6cdc6443b6 Project import generated by Copybara.
GitOrigin-RevId: afbe148020a643ac800e34d91dc515995a9abb5c
2022-09-09 01:38:52 +00:00
MediaPipe TeamandSebastian Schmidt b65602fd31 Project import generated by Copybara.
GitOrigin-RevId: 777962478d88650e311af635e3ac3fa58e5a530b
2022-09-09 01:35:36 +00:00
MediaPipe TeamandSebastian Schmidt 06c30f1931 Project import generated by Copybara.
GitOrigin-RevId: ca9878d6e9c5beb87512e7536b200c55d150ede8
2022-09-08 19:08:07 +00:00
MediaPipe TeamandSebastian Schmidt ebec590cfe Project import generated by Copybara.
GitOrigin-RevId: e207bb2a1b26cd799055d7735ed35ad2f0e56b83
2022-09-07 20:55:41 +00:00
MediaPipe TeamandSebastian Schmidt d3f98334bf Project import generated by Copybara.
GitOrigin-RevId: 3ce19771d2586aeb611fff75bb7627721cf5d36b
2022-09-07 17:47:04 +00:00
MediaPipe TeamandSebastian Schmidt 4dc4b19ddb Project import generated by Copybara.
GitOrigin-RevId: 1e13be30e2c6838d4a2ff768a39c414bc80534bb
2022-09-06 21:46:17 +00:00
MediaPipe TeamandSebastian Schmidt 63e679d99c Project import generated by Copybara.
GitOrigin-RevId: 0f2489d226f1e2a5d718a8b9efe5e8198ba4ab3b
2022-06-29 18:45:02 +00:00
MediaPipe TeamandSebastian Schmidt c688862570 Project import generated by Copybara.
GitOrigin-RevId: 6e5aa035cd1f6a9333962df5d3ab97a05bd5744e
2022-06-28 12:11:05 +00:00
MediaPipe Teamandjqtang 4a20e9909d Project import generated by Copybara.
GitOrigin-RevId: b66251317fbebfbb8e1f2ddc64ea5da84bceb7e5
2022-05-06 17:05:30 -07:00
MediaPipe Teamandschmidt-sebastian 7fb37c80e8 Project import generated by Copybara.
GitOrigin-RevId: 19a829ffd755edb43e54d20c0e7b9348512d5108
2022-05-05 19:57:20 +00:00
MediaPipe Teamandjqtang c6c80c3745 Project import generated by Copybara.
GitOrigin-RevId: 17113e259b160929c49262e7aa78ac22d228f9fc
2022-03-22 17:48:17 -07:00
MediaPipe Teamandjqtang cc6a2f7af6 Project import generated by Copybara.
GitOrigin-RevId: 73d686c40057684f8bfaca285368bf1813f9fc26
2022-03-21 12:12:39 -07:00
MediaPipe Teamandjqtang e6c19885c6 Project import generated by Copybara.
GitOrigin-RevId: bb059a0721c92e8154d33ce8057b3915a25b3d7d
2021-12-13 15:56:02 -08:00
MediaPipe Teamandchuoling cf101e62a9 Project import generated by Copybara.
GitOrigin-RevId: 7e1d382a1788ebd8412c5626581b4c4cf2fe75ea
2021-11-16 14:32:04 -05:00
MediaPipe Teamandchuoling f4e7f6cc48 Project import generated by Copybara.
GitOrigin-RevId: 412b20ea6bc8e49ba5b50798a6114ad6173ff073
2021-11-04 01:53:16 -04:00
MediaPipe Teamandjqtang d4bb35fe5a Project import generated by Copybara.
GitOrigin-RevId: d4a11282d20fe4d2e137f9032cf349750030dcb9
2021-11-03 17:27:30 -07:00
MediaPipe Teamandchuoling 1faeaae7e5 Project import generated by Copybara.
GitOrigin-RevId: bbbbcb4f5174dea33525729ede47c770069157cd
2021-10-18 17:00:29 -04:00
MediaPipe Teamandjqtang 33d683c671 Project import generated by Copybara.
GitOrigin-RevId: 373e3ac1e5839befd95bf7d73ceff3c5f1171969
2021-10-06 14:27:49 -07:00
MediaPipe Teamandchuoling 137e1cc763 Project import generated by Copybara.
GitOrigin-RevId: 283c1a295de0a53e47d7a94996bda0c52dcfd677
2021-09-13 21:35:51 -04:00
MediaPipe Teamandjqtang 6abec128ed Project import generated by Copybara.
GitOrigin-RevId: f4b1fe3f15810450fb6539e733f6a260d3ee082c
2021-09-01 18:15:31 -07:00
MediaPipe Teamandjqtang 710fb3de58 Project import generated by Copybara.
GitOrigin-RevId: 1610e588e497817fae2d9a458093ab6a370e2972
2021-08-18 17:45:46 -07:00
MediaPipe Teamandchuoling b899d17f18 Project import generated by Copybara.
GitOrigin-RevId: 8e1da4611d93ccb7d9674713157d43be0348d98f
2021-07-27 22:36:23 -04:00
MediaPipe Teamandchuoling 50c92c6623 Project import generated by Copybara.
GitOrigin-RevId: 27c70b5fe62ab71189d358ca122ee4b19c817a8f
2021-07-27 19:36:32 -04:00
MediaPipe Teamandjqtang 374f5e2e7e Project import generated by Copybara.
GitOrigin-RevId: 65b427572550bd9c5bc5f053eeea0f44340d5673
2021-06-28 10:17:10 -07:00
MediaPipe Teamandchuoling 139237092f Project import generated by Copybara.
GitOrigin-RevId: 33adfdf31f3a5cbf9edc07ee1ea583e95080bdc5
2021-06-24 17:55:26 -04:00
MediaPipe Teamandchuoling b544a314b3 Project import generated by Copybara.
GitOrigin-RevId: ec25bf2e416c3689477e82946fb69de2e53b9161
2021-06-10 01:38:18 -04:00
MediaPipe Teamandchuoling b48d72e43f Project import generated by Copybara.
GitOrigin-RevId: 1e221238b0bc717115c8152ad3092da3309a63a1
2021-06-03 17:32:02 -04:00
MediaPipe Teamandchuoling 8b57bf879b Project import generated by Copybara.
GitOrigin-RevId: 08c2016a4df5aef571b464a4d4491f38c6b2af10
2021-06-03 17:04:35 -04:00
MediaPipe Teamandchuoling ae05ad04b3 Project import generated by Copybara.
GitOrigin-RevId: 016275ca4057540b2370ed4531dbc81eb92caae2
2021-05-11 01:00:51 -04:00
MediaPipe Teamandchuoling 017c1dc7ea Project import generated by Copybara.
GitOrigin-RevId: 2146b10f0a498f665f246e16033b686c7947b92d
2021-05-10 16:42:02 -04:00
MediaPipe Teamandchuoling a9b643e0f5 Project import generated by Copybara.
GitOrigin-RevId: ff83882955f1a1e2a043ff4e71278be9d7217bbe
2021-05-05 14:56:16 -04:00
MediaPipe Teamandchuoling ecb5b5f44a Project import generated by Copybara.
GitOrigin-RevId: 6a704ded0bf489614797082e7e7cda1068477ef5
2021-03-31 20:33:42 -04:00
MediaPipe Teamandchuoling 7c331ad58b Project import generated by Copybara.
GitOrigin-RevId: 6e4aff1cc351be3ae4537b677f36d139ee50ce09
2021-03-25 22:09:18 -04:00
1575 changed files with 133895 additions and 15272 deletions
+8
View File
@@ -32,6 +32,9 @@ build:macos --copt=-w
# Sets the default Apple platform to macOS. # Sets the default Apple platform to macOS.
build --apple_platform_type=macos build --apple_platform_type=macos
# Compile ObjC++ files with C++17
build --per_file_copt=.*\.mm\$@-std=c++17
# Allow debugging with XCODE # Allow debugging with XCODE
build --apple_generate_dsym build --apple_generate_dsym
@@ -58,6 +61,7 @@ build:android_arm64 --fat_apk_cpu=arm64-v8a
# iOS configs. # iOS configs.
build:ios --apple_platform_type=ios build:ios --apple_platform_type=ios
build:ios --copt=-fno-aligned-allocation
build:ios_i386 --config=ios build:ios_i386 --config=ios
build:ios_i386 --cpu=ios_i386 build:ios_i386 --cpu=ios_i386
@@ -87,6 +91,10 @@ build:darwin_x86_64 --apple_platform_type=macos
build:darwin_x86_64 --macos_minimum_os=10.12 build:darwin_x86_64 --macos_minimum_os=10.12
build:darwin_x86_64 --cpu=darwin_x86_64 build:darwin_x86_64 --cpu=darwin_x86_64
build:darwin_arm64 --apple_platform_type=macos
build:darwin_arm64 --macos_minimum_os=10.16
build:darwin_arm64 --cpu=darwin_arm64
# This bazelrc file is meant to be written by a setup script. # This bazelrc file is meant to be written by a setup script.
try-import %workspace%/.configure.bazelrc try-import %workspace%/.configure.bazelrc
+1
View File
@@ -0,0 +1 @@
5.2.0
@@ -0,0 +1,27 @@
---
name: "Build/Installation Issue"
about: Use this template for build/installation issues
labels: type:build/install
---
<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
**System information** (Please provide as much relevant information as possible)
- OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
- Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
- Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
- Installed using virtualenv? pip? Conda? (if python):
- [MediaPipe version](https://github.com/google/mediapipe/releases):
- Bazel version:
- XCode and Tulsi versions (if iOS):
- Android SDK and NDK versions (if android):
- Android [AAR](https://google.github.io/mediapipe/getting_started/android_archive_library.html) ( if android):
- OpenCV version (if running on desktop):
**Describe the problem**:
**[Provide the exact sequence of commands / steps that you executed before running into the problem](https://google.github.io/mediapipe/getting_started/getting_started.html):**
**Complete Logs:**
Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:
@@ -0,0 +1,26 @@
---
name: "Solution Issue"
about: Use this template for assistance with a specific mediapipe solution, such as "Pose" or "Iris", including inference model usage/training, solution-specific calculators, etc.
labels: type:support
---
<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
**System information** (Please provide as much relevant information as possible)
- Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
- [MediaPipe version](https://github.com/google/mediapipe/releases):
- Bazel version:
- Solution (e.g. FaceMesh, Pose, Holistic):
- Programming Language and version ( e.g. C++, Python, Java):
**Describe the expected behavior:**
**Standalone code you may have used to try to get what you need :**
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
**Other info / Complete Logs :**
Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached:
@@ -0,0 +1,51 @@
---
name: "Documentation Issue"
about: Use this template for documentation related issues
labels: type:docs
---
Thank you for submitting a MediaPipe documentation issue.
The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
## URL(s) with the issue:
Please provide a link to the documentation entry, for example: https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
## Description of issue (what needs changing):
Kinds of documentation problems:
### Clear description
For example, why should someone use this method? How is it useful?
### Correct links
Is the link to the source code correct?
### Parameters defined
Are all parameters defined and formatted correctly?
### Returns defined
Are return values defined?
### Raises listed and defined
Are the errors defined? For example,
### Usage example
Is there a usage example?
See the API guide:
on how to write testable usage examples.
### Request visuals, if applicable
Are there currently visuals? If not, will it clarify the content?
### Submit a pull request?
Are you planning to also submit a pull request to fix the issue? See the docs
https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md
+32
View File
@@ -0,0 +1,32 @@
---
name: "Bug Issue"
about: Use this template for reporting a bug
labels: type:bug
---
<em>Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.</em>
**System information** (Please provide as much relevant information as possible)
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser:
- Programming Language and version ( e.g. C++, Python, Java):
- [MediaPipe version](https://github.com/google/mediapipe/releases):
- Bazel version (if compiling from source):
- Solution ( e.g. FaceMesh, Pose, Holistic ):
- Android Studio, NDK, SDK versions (if issue is related to building in Android environment):
- Xcode & Tulsi version (if issue is related to building for iOS):
**Describe the current behavior:**
**Describe the expected behavior:**
**Standalone code to reproduce the issue:**
Provide a reproducible test case that is the bare minimum necessary to replicate the problem. If possible, please share a link to Colab/repo link /any notebook:
**Other info / Complete Logs :**
Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached
@@ -0,0 +1,24 @@
---
name: "Feature Request"
about: Use this template for raising a feature request
labels: type:feature
---
<em>Please make sure that this is a feature request.</em>
**System information** (Please provide as much relevant information as possible)
- MediaPipe Solution (you are using):
- Programming language : C++/typescript/Python/Objective C/Android Java
- Are you willing to contribute it (Yes/No):
**Describe the feature and the current behavior/state:**
**Will this change the current api? How?**
**Who will benefit with this feature?**
**Please specify the use cases for this feature:**
**Any Other info:**
+12
View File
@@ -0,0 +1,12 @@
---
name: "Other Issue"
about: Use this template for any other non-support related issues.
labels: type:others
---
This template is for miscellaneous issues not covered by the other issue categories
For questions on how to work with MediaPipe, or support for problems that are not verified bugs in MediaPipe, please go to [StackOverflow](https://stackoverflow.com/questions/tagged/mediapipe) and [Slack](https://mediapipe.page.link/joinslack) communities.
If you are reporting a vulnerability, please use the [dedicated reporting process](https://github.com/google/mediapipe/security).
+18
View File
@@ -0,0 +1,18 @@
# Copyright 2021 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
# A list of assignees
assignees:
- sureshdagooglecom
+34
View File
@@ -0,0 +1,34 @@
# Copyright 2021 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
#
# This file was assembled from multiple pieces, whose use is documented
# throughout. Please refer to the TensorFlow dockerfiles documentation
# for more information.
# Number of days of inactivity before an Issue or Pull Request becomes stale
daysUntilStale: 7
# Number of days of inactivity before a stale Issue or Pull Request is closed
daysUntilClose: 7
# Only issues or pull requests with all of these labels are checked if stale. Defaults to `[]` (disabled)
onlyLabels:
- stat:awaiting response
# Comment to post when marking as stale. Set to `false` to disable
markComment: >
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you.
# Comment to post when removing the stale label. Set to `false` to disable
unmarkComment: false
closeComment: >
Closing as stale. Please reopen if you'd like to work on this further.
+1 -1
View File
@@ -5,7 +5,7 @@
* Bug fixes * Bug fixes
* Documentation fixes * Documentation fixes
For new feature additions (e.g., new graphs and calculators), we are currently not planning to accept new feature pull requests into the MediaPipe repository. Instead, we like to get contributors to create their own repositories of the new feature and list it at [Awesome MediaPipe](https://mediapipe.org). This will allow contributors to more quickly get their code out to the community. For new feature additions (e.g., new graphs and calculators), we are currently not planning to accept new feature pull requests into the MediaPipe repository. Instead, we like to get contributors to create their own repositories of the new feature and list it at [Awesome MediaPipe](https://mediapipe.page.link/awesome-mediapipe). This will allow contributors to more quickly get their code out to the community.
Before sending your pull requests, make sure you followed this list. Before sending your pull requests, make sure you followed this list.
+8 -3
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
FROM ubuntu:18.04 FROM ubuntu:20.04
MAINTAINER <[email protected]> MAINTAINER <[email protected]>
@@ -23,6 +23,7 @@ ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \ build-essential \
gcc-8 g++-8 \
ca-certificates \ ca-certificates \
curl \ curl \
ffmpeg \ ffmpeg \
@@ -41,20 +42,24 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common && \ software-properties-common && \
add-apt-repository -y ppa:openjdk-r/ppa && \ add-apt-repository -y ppa:openjdk-r/ppa && \
apt-get update && apt-get install -y openjdk-8-jdk && \ apt-get update && apt-get install -y openjdk-8-jdk && \
apt-get install -y mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev && \
apt-get install -y mesa-utils && \
apt-get clean && \ apt-get clean && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 100 --slave /usr/bin/g++ g++ /usr/bin/g++-8
RUN pip3 install --upgrade setuptools RUN pip3 install --upgrade setuptools
RUN pip3 install wheel RUN pip3 install wheel
RUN pip3 install future RUN pip3 install future
RUN pip3 install absl-py numpy opencv-contrib-python protobuf==3.20.1
RUN pip3 install six==1.14.0 RUN pip3 install six==1.14.0
RUN pip3 install tensorflow==1.14.0 RUN pip3 install tensorflow==2.2.0
RUN pip3 install tf_slim RUN pip3 install tf_slim
RUN ln -s /usr/bin/python3 /usr/bin/python RUN ln -s /usr/bin/python3 /usr/bin/python
# Install bazel # Install bazel
ARG BAZEL_VERSION=3.4.1 ARG BAZEL_VERSION=5.2.0
RUN mkdir /bazel && \ RUN mkdir /bazel && \
wget --no-check-certificate -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/b\ wget --no-check-certificate -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/b\
azel-${BAZEL_VERSION}-installer-linux-x86_64.sh" && \ azel-${BAZEL_VERSION}-installer-linux-x86_64.sh" && \
+1 -3
View File
@@ -7,6 +7,4 @@ include MANIFEST.in
include README.md include README.md
include requirements.txt include requirements.txt
recursive-include mediapipe/modules *.tflite *.txt *.binarypb recursive-include mediapipe/modules *.txt
exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite
+30 -46
View File
@@ -4,7 +4,7 @@ title: Home
nav_order: 1 nav_order: 1
--- ---
![MediaPipe](docs/images/mediapipe_small.png) ![MediaPipe](https://mediapipe.dev/images/mediapipe_small.png)
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
@@ -13,21 +13,21 @@ nav_order: 1
[MediaPipe](https://google.github.io/mediapipe/) offers cross-platform, customizable [MediaPipe](https://google.github.io/mediapipe/) offers cross-platform, customizable
ML solutions for live and streaming media. ML solutions for live and streaming media.
![accelerated.png](docs/images/accelerated_small.png) | ![cross_platform.png](docs/images/cross_platform_small.png) ![accelerated.png](https://mediapipe.dev/images/accelerated_small.png) | ![cross_platform.png](https://mediapipe.dev/images/cross_platform_small.png)
:------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------: :------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------:
***End-to-End acceleration***: *Built-in fast ML inference and processing accelerated even on common hardware* | ***Build once, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT* ***End-to-End acceleration***: *Built-in fast ML inference and processing accelerated even on common hardware* | ***Build once, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT*
![ready_to_use.png](docs/images/ready_to_use_small.png) | ![open_source.png](docs/images/open_source_small.png) ![ready_to_use.png](https://mediapipe.dev/images/ready_to_use_small.png) | ![open_source.png](https://mediapipe.dev/images/open_source_small.png)
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable* ***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
## ML solutions in MediaPipe ## ML solutions in MediaPipe
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :------: :----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :------:
[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](docs/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose) | [![hair_segmentation](docs/images/mobile/holistic_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/holistic) [![face_detection](https://mediapipe.dev/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](https://mediapipe.dev/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](https://mediapipe.dev/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](https://mediapipe.dev/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](https://mediapipe.dev/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose) | [![hair_segmentation](https://mediapipe.dev/images/mobile/holistic_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/holistic)
Hair Segmentation | Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT Hair Segmentation | Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT
:-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---: :-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
[![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) | [![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![instant_motion_tracking](docs/images/mobile/instant_motion_tracking_android_small.gif)](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift) [![hair_segmentation](https://mediapipe.dev/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) | [![object_detection](https://mediapipe.dev/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](https://mediapipe.dev/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![instant_motion_tracking](https://mediapipe.dev/images/mobile/instant_motion_tracking_android_small.gif)](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [![objectron](https://mediapipe.dev/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](https://mediapipe.dev/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. --> <!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
<!-- Whenever this table is updated, paste a copy to solutions/solutions.md. --> <!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
@@ -40,11 +40,12 @@ Hair Segmentation
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ | ✅ | [Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ | ✅ |
[Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ | [Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ |
[Holistic](https://google.github.io/mediapipe/solutions/holistic) | ✅ | ✅ | ✅ | ✅ | ✅ | [Holistic](https://google.github.io/mediapipe/solutions/holistic) | ✅ | ✅ | ✅ | ✅ | ✅ |
[Selfie Segmentation](https://google.github.io/mediapipe/solutions/selfie_segmentation) | ✅ | ✅ | ✅ | ✅ | ✅ |
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | | [Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | |
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅ [Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | | [Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | |
[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | | [Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | |
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | ✅ | | [Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | ✅ | |
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | | [KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | |
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | | [AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | |
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | | [MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | |
@@ -54,46 +55,22 @@ See also
[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models) [MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
for ML models released in MediaPipe. for ML models released in MediaPipe.
## MediaPipe in Python
MediaPipe offers customizable Python solutions as a prebuilt Python package on
[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
`pip install mediapipe`. It also provides tools for users to build their own
solutions. Please see
[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
for more info.
## MediaPipe on the Web
MediaPipe on the Web is an effort to run the same ML solutions built for mobile
and desktop also in web browsers. The official API is under construction, but
the core technology has been proven effective. Please see
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
in Google Developers Blog for details.
You can use the following links to load a demo in the MediaPipe Visualizer, and
over there click the "Runner" icon in the top bar like shown below. The demos
use your webcam video as input, which is processed all locally in real-time and
never leaves your device.
![visualizer_runner](docs/images/visualizer_runner.png)
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
## Getting started ## Getting started
Learn how to [install](https://google.github.io/mediapipe/getting_started/install) To start using MediaPipe
MediaPipe and [solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
[build example applications](https://google.github.io/mediapipe/getting_started/building_examples), lines code, see example code and demos in
and start exploring our ready-to-use [MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can [MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
further extend and customize.
To use MediaPipe in C++, Android and iOS, which allow further customization of
the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
building your own, learn how to
[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
start building example applications in
[C++](https://google.github.io/mediapipe/getting_started/cpp),
[Android](https://google.github.io/mediapipe/getting_started/android) and
[iOS](https://google.github.io/mediapipe/getting_started/ios).
The source code is hosted in the The source code is hosted in the
[MediaPipe Github repository](https://github.com/google/mediapipe), and you can [MediaPipe Github repository](https://github.com/google/mediapipe), and you can
@@ -102,6 +79,13 @@ run code search using
## Publications ## Publications
* [Bringing artworks to life with AR](https://developers.googleblog.com/2021/07/bringing-artworks-to-life-with-ar.html)
in Google Developers Blog
* [Prosthesis control via Mirru App using MediaPipe hand tracking](https://developers.googleblog.com/2021/05/control-your-mirru-prosthesis-with-mediapipe-hand-tracking.html)
in Google Developers Blog
* [SignAll SDK: Sign language interface using MediaPipe is now available for
developers](https://developers.googleblog.com/2021/04/signall-sdk-sign-language-interface-using-mediapipe-now-available.html)
in Google Developers Blog
* [MediaPipe Holistic - Simultaneous Face, Hand and Pose Prediction, on Device](https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html) * [MediaPipe Holistic - Simultaneous Face, Hand and Pose Prediction, on Device](https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html)
in Google AI Blog in Google AI Blog
* [Background Features in Google Meet, Powered by Web ML](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html) * [Background Features in Google Meet, Powered by Web ML](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html)
@@ -152,8 +136,8 @@ run code search using
## Community ## Community
* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome * [Awesome MediaPipe](https://mediapipe.page.link/awesome-mediapipe) - A
MediaPipe related frameworks, libraries and software curated list of awesome MediaPipe related frameworks, libraries and software
* [Slack community](https://mediapipe.page.link/joinslack) for MediaPipe users * [Slack community](https://mediapipe.page.link/joinslack) for MediaPipe users
* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General * [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General
community discussion around MediaPipe community discussion around MediaPipe
+176 -69
View File
@@ -2,22 +2,31 @@ workspace(name = "mediapipe")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
skylib_version = "0.9.0" # Protobuf expects an //external:python_headers target
bind(
name = "python_headers",
actual = "@local_config_python//:python_headers",
)
http_archive( http_archive(
name = "bazel_skylib", name = "bazel_skylib",
type = "tar.gz", type = "tar.gz",
url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel_skylib-{}.tar.gz".format (skylib_version, skylib_version), urls = [
sha256 = "1dde365491125a3db70731e25658dfdd3bc5dbdfd11b840b3e987ecf043c7ca0", "https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz",
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz",
],
sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
) )
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()
load("@bazel_skylib//lib:versions.bzl", "versions") load("@bazel_skylib//lib:versions.bzl", "versions")
versions.check(minimum_bazel_version = "3.4.0") versions.check(minimum_bazel_version = "3.7.2")
# ABSL cpp library lts_2021_03_24, patch 2.
# ABSL cpp library lts_2020_09_23
http_archive( http_archive(
name = "com_google_absl", name = "com_google_absl",
urls = [ urls = [
"https://github.com/abseil/abseil-cpp/archive/20200923.tar.gz", "https://github.com/abseil/abseil-cpp/archive/refs/tags/20210324.2.tar.gz",
], ],
# Remove after https://github.com/abseil/abseil-cpp/issues/326 is solved. # Remove after https://github.com/abseil/abseil-cpp/issues/326 is solved.
patches = [ patches = [
@@ -26,20 +35,21 @@ http_archive(
patch_args = [ patch_args = [
"-p1", "-p1",
], ],
strip_prefix = "abseil-cpp-20200923", strip_prefix = "abseil-cpp-20210324.2",
sha256 = "b3744a4f7a249d5eaf2309daad597631ce77ea62e0fc6abffbab4b4c3dc0fc08" sha256 = "59b862f50e710277f8ede96f083a5bb8d7c9595376146838b9580be90374ee1f"
) )
http_archive( http_archive(
name = "rules_cc", name = "rules_cc",
strip_prefix = "rules_cc-master", strip_prefix = "rules_cc-2f8c04c04462ab83c545ab14c0da68c3b4c96191",
urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"], # The commit can be updated if the build passes. Last updated 6/23/22.
urls = ["https://github.com/bazelbuild/rules_cc/archive/2f8c04c04462ab83c545ab14c0da68c3b4c96191.zip"],
) )
http_archive( http_archive(
name = "rules_foreign_cc", name = "rules_foreign_cc",
strip_prefix = "rules_foreign_cc-main", strip_prefix = "rules_foreign_cc-0.1.0",
url = "https://github.com/bazelbuild/rules_foreign_cc/archive/main.zip", url = "https://github.com/bazelbuild/rules_foreign_cc/archive/0.1.0.zip",
) )
load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies") load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
@@ -50,26 +60,20 @@ rules_foreign_cc_dependencies()
all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])""" all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
# GoogleTest/GoogleMock framework. Used by most unit-tests. # GoogleTest/GoogleMock framework. Used by most unit-tests.
# Last updated 2020-06-30. # Last updated 2021-07-02.
http_archive( http_archive(
name = "com_google_googletest", name = "com_google_googletest",
urls = ["https://github.com/google/googletest/archive/aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e.zip"], urls = ["https://github.com/google/googletest/archive/4ec4cd23f486bf70efcc5d2caa40f24368f752e3.zip"],
patches = [ strip_prefix = "googletest-4ec4cd23f486bf70efcc5d2caa40f24368f752e3",
# fix for https://github.com/google/googletest/issues/2817 sha256 = "de682ea824bfffba05b4e33b67431c247397d6175962534305136aa06f92e049",
"@//third_party:com_google_googletest_9d580ea80592189e6d44fa35bcf9cdea8bf620d6.diff"
],
patch_args = [
"-p1",
],
strip_prefix = "googletest-aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e",
sha256 = "04a1751f94244307cebe695a69cc945f9387a80b0ef1af21394a490697c5c895",
) )
# Google Benchmark library. # Google Benchmark library v1.6.1 released on 2022-01-10.
http_archive( http_archive(
name = "com_google_benchmark", name = "com_google_benchmark",
urls = ["https://github.com/google/benchmark/archive/master.zip"], urls = ["https://github.com/google/benchmark/archive/refs/tags/v1.6.1.tar.gz"],
strip_prefix = "benchmark-master", strip_prefix = "benchmark-1.6.1",
sha256 = "6132883bc8c9b0df5375b16ab520fac1a85dc9e4cf5be59480448ece74b278d4",
build_file = "@//third_party:benchmark.BUILD", build_file = "@//third_party:benchmark.BUILD",
) )
@@ -117,7 +121,8 @@ http_archive(
# libyuv # libyuv
http_archive( http_archive(
name = "libyuv", name = "libyuv",
urls = ["https://chromium.googlesource.com/libyuv/libyuv/+archive/refs/heads/master.tar.gz"], # Error: operand type mismatch for `vbroadcastss' caused by commit 8a13626e42f7fdcf3a6acbb0316760ee54cda7d8.
urls = ["https://chromium.googlesource.com/libyuv/libyuv/+archive/2525698acba9bf9b701ba6b4d9584291a1f62257.tar.gz"],
build_file = "@//third_party:libyuv.BUILD", build_file = "@//third_party:libyuv.BUILD",
) )
@@ -125,16 +130,16 @@ http_archive(
# ...but the Java download is currently broken, so we use the "source" download. # ...but the Java download is currently broken, so we use the "source" download.
http_archive( http_archive(
name = "com_google_protobuf_javalite", name = "com_google_protobuf_javalite",
sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288ed4380426f9", sha256 = "87407cd28e7a9c95d9f61a098a53cf031109d451a7763e7dd1253abf8b4df422",
strip_prefix = "protobuf-3.11.4", strip_prefix = "protobuf-3.19.1",
urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz"], urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.19.1.tar.gz"],
) )
http_archive( http_archive(
name = "com_google_protobuf", name = "com_google_protobuf",
sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288ed4380426f9", sha256 = "87407cd28e7a9c95d9f61a098a53cf031109d451a7763e7dd1253abf8b4df422",
strip_prefix = "protobuf-3.11.4", strip_prefix = "protobuf-3.19.1",
urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz"], urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.19.1.tar.gz"],
patches = [ patches = [
"@//third_party:com_google_protobuf_fixes.diff" "@//third_party:com_google_protobuf_fixes.diff"
], ],
@@ -143,12 +148,50 @@ http_archive(
], ],
) )
load("//third_party/flatbuffers:workspace.bzl", flatbuffers = "repo")
flatbuffers()
http_archive( http_archive(
name = "com_google_audio_tools", name = "com_google_audio_tools",
strip_prefix = "multichannel-audio-tools-master", strip_prefix = "multichannel-audio-tools-master",
urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"], urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
) )
# sentencepiece
http_archive(
name = "com_google_sentencepiece",
strip_prefix = "sentencepiece-1.0.0",
sha256 = "c05901f30a1d0ed64cbcf40eba08e48894e1b0e985777217b7c9036cac631346",
urls = [
"https://github.com/google/sentencepiece/archive/1.0.0.zip",
],
repo_mapping = {"@com_google_glog" : "@com_github_glog_glog"},
)
http_archive(
name = "org_tensorflow_text",
sha256 = "f64647276f7288d1b1fe4c89581d51404d0ce4ae97f2bcc4c19bd667549adca8",
strip_prefix = "text-2.2.0",
urls = [
"https://github.com/tensorflow/text/archive/v2.2.0.zip",
],
patches = [
"//third_party:tensorflow_text_remove_tf_deps.diff",
"//third_party:tensorflow_text_a0f49e63.diff",
],
patch_args = ["-p1"],
repo_mapping = {"@com_google_re2": "@com_googlesource_code_re2"},
)
http_archive(
name = "com_googlesource_code_re2",
sha256 = "e06b718c129f4019d6e7aa8b7631bee38d3d450dd980246bfaf493eb7db67868",
strip_prefix = "re2-fe4a310131c37f9a7e7f7816fa6ce2a8b27d65a8",
urls = [
"https://github.com/google/re2/archive/fe4a310131c37f9a7e7f7816fa6ce2a8b27d65a8.tar.gz",
],
)
# 2020-07-09 # 2020-07-09
http_archive( http_archive(
name = "pybind11_bazel", name = "pybind11_bazel",
@@ -157,28 +200,38 @@ http_archive(
sha256 = "75922da3a1bdb417d820398eb03d4e9bd067c4905a4246d35a44c01d62154d91", sha256 = "75922da3a1bdb417d820398eb03d4e9bd067c4905a4246d35a44c01d62154d91",
) )
# Point to the commit that deprecates the usage of Eigen::MappedSparseMatrix.
http_archive( http_archive(
name = "pybind11", name = "pybind11",
urls = [ urls = [
"https://storage.googleapis.com/mirror.tensorflow.org/github.com/pybind/pybind11/archive/v2.4.3.tar.gz", "https://github.com/pybind/pybind11/archive/70a58c577eaf067748c2ec31bfd0b0a614cffba6.zip",
"https://github.com/pybind/pybind11/archive/v2.4.3.tar.gz",
], ],
sha256 = "1eed57bc6863190e35637290f97a20c81cfe4d9090ac0a24f3bbf08f265eb71d", sha256 = "b971842fab1b5b8f3815a2302331782b7d137fef0e06502422bc4bc360f4956c",
strip_prefix = "pybind11-2.4.3", strip_prefix = "pybind11-70a58c577eaf067748c2ec31bfd0b0a614cffba6",
build_file = "@pybind11_bazel//:pybind11.BUILD", build_file = "@pybind11_bazel//:pybind11.BUILD",
) )
http_archive(
name = "pybind11_protobuf",
sha256 = "baa1f53568283630a5055c85f0898b8810f7a6431bd01bbaedd32b4c1defbcb1",
strip_prefix = "pybind11_protobuf-3594106f2df3d725e65015ffb4c7886d6eeee683",
urls = [
"https://github.com/pybind/pybind11_protobuf/archive/3594106f2df3d725e65015ffb4c7886d6eeee683.tar.gz",
],
)
# Point to the commit that deprecates the usage of Eigen::MappedSparseMatrix.
http_archive( http_archive(
name = "ceres_solver", name = "ceres_solver",
url = "https://github.com/ceres-solver/ceres-solver/archive/2.0.0.zip", url = "https://github.com/ceres-solver/ceres-solver/archive/123fba61cf2611a3c8bddc9d91416db26b10b558.zip",
patches = [ patches = [
"@//third_party:ceres_solver_compatibility_fixes.diff" "@//third_party:ceres_solver_compatibility_fixes.diff"
], ],
patch_args = [ patch_args = [
"-p1", "-p1",
], ],
strip_prefix = "ceres-solver-2.0.0", strip_prefix = "ceres-solver-123fba61cf2611a3c8bddc9d91416db26b10b558",
sha256 = "db12d37b4cebb26353ae5b7746c7985e00877baa8e7b12dc4d3a1512252fff3b" sha256 = "8b7b16ceb363420e0fd499576daf73fa338adb0b1449f58bea7862766baa1ac7"
) )
http_archive( http_archive(
@@ -203,7 +256,10 @@ new_local_repository(
new_local_repository( new_local_repository(
name = "macos_opencv", name = "macos_opencv",
build_file = "@//third_party:opencv_macos.BUILD", build_file = "@//third_party:opencv_macos.BUILD",
path = "/usr/local/opt/opencv@3", # For local MacOS builds, the path should point to an opencv@3 installation.
# If you edit the path here, you will also need to update the corresponding
# prefix in "opencv_macos.BUILD".
path = "/usr/local",
) )
new_local_repository( new_local_repository(
@@ -238,21 +294,26 @@ http_archive(
url = "https://github.com/opencv/opencv/releases/download/3.2.0/opencv-3.2.0-ios-framework.zip", url = "https://github.com/opencv/opencv/releases/download/3.2.0/opencv-3.2.0-ios-framework.zip",
) )
# You may run setup_android.sh to install Android SDK and NDK. http_archive(
android_ndk_repository( name = "stblib",
name = "androidndk", strip_prefix = "stb-b42009b3b9d4ca35bc703f5310eedc74f584be58",
) sha256 = "13a99ad430e930907f5611325ec384168a958bf7610e63e60e2fd8e7b7379610",
urls = ["https://github.com/nothings/stb/archive/b42009b3b9d4ca35bc703f5310eedc74f584be58.tar.gz"],
android_sdk_repository( build_file = "@//third_party:stblib.BUILD",
name = "androidsdk", patches = [
"@//third_party:stb_image_impl.diff"
],
patch_args = [
"-p1",
],
) )
# iOS basic build deps. # iOS basic build deps.
http_archive( http_archive(
name = "build_bazel_rules_apple", name = "build_bazel_rules_apple",
sha256 = "7a7afdd4869bb201c9352eed2daf37294d42b093579b70423490c1b4d4f6ce42", sha256 = "77e8bf6fda706f420a55874ae6ee4df0c9d95da6c7838228b26910fc82eea5a2",
url = "https://github.com/bazelbuild/rules_apple/releases/download/0.19.0/rules_apple.0.19.0.tar.gz", url = "https://github.com/bazelbuild/rules_apple/releases/download/0.32.0/rules_apple.0.32.0.tar.gz",
patches = [ patches = [
# Bypass checking ios unit test runner when building MP ios applications. # Bypass checking ios unit test runner when building MP ios applications.
"@//third_party:build_bazel_rules_apple_bypass_test_runner_check.diff" "@//third_party:build_bazel_rules_apple_bypass_test_runner_check.diff"
@@ -278,10 +339,9 @@ swift_rules_dependencies()
http_archive( http_archive(
name = "build_bazel_apple_support", name = "build_bazel_apple_support",
sha256 = "122ebf7fe7d1c8e938af6aeaee0efe788a3a2449ece5a8d6a428cb18d6f88033", sha256 = "741366f79d900c11e11d8efd6cc6c66a31bfb2451178b58e0b5edc6f1db17b35",
urls = [ urls = [
"https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/apple_support/releases/download/0.7.1/apple_support.0.7.1.tar.gz", "https://github.com/bazelbuild/apple_support/releases/download/0.10.0/apple_support.0.10.0.tar.gz"
"https://github.com/bazelbuild/apple_support/releases/download/0.7.1/apple_support.0.7.1.tar.gz",
], ],
) )
@@ -304,8 +364,8 @@ http_archive(
# Maven dependencies. # Maven dependencies.
RULES_JVM_EXTERNAL_TAG = "3.2" RULES_JVM_EXTERNAL_TAG = "4.0"
RULES_JVM_EXTERNAL_SHA = "82262ff4223c5fda6fb7ff8bd63db8131b51b413d26eb49e3131037e79e324af" RULES_JVM_EXTERNAL_SHA = "31701ad93dbfe544d597dbe62c9a1fdd76d81d8a9150c2bf1ecf928ecdf97169"
http_archive( http_archive(
name = "rules_jvm_external", name = "rules_jvm_external",
@@ -318,10 +378,12 @@ load("@rules_jvm_external//:defs.bzl", "maven_install")
# Important: there can only be one maven_install rule. Add new maven deps here. # Important: there can only be one maven_install rule. Add new maven deps here.
maven_install( maven_install(
name = "maven",
artifacts = [ artifacts = [
"androidx.concurrent:concurrent-futures:1.0.0-alpha03", "androidx.concurrent:concurrent-futures:1.0.0-alpha03",
"androidx.lifecycle:lifecycle-common:2.2.0", "androidx.lifecycle:lifecycle-common:2.3.1",
"androidx.activity:activity:1.2.2",
"androidx.exifinterface:exifinterface:1.3.3",
"androidx.fragment:fragment:1.3.4",
"androidx.annotation:annotation:aar:1.1.0", "androidx.annotation:annotation:aar:1.1.0",
"androidx.appcompat:appcompat:aar:1.1.0-rc01", "androidx.appcompat:appcompat:aar:1.1.0-rc01",
"androidx.camera:camera-core:1.0.0-beta10", "androidx.camera:camera-core:1.0.0-beta10",
@@ -334,19 +396,24 @@ maven_install(
"androidx.test.espresso:espresso-core:3.1.1", "androidx.test.espresso:espresso-core:3.1.1",
"com.github.bumptech.glide:glide:4.11.0", "com.github.bumptech.glide:glide:4.11.0",
"com.google.android.material:material:aar:1.0.0-rc01", "com.google.android.material:material:aar:1.0.0-rc01",
"com.google.code.findbugs:jsr305:3.0.2", "com.google.auto.value:auto-value:1.8.1",
"com.google.flogger:flogger-system-backend:0.3.1", "com.google.auto.value:auto-value-annotations:1.8.1",
"com.google.flogger:flogger:0.3.1", "com.google.code.findbugs:jsr305:latest.release",
"com.google.android.datatransport:transport-api:3.0.0",
"com.google.android.datatransport:transport-backend-cct:3.1.0",
"com.google.android.datatransport:transport-runtime:3.1.0",
"com.google.flogger:flogger-system-backend:0.6",
"com.google.flogger:flogger:0.6",
"com.google.guava:guava:27.0.1-android", "com.google.guava:guava:27.0.1-android",
"com.google.guava:listenablefuture:1.0", "com.google.guava:listenablefuture:1.0",
"junit:junit:4.12", "junit:junit:4.12",
"org.hamcrest:hamcrest-library:1.3", "org.hamcrest:hamcrest-library:1.3",
], ],
repositories = [ repositories = [
"https://jcenter.bintray.com",
"https://maven.google.com", "https://maven.google.com",
"https://dl.google.com/dl/android/maven2", "https://dl.google.com/dl/android/maven2",
"https://repo1.maven.org/maven2", "https://repo1.maven.org/maven2",
"https://jcenter.bintray.com",
], ],
fetch_sources = True, fetch_sources = True,
version_conflict_policy = "pinned", version_conflict_policy = "pinned",
@@ -363,10 +430,29 @@ http_archive(
], ],
) )
#Tensorflow repo should always go after the other external dependencies. # Load Zlib before initializing TensorFlow to guarantee that the target
# 2020-12-09 # @zlib//:mini_zlib is available
_TENSORFLOW_GIT_COMMIT = "0eadbb13cef1226b1bae17c941f7870734d97f8a" http_archive(
_TENSORFLOW_SHA256= "4ae06daa5b09c62f31b7bc1f781fd59053f286dd64355830d8c2ac601b795ef0" name = "zlib",
build_file = "//third_party:zlib.BUILD",
sha256 = "c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1",
strip_prefix = "zlib-1.2.11",
urls = [
"http://mirror.bazel.build/zlib.net/fossils/zlib-1.2.11.tar.gz",
"http://zlib.net/fossils/zlib-1.2.11.tar.gz", # 2017-01-15
],
patches = [
"@//third_party:zlib.diff",
],
patch_args = [
"-p1",
],
)
# TensorFlow repo should always go after the other external dependencies.
# TF on 2022-08-10.
_TENSORFLOW_GIT_COMMIT = "af1d5bc4fbb66d9e6cc1cf89503014a99233583b"
_TENSORFLOW_SHA256 = "f85a5443264fc58a12d136ca6a30774b5bc25ceaf7d114d97f252351b3c3a2cb"
http_archive( http_archive(
name = "org_tensorflow", name = "org_tensorflow",
urls = [ urls = [
@@ -374,7 +460,8 @@ http_archive(
], ],
patches = [ patches = [
"@//third_party:org_tensorflow_compatibility_fixes.diff", "@//third_party:org_tensorflow_compatibility_fixes.diff",
"@//third_party:org_tensorflow_objc_cxx17.diff", # Diff is generated with a script, don't update it manually.
"@//third_party:org_tensorflow_custom_ops.diff",
], ],
patch_args = [ patch_args = [
"-p1", "-p1",
@@ -383,5 +470,25 @@ http_archive(
sha256 = _TENSORFLOW_SHA256, sha256 = _TENSORFLOW_SHA256,
) )
load("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace") load("@org_tensorflow//tensorflow:workspace3.bzl", "tf_workspace3")
tf_workspace(tf_repo_name = "org_tensorflow") tf_workspace3()
load("@org_tensorflow//tensorflow:workspace2.bzl", "tf_workspace2")
tf_workspace2()
# Edge TPU
http_archive(
name = "libedgetpu",
sha256 = "14d5527a943a25bc648c28a9961f954f70ba4d79c0a9ca5ae226e1831d72fe80",
strip_prefix = "libedgetpu-3164995622300286ef2bb14d7fdc2792dae045b7",
urls = [
"https://github.com/google-coral/libedgetpu/archive/3164995622300286ef2bb14d7fdc2792dae045b7.tar.gz"
],
)
load("@libedgetpu//:workspace.bzl", "libedgetpu_dependencies")
libedgetpu_dependencies()
load("@coral_crosstool//:configure.bzl", "cc_crosstool")
cc_crosstool(name = "crosstool")
load("//third_party:external_files.bzl", "external_files")
external_files()
+2 -2
View File
@@ -109,7 +109,7 @@ for app in ${apps}; do
if [[ ${category} != "shoe" ]]; then if [[ ${category} != "shoe" ]]; then
bazel_flags_extended+=(--define ${category}=true) bazel_flags_extended+=(--define ${category}=true)
fi fi
bazel "${bazel_flags_extended[@]}" bazelisk "${bazel_flags_extended[@]}"
cp -f "${bin}" "${apk}" cp -f "${bin}" "${apk}"
fi fi
apks+=(${apk}) apks+=(${apk})
@@ -120,7 +120,7 @@ for app in ${apps}; do
if [[ ${app_name} == "templatematchingcpu" ]]; then if [[ ${app_name} == "templatematchingcpu" ]]; then
switch_to_opencv_4 switch_to_opencv_4
fi fi
bazel "${bazel_flags[@]}" bazelisk "${bazel_flags[@]}"
cp -f "${bin}" "${apk}" cp -f "${bin}" "${apk}"
if [[ ${app_name} == "templatematchingcpu" ]]; then if [[ ${app_name} == "templatematchingcpu" ]]; then
switch_to_opencv_3 switch_to_opencv_3
+5 -4
View File
@@ -17,15 +17,15 @@
# Script to build/run all MediaPipe desktop example apps (with webcam input). # Script to build/run all MediaPipe desktop example apps (with webcam input).
# #
# To build and run all apps and store them in out_dir: # To build and run all apps and store them in out_dir:
# $ ./build_ios_examples.sh -d out_dir # $ ./build_desktop_examples.sh -d out_dir
# Omitting -d and the associated directory saves all generated apps in the # Omitting -d and the associated directory saves all generated apps in the
# current directory. # current directory.
# To build all apps and store them in out_dir: # To build all apps and store them in out_dir:
# $ ./build_ios_examples.sh -d out_dir -b # $ ./build_desktop_examples.sh -d out_dir -b
# Omitting -d and the associated directory saves all generated apps in the # Omitting -d and the associated directory saves all generated apps in the
# current directory. # current directory.
# To run all apps already stored in out_dir: # To run all apps already stored in out_dir:
# $ ./build_ios_examples.sh -d out_dir -r # $ ./build_desktop_examples.sh -d out_dir -r
# Omitting -d and the associated directory assumes all apps are in the current # Omitting -d and the associated directory assumes all apps are in the current
# directory. # directory.
@@ -83,7 +83,7 @@ for app in ${apps}; do
bazel_flags=("${default_bazel_flags[@]}") bazel_flags=("${default_bazel_flags[@]}")
bazel_flags+=(${target}) bazel_flags+=(${target})
bazel "${bazel_flags[@]}" bazelisk "${bazel_flags[@]}"
cp -f "${bin_dir}/${app}/"*"_cpu" "${out_dir}" cp -f "${bin_dir}/${app}/"*"_cpu" "${out_dir}"
fi fi
if [[ $build_only == false ]]; then if [[ $build_only == false ]]; then
@@ -97,6 +97,7 @@ for app in ${apps}; do
if [[ ${target_name} == "holistic_tracking" || if [[ ${target_name} == "holistic_tracking" ||
${target_name} == "iris_tracking" || ${target_name} == "iris_tracking" ||
${target_name} == "pose_tracking" || ${target_name} == "pose_tracking" ||
${target_name} == "selfie_segmentation" ||
${target_name} == "upper_body_pose_tracking" ]]; then ${target_name} == "upper_body_pose_tracking" ]]; then
graph_suffix="cpu" graph_suffix="cpu"
else else
+1 -1
View File
@@ -71,7 +71,7 @@ for app in ${apps}; do
bazel_flags+=(--linkopt=-s) bazel_flags+=(--linkopt=-s)
fi fi
bazel "${bazel_flags[@]}" bazelisk "${bazel_flags[@]}"
cp -f "${bin_dir}/${app}/"*".ipa" "${out_dir}" cp -f "${bin_dir}/${app}/"*".ipa" "${out_dir}"
fi fi
done done
+1 -1
View File
@@ -20,7 +20,7 @@ aux_links:
- "//github.com/google/mediapipe" - "//github.com/google/mediapipe"
# Footer content appears at the bottom of every page's main content # Footer content appears at the bottom of every page's main content
footer_content: "&copy; 2020 GOOGLE LLC | <a href=\"https://policies.google.com/privacy\">PRIVACY POLICY</a> | <a href=\"https://policies.google.com/terms\">TERMS OF SERVICE</a>" footer_content: "&copy; GOOGLE LLC | <a href=\"https://policies.google.com/privacy\">PRIVACY POLICY</a> | <a href=\"https://policies.google.com/terms\">TERMS OF SERVICE</a>"
# Color scheme currently only supports "dark", "light"/nil (default), or a custom scheme that you define # Color scheme currently only supports "dark", "light"/nil (default), or a custom scheme that you define
color_scheme: mediapipe color_scheme: mediapipe
+68 -11
View File
@@ -133,7 +133,7 @@ write outputs. After Close returns, the calculator is destroyed.
Calculators with no inputs are referred to as sources. A source calculator Calculators with no inputs are referred to as sources. A source calculator
continues to have `Process()` called as long as it returns an `Ok` status. A continues to have `Process()` called as long as it returns an `Ok` status. A
source calculator indicates that it is exhausted by returning a stop status source calculator indicates that it is exhausted by returning a stop status
(i.e. MediaPipe::tool::StatusStop). (i.e. [`mediaPipe::tool::StatusStop()`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/tool/status_util.cc).).
## Identifying inputs and outputs ## Identifying inputs and outputs
@@ -187,7 +187,7 @@ node {
``` ```
In the calculator implementation, inputs and outputs are also identified by tag In the calculator implementation, inputs and outputs are also identified by tag
name and index number. In the function below input are output are identified: name and index number. In the function below input and output are identified:
* By index number: The combined input stream is identified simply by index * By index number: The combined input stream is identified simply by index
`0`. `0`.
@@ -248,12 +248,70 @@ absl::Status MyCalculator::Process() {
} }
``` ```
## Calculator options
Calculators accept processing parameters through (1) input stream packets (2)
input side packets, and (3) calculator options. Calculator options, if
specified, appear as literal values in the `node_options` field of the
`CalculatorGraphConfiguration.Node` message.
```
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS:main_model_input"
output_stream: "TENSORS:main_model_output"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "mediapipe/models/detection_model.tflite"
}
}
}
```
The `node_options` field accepts the proto3 syntax. Alternatively, calculator
options can be specified in the `options` field using proto2 syntax.
```
node {
calculator: "TfLiteInferenceCalculator"
input_stream: "TENSORS:main_model_input"
output_stream: "TENSORS:main_model_output"
node_options: {
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
model_path: "mediapipe/models/detection_model.tflite"
}
}
}
```
Not all calculators accept calcuator options. In order to accept options, a
calculator will normally define a new protobuf message type to represent its
options, such as `PacketClonerCalculatorOptions`. The calculator will then
read that protobuf message in its `CalculatorBase::Open` method, and possibly
also in its `CalculatorBase::GetContract` function or its
`CalculatorBase::Process` method. Normally, the new protobuf message type will
be defined as a protobuf schema using a ".proto" file and a
`mediapipe_proto_library()` build rule.
```
mediapipe_proto_library(
name = "packet_cloner_calculator_proto",
srcs = ["packet_cloner_calculator.proto"],
visibility = ["//visibility:public"],
deps = [
"//mediapipe/framework:calculator_options_proto",
"//mediapipe/framework:calculator_proto",
],
)
```
## Example calculator ## Example calculator
This section discusses the implementation of `PacketClonerCalculator`, which This section discusses the implementation of `PacketClonerCalculator`, which
does a relatively simple job, and is used in many calculator graphs. does a relatively simple job, and is used in many calculator graphs.
`PacketClonerCalculator` simply produces a copy of its most recent input `PacketClonerCalculator` simply produces a copy of its most recent input packets
packets on demand. on demand.
`PacketClonerCalculator` is useful when the timestamps of arriving data packets `PacketClonerCalculator` is useful when the timestamps of arriving data packets
are not aligned perfectly. Suppose we have a room with a microphone, light are not aligned perfectly. Suppose we have a room with a microphone, light
@@ -279,8 +337,8 @@ input streams:
imageframe of video data representing video collected from camera in the imageframe of video data representing video collected from camera in the
room with timestamp. room with timestamp.
Below is the implementation of the `PacketClonerCalculator`. You can see Below is the implementation of the `PacketClonerCalculator`. You can see the
the `GetContract()`, `Open()`, and `Process()` methods as well as the instance `GetContract()`, `Open()`, and `Process()` methods as well as the instance
variable `current_` which holds the most recent input packets. variable `current_` which holds the most recent input packets.
```c++ ```c++
@@ -355,7 +413,6 @@ class PacketClonerCalculator : public CalculatorBase {
current_[i].At(cc->InputTimestamp())); current_[i].At(cc->InputTimestamp()));
// Add a packet to output stream of index i a packet from inputstream i // Add a packet to output stream of index i a packet from inputstream i
// with timestamp common to all present inputs // with timestamp common to all present inputs
//
} else { } else {
cc->Outputs().Index(i).SetNextTimestampBound( cc->Outputs().Index(i).SetNextTimestampBound(
cc->InputTimestamp().NextAllowedInStream()); cc->InputTimestamp().NextAllowedInStream());
@@ -382,7 +439,7 @@ defined your calculator class, register it with a macro invocation
REGISTER_CALCULATOR(calculator_class_name). REGISTER_CALCULATOR(calculator_class_name).
Below is a trivial MediaPipe graph that has 3 input streams, 1 node Below is a trivial MediaPipe graph that has 3 input streams, 1 node
(PacketClonerCalculator) and 3 output streams. (PacketClonerCalculator) and 2 output streams.
```proto ```proto
input_stream: "room_mic_signal" input_stream: "room_mic_signal"
@@ -402,6 +459,6 @@ node {
The diagram below shows how the `PacketClonerCalculator` defines its output The diagram below shows how the `PacketClonerCalculator` defines its output
packets (bottom) based on its series of input packets (top). packets (bottom) based on its series of input packets (top).
| ![Graph using PacketClonerCalculator](../images/packet_cloner_calculator.png) | ![Graph using PacketClonerCalculator](https://mediapipe.dev/images/packet_cloner_calculator.png) |
| :---------------------------------------------------------------------------: | :--------------------------------------------------------------------------: |
| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* | *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |
@@ -110,3 +110,12 @@ Other policies are also available, implemented using a separate kind of
component known as an InputStreamHandler. component known as an InputStreamHandler.
See [Synchronization](synchronization.md) for more details. See [Synchronization](synchronization.md) for more details.
### Real-time streams
MediaPipe calculator graphs are often used to process streams of video or audio
frames for interactive applications. Normally, each Calculator runs as soon as
all of its input packets for a given timestamp become available. Calculators
used in real-time graphs need to define output timestamp bounds based on input
timestamp bounds in order to allow downstream calculators to be scheduled
promptly. See [Real-time Streams](realtime_streams.md) for details.
+1 -1
View File
@@ -149,7 +149,7 @@ When possible, these calculators use platform-specific functionality to share da
The below diagram shows the data flow in a mobile application that captures video from the camera, runs it through a MediaPipe graph, and renders the output on the screen in real time. The dashed line indicates which parts are inside the MediaPipe graph proper. This application runs a Canny edge-detection filter on the CPU using OpenCV, and overlays it on top of the original video using the GPU. The below diagram shows the data flow in a mobile application that captures video from the camera, runs it through a MediaPipe graph, and renders the output on the screen in real time. The dashed line indicates which parts are inside the MediaPipe graph proper. This application runs a Canny edge-detection filter on the CPU using OpenCV, and overlays it on top of the original video using the GPU.
![How GPU calculators interact](../images/gpu_example_graph.png) ![How GPU calculators interact](https://mediapipe.dev/images/gpu_example_graph.png)
Video frames from the camera are fed into the graph as `GpuBuffer` packets. The Video frames from the camera are fed into the graph as `GpuBuffer` packets. The
input stream is accessed by two calculators in parallel. input stream is accessed by two calculators in parallel.
+3 -3
View File
@@ -83,12 +83,12 @@ Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`.
output_stream: "out3" output_stream: "out3"
node { node {
calculator: "PassThroughculator" calculator: "PassThroughCalculator"
input_stream: "out1" input_stream: "out1"
output_stream: "out2" output_stream: "out2"
} }
node { node {
calculator: "PassThroughculator" calculator: "PassThroughCalculator"
input_stream: "out2" input_stream: "out2"
output_stream: "out3" output_stream: "out3"
} }
@@ -159,7 +159,7 @@ Please use the `CalculatorGraphTest.Cycle` unit test in
below is the cyclic graph in the test. The `sum` output of the adder is the sum below is the cyclic graph in the test. The `sum` output of the adder is the sum
of the integers generated by the integer source calculator. of the integers generated by the integer source calculator.
![a cyclic graph that adds a stream of integers](../images/cyclic_integer_sum_graph.svg "A cyclic graph") ![a cyclic graph that adds a stream of integers](https://mediapipe.dev/images/cyclic_integer_sum_graph.svg "A cyclic graph")
This simple graph illustrates all the issues in supporting cyclic graphs. This simple graph illustrates all the issues in supporting cyclic graphs.
+17 -6
View File
@@ -12,19 +12,30 @@ nav_order: 3
{:toc} {:toc}
--- ---
Each calculator is a node of of a graph. We describe how to create a new calculator, how to initialize a calculator, how to perform its calculations, input and output streams, timestamps, and options Calculators communicate by sending and receiving packets. Typically a single
packet is sent along each input stream at each input timestamp. A packet can
contain any kind of data, such as a single frame of video or a single integer
detection count.
## Creating a packet ## Creating a packet
Packets are generally created with `MediaPipe::Adopt()` (from packet.h). Packets are generally created with `mediapipe::MakePacket<T>()` or
`mediapipe::Adopt()` (from packet.h).
```c++ ```c++
// Create some data. // Create a packet containing some new data.
auto data = absl::make_unique<MyDataClass>("constructor_argument"); Packet p = MakePacket<MyDataClass>("constructor_argument");
// Create a packet to own the data.
Packet p = Adopt(data.release());
// Make a new packet with the same data and a different timestamp. // Make a new packet with the same data and a different timestamp.
Packet p2 = p.At(Timestamp::PostStream()); Packet p2 = p.At(Timestamp::PostStream());
``` ```
or:
```c++
// Create some new data.
auto data = absl::make_unique<MyDataClass>("constructor_argument");
// Create a packet to own the data.
Packet p = Adopt(data.release()).At(Timestamp::PostStream());
```
Data within a packet is accessed with `Packet::Get<T>()` Data within a packet is accessed with `Packet::Get<T>()`
+186
View File
@@ -0,0 +1,186 @@
---
layout: default
title: Real-time Streams
parent: Framework Concepts
nav_order: 6
---
# Real-time Streams
{: .no_toc }
1. TOC
{:toc}
---
## Real-time timestamps
MediaPipe calculator graphs are often used to process streams of video or audio
frames for interactive applications. The MediaPipe framework requires only that
successive packets be assigned monotonically increasing timestamps. By
convention, real-time calculators and graphs use the recording time or the
presentation time of each frame as its timestamp, with each timestamp indicating
the microseconds since `Jan/1/1970:00:00:00`. This allows packets from various
sources to be processed in a globally consistent sequence.
## Real-time scheduling
Normally, each Calculator runs as soon as all of its input packets for a given
timestamp become available. Normally, this happens when the calculator has
finished processing the previous frame, and each of the calculators producing
its inputs have finished processing the current frame. The MediaPipe scheduler
invokes each calculator as soon as these conditions are met. See
[Synchronization](synchronization.md) for more details.
## Timestamp bounds
When a calculator does not produce any output packets for a given timestamp, it
can instead output a "timestamp bound" indicating that no packet will be
produced for that timestamp. This indication is necessary to allow downstream
calculators to run at that timestamp, even though no packet has arrived for
certain streams for that timestamp. This is especially important for real-time
graphs in interactive applications, where it is crucial that each calculator
begin processing as soon as possible.
Consider a graph like the following:
```
node {
calculator: "A"
input_stream: "alpha_in"
output_stream: "alpha"
}
node {
calculator: "B"
input_stream: "alpha"
input_stream: "foo"
output_stream: "beta"
}
```
Suppose: at timestamp `T`, node `A` doesn't send a packet in its output stream
`alpha`. Node `B` gets a packet in `foo` at timestamp `T` and is waiting for a
packet in `alpha` at timestamp `T`. If `A` doesn't send `B` a timestamp bound
update for `alpha`, `B` will keep waiting for a packet to arrive in `alpha`.
Meanwhile, the packet queue of `foo` will accumulate packets at `T`, `T+1` and
so on.
To output a packet on a stream, a calculator uses the API functions
`CalculatorContext::Outputs` and `OutputStream::Add`. To instead output a
timestamp bound on a stream, a calculator can use the API functions
`CalculatorContext::Outputs` and `CalculatorContext::SetNextTimestampBound`. The
specified bound is the lowest allowable timestamp for the next packet on the
specified output stream. When no packet is output, a calculator will typically
do something like:
```
cc->Outputs().Tag("output_frame").SetNextTimestampBound(
cc->InputTimestamp().NextAllowedInStream());
```
The function `Timestamp::NextAllowedInStream` returns the successive timestamp.
For example, `Timestamp(1).NextAllowedInStream() == Timestamp(2)`.
## Propagating timestamp bounds
Calculators that will be used in real-time graphs need to define output
timestamp bounds based on input timestamp bounds in order to allow downstream
calculators to be scheduled promptly. A common pattern is for calculators to
output packets with the same timestamps as their input packets. In this case,
simply outputting a packet on every call to `Calculator::Process` is sufficient
to define output timestamp bounds.
However, calculators are not required to follow this common pattern for output
timestamps, they are only required to choose monotonically increasing output
timestamps. As a result, certain calculators must calculate timestamp bounds
explicitly. MediaPipe provides several tools for computing appropriate timestamp
bound for each calculator.
1\. **SetNextTimestampBound()** can be used to specify the timestamp bound, `t +
1`, for an output stream.
```
cc->Outputs.Tag("OUT").SetNextTimestampBound(t.NextAllowedInStream());
```
Alternatively, an empty packet with timestamp `t` can be produced to specify the
timestamp bound `t + 1`.
```
cc->Outputs.Tag("OUT").Add(Packet(), t);
```
The timestamp bound of an input stream is indicated by the packet or the empty
packet on the input stream.
```
Timestamp bound = cc->Inputs().Tag("IN").Value().Timestamp();
```
2\. **TimestampOffset()** can be specified in order to automatically copy the
timestamp bound from input streams to output streams.
```
cc->SetTimestampOffset(0);
```
This setting has the advantage of propagating timestamp bounds automatically,
even when only timestamp bounds arrive and Calculator::Process is not invoked.
3\. **ProcessTimestampBounds()** can be specified in order to invoke
`Calculator::Process` for each new "settled timestamp", where the "settled
timestamp" is the new highest timestamp below the current timestamp bounds.
Without `ProcessTimestampBounds()`, `Calculator::Process` is invoked only with
one or more arriving packets.
```
cc->SetProcessTimestampBounds(true);
```
This setting allows a calculator to perform its own timestamp bounds calculation
and propagation, even when only input timestamps are updated. It can be used to
replicate the effect of `TimestampOffset()`, but it can also be used to
calculate a timestamp bound that takes into account additional factors.
For example, in order to replicate `SetTimestampOffset(0)`, a calculator could
do the following:
```
absl::Status Open(CalculatorContext* cc) {
cc->SetProcessTimestampBounds(true);
}
absl::Status Process(CalculatorContext* cc) {
cc->Outputs.Tag("OUT").SetNextTimestampBound(
cc->InputTimestamp().NextAllowedInStream());
}
```
## Scheduling of Calculator::Open and Calculator::Close
`Calculator::Open` is invoked when all required input side-packets have been
produced. Input side-packets can be provided by the enclosing application or by
"side-packet calculators" inside the graph. Side-packets can be specified from
outside the graph using the API's `CalculatorGraph::Initialize` and
`CalculatorGraph::StartRun`. Side packets can be specified by calculators within
the graph using `CalculatorGraphConfig::OutputSidePackets` and
`OutputSidePacket::Set`.
Calculator::Close is invoked when all of the input streams have become `Done` by
being closed or reaching timestamp bound `Timestamp::Done`.
**Note:** If the graph finishes all pending calculator execution and becomes
`Done`, before some streams become `Done`, then MediaPipe will invoke the
remaining calls to `Calculator::Close`, so that every calculator can produce its
final outputs.
The use of `TimestampOffset` has some implications for `Calculator::Close`. A
calculator specifying `SetTimestampOffset(0)` will by design signal that all of
its output streams have reached `Timestamp::Done` when all of its input streams
have reached `Timestamp::Done`, and therefore no further outputs are possible.
This prevents such a calculator from emitting any packets during
`Calculator::Close`. If a calculator needs to produce a summary packet during
`Calculator::Close`, `Calculator::Process` must specify timestamp bounds such
that at least one timestamp (such as `Timestamp::Max`) remains available during
`Calculator::Close`. This means that such a calculator normally cannot rely upon
`SetTimestampOffset(0)` and must instead specify timestamp bounds explicitly
using `SetNextTimestampBounds()`.
+2 -2
View File
@@ -169,7 +169,7 @@ behavior depending on resource constraints.
[`CalculatorBase`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h [`CalculatorBase`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h
[`DefaultInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/default_input_stream_handler.h [`DefaultInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/default_input_stream_handler.h
[`SyncSetInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/sync_set_input_stream_handler.h [`SyncSetInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/sync_set_input_stream_handler.cc
[`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.h [`ImmediateInputStreamHandler`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/stream_handler/immediate_input_stream_handler.cc
[`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto [`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto
[`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc [`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc
+18 -122
View File
@@ -16,19 +16,21 @@ nav_order: 1
Please follow instructions below to build Android example apps in the supported Please follow instructions below to build Android example apps in the supported
MediaPipe [solutions](../solutions/solutions.md). To learn more about these MediaPipe [solutions](../solutions/solutions.md). To learn more about these
example apps, start from [Hello World! on Android](./hello_world_android.md). To example apps, start from [Hello World! on Android](./hello_world_android.md).
incorporate MediaPipe into an existing Android Studio project, see these
[instructions](./android_archive_library.md) that use Android Archive (AAR) and
Gradle.
## Building Android example apps To incorporate MediaPipe into Android Studio projects, see these
[instructions](./android_solutions.md) to use the MediaPipe Android Solution
APIs (currently in alpha) that are now available in
[Google's Maven Repository](https://maven.google.com/web/index.html?#com.google.mediapipe).
## Building Android example apps with Bazel
### Prerequisite ### Prerequisite
* Install MediaPipe following these [instructions](./install.md). * Install MediaPipe following these [instructions](./install.md).
* Setup Java Runtime. * Setup Java Runtime.
* Setup Android SDK release 28.0.3 and above. * Setup Android SDK release 30.0.0 and above.
* Setup Android NDK r18b and above. * Setup Android NDK version between 18 and 21.
MediaPipe recommends setting up Android SDK and NDK via Android Studio (and see MediaPipe recommends setting up Android SDK and NDK via Android Studio (and see
below for Android Studio setup). However, if you prefer using MediaPipe without below for Android Studio setup). However, if you prefer using MediaPipe without
@@ -45,22 +47,21 @@ export ANDROID_HOME=<path to the Android SDK>
export ANDROID_NDK_HOME=<path to the Android NDK> export ANDROID_NDK_HOME=<path to the Android NDK>
``` ```
and add android_ndk_repository() and android_sdk_repository() rules into the
[`WORKSPACE`](https://github.com/google/mediapipe/blob/master/WORKSPACE) file as
the following:
```bash
$ echo "android_sdk_repository(name = \"androidsdk\")" >> WORKSPACE
$ echo "android_ndk_repository(name = \"androidndk\", api_level=21)" >> WORKSPACE
```
In order to use MediaPipe on earlier Android versions, MediaPipe needs to switch In order to use MediaPipe on earlier Android versions, MediaPipe needs to switch
to a lower Android API level. You can achieve this by specifying `api_level = to a lower Android API level. You can achieve this by specifying `api_level =
$YOUR_INTENDED_API_LEVEL` in android_ndk_repository() and/or $YOUR_INTENDED_API_LEVEL` in android_ndk_repository() and/or
android_sdk_repository() in the android_sdk_repository() in the
[`WORKSPACE`](https://github.com/google/mediapipe/blob/master/WORKSPACE) file. [`WORKSPACE`](https://github.com/google/mediapipe/blob/master/WORKSPACE) file.
Please verify all the necessary packages are installed.
* Android SDK Platform API Level 28 or 29
* Android SDK Build-Tools 28 or 29
* Android SDK Platform-Tools 28 or 29
* Android SDK Tools 26.1.1
* Android NDK 17c or above
### Option 1: Build with Bazel in Command Line
Tip: You can run this Tip: You can run this
[script](https://github.com/google/mediapipe/blob/master/build_android_examples.sh) [script](https://github.com/google/mediapipe/blob/master/build_android_examples.sh)
to build (and install) all MediaPipe Android example apps. to build (and install) all MediaPipe Android example apps.
@@ -84,108 +85,3 @@ to build (and install) all MediaPipe Android example apps.
```bash ```bash
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
``` ```
### Option 2: Build with Bazel in Android Studio
The MediaPipe project can be imported into Android Studio using the Bazel
plugins. This allows the MediaPipe examples to be built and modified in Android
Studio.
To incorporate MediaPipe into an existing Android Studio project, see these
[instructions](./android_archive_library.md) that use Android Archive (AAR) and
Gradle.
The steps below use Android Studio 3.5 to build and install a MediaPipe example
app:
1. Install and launch Android Studio 3.5.
2. Select `Configure` -> `SDK Manager` -> `SDK Platforms`.
* Verify that Android SDK Platform API Level 28 or 29 is installed.
* Take note of the Android SDK Location, e.g.,
`/usr/local/home/Android/Sdk`.
3. Select `Configure` -> `SDK Manager` -> `SDK Tools`.
* Verify that Android SDK Build-Tools 28 or 29 is installed.
* Verify that Android SDK Platform-Tools 28 or 29 is installed.
* Verify that Android SDK Tools 26.1.1 is installed.
* Verify that Android NDK 17c or above is installed.
* Take note of the Android NDK Location, e.g.,
`/usr/local/home/Android/Sdk/ndk-bundle` or
`/usr/local/home/Android/Sdk/ndk/20.0.5594570`.
4. Set environment variables `$ANDROID_HOME` and `$ANDROID_NDK_HOME` to point
to the installed SDK and NDK.
```bash
export ANDROID_HOME=/usr/local/home/Android/Sdk
# If the NDK libraries are installed by a previous version of Android Studio, do
export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk-bundle
# If the NDK libraries are installed by Android Studio 3.5, do
export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk/<version number>
```
5. Select `Configure` -> `Plugins` to install `Bazel`.
6. On Linux, select `File` -> `Settings` -> `Bazel settings`. On macos, select
`Android Studio` -> `Preferences` -> `Bazel settings`. Then, modify `Bazel
binary location` to be the same as the output of `$ which bazel`.
7. Select `Import Bazel Project`.
* Select `Workspace`: `/path/to/mediapipe` and select `Next`.
* Select `Generate from BUILD file`: `/path/to/mediapipe/BUILD` and select
`Next`.
* Modify `Project View` to be the following and select `Finish`.
```
directories:
# read project settings, e.g., .bazelrc
.
-mediapipe/objc
-mediapipe/examples/ios
targets:
//mediapipe/examples/android/...:all
//mediapipe/java/...:all
android_sdk_platform: android-29
sync_flags:
--host_crosstool_top=@bazel_tools//tools/cpp:toolchain
```
8. Select `Bazel` -> `Sync` -> `Sync project with Build files`.
Note: Even after doing step 4, if you still see the error: `"no such package
'@androidsdk//': Either the path attribute of android_sdk_repository or the
ANDROID_HOME environment variable must be set."`, please modify the
[`WORKSPACE`](https://github.com/google/mediapipe/blob/master/WORKSPACE)
file to point to your SDK and NDK library locations, as below:
```
android_sdk_repository(
name = "androidsdk",
path = "/path/to/android/sdk"
)
android_ndk_repository(
name = "androidndk",
path = "/path/to/android/ndk"
)
```
9. Connect an Android device to the workstation.
10. Select `Run...` -> `Edit Configurations...`.
* Select `Templates` -> `Bazel Command`.
* Enter Target Expression:
`//mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu`
* Enter Bazel command: `mobile-install`.
* Enter Bazel flags: `-c opt --config=android_arm64`.
* Press the `[+]` button to add the new configuration.
* Select `Run` to run the example app on the connected Android device.
+51 -40
View File
@@ -3,7 +3,7 @@ layout: default
title: MediaPipe Android Archive title: MediaPipe Android Archive
parent: MediaPipe on Android parent: MediaPipe on Android
grand_parent: Getting Started grand_parent: Getting Started
nav_order: 2 nav_order: 3
--- ---
# MediaPipe Android Archive # MediaPipe Android Archive
@@ -37,7 +37,7 @@ each project.
load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar") load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")
mediapipe_aar( mediapipe_aar(
name = "mp_face_detection_aar", name = "mediapipe_face_detection",
calculators = ["//mediapipe/graphs/face_detection:mobile_calculators"], calculators = ["//mediapipe/graphs/face_detection:mobile_calculators"],
) )
``` ```
@@ -45,26 +45,49 @@ each project.
2. Run the Bazel build command to generate the AAR. 2. Run the Bazel build command to generate the AAR.
```bash ```bash
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \ bazel build -c opt --strip=ALWAYS \
--fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS \ --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
//path/to/the/aar/build/file:aar_name --fat_apk_cpu=arm64-v8a,armeabi-v7a \
--legacy_whole_archive=0 \
--features=-legacy_whole_archive \
--copt=-fvisibility=hidden \
--copt=-ffunction-sections \
--copt=-fdata-sections \
--copt=-fstack-protector \
--copt=-Oz \
--copt=-fomit-frame-pointer \
--copt=-DABSL_MIN_LOG_LEVEL=2 \
--linkopt=-Wl,--gc-sections,--strip-all \
//path/to/the/aar/build/file:aar_name.aar
``` ```
For the face detection AAR target we made in the step 1, run: For the face detection AAR target we made in step 1, run:
```bash ```bash
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a \ bazel build -c opt --strip=ALWAYS \
//mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \
--fat_apk_cpu=arm64-v8a,armeabi-v7a \
--legacy_whole_archive=0 \
--features=-legacy_whole_archive \
--copt=-fvisibility=hidden \
--copt=-ffunction-sections \
--copt=-fdata-sections \
--copt=-fstack-protector \
--copt=-Oz \
--copt=-fomit-frame-pointer \
--copt=-DABSL_MIN_LOG_LEVEL=2 \
--linkopt=-Wl,--gc-sections,--strip-all \
//mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mediapipe_face_detection.aar
# It should print: # It should print:
# Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mp_face_detection_aar up-to-date: # Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example:mediapipe_face_detection.aar up-to-date:
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar # bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mediapipe_face_detection.aar
``` ```
3. (Optional) Save the AAR to your preferred location. 3. (Optional) Save the AAR to your preferred location.
```bash ```bash
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mediapipe_face_detection.aar
/absolute/path/to/your/preferred/location /absolute/path/to/your/preferred/location
``` ```
@@ -75,11 +98,11 @@ each project.
2. Copy the AAR into app/libs. 2. Copy the AAR into app/libs.
```bash ```bash
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mp_face_detection_aar.aar cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/mediapipe_face_detection.aar
/path/to/your/app/libs/ /path/to/your/app/libs/
``` ```
![Screenshot](../images/mobile/aar_location.png) ![Screenshot](https://mediapipe.dev/images/mobile/aar_location.png)
3. Make app/src/main/assets and copy assets (graph, model, and etc) into 3. Make app/src/main/assets and copy assets (graph, model, and etc) into
app/src/main/assets. app/src/main/assets.
@@ -89,32 +112,17 @@ each project.
and copy and copy
[the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41) [the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41)
and and
[the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite). [the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite).
```bash ```bash
bazel build -c opt mediapipe/mediapipe/graphs/face_detection:mobile_gpu_binary_graph bazel build -c opt mediapipe/graphs/face_detection:face_detection_mobile_gpu_binary_graph
cp bazel-bin/mediapipe/graphs/face_detection/mobile_gpu.binarypb /path/to/your/app/src/main/assets/ cp bazel-bin/mediapipe/graphs/face_detection/face_detection_mobile_gpu.binarypb /path/to/your/app/src/main/assets/
cp mediapipe/modules/face_detection/face_detection_front.tflite /path/to/your/app/src/main/assets/ cp mediapipe/modules/face_detection/face_detection_short_range.tflite /path/to/your/app/src/main/assets/
``` ```
![Screenshot](../images/mobile/assets_location.png) ![Screenshot](https://mediapipe.dev/images/mobile/assets_location.png)
4. Make app/src/main/jniLibs and copy OpenCV JNI libraries into 4. Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR.
app/src/main/jniLibs.
MediaPipe depends on OpenCV, you will need to copy the precompiled OpenCV so
files into app/src/main/jniLibs. You can download the official OpenCV
Android SDK from
[here](https://github.com/opencv/opencv/releases/download/3.4.3/opencv-3.4.3-android-sdk.zip)
and run:
```bash
cp -R ~/Downloads/OpenCV-android-sdk/sdk/native/libs/arm* /path/to/your/app/src/main/jniLibs/
```
![Screenshot](../images/mobile/android_studio_opencv_location.png)
5. Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR.
``` ```
dependencies { dependencies {
@@ -125,21 +133,24 @@ each project.
androidTestImplementation 'androidx.test.ext:junit:1.1.0' androidTestImplementation 'androidx.test.ext:junit:1.1.0'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.1.1' androidTestImplementation 'androidx.test.espresso:espresso-core:3.1.1'
// MediaPipe deps // MediaPipe deps
implementation 'com.google.flogger:flogger:0.3.1' implementation 'com.google.flogger:flogger:latest.release'
implementation 'com.google.flogger:flogger-system-backend:0.3.1' implementation 'com.google.flogger:flogger-system-backend:latest.release'
implementation 'com.google.code.findbugs:jsr305:3.0.2' implementation 'com.google.code.findbugs:jsr305:latest.release'
implementation 'com.google.guava:guava:27.0.1-android' implementation 'com.google.guava:guava:27.0.1-android'
implementation 'com.google.guava:guava:27.0.1-android' implementation 'com.google.protobuf:protobuf-javalite:3.19.1'
implementation 'com.google.protobuf:protobuf-java:3.11.4'
// CameraX core library // CameraX core library
def camerax_version = "1.0.0-beta10" def camerax_version = "1.0.0-beta10"
implementation "androidx.camera:camera-core:$camerax_version" implementation "androidx.camera:camera-core:$camerax_version"
implementation "androidx.camera:camera-camera2:$camerax_version" implementation "androidx.camera:camera-camera2:$camerax_version"
implementation "androidx.camera:camera-lifecycle:$camerax_version" implementation "androidx.camera:camera-lifecycle:$camerax_version"
// AutoValue
def auto_value_version = "1.8.1"
implementation "com.google.auto.value:auto-value-annotations:$auto_value_version"
annotationProcessor "com.google.auto.value:auto-value:$auto_value_version"
} }
``` ```
6. Follow our Android app examples to use MediaPipe in Android Studio for your 5. Follow our Android app examples to use MediaPipe in Android Studio for your
use case. If you are looking for an example, a face detection example can be use case. If you are looking for an example, a face detection example can be
found found
[here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and [here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and
+131
View File
@@ -0,0 +1,131 @@
---
layout: default
title: MediaPipe Android Solutions
parent: MediaPipe on Android
grand_parent: Getting Started
nav_order: 2
---
# MediaPipe Android Solutions
{: .no_toc }
1. TOC
{:toc}
---
MediaPipe Android Solution APIs (currently in alpha) are available in:
* [MediaPipe Face Detection](../solutions/face_detection#android-solution-api)
* [MediaPipe Face Mesh](../solutions/face_mesh#android-solution-api)
* [MediaPipe Hands](../solutions/hands#android-solution-api)
## Incorporation in Android Studio
Prebuilt packages of Android Solution APIs can be found in
[Google's Maven Repository](https://maven.google.com/web/index.html?#com.google.mediapipe).
To incorporate them into an Android Studio project, add the following into the
project's Gradle dependencies:
```
dependencies {
// MediaPipe solution-core is the foundation of any MediaPipe Solutions.
implementation 'com.google.mediapipe:solution-core:latest.release'
// Optional: MediaPipe Face Detection Solution.
implementation 'com.google.mediapipe:facedetection:latest.release'
// Optional: MediaPipe Face Mesh Solution.
implementation 'com.google.mediapipe:facemesh:latest.release'
// Optional: MediaPipe Hands Solution.
implementation 'com.google.mediapipe:hands:latest.release'
}
```
If you need further customization, instead of using the prebuilt maven packages
consider building a MediaPipe Android Archive library locally from source by
following these [instructions](./android_archive_library.md).
## Building solution example apps
Detailed usage examples of the Android Solution APIs can be found in the
[source code](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/solutions)
of the solution example apps.
To build these apps:
1. Open Android Studio Arctic Fox on Linux, macOS, or Windows.
2. Import mediapipe/examples/android/solutions directory into Android Studio.
![Screenshot](https://mediapipe.dev/images/import_mp_android_studio_project.png)
3. For Windows users, run `create_win_symlinks.bat` as administrator to create
res directory symlinks.
![Screenshot](https://mediapipe.dev/images/run_create_win_symlinks.png)
4. Select "File" -> "Sync Project with Gradle Files" to sync project.
5. Run solution example app in Android Studio.
![Screenshot](https://mediapipe.dev/images/run_android_solution_app.png)
6. (Optional) Run solutions on CPU.
MediaPipe solution example apps run the pipeline and model inference on GPU
by default. If needed, for example to run the apps on Android Emulator, set
the `RUN_ON_GPU` boolean variable to `false` in the app's
`MainActivity.java` to run the pipeline and model inference on CPU.
## MediaPipe Solution APIs Terms of Service
Last modified: November 12, 2021
Use of MediaPipe Solution APIs is subject to the
[Google APIs Terms of Service](https://developers.google.com/terms),
[Google API Services User Data Policy](https://developers.google.com/terms/api-services-user-data-policy),
and the terms below. Please check back from time to time as these terms and
policies are occasionally updated.
**Privacy**
When you use MediaPipe Solution APIs, processing of the input data (e.g. images,
video, text) fully happens on-device, and **MediaPipe does not send that input
data to Google servers**. As a result, you can use our APIs for processing data
that should not leave the device.
MediaPipe Android Solution APIs will contact Google servers from time to time in
order to receive things like bug fixes, updated models, and hardware accelerator
compatibility information. MediaPipe Android Solution APIs also send metrics
about the performance and utilization of the APIs in your app to Google. Google
uses this metrics data to measure performance, API usage, debug, maintain and
improve the APIs, and detect misuse or abuse, as further described in our
[Privacy Policy](https://policies.google.com/privacy).
**You are responsible for obtaining informed consent from your app users about
Googles processing of MediaPipe metrics data as required by applicable law.**
Data we collect may include the following, across all MediaPipe Android Solution
APIs:
- Device information (such as manufacturer, model, OS version and build) and
available ML hardware accelerators (GPU and DSP). Used for diagnostics and
usage analytics.
- App identification information (package name / bundle id, app version). Used
for diagnostics and usage analytics.
- API configuration (such as image format, resolution, and MediaPipe version
used). Used for diagnostics and usage analytics.
- Event type (such as initialize, download model, update, run, and detection).
Used for diagnostics and usage analytics.
- Error codes. Used for diagnostics.
- Performance metrics. Used for diagnostics.
- Per-installation identifiers that do not uniquely identify a user or
physical device. Used for operation of remote configuration and usage
analytics.
- Network request sender IP addresses. Used for remote configuration
diagnostics. Collected IP addresses are retained temporarily.
+1 -1
View File
@@ -103,7 +103,7 @@ monotonically increasing timestamps. By convention, realtime calculators and
graphs use the recording time or the presentation time as the timestamp for each graphs use the recording time or the presentation time as the timestamp for each
packet, with each timestamp representing microseconds since packet, with each timestamp representing microseconds since
`Jan/1/1970:00:00:00`. This allows packets from various sources to be processed `Jan/1/1970:00:00:00`. This allows packets from various sources to be processed
in a gloablly consistent order. in a globally consistent order.
Normally for offline processing, every input packet is processed and processing Normally for offline processing, every input packet is processed and processing
continues as long as necessary. For online processing, it is often necessary to continues as long as necessary. For online processing, it is often necessary to
+15
View File
@@ -59,6 +59,21 @@ OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions: OpenGL ES profile extensions:
``` ```
If you have connected to your computer through SSH and find when you probe for
GPU information you see the output:
```bash
glxinfo | grep -i opengl
Error: unable to open display
```
Try re-establishing your SSH connection with the `-X` option and try again. For
example:
```bash
ssh -X <user>@<host>
```
*Notice the ES 3.20 text above.* *Notice the ES 3.20 text above.*
You need to see ES 3.1 or greater printed in order to perform TFLite inference You need to see ES 3.1 or greater printed in order to perform TFLite inference
+7 -8
View File
@@ -27,12 +27,12 @@ graph on Android.
A simple camera app for real-time Sobel edge detection applied to a live video A simple camera app for real-time Sobel edge detection applied to a live video
stream on an Android device. stream on an Android device.
![edge_detection_android_gpu_gif](../images/mobile/edge_detection_android_gpu.gif) ![edge_detection_android_gpu_gif](https://mediapipe.dev/images/mobile/edge_detection_android_gpu.gif)
## Setup ## Setup
1. Install MediaPipe on your system, see [MediaPipe installation guide] for 1. Install MediaPipe on your system, see
details. [MediaPipe installation guide](./install.md) for details.
2. Install Android Development SDK and Android NDK. See how to do so also in 2. Install Android Development SDK and Android NDK. See how to do so also in
[MediaPipe installation guide]. [MediaPipe installation guide].
3. Enable [developer options] on your Android device. 3. Enable [developer options] on your Android device.
@@ -69,7 +69,7 @@ node: {
A visualization of the graph is shown below: A visualization of the graph is shown below:
![edge_detection_mobile_gpu](../images/mobile/edge_detection_mobile_gpu.png) ![edge_detection_mobile_gpu](https://mediapipe.dev/images/mobile/edge_detection_mobile_gpu.png)
This graph has a single input stream named `input_video` for all incoming frames This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera. that will be provided by your device's camera.
@@ -260,7 +260,7 @@ adb install bazel-bin/$APPLICATION_PATH/helloworld.apk
Open the application on your device. It should display a screen with the text Open the application on your device. It should display a screen with the text
`Hello World!`. `Hello World!`.
![bazel_hello_world_android](../images/mobile/bazel_hello_world_android.png) ![bazel_hello_world_android](https://mediapipe.dev/images/mobile/bazel_hello_world_android.png)
## Using the camera via `CameraX` ## Using the camera via `CameraX`
@@ -377,7 +377,7 @@ Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file:
When the user doesn't grant camera permission, the screen will now look like When the user doesn't grant camera permission, the screen will now look like
this: this:
![missing_camera_permission_android](../images/mobile/missing_camera_permission_android.png) ![missing_camera_permission_android](https://mediapipe.dev/images/mobile/missing_camera_permission_android.png)
Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
`MainActivity`: `MainActivity`:
@@ -753,7 +753,7 @@ And that's it! You should now be able to successfully build and run the
application on the device and see Sobel edge detection running on a live camera application on the device and see Sobel edge detection running on a live camera
feed! Congrats! feed! Congrats!
![edge_detection_android_gpu_gif](../images/mobile/edge_detection_android_gpu.gif) ![edge_detection_android_gpu_gif](https://mediapipe.dev/images/mobile/edge_detection_android_gpu.gif)
If you ran into any issues, please see the full code of the tutorial If you ran into any issues, please see the full code of the tutorial
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic). [here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic).
@@ -770,7 +770,6 @@ If you ran into any issues, please see the full code of the tutorial
[`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java [`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java
[`FrameLayout`]:https://developer.android.com/reference/android/widget/FrameLayout [`FrameLayout`]:https://developer.android.com/reference/android/widget/FrameLayout
[`FrameProcessor`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/FrameProcessor.java [`FrameProcessor`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/FrameProcessor.java
[MediaPipe installation guide]:./install.md
[`PermissionHelper`]: https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/PermissionHelper.java [`PermissionHelper`]: https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/PermissionHelper.java
[`SurfaceHolder.Callback`]:https://developer.android.com/reference/android/view/SurfaceHolder.Callback.html [`SurfaceHolder.Callback`]:https://developer.android.com/reference/android/view/SurfaceHolder.Callback.html
[`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView [`SurfaceView`]:https://developer.android.com/reference/android/view/SurfaceView
+1 -1
View File
@@ -85,7 +85,7 @@ nav_order: 1
This graph consists of 1 graph input stream (`in`) and 1 graph output stream This graph consists of 1 graph input stream (`in`) and 1 graph output stream
(`out`), and 2 [`PassThroughCalculator`]s connected serially. (`out`), and 2 [`PassThroughCalculator`]s connected serially.
![hello_world graph](../images/hello_world.png) ![hello_world graph](https://mediapipe.dev/images/hello_world.png)
4. Before running the graph, an `OutputStreamPoller` object is connected to the 4. Before running the graph, an `OutputStreamPoller` object is connected to the
output stream in order to later retrieve the graph output, and a graph run output stream in order to later retrieve the graph output, and a graph run
+36 -9
View File
@@ -27,12 +27,12 @@ on iOS.
A simple camera app for real-time Sobel edge detection applied to a live video A simple camera app for real-time Sobel edge detection applied to a live video
stream on an iOS device. stream on an iOS device.
![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif) ![edge_detection_ios_gpu_gif](https://mediapipe.dev/images/mobile/edge_detection_ios_gpu.gif)
## Setup ## Setup
1. Install MediaPipe on your system, see [MediaPipe installation guide] for 1. Install MediaPipe on your system, see
details. [MediaPipe installation guide](./install.md) for details.
2. Setup your iOS device for development. 2. Setup your iOS device for development.
3. Setup [Bazel] on your system to build and deploy the iOS app. 3. Setup [Bazel] on your system to build and deploy the iOS app.
@@ -67,7 +67,7 @@ node: {
A visualization of the graph is shown below: A visualization of the graph is shown below:
![edge_detection_mobile_gpu](../images/mobile/edge_detection_mobile_gpu.png) ![edge_detection_mobile_gpu](https://mediapipe.dev/images/mobile/edge_detection_mobile_gpu.png)
This graph has a single input stream named `input_video` for all incoming frames This graph has a single input stream named `input_video` for all incoming frames
that will be provided by your device's camera. that will be provided by your device's camera.
@@ -113,6 +113,10 @@ bazel to build the iOS application. The content of the
5. `Main.storyboard` and `Launch.storyboard` 5. `Main.storyboard` and `Launch.storyboard`
6. `Assets.xcassets` directory. 6. `Assets.xcassets` directory.
Note: In newer versions of Xcode, you may see additional files `SceneDelegate.h`
and `SceneDelegate.m`. Make sure to copy them too and add them to the `BUILD`
file mentioned below.
Copy these files to a directory named `HelloWorld` to a location that can access Copy these files to a directory named `HelloWorld` to a location that can access
the MediaPipe source code. For example, the source code of the application that the MediaPipe source code. For example, the source code of the application that
we will build in this tutorial is located in we will build in this tutorial is located in
@@ -127,7 +131,7 @@ Create a `BUILD` file in the `$APPLICATION_PATH` and add the following build
rules: rules:
``` ```
MIN_IOS_VERSION = "10.0" MIN_IOS_VERSION = "11.0"
load( load(
"@build_bazel_rules_apple//apple:ios.bzl", "@build_bazel_rules_apple//apple:ios.bzl",
@@ -247,6 +251,12 @@ We need to get frames from the `_cameraSource` into our application
`MPPInputSourceDelegate`. So our application `ViewController` can be a delegate `MPPInputSourceDelegate`. So our application `ViewController` can be a delegate
of `_cameraSource`. of `_cameraSource`.
Update the interface definition of `ViewController` accordingly:
```
@interface ViewController () <MPPInputSourceDelegate>
```
To handle camera setup and process incoming frames, we should use a queue To handle camera setup and process incoming frames, we should use a queue
different from the main queue. Add the following to the implementation block of different from the main queue. Add the following to the implementation block of
the `ViewController`: the `ViewController`:
@@ -288,6 +298,12 @@ utility called `MPPLayerRenderer` to display images on the screen. This utility
can be used to display `CVPixelBufferRef` objects, which is the type of the can be used to display `CVPixelBufferRef` objects, which is the type of the
images provided by `MPPCameraInputSource` to its delegates. images provided by `MPPCameraInputSource` to its delegates.
In `ViewController.m`, add the following import line:
```
#import "mediapipe/objc/MPPLayerRenderer.h"
```
To display images of the screen, we need to add a new `UIView` object called To display images of the screen, we need to add a new `UIView` object called
`_liveView` to the `ViewController`. `_liveView` to the `ViewController`.
@@ -411,6 +427,12 @@ Objective-C++.
### Use the graph in `ViewController` ### Use the graph in `ViewController`
In `ViewController.m`, add the following import line:
```
#import "mediapipe/objc/MPPGraph.h"
```
Declare a static constant with the name of the graph, the input stream and the Declare a static constant with the name of the graph, the input stream and the
output stream: output stream:
@@ -549,10 +571,16 @@ method to receive packets on this output stream and display them on the screen:
} }
``` ```
Update the interface definition of `ViewController` with `MPPGraphDelegate`:
```
@interface ViewController () <MPPGraphDelegate, MPPInputSourceDelegate>
```
And that is all! Build and run the app on your iOS device. You should see the And that is all! Build and run the app on your iOS device. You should see the
results of running the edge detection graph on a live video feed. Congrats! results of running the edge detection graph on a live video feed. Congrats!
![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif) ![edge_detection_ios_gpu_gif](https://mediapipe.dev/images/mobile/edge_detection_ios_gpu.gif)
Please note that the iOS examples now use a [common] template app. The code in Please note that the iOS examples now use a [common] template app. The code in
this tutorial is used in the [common] template app. The [helloworld] app has the this tutorial is used in the [common] template app. The [helloworld] app has the
@@ -560,6 +588,5 @@ appropriate `BUILD` file dependencies for the edge detection graph.
[Bazel]:https://bazel.build/ [Bazel]:https://bazel.build/
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/edge_detection/edge_detection_mobile_gpu.pbtxt [`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/edge_detection/edge_detection_mobile_gpu.pbtxt
[MediaPipe installation guide]:./install.md [common]:https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/common
[common]:(https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/common) [helloworld]:https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/helloworld
[helloworld]:(https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/helloworld)
+196 -126
View File
@@ -25,25 +25,11 @@ install --user six`.
## Installing on Debian and Ubuntu ## Installing on Debian and Ubuntu
1. Install Bazel. 1. Install Bazelisk.
Follow the official Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-ubuntu.html) [Bazel documentation](https://docs.bazel.build/versions/master/install-bazelisk.html)
to install Bazel 3.4 or higher. to install Bazelisk.
For Nvidia Jetson and Raspberry Pi devices with aarch64 Linux, Bazel needs
to be built from source:
```bash
# For Bazel 3.4.1
mkdir $HOME/bazel-3.4.1
cd $HOME/bazel-3.4.1
wget https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-dist.zip
sudo apt-get install build-essential openjdk-8-jdk python zip unzip
unzip bazel-3.4.1-dist.zip
env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
sudo cp output/bazel /usr/local/bin/
```
2. Checkout MediaPipe repository. 2. Checkout MediaPipe repository.
@@ -57,104 +43,189 @@ install --user six`.
3. Install OpenCV and FFmpeg. 3. Install OpenCV and FFmpeg.
Option 1. Use package manager tool to install the pre-compiled OpenCV **Option 1**. Use package manager tool to install the pre-compiled OpenCV
libraries. FFmpeg will be installed via libopencv-video-dev. libraries. FFmpeg will be installed via `libopencv-video-dev`.
Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to take OS | OpenCV
option 2 or 3 to install OpenCV 3 or above. -------------------- | ------
Debian 9 (stretch) | 2.4
Debian 10 (buster) | 3.2
Debian 11 (bullseye) | 4.5
Ubuntu 16.04 LTS | 2.4
Ubuntu 18.04 LTS | 3.2
Ubuntu 20.04 LTS | 4.2
Ubuntu 20.04 LTS | 4.2
Ubuntu 21.04 | 4.5
```bash ```bash
$ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \ $ sudo apt-get install -y \
libopencv-calib3d-dev libopencv-features2d-dev \ libopencv-core-dev \
libopencv-imgproc-dev libopencv-video-dev libopencv-highgui-dev \
libopencv-calib3d-dev \
libopencv-features2d-dev \
libopencv-imgproc-dev \
libopencv-video-dev
``` ```
Debian 9 and Ubuntu 18.04 install the packages in MediaPipe's [`opencv_linux.BUILD`] and [`WORKSPACE`] are already configured
`/usr/lib/x86_64-linux-gnu`. MediaPipe's [`opencv_linux.BUILD`] and for OpenCV 2/3 and should work correctly on any architecture:
[`ffmpeg_linux.BUILD`] are configured for this library path. Ubuntu 20.04
may install the OpenCV and FFmpeg packages in `/usr/local`, Please follow
the option 3 below to modify the [`WORKSPACE`], [`opencv_linux.BUILD`] and
[`ffmpeg_linux.BUILD`] files accordingly.
Moreover, for Nvidia Jetson and Raspberry Pi devices with ARM Ubuntu, the
library path needs to be modified like the following:
```bash ```bash
sed -i "s/x86_64-linux-gnu/aarch64-linux-gnu/g" third_party/opencv_linux.BUILD # WORKSPACE
new_local_repository(
name = "linux_opencv",
build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr",
)
# opencv_linux.BUILD for OpenCV 2/3 installed from Debian package
cc_library(
name = "opencv",
linkopts = [
"-l:libopencv_core.so",
"-l:libopencv_calib3d.so",
"-l:libopencv_features2d.so",
"-l:libopencv_highgui.so",
"-l:libopencv_imgcodecs.so",
"-l:libopencv_imgproc.so",
"-l:libopencv_video.so",
"-l:libopencv_videoio.so",
],
)
``` ```
Option 2. Run [`setup_opencv.sh`] to automatically build OpenCV from source For OpenCV 4 you need to modify [`opencv_linux.BUILD`] taking into account
and modify MediaPipe's OpenCV config. current architecture:
Option 3. Follow OpenCV's ```bash
# WORKSPACE
new_local_repository(
name = "linux_opencv",
build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr",
)
# opencv_linux.BUILD for OpenCV 4 installed from Debian package
cc_library(
name = "opencv",
hdrs = glob([
# Uncomment according to your multiarch value (gcc -print-multiarch):
# "include/aarch64-linux-gnu/opencv4/opencv2/cvconfig.h",
# "include/arm-linux-gnueabihf/opencv4/opencv2/cvconfig.h",
# "include/x86_64-linux-gnu/opencv4/opencv2/cvconfig.h",
"include/opencv4/opencv2/**/*.h*",
]),
includes = [
# Uncomment according to your multiarch value (gcc -print-multiarch):
# "include/aarch64-linux-gnu/opencv4/",
# "include/arm-linux-gnueabihf/opencv4/",
# "include/x86_64-linux-gnu/opencv4/",
"include/opencv4/",
],
linkopts = [
"-l:libopencv_core.so",
"-l:libopencv_calib3d.so",
"-l:libopencv_features2d.so",
"-l:libopencv_highgui.so",
"-l:libopencv_imgcodecs.so",
"-l:libopencv_imgproc.so",
"-l:libopencv_video.so",
"-l:libopencv_videoio.so",
],
)
```
**Option 2**. Run [`setup_opencv.sh`] to automatically build OpenCV from
source and modify MediaPipe's OpenCV config. This option will do all steps
defined in Option 3 automatically.
**Option 3**. Follow OpenCV's
[documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html) [documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
to manually build OpenCV from source code. to manually build OpenCV from source code.
Note: You may need to modify [`WORKSPACE`], [`opencv_linux.BUILD`] and You may need to modify [`WORKSPACE`] and [`opencv_linux.BUILD`] to point
[`ffmpeg_linux.BUILD`] to point MediaPipe to your own OpenCV and FFmpeg MediaPipe to your own OpenCV libraries. Assume OpenCV would be installed to
libraries. For example if OpenCV and FFmpeg are both manually installed in `/usr/local/` which is recommended by default.
"/usr/local/", you will need to update: (1) the "linux_opencv" and
"linux_ffmpeg" new_local_repository rules in [`WORKSPACE`], (2) the "opencv" OpenCV 2/3 setup:
cc_library rule in [`opencv_linux.BUILD`], and (3) the "libffmpeg"
cc_library rule in [`ffmpeg_linux.BUILD`]. These 3 changes are shown below:
```bash ```bash
# WORKSPACE
new_local_repository( new_local_repository(
name = "linux_opencv", name = "linux_opencv",
build_file = "@//third_party:opencv_linux.BUILD", build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr/local", path = "/usr/local",
) )
# opencv_linux.BUILD for OpenCV 2/3 installed to /usr/local
cc_library(
name = "opencv",
linkopts = [
"-L/usr/local/lib",
"-l:libopencv_core.so",
"-l:libopencv_calib3d.so",
"-l:libopencv_features2d.so",
"-l:libopencv_highgui.so",
"-l:libopencv_imgcodecs.so",
"-l:libopencv_imgproc.so",
"-l:libopencv_video.so",
"-l:libopencv_videoio.so",
],
)
```
OpenCV 4 setup:
```bash
# WORKSPACE
new_local_repository( new_local_repository(
name = "linux_ffmpeg", name = "linux_opencv",
build_file = "@//third_party:ffmpeg_linux.BUILD", build_file = "@//third_party:opencv_linux.BUILD",
path = "/usr/local", path = "/usr/local",
) )
# opencv_linux.BUILD for OpenCV 4 installed to /usr/local
cc_library( cc_library(
name = "opencv", name = "opencv",
srcs = glob( hdrs = glob([
[ "include/opencv4/opencv2/**/*.h*",
"lib/libopencv_core.so", ]),
"lib/libopencv_highgui.so", includes = [
"lib/libopencv_imgcodecs.so", "include/opencv4/",
"lib/libopencv_imgproc.so", ],
"lib/libopencv_video.so", linkopts = [
"lib/libopencv_videoio.so", "-L/usr/local/lib",
], "-l:libopencv_core.so",
), "-l:libopencv_calib3d.so",
hdrs = glob([ "-l:libopencv_features2d.so",
# For OpenCV 3.x "-l:libopencv_highgui.so",
"include/opencv2/**/*.h*", "-l:libopencv_imgcodecs.so",
# For OpenCV 4.x "-l:libopencv_imgproc.so",
# "include/opencv4/opencv2/**/*.h*", "-l:libopencv_video.so",
]), "-l:libopencv_videoio.so",
includes = [ ],
# For OpenCV 3.x )
"include/", ```
# For OpenCV 4.x
# "include/opencv4/", Current FFmpeg setup is defined in [`ffmpeg_linux.BUILD`] and should work
], for any architecture:
linkstatic = 1,
visibility = ["//visibility:public"], ```bash
# WORKSPACE
new_local_repository(
name = "linux_ffmpeg",
build_file = "@//third_party:ffmpeg_linux.BUILD",
path = "/usr"
) )
# ffmpeg_linux.BUILD for FFmpeg installed from Debian package
cc_library( cc_library(
name = "libffmpeg", name = "libffmpeg",
srcs = glob( linkopts = [
[ "-l:libavcodec.so",
"lib/libav*.so", "-l:libavformat.so",
], "-l:libavutil.so",
), ],
hdrs = glob(["include/libav*/*.h"]),
includes = ["include"],
linkopts = [
"-lavcodec",
"-lavformat",
"-lavutil",
],
linkstatic = 1,
visibility = ["//visibility:public"],
) )
``` ```
@@ -207,11 +278,11 @@ build issues.
**Disclaimer**: Running MediaPipe on CentOS is experimental. **Disclaimer**: Running MediaPipe on CentOS is experimental.
1. Install Bazel. 1. Install Bazelisk.
Follow the official Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-redhat.html) [Bazel documentation](https://docs.bazel.build/versions/master/install-bazelisk.html)
to install Bazel 3.4 or higher. to install Bazelisk.
2. Checkout MediaPipe repository. 2. Checkout MediaPipe repository.
@@ -336,11 +407,11 @@ build issues.
* Install [Xcode](https://developer.apple.com/xcode/) and its Command Line * Install [Xcode](https://developer.apple.com/xcode/) and its Command Line
Tools by `xcode-select --install`. Tools by `xcode-select --install`.
2. Install Bazel. 2. Install Bazelisk.
Follow the official Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-os-x.html#install-with-installer-mac-os-x) [Bazel documentation](https://docs.bazel.build/versions/master/install-bazelisk.html)
to install Bazel 3.4 or higher. to install Bazelisk.
3. Checkout MediaPipe repository. 3. Checkout MediaPipe repository.
@@ -353,7 +424,7 @@ build issues.
4. Install OpenCV and FFmpeg. 4. Install OpenCV and FFmpeg.
Option 1. Use HomeBrew package manager tool to install the pre-compiled Option 1. Use HomeBrew package manager tool to install the pre-compiled
OpenCV 3.4.5 libraries. FFmpeg will be installed via OpenCV. OpenCV 3 libraries. FFmpeg will be installed via OpenCV.
```bash ```bash
$ brew install opencv@3 $ brew install opencv@3
@@ -484,29 +555,36 @@ next section.
4. Install Visual C++ Build Tools 2019 and WinSDK 4. Install Visual C++ Build Tools 2019 and WinSDK
Go to https://visualstudio.microsoft.com/visual-cpp-build-tools, download Go to
build tools, and install Microsoft Visual C++ 2019 Redistributable and [the VisualStudio website](https://visualstudio.microsoft.com/visual-cpp-build-tools),
Microsoft Build Tools 2019. download build tools, and install Microsoft Visual C++ 2019 Redistributable
and Microsoft Build Tools 2019.
Download the WinSDK from Download the WinSDK from
https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/ and [the official MicroSoft website](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk/)
install. and install.
5. Install Bazel and add the location of the Bazel executable to the `%PATH%` 5. Install Bazel or Bazelisk and add the location of the Bazel executable to
environment variable. the `%PATH%` environment variable.
Follow the official Option 1. Follow
[Bazel documentation](https://docs.bazel.build/versions/master/install-windows.html) [the official Bazel documentation](https://docs.bazel.build/versions/master/install-windows.html)
to install Bazel 3.4 or higher. to install Bazel 5.2.0 or higher.
6. Set Bazel variables. Option 2. Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-bazelisk.html)
to install Bazelisk.
6. Set Bazel variables. Learn more details about
["Build on Windows"](https://docs.bazel.build/versions/master/windows.html#build-c-with-msvc)
in the Bazel official documentation.
``` ```
# Find the exact paths and version numbers from your local version. # Please find the exact paths and version numbers from your local version.
C:\> set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools C:\> set BAZEL_VS=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools
C:\> set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC C:\> set BAZEL_VC=C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC
C:\> set BAZEL_VC_FULL_VERSION=14.25.28610 C:\> set BAZEL_VC_FULL_VERSION=<Your local VC version>
C:\> set BAZEL_WINSDK_FULL_VERSION=10.1.18362.1 C:\> set BAZEL_WINSDK_FULL_VERSION=<Your local WinSDK version>
``` ```
7. Checkout MediaPipe repository. 7. Checkout MediaPipe repository.
@@ -579,7 +657,7 @@ cameras. Alternatively, you use a video file as input.
Note: Windows' and WSLs adb versions must be the same version, e.g., if WSL Note: Windows' and WSLs adb versions must be the same version, e.g., if WSL
has ADB 1.0.39, you need to download the corresponding Windows ADB from has ADB 1.0.39, you need to download the corresponding Windows ADB from
[here](https://dl.google.com/android/repository/platform-tools_r26.0.1-windows.zip). [here](https://dl.google.com/android/repository/platform-tools_r30.0.3-windows.zip).
3. Launch WSL. 3. Launch WSL.
@@ -593,19 +671,11 @@ cameras. Alternatively, you use a video file as input.
username@DESKTOP-TMVLBJ1:~$ sudo apt-get update && sudo apt-get install -y build-essential git python zip adb openjdk-8-jdk username@DESKTOP-TMVLBJ1:~$ sudo apt-get update && sudo apt-get install -y build-essential git python zip adb openjdk-8-jdk
``` ```
5. Install Bazel. 5. Install Bazelisk.
```bash Follow the official
username@DESKTOP-TMVLBJ1:~$ curl -sLO --retry 5 --retry-max-time 10 \ [Bazel documentation](https://docs.bazel.build/versions/master/install-bazelisk.html)
https://storage.googleapis.com/bazel/3.4.1/release/bazel-3.4.1-installer-linux-x86_64.sh && \ to install Bazelisk.
sudo mkdir -p /usr/local/bazel/3.4.1 && \
chmod 755 bazel-3.4.1-installer-linux-x86_64.sh && \
sudo ./bazel-3.4.1-installer-linux-x86_64.sh --prefix=/usr/local/bazel/3.4.1 && \
source /usr/local/bazel/3.4.1/lib/bazel/bin/bazel-complete.bash
username@DESKTOP-TMVLBJ1:~$ /usr/local/bazel/3.4.1/lib/bazel/bin/bazel version && \
alias bazel='/usr/local/bazel/3.4.1/lib/bazel/bin/bazel'
```
6. Checkout MediaPipe repository. 6. Checkout MediaPipe repository.
@@ -726,7 +796,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
```bash ```bash
$ docker run -it --name mediapipe mediapipe:latest $ docker run -it --name mediapipe mediapipe:latest
root@bca08b91ff63:/mediapipe# GLOG_logtostderr=1 bazel run --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world:hello_world root@bca08b91ff63:/mediapipe# GLOG_logtostderr=1 bazel run --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hello_world
# Should print: # Should print:
# Hello World! # Hello World!
@@ -753,7 +823,7 @@ common build issues.
root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk.sh root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk.sh
# Should print: # Should print:
# Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r18b # Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r19c
# Set android_ndk_repository and android_sdk_repository in WORKSPACE # Set android_ndk_repository and android_sdk_repository in WORKSPACE
# Done # Done
+10 -2
View File
@@ -32,9 +32,14 @@ example apps, start from, start from
xcode-select --install xcode-select --install
``` ```
3. Install [Bazel](https://bazel.build/). 3. Install [Bazelisk](https://github.com/bazelbuild/bazelisk)
.
We recommend using [Homebrew](https://brew.sh/) to get the latest version. We recommend using [Homebrew](https://brew.sh/) to get the latest versions.
```bash
brew install bazelisk
```
4. Set Python 3.7 as the default Python version and install the Python "six" 4. Set Python 3.7 as the default Python version and install the Python "six"
library. This is needed for TensorFlow. library. This is needed for TensorFlow.
@@ -187,6 +192,9 @@ Note: When you ask Xcode to run an app, by default it will use the Debug
configuration. Some of our demos are computationally heavy; you may want to use configuration. Some of our demos are computationally heavy; you may want to use
the Release configuration for better performance. the Release configuration for better performance.
Note: Due to an imcoptibility caused by one of our dependencies, MediaPipe
cannot be used for apps running on the iPhone Simulator on Apple Silicon (M1).
Tip: To switch build configuration in Xcode, click on the target menu, choose Tip: To switch build configuration in Xcode, click on the target menu, choose
"Edit Scheme...", select the Run action, and switch the Build Configuration from "Edit Scheme...", select the Run action, and switch the Build Configuration from
Debug to Release. Note that this is set independently for each target. Debug to Release. Note that this is set independently for each target.
+28 -20
View File
@@ -16,17 +16,29 @@ nav_order: 4
MediaPipe currently offers the following solutions: MediaPipe currently offers the following solutions:
Solution | NPM Package | Example Solution | NPM Package | Example
----------------- | ----------------------------- | ------- --------------------------- | --------------------------------------- | -------
[Face Mesh][F-pg] | [@mediapipe/face_mesh][F-npm] | [mediapipe.dev/demo/face_mesh][F-demo] [Face Mesh][F-pg] | [@mediapipe/face_mesh][F-npm] | [mediapipe.dev/demo/face_mesh][F-demo]
[Face Detection][Fd-pg] | [@mediapipe/face_detection][Fd-npm] | [mediapipe.dev/demo/face_detection][Fd-demo] [Face Detection][Fd-pg] | [@mediapipe/face_detection][Fd-npm] | [mediapipe.dev/demo/face_detection][Fd-demo]
[Hands][H-pg] | [@mediapipe/hands][H-npm] | [mediapipe.dev/demo/hands][H-demo] [Hands][H-pg] | [@mediapipe/hands][H-npm] | [mediapipe.dev/demo/hands][H-demo]
[Holistic][Ho-pg] | [@mediapipe/holistic][Ho-npm] | [mediapipe.dev/demo/holistic][Ho-demo] [Holistic][Ho-pg] | [@mediapipe/holistic][Ho-npm] | [mediapipe.dev/demo/holistic][Ho-demo]
[Pose][P-pg] | [@mediapipe/pose][P-npm] | [mediapipe.dev/demo/pose][P-demo] [Objectron][Ob-pg] | [@mediapipe/objectron][Ob-npm] | [mediapipe.dev/demo/objectron][Ob-demo]
[Pose][P-pg] | [@mediapipe/pose][P-npm] | [mediapipe.dev/demo/pose][P-demo]
[Selfie Segmentation][S-pg] | [@mediapipe/selfie_segmentation][S-npm] | [mediapipe.dev/demo/selfie_segmentation][S-demo]
Click on a solution link above for more information, including API and code Click on a solution link above for more information, including API and code
snippets. snippets.
### Supported plaforms:
| Browser | Platform | Notes |
| ------- | ----------------------- | -------------------------------------- |
| Chrome | Android / Windows / Mac | Pixel 4 and older unsupported. Fuschia |
| | | unsupported. |
| Chrome | iOS | Camera unavailable in Chrome on iOS. |
| Safari | iPad/iPhone/Mac | iOS and Safari on iPad / iPhone / |
| | | MacBook |
The quickest way to get acclimated is to look at the examples above. Each demo The quickest way to get acclimated is to look at the examples above. Each demo
has a link to a [CodePen][codepen] so that you can edit the code and try it has a link to a [CodePen][codepen] so that you can edit the code and try it
yourself. We have included a number of utility packages to help you get started: yourself. We have included a number of utility packages to help you get started:
@@ -66,29 +78,25 @@ affecting your work, restrict your request to a `<minor>` number. e.g.,
[F-pg]: ../solutions/face_mesh#javascript-solution-api [F-pg]: ../solutions/face_mesh#javascript-solution-api
[Fd-pg]: ../solutions/face_detection#javascript-solution-api [Fd-pg]: ../solutions/face_detection#javascript-solution-api
[H-pg]: ../solutions/hands#javascript-solution-api [H-pg]: ../solutions/hands#javascript-solution-api
[Ob-pg]: ../solutions/objectron#javascript-solution-api
[P-pg]: ../solutions/pose#javascript-solution-api [P-pg]: ../solutions/pose#javascript-solution-api
[S-pg]: ../solutions/selfie_segmentation#javascript-solution-api
[Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic [Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic
[F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh [F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh
[Fd-npm]: https://www.npmjs.com/package/@mediapipe/face_detection [Fd-npm]: https://www.npmjs.com/package/@mediapipe/face_detection
[H-npm]: https://www.npmjs.com/package/@mediapipe/hands [H-npm]: https://www.npmjs.com/package/@mediapipe/hands
[Ob-npm]: https://www.npmjs.com/package/@mediapipe/objectron
[P-npm]: https://www.npmjs.com/package/@mediapipe/pose [P-npm]: https://www.npmjs.com/package/@mediapipe/pose
[draw-npm]: https://www.npmjs.com/package/@mediapipe/pose [S-npm]: https://www.npmjs.com/package/@mediapipe/selfie_segmentation
[cam-npm]: https://www.npmjs.com/package/@mediapipe/pose [draw-npm]: https://www.npmjs.com/package/@mediapipe/drawing_utils
[ctrl-npm]: https://www.npmjs.com/package/@mediapipe/pose [cam-npm]: https://www.npmjs.com/package/@mediapipe/camera_utils
[Ho-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/holistic [ctrl-npm]: https://www.npmjs.com/package/@mediapipe/control_utils
[F-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/face_mesh
[Fd-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/face_detection
[H-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/hands
[P-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/pose
[Ho-pen]: https://code.mediapipe.dev/codepen/holistic
[F-pen]: https://code.mediapipe.dev/codepen/face_mesh
[Fd-pen]: https://code.mediapipe.dev/codepen/face_detection
[H-pen]: https://code.mediapipe.dev/codepen/hands
[P-pen]: https://code.mediapipe.dev/codepen/pose
[Ho-demo]: https://mediapipe.dev/demo/holistic [Ho-demo]: https://mediapipe.dev/demo/holistic
[F-demo]: https://mediapipe.dev/demo/face_mesh [F-demo]: https://mediapipe.dev/demo/face_mesh
[Fd-demo]: https://mediapipe.dev/demo/face_detection [Fd-demo]: https://mediapipe.dev/demo/face_detection
[H-demo]: https://mediapipe.dev/demo/hands [H-demo]: https://mediapipe.dev/demo/hands
[Ob-demo]: https://mediapipe.dev/demo/objectron
[P-demo]: https://mediapipe.dev/demo/pose [P-demo]: https://mediapipe.dev/demo/pose
[S-demo]: https://mediapipe.dev/demo/selfie_segmentation
[npm]: https://www.npmjs.com/package/@mediapipe [npm]: https://www.npmjs.com/package/@mediapipe
[codepen]: https://code.mediapipe.dev/codepen [codepen]: https://code.mediapipe.dev/codepen
+6 -7
View File
@@ -26,7 +26,7 @@ You can, for instance, activate a Python virtual environment:
$ python3 -m venv mp_env && source mp_env/bin/activate $ python3 -m venv mp_env && source mp_env/bin/activate
``` ```
Install MediaPipe Python package and start Python intepreter: Install MediaPipe Python package and start Python interpreter:
```bash ```bash
(mp_env)$ pip install mediapipe (mp_env)$ pip install mediapipe
@@ -51,6 +51,7 @@ details in each solution via the links below:
* [MediaPipe Holistic](../solutions/holistic#python-solution-api) * [MediaPipe Holistic](../solutions/holistic#python-solution-api)
* [MediaPipe Objectron](../solutions/objectron#python-solution-api) * [MediaPipe Objectron](../solutions/objectron#python-solution-api)
* [MediaPipe Pose](../solutions/pose#python-solution-api) * [MediaPipe Pose](../solutions/pose#python-solution-api)
* [MediaPipe Selfie Segmentation](../solutions/selfie_segmentation#python-solution-api)
## MediaPipe on Google Colab ## MediaPipe on Google Colab
@@ -62,6 +63,7 @@ details in each solution via the links below:
* [MediaPipe Pose Colab](https://mediapipe.page.link/pose_py_colab) * [MediaPipe Pose Colab](https://mediapipe.page.link/pose_py_colab)
* [MediaPipe Pose Classification Colab (Basic)](https://mediapipe.page.link/pose_classification_basic) * [MediaPipe Pose Classification Colab (Basic)](https://mediapipe.page.link/pose_classification_basic)
* [MediaPipe Pose Classification Colab (Extended)](https://mediapipe.page.link/pose_classification_extended) * [MediaPipe Pose Classification Colab (Extended)](https://mediapipe.page.link/pose_classification_extended)
* [MediaPipe Selfie Segmentation Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)
## MediaPipe Python Framework ## MediaPipe Python Framework
@@ -111,9 +113,8 @@ Nvidia Jetson and Raspberry Pi, please read
Download the latest protoc win64 zip from Download the latest protoc win64 zip from
[the Protobuf GitHub repo](https://github.com/protocolbuffers/protobuf/releases), [the Protobuf GitHub repo](https://github.com/protocolbuffers/protobuf/releases),
unzip the file, and copy the protoc.exe executable to a preferred unzip the file, and copy the protoc.exe executable to a preferred location.
location. Please ensure that location is added into the Path environment Please ensure that location is added into the Path environment variable.
variable.
3. Activate a Python virtual environment. 3. Activate a Python virtual environment.
@@ -129,16 +130,14 @@ Nvidia Jetson and Raspberry Pi, please read
(mp_env)mediapipe$ pip3 install -r requirements.txt (mp_env)mediapipe$ pip3 install -r requirements.txt
``` ```
6. Generate and install MediaPipe package. 6. Build and install MediaPipe package.
```bash ```bash
(mp_env)mediapipe$ python3 setup.py gen_protos
(mp_env)mediapipe$ python3 setup.py install --link-opencv (mp_env)mediapipe$ python3 setup.py install --link-opencv
``` ```
or or
```bash ```bash
(mp_env)mediapipe$ python3 setup.py gen_protos
(mp_env)mediapipe$ python3 setup.py bdist_wheel (mp_env)mediapipe$ python3 setup.py bdist_wheel
``` ```
+4 -3
View File
@@ -74,7 +74,7 @@ Mapping\[str, Packet\] | std::map<std::string, Packet> | create_st
np.ndarray<br>(cv.mat and PIL.Image) | mp::ImageFrame | create_image_frame(<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;format=ImageFormat.SRGB,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data=mat) | get_image_frame(packet) np.ndarray<br>(cv.mat and PIL.Image) | mp::ImageFrame | create_image_frame(<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;format=ImageFormat.SRGB,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data=mat) | get_image_frame(packet)
np.ndarray | mp::Matrix | create_matrix(data) | get_matrix(packet) np.ndarray | mp::Matrix | create_matrix(data) | get_matrix(packet)
Google Proto Message | Google Proto Message | create_proto(proto) | get_proto(packet) Google Proto Message | Google Proto Message | create_proto(proto) | get_proto(packet)
List\[Proto\] | std::vector\<Proto\> | create_proto_vector(proto_list) | get_proto_list(packet) List\[Proto\] | std::vector\<Proto\> | n/a | get_proto_list(packet)
It's not uncommon that users create custom C++ classes and and send those into It's not uncommon that users create custom C++ classes and and send those into
the graphs and calculators. To allow the custom classes to be used in Python the graphs and calculators. To allow the custom classes to be used in Python
@@ -126,6 +126,7 @@ following steps:
} }
return packet.Get<MyType>(); return packet.Get<MyType>();
}); });
}
} // namespace mediapipe } // namespace mediapipe
``` ```
@@ -249,12 +250,12 @@ three stages: initialization and setup, graph run, and graph shutdown.
graph.start_run() graph.start_run()
graph.add_packet_to_input_stream( graph.add_packet_to_input_stream(
'in_stream', mp.packet_creator.create_str('abc').at(0)) 'in_stream', mp.packet_creator.create_string('abc').at(0))
rgb_img = cv2.cvtColor(cv2.imread('/path/to/your/image.png'), cv2.COLOR_BGR2RGB) rgb_img = cv2.cvtColor(cv2.imread('/path/to/your/image.png'), cv2.COLOR_BGR2RGB)
graph.add_packet_to_input_stream( graph.add_packet_to_input_stream(
'in_stream', 'in_stream',
mp.packet_creator.create_image_frame(format=mp.ImageFormat.SRGB, mp.packet_creator.create_image_frame(image_format=mp.ImageFormat.SRGB,
data=rgb_img).at(1)) data=rgb_img).at(1))
``` ```
+43
View File
@@ -97,6 +97,49 @@ linux_opencv/macos_opencv/windows_opencv.BUILD files for your local opencv
libraries. [This GitHub issue](https://github.com/google/mediapipe/issues/666) libraries. [This GitHub issue](https://github.com/google/mediapipe/issues/666)
may also help. may also help.
## Python pip install failure
The error message:
```
ERROR: Could not find a version that satisfies the requirement mediapipe
ERROR: No matching distribution found for mediapipe
```
after running `pip install mediapipe` usually indicates that there is no qualified MediaPipe Python for your system.
Please note that MediaPipe Python PyPI officially supports the **64-bit**
version of Python 3.7 to 3.10 on the following OS:
- x86_64 Linux
- x86_64 macOS 10.15+
- amd64 Windows
If the OS is currently supported and you still see this error, please make sure
that both the Python and pip binary are for Python 3.7 to 3.10. Otherwise,
please consider building the MediaPipe Python package locally by following the
instructions [here](python.md#building-mediapipe-python-package).
## Python DLL load failure on Windows
The error message:
```
ImportError: DLL load failed: The specified module could not be found
```
usually indicates that the local Windows system is missing Visual C++
redistributable packages and/or Visual C++ runtime DLLs. This can be solved by
either installing the official
[vc_redist.x64.exe](https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0)
or installing the "msvc-runtime" Python package by running
```bash
$ python -m pip install msvc-runtime
```
Please note that the "msvc-runtime" Python package is not released or maintained
by Microsoft.
## Native method not found ## Native method not found
The error message: The error message:
Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 885 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 8.2 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.5 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 39 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 7.7 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 11 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 361 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 5.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 11 KiB

File diff suppressed because one or more lines are too long

Before

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 195 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 524 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 808 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.7 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 923 B

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 217 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 163 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.8 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.6 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 302 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 9.4 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 666 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 529 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 12 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 808 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 121 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 121 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.0 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 59 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 460 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 299 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 3.1 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 383 KiB

Some files were not shown because too many files have changed in this diff Show More