Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 MediaPipe Teamandchuoling
|
c0124fb83c | ||
|
MediaPipe Teamandchuoling
|
a7225b938a | ||
|
MediaPipe Teamandchuoling
|
73f4475c17 | ||
|
MediaPipe Teamandchuoling
|
f57ff46845 | ||
|
MediaPipe Teamandchuoling
|
d7c287c4e9 | ||
|
MediaPipe Teamandchuoling
|
6b0ab0e012 | ||
|
MediaPipe Teamandchuoling
|
2f86a459b6 | ||
|
MediaPipe Teamandchuoling
|
bdfdaef305 |
+1
-1
@@ -12,6 +12,6 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
exports_files(["LICENSE"])
|
||||
+8
-99
@@ -1,15 +1,17 @@
|
||||
# Contributing guidelines
|
||||
|
||||
## Pull Request Checklist
|
||||
## What type of pull request do we accept into MediaPipe repository?
|
||||
|
||||
* Bug fixes
|
||||
* Documentation fixes
|
||||
|
||||
For new feature additions (e.g., new graphs and calculators), we are currently not planning to accept new feature pull requests into the MediaPipe repository. Instead, we like to get contributors to create their own repositories of the new feature and list it at [Awesome MediaPipe](https://mediapipe.org). This will allow contributors to more quickly get their code out to the community.
|
||||
|
||||
Before sending your pull requests, make sure you followed this list.
|
||||
|
||||
- Read [contributing guidelines](CONTRIBUTING.md).
|
||||
- Read [Code of Conduct](CODE_OF_CONDUCT.md).
|
||||
- Ensure you have signed the [Contributor License Agreement (CLA)](https://cla.developers.google.com/).
|
||||
- Check if my changes are consistent with the [guidelines](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md#general-guidelines-and-philosophy-for-contribution).
|
||||
- Changes are consistent with the [Coding Style](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md#c-coding-style).
|
||||
- Run [Unit Tests](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md#running-unit-tests).
|
||||
|
||||
## How to become a contributor and submit your own code
|
||||
|
||||
@@ -28,100 +30,7 @@ Follow either of the two links above to access the appropriate CLA and instructi
|
||||
|
||||
### Contributing code
|
||||
|
||||
If you have improvements to MediaPipe, send us your pull requests! For those
|
||||
If you have bug fixes and documentation fixes to MediaPipe, send us your pull requests! For those
|
||||
just getting started, GitHub has a [howto](https://help.github.com/articles/using-pull-requests/).
|
||||
|
||||
MediaPipe team members will be assigned to review your pull requests. Once the
|
||||
pull requests are approved and pass continuous integration checks, a MediaPipe
|
||||
team member will apply `ready to pull` label to your change. This means we are
|
||||
working on getting your pull request submitted to our internal repository. After
|
||||
the change has been submitted internally, your pull request will be merged
|
||||
automatically on GitHub.
|
||||
|
||||
If you want to contribute but you're not sure where to start, take a look at the
|
||||
[issues with the "contributions welcome" label](https://github.com/google/mediapipe/labels/stat%3Acontributions%20welcome).
|
||||
These are issues that we believe are particularly well suited for outside
|
||||
contributions, often because we probably won't get to them right now. If you
|
||||
decide to start on an issue, leave a comment so that other people know that
|
||||
you're working on it. If you want to help out, but not alone, use the issue
|
||||
comment thread to coordinate.
|
||||
|
||||
### Contribution guidelines and standards
|
||||
|
||||
Before sending your pull request for
|
||||
[review](https://github.com/google/mediapipe/pulls),
|
||||
make sure your changes are consistent with the guidelines and follow the
|
||||
MediaPipe coding style.
|
||||
|
||||
#### General guidelines and philosophy for contribution
|
||||
|
||||
* Include unit tests when you contribute new features, as they help to a)
|
||||
prove that your code works correctly, and b) guard against future breaking
|
||||
changes to lower the maintenance cost.
|
||||
* Bug fixes also generally require unit tests, because the presence of bugs
|
||||
usually indicates insufficient test coverage.
|
||||
* Keep API compatibility in mind when you change code in MediaPipe framework
|
||||
e.g., code in
|
||||
[mediapipe/framework](https://github.com/google/mediapipe/tree/master/mediapipe/framework)
|
||||
and
|
||||
[mediapipe/calculators](https://github.com/google/mediapipe/tree/master/mediapipe/calculators).
|
||||
Once MediaPipe has reached version 1 and we will not make
|
||||
non-backward-compatible API changes without a major release. Reviewers of
|
||||
your pull request will comment on any API compatibility issues.
|
||||
* When you contribute a new feature to MediaPipe, the maintenance burden is
|
||||
(by default) transferred to the MediaPipe team. This means that benefit of
|
||||
the contribution must be compared against the cost of maintaining the
|
||||
feature.
|
||||
* Full new features (e.g., a new op implementing a cutting-edge algorithm)
|
||||
typically will live in
|
||||
[mediapipe/addons](https://github.com/google/mediapipe/addons) to get some
|
||||
airtime before decision is made regarding whether they are to be migrated to
|
||||
the core.
|
||||
|
||||
#### License
|
||||
|
||||
Include a license at the top of new files.
|
||||
|
||||
* [C/C++ license example](https://github.com/google/mediapipe/blob/master/mediapipe/framework/calculator_base.cc#L1)
|
||||
* [Java license example](https://github.com/google/mediapipe/blob/master/mediapipe/java/com/google/mediapipe/components/CameraHelper.java)
|
||||
|
||||
Bazel BUILD files also need to include a license section, e.g.,
|
||||
[BUILD example](https://github.com/google/mediapipe/blob/master/mediapipe/framework/BUILD#L61).
|
||||
|
||||
#### C++ coding style
|
||||
|
||||
Changes to MediaPipe C++ code should conform to
|
||||
[Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html).
|
||||
|
||||
Use `clang-tidy` to check your C/C++ changes. To install `clang-tidy` on ubuntu:16.04, do:
|
||||
|
||||
```bash
|
||||
apt-get install -y clang-tidy
|
||||
```
|
||||
|
||||
You can check a C/C++ file by doing:
|
||||
|
||||
|
||||
```bash
|
||||
clang-format <my_cc_file> --style=google > /tmp/my_cc_file.cc
|
||||
diff <my_cc_file> /tmp/my_cc_file.cc
|
||||
```
|
||||
|
||||
#### Coding style for other languages
|
||||
|
||||
* [Google Java Style Guide](https://google.github.io/styleguide/javaguide.html)
|
||||
* [Google JavaScript Style Guide](https://google.github.io/styleguide/jsguide.html)
|
||||
* [Google Shell Style Guide](https://google.github.io/styleguide/shell.xml)
|
||||
* [Google Objective-C Style Guide](https://google.github.io/styleguide/objcguide.html)
|
||||
|
||||
#### Running sanity check
|
||||
|
||||
If you have Docker installed on your system, you can perform a sanity check on
|
||||
your changes by running the command:
|
||||
|
||||
```bash
|
||||
mediapipe/tools/ci_build/ci_build.sh CPU mediapipe/tools/ci_build/ci_sanity.sh
|
||||
```
|
||||
|
||||
This will catch most license, Python coding style and BUILD file issues that
|
||||
may exist in your changes.
|
||||
MediaPipe team members will be assigned to review your pull requests. Once the bug/documentation fixes are verified, a MediaPipe team member will acknowledge your contribution in the pull request comments, manually merge the fixes into our internal codebase upstream, and apply the `to be closed` label to the pull request. These fixes will later be pushed to GitHub in the next release, and a MediaPipe team member will then close the pull request.
|
||||
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
global-exclude .git*
|
||||
global-exclude *_test.py
|
||||
|
||||
include CONTRIBUTING.md
|
||||
include LICENSE
|
||||
include MANIFEST.in
|
||||
include README.md
|
||||
include requirements.txt
|
||||
|
||||
recursive-include mediapipe/modules *.tflite *.txt
|
||||
recursive-include mediapipe/graphs *.binarypb
|
||||
@@ -22,30 +22,33 @@ desktop/cloud, web and IoT devices.
|
||||
|
||||
## ML solutions in MediaPipe
|
||||
|
||||
Face Detection | Face Mesh | Hands | Hair Segmentation
|
||||
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :---------------:
|
||||
[](https://google.github.io/mediapipe/solutions/face_detection) | [](https://google.github.io/mediapipe/solutions/face_mesh) | [](https://google.github.io/mediapipe/solutions/hands) | [](https://google.github.io/mediapipe/solutions/hair_segmentation)
|
||||
Face Detection | Face Mesh | Iris | Hands | Pose | Hair Segmentation
|
||||
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------:
|
||||
[](https://google.github.io/mediapipe/solutions/face_detection) | [](https://google.github.io/mediapipe/solutions/face_mesh) | [](https://google.github.io/mediapipe/solutions/iris) | [](https://google.github.io/mediapipe/solutions/hands) | [](https://google.github.io/mediapipe/solutions/pose) | [](https://google.github.io/mediapipe/solutions/hair_segmentation)
|
||||
|
||||
Object Detection | Box Tracking | Objectron | KNIFT
|
||||
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
|
||||
[](https://google.github.io/mediapipe/solutions/object_detection) | [](https://google.github.io/mediapipe/solutions/box_tracking) | [](https://google.github.io/mediapipe/solutions/objectron) | [](https://google.github.io/mediapipe/solutions/knift)
|
||||
Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT
|
||||
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
|
||||
[](https://google.github.io/mediapipe/solutions/object_detection) | [](https://google.github.io/mediapipe/solutions/box_tracking) | [](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [](https://google.github.io/mediapipe/solutions/objectron) | [](https://google.github.io/mediapipe/solutions/knift)
|
||||
|
||||
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
|
||||
<!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
|
||||
|
||||
[]() | Android | iOS | Desktop | Web | Coral
|
||||
:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
|
||||
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅
|
||||
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | |
|
||||
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ |
|
||||
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ |
|
||||
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅
|
||||
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | |
|
||||
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | |
|
||||
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | |
|
||||
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | |
|
||||
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | |
|
||||
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | |
|
||||
[]() | Android | iOS | Desktop | Python | Web | Coral
|
||||
:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
|
||||
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | | ✅ | ✅
|
||||
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | | |
|
||||
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | | ✅ |
|
||||
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | | ✅ |
|
||||
[Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | ✅ |
|
||||
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅
|
||||
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | |
|
||||
[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | |
|
||||
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | |
|
||||
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | |
|
||||
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | |
|
||||
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | |
|
||||
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | |
|
||||
|
||||
## MediaPipe on the Web
|
||||
|
||||
@@ -63,8 +66,11 @@ never leaves your device.
|
||||
|
||||
|
||||
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
|
||||
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
|
||||
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
|
||||
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
|
||||
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
|
||||
* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
|
||||
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
|
||||
|
||||
## Getting started
|
||||
@@ -83,6 +89,12 @@ run code search using
|
||||
|
||||
## Publications
|
||||
|
||||
* [Instant Motion Tracking With MediaPipe](https://mediapipe.page.link/instant-motion-tracking-blog)
|
||||
in Google Developers Blog
|
||||
* [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
|
||||
in Google AI Blog
|
||||
* [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
|
||||
in Google AI Blog
|
||||
* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
|
||||
in Google Developers Blog
|
||||
* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html)
|
||||
@@ -123,7 +135,7 @@ run code search using
|
||||
|
||||
* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome
|
||||
MediaPipe related frameworks, libraries and software
|
||||
* [Slack community](https://https://mediapipe.page.link/joinslack) for MediaPipe users
|
||||
* [Slack community](https://mediapipe.page.link/joinslack) for MediaPipe users
|
||||
* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General
|
||||
community discussion around MediaPipe
|
||||
|
||||
|
||||
@@ -36,6 +36,19 @@ http_archive(
|
||||
urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "rules_foreign_cc",
|
||||
strip_prefix = "rules_foreign_cc-master",
|
||||
url = "https://github.com/bazelbuild/rules_foreign_cc/archive/master.zip",
|
||||
)
|
||||
|
||||
load("@rules_foreign_cc//:workspace_definitions.bzl", "rules_foreign_cc_dependencies")
|
||||
|
||||
rules_foreign_cc_dependencies()
|
||||
|
||||
# This is used to select all contents of the archives for CMake-based packages to give CMake access to them.
|
||||
all_content = """filegroup(name = "all", srcs = glob(["**"]), visibility = ["//visibility:public"])"""
|
||||
|
||||
# GoogleTest/GoogleMock framework. Used by most unit-tests.
|
||||
# Last updated 2020-06-30.
|
||||
http_archive(
|
||||
@@ -68,14 +81,23 @@ http_archive(
|
||||
url = "https://github.com/gflags/gflags/archive/v2.2.2.zip",
|
||||
)
|
||||
|
||||
# glog v0.3.5
|
||||
# TODO: Migrate MediaPipe to use com_github_glog_glog on all platforms.
|
||||
# 2020-08-21
|
||||
http_archive(
|
||||
name = "com_github_glog_glog_v_0_3_5",
|
||||
url = "https://github.com/google/glog/archive/v0.3.5.zip",
|
||||
sha256 = "267103f8a1e9578978aa1dc256001e6529ef593e5aea38193d31c2872ee025e8",
|
||||
strip_prefix = "glog-0.3.5",
|
||||
build_file = "@//third_party:glog.BUILD",
|
||||
name = "com_github_glog_glog",
|
||||
strip_prefix = "glog-0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6",
|
||||
sha256 = "58c9b3b6aaa4dd8b836c0fd8f65d0f941441fb95e27212c5eeb9979cfd3592ab",
|
||||
urls = [
|
||||
"https://github.com/google/glog/archive/0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6.zip",
|
||||
],
|
||||
)
|
||||
http_archive(
|
||||
name = "com_github_glog_glog_no_gflags",
|
||||
strip_prefix = "glog-0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6",
|
||||
sha256 = "58c9b3b6aaa4dd8b836c0fd8f65d0f941441fb95e27212c5eeb9979cfd3592ab",
|
||||
build_file = "@//third_party:glog_no_gflags.BUILD",
|
||||
urls = [
|
||||
"https://github.com/google/glog/archive/0a2e5931bd5ff22fd3bf8999eb8ce776f159cda6.zip",
|
||||
],
|
||||
patches = [
|
||||
"@//third_party:com_github_glog_glog_9779e5ea6ef59562b030248947f787d1256132ae.diff"
|
||||
],
|
||||
@@ -84,16 +106,6 @@ http_archive(
|
||||
],
|
||||
)
|
||||
|
||||
# 2020-02-16
|
||||
http_archive(
|
||||
name = "com_github_glog_glog",
|
||||
strip_prefix = "glog-3ba8976592274bc1f907c402ce22558011d6fc5e",
|
||||
sha256 = "feca3c7e29a693cab7887409756d89d342d4a992d54d7c5599bebeae8f7b50be",
|
||||
urls = [
|
||||
"https://github.com/google/glog/archive/3ba8976592274bc1f907c402ce22558011d6fc5e.zip",
|
||||
],
|
||||
)
|
||||
|
||||
# easyexif
|
||||
http_archive(
|
||||
name = "easyexif",
|
||||
@@ -137,6 +149,25 @@ http_archive(
|
||||
urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
|
||||
)
|
||||
|
||||
# 2020-07-09
|
||||
http_archive(
|
||||
name = "pybind11_bazel",
|
||||
strip_prefix = "pybind11_bazel-203508e14aab7309892a1c5f7dd05debda22d9a5",
|
||||
urls = ["https://github.com/pybind/pybind11_bazel/archive/203508e14aab7309892a1c5f7dd05debda22d9a5.zip"],
|
||||
sha256 = "75922da3a1bdb417d820398eb03d4e9bd067c4905a4246d35a44c01d62154d91",
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "pybind11",
|
||||
urls = [
|
||||
"https://storage.googleapis.com/mirror.tensorflow.org/github.com/pybind/pybind11/archive/v2.4.3.tar.gz",
|
||||
"https://github.com/pybind/pybind11/archive/v2.4.3.tar.gz",
|
||||
],
|
||||
sha256 = "1eed57bc6863190e35637290f97a20c81cfe4d9090ac0a24f3bbf08f265eb71d",
|
||||
strip_prefix = "pybind11-2.4.3",
|
||||
build_file = "@pybind11_bazel//:pybind11.BUILD",
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "ceres_solver",
|
||||
url = "https://github.com/ceres-solver/ceres-solver/archive/1.14.0.zip",
|
||||
@@ -150,6 +181,13 @@ http_archive(
|
||||
sha256 = "5ba6d0db4e784621fda44a50c58bb23b0892684692f0c623e2063f9c19f192f1"
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "opencv",
|
||||
build_file_content = all_content,
|
||||
strip_prefix = "opencv-3.4.10",
|
||||
urls = ["https://github.com/opencv/opencv/archive/3.4.10.tar.gz"],
|
||||
)
|
||||
|
||||
new_local_repository(
|
||||
name = "linux_opencv",
|
||||
build_file = "@//third_party:opencv_linux.BUILD",
|
||||
@@ -165,13 +203,13 @@ new_local_repository(
|
||||
new_local_repository(
|
||||
name = "macos_opencv",
|
||||
build_file = "@//third_party:opencv_macos.BUILD",
|
||||
path = "/usr",
|
||||
path = "/usr/local/opt/opencv@3",
|
||||
)
|
||||
|
||||
new_local_repository(
|
||||
name = "macos_ffmpeg",
|
||||
build_file = "@//third_party:ffmpeg_macos.BUILD",
|
||||
path = "/usr",
|
||||
path = "/usr/local/opt/ffmpeg",
|
||||
)
|
||||
|
||||
new_local_repository(
|
||||
@@ -282,9 +320,6 @@ load("@rules_jvm_external//:defs.bzl", "maven_install")
|
||||
maven_install(
|
||||
name = "maven",
|
||||
artifacts = [
|
||||
"junit:junit:4.12",
|
||||
"androidx.test.espresso:espresso-core:3.1.1",
|
||||
"org.hamcrest:hamcrest-library:1.3",
|
||||
"androidx.concurrent:concurrent-futures:1.0.0-alpha03",
|
||||
"androidx.lifecycle:lifecycle-common:2.2.0",
|
||||
"androidx.annotation:annotation:aar:1.1.0",
|
||||
@@ -295,11 +330,15 @@ maven_install(
|
||||
"androidx.core:core:aar:1.1.0-rc03",
|
||||
"androidx.legacy:legacy-support-v4:aar:1.0.0",
|
||||
"androidx.recyclerview:recyclerview:aar:1.1.0-beta02",
|
||||
"androidx.test.espresso:espresso-core:3.1.1",
|
||||
"com.github.bumptech.glide:glide:4.11.0",
|
||||
"com.google.android.material:material:aar:1.0.0-rc01",
|
||||
"com.google.code.findbugs:jsr305:3.0.2",
|
||||
"com.google.flogger:flogger-system-backend:0.3.1",
|
||||
"com.google.flogger:flogger:0.3.1",
|
||||
"com.google.guava:guava:27.0.1-android",
|
||||
"junit:junit:4.12",
|
||||
"org.hamcrest:hamcrest-library:1.3",
|
||||
],
|
||||
repositories = [
|
||||
"https://jcenter.bintray.com",
|
||||
|
||||
@@ -58,6 +58,9 @@ apps="${app_dir}/*"
|
||||
for app in ${apps}; do
|
||||
if [[ -d "${app}" ]]; then
|
||||
target_name=${app##*/}
|
||||
if [[ "${target_name}" == "common" ]]; then
|
||||
continue
|
||||
fi
|
||||
target="${app}:${target_name}"
|
||||
|
||||
echo "=== Target: ${target}"
|
||||
|
||||
+2
-2
@@ -22,8 +22,8 @@ aux_links:
|
||||
# Footer content appears at the bottom of every page's main content
|
||||
footer_content: "© 2020 GOOGLE LLC | <a href=\"https://policies.google.com/privacy\">PRIVACY POLICY</a> | <a href=\"https://policies.google.com/terms\">TERMS OF SERVICE</a>"
|
||||
|
||||
# Color scheme currently only supports "dark" or nil (default)
|
||||
color_scheme: nil
|
||||
# Color scheme currently only supports "dark", "light"/nil (default), or a custom scheme that you define
|
||||
color_scheme: mediapipe
|
||||
|
||||
# Google Analytics Tracking (optional)
|
||||
ga_tracking: UA-140696581-2
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
$link-color: #0097A7;
|
||||
Binary file not shown.
@@ -207,8 +207,8 @@ class SomeAudioVideoCalculator : public CalculatorBase {
|
||||
// particular type. SetAny() has the same effect as explicitly
|
||||
// setting the type to be the stream's type.
|
||||
cc->Outputs().Tag("VIDEO").Set<ImageFrame>();
|
||||
cc->Outputs().Get("AUDIO", 0).Set<Matrix>;
|
||||
cc->Outputs().Get("AUDIO", 1).Set<Matrix>;
|
||||
cc->Outputs().Get("AUDIO", 0).Set<Matrix>();
|
||||
cc->Outputs().Get("AUDIO", 1).Set<Matrix>();
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
```
|
||||
@@ -400,13 +400,8 @@ node {
|
||||
```
|
||||
|
||||
The diagram below shows how the `PacketClonerCalculator` defines its output
|
||||
packets based on its series of input packets.
|
||||
packets (bottom) based on its series of input packets (top).
|
||||
|
||||
|  :
|
||||
| :--------------------------------------------------------------------------: |
|
||||
| *Each time it receives a packet on its TICK input stream, the |
|
||||
: PacketClonerCalculator outputs the most recent packet from each of its input :
|
||||
: streams. The sequence of output packets is determined by the sequene of :
|
||||
: input packets and their timestamps. The timestamps are shows along the right :
|
||||
: side of the diagram.* :
|
||||
|  |
|
||||
| :---------------------------------------------------------------------------: |
|
||||
| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |
|
||||
|
||||
@@ -20,7 +20,7 @@ Packets are generally created with `MediaPipe::Adopt()` (from packet.h).
|
||||
|
||||
```c++
|
||||
// Create some data.
|
||||
auto data = gtl::MakeUnique<MyDataClass>("constructor_argument");
|
||||
auto data = absl::make_unique<MyDataClass>("constructor_argument");
|
||||
// Create a packet to own the data.
|
||||
Packet p = Adopt(data.release());
|
||||
// Make a new packet with the same data and a different timestamp.
|
||||
|
||||
@@ -184,8 +184,8 @@ app:
|
||||
|
||||
### Prerequisite
|
||||
|
||||
1. Install [Xcode](https://developer.apple.com/xcode/), and additionally
|
||||
install the Command Line Tools by:
|
||||
1. Install [Xcode](https://developer.apple.com/xcode/), then install the
|
||||
Command Line Tools using:
|
||||
|
||||
```bash
|
||||
xcode-select --install
|
||||
@@ -196,74 +196,38 @@ app:
|
||||
We recommend using [Homebrew](https://brew.sh/) to get the latest version.
|
||||
|
||||
3. Set Python 3.7 as the default Python version and install the Python "six"
|
||||
library.
|
||||
|
||||
To make Mediapipe work with TensorFlow, please set Python 3.7 as the default
|
||||
Python version and install the Python "six" library.
|
||||
library. This is needed for TensorFlow.
|
||||
|
||||
```bash
|
||||
pip3 install --user six
|
||||
```
|
||||
|
||||
4. Follow
|
||||
[Apple's instructions](https://developer.apple.com/support/certificates/) to
|
||||
obtain the required development certificates and provisioning profiles for
|
||||
your iOS device.
|
||||
|
||||
Tip: You can the following command to see the provisioning profiles you have
|
||||
previously downloaded using Xcode: `open
|
||||
~/Library/MobileDevice/"Provisioning Profiles"`. If there are none, generate
|
||||
and download a profile on
|
||||
[Apple's developer site](https://developer.apple.com/account/resources/).
|
||||
|
||||
5. Clone the MediaPipe repository.
|
||||
4. Clone the MediaPipe repository.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/google/mediapipe.git
|
||||
```
|
||||
|
||||
6. In the cloned MediaPipe repository, symlink or copy your provisioning profile
|
||||
to `mediapipe/provisioning_profile.mobileprovision`, e.g.,
|
||||
### Set up a bundle ID prefix
|
||||
|
||||
```bash
|
||||
cd mediapipe
|
||||
ln -s ~/Downloads/MyProvisioningProfile.mobileprovision mediapipe/provisioning_profile.mobileprovision
|
||||
```
|
||||
All iOS apps must have a bundle ID, and you must have a provisioning profile
|
||||
that lets you install an app with that ID onto your phone. To avoid clashes
|
||||
between different MediaPipe users, you need to configure a unique prefix for the
|
||||
bundle IDs of our iOS demo apps.
|
||||
|
||||
### Option 1: Build with Bazel in Command Line
|
||||
If you have a custom provisioning profile, see
|
||||
[Custom provisioning](#custom-provisioning) below.
|
||||
|
||||
1. Modify the `bundle_id` field of the app's `ios_application` build target to
|
||||
use your own identifier. For instance, for
|
||||
[MediaPipe Hands](../solutions/hands.md), the `bundle_id` is in the
|
||||
`HandTrackingGpuApp` target in the
|
||||
[BUILD](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu/BUILD)
|
||||
file.
|
||||
Otherwise, run this command to generate a unique prefix:
|
||||
|
||||
2. Again using [MediaPipe Hands](../solutions/hands.md) for example, run:
|
||||
```bash
|
||||
python3 mediapipe/examples/ios/link_local_profiles.py
|
||||
```
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
|
||||
```
|
||||
### Create an Xcode project
|
||||
|
||||
You may see a permission request from `codesign` in order to sign the app.
|
||||
|
||||
Tip: You can run this
|
||||
[script](https://github.com/google/mediapipe/blob/master/build_ios_examples.sh)
|
||||
to build all MediaPipe iOS example apps.
|
||||
|
||||
3. In Xcode, open the `Devices and Simulators` window (command-shift-2).
|
||||
|
||||
4. Make sure your device is connected. You will see a list of installed apps.
|
||||
Press the "+" button under the list, and select the `.ipa` file built by
|
||||
Bazel.
|
||||
|
||||
5. You can now run the app on your device.
|
||||
|
||||
### Option 2: Build in Xcode
|
||||
|
||||
Note: This workflow requires a separate tool in addition to Bazel. If it fails
|
||||
to work for some reason, please resort to the command-line build instructions in
|
||||
the previous section.
|
||||
This allows you to edit and debug one of the example apps in Xcode. It also
|
||||
allows you to make use of automatic provisioning (see later section).
|
||||
|
||||
1. We will use a tool called [Tulsi](https://tulsi.bazel.build/) for generating
|
||||
Xcode projects from Bazel build configurations.
|
||||
@@ -283,25 +247,138 @@ the previous section.
|
||||
|
||||
2. Open `mediapipe/Mediapipe.tulsiproj` using the Tulsi app.
|
||||
|
||||
Important: If Tulsi displays an error saying "Bazel could not be found",
|
||||
press the "Bazel..." button in the Packages tab and select the `bazel`
|
||||
executable in your homebrew `/bin/` directory.
|
||||
Tip: If Tulsi displays an error saying "Bazel could not be found", press the
|
||||
"Bazel..." button in the Packages tab and select the `bazel` executable in
|
||||
your homebrew `/bin/` directory.
|
||||
|
||||
3. Select the MediaPipe config in the Configs tab, then press the Generate
|
||||
button below. You will be asked for a location to save the Xcode project.
|
||||
Once the project is generated, it will be opened in Xcode.
|
||||
|
||||
4. You can now select any of the MediaPipe demos in the target menu, and build
|
||||
If you get an error about bundle IDs, see the
|
||||
[previous section](#set-up-a-bundle-id-prefix).
|
||||
|
||||
### Set up provisioning
|
||||
|
||||
To install applications on an iOS device, you need a provisioning profile. There
|
||||
are two options:
|
||||
|
||||
1. Automatic provisioning. This allows you to build and install an app to your
|
||||
personal device. The provisining profile is managed by Xcode, and has to be
|
||||
updated often (it is valid for about a week).
|
||||
|
||||
2. Custom provisioning. This uses a provisioning profile associated with an
|
||||
Apple developer account. These profiles have a longer validity period and
|
||||
can target multiple devices, but you need a paid developer account with
|
||||
Apple to obtain one.
|
||||
|
||||
#### Automatic provisioning
|
||||
|
||||
1. Create an Xcode project for MediaPipe, as discussed
|
||||
[earlier](#create-an-xcode-project).
|
||||
|
||||
2. In the project navigator in the left sidebar, select the "Mediapipe"
|
||||
project.
|
||||
|
||||
3. Select one of the application targets, e.g. HandTrackingGpuApp.
|
||||
|
||||
4. Select the "Signing & Capabilities" tab.
|
||||
|
||||
5. Check "Automatically manage signing", and confirm the dialog box.
|
||||
|
||||
6. Select "_Your Name_ (Personal Team)" in the Team pop-up menu.
|
||||
|
||||
7. This set-up needs to be done once for each application you want to install.
|
||||
Repeat steps 3-6 as needed.
|
||||
|
||||
This generates provisioning profiles for each app you have selected. Now we need
|
||||
to tell Bazel to use them. We have provided a script to make this easier.
|
||||
|
||||
1. In the terminal, to the `mediapipe` directory where you cloned the
|
||||
repository.
|
||||
|
||||
2. Run this command:
|
||||
|
||||
```bash
|
||||
python3 mediapipe/examples/ios/link_local_profiles.py
|
||||
```
|
||||
|
||||
This will find and link the provisioning profile for all applications for which
|
||||
you have enabled automatic provisioning in Xcode.
|
||||
|
||||
Note: once a profile expires, Xcode will generate a new one; you must then run
|
||||
this script again to link the updated profiles.
|
||||
|
||||
#### Custom provisioning
|
||||
|
||||
1. Obtain a provisioning profile from Apple.
|
||||
|
||||
Tip: You can use this command to see the provisioning profiles you have
|
||||
previously downloaded using Xcode: `open ~/Library/MobileDevice/"Provisioning
|
||||
Profiles"`. If there are none, generate and download a profile on
|
||||
[Apple's developer site](https://developer.apple.com/account/resources/).
|
||||
|
||||
1. Symlink or copy your provisioning profile to
|
||||
`mediapipe/mediapipe/provisioning_profile.mobileprovision`.
|
||||
|
||||
```bash
|
||||
cd mediapipe
|
||||
ln -s ~/Downloads/MyProvisioningProfile.mobileprovision mediapipe/provisioning_profile.mobileprovision
|
||||
```
|
||||
|
||||
Note: if you had previously set up automatic provisioning, you should remove the
|
||||
`provisioning_profile.mobileprovision` symlink in each example's directory,
|
||||
since it will take precedence over the common one. You can also overwrite it
|
||||
with you own profile if you need a different profile for different apps.
|
||||
|
||||
1. Open `mediapipe/examples/ios/bundle_id.bzl`, and change the
|
||||
`BUNDLE_ID_PREFIX` to a prefix associated with your provisioning profile.
|
||||
|
||||
### Build and run an app using Xcode
|
||||
|
||||
1. Create the Xcode project, and make sure you have set up either automatic or
|
||||
custom provisioning.
|
||||
|
||||
2. You can now select any of the MediaPipe demos in the target menu, and build
|
||||
and run them as normal.
|
||||
|
||||
Note: When you ask Xcode to run an app, by default it will use the Debug
|
||||
configuration. Some of our demos are computationally heavy; you may want to
|
||||
use the Release configuration for better performance.
|
||||
Note: When you ask Xcode to run an app, by default it will use the Debug
|
||||
configuration. Some of our demos are computationally heavy; you may want to use
|
||||
the Release configuration for better performance.
|
||||
|
||||
Tip: To switch build configuration in Xcode, click on the target menu,
|
||||
choose "Edit Scheme...", select the Run action, and switch the Build
|
||||
Configuration from Debug to Release. Note that this is set independently for
|
||||
each target.
|
||||
Tip: To switch build configuration in Xcode, click on the target menu, choose
|
||||
"Edit Scheme...", select the Run action, and switch the Build Configuration from
|
||||
Debug to Release. Note that this is set independently for each target.
|
||||
|
||||
Tip: On the device, in Settings > General > Device Management, make sure the
|
||||
developer (yourself) is trusted.
|
||||
|
||||
### Build an app using the command line
|
||||
|
||||
1. Make sure you have set up either automatic or custom provisioning.
|
||||
|
||||
2. Using [MediaPipe Hands](../solutions/hands.md) for example, run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
|
||||
```
|
||||
|
||||
You may see a permission request from `codesign` in order to sign the app.
|
||||
|
||||
Tip: If you are using custom provisioning, you can run this
|
||||
[script](https://github.com/google/mediapipe/blob/master/build_ios_examples.sh)
|
||||
to build all MediaPipe iOS example apps.
|
||||
|
||||
3. In Xcode, open the `Devices and Simulators` window (command-shift-2).
|
||||
|
||||
4. Make sure your device is connected. You will see a list of installed apps.
|
||||
Press the "+" button under the list, and select the `.ipa` file built by
|
||||
Bazel.
|
||||
|
||||
5. You can now run the app on your device.
|
||||
|
||||
Tip: On the device, in Settings > General > Device Management, make sure the
|
||||
developer (yourself) is trusted.
|
||||
|
||||
## Desktop
|
||||
|
||||
@@ -313,9 +390,6 @@ the previous section.
|
||||
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
|
||||
```
|
||||
|
||||
This will open up your webcam as long as it is connected and on. Any errors
|
||||
is likely due to your webcam being not accessible.
|
||||
|
||||
2. To run the application:
|
||||
|
||||
```bash
|
||||
@@ -323,6 +397,9 @@ the previous section.
|
||||
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
|
||||
```
|
||||
|
||||
This will open up your webcam as long as it is connected and on. Any errors
|
||||
is likely due to your webcam being not accessible.
|
||||
|
||||
### Option 2: Running on GPU
|
||||
|
||||
Note: This currently works only on Linux, and please first follow
|
||||
@@ -335,13 +412,96 @@ Note: This currently works only on Linux, and please first follow
|
||||
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
|
||||
```
|
||||
|
||||
This will open up your webcam as long as it is connected and on. Any errors
|
||||
is likely due to your webcam being not accessible, or GPU drivers not setup
|
||||
properly.
|
||||
|
||||
2. To run the application:
|
||||
|
||||
```bash
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
|
||||
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
|
||||
```
|
||||
|
||||
This will open up your webcam as long as it is connected and on. Any errors
|
||||
is likely due to your webcam being not accessible, or GPU drivers not setup
|
||||
properly.
|
||||
|
||||
## Python
|
||||
|
||||
MediaPipe Python package is available on
|
||||
[PyPI](https://pypi.org/project/mediapipe/), and can be installed simply by `pip
|
||||
install mediapipe` on Linux and macOS, as described below in
|
||||
[Run in python interpreter](#run-in-python-interpreter) and in this
|
||||
[colab](https://mediapipe.page.link/mp-py-colab).
|
||||
|
||||
### Run in Python interpreter
|
||||
|
||||
Using [MediaPipe Pose](../solutions/pose.md) as an example:
|
||||
|
||||
```bash
|
||||
# Activate a Python virtual environment.
|
||||
$ python3 -m venv mp_env && source mp_env/bin/activate
|
||||
|
||||
# Install MediaPipe Python package
|
||||
(mp_env)$ pip install mediapipe
|
||||
|
||||
# Run in Python interpreter
|
||||
(mp_env)$ python3
|
||||
>>> import mediapipe as mp
|
||||
>>> pose_tracker = mp.examples.UpperBodyPoseTracker()
|
||||
|
||||
# For image input
|
||||
>>> pose_landmarks, _ = pose_tracker.run(input_file='/path/to/input/file', output_file='/path/to/output/file')
|
||||
>>> pose_landmarks, annotated_image = pose_tracker.run(input_file='/path/to/file')
|
||||
|
||||
# For live camera input
|
||||
# (Press Esc within the output image window to stop the run or let it self terminate after 30 seconds.)
|
||||
>>> pose_tracker.run_live()
|
||||
|
||||
# Close the tracker.
|
||||
>>> pose_tracker.close()
|
||||
```
|
||||
|
||||
Tip: Use command `deactivate` to exit the Python virtual environment.
|
||||
|
||||
### Building Python package from source
|
||||
|
||||
Follow these steps only if you have local changes and need to build the Python
|
||||
package from source. Otherwise, we strongly encourage our users to simply run
|
||||
`pip install mediapipe`, more convenient and much faster.
|
||||
|
||||
1. Make sure that Bazel and OpenCV are correctly installed and configured for
|
||||
MediaPipe. Please see [Installation](./install.md) for how to setup Bazel
|
||||
and OpenCV for MediaPipe on Linux and macOS.
|
||||
|
||||
2. Install the following dependencies.
|
||||
|
||||
```bash
|
||||
# Debian or Ubuntu
|
||||
$ sudo apt install python3-dev
|
||||
$ sudo apt install python3-venv
|
||||
$ sudo apt install -y protobuf-compiler
|
||||
```
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
$ brew install protobuf
|
||||
```
|
||||
|
||||
3. Activate a Python virtual environment.
|
||||
|
||||
```bash
|
||||
$ python3 -m venv mp_env && source mp_env/bin/activate
|
||||
```
|
||||
|
||||
4. In the virtual environment, go to the MediaPipe repo directory.
|
||||
|
||||
5. Install the required Python packages.
|
||||
|
||||
```bash
|
||||
(mp_env)mediapipe$ pip3 install -r requirements.txt
|
||||
```
|
||||
|
||||
6. Generate and install MediaPipe package.
|
||||
|
||||
```bash
|
||||
(mp_env)mediapipe$ python3 setup.py gen_protos
|
||||
(mp_env)mediapipe$ python3 setup.py install --link-opencv
|
||||
```
|
||||
|
||||
@@ -43,8 +43,8 @@ We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
|
||||
|
||||
```
|
||||
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
|
||||
# Used in the examples
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/basic.
|
||||
# Used in the examples in
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/basic and
|
||||
# mediapipe/examples/ios/edgedetectiongpu.
|
||||
|
||||
# Images coming into and out of the graph.
|
||||
@@ -764,7 +764,7 @@ If you ran into any issues, please see the full code of the tutorial
|
||||
[CameraX]:https://developer.android.com/training/camerax
|
||||
[`CameraXPreviewHelper`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/CameraXPreviewHelper.java
|
||||
[developer options]:https://developer.android.com/studio/debug/dev-options
|
||||
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
|
||||
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/edge_detection/edge_detection_mobile_gpu.pbtxt
|
||||
[`EglManager`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/glutil/EglManager.java
|
||||
[`ExternalTextureConverter`]:https://github.com/google/mediapipe/tree/master/mediapipe/java/com/google/mediapipe/components/ExternalTextureConverter.java
|
||||
[`FrameLayout`]:https://developer.android.com/reference/android/widget/FrameLayout
|
||||
|
||||
@@ -18,7 +18,7 @@ nav_order: 5
|
||||
2. To run the [`hello world`] example:
|
||||
|
||||
```bash
|
||||
$ git clone https://github.com/google/mediapipe/mediapipe.git
|
||||
$ git clone https://github.com/google/mediapipe.git
|
||||
$ cd mediapipe
|
||||
|
||||
$ export GLOG_logtostderr=1
|
||||
@@ -92,10 +92,10 @@ nav_order: 5
|
||||
|
||||
```c++
|
||||
CalculatorGraph graph;
|
||||
RETURN_IF_ERROR(graph.Initialize(config));
|
||||
ASSIGN_OR_RETURN(OutputStreamPoller poller,
|
||||
graph.AddOutputStreamPoller("out"));
|
||||
RETURN_IF_ERROR(graph.StartRun({}));
|
||||
MP_RETURN_IF_ERROR(graph.Initialize(config));
|
||||
MP_ASSIGN_OR_RETURN(OutputStreamPoller poller,
|
||||
graph.AddOutputStreamPoller("out"));
|
||||
MP_RETURN_IF_ERROR(graph.StartRun({}));
|
||||
```
|
||||
|
||||
5. The example then creates 10 packets (each packet contains a string "Hello
|
||||
@@ -105,9 +105,10 @@ nav_order: 5
|
||||
|
||||
```c++
|
||||
for (int i = 0; i < 10; ++i) {
|
||||
RETURN_IF_ERROR(graph.AddPacketToInputStream("in", MakePacket<std::string>("Hello World!").At(Timestamp(i))));
|
||||
MP_RETURN_IF_ERROR(graph.AddPacketToInputStream("in",
|
||||
MakePacket<std::string>("Hello World!").At(Timestamp(i))));
|
||||
}
|
||||
RETURN_IF_ERROR(graph.CloseInputStream("in"));
|
||||
MP_RETURN_IF_ERROR(graph.CloseInputStream("in"));
|
||||
```
|
||||
|
||||
6. Through the `OutputStreamPoller` object the example then retrieves all 10
|
||||
|
||||
@@ -18,8 +18,8 @@ This codelab uses MediaPipe on an iOS device.
|
||||
|
||||
### What you will learn
|
||||
|
||||
How to develop an iOS application that uses MediaPipe and run a MediaPipe
|
||||
graph on iOS.
|
||||
How to develop an iOS application that uses MediaPipe and run a MediaPipe graph
|
||||
on iOS.
|
||||
|
||||
### What you will build
|
||||
|
||||
@@ -42,8 +42,8 @@ We will be using the following graph, [`edge_detection_mobile_gpu.pbtxt`]:
|
||||
```
|
||||
# MediaPipe graph that performs GPU Sobel edge detection on a live video stream.
|
||||
# Used in the examples
|
||||
# mediapipe/examples/android/src/java/com/mediapipe/apps/edgedetectiongpu.
|
||||
# mediapipe/examples/ios/edgedetectiongpu.
|
||||
# mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:helloworld
|
||||
# and mediapipe/examples/ios/helloworld.
|
||||
|
||||
# Images coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
@@ -56,7 +56,7 @@ node: {
|
||||
output_stream: "luma_video"
|
||||
}
|
||||
|
||||
# Applies the Sobel filter to luminance images sotred in RGB format.
|
||||
# Applies the Sobel filter to luminance images stored in RGB format.
|
||||
node: {
|
||||
calculator: "SobelEdgesCalculator"
|
||||
input_stream: "luma_video"
|
||||
@@ -89,21 +89,21 @@ to build it.
|
||||
|
||||
First, create an XCode project via File > New > Single View App.
|
||||
|
||||
Set the product name to "EdgeDetectionGpu", and use an appropriate organization
|
||||
Set the product name to "HelloWorld", and use an appropriate organization
|
||||
identifier, such as `com.google.mediapipe`. The organization identifier
|
||||
alongwith the product name will be the `bundle_id` for the application, such as
|
||||
`com.google.mediapipe.EdgeDetectionGpu`.
|
||||
`com.google.mediapipe.HelloWorld`.
|
||||
|
||||
Set the language to Objective-C.
|
||||
|
||||
Save the project to an appropriate location. Let's call this
|
||||
`$PROJECT_TEMPLATE_LOC`. So your project will be in the
|
||||
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu` directory. This directory will contain
|
||||
another directory named `EdgeDetectionGpu` and an `EdgeDetectionGpu.xcodeproj` file.
|
||||
`$PROJECT_TEMPLATE_LOC/HelloWorld` directory. This directory will contain
|
||||
another directory named `HelloWorld` and an `HelloWorld.xcodeproj` file.
|
||||
|
||||
The `EdgeDetectionGpu.xcodeproj` will not be useful for this tutorial, as we will
|
||||
use bazel to build the iOS application. The content of the
|
||||
`$PROJECT_TEMPLATE_LOC/EdgeDetectionGpu/EdgeDetectionGpu` directory is listed below:
|
||||
The `HelloWorld.xcodeproj` will not be useful for this tutorial, as we will use
|
||||
bazel to build the iOS application. The content of the
|
||||
`$PROJECT_TEMPLATE_LOC/HelloWorld/HelloWorld` directory is listed below:
|
||||
|
||||
1. `AppDelegate.h` and `AppDelegate.m`
|
||||
2. `ViewController.h` and `ViewController.m`
|
||||
@@ -112,10 +112,10 @@ use bazel to build the iOS application. The content of the
|
||||
5. `Main.storyboard` and `Launch.storyboard`
|
||||
6. `Assets.xcassets` directory.
|
||||
|
||||
Copy these files to a directory named `EdgeDetectionGpu` to a location that can
|
||||
access the MediaPipe source code. For example, the source code of the
|
||||
application that we will build in this tutorial is located in
|
||||
`mediapipe/examples/ios/EdgeDetectionGpu`. We will refer to this path as the
|
||||
Copy these files to a directory named `HelloWorld` to a location that can access
|
||||
the MediaPipe source code. For example, the source code of the application that
|
||||
we will build in this tutorial is located in
|
||||
`mediapipe/examples/ios/HelloWorld`. We will refer to this path as the
|
||||
`$APPLICATION_PATH` throughout the codelab.
|
||||
|
||||
Note: MediaPipe provides Objective-C bindings for iOS. The edge detection
|
||||
@@ -134,8 +134,8 @@ load(
|
||||
)
|
||||
|
||||
ios_application(
|
||||
name = "EdgeDetectionGpuApp",
|
||||
bundle_id = "com.google.mediapipe.EdgeDetectionGpu",
|
||||
name = "HelloWorldApp",
|
||||
bundle_id = "com.google.mediapipe.HelloWorld",
|
||||
families = [
|
||||
"iphone",
|
||||
"ipad",
|
||||
@@ -143,11 +143,11 @@ ios_application(
|
||||
infoplists = ["Info.plist"],
|
||||
minimum_os_version = MIN_IOS_VERSION,
|
||||
provisioning_profile = "//mediapipe/examples/ios:developer_provisioning_profile",
|
||||
deps = [":EdgeDetectionGpuAppLibrary"],
|
||||
deps = [":HelloWorldAppLibrary"],
|
||||
)
|
||||
|
||||
objc_library(
|
||||
name = "EdgeDetectionGpuAppLibrary",
|
||||
name = "HelloWorldAppLibrary",
|
||||
srcs = [
|
||||
"AppDelegate.m",
|
||||
"ViewController.m",
|
||||
@@ -172,9 +172,8 @@ The `objc_library` rule adds dependencies for the `AppDelegate` and
|
||||
`ViewController` classes, `main.m` and the application storyboards. The
|
||||
templated app depends only on the `UIKit` SDK.
|
||||
|
||||
The `ios_application` rule uses the `EdgeDetectionGpuAppLibrary` Objective-C
|
||||
library generated to build an iOS application for installation on your iOS
|
||||
device.
|
||||
The `ios_application` rule uses the `HelloWorldAppLibrary` Objective-C library
|
||||
generated to build an iOS application for installation on your iOS device.
|
||||
|
||||
Note: You need to point to your own iOS developer provisioning profile to be
|
||||
able to run the application on your iOS device.
|
||||
@@ -182,21 +181,20 @@ able to run the application on your iOS device.
|
||||
To build the app, use the following command in a terminal:
|
||||
|
||||
```
|
||||
bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
|
||||
bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:HelloWorldApp'
|
||||
```
|
||||
|
||||
For example, to build the `EdgeDetectionGpuApp` application in
|
||||
`mediapipe/examples/ios/edgedetectiongpu`, use the following
|
||||
command:
|
||||
For example, to build the `HelloWorldApp` application in
|
||||
`mediapipe/examples/ios/helloworld`, use the following command:
|
||||
|
||||
```
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp
|
||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/helloworld:HelloWorldApp
|
||||
```
|
||||
|
||||
Then, go back to XCode, open Window > Devices and Simulators, select your
|
||||
device, and add the `.ipa` file generated by the command above to your device.
|
||||
Here is the document on [setting up and compiling](./building_examples.md#ios) iOS
|
||||
MediaPipe apps.
|
||||
Here is the document on [setting up and compiling](./building_examples.md#ios)
|
||||
iOS MediaPipe apps.
|
||||
|
||||
Open the application on your device. Since it is empty, it should display a
|
||||
blank white screen.
|
||||
@@ -502,8 +500,8 @@ in our app:
|
||||
}];
|
||||
```
|
||||
|
||||
Note: It is important to start the graph before starting the camera, so that
|
||||
the graph is ready to process frames as soon as the camera starts sending them.
|
||||
Note: It is important to start the graph before starting the camera, so that the
|
||||
graph is ready to process frames as soon as the camera starts sending them.
|
||||
|
||||
Earlier, when we received frames from the camera in the `processVideoFrame`
|
||||
function, we displayed them in the `_liveView` using the `_renderer`. Now, we
|
||||
@@ -552,9 +550,12 @@ results of running the edge detection graph on a live video feed. Congrats!
|
||||
|
||||
|
||||
|
||||
If you ran into any issues, please see the full code of the tutorial
|
||||
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
|
||||
Please note that the iOS examples now use a [common] template app. The code in
|
||||
this tutorial is used in the [common] template app. The [helloworld] app has the
|
||||
appropriate `BUILD` file dependencies for the edge detection graph.
|
||||
|
||||
[Bazel]:https://bazel.build/
|
||||
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt
|
||||
[`edge_detection_mobile_gpu.pbtxt`]:https://github.com/google/mediapipe/tree/master/mediapipe/graphs/edge_detection/edge_detection_mobile_gpu.pbtxt
|
||||
[MediaPipe installation guide]:./install.md
|
||||
[common]:(https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/common)
|
||||
[helloworld]:(https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/helloworld)
|
||||
|
||||
+122
-19
@@ -70,9 +70,15 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
libopencv-imgproc-dev libopencv-video-dev
|
||||
```
|
||||
|
||||
[`opencv_linux.BUILD`] is configured for x86_64 by default. For Nvidia
|
||||
Jetson and Raspberry Pi devices with ARM Ubuntu, the lib paths need to be
|
||||
modified.
|
||||
Debian 9 and Ubuntu 18.04 install the packages in
|
||||
`/usr/lib/x86_64-linux-gnu`. MediaPipe's [`opencv_linux.BUILD`] and
|
||||
[`ffmpeg_linux.BUILD`] are configured for this library path. Ubuntu 20.04
|
||||
may install the OpenCV and FFmpeg packages in `/usr/local`, Please follow
|
||||
the option 3 below to modify the [`WORKSPACE`], [`opencv_linux.BUILD`] and
|
||||
[`ffmpeg_linux.BUILD`] files accordingly.
|
||||
|
||||
Moreover, for Nvidia Jetson and Raspberry Pi devices with ARM Ubuntu, the
|
||||
library path needs to be modified like the following:
|
||||
|
||||
```bash
|
||||
sed -i "s/x86_64-linux-gnu/aarch64-linux-gnu/g" third_party/opencv_linux.BUILD
|
||||
@@ -85,11 +91,13 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
[documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
|
||||
to manually build OpenCV from source code.
|
||||
|
||||
Note: You may need to modify [`WORKSPACE`] and [`opencv_linux.BUILD`] to
|
||||
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
|
||||
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
|
||||
rule in [`WORKSPACE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
|
||||
like the following:
|
||||
Note: You may need to modify [`WORKSPACE`], [`opencv_linux.BUILD`] and
|
||||
[`ffmpeg_linux.BUILD`] to point MediaPipe to your own OpenCV and FFmpeg
|
||||
libraries. For example if OpenCV and FFmpeg are both manually installed in
|
||||
"/usr/local/", you will need to update: (1) the "linux_opencv" and
|
||||
"linux_ffmpeg" new_local_repository rules in [`WORKSPACE`], (2) the "opencv"
|
||||
cc_library rule in [`opencv_linux.BUILD`], and (3) the "libffmpeg"
|
||||
cc_library rule in [`ffmpeg_linux.BUILD`]. These 3 changes are shown below:
|
||||
|
||||
```bash
|
||||
new_local_repository(
|
||||
@@ -98,6 +106,12 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
path = "/usr/local",
|
||||
)
|
||||
|
||||
new_local_repository(
|
||||
name = "linux_ffmpeg",
|
||||
build_file = "@//third_party:ffmpeg_linux.BUILD",
|
||||
path = "/usr/local",
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "opencv",
|
||||
srcs = glob(
|
||||
@@ -110,8 +124,36 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
"lib/libopencv_videoio.so",
|
||||
],
|
||||
),
|
||||
hdrs = glob(["include/opencv4/**/*.h*"]),
|
||||
includes = ["include/opencv4/"],
|
||||
hdrs = glob([
|
||||
# For OpenCV 3.x
|
||||
"include/opencv2/**/*.h*",
|
||||
# For OpenCV 4.x
|
||||
# "include/opencv4/opencv2/**/*.h*",
|
||||
]),
|
||||
includes = [
|
||||
# For OpenCV 3.x
|
||||
"include/",
|
||||
# For OpenCV 4.x
|
||||
# "include/opencv4/",
|
||||
],
|
||||
linkstatic = 1,
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "libffmpeg",
|
||||
srcs = glob(
|
||||
[
|
||||
"lib/libav*.so",
|
||||
],
|
||||
),
|
||||
hdrs = glob(["include/libav*/*.h"]),
|
||||
includes = ["include"],
|
||||
linkopts = [
|
||||
"-lavcodec",
|
||||
"-lavformat",
|
||||
"-lavutil",
|
||||
],
|
||||
linkstatic = 1,
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
@@ -158,6 +200,10 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
# Hello World!
|
||||
```
|
||||
|
||||
If you run into a build error, please read
|
||||
[Troubleshooting](./troubleshooting.md) to find the solutions of several common
|
||||
build issues.
|
||||
|
||||
## Installing on CentOS
|
||||
|
||||
**Disclaimer**: Running MediaPipe on CentOS is experimental.
|
||||
@@ -190,11 +236,13 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
|
||||
Option 2. Build OpenCV from source code.
|
||||
|
||||
Note: You may need to modify [`WORKSPACE`] and [`opencv_linux.BUILD`] to
|
||||
point MediaPipe to your own OpenCV libraries, e.g., if OpenCV 4 is installed
|
||||
in "/usr/local/", you need to update the "linux_opencv" new_local_repository
|
||||
rule in [`WORKSPACE`] and "opencv" cc_library rule in [`opencv_linux.BUILD`]
|
||||
like the following:
|
||||
Note: You may need to modify [`WORKSPACE`], [`opencv_linux.BUILD`] and
|
||||
[`ffmpeg_linux.BUILD`] to point MediaPipe to your own OpenCV and FFmpeg
|
||||
libraries. For example if OpenCV and FFmpeg are both manually installed in
|
||||
"/usr/local/", you will need to update: (1) the "linux_opencv" and
|
||||
"linux_ffmpeg" new_local_repository rules in [`WORKSPACE`], (2) the "opencv"
|
||||
cc_library rule in [`opencv_linux.BUILD`], and (3) the "libffmpeg"
|
||||
cc_library rule in [`ffmpeg_linux.BUILD`]. These 3 changes are shown below:
|
||||
|
||||
```bash
|
||||
new_local_repository(
|
||||
@@ -203,6 +251,12 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
path = "/usr/local",
|
||||
)
|
||||
|
||||
new_local_repository(
|
||||
name = "linux_ffmpeg",
|
||||
build_file = "@//third_party:ffmpeg_linux.BUILD",
|
||||
path = "/usr/local",
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "opencv",
|
||||
srcs = glob(
|
||||
@@ -215,8 +269,36 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
"lib/libopencv_videoio.so",
|
||||
],
|
||||
),
|
||||
hdrs = glob(["include/opencv4/**/*.h*"]),
|
||||
includes = ["include/opencv4/"],
|
||||
hdrs = glob([
|
||||
# For OpenCV 3.x
|
||||
"include/opencv2/**/*.h*",
|
||||
# For OpenCV 4.x
|
||||
# "include/opencv4/opencv2/**/*.h*",
|
||||
]),
|
||||
includes = [
|
||||
# For OpenCV 3.x
|
||||
"include/",
|
||||
# For OpenCV 4.x
|
||||
# "include/opencv4/",
|
||||
],
|
||||
linkstatic = 1,
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "libffmpeg",
|
||||
srcs = glob(
|
||||
[
|
||||
"lib/libav*.so",
|
||||
],
|
||||
),
|
||||
hdrs = glob(["include/libav*/*.h"]),
|
||||
includes = ["include"],
|
||||
linkopts = [
|
||||
"-lavcodec",
|
||||
"-lavformat",
|
||||
"-lavutil",
|
||||
],
|
||||
linkstatic = 1,
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
@@ -243,6 +325,10 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
# Hello World!
|
||||
```
|
||||
|
||||
If you run into a build error, please read
|
||||
[Troubleshooting](./troubleshooting.md) to find the solutions of several common
|
||||
build issues.
|
||||
|
||||
## Installing on macOS
|
||||
|
||||
1. Prework:
|
||||
@@ -375,6 +461,10 @@ apps, see these [instructions](./building_examples.md#ios).
|
||||
# Hello World!
|
||||
```
|
||||
|
||||
If you run into a build error, please read
|
||||
[Troubleshooting](./troubleshooting.md) to find the solutions of several common
|
||||
build issues.
|
||||
|
||||
## Installing on Windows
|
||||
|
||||
**Disclaimer**: Running MediaPipe on Windows is experimental.
|
||||
@@ -454,13 +544,13 @@ next section.
|
||||
9. Run the [Hello World desktop example](./hello_world_desktop.md).
|
||||
|
||||
Note: For building MediaPipe on Windows, please add `--action_env
|
||||
PYTHON_BIN_PATH="C:/path/to/python.exe"` to the build command.
|
||||
PYTHON_BIN_PATH="C://path//to//python.exe"` to the build command.
|
||||
Alternatively, you can follow
|
||||
[issue 724](https://github.com/google/mediapipe/issues/724) to fix the
|
||||
python configuration manually.
|
||||
|
||||
```
|
||||
C:\Users\Username\mediapipe_repo>bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH="C:/python_36/python.exe" mediapipe/examples/desktop/hello_world
|
||||
C:\Users\Username\mediapipe_repo>bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 --action_env PYTHON_BIN_PATH="C://python_36//python.exe" mediapipe/examples/desktop/hello_world
|
||||
|
||||
C:\Users\Username\mediapipe_repo>set GLOG_logtostderr=1
|
||||
|
||||
@@ -480,6 +570,10 @@ next section.
|
||||
|
||||
```
|
||||
|
||||
If you run into a build error, please read
|
||||
[Troubleshooting](./troubleshooting.md) to find the solutions of several common
|
||||
build issues.
|
||||
|
||||
## Installing on Windows Subsystem for Linux (WSL)
|
||||
|
||||
Note: The pre-built OpenCV packages don't support cameras in WSL. Unless you
|
||||
@@ -603,6 +697,10 @@ cameras. Alternatively, you use a video file as input.
|
||||
# Hello World!
|
||||
```
|
||||
|
||||
If you run into a build error, please
|
||||
read [Troubleshooting](./troubleshooting.md) to find the solutions of several
|
||||
common build issues.
|
||||
|
||||
## Installing using Docker
|
||||
|
||||
This will use a Docker image that will isolate mediapipe's installation from the rest of the system.
|
||||
@@ -653,6 +751,10 @@ This will use a Docker image that will isolate mediapipe's installation from the
|
||||
# Hello World!
|
||||
```
|
||||
|
||||
If you run into a build error, please
|
||||
read [Troubleshooting](./troubleshooting.md) to find the solutions of several
|
||||
common build issues.
|
||||
|
||||
4. Build a MediaPipe Android example.
|
||||
|
||||
```bash
|
||||
@@ -692,6 +794,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
|
||||
|
||||
[`WORKSPACE`]: https://github.com/google/mediapipe/blob/master/WORKSPACE
|
||||
[`opencv_linux.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_linux.BUILD
|
||||
[`ffmpeg_linux.BUILD`]:https://github.com/google/mediapipe/tree/master/third_party/ffmpeg_linux.BUILD
|
||||
[`opencv_macos.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_macos.BUILD
|
||||
[`ffmpeg_macos.BUILD`]:https://github.com/google/mediapipe/tree/master/third_party/ffmpeg_macos.BUILD
|
||||
[`setup_opencv.sh`]: https://github.com/google/mediapipe/blob/master/setup_opencv.sh
|
||||
|
||||
@@ -12,6 +12,91 @@ nav_order: 10
|
||||
{:toc}
|
||||
---
|
||||
|
||||
## Missing Python binary path
|
||||
|
||||
The error message:
|
||||
|
||||
```
|
||||
ERROR: An error occurred during the fetch of repository 'local_execution_config_python':
|
||||
Traceback (most recent call last):
|
||||
File "/sandbox_path/external/org_tensorflow/third_party/py/python_configure.bzl", line 208
|
||||
get_python_bin(repository_ctx)
|
||||
...
|
||||
Repository command failed
|
||||
```
|
||||
|
||||
usually indicates that Bazel fails to find the local Python binary. To solve
|
||||
this issue, please first find where the python binary is and then add
|
||||
`--action_env PYTHON_BIN_PATH=<path to python binary>` to the Bazel command. For
|
||||
example, you can switch to use the system default python3 binary by the
|
||||
following command:
|
||||
|
||||
```
|
||||
bazel build -c opt \
|
||||
--define MEDIAPIPE_DISABLE_GPU=1 \
|
||||
--action_env PYTHON_BIN_PATH=$(which python3) \
|
||||
mediapipe/examples/desktop/hello_world
|
||||
```
|
||||
|
||||
## Missing necessary Python packages
|
||||
|
||||
The error message:
|
||||
|
||||
```
|
||||
ImportError: No module named numpy
|
||||
Is numpy installed?
|
||||
```
|
||||
|
||||
usually indicates that certain Python packages are not installed. Please run
|
||||
`pip install` or `pip3 install` depending on your Python binary version to
|
||||
install those packages.
|
||||
|
||||
## Fail to fetch remote dependency repositories
|
||||
|
||||
The error message:
|
||||
|
||||
```
|
||||
ERROR: An error occurred during the fetch of repository 'org_tensorflow':
|
||||
java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/77e9ffb9b2bfb1a4f7056e62d84039626923e328.tar.gz, https://github.com/tensorflow/tensorflow/archive/77e9ffb9b2bfb1a4f7056e62d84039626923e328.tar.gz] to /sandbox_path/external/org_tensorflow/77e9ffb9b2bfb1a4f7056e62d84039626923e328.tar.gz: Tried to reconnect at offset 9,944,151 but server didn't support it
|
||||
|
||||
or
|
||||
|
||||
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/rules_swift/releases/download/0.12.1/rules_swift.0.12.1.tar.gz failed: class java.net.ConnectException Connection timed out (Connection timed out)
|
||||
```
|
||||
|
||||
usually indicates that Bazel fails to download necessary dependency repositories
|
||||
that MediaPipe needs. MedaiPipe has several dependency repositories that are
|
||||
hosted by Google sites. In some regions, you may need to set up a network proxy
|
||||
or use a VPN to access those resources. You may also need to append
|
||||
`--host_jvm_args "-DsocksProxyHost=<ip address> -DsocksProxyPort=<port number>"`
|
||||
to the Bazel command. See
|
||||
[this GitHub issue](https://github.com/google/mediapipe/issues/581#issuecomment-610356857)
|
||||
for more details.
|
||||
|
||||
If you believe that it's not a network issue, another possibility is that some
|
||||
resources could be temporarily unavailable, please run `bazel clean --expunge`
|
||||
and retry it later. If it's still not working, please file a GitHub issue with
|
||||
the detailed error message.
|
||||
|
||||
## Incorrect MediaPipe OpenCV config
|
||||
|
||||
The error message:
|
||||
|
||||
```
|
||||
error: undefined reference to 'cv::String::deallocate()'
|
||||
error: undefined reference to 'cv::String::allocate(unsigned long)'
|
||||
error: undefined reference to 'cv::VideoCapture::VideoCapture(cv::String const&)'
|
||||
...
|
||||
error: undefined reference to 'cv::putText(cv::InputOutputArray const&, cv::String const&, cv::Point, int, double, cv::Scalar, int, int, bool)'
|
||||
```
|
||||
|
||||
usually indicates that OpenCV is not properly configured for MediaPipe. Please
|
||||
take a look at the "Install OpenCV and FFmpeg" sections in
|
||||
[Installation](./install.md) to see how to modify MediaPipe's WORKSPACE and
|
||||
linux_opencv/macos_opencv/windows_opencv.BUILD files for your local opencv
|
||||
libraries. [This GitHub issue](https://github.com/google/mediapipe/issues/666)
|
||||
may also help.
|
||||
|
||||
## Native method not found
|
||||
|
||||
The error message:
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 925 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.4 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1019 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.8 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 182 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 996 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 313 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 53 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.9 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 224 KiB |
+32
-20
@@ -22,30 +22,33 @@ desktop/cloud, web and IoT devices.
|
||||
|
||||
## ML solutions in MediaPipe
|
||||
|
||||
Face Detection | Face Mesh | Hands | Hair Segmentation
|
||||
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :---------------:
|
||||
[](https://google.github.io/mediapipe/solutions/face_detection) | [](https://google.github.io/mediapipe/solutions/face_mesh) | [](https://google.github.io/mediapipe/solutions/hands) | [](https://google.github.io/mediapipe/solutions/hair_segmentation)
|
||||
Face Detection | Face Mesh | Iris | Hands | Pose | Hair Segmentation
|
||||
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------:
|
||||
[](https://google.github.io/mediapipe/solutions/face_detection) | [](https://google.github.io/mediapipe/solutions/face_mesh) | [](https://google.github.io/mediapipe/solutions/iris) | [](https://google.github.io/mediapipe/solutions/hands) | [](https://google.github.io/mediapipe/solutions/pose) | [](https://google.github.io/mediapipe/solutions/hair_segmentation)
|
||||
|
||||
Object Detection | Box Tracking | Objectron | KNIFT
|
||||
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
|
||||
[](https://google.github.io/mediapipe/solutions/object_detection) | [](https://google.github.io/mediapipe/solutions/box_tracking) | [](https://google.github.io/mediapipe/solutions/objectron) | [](https://google.github.io/mediapipe/solutions/knift)
|
||||
Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT
|
||||
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
|
||||
[](https://google.github.io/mediapipe/solutions/object_detection) | [](https://google.github.io/mediapipe/solutions/box_tracking) | [](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [](https://google.github.io/mediapipe/solutions/objectron) | [](https://google.github.io/mediapipe/solutions/knift)
|
||||
|
||||
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
|
||||
<!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
|
||||
|
||||
[]() | Android | iOS | Desktop | Web | Coral
|
||||
:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
|
||||
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅
|
||||
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | |
|
||||
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ |
|
||||
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ |
|
||||
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅
|
||||
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | |
|
||||
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | |
|
||||
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | |
|
||||
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | |
|
||||
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | |
|
||||
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | |
|
||||
[]() | Android | iOS | Desktop | Python | Web | Coral
|
||||
:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
|
||||
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | | ✅ | ✅
|
||||
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | | |
|
||||
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | | ✅ |
|
||||
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | | ✅ |
|
||||
[Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | ✅ |
|
||||
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅
|
||||
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | |
|
||||
[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | |
|
||||
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | |
|
||||
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | |
|
||||
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | |
|
||||
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | |
|
||||
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | |
|
||||
|
||||
## MediaPipe on the Web
|
||||
|
||||
@@ -63,8 +66,11 @@ never leaves your device.
|
||||
|
||||
|
||||
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
|
||||
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
|
||||
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
|
||||
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
|
||||
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
|
||||
* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
|
||||
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
|
||||
|
||||
## Getting started
|
||||
@@ -83,6 +89,12 @@ run code search using
|
||||
|
||||
## Publications
|
||||
|
||||
* [Instant Motion Tracking With MediaPipe](https://mediapipe.page.link/instant-motion-tracking-blog)
|
||||
in Google Developers Blog
|
||||
* [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
|
||||
in Google AI Blog
|
||||
* [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
|
||||
in Google AI Blog
|
||||
* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
|
||||
in Google Developers Blog
|
||||
* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html)
|
||||
@@ -123,7 +135,7 @@ run code search using
|
||||
|
||||
* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome
|
||||
MediaPipe related frameworks, libraries and software
|
||||
* [Slack community](https://https://mediapipe.page.link/joinslack) for MediaPipe users
|
||||
* [Slack community](https://mediapipe.page.link/joinslack) for MediaPipe users
|
||||
* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General
|
||||
community discussion around MediaPipe
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: AutoFlip (Saliency-aware Video Cropping)
|
||||
parent: Solutions
|
||||
nav_order: 9
|
||||
nav_order: 12
|
||||
---
|
||||
|
||||
# AutoFlip: Saliency-aware Video Cropping
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: Box Tracking
|
||||
parent: Solutions
|
||||
nav_order: 6
|
||||
nav_order: 8
|
||||
---
|
||||
|
||||
# MediaPipe Box Tracking
|
||||
@@ -112,7 +112,7 @@ examples.
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ examples.
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
@@ -107,4 +107,4 @@ to cross-compile and run MediaPipe examples on the
|
||||
[TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
|
||||
* For back-facing camera:
|
||||
[TFLite model ](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_back.tflite)
|
||||
* [Model card](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view)
|
||||
* [Model card](https://mediapipe.page.link/blazeface-mc)
|
||||
|
||||
@@ -65,7 +65,7 @@ from the
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
## Models
|
||||
|
||||
@@ -109,7 +109,7 @@ Please first see general instructions for
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
@@ -125,7 +125,7 @@ Tip: Maximum number of faces to detect/process is set to 1 by default. To change
|
||||
it, for Android modify `NUM_FACES` in
|
||||
[MainActivity.java](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/MainActivity.java),
|
||||
and for iOS modify `kNumFaces` in
|
||||
[ViewController.mm](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facemeshgpu/ViewController.mm).
|
||||
[FaceMeshGpuViewController.mm](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facemeshgpu/FaceMeshGpuViewController.mm).
|
||||
|
||||
### Desktop
|
||||
|
||||
@@ -153,8 +153,8 @@ it, in the graph file modify the option of `ConstantSidePacketCalculator`.
|
||||
[Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://arxiv.org/abs/1907.06724)
|
||||
([poster](https://docs.google.com/presentation/d/1-LWwOMO9TzEVdrZ1CS1ndJzciRHfYDJfbSxH_ke_JRg/present?slide=id.g5986dd4b4c_4_212))
|
||||
* Face detection model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite)
|
||||
* Face landmark mode:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_landmark.tflite),
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite)
|
||||
* Face landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
|
||||
* [Model card](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view)
|
||||
* [Model card](https://mediapipe.page.link/facemesh-mc)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: Hair Segmentation
|
||||
parent: Solutions
|
||||
nav_order: 4
|
||||
nav_order: 6
|
||||
---
|
||||
|
||||
# MediaPipe Hair Segmentation
|
||||
@@ -24,7 +24,7 @@ examples.
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
@@ -55,4 +55,4 @@ Please refer to [these instructions](../index.md#mediapipe-on-the-web).
|
||||
([presentation](https://drive.google.com/file/d/1C8WYlWdDRNtU1_pYBvkkG5Z5wqYqf0yj/view))
|
||||
([supplementary video](https://drive.google.com/file/d/1LPtM99Ch2ogyXYbDNpEqnUfhFq0TfLuf/view))
|
||||
* [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite)
|
||||
* [Model card](https://drive.google.com/file/d/1lPwJ8BD_-3UUor4LayQ0xpa_RIC_hoRh/view)
|
||||
* [Model card](https://mediapipe.page.link/hairsegmentation-mc)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: Hands
|
||||
parent: Solutions
|
||||
nav_order: 3
|
||||
nav_order: 4
|
||||
---
|
||||
|
||||
# MediaPipe Hands
|
||||
@@ -66,7 +66,7 @@ and a
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
## Models
|
||||
|
||||
@@ -132,7 +132,7 @@ examples.
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
layout: default
|
||||
title: Instant Motion Tracking
|
||||
parent: Solutions
|
||||
nav_order: 9
|
||||
---
|
||||
|
||||
# MediaPipe Instant Motion Tracking
|
||||
{: .no_toc }
|
||||
|
||||
1. TOC
|
||||
{:toc}
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Augmented Reality (AR) technology creates fun, engaging, and immersive user
|
||||
experiences. The ability to perform AR tracking across devices and platforms,
|
||||
without initialization, remains important to power AR applications at scale.
|
||||
|
||||
MediaPipe Instant Motion Tracking provides AR tracking across devices and
|
||||
platforms without initialization or calibration. It is built upon the
|
||||
[MediaPipe Box Tracking](./box_tracking.md) solution. With Instant Motion
|
||||
Tracking, you can easily place virtual 2D and 3D content on static or moving
|
||||
surfaces, allowing them to seamlessly interact with the real-world environment.
|
||||
|
||||
 |
|
||||
:-----------------------------------------------------------------------: |
|
||||
*Fig 1. Instant Motion Tracking is used to augment the world with a 3D sticker.* |
|
||||
|
||||
## Pipeline
|
||||
|
||||
The Instant Motion Tracking pipeline is implemented as a MediaPipe
|
||||
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt),
|
||||
which internally utilizes a
|
||||
[RegionTrackingSubgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/subgraphs/region_tracking.pbtxt)
|
||||
in order to perform anchor tracking for each individual 3D sticker.
|
||||
|
||||
We first use a
|
||||
[StickerManagerCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/calculators/sticker_manager_calculator.cc)
|
||||
to prepare the individual sticker data for the rest of the application. This
|
||||
information is then sent to the
|
||||
[RegionTrackingSubgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/subgraphs/region_tracking.pbtxt)
|
||||
that performs 3D region tracking for sticker placement and rendering. Once
|
||||
acquired, our tracked sticker regions are sent with user transformations (i.e.
|
||||
gestures from the user to rotate and zoom the sticker) and IMU data to the
|
||||
[MatricesManagerCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/calculators/matrices_manager_calculator.cc),
|
||||
which turns all our sticker transformation data into a set of model matrices.
|
||||
This data is handled directly by our
|
||||
[GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
|
||||
as an input stream, which will render the provided texture and object file using
|
||||
our matrix specifications. The output of
|
||||
[GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
|
||||
is a video stream depicting the virtual 3D content rendered on top of the real
|
||||
world, creating immersive AR experiences for users.
|
||||
|
||||
## Using Instant Motion Tracking
|
||||
|
||||
With the Instant Motion Tracking MediaPipe [graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt),
|
||||
an application can create an interactive and realistic AR experience by
|
||||
specifying the required input streams, side packets, and output streams.
|
||||
The input streams are the following:
|
||||
|
||||
* Input Video (GpuBuffer): Video frames to render augmented stickers onto.
|
||||
* Rotation Matrix (9-element Float Array): The 3x3 row-major rotation
|
||||
matrix from the device IMU to determine proper orientation of the device.
|
||||
* Sticker Proto String (String): A string representing the
|
||||
serialized [sticker buffer protobuf message](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/calculators/sticker_buffer.proto),
|
||||
containing a list of all stickers and their attributes.
|
||||
* Each sticker in the Protobuffer has a unique ID to find associated
|
||||
anchors and transforms, an initial anchor placement in a normalized [0.0, 1.0]
|
||||
3D space, a user rotation and user scaling transform on the sticker,
|
||||
and an integer indicating which type of objects to render for the
|
||||
sticker (e.g. 3D asset or GIF).
|
||||
* Sticker Sentinel (Integer): When an anchor must be initially placed or
|
||||
repositioned, this value must be changed to the ID of the anchor to reset from
|
||||
the sticker buffer protobuf message. If no valid ID is provided, the system
|
||||
will simply maintain tracking.
|
||||
|
||||
Side packets are also an integral part of the Instant Motion Tracking solution
|
||||
to provide device-specific information for the rendering system:
|
||||
|
||||
* Field of View (Float): The field of view of the camera in radians.
|
||||
* Aspect Ratio (Float): The aspect ratio (width / height) of the camera frames
|
||||
(this ratio corresponds to the image frames themselves, not necessarily the
|
||||
screen bounds).
|
||||
* Object Asset (String): The
|
||||
[GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
|
||||
must be provided with an associated asset file name pointing to the 3D model
|
||||
to render in the viewfinder.
|
||||
* (Optional) Texture (ImageFrame on Android, GpuBuffer on iOS): Textures for
|
||||
the
|
||||
[GlAnimationOverlayCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/calculators/gl_animation_overlay_calculator.cc)
|
||||
can be provided either via an input stream (dynamic texturing) or as a side
|
||||
packet (unchanging texture).
|
||||
|
||||
The rendering system for the Instant Motion Tracking is powered by OpenGL. For
|
||||
more information regarding the structure of model matrices and OpenGL rendering,
|
||||
please visit [OpenGL Wiki](https://www.khronos.org/opengl/wiki/). With the
|
||||
specifications above, the Instant Motion Tracking capabilities can be adapted to
|
||||
any device that is able to run the MediaPipe framework with a working IMU system
|
||||
and connected camera.
|
||||
|
||||
## Example Apps
|
||||
|
||||
Please first see general instructions for
|
||||
[Android](../getting_started/building_examples.md#android) on how to build
|
||||
MediaPipe examples.
|
||||
|
||||
* Graph: [mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/instant_motion_tracking/instant_motion_tracking.pbtxt)
|
||||
|
||||
* Android target (or download prebuilt [ARM64 APK](https://drive.google.com/file/d/1KnaBBoKpCHR73nOBJ4fL_YdWVTAcwe6L/view?usp=sharing)):
|
||||
[`mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking:instantmotiontracking`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/instantmotiontracking/BUILD)
|
||||
|
||||
## Resources
|
||||
|
||||
* Google Developers Blog:
|
||||
[Instant Motion Tracking With MediaPipe](https://mediapipe.page.link/instant-motion-tracking-blog)
|
||||
* Google AI Blog:
|
||||
[The Instant Motion Tracking Behind Motion Stills AR](https://ai.googleblog.com/2018/02/the-instant-motion-tracking-behind.html)
|
||||
* Paper:
|
||||
[Instant Motion Tracking and Its Applications to Augmented Reality](https://arxiv.org/abs/1907.06796)
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
layout: default
|
||||
title: Iris
|
||||
parent: Solutions
|
||||
nav_order: 3
|
||||
---
|
||||
|
||||
# MediaPipe Iris
|
||||
{: .no_toc }
|
||||
|
||||
1. TOC
|
||||
{:toc}
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
A wide range of real-world applications, including computational photography
|
||||
(glint reflection) and augmented reality effects (virtual avatars) rely on
|
||||
accurately tracking the iris within an eye. This is a challenging task to solve
|
||||
on mobile devices, due to the limited computing resources, variable light
|
||||
conditions and the presence of occlusions, such as hair or people squinting.
|
||||
Iris tracking can also be utilized to determine the metric distance of the
|
||||
camera to the user. This can improve a variety of use cases, ranging from
|
||||
virtual try-on of properly sized glasses and hats to accessibility features that
|
||||
adopt the font size depending on the viewer’s distance. Often, sophisticated
|
||||
specialized hardware is employed to compute the metric distance, limiting the
|
||||
range of devices on which the solution could be applied.
|
||||
|
||||
MediaPipe Iris is a ML solution for accurate iris estimation, able to track
|
||||
landmarks involving the iris, pupil and the eye contours using a single RGB
|
||||
camera, in real-time, without the need for specialized hardware. Through use of
|
||||
iris landmarks, the solution is also able to determine the metric distance
|
||||
between the subject and the camera with relative error less than 10%. Note that
|
||||
iris tracking does not infer the location at which people are looking, nor does
|
||||
it provide any form of identity recognition. With the cross-platfrom capability
|
||||
of the MediaPipe framework, MediaPipe Iris can run on most modern
|
||||
[mobile phones](#mobile), [desktops/laptops](#desktop) and even on the
|
||||
[web](#web).
|
||||
|
||||
 |
|
||||
:------------------------------------------------------------------------: |
|
||||
*Fig 1. Example of MediaPipe Iris: eyelid (red) and iris (blue) contours.* |
|
||||
|
||||
## ML Pipeline
|
||||
|
||||
The first step in the pipeline leverages [MediaPipe Face Mesh](./face_mesh.md),
|
||||
which generates a mesh of the approximate face geometry. From this mesh, we
|
||||
isolate the eye region in the original image for use in the subsequent iris
|
||||
tracking step.
|
||||
|
||||
The pipeline is implemented as a MediaPipe
|
||||
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt)
|
||||
that uses a
|
||||
[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
|
||||
from the
|
||||
[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark),
|
||||
an
|
||||
[iris landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark_left_and_right_gpu.pbtxt)
|
||||
from the
|
||||
[iris landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark),
|
||||
and renders using a dedicated
|
||||
[iris-and-depth renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/subgraphs/iris_and_depth_renderer_gpu.pbtxt).
|
||||
The
|
||||
[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
|
||||
internally uses a
|
||||
[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
|
||||
from the
|
||||
[face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
|
||||
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
The output of the pipeline is a set of 478 3D landmarks, including 468 face
|
||||
landmarks from [MediaPipe Face Mesh](./face_mesh.md), with those around the eyes
|
||||
further refined (see Fig 2), and 10 additional iris landmarks appended at the
|
||||
end (5 for each eye, and see Fig 2 also).
|
||||
|
||||
## Models
|
||||
|
||||
### Face Detection Model
|
||||
|
||||
The face detector is the same [BlazeFace](https://arxiv.org/abs/1907.05047)
|
||||
model used in [MediaPipe Face Detection](./face_detection.md).
|
||||
|
||||
### Face Landmark Model
|
||||
|
||||
The face landmark model is the same as in [MediaPipe Face Mesh](./face_mesh.md).
|
||||
You can also find more details in this
|
||||
[paper](https://arxiv.org/abs/1907.06724).
|
||||
|
||||
### Iris Landmark Model
|
||||
|
||||
The iris model takes an image patch of the eye region and estimates both the eye
|
||||
landmarks (along the eyelid) and iris landmarks (along ths iris contour). You
|
||||
can find more details in this [paper](https://arxiv.org/abs/2006.11341).
|
||||
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------------------: |
|
||||
*Fig 2. Eye landmarks (red) and iris landmarks (green).* |
|
||||
|
||||
## Depth-from-Iris
|
||||
|
||||
MediaPipe Iris is able to determine the metric distance of a subject to the
|
||||
camera with less than 10% error, without requiring any specialized hardware.
|
||||
This is done by relying on the fact that the horizontal iris diameter of the
|
||||
human eye remains roughly constant at 11.7±0.5 mm across a wide population,
|
||||
along with some simple geometric arguments. For more details please refer to our
|
||||
[Google AI Blog post](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html).
|
||||
|
||||
 |
|
||||
:--------------------------------------------------------------------------------------------: |
|
||||
*Fig 3. (Left) MediaPipe Iris predicting metric distance in cm on a Pixel 2 from iris tracking without use of a depth sensor. (Right) Ground-truth depth.* |
|
||||
|
||||
## Example Apps
|
||||
|
||||
Please first see general instructions for
|
||||
[Android](../getting_started/building_examples.md#android),
|
||||
[iOS](../getting_started/building_examples.md#ios) and
|
||||
[desktop](../getting_started/building_examples.md#desktop) on how to build
|
||||
MediaPipe examples.
|
||||
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
* Graph:
|
||||
[`mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt)
|
||||
* Android target:
|
||||
[(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1cywcNtqk764TlZf1lvSTV4F3NGB2aL1R/view?usp=sharing)
|
||||
[`mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu:iristrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu/BUILD)
|
||||
* iOS target:
|
||||
[`mediapipe/examples/ios/iristrackinggpu:IrisTrackingGpuApp`](http:/mediapipe/examples/ios/iristrackinggpu/BUILD)
|
||||
|
||||
### Desktop
|
||||
|
||||
#### Live Camera Input
|
||||
|
||||
Please first see general instructions for
|
||||
[desktop](../getting_started/building_examples.md#desktop) on how to build
|
||||
MediaPipe examples.
|
||||
|
||||
* Running on CPU
|
||||
* Graph:
|
||||
[`mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt)
|
||||
* Target:
|
||||
[`mediapipe/examples/desktop/iris_tracking:iris_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/iris_tracking/BUILD)
|
||||
* Running on GPU
|
||||
* Graph:
|
||||
[`mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt)
|
||||
* Target:
|
||||
[`mediapipe/examples/desktop/iris_tracking:iris_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/iris_tracking/BUILD)
|
||||
|
||||
#### Video File Input
|
||||
|
||||
1. To build the application, run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/iris_tracking:iris_tracking_cpu_video_input
|
||||
```
|
||||
|
||||
2. To run the application, replace `<input video path>` and `<output video
|
||||
path>` in the command below with your own paths:
|
||||
|
||||
```
|
||||
bazel-bin/mediapipe/examples/desktop/iris_tracking/iris_tracking_cpu_video_input \
|
||||
--calculator_graph_config_file=mediapipe/graphs/iris_tracking/iris_tracking_cpu_video_input.pbtxt \
|
||||
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
|
||||
```
|
||||
|
||||
#### Single-image Depth Estimation
|
||||
|
||||
1. To build the application, run:
|
||||
|
||||
```bash
|
||||
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/iris_tracking:iris_depth_from_image_desktop
|
||||
```
|
||||
|
||||
2. To run the application, replace `<input image path>` and `<output image
|
||||
path>` in the command below with your own paths:
|
||||
|
||||
```bash
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/iris_tracking/iris_depth_from_image_desktop \
|
||||
--input_image_path=<input image path> --output_image_path=<output image path>
|
||||
```
|
||||
|
||||
### Web
|
||||
|
||||
Please refer to [these instructions](../index.md#mediapipe-on-the-web).
|
||||
|
||||
## Resources
|
||||
|
||||
* Google AI Blog:
|
||||
[MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
|
||||
* Paper:
|
||||
[Real-time Pupil Tracking from Monocular Video for Digital Puppetry](https://arxiv.org/abs/2006.11341)
|
||||
([presentation](https://youtu.be/cIhXkiiapQI))
|
||||
* Face detection model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite)
|
||||
* Face landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
|
||||
[TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
|
||||
* Iris landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark.tflite)
|
||||
* [Model card](https://mediapipe.page.link/iris-mc)
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: KNIFT (Template-based Feature Matching)
|
||||
parent: Solutions
|
||||
nav_order: 8
|
||||
nav_order: 11
|
||||
---
|
||||
|
||||
# MediaPipe KNIFT
|
||||
@@ -72,7 +72,7 @@ Please first see general instructions for
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
* Graph:
|
||||
[`mediapipe/graphs/template_matching/template_matching_mobile_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/template_matching/template_matching_mobile_cpu.pbtxt)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: Dataset Preparation with MediaSequence
|
||||
parent: Solutions
|
||||
nav_order: 10
|
||||
nav_order: 13
|
||||
---
|
||||
|
||||
# Dataset Preparation with MediaSequence
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: Object Detection
|
||||
parent: Solutions
|
||||
nav_order: 5
|
||||
nav_order: 7
|
||||
---
|
||||
|
||||
# MediaPipe Object Detection
|
||||
@@ -19,7 +19,7 @@ nav_order: 5
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
@@ -95,8 +95,8 @@ Please first see general instructions for
|
||||
|
||||
```
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite \
|
||||
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tflite_graph.pbtxt \
|
||||
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
|
||||
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tflite_graph.pbtxt \
|
||||
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
|
||||
```
|
||||
|
||||
* With a TensorFlow Model
|
||||
@@ -131,8 +131,8 @@ Please first see general instructions for
|
||||
|
||||
```bash
|
||||
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite \
|
||||
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tensorflow_graph.pbtxt \
|
||||
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
|
||||
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tensorflow_graph.pbtxt \
|
||||
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
|
||||
```
|
||||
|
||||
### Coral
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: Objectron (3D Object Detection)
|
||||
parent: Solutions
|
||||
nav_order: 7
|
||||
nav_order: 10
|
||||
---
|
||||
|
||||
# MediaPipe Objectron
|
||||
@@ -156,12 +156,12 @@ Please first see general instructions for
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../visualizer.md).
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Objectron for Shoes
|
||||
|
||||
* Graph:
|
||||
[`mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/shoe_classic_occlusion_tracking.pbtxt)
|
||||
[`mediapipe/graphs/object_detection_3d/shoe_classic_occlusion_tracking.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection_3d/shoe_classic_occlusion_tracking.pbtxt)
|
||||
* Android target:
|
||||
[(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1S0K4hbWt3o31FfQ4QU3Rz7IHrvOUMx1d)
|
||||
[`mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetection3d:objectdetection3d`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetection3d/BUILD)
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
---
|
||||
layout: default
|
||||
title: Pose
|
||||
parent: Solutions
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
# MediaPipe BlazePose
|
||||
{: .no_toc }
|
||||
|
||||
1. TOC
|
||||
{:toc}
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Human pose estimation from video plays a critical role in various applications
|
||||
such as quantifying physical exercises, sign language recognition, and full-body
|
||||
gesture control. For example, it can form the basis for yoga, dance, and fitness
|
||||
applications. It can also enable the overlay of digital content and information
|
||||
on top of the physical world in augmented reality.
|
||||
|
||||
MediaPipe Pose is a ML solution for high-fidelity upper-body pose tracking,
|
||||
inferring 25 2D upper-body landmarks from RGB video frames utilizing our
|
||||
[BlazePose](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
|
||||
research. Current state-of-the-art approaches rely primarily on powerful desktop
|
||||
environments for inference, whereas our method achieves real-time performance on
|
||||
most modern [mobile phones](#mobile), [desktops/laptops](#desktop), in
|
||||
[python](#python) and even on the [web](#web). A variant of MediaPipe Pose that
|
||||
performs full-body pose tracking on mobile phones will be included in an
|
||||
upcoming release of
|
||||
[ML Kit](https://developers.google.com/ml-kit/early-access/pose-detection).
|
||||
|
||||
 |
|
||||
:--------------------------------------------------------------------------------------------: |
|
||||
*Fig 1. Example of MediaPipe Pose for upper-body pose tracking.* |
|
||||
|
||||
## ML Pipeline
|
||||
|
||||
The solution utilizes a two-step detector-tracker ML pipeline, proven to be
|
||||
effective in our [MediaPipe Hands](./hands.md) and
|
||||
[MediaPipe Face Mesh](./face_mesh.md) solutions. Using a detector, the pipeline
|
||||
first locates the pose region-of-interest (ROI) within the frame. The tracker
|
||||
subsequently predicts the pose landmarks within the ROI using the ROI-cropped
|
||||
frame as input. Note that for video use cases the detector is invoked only as
|
||||
needed, i.e., for the very first frame and when the tracker could no longer
|
||||
identify body pose presence in the previous frame. For other frames the pipeline
|
||||
simply derives the ROI from the previous frame’s pose landmarks.
|
||||
|
||||
The pipeline is implemented as a MediaPipe
|
||||
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt)
|
||||
that uses a
|
||||
[pose landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_upper_body_gpu.pbtxt)
|
||||
from the
|
||||
[pose landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark)
|
||||
and renders using a dedicated
|
||||
[upper-body pose renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/subgraphs/upper_body_pose_renderer_gpu.pbtxt).
|
||||
The
|
||||
[pose landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_upper_body_gpu.pbtxt)
|
||||
internally uses a
|
||||
[pose detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection/pose_detection_gpu.pbtxt)
|
||||
from the
|
||||
[pose detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection).
|
||||
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
## Models
|
||||
|
||||
### Pose Detection Model (BlazePose Detector)
|
||||
|
||||
The detector is inspired by our own lightweight
|
||||
[BlazeFace](https://arxiv.org/abs/1907.05047) model, used in
|
||||
[MediaPipe Face Detection](./face_detection.md), as a proxy for a person
|
||||
detector. It explicitly predicts two additional virtual keypoints that firmly
|
||||
describe the human body center, rotation and scale as a circle. Inspired by
|
||||
[Leonardo’s Vitruvian man](https://en.wikipedia.org/wiki/Vitruvian_Man), we
|
||||
predict the midpoint of a person's hips, the radius of a circle circumscribing
|
||||
the whole person, and the incline angle of the line connecting the shoulder and
|
||||
hip midpoints.
|
||||
|
||||
 |
|
||||
:----------------------------------------------------------------------------------------------------: |
|
||||
*Fig 2. Vitruvian man aligned via two virtual keypoints predicted by BlazePose detector in addition to the face bounding box.* |
|
||||
|
||||
### Pose Landmark Model (BlazePose Tracker)
|
||||
|
||||
The landmark model currently included in MediaPipe Pose predicts the location of
|
||||
25 upper-body landmarks (see figure below), with three degrees of freedom each
|
||||
(x, y location and visibility), plus two virtual alignment keypoints. It shares
|
||||
the same architecture as the full-body version that predicts 33 landmarks,
|
||||
described in more detail in the
|
||||
[BlazePose Google AI Blog](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
|
||||
and in this [paper](https://arxiv.org/abs/2006.10204).
|
||||
|
||||
 |
|
||||
:------------------------------------------------------------------------------------------------: |
|
||||
*Fig 3. 25 upper-body pose landmarks.* |
|
||||
|
||||
## Example Apps
|
||||
|
||||
Please first see general instructions for
|
||||
[Android](../getting_started/building_examples.md#android),
|
||||
[iOS](../getting_started/building_examples.md#ios),
|
||||
[desktop](../getting_started/building_examples.md#desktop) and
|
||||
[Python](../getting_started/building_examples.md#python) on how to build
|
||||
MediaPipe examples.
|
||||
|
||||
Note: To visualize a graph, copy the graph and paste it into
|
||||
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
|
||||
to visualize its associated subgraphs, please see
|
||||
[visualizer documentation](../tools/visualizer.md).
|
||||
|
||||
### Mobile
|
||||
|
||||
* Graph:
|
||||
[`mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt)
|
||||
* Android target:
|
||||
[(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1uKc6T7KSuA0Mlq2URi5YookHu0U3yoh_/view?usp=sharing)
|
||||
[`mediapipe/examples/android/src/java/com/google/mediapipe/apps/upperbodyposetrackinggpu:upperbodyposetrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/upperbodyposetrackinggpu/BUILD)
|
||||
* iOS target:
|
||||
[`mediapipe/examples/ios/upperbodyposetrackinggpu:UpperBodyPoseTrackingGpuApp`](http:/mediapipe/examples/ios/upperbodyposetrackinggpu/BUILD)
|
||||
|
||||
### Desktop
|
||||
|
||||
Please first see general instructions for
|
||||
[desktop](../getting_started/building_examples.md#desktop) on how to build
|
||||
MediaPipe examples.
|
||||
|
||||
* Running on CPU
|
||||
* Graph:
|
||||
[`mediapipe/graphs/pose_tracking/upper_body_pose_tracking_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_cpu.pbtxt)
|
||||
* Target:
|
||||
[`mediapipe/examples/desktop/upper_body_pose_tracking:upper_body_pose_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/upper_body_pose_tracking/BUILD)
|
||||
* Running on GPU
|
||||
* Graph:
|
||||
[`mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/pose_tracking/upper_body_pose_tracking_gpu.pbtxt)
|
||||
* Target:
|
||||
[`mediapipe/examples/desktop/upper_body_pose_tracking:upper_body_pose_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/upper_body_pose_tracking/BUILD)
|
||||
|
||||
### Python
|
||||
|
||||
MediaPipe Python package is available on
|
||||
[PyPI](https://pypi.org/project/mediapipe/), and can be installed simply by `pip
|
||||
install mediapipe` on Linux and macOS, as described below and in this
|
||||
[colab](https://mediapipe.page.link/mp-py-colab). If you do need to build the
|
||||
Python package from source, see
|
||||
[additional instructions](../getting_started/building_examples.md#python).
|
||||
|
||||
```bash
|
||||
# Activate a Python virtual environment.
|
||||
$ python3 -m venv mp_env && source mp_env/bin/activate
|
||||
|
||||
# Install MediaPipe Python package
|
||||
(mp_env)$ pip install mediapipe
|
||||
|
||||
# Run in Python interpreter
|
||||
(mp_env)$ python3
|
||||
>>> import mediapipe as mp
|
||||
>>> pose_tracker = mp.examples.UpperBodyPoseTracker()
|
||||
|
||||
# For image input
|
||||
>>> pose_landmarks, _ = pose_tracker.run(input_file='/path/to/input/file', output_file='/path/to/output/file')
|
||||
>>> pose_landmarks, annotated_image = pose_tracker.run(input_file='/path/to/file')
|
||||
# To print out the pose landmarks, you can simply do "print(pose_landmarks)".
|
||||
# However, the data points can be more accessible with the following approach.
|
||||
>>> [print('x is', data_point.x, 'y is', data_point.y, 'z is', data_point.z, 'visibility is', data_point.visibility) for data_point in pose_landmarks.landmark]
|
||||
|
||||
# For live camera input
|
||||
# (Press Esc within the output image window to stop the run or let it self terminate after 30 seconds.)
|
||||
>>> pose_tracker.run_live()
|
||||
|
||||
# Close the tracker.
|
||||
>>> pose_tracker.close()
|
||||
```
|
||||
|
||||
Tip: Use command `deactivate` to exit the Python virtual environment.
|
||||
|
||||
### Web
|
||||
|
||||
Please refer to [these instructions](../index.md#mediapipe-on-the-web).
|
||||
|
||||
## Resources
|
||||
|
||||
* Google AI Blog:
|
||||
[BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
|
||||
* Paper:
|
||||
[BlazePose: On-device Real-time Body Pose Tracking](https://arxiv.org/abs/2006.10204)
|
||||
([presentation](https://youtu.be/YPpUOTRn5tA))
|
||||
* Pose detection model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_detection/pose_detection.tflite)
|
||||
* Upper-body pose landmark model:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/pose_landmark/pose_landmark_upper_body.tflite)
|
||||
* [Model card](https://mediapipe.page.link/blazepose-mc)
|
||||
+17
-14
@@ -14,18 +14,21 @@ has_toc: false
|
||||
---
|
||||
|
||||
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
|
||||
<!-- Whenever this table is updated, paste a copy to ../index.md. -->
|
||||
<!-- Whenever this table is updated, paste a copy to ../external_index.md. -->
|
||||
|
||||
[]() | Android | iOS | Desktop | Web | Coral
|
||||
:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
|
||||
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅
|
||||
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | |
|
||||
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ |
|
||||
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ |
|
||||
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅
|
||||
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | |
|
||||
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | |
|
||||
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | |
|
||||
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | |
|
||||
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | |
|
||||
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | |
|
||||
[]() | Android | iOS | Desktop | Python | Web | Coral
|
||||
:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---:
|
||||
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | | ✅ | ✅
|
||||
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | | |
|
||||
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | | ✅ |
|
||||
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | | ✅ |
|
||||
[Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | ✅ |
|
||||
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅
|
||||
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | |
|
||||
[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | |
|
||||
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | |
|
||||
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | |
|
||||
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | |
|
||||
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | |
|
||||
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | |
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layout: default
|
||||
title: YouTube-8M Feature Extraction and Model Inference
|
||||
parent: Solutions
|
||||
nav_order: 11
|
||||
nav_order: 14
|
||||
---
|
||||
|
||||
# YouTube-8M Feature Extraction and Model Inference
|
||||
|
||||
@@ -33,13 +33,12 @@ command line option: `--define MEDIAPIPE_PROFILING=0`.
|
||||
To enable tracing and profiling, the `CalculatorGraphConfig` (in
|
||||
[calculator.proto](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto))
|
||||
representing the graph must have a `profiler_config` message at its root. Here
|
||||
is a simple setup that turns on a few extra options:
|
||||
is a simple setup that turns on tracing and keeps 100 seconds of timing events:
|
||||
|
||||
```
|
||||
profiler_config {
|
||||
enable_profiler: true
|
||||
trace_enabled: true
|
||||
trace_log_count: 5
|
||||
trace_log_interval_count: 200
|
||||
}
|
||||
```
|
||||
|
||||
@@ -145,10 +144,13 @@ we record ten intervals of half a second each. This can be overridden by adding
|
||||
```bash
|
||||
profiler_config {
|
||||
trace_enabled: true
|
||||
trace_log_path: "/sdcard/profiles"
|
||||
trace_log_path: "/sdcard/profiles/"
|
||||
}
|
||||
```
|
||||
|
||||
Note: The forward slash at the end of the `trace_log_path` is necessary for
|
||||
indicating that `profiles` is a directory (that *should* exist).
|
||||
|
||||
* Download the trace files from the device.
|
||||
|
||||
```bash
|
||||
@@ -295,7 +297,7 @@ trace_log_margin_usec
|
||||
in trace log output. This margin allows time for events to be appended to
|
||||
the TraceBuffer.
|
||||
|
||||
trace_log_duration_events
|
||||
trace_log_instant_events
|
||||
: False specifies an event for each calculator invocation. True specifies a
|
||||
separate event for each start and finish time.
|
||||
|
||||
|
||||
@@ -3,27 +3,34 @@
|
||||
"/BUILD",
|
||||
"mediapipe/BUILD",
|
||||
"mediapipe/objc/BUILD",
|
||||
"mediapipe/examples/ios/BUILD",
|
||||
"mediapipe/examples/ios/edgedetectiongpu/BUILD",
|
||||
"mediapipe/framework/BUILD",
|
||||
"mediapipe/gpu/BUILD",
|
||||
"mediapipe/objc/testing/app/BUILD",
|
||||
"mediapipe/examples/ios/common/BUILD",
|
||||
"mediapipe/examples/ios/helloworld/BUILD",
|
||||
"mediapipe/examples/ios/facedetectioncpu/BUILD",
|
||||
"mediapipe/examples/ios/facedetectiongpu/BUILD",
|
||||
"mediapipe/examples/ios/facemeshgpu/BUILD",
|
||||
"mediapipe/examples/ios/handdetectiongpu/BUILD",
|
||||
"mediapipe/examples/ios/handtrackinggpu/BUILD",
|
||||
"mediapipe/examples/ios/iristrackinggpu/BUILD",
|
||||
"mediapipe/examples/ios/multihandtrackinggpu/BUILD",
|
||||
"mediapipe/examples/ios/objectdetectioncpu/BUILD",
|
||||
"mediapipe/examples/ios/objectdetectiongpu/BUILD"
|
||||
"mediapipe/examples/ios/objectdetectiongpu/BUILD",
|
||||
"mediapipe/examples/ios/upperbodyposetrackinggpu/BUILD"
|
||||
],
|
||||
"buildTargets" : [
|
||||
"//mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp",
|
||||
"//mediapipe/examples/ios/helloworld:HelloWorldApp",
|
||||
"//mediapipe/examples/ios/facedetectioncpu:FaceDetectionCpuApp",
|
||||
"//mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp",
|
||||
"//mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp",
|
||||
"//mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp",
|
||||
"//mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp",
|
||||
"//mediapipe/examples/ios/iristrackinggpu:IrisTrackingGpuApp",
|
||||
"//mediapipe/examples/ios/multihandtrackinggpu:MultiHandTrackingGpuApp",
|
||||
"//mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp",
|
||||
"//mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp",
|
||||
"//mediapipe/examples/ios/upperbodyposetrackinggpu:UpperBodyPoseTrackingGpuApp",
|
||||
"//mediapipe/objc:mediapipe_framework_ios"
|
||||
],
|
||||
"optionSet" : {
|
||||
@@ -78,22 +85,18 @@
|
||||
"mediapipe/calculators/util",
|
||||
"mediapipe/examples",
|
||||
"mediapipe/examples/ios",
|
||||
"mediapipe/examples/ios/edgedetectiongpu",
|
||||
"mediapipe/examples/ios/edgedetectiongpu/Base.lproj",
|
||||
"mediapipe/examples/ios/common",
|
||||
"mediapipe/examples/ios/common/Base.lproj",
|
||||
"mediapipe/examples/ios/helloworld",
|
||||
"mediapipe/examples/ios/facedetectioncpu",
|
||||
"mediapipe/examples/ios/facedetectioncpu/Base.lproj",
|
||||
"mediapipe/examples/ios/facedetectiongpu",
|
||||
"mediapipe/examples/ios/facedetectiongpu/Base.lproj",
|
||||
"mediapipe/examples/ios/handdetectiongpu",
|
||||
"mediapipe/examples/ios/handdetectiongpu/Base.lproj",
|
||||
"mediapipe/examples/ios/handtrackinggpu",
|
||||
"mediapipe/examples/ios/handtrackinggpu/Base.lproj",
|
||||
"mediapipe/examples/ios/iristrackinggpu",
|
||||
"mediapipe/examples/ios/multihandtrackinggpu",
|
||||
"mediapipe/examples/ios/multihandtrackinggpu/Base.lproj",
|
||||
"mediapipe/examples/ios/objectdetectioncpu",
|
||||
"mediapipe/examples/ios/objectdetectioncpu/Base.lproj",
|
||||
"mediapipe/examples/ios/objectdetectiongpu",
|
||||
"mediapipe/examples/ios/objectdetectiongpu/Base.lproj",
|
||||
"mediapipe/examples/ios/upperbodyposetrackinggpu",
|
||||
"mediapipe/framework",
|
||||
"mediapipe/framework/deps",
|
||||
"mediapipe/framework/formats",
|
||||
@@ -109,7 +112,9 @@
|
||||
"mediapipe/graphs/face_detection",
|
||||
"mediapipe/graphs/hand_tracking",
|
||||
"mediapipe/graphs/object_detection",
|
||||
"mediapipe/graphs/pose_tracking",
|
||||
"mediapipe/models",
|
||||
"mediapipe/modules",
|
||||
"mediapipe/objc",
|
||||
"mediapipe/util",
|
||||
"mediapipe/util/android",
|
||||
|
||||
@@ -11,15 +11,16 @@
|
||||
"mediapipe",
|
||||
"mediapipe/objc",
|
||||
"mediapipe/examples/ios",
|
||||
"mediapipe/examples/ios/edgedetectiongpu",
|
||||
"mediapipe/examples/ios/facedetectioncpu",
|
||||
"mediapipe/examples/ios/facedetectiongpu",
|
||||
"mediapipe/examples/ios/facemeshgpu",
|
||||
"mediapipe/examples/ios/handdetectiongpu",
|
||||
"mediapipe/examples/ios/handtrackinggpu",
|
||||
"mediapipe/examples/ios/iristrackinggpu",
|
||||
"mediapipe/examples/ios/multihandtrackinggpu",
|
||||
"mediapipe/examples/ios/objectdetectioncpu",
|
||||
"mediapipe/examples/ios/objectdetectiongpu"
|
||||
"mediapipe/examples/ios/objectdetectiongpu",
|
||||
"mediapipe/examples/ios/upperbodyposetrackinggpu"
|
||||
],
|
||||
"projectName" : "Mediapipe",
|
||||
"workspaceRoot" : "../.."
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Copyright 2019 The MediaPipe Authors.
|
||||
"""Copyright 2019 - 2020 The MediaPipe Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//visibility:private"])
|
||||
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
filegroup(
|
||||
name = "test_audios",
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
|
||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//visibility:private"])
|
||||
|
||||
@@ -290,7 +290,9 @@ cc_library(
|
||||
deps = [
|
||||
":concatenate_vector_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/formats:classification_cc_proto",
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/port:integral_types",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
"@org_tensorflow//tensorflow/lite:framework",
|
||||
@@ -316,6 +318,37 @@ cc_library(
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "concatenate_normalized_landmark_list_calculator",
|
||||
srcs = ["concatenate_normalized_landmark_list_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":concatenate_vector_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "concatenate_normalized_landmark_list_calculator_test",
|
||||
srcs = ["concatenate_normalized_landmark_list_calculator_test.cc"],
|
||||
deps = [
|
||||
":concatenate_normalized_landmark_list_calculator",
|
||||
":concatenate_vector_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/port:status",
|
||||
"@com_google_absl//absl/strings",
|
||||
],
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "concatenate_vector_calculator_test",
|
||||
srcs = ["concatenate_vector_calculator_test.cc"],
|
||||
@@ -450,6 +483,21 @@ cc_library(
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "mux_calculator_test",
|
||||
srcs = ["mux_calculator_test.cc"],
|
||||
deps = [
|
||||
":mux_calculator",
|
||||
":round_robin_demux_calculator",
|
||||
":split_vector_calculator",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "packet_cloner_calculator",
|
||||
srcs = ["packet_cloner_calculator.cc"],
|
||||
@@ -560,6 +608,35 @@ cc_library(
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "packet_presence_calculator",
|
||||
srcs = ["packet_presence_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:packet",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "packet_presence_calculator_test",
|
||||
srcs = ["packet_presence_calculator_test.cc"],
|
||||
deps = [
|
||||
":gate_calculator",
|
||||
":packet_presence_calculator",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/tool:sink",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "previous_loopback_calculator",
|
||||
srcs = ["previous_loopback_calculator.cc"],
|
||||
@@ -947,7 +1024,6 @@ cc_test(
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -1045,6 +1121,7 @@ cc_library(
|
||||
":constant_side_packet_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:collection_item_id",
|
||||
"//mediapipe/framework/port:integral_types",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
// Copyright 2019 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_NORMALIZED_LIST_CALCULATOR_H_ // NOLINT
|
||||
#define MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_NORMALIZED_LIST_CALCULATOR_H_ // NOLINT
|
||||
|
||||
#include "mediapipe/calculators/core/concatenate_vector_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||
#include "mediapipe/framework/port/canonical_errors.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
// Concatenates several NormalizedLandmarkList protos following stream index
|
||||
// order. This class assumes that every input stream contains a
|
||||
// NormalizedLandmarkList proto object.
|
||||
class ConcatenateNormalizedLandmarkListCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
RET_CHECK(cc->Inputs().NumEntries() != 0);
|
||||
RET_CHECK(cc->Outputs().NumEntries() == 1);
|
||||
|
||||
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
||||
cc->Inputs().Index(i).Set<NormalizedLandmarkList>();
|
||||
}
|
||||
|
||||
cc->Outputs().Index(0).Set<NormalizedLandmarkList>();
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) override {
|
||||
cc->SetOffset(TimestampDiff(0));
|
||||
only_emit_if_all_present_ =
|
||||
cc->Options<::mediapipe::ConcatenateVectorCalculatorOptions>()
|
||||
.only_emit_if_all_present();
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) override {
|
||||
if (only_emit_if_all_present_) {
|
||||
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
||||
if (cc->Inputs().Index(i).IsEmpty()) return ::mediapipe::OkStatus();
|
||||
}
|
||||
}
|
||||
|
||||
NormalizedLandmarkList output;
|
||||
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
||||
if (cc->Inputs().Index(i).IsEmpty()) continue;
|
||||
const NormalizedLandmarkList& input =
|
||||
cc->Inputs().Index(i).Get<NormalizedLandmarkList>();
|
||||
for (int j = 0; j < input.landmark_size(); ++j) {
|
||||
const NormalizedLandmark& input_landmark = input.landmark(j);
|
||||
*output.add_landmark() = input_landmark;
|
||||
}
|
||||
}
|
||||
cc->Outputs().Index(0).AddPacket(
|
||||
MakePacket<NormalizedLandmarkList>(output).At(cc->InputTimestamp()));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
private:
|
||||
bool only_emit_if_all_present_;
|
||||
};
|
||||
|
||||
REGISTER_CALCULATOR(ConcatenateNormalizedLandmarkListCalculator);
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
// NOLINTNEXTLINE
|
||||
#endif // MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_NORMALIZED_LIST_CALCULATOR_H_
|
||||
@@ -0,0 +1,184 @@
|
||||
// Copyright 2019 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||
#include "mediapipe/framework/port/gmock.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h" // NOLINT
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
constexpr float kLocationValue = 3;
|
||||
|
||||
NormalizedLandmarkList GenerateLandmarks(int landmarks_size,
|
||||
int value_multiplier) {
|
||||
NormalizedLandmarkList landmarks;
|
||||
for (int i = 0; i < landmarks_size; ++i) {
|
||||
NormalizedLandmark* landmark = landmarks.add_landmark();
|
||||
landmark->set_x(value_multiplier * kLocationValue);
|
||||
landmark->set_y(value_multiplier * kLocationValue);
|
||||
landmark->set_z(value_multiplier * kLocationValue);
|
||||
}
|
||||
return landmarks;
|
||||
}
|
||||
|
||||
void ValidateCombinedLandmarks(
|
||||
const std::vector<NormalizedLandmarkList>& inputs,
|
||||
const NormalizedLandmarkList& result) {
|
||||
int element_id = 0;
|
||||
int expected_size = 0;
|
||||
for (int i = 0; i < inputs.size(); ++i) {
|
||||
const NormalizedLandmarkList& landmarks_i = inputs[i];
|
||||
expected_size += landmarks_i.landmark_size();
|
||||
for (int j = 0; j < landmarks_i.landmark_size(); ++j) {
|
||||
const NormalizedLandmark& expected = landmarks_i.landmark(j);
|
||||
const NormalizedLandmark& got = result.landmark(element_id);
|
||||
EXPECT_FLOAT_EQ(expected.x(), got.x());
|
||||
EXPECT_FLOAT_EQ(expected.y(), got.y());
|
||||
EXPECT_FLOAT_EQ(expected.z(), got.z());
|
||||
++element_id;
|
||||
}
|
||||
}
|
||||
EXPECT_EQ(expected_size, result.landmark_size());
|
||||
}
|
||||
|
||||
void AddInputLandmarkLists(
|
||||
const std::vector<NormalizedLandmarkList>& input_landmarks_vec,
|
||||
int64 timestamp, CalculatorRunner* runner) {
|
||||
for (int i = 0; i < input_landmarks_vec.size(); ++i) {
|
||||
runner->MutableInputs()->Index(i).packets.push_back(
|
||||
MakePacket<NormalizedLandmarkList>(input_landmarks_vec[i])
|
||||
.At(Timestamp(timestamp)));
|
||||
}
|
||||
}
|
||||
|
||||
TEST(ConcatenateNormalizedLandmarkListCalculatorTest, EmptyVectorInputs) {
|
||||
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
NormalizedLandmarkList empty_list;
|
||||
std::vector<NormalizedLandmarkList> inputs = {empty_list, empty_list,
|
||||
empty_list};
|
||||
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(0, outputs[0].Get<NormalizedLandmarkList>().landmark_size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
}
|
||||
|
||||
TEST(ConcatenateNormalizedLandmarkListCalculatorTest, OneTimestamp) {
|
||||
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
NormalizedLandmarkList input_0 =
|
||||
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
|
||||
NormalizedLandmarkList input_1 =
|
||||
GenerateLandmarks(/*landmarks_size=*/1, /*value_multiplier=*/1);
|
||||
NormalizedLandmarkList input_2 =
|
||||
GenerateLandmarks(/*landmarks_size=*/2, /*value_multiplier=*/2);
|
||||
std::vector<NormalizedLandmarkList> inputs = {input_0, input_1, input_2};
|
||||
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
const NormalizedLandmarkList& result =
|
||||
outputs[0].Get<NormalizedLandmarkList>();
|
||||
ValidateCombinedLandmarks(inputs, result);
|
||||
}
|
||||
|
||||
TEST(ConcatenateNormalizedLandmarkListCalculatorTest,
|
||||
TwoInputsAtTwoTimestamps) {
|
||||
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
NormalizedLandmarkList input_0 =
|
||||
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
|
||||
NormalizedLandmarkList input_1 =
|
||||
GenerateLandmarks(/*landmarks_size=*/1, /*value_multiplier=*/1);
|
||||
NormalizedLandmarkList input_2 =
|
||||
GenerateLandmarks(/*landmarks_size=*/2, /*value_multiplier=*/2);
|
||||
std::vector<NormalizedLandmarkList> inputs = {input_0, input_1, input_2};
|
||||
{ AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner); }
|
||||
{ AddInputLandmarkLists(inputs, /*timestamp=*/2, &runner); }
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(2, outputs.size());
|
||||
{
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
const NormalizedLandmarkList& result =
|
||||
outputs[0].Get<NormalizedLandmarkList>();
|
||||
ValidateCombinedLandmarks(inputs, result);
|
||||
}
|
||||
{
|
||||
EXPECT_EQ(Timestamp(2), outputs[1].Timestamp());
|
||||
const NormalizedLandmarkList& result =
|
||||
outputs[1].Get<NormalizedLandmarkList>();
|
||||
ValidateCombinedLandmarks(inputs, result);
|
||||
}
|
||||
}
|
||||
|
||||
TEST(ConcatenateNormalizedLandmarkListCalculatorTest,
|
||||
OneEmptyStreamStillOutput) {
|
||||
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/2,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
NormalizedLandmarkList input_0 =
|
||||
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
|
||||
std::vector<NormalizedLandmarkList> inputs = {input_0};
|
||||
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
const NormalizedLandmarkList& result =
|
||||
outputs[0].Get<NormalizedLandmarkList>();
|
||||
ValidateCombinedLandmarks(inputs, result);
|
||||
}
|
||||
|
||||
TEST(ConcatenateNormalizedLandmarkListCalculatorTest, OneEmptyStreamNoOutput) {
|
||||
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
|
||||
/*options_string=*/
|
||||
"[mediapipe.ConcatenateVectorCalculatorOptions.ext]: "
|
||||
"{only_emit_if_all_present: true}",
|
||||
/*num_inputs=*/2,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
NormalizedLandmarkList input_0 =
|
||||
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
|
||||
std::vector<NormalizedLandmarkList> inputs = {input_0};
|
||||
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(0, outputs.size());
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
@@ -16,7 +16,9 @@
|
||||
|
||||
#include <vector>
|
||||
|
||||
#include "mediapipe/framework/formats/classification.pb.h"
|
||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||
#include "mediapipe/framework/port/integral_types.h"
|
||||
#include "tensorflow/lite/interpreter.h"
|
||||
|
||||
#if !defined(MEDIAPIPE_DISABLE_GL_COMPUTE)
|
||||
@@ -45,6 +47,9 @@ REGISTER_CALCULATOR(ConcatenateFloatVectorCalculator);
|
||||
typedef ConcatenateVectorCalculator<int32> ConcatenateInt32VectorCalculator;
|
||||
REGISTER_CALCULATOR(ConcatenateInt32VectorCalculator);
|
||||
|
||||
typedef ConcatenateVectorCalculator<uint64> ConcatenateUInt64VectorCalculator;
|
||||
REGISTER_CALCULATOR(ConcatenateUInt64VectorCalculator);
|
||||
|
||||
// Example config:
|
||||
// node {
|
||||
// calculator: "ConcatenateTfLiteTensorVectorCalculator"
|
||||
@@ -60,6 +65,14 @@ typedef ConcatenateVectorCalculator<::mediapipe::NormalizedLandmark>
|
||||
ConcatenateLandmarkVectorCalculator;
|
||||
REGISTER_CALCULATOR(ConcatenateLandmarkVectorCalculator);
|
||||
|
||||
typedef ConcatenateVectorCalculator<::mediapipe::NormalizedLandmarkList>
|
||||
ConcatenateLandmarListVectorCalculator;
|
||||
REGISTER_CALCULATOR(ConcatenateLandmarListVectorCalculator);
|
||||
|
||||
typedef ConcatenateVectorCalculator<mediapipe::ClassificationList>
|
||||
ConcatenateClassificationListVectorCalculator;
|
||||
REGISTER_CALCULATOR(ConcatenateClassificationListVectorCalculator);
|
||||
|
||||
#if !defined(MEDIAPIPE_DISABLE_GL_COMPUTE)
|
||||
typedef ConcatenateVectorCalculator<::tflite::gpu::gl::GlBuffer>
|
||||
ConcatenateGlBufferVectorCalculator;
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_VECTOR_CALCULATOR_H_
|
||||
#define MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_VECTOR_CALCULATOR_H_
|
||||
|
||||
#include <string>
|
||||
#include <type_traits>
|
||||
#include <vector>
|
||||
|
||||
@@ -26,10 +27,10 @@
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
// Concatenates several std::vector<T> following stream index order. This class
|
||||
// assumes that every input stream contains the vector<T> type. To use this
|
||||
// class for a particular type T, regisiter a calculator using
|
||||
// ConcatenateVectorCalculator<T>.
|
||||
// Concatenates several objects of type T or std::vector<T> following stream
|
||||
// index order. This class assumes that every input stream contains either T or
|
||||
// vector<T> type. To use this class for a particular type T, regisiter a
|
||||
// calculator using ConcatenateVectorCalculator<T>.
|
||||
template <typename T>
|
||||
class ConcatenateVectorCalculator : public CalculatorBase {
|
||||
public:
|
||||
@@ -38,7 +39,8 @@ class ConcatenateVectorCalculator : public CalculatorBase {
|
||||
RET_CHECK(cc->Outputs().NumEntries() == 1);
|
||||
|
||||
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
||||
cc->Inputs().Index(i).Set<std::vector<T>>();
|
||||
// Actual type T or vector<T> will be validated in Process().
|
||||
cc->Inputs().Index(i).SetAny();
|
||||
}
|
||||
|
||||
cc->Outputs().Index(0).Set<std::vector<T>>();
|
||||
@@ -69,9 +71,19 @@ class ConcatenateVectorCalculator : public CalculatorBase {
|
||||
CalculatorContext* cc) {
|
||||
auto output = absl::make_unique<std::vector<U>>();
|
||||
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
||||
if (cc->Inputs().Index(i).IsEmpty()) continue;
|
||||
const std::vector<U>& input = cc->Inputs().Index(i).Get<std::vector<U>>();
|
||||
output->insert(output->end(), input.begin(), input.end());
|
||||
auto& input = cc->Inputs().Index(i);
|
||||
|
||||
if (input.IsEmpty()) continue;
|
||||
|
||||
if (input.Value().ValidateAsType<U>().ok()) {
|
||||
const U& value = input.Get<U>();
|
||||
output->push_back(value);
|
||||
} else if (input.Value().ValidateAsType<std::vector<U>>().ok()) {
|
||||
const std::vector<U>& value = input.Get<std::vector<U>>();
|
||||
output->insert(output->end(), value.begin(), value.end());
|
||||
} else {
|
||||
return ::mediapipe::InvalidArgumentError("Invalid input stream type.");
|
||||
}
|
||||
}
|
||||
cc->Outputs().Index(0).Add(output.release(), cc->InputTimestamp());
|
||||
return ::mediapipe::OkStatus();
|
||||
@@ -88,17 +100,32 @@ class ConcatenateVectorCalculator : public CalculatorBase {
|
||||
CalculatorContext* cc) {
|
||||
auto output = absl::make_unique<std::vector<U>>();
|
||||
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
||||
if (cc->Inputs().Index(i).IsEmpty()) continue;
|
||||
::mediapipe::StatusOr<std::unique_ptr<std::vector<U>>> input_status =
|
||||
cc->Inputs().Index(i).Value().Consume<std::vector<U>>();
|
||||
if (input_status.ok()) {
|
||||
std::unique_ptr<std::vector<U>> input_vector =
|
||||
std::move(input_status).ValueOrDie();
|
||||
output->insert(output->end(),
|
||||
std::make_move_iterator(input_vector->begin()),
|
||||
std::make_move_iterator(input_vector->end()));
|
||||
auto& input = cc->Inputs().Index(i);
|
||||
|
||||
if (input.IsEmpty()) continue;
|
||||
|
||||
if (input.Value().ValidateAsType<U>().ok()) {
|
||||
::mediapipe::StatusOr<std::unique_ptr<U>> value_status =
|
||||
input.Value().Consume<U>();
|
||||
if (value_status.ok()) {
|
||||
std::unique_ptr<U> value = std::move(value_status).ValueOrDie();
|
||||
output->push_back(std::move(*value));
|
||||
} else {
|
||||
return value_status.status();
|
||||
}
|
||||
} else if (input.Value().ValidateAsType<std::vector<U>>().ok()) {
|
||||
::mediapipe::StatusOr<std::unique_ptr<std::vector<U>>> value_status =
|
||||
input.Value().Consume<std::vector<U>>();
|
||||
if (value_status.ok()) {
|
||||
std::unique_ptr<std::vector<U>> value =
|
||||
std::move(value_status).ValueOrDie();
|
||||
output->insert(output->end(), std::make_move_iterator(value->begin()),
|
||||
std::make_move_iterator(value->end()));
|
||||
} else {
|
||||
return value_status.status();
|
||||
}
|
||||
} else {
|
||||
return input_status.status();
|
||||
return ::mediapipe::InvalidArgumentError("Invalid input stream type.");
|
||||
}
|
||||
}
|
||||
cc->Outputs().Index(0).Add(output.release(), cc->InputTimestamp());
|
||||
@@ -109,7 +136,7 @@ class ConcatenateVectorCalculator : public CalculatorBase {
|
||||
::mediapipe::Status ConsumeAndConcatenateVectors(std::false_type,
|
||||
CalculatorContext* cc) {
|
||||
return ::mediapipe::InternalError(
|
||||
"Cannot copy or move input vectors to concatenate them");
|
||||
"Cannot copy or move inputs to concatenate them");
|
||||
}
|
||||
|
||||
private:
|
||||
|
||||
@@ -30,11 +30,29 @@ namespace mediapipe {
|
||||
typedef ConcatenateVectorCalculator<int> TestConcatenateIntVectorCalculator;
|
||||
REGISTER_CALCULATOR(TestConcatenateIntVectorCalculator);
|
||||
|
||||
void AddInputVector(int index, const std::vector<int>& input, int64 timestamp,
|
||||
CalculatorRunner* runner) {
|
||||
runner->MutableInputs()->Index(index).packets.push_back(
|
||||
MakePacket<std::vector<int>>(input).At(Timestamp(timestamp)));
|
||||
}
|
||||
|
||||
void AddInputVectors(const std::vector<std::vector<int>>& inputs,
|
||||
int64 timestamp, CalculatorRunner* runner) {
|
||||
for (int i = 0; i < inputs.size(); ++i) {
|
||||
runner->MutableInputs()->Index(i).packets.push_back(
|
||||
MakePacket<std::vector<int>>(inputs[i]).At(Timestamp(timestamp)));
|
||||
AddInputVector(i, inputs[i], timestamp, runner);
|
||||
}
|
||||
}
|
||||
|
||||
void AddInputItem(int index, int input, int64 timestamp,
|
||||
CalculatorRunner* runner) {
|
||||
runner->MutableInputs()->Index(index).packets.push_back(
|
||||
MakePacket<int>(input).At(Timestamp(timestamp)));
|
||||
}
|
||||
|
||||
void AddInputItems(const std::vector<int>& inputs, int64 timestamp,
|
||||
CalculatorRunner* runner) {
|
||||
for (int i = 0; i < inputs.size(); ++i) {
|
||||
AddInputItem(i, inputs[i], timestamp, runner);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -131,6 +149,135 @@ TEST(TestConcatenateIntVectorCalculatorTest, OneEmptyStreamNoOutput) {
|
||||
EXPECT_EQ(0, outputs.size());
|
||||
}
|
||||
|
||||
TEST(TestConcatenateIntVectorCalculatorTest, ItemsOneTimestamp) {
|
||||
CalculatorRunner runner("TestConcatenateIntVectorCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
std::vector<int> inputs = {1, 2, 3};
|
||||
AddInputItems(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
std::vector<int> expected_vector = {1, 2, 3};
|
||||
EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
|
||||
}
|
||||
|
||||
TEST(TestConcatenateIntVectorCalculatorTest, ItemsTwoInputsAtTwoTimestamps) {
|
||||
CalculatorRunner runner("TestConcatenateIntVectorCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
{
|
||||
std::vector<int> inputs = {1, 2, 3};
|
||||
AddInputItems(inputs, /*timestamp=*/1, &runner);
|
||||
}
|
||||
{
|
||||
std::vector<int> inputs = {4, 5, 6};
|
||||
AddInputItems(inputs, /*timestamp=*/2, &runner);
|
||||
}
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(2, outputs.size());
|
||||
{
|
||||
EXPECT_EQ(3, outputs[0].Get<std::vector<int>>().size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
std::vector<int> expected_vector = {1, 2, 3};
|
||||
EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
|
||||
}
|
||||
{
|
||||
EXPECT_EQ(3, outputs[1].Get<std::vector<int>>().size());
|
||||
EXPECT_EQ(Timestamp(2), outputs[1].Timestamp());
|
||||
std::vector<int> expected_vector = {4, 5, 6};
|
||||
EXPECT_EQ(expected_vector, outputs[1].Get<std::vector<int>>());
|
||||
}
|
||||
}
|
||||
|
||||
TEST(TestConcatenateIntVectorCalculatorTest, ItemsOneEmptyStreamStillOutput) {
|
||||
CalculatorRunner runner("TestConcatenateIntVectorCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
// No third input item.
|
||||
std::vector<int> inputs = {1, 2};
|
||||
AddInputItems(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
std::vector<int> expected_vector = {1, 2};
|
||||
EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
|
||||
}
|
||||
|
||||
TEST(TestConcatenateIntVectorCalculatorTest, ItemsOneEmptyStreamNoOutput) {
|
||||
CalculatorRunner runner("TestConcatenateIntVectorCalculator",
|
||||
/*options_string=*/
|
||||
"[mediapipe.ConcatenateVectorCalculatorOptions.ext]: "
|
||||
"{only_emit_if_all_present: true}",
|
||||
/*num_inputs=*/3,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
// No third input item.
|
||||
std::vector<int> inputs = {1, 2};
|
||||
AddInputItems(inputs, /*timestamp=*/1, &runner);
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(0, outputs.size());
|
||||
}
|
||||
|
||||
TEST(TestConcatenateIntVectorCalculatorTest, MixedVectorsAndItems) {
|
||||
CalculatorRunner runner("TestConcatenateIntVectorCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/4,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
std::vector<int> vector_0 = {1, 2};
|
||||
std::vector<int> vector_1 = {3, 4, 5};
|
||||
int item_0 = 6;
|
||||
int item_1 = 7;
|
||||
|
||||
AddInputVector(/*index*/ 0, vector_0, /*timestamp=*/1, &runner);
|
||||
AddInputVector(/*index*/ 1, vector_1, /*timestamp=*/1, &runner);
|
||||
AddInputItem(/*index*/ 2, item_0, /*timestamp=*/1, &runner);
|
||||
AddInputItem(/*index*/ 3, item_1, /*timestamp=*/1, &runner);
|
||||
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
std::vector<int> expected_vector = {1, 2, 3, 4, 5, 6, 7};
|
||||
EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
|
||||
}
|
||||
|
||||
TEST(TestConcatenateIntVectorCalculatorTest, MixedVectorsAndItemsAnother) {
|
||||
CalculatorRunner runner("TestConcatenateIntVectorCalculator",
|
||||
/*options_string=*/"", /*num_inputs=*/4,
|
||||
/*num_outputs=*/1, /*num_side_packets=*/0);
|
||||
|
||||
int item_0 = 1;
|
||||
std::vector<int> vector_0 = {2, 3};
|
||||
std::vector<int> vector_1 = {4, 5, 6};
|
||||
int item_1 = 7;
|
||||
|
||||
AddInputItem(/*index*/ 0, item_0, /*timestamp=*/1, &runner);
|
||||
AddInputVector(/*index*/ 1, vector_0, /*timestamp=*/1, &runner);
|
||||
AddInputVector(/*index*/ 2, vector_1, /*timestamp=*/1, &runner);
|
||||
AddInputItem(/*index*/ 3, item_1, /*timestamp=*/1, &runner);
|
||||
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
|
||||
EXPECT_EQ(1, outputs.size());
|
||||
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
|
||||
std::vector<int> expected_vector = {1, 2, 3, 4, 5, 6, 7};
|
||||
EXPECT_EQ(expected_vector, outputs[0].Get<std::vector<int>>());
|
||||
}
|
||||
|
||||
void AddInputVectors(const std::vector<std::vector<float>>& inputs,
|
||||
int64 timestamp, CalculatorRunner* runner) {
|
||||
for (int i = 0; i < inputs.size(); ++i) {
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/collection_item_id.h"
|
||||
#include "mediapipe/framework/port/canonical_errors.h"
|
||||
#include "mediapipe/framework/port/integral_types.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
|
||||
@@ -71,6 +72,8 @@ class ConstantSidePacketCalculator : public CalculatorBase {
|
||||
packet.Set<bool>();
|
||||
} else if (packet_options.has_string_value()) {
|
||||
packet.Set<std::string>();
|
||||
} else if (packet_options.has_uint64_value()) {
|
||||
packet.Set<uint64>();
|
||||
} else {
|
||||
return ::mediapipe::InvalidArgumentError(
|
||||
"None of supported values were specified in options.");
|
||||
@@ -95,6 +98,8 @@ class ConstantSidePacketCalculator : public CalculatorBase {
|
||||
packet.Set(MakePacket<bool>(packet_options.bool_value()));
|
||||
} else if (packet_options.has_string_value()) {
|
||||
packet.Set(MakePacket<std::string>(packet_options.string_value()));
|
||||
} else if (packet_options.has_uint64_value()) {
|
||||
packet.Set(MakePacket<uint64>(packet_options.uint64_value()));
|
||||
} else {
|
||||
return ::mediapipe::InvalidArgumentError(
|
||||
"None of supported values were specified in options.");
|
||||
|
||||
@@ -29,6 +29,7 @@ message ConstantSidePacketCalculatorOptions {
|
||||
float float_value = 2;
|
||||
bool bool_value = 3;
|
||||
string string_value = 4;
|
||||
uint64 uint64_value = 5;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -56,12 +56,19 @@ std::string ToString(GateState state) {
|
||||
// disallowing the corresponding packets in other input streams. The behavior
|
||||
// can be inverted with a calculator option.
|
||||
//
|
||||
// ALLOW or DISALLOW can also be specified as an input side packet. The rules
|
||||
// for evaluation remain the same as above.
|
||||
//
|
||||
// ALLOW/DISALLOW inputs must be specified either using input stream or
|
||||
// via input side packet but not both.
|
||||
//
|
||||
// Intended to be used with the default input stream handler, which synchronizes
|
||||
// all data input streams with the ALLOW/DISALLOW control input stream.
|
||||
//
|
||||
// Example config:
|
||||
// node {
|
||||
// calculator: "GateCalculator"
|
||||
// input_side_packet: "ALLOW:allow" or "DISALLOW:disallow"
|
||||
// input_stream: "input_stream0"
|
||||
// input_stream: "input_stream1"
|
||||
// input_stream: "input_streamN"
|
||||
@@ -75,10 +82,40 @@ class GateCalculator : public CalculatorBase {
|
||||
public:
|
||||
GateCalculator() {}
|
||||
|
||||
static ::mediapipe::Status CheckAndInitAllowDisallowInputs(
|
||||
CalculatorContract* cc) {
|
||||
bool input_via_side_packet = cc->InputSidePackets().HasTag("ALLOW") ||
|
||||
cc->InputSidePackets().HasTag("DISALLOW");
|
||||
bool input_via_stream =
|
||||
cc->Inputs().HasTag("ALLOW") || cc->Inputs().HasTag("DISALLOW");
|
||||
// Only one of input_side_packet or input_stream may specify ALLOW/DISALLOW
|
||||
// input.
|
||||
RET_CHECK(input_via_side_packet ^ input_via_stream);
|
||||
|
||||
if (input_via_side_packet) {
|
||||
RET_CHECK(cc->InputSidePackets().HasTag("ALLOW") ^
|
||||
cc->InputSidePackets().HasTag("DISALLOW"));
|
||||
|
||||
if (cc->InputSidePackets().HasTag("ALLOW")) {
|
||||
cc->InputSidePackets().Tag("ALLOW").Set<bool>();
|
||||
} else {
|
||||
cc->InputSidePackets().Tag("DISALLOW").Set<bool>();
|
||||
}
|
||||
} else {
|
||||
RET_CHECK(cc->Inputs().HasTag("ALLOW") ^ cc->Inputs().HasTag("DISALLOW"));
|
||||
|
||||
if (cc->Inputs().HasTag("ALLOW")) {
|
||||
cc->Inputs().Tag("ALLOW").Set<bool>();
|
||||
} else {
|
||||
cc->Inputs().Tag("DISALLOW").Set<bool>();
|
||||
}
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
// Assume that input streams do not have a tag and that gating signal is
|
||||
// tagged either ALLOW or DISALLOW.
|
||||
RET_CHECK(cc->Inputs().HasTag("ALLOW") ^ cc->Inputs().HasTag("DISALLOW"));
|
||||
RET_CHECK_OK(CheckAndInitAllowDisallowInputs(cc));
|
||||
|
||||
const int num_data_streams = cc->Inputs().NumEntries("");
|
||||
RET_CHECK_GE(num_data_streams, 1);
|
||||
RET_CHECK_EQ(cc->Outputs().NumEntries(""), num_data_streams)
|
||||
@@ -88,11 +125,6 @@ class GateCalculator : public CalculatorBase {
|
||||
cc->Inputs().Get("", i).SetAny();
|
||||
cc->Outputs().Get("", i).SetSameAs(&cc->Inputs().Get("", i));
|
||||
}
|
||||
if (cc->Inputs().HasTag("ALLOW")) {
|
||||
cc->Inputs().Tag("ALLOW").Set<bool>();
|
||||
} else {
|
||||
cc->Inputs().Tag("DISALLOW").Set<bool>();
|
||||
}
|
||||
|
||||
if (cc->Outputs().HasTag("STATE_CHANGE")) {
|
||||
cc->Outputs().Tag("STATE_CHANGE").Set<bool>();
|
||||
@@ -102,6 +134,17 @@ class GateCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) final {
|
||||
use_side_packet_for_allow_disallow_ = false;
|
||||
if (cc->InputSidePackets().HasTag("ALLOW")) {
|
||||
use_side_packet_for_allow_disallow_ = true;
|
||||
allow_by_side_packet_decision_ =
|
||||
cc->InputSidePackets().Tag("ALLOW").Get<bool>();
|
||||
} else if (cc->InputSidePackets().HasTag("DISALLOW")) {
|
||||
use_side_packet_for_allow_disallow_ = true;
|
||||
allow_by_side_packet_decision_ =
|
||||
!cc->InputSidePackets().Tag("DISALLOW").Get<bool>();
|
||||
}
|
||||
|
||||
cc->SetOffset(TimestampDiff(0));
|
||||
num_data_streams_ = cc->Inputs().NumEntries("");
|
||||
last_gate_state_ = GATE_UNINITIALIZED;
|
||||
@@ -115,14 +158,18 @@ class GateCalculator : public CalculatorBase {
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) final {
|
||||
bool allow = empty_packets_as_allow_;
|
||||
if (cc->Inputs().HasTag("ALLOW") && !cc->Inputs().Tag("ALLOW").IsEmpty()) {
|
||||
allow = cc->Inputs().Tag("ALLOW").Get<bool>();
|
||||
if (use_side_packet_for_allow_disallow_) {
|
||||
allow = allow_by_side_packet_decision_;
|
||||
} else {
|
||||
if (cc->Inputs().HasTag("ALLOW") &&
|
||||
!cc->Inputs().Tag("ALLOW").IsEmpty()) {
|
||||
allow = cc->Inputs().Tag("ALLOW").Get<bool>();
|
||||
}
|
||||
if (cc->Inputs().HasTag("DISALLOW") &&
|
||||
!cc->Inputs().Tag("DISALLOW").IsEmpty()) {
|
||||
allow = !cc->Inputs().Tag("DISALLOW").Get<bool>();
|
||||
}
|
||||
}
|
||||
if (cc->Inputs().HasTag("DISALLOW") &&
|
||||
!cc->Inputs().Tag("DISALLOW").IsEmpty()) {
|
||||
allow = !cc->Inputs().Tag("DISALLOW").Get<bool>();
|
||||
}
|
||||
|
||||
const GateState new_gate_state = allow ? GATE_ALLOW : GATE_DISALLOW;
|
||||
|
||||
if (cc->Outputs().HasTag("STATE_CHANGE")) {
|
||||
@@ -157,6 +204,8 @@ class GateCalculator : public CalculatorBase {
|
||||
GateState last_gate_state_ = GATE_UNINITIALIZED;
|
||||
int num_data_streams_;
|
||||
bool empty_packets_as_allow_;
|
||||
bool use_side_packet_for_allow_disallow_;
|
||||
bool allow_by_side_packet_decision_;
|
||||
};
|
||||
REGISTER_CALCULATOR(GateCalculator);
|
||||
|
||||
|
||||
@@ -24,6 +24,21 @@ namespace {
|
||||
|
||||
class GateCalculatorTest : public ::testing::Test {
|
||||
protected:
|
||||
// Helper to run a graph and return status.
|
||||
static ::mediapipe::Status RunGraph(const std::string& proto) {
|
||||
auto runner = absl::make_unique<CalculatorRunner>(
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(proto));
|
||||
return runner->Run();
|
||||
}
|
||||
|
||||
// Use this when ALLOW/DISALLOW input is provided as a side packet.
|
||||
void RunTimeStep(int64 timestamp, bool stream_payload) {
|
||||
runner_->MutableInputs()->Get("", 0).packets.push_back(
|
||||
MakePacket<bool>(stream_payload).At(Timestamp(timestamp)));
|
||||
MP_ASSERT_OK(runner_->Run()) << "Calculator execution failed.";
|
||||
}
|
||||
|
||||
// Use this when ALLOW/DISALLOW input is provided as an input stream.
|
||||
void RunTimeStep(int64 timestamp, const std::string& control_tag,
|
||||
bool control) {
|
||||
runner_->MutableInputs()->Get("", 0).packets.push_back(
|
||||
@@ -31,7 +46,6 @@ class GateCalculatorTest : public ::testing::Test {
|
||||
runner_->MutableInputs()
|
||||
->Tag(control_tag)
|
||||
.packets.push_back(MakePacket<bool>(control).At(Timestamp(timestamp)));
|
||||
|
||||
MP_ASSERT_OK(runner_->Run()) << "Calculator execution failed.";
|
||||
}
|
||||
|
||||
@@ -46,6 +60,136 @@ class GateCalculatorTest : public ::testing::Test {
|
||||
std::unique_ptr<CalculatorRunner> runner_;
|
||||
};
|
||||
|
||||
TEST_F(GateCalculatorTest, InvalidInputs) {
|
||||
EXPECT_TRUE(absl::IsInternal(GateCalculatorTest::RunGraph(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "test_input"
|
||||
input_stream: "ALLOW:gating_stream"
|
||||
input_stream: "DISALLOW:gating_stream"
|
||||
output_stream: "test_output"
|
||||
)")));
|
||||
|
||||
EXPECT_TRUE(absl::IsInternal(GateCalculatorTest::RunGraph(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "test_input"
|
||||
input_side_packet: "ALLOW:gating_stream"
|
||||
input_side_packet: "DISALLOW:gating_stream"
|
||||
output_stream: "test_output"
|
||||
)")));
|
||||
|
||||
EXPECT_TRUE(absl::IsInternal(GateCalculatorTest::RunGraph(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "test_input"
|
||||
input_stream: "ALLOW:gating_stream"
|
||||
input_side_packet: "ALLOW:gating_stream"
|
||||
output_stream: "test_output"
|
||||
)")));
|
||||
|
||||
EXPECT_TRUE(absl::IsInternal(GateCalculatorTest::RunGraph(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "test_input"
|
||||
input_stream: "DISALLOW:gating_stream"
|
||||
input_side_packet: "DISALLOW:gating_stream"
|
||||
output_stream: "test_output"
|
||||
)")));
|
||||
|
||||
EXPECT_TRUE(absl::IsInternal(GateCalculatorTest::RunGraph(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "test_input"
|
||||
input_stream: "ALLOW:gating_stream"
|
||||
input_side_packet: "DISALLOW:gating_stream"
|
||||
output_stream: "test_output"
|
||||
)")));
|
||||
|
||||
EXPECT_TRUE(absl::IsInternal(GateCalculatorTest::RunGraph(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_stream: "test_input"
|
||||
input_stream: "DISALLOW:gating_stream"
|
||||
input_side_packet: "ALLOW:gating_stream"
|
||||
output_stream: "test_output"
|
||||
)")));
|
||||
}
|
||||
|
||||
TEST_F(GateCalculatorTest, AllowByALLOWSidePacketSetToTrue) {
|
||||
SetRunner(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_side_packet: "ALLOW:gating_stream"
|
||||
input_stream: "test_input"
|
||||
output_stream: "test_output"
|
||||
)");
|
||||
runner()->MutableSidePackets()->Tag("ALLOW") = Adopt(new bool(true));
|
||||
|
||||
constexpr int64 kTimestampValue0 = 42;
|
||||
RunTimeStep(kTimestampValue0, true);
|
||||
constexpr int64 kTimestampValue1 = 43;
|
||||
RunTimeStep(kTimestampValue1, false);
|
||||
|
||||
const std::vector<Packet>& output = runner()->Outputs().Get("", 0).packets;
|
||||
ASSERT_EQ(2, output.size());
|
||||
EXPECT_EQ(kTimestampValue0, output[0].Timestamp().Value());
|
||||
EXPECT_EQ(kTimestampValue1, output[1].Timestamp().Value());
|
||||
EXPECT_EQ(true, output[0].Get<bool>());
|
||||
EXPECT_EQ(false, output[1].Get<bool>());
|
||||
}
|
||||
|
||||
TEST_F(GateCalculatorTest, AllowByDisallowSidePacketSetToFalse) {
|
||||
SetRunner(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_side_packet: "DISALLOW:gating_stream"
|
||||
input_stream: "test_input"
|
||||
output_stream: "test_output"
|
||||
)");
|
||||
runner()->MutableSidePackets()->Tag("DISALLOW") = Adopt(new bool(false));
|
||||
|
||||
constexpr int64 kTimestampValue0 = 42;
|
||||
RunTimeStep(kTimestampValue0, true);
|
||||
constexpr int64 kTimestampValue1 = 43;
|
||||
RunTimeStep(kTimestampValue1, false);
|
||||
|
||||
const std::vector<Packet>& output = runner()->Outputs().Get("", 0).packets;
|
||||
ASSERT_EQ(2, output.size());
|
||||
EXPECT_EQ(kTimestampValue0, output[0].Timestamp().Value());
|
||||
EXPECT_EQ(kTimestampValue1, output[1].Timestamp().Value());
|
||||
EXPECT_EQ(true, output[0].Get<bool>());
|
||||
EXPECT_EQ(false, output[1].Get<bool>());
|
||||
}
|
||||
|
||||
TEST_F(GateCalculatorTest, DisallowByALLOWSidePacketSetToFalse) {
|
||||
SetRunner(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_side_packet: "ALLOW:gating_stream"
|
||||
input_stream: "test_input"
|
||||
output_stream: "test_output"
|
||||
)");
|
||||
runner()->MutableSidePackets()->Tag("ALLOW") = Adopt(new bool(false));
|
||||
|
||||
constexpr int64 kTimestampValue0 = 42;
|
||||
RunTimeStep(kTimestampValue0, true);
|
||||
constexpr int64 kTimestampValue1 = 43;
|
||||
RunTimeStep(kTimestampValue1, false);
|
||||
|
||||
const std::vector<Packet>& output = runner()->Outputs().Get("", 0).packets;
|
||||
ASSERT_EQ(0, output.size());
|
||||
}
|
||||
|
||||
TEST_F(GateCalculatorTest, DisallowByDISALLOWSidePacketSetToTrue) {
|
||||
SetRunner(R"(
|
||||
calculator: "GateCalculator"
|
||||
input_side_packet: "DISALLOW:gating_stream"
|
||||
input_stream: "test_input"
|
||||
output_stream: "test_output"
|
||||
)");
|
||||
runner()->MutableSidePackets()->Tag("DISALLOW") = Adopt(new bool(true));
|
||||
|
||||
constexpr int64 kTimestampValue0 = 42;
|
||||
RunTimeStep(kTimestampValue0, true);
|
||||
constexpr int64 kTimestampValue1 = 43;
|
||||
RunTimeStep(kTimestampValue1, false);
|
||||
|
||||
const std::vector<Packet>& output = runner()->Outputs().Get("", 0).packets;
|
||||
ASSERT_EQ(0, output.size());
|
||||
}
|
||||
|
||||
TEST_F(GateCalculatorTest, Allow) {
|
||||
SetRunner(R"(
|
||||
calculator: "GateCalculator"
|
||||
|
||||
@@ -37,6 +37,10 @@ namespace mediapipe {
|
||||
// the RoundRobinDemuxCalculator. Therefore, packets from different
|
||||
// input streams are normally not expected to have the same timestamp.
|
||||
//
|
||||
// NOTE: this calculator can drop packets non-deterministically, depending on
|
||||
// how fast the input streams are fed. In most cases, MuxCalculator should be
|
||||
// preferred. In particular, dropping packets can interfere with rate limiting
|
||||
// mechanisms.
|
||||
class ImmediateMuxCalculator : public CalculatorBase {
|
||||
public:
|
||||
// This calculator combines any set of input streams into a single
|
||||
@@ -76,6 +80,9 @@ REGISTER_CALCULATOR(ImmediateMuxCalculator);
|
||||
if (!packet.IsEmpty()) {
|
||||
if (packet.Timestamp() >= cc->Outputs().Index(0).NextTimestampBound()) {
|
||||
cc->Outputs().Index(0).AddPacket(packet);
|
||||
} else {
|
||||
LOG_FIRST_N(WARNING, 5)
|
||||
<< "Dropping a packet with timestamp " << packet.Timestamp();
|
||||
}
|
||||
if (cc->Outputs().NumEntries() >= 2) {
|
||||
Timestamp output_timestamp = std::max(
|
||||
|
||||
@@ -17,28 +17,49 @@
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
namespace {
|
||||
constexpr char kSelectTag[] = "SELECT";
|
||||
constexpr char kInputTag[] = "INPUT";
|
||||
} // namespace
|
||||
|
||||
// A Calculator that selects an input stream from "INPUT:0", "INPUT:1", ...,
|
||||
// using the integer value (0, 1, ...) in the packet on the "SELECT" input
|
||||
// using the integer value (0, 1, ...) in the packet on the kSelectTag input
|
||||
// stream, and passes the packet on the selected input stream to the "OUTPUT"
|
||||
// output stream.
|
||||
// The kSelectTag input can also be passed in as an input side packet, instead
|
||||
// of as an input stream. Either of input stream or input side packet must be
|
||||
// specified but not both.
|
||||
//
|
||||
// Note that this calculator defaults to use MuxInputStreamHandler, which is
|
||||
// required for this calculator.
|
||||
// required for this calculator. However, it can be overridden to work with
|
||||
// other InputStreamHandlers. Check out the unit tests on for an example usage
|
||||
// with DefaultInputStreamHandler.
|
||||
class MuxCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status CheckAndInitAllowDisallowInputs(
|
||||
CalculatorContract* cc) {
|
||||
RET_CHECK(cc->Inputs().HasTag(kSelectTag) ^
|
||||
cc->InputSidePackets().HasTag(kSelectTag));
|
||||
if (cc->Inputs().HasTag(kSelectTag)) {
|
||||
cc->Inputs().Tag(kSelectTag).Set<int>();
|
||||
} else {
|
||||
cc->InputSidePackets().Tag(kSelectTag).Set<int>();
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
cc->Inputs().Tag("SELECT").Set<int>();
|
||||
CollectionItemId data_input_id = cc->Inputs().BeginId("INPUT");
|
||||
RET_CHECK_OK(CheckAndInitAllowDisallowInputs(cc));
|
||||
CollectionItemId data_input_id = cc->Inputs().BeginId(kInputTag);
|
||||
PacketType* data_input0 = &cc->Inputs().Get(data_input_id);
|
||||
data_input0->SetAny();
|
||||
++data_input_id;
|
||||
for (; data_input_id < cc->Inputs().EndId("INPUT"); ++data_input_id) {
|
||||
for (; data_input_id < cc->Inputs().EndId(kInputTag); ++data_input_id) {
|
||||
cc->Inputs().Get(data_input_id).SetSameAs(data_input0);
|
||||
}
|
||||
RET_CHECK_EQ(cc->Outputs().NumEntries(), 1);
|
||||
cc->Outputs().Tag("OUTPUT").SetSameAs(data_input0);
|
||||
|
||||
// Assign this calculator's default InputStreamHandler.
|
||||
cc->SetInputStreamHandler("MuxInputStreamHandler");
|
||||
MediaPipeOptions options;
|
||||
cc->SetInputStreamHandlerOptions(options);
|
||||
@@ -47,16 +68,24 @@ class MuxCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) final {
|
||||
select_input_ = cc->Inputs().GetId("SELECT", 0);
|
||||
data_input_base_ = cc->Inputs().GetId("INPUT", 0);
|
||||
num_data_inputs_ = cc->Inputs().NumEntries("INPUT");
|
||||
use_side_packet_select_ = false;
|
||||
if (cc->InputSidePackets().HasTag(kSelectTag)) {
|
||||
use_side_packet_select_ = true;
|
||||
selected_index_ = cc->InputSidePackets().Tag(kSelectTag).Get<int>();
|
||||
} else {
|
||||
select_input_ = cc->Inputs().GetId(kSelectTag, 0);
|
||||
}
|
||||
data_input_base_ = cc->Inputs().GetId(kInputTag, 0);
|
||||
num_data_inputs_ = cc->Inputs().NumEntries(kInputTag);
|
||||
output_ = cc->Outputs().GetId("OUTPUT", 0);
|
||||
cc->SetOffset(TimestampDiff(0));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) final {
|
||||
int select = cc->Inputs().Get(select_input_).Get<int>();
|
||||
int select = use_side_packet_select_
|
||||
? selected_index_
|
||||
: cc->Inputs().Get(select_input_).Get<int>();
|
||||
RET_CHECK(0 <= select && select < num_data_inputs_);
|
||||
if (!cc->Inputs().Get(data_input_base_ + select).IsEmpty()) {
|
||||
cc->Outputs().Get(output_).AddPacket(
|
||||
@@ -70,6 +99,8 @@ class MuxCalculator : public CalculatorBase {
|
||||
CollectionItemId data_input_base_;
|
||||
int num_data_inputs_ = 0;
|
||||
CollectionItemId output_;
|
||||
bool use_side_packet_select_;
|
||||
int selected_index_;
|
||||
};
|
||||
|
||||
REGISTER_CALCULATOR(MuxCalculator);
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
// Copyright 2019 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include "mediapipe/calculators/core/split_vector_calculator.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
typedef SplitVectorCalculator<int, false> SplitIntVectorCalculator;
|
||||
REGISTER_CALCULATOR(SplitIntVectorCalculator);
|
||||
|
||||
namespace {
|
||||
|
||||
// Graph with default input stream handler, and the input selection is driven
|
||||
// by an input stream. All MuxCalculator inputs are present at each timestamp.
|
||||
constexpr char kTestGraphConfig1[] = R"proto(
|
||||
input_stream: "input"
|
||||
output_stream: "test_output"
|
||||
node {
|
||||
calculator: "SplitIntVectorCalculator"
|
||||
input_stream: "input"
|
||||
output_stream: "stream0"
|
||||
output_stream: "stream1"
|
||||
output_stream: "stream2"
|
||||
output_stream: "input_select"
|
||||
options {
|
||||
[mediapipe.SplitVectorCalculatorOptions.ext] {
|
||||
ranges: { begin: 0 end: 1 }
|
||||
ranges: { begin: 1 end: 2 }
|
||||
ranges: { begin: 2 end: 3 }
|
||||
ranges: { begin: 3 end: 4 }
|
||||
element_only: true
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
calculator: "MuxCalculator"
|
||||
input_stream: "INPUT:0:stream0"
|
||||
input_stream: "INPUT:1:stream1"
|
||||
input_stream: "INPUT:2:stream2"
|
||||
input_stream: "SELECT:input_select"
|
||||
output_stream: "OUTPUT:test_output"
|
||||
input_stream_handler { input_stream_handler: "DefaultInputStreamHandler" }
|
||||
}
|
||||
)proto";
|
||||
|
||||
// Graph with default input stream handler, and the input selection is driven
|
||||
// by an input side packet. All MuxCalculator inputs are present at each
|
||||
// timestamp.
|
||||
constexpr char kTestGraphConfig2[] = R"proto(
|
||||
input_side_packet: "input_selector"
|
||||
input_stream: "input"
|
||||
output_stream: "test_output"
|
||||
node {
|
||||
calculator: "SplitIntVectorCalculator"
|
||||
input_stream: "input"
|
||||
output_stream: "stream0"
|
||||
output_stream: "stream1"
|
||||
output_stream: "stream2"
|
||||
options {
|
||||
[mediapipe.SplitVectorCalculatorOptions.ext] {
|
||||
ranges: { begin: 0 end: 1 }
|
||||
ranges: { begin: 1 end: 2 }
|
||||
ranges: { begin: 2 end: 3 }
|
||||
element_only: true
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
calculator: "MuxCalculator"
|
||||
input_stream: "INPUT:0:stream0"
|
||||
input_stream: "INPUT:1:stream1"
|
||||
input_stream: "INPUT:2:stream2"
|
||||
input_side_packet: "SELECT:input_selector"
|
||||
output_stream: "OUTPUT:test_output"
|
||||
input_stream_handler { input_stream_handler: "DefaultInputStreamHandler" }
|
||||
}
|
||||
)proto";
|
||||
|
||||
// Graph with mux input stream handler, and the input selection is driven
|
||||
// by an input stream. Only one MuxCalculator input is present at each
|
||||
// timestamp.
|
||||
constexpr char kTestGraphConfig3[] = R"proto(
|
||||
input_stream: "input"
|
||||
output_stream: "test_output"
|
||||
node {
|
||||
calculator: "RoundRobinDemuxCalculator"
|
||||
input_stream: "input"
|
||||
output_stream: "OUTPUT:0:stream0"
|
||||
output_stream: "OUTPUT:1:stream1"
|
||||
output_stream: "OUTPUT:2:stream2"
|
||||
output_stream: "SELECT:input_select"
|
||||
}
|
||||
node {
|
||||
calculator: "MuxCalculator"
|
||||
input_stream: "INPUT:0:stream0"
|
||||
input_stream: "INPUT:1:stream1"
|
||||
input_stream: "INPUT:2:stream2"
|
||||
input_stream: "SELECT:input_select"
|
||||
output_stream: "OUTPUT:test_output"
|
||||
}
|
||||
)proto";
|
||||
|
||||
constexpr char kOutputName[] = "test_output";
|
||||
constexpr char kInputName[] = "input";
|
||||
constexpr char kInputSelector[] = "input_selector";
|
||||
|
||||
// Helper to run a graph with the given inputs and generate outputs, asserting
|
||||
// each step along the way.
|
||||
// Inputs:
|
||||
// graph_config_proto - graph config protobuf
|
||||
// extra_side_packets - input side packets name to value map
|
||||
// input_stream_name - name of the input
|
||||
void RunGraph(const std::string& graph_config_proto,
|
||||
const std::map<std::string, Packet>& extra_side_packets,
|
||||
const std::string& input_stream_name, int num_input_packets,
|
||||
std::function<Packet(int)> input_fn,
|
||||
const std::string& output_stream_name,
|
||||
std::function<::mediapipe::Status(const Packet&)> output_fn) {
|
||||
CalculatorGraphConfig config =
|
||||
::mediapipe::ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
graph_config_proto);
|
||||
CalculatorGraph graph;
|
||||
MP_ASSERT_OK(graph.Initialize(config));
|
||||
MP_ASSERT_OK(graph.ObserveOutputStream(output_stream_name, output_fn));
|
||||
MP_ASSERT_OK(graph.StartRun(extra_side_packets));
|
||||
for (int i = 0; i < num_input_packets; ++i) {
|
||||
MP_ASSERT_OK(graph.AddPacketToInputStream(input_stream_name, input_fn(i)));
|
||||
}
|
||||
MP_ASSERT_OK(graph.CloseAllInputStreams());
|
||||
MP_ASSERT_OK(graph.WaitUntilDone());
|
||||
}
|
||||
|
||||
TEST(MuxCalculatorTest, InputStreamSelector_DefaultInputStreamHandler) {
|
||||
// Input and handling.
|
||||
std::vector<std::vector<int>> input_packets = {
|
||||
{1, 1, 2, 1}, {3, 5, 8, 2}, {13, 21, 34, 0},
|
||||
{55, 89, 144, 2}, {233, 377, 610, 0}, {987, 1597, 2584, 1},
|
||||
{4181, 6765, 10946, 2},
|
||||
};
|
||||
int packet_time_stamp = 22;
|
||||
// This function will return the i-th input packet.
|
||||
auto input_fn = [&packet_time_stamp, &input_packets](int i) -> Packet {
|
||||
return MakePacket<std::vector<int>>(input_packets[i])
|
||||
.At(Timestamp(packet_time_stamp++));
|
||||
};
|
||||
|
||||
// Output and handling.
|
||||
std::vector<int> output;
|
||||
// This function collects the output from the packet.
|
||||
auto output_fn = [&output](const Packet& p) -> ::mediapipe::Status {
|
||||
output.push_back(p.Get<int>());
|
||||
return ::mediapipe::OkStatus();
|
||||
};
|
||||
|
||||
RunGraph(kTestGraphConfig1, {}, kInputName, input_packets.size(), input_fn,
|
||||
kOutputName, output_fn);
|
||||
EXPECT_THAT(output, testing::ElementsAre(1, 8, 13, 144, 233, 1597, 10946));
|
||||
}
|
||||
|
||||
TEST(MuxCalculatorTest, InputSidePacketSelector_DefaultInputStreamHandler) {
|
||||
// Input and handling.
|
||||
std::vector<std::vector<int>> input_packets = {
|
||||
{1, 1, 2}, {3, 5, 8}, {13, 21, 34}, {55, 89, 144},
|
||||
{233, 377, 610}, {987, 1597, 2584}, {4181, 6765, 10946},
|
||||
};
|
||||
int packet_time_stamp = 22;
|
||||
// This function will return the i-th input packet.
|
||||
auto input_fn = [&packet_time_stamp, &input_packets](int i) -> Packet {
|
||||
return MakePacket<std::vector<int>>(input_packets[i])
|
||||
.At(Timestamp(packet_time_stamp++));
|
||||
};
|
||||
|
||||
// Output and handling.
|
||||
std::vector<int> output;
|
||||
// This function collects the output from the packet.
|
||||
auto output_fn = [&output](const Packet& p) -> ::mediapipe::Status {
|
||||
output.push_back(p.Get<int>());
|
||||
return ::mediapipe::OkStatus();
|
||||
};
|
||||
|
||||
RunGraph(kTestGraphConfig2, {{kInputSelector, MakePacket<int>(0)}},
|
||||
kInputName, input_packets.size(), input_fn, kOutputName, output_fn);
|
||||
EXPECT_THAT(output, testing::ElementsAre(1, 3, 13, 55, 233, 987, 4181));
|
||||
|
||||
output.clear();
|
||||
RunGraph(kTestGraphConfig2, {{kInputSelector, MakePacket<int>(1)}},
|
||||
kInputName, input_packets.size(), input_fn, kOutputName, output_fn);
|
||||
EXPECT_THAT(output, testing::ElementsAre(1, 5, 21, 89, 377, 1597, 6765));
|
||||
|
||||
output.clear();
|
||||
RunGraph(kTestGraphConfig2, {{kInputSelector, MakePacket<int>(2)}},
|
||||
kInputName, input_packets.size(), input_fn, kOutputName, output_fn);
|
||||
EXPECT_THAT(output, testing::ElementsAre(2, 8, 34, 144, 610, 2584, 10946));
|
||||
}
|
||||
|
||||
TEST(MuxCalculatorTest, InputStreamSelector_MuxInputStreamHandler) {
|
||||
// Input and handling.
|
||||
std::vector<int> input_packets = {1, 1, 2, 3, 5, 8, 13,
|
||||
21, 34, 55, 89, 144, 233, 377,
|
||||
610, 987, 1597, 2584, 4181, 6765, 10946};
|
||||
int packet_time_stamp = 22;
|
||||
// This function will return the i-th input packet.
|
||||
auto input_fn = [&packet_time_stamp, &input_packets](int i) -> Packet {
|
||||
return MakePacket<int>(input_packets[i]).At(Timestamp(packet_time_stamp++));
|
||||
};
|
||||
|
||||
// Output and handling.
|
||||
std::vector<int> output;
|
||||
// This function collects the output from the packet.
|
||||
auto output_fn = [&output](const Packet& p) -> ::mediapipe::Status {
|
||||
output.push_back(p.Get<int>());
|
||||
return ::mediapipe::OkStatus();
|
||||
};
|
||||
|
||||
RunGraph(kTestGraphConfig3, {}, kInputName, input_packets.size(), input_fn,
|
||||
kOutputName, output_fn);
|
||||
EXPECT_EQ(output, input_packets);
|
||||
}
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
@@ -0,0 +1,84 @@
|
||||
// Copyright 2020 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
// For each non empty input packet, emits a single output packet containing a
|
||||
// boolean value "true", "false" in response to empty packets (a.k.a. timestamp
|
||||
// bound updates) This can be used to "flag" the presence of an arbitrary packet
|
||||
// type as input into a downstream calculator.
|
||||
//
|
||||
// Inputs:
|
||||
// PACKET - any type.
|
||||
//
|
||||
// Outputs:
|
||||
// PRESENCE - bool.
|
||||
// "true" if packet is not empty, "false" if there's timestamp bound update
|
||||
// instead.
|
||||
//
|
||||
// Examples:
|
||||
// node: {
|
||||
// calculator: "PacketPresenceCalculator"
|
||||
// input_stream: "PACKET:packet"
|
||||
// output_stream: "PRESENCE:presence"
|
||||
// }
|
||||
//
|
||||
// This calculator can be used in conjuction with GateCalculator in order to
|
||||
// allow/disallow processing. For instance:
|
||||
// node: {
|
||||
// calculator: "PacketPresenceCalculator"
|
||||
// input_stream: "PACKET:value"
|
||||
// output_stream: "PRESENCE:disallow_if_present"
|
||||
// }
|
||||
// node {
|
||||
// calculator: "GateCalculator"
|
||||
// input_stream: "image"
|
||||
// input_stream: "DISALLOW:disallow_if_present"
|
||||
// output_stream: "image_for_processing"
|
||||
// options: {
|
||||
// [mediapipe.GateCalculatorOptions.ext] {
|
||||
// empty_packets_as_allow: true
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
class PacketPresenceCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
cc->Inputs().Tag("PACKET").SetAny();
|
||||
cc->Outputs().Tag("PRESENCE").Set<bool>();
|
||||
// Process() function is invoked in response to input stream timestamp
|
||||
// bound updates.
|
||||
cc->SetProcessTimestampBounds(true);
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) override {
|
||||
cc->SetOffset(TimestampDiff(0));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) final {
|
||||
cc->Outputs()
|
||||
.Tag("PRESENCE")
|
||||
.AddPacket(MakePacket<bool>(!cc->Inputs().Tag("PACKET").IsEmpty())
|
||||
.At(cc->InputTimestamp()));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
};
|
||||
REGISTER_CALCULATOR(PacketPresenceCalculator);
|
||||
|
||||
} // namespace mediapipe
|
||||
@@ -0,0 +1,85 @@
|
||||
// Copyright 2020 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <functional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/port/gmock.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
#include "mediapipe/framework/timestamp.h"
|
||||
#include "mediapipe/framework/tool/sink.h"
|
||||
|
||||
namespace mediapipe {
|
||||
using ::testing::ElementsAre;
|
||||
using ::testing::Eq;
|
||||
using ::testing::Value;
|
||||
namespace {
|
||||
|
||||
MATCHER_P2(BoolPacket, value, timestamp, "") {
|
||||
return Value(arg.template Get<bool>(), Eq(value)) &&
|
||||
Value(arg.Timestamp(), Eq(timestamp));
|
||||
}
|
||||
|
||||
TEST(PreviousLoopbackCalculator, CorrectTimestamps) {
|
||||
std::vector<Packet> output_packets;
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(R"(
|
||||
input_stream: 'allow'
|
||||
input_stream: 'value'
|
||||
node {
|
||||
calculator: "GateCalculator"
|
||||
input_stream: 'value'
|
||||
input_stream: 'ALLOW:allow'
|
||||
output_stream: 'gated_value'
|
||||
}
|
||||
node {
|
||||
calculator: 'PacketPresenceCalculator'
|
||||
input_stream: 'PACKET:gated_value'
|
||||
output_stream: 'PRESENCE:presence'
|
||||
}
|
||||
)");
|
||||
tool::AddVectorSink("presence", &graph_config, &output_packets);
|
||||
|
||||
CalculatorGraph graph;
|
||||
MP_ASSERT_OK(graph.Initialize(graph_config, {}));
|
||||
MP_ASSERT_OK(graph.StartRun({}));
|
||||
|
||||
auto send_packet = [&graph](int value, bool allow, Timestamp timestamp) {
|
||||
MP_ASSERT_OK(graph.AddPacketToInputStream(
|
||||
"value", MakePacket<int>(value).At(timestamp)));
|
||||
MP_ASSERT_OK(graph.AddPacketToInputStream(
|
||||
"allow", MakePacket<bool>(allow).At(timestamp)));
|
||||
};
|
||||
|
||||
send_packet(10, false, Timestamp(10));
|
||||
MP_EXPECT_OK(graph.WaitUntilIdle());
|
||||
EXPECT_THAT(output_packets, ElementsAre(BoolPacket(false, Timestamp(10))));
|
||||
|
||||
output_packets.clear();
|
||||
send_packet(20, true, Timestamp(11));
|
||||
MP_EXPECT_OK(graph.WaitUntilIdle());
|
||||
EXPECT_THAT(output_packets, ElementsAre(BoolPacket(true, Timestamp(11))));
|
||||
|
||||
MP_EXPECT_OK(graph.CloseAllInputStreams());
|
||||
MP_EXPECT_OK(graph.WaitUntilDone());
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
@@ -128,11 +128,17 @@ class PreviousLoopbackCalculator : public CalculatorBase {
|
||||
loop_packets_.pop_front();
|
||||
main_packet_specs_.pop_front();
|
||||
}
|
||||
|
||||
// We can close PREV_LOOP output stream as soon as we processed last
|
||||
// possible MAIN packet. That can happen in two cases:
|
||||
// a) Non-empty MAIN packet has been received with Timestamp::Max()
|
||||
// b) Empty MAIN packet has been received with Timestamp::Max() indicating
|
||||
// MAIN is done.
|
||||
if (main_spec.timestamp == Timestamp::Done().PreviousAllowedInStream()) {
|
||||
prev_loop.Close();
|
||||
}
|
||||
}
|
||||
|
||||
if (main_packet_specs_.empty() && cc->Inputs().Get(main_id_).IsDone()) {
|
||||
prev_loop.Close();
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
|
||||
@@ -228,6 +228,104 @@ TEST(PreviousLoopbackCalculator, ClosesCorrectly) {
|
||||
MP_EXPECT_OK(graph_.WaitUntilDone());
|
||||
}
|
||||
|
||||
TEST(PreviousLoopbackCalculator, ProcessesMaxTimestamp) {
|
||||
std::vector<Packet> out_and_previous_packets;
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(R"(
|
||||
input_stream: 'in'
|
||||
node {
|
||||
calculator: 'PreviousLoopbackCalculator'
|
||||
input_stream: 'MAIN:in'
|
||||
input_stream: 'LOOP:out'
|
||||
input_stream_info: { tag_index: 'LOOP' back_edge: true }
|
||||
output_stream: 'PREV_LOOP:previous'
|
||||
}
|
||||
node {
|
||||
calculator: 'PassThroughCalculator'
|
||||
input_stream: 'in'
|
||||
input_stream: 'previous'
|
||||
output_stream: 'out'
|
||||
output_stream: 'previous2'
|
||||
}
|
||||
node {
|
||||
calculator: 'MakePairCalculator'
|
||||
input_stream: 'out'
|
||||
input_stream: 'previous'
|
||||
output_stream: 'out_and_previous'
|
||||
}
|
||||
)");
|
||||
tool::AddVectorSink("out_and_previous", &graph_config,
|
||||
&out_and_previous_packets);
|
||||
|
||||
CalculatorGraph graph;
|
||||
MP_ASSERT_OK(graph.Initialize(graph_config, {}));
|
||||
MP_ASSERT_OK(graph.StartRun({}));
|
||||
|
||||
MP_EXPECT_OK(graph.AddPacketToInputStream(
|
||||
"in", MakePacket<int>(1).At(Timestamp::Max())));
|
||||
|
||||
MP_EXPECT_OK(graph.WaitUntilIdle());
|
||||
|
||||
EXPECT_THAT(out_and_previous_packets,
|
||||
ElementsAre(PairPacket(Timestamp::Max(),
|
||||
Pair(IntPacket(1), EmptyPacket()))));
|
||||
|
||||
MP_EXPECT_OK(graph.CloseAllInputStreams());
|
||||
MP_EXPECT_OK(graph.WaitUntilIdle());
|
||||
MP_EXPECT_OK(graph.WaitUntilDone());
|
||||
}
|
||||
|
||||
TEST(PreviousLoopbackCalculator, ProcessesMaxTimestampNonEmptyPrevious) {
|
||||
std::vector<Packet> out_and_previous_packets;
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(R"(
|
||||
input_stream: 'in'
|
||||
node {
|
||||
calculator: 'PreviousLoopbackCalculator'
|
||||
input_stream: 'MAIN:in'
|
||||
input_stream: 'LOOP:out'
|
||||
input_stream_info: { tag_index: 'LOOP' back_edge: true }
|
||||
output_stream: 'PREV_LOOP:previous'
|
||||
}
|
||||
node {
|
||||
calculator: 'PassThroughCalculator'
|
||||
input_stream: 'in'
|
||||
input_stream: 'previous'
|
||||
output_stream: 'out'
|
||||
output_stream: 'previous2'
|
||||
}
|
||||
node {
|
||||
calculator: 'MakePairCalculator'
|
||||
input_stream: 'out'
|
||||
input_stream: 'previous'
|
||||
output_stream: 'out_and_previous'
|
||||
}
|
||||
)");
|
||||
tool::AddVectorSink("out_and_previous", &graph_config,
|
||||
&out_and_previous_packets);
|
||||
|
||||
CalculatorGraph graph;
|
||||
MP_ASSERT_OK(graph.Initialize(graph_config, {}));
|
||||
MP_ASSERT_OK(graph.StartRun({}));
|
||||
|
||||
MP_EXPECT_OK(graph.AddPacketToInputStream(
|
||||
"in", MakePacket<int>(1).At(Timestamp::Min())));
|
||||
MP_EXPECT_OK(graph.AddPacketToInputStream(
|
||||
"in", MakePacket<int>(2).At(Timestamp::Max())));
|
||||
|
||||
MP_EXPECT_OK(graph.WaitUntilIdle());
|
||||
|
||||
EXPECT_THAT(
|
||||
out_and_previous_packets,
|
||||
ElementsAre(
|
||||
PairPacket(Timestamp::Min(), Pair(IntPacket(1), EmptyPacket())),
|
||||
PairPacket(Timestamp::Max(), Pair(IntPacket(2), IntPacket(1)))));
|
||||
|
||||
MP_EXPECT_OK(graph.CloseAllInputStreams());
|
||||
MP_EXPECT_OK(graph.WaitUntilIdle());
|
||||
MP_EXPECT_OK(graph.WaitUntilDone());
|
||||
}
|
||||
|
||||
// Demonstrates that downstream calculators won't be blocked by
|
||||
// always-empty-LOOP-stream.
|
||||
TEST(PreviousLoopbackCalculator, EmptyLoopForever) {
|
||||
|
||||
@@ -34,6 +34,8 @@ constexpr char kTagAtPostStream[] = "AT_POSTSTREAM";
|
||||
constexpr char kTagAtZero[] = "AT_ZERO";
|
||||
constexpr char kTagAtTick[] = "AT_TICK";
|
||||
constexpr char kTagTick[] = "TICK";
|
||||
constexpr char kTagAtTimestamp[] = "AT_TIMESTAMP";
|
||||
constexpr char kTagSideInputTimestamp[] = "TIMESTAMP";
|
||||
|
||||
static std::map<std::string, Timestamp>* kTimestampMap = []() {
|
||||
auto* res = new std::map<std::string, Timestamp>();
|
||||
@@ -41,6 +43,7 @@ static std::map<std::string, Timestamp>* kTimestampMap = []() {
|
||||
res->emplace(kTagAtPostStream, Timestamp::PostStream());
|
||||
res->emplace(kTagAtZero, Timestamp(0));
|
||||
res->emplace(kTagAtTick, Timestamp::Unset());
|
||||
res->emplace(kTagAtTimestamp, Timestamp::Unset());
|
||||
return res;
|
||||
}();
|
||||
|
||||
@@ -56,9 +59,10 @@ std::string GetOutputTag(const CC& cc) {
|
||||
// timestamp, depending on the tag used to define output stream(s). (One tag can
|
||||
// be used only.)
|
||||
//
|
||||
// Valid tags are AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO and AT_TICK and
|
||||
// corresponding timestamps are Timestamp::PreStream(), Timestamp::PostStream(),
|
||||
// Timestamp(0) and timestamp of a packet received in TICK input.
|
||||
// Valid tags are AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO, AT_TICK, AT_TIMESTAMP
|
||||
// and corresponding timestamps are Timestamp::PreStream(),
|
||||
// Timestamp::PostStream(), Timestamp(0), timestamp of a packet received in TICK
|
||||
// input, and timestamp received from a side input.
|
||||
//
|
||||
// Examples:
|
||||
// node {
|
||||
@@ -73,6 +77,13 @@ std::string GetOutputTag(const CC& cc) {
|
||||
// input_side_packet: "side_packet"
|
||||
// output_stream: "AT_TICK:packet"
|
||||
// }
|
||||
//
|
||||
// node {
|
||||
// calculator: "SidePacketToStreamCalculator"
|
||||
// input_side_packet: "TIMESTAMP:timestamp"
|
||||
// input_side_packet: "side_packet"
|
||||
// output_stream: "AT_TIMESTAMP:packet"
|
||||
// }
|
||||
class SidePacketToStreamCalculator : public CalculatorBase {
|
||||
public:
|
||||
SidePacketToStreamCalculator() = default;
|
||||
@@ -93,16 +104,29 @@ REGISTER_CALCULATOR(SidePacketToStreamCalculator);
|
||||
CalculatorContract* cc) {
|
||||
const auto& tags = cc->Outputs().GetTags();
|
||||
RET_CHECK(tags.size() == 1 && kTimestampMap->count(*tags.begin()) == 1)
|
||||
<< "Only one of AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO and AT_TICK tags is "
|
||||
"allowed and required to specify output stream(s).";
|
||||
<< "Only one of AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO, AT_TICK and "
|
||||
"AT_TIMESTAMP tags is allowed and required to specify output "
|
||||
"stream(s).";
|
||||
RET_CHECK(
|
||||
(cc->Outputs().HasTag(kTagAtTick) && cc->Inputs().HasTag(kTagTick)) ||
|
||||
(!cc->Outputs().HasTag(kTagAtTick) && !cc->Inputs().HasTag(kTagTick)))
|
||||
<< "Either both of TICK and AT_TICK should be used or none of them.";
|
||||
RET_CHECK((cc->Outputs().HasTag(kTagAtTimestamp) &&
|
||||
cc->InputSidePackets().HasTag(kTagSideInputTimestamp)) ||
|
||||
(!cc->Outputs().HasTag(kTagAtTimestamp) &&
|
||||
!cc->InputSidePackets().HasTag(kTagSideInputTimestamp)))
|
||||
<< "Either both TIMESTAMP and AT_TIMESTAMP should be used or none of "
|
||||
"them.";
|
||||
const std::string output_tag = GetOutputTag(*cc);
|
||||
const int num_entries = cc->Outputs().NumEntries(output_tag);
|
||||
RET_CHECK_EQ(num_entries, cc->InputSidePackets().NumEntries())
|
||||
<< "Same number of input side packets and output streams is required.";
|
||||
if (cc->Outputs().HasTag(kTagAtTimestamp)) {
|
||||
RET_CHECK_EQ(num_entries + 1, cc->InputSidePackets().NumEntries())
|
||||
<< "For AT_TIMESTAMP tag, 2 input side packets are required.";
|
||||
cc->InputSidePackets().Tag(kTagSideInputTimestamp).Set<int64>();
|
||||
} else {
|
||||
RET_CHECK_EQ(num_entries, cc->InputSidePackets().NumEntries())
|
||||
<< "Same number of input side packets and output streams is required.";
|
||||
}
|
||||
for (int i = 0; i < num_entries; ++i) {
|
||||
cc->InputSidePackets().Index(i).SetAny();
|
||||
cc->Outputs()
|
||||
@@ -147,13 +171,22 @@ REGISTER_CALCULATOR(SidePacketToStreamCalculator);
|
||||
}
|
||||
|
||||
::mediapipe::Status SidePacketToStreamCalculator::Close(CalculatorContext* cc) {
|
||||
if (!cc->Outputs().HasTag(kTagAtTick)) {
|
||||
if (!cc->Outputs().HasTag(kTagAtTick) &&
|
||||
!cc->Outputs().HasTag(kTagAtTimestamp)) {
|
||||
const auto& timestamp = kTimestampMap->at(output_tag_);
|
||||
for (int i = 0; i < cc->Outputs().NumEntries(output_tag_); ++i) {
|
||||
cc->Outputs()
|
||||
.Get(output_tag_, i)
|
||||
.AddPacket(cc->InputSidePackets().Index(i).At(timestamp));
|
||||
}
|
||||
} else if (cc->Outputs().HasTag(kTagAtTimestamp)) {
|
||||
int64 timestamp =
|
||||
cc->InputSidePackets().Tag(kTagSideInputTimestamp).Get<int64>();
|
||||
for (int i = 0; i < cc->Outputs().NumEntries(output_tag_); ++i) {
|
||||
cc->Outputs()
|
||||
.Get(output_tag_, i)
|
||||
.AddPacket(cc->InputSidePackets().Index(i).At(Timestamp(timestamp)));
|
||||
}
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
@@ -51,6 +51,27 @@ TEST(SidePacketToStreamCalculator, WrongConfig_MissingTick) {
|
||||
"Either both of TICK and AT_TICK should be used or none of them.");
|
||||
}
|
||||
|
||||
TEST(SidePacketToStreamCalculator, WrongConfig_MissingTimestampSideInput) {
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
R"(
|
||||
input_stream: "timestamp"
|
||||
input_side_packet: "side_packet"
|
||||
output_stream: "packet"
|
||||
node {
|
||||
calculator: "SidePacketToStreamCalculator"
|
||||
input_side_packet: "side_packet"
|
||||
output_stream: "AT_TIMESTAMP:packet"
|
||||
}
|
||||
)");
|
||||
CalculatorGraph graph;
|
||||
auto status = graph.Initialize(graph_config);
|
||||
EXPECT_FALSE(status.ok());
|
||||
EXPECT_PRED2(
|
||||
absl::StrContains, status.message(),
|
||||
"Either both TIMESTAMP and AT_TIMESTAMP should be used or none of them.");
|
||||
}
|
||||
|
||||
TEST(SidePacketToStreamCalculator, WrongConfig_NonExistentTag) {
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
@@ -68,8 +89,9 @@ TEST(SidePacketToStreamCalculator, WrongConfig_NonExistentTag) {
|
||||
auto status = graph.Initialize(graph_config);
|
||||
EXPECT_FALSE(status.ok());
|
||||
EXPECT_PRED2(absl::StrContains, status.message(),
|
||||
"Only one of AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO and AT_TICK "
|
||||
"tags is allowed and required to specify output stream(s).");
|
||||
"Only one of AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO, AT_TICK and "
|
||||
"AT_TIMESTAMP tags is allowed and required to specify output "
|
||||
"stream(s).");
|
||||
}
|
||||
|
||||
TEST(SidePacketToStreamCalculator, WrongConfig_MixedTags) {
|
||||
@@ -91,8 +113,9 @@ TEST(SidePacketToStreamCalculator, WrongConfig_MixedTags) {
|
||||
auto status = graph.Initialize(graph_config);
|
||||
EXPECT_FALSE(status.ok());
|
||||
EXPECT_PRED2(absl::StrContains, status.message(),
|
||||
"Only one of AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO and AT_TICK "
|
||||
"tags is allowed and required to specify output stream(s).");
|
||||
"Only one of AT_PRESTREAM, AT_POSTSTREAM, AT_ZERO, AT_TICK and "
|
||||
"AT_TIMESTAMP tags is allowed and required to specify output "
|
||||
"stream(s).");
|
||||
}
|
||||
|
||||
TEST(SidePacketToStreamCalculator, WrongConfig_NotEnoughSidePackets) {
|
||||
@@ -271,5 +294,79 @@ TEST(SidePacketToStreamCalculator, AtTick_MultipleSidePackets) {
|
||||
tick_and_verify(/*at_timestamp=*/1025);
|
||||
}
|
||||
|
||||
TEST(SidePacketToStreamCalculator, AtTimestamp) {
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
R"(
|
||||
input_side_packet: "timestamp"
|
||||
input_side_packet: "side_packet"
|
||||
output_stream: "packet"
|
||||
node {
|
||||
calculator: "SidePacketToStreamCalculator"
|
||||
input_side_packet: "TIMESTAMP:timestamp"
|
||||
input_side_packet: "side_packet"
|
||||
output_stream: "AT_TIMESTAMP:packet"
|
||||
}
|
||||
)");
|
||||
std::vector<Packet> output_packets;
|
||||
tool::AddVectorSink("packet", &graph_config, &output_packets);
|
||||
CalculatorGraph graph;
|
||||
|
||||
MP_ASSERT_OK(graph.Initialize(graph_config));
|
||||
const int expected_value = 20;
|
||||
const int64 expected_timestamp = 5;
|
||||
MP_ASSERT_OK(
|
||||
graph.StartRun({{"side_packet", MakePacket<int>(expected_value)},
|
||||
{"timestamp", MakePacket<int64>(expected_timestamp)}}));
|
||||
|
||||
MP_ASSERT_OK(graph.WaitUntilDone());
|
||||
|
||||
ASSERT_FALSE(output_packets.empty());
|
||||
EXPECT_EQ(Timestamp(expected_timestamp), output_packets.back().Timestamp());
|
||||
EXPECT_EQ(expected_value, output_packets.back().Get<int>());
|
||||
}
|
||||
|
||||
TEST(SidePacketToStreamCalculator, AtTimestamp_MultipleOutputs) {
|
||||
CalculatorGraphConfig graph_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
R"(
|
||||
input_side_packet: "timestamp"
|
||||
input_side_packet: "side_packet0"
|
||||
input_side_packet: "side_packet1"
|
||||
output_stream: "packet"
|
||||
node {
|
||||
calculator: "SidePacketToStreamCalculator"
|
||||
input_side_packet: "TIMESTAMP:timestamp"
|
||||
input_side_packet: "side_packet0"
|
||||
input_side_packet: "side_packet1"
|
||||
output_stream: "AT_TIMESTAMP:0:packet0"
|
||||
output_stream: "AT_TIMESTAMP:1:packet1"
|
||||
}
|
||||
)");
|
||||
std::vector<Packet> output_packets0;
|
||||
tool::AddVectorSink("packet0", &graph_config, &output_packets0);
|
||||
std::vector<Packet> output_packets1;
|
||||
tool::AddVectorSink("packet1", &graph_config, &output_packets1);
|
||||
CalculatorGraph graph;
|
||||
|
||||
MP_ASSERT_OK(graph.Initialize(graph_config));
|
||||
const int expected_value0 = 20;
|
||||
const int expected_value1 = 15;
|
||||
const int64 expected_timestamp = 5;
|
||||
MP_ASSERT_OK(
|
||||
graph.StartRun({{"side_packet0", MakePacket<int>(expected_value0)},
|
||||
{"side_packet1", MakePacket<int>(expected_value1)},
|
||||
{"timestamp", MakePacket<int64>(expected_timestamp)}}));
|
||||
|
||||
MP_ASSERT_OK(graph.WaitUntilDone());
|
||||
|
||||
ASSERT_FALSE(output_packets0.empty());
|
||||
EXPECT_EQ(Timestamp(expected_timestamp), output_packets0.back().Timestamp());
|
||||
EXPECT_EQ(expected_value0, output_packets0.back().Get<int>());
|
||||
ASSERT_FALSE(output_packets1.empty());
|
||||
EXPECT_EQ(Timestamp(expected_timestamp), output_packets1.back().Timestamp());
|
||||
EXPECT_EQ(expected_value1, output_packets1.back().Get<int>());
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
|
||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//visibility:private"])
|
||||
|
||||
@@ -630,3 +630,34 @@ cc_library(
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "image_file_properties_calculator",
|
||||
srcs = ["image_file_properties_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/formats:image_file_properties_cc_proto",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
"@easyexif",
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "image_file_properties_calculator_test",
|
||||
srcs = ["image_file_properties_calculator_test.cc"],
|
||||
data = ["//mediapipe/calculators/image/testdata:test_images"],
|
||||
deps = [
|
||||
":image_file_properties_calculator",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework/deps:file_path",
|
||||
"//mediapipe/framework/formats:image_file_properties_cc_proto",
|
||||
"//mediapipe/framework/port:file_helpers",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -127,6 +127,10 @@ REGISTER_CALCULATOR(ImageCroppingCalculator);
|
||||
}
|
||||
|
||||
options_ = cc->Options<mediapipe::ImageCroppingCalculatorOptions>();
|
||||
output_max_width_ =
|
||||
options_.has_output_max_width() ? options_.output_max_width() : FLT_MAX;
|
||||
output_max_height_ =
|
||||
options_.has_output_max_height() ? options_.output_max_height() : FLT_MAX;
|
||||
|
||||
if (use_gpu_) {
|
||||
#if !defined(MEDIAPIPE_DISABLE_GPU)
|
||||
@@ -234,20 +238,27 @@ REGISTER_CALCULATOR(ImageCroppingCalculator);
|
||||
cv::Mat src_points;
|
||||
cv::boxPoints(min_rect, src_points);
|
||||
|
||||
float output_width = min_rect.size.width;
|
||||
float output_height = min_rect.size.height;
|
||||
float scale = std::min({1.0f, output_max_width_ / output_width,
|
||||
output_max_height_ / output_height});
|
||||
output_width *= scale;
|
||||
output_height *= scale;
|
||||
|
||||
float dst_corners[8] = {0,
|
||||
min_rect.size.height - 1,
|
||||
output_height - 1,
|
||||
0,
|
||||
0,
|
||||
min_rect.size.width - 1,
|
||||
output_width - 1,
|
||||
0,
|
||||
min_rect.size.width - 1,
|
||||
min_rect.size.height - 1};
|
||||
output_width - 1,
|
||||
output_height - 1};
|
||||
cv::Mat dst_points = cv::Mat(4, 2, CV_32F, dst_corners);
|
||||
cv::Mat projection_matrix =
|
||||
cv::getPerspectiveTransform(src_points, dst_points);
|
||||
cv::Mat cropped_image;
|
||||
cv::warpPerspective(input_mat, cropped_image, projection_matrix,
|
||||
cv::Size(min_rect.size.width, min_rect.size.height),
|
||||
cv::Size(output_width, output_height),
|
||||
/* flags = */ 0,
|
||||
/* borderMode = */ border_mode);
|
||||
|
||||
@@ -439,6 +450,12 @@ void ImageCroppingCalculator::GetOutputDimensions(CalculatorContext* cc,
|
||||
|
||||
int width = static_cast<int>(std::round((col_max - col_min) * src_width));
|
||||
int height = static_cast<int>(std::round((row_max - row_min) * src_height));
|
||||
|
||||
float scale =
|
||||
std::min({1.0f, output_max_width_ / width, output_max_height_ / height});
|
||||
width *= scale;
|
||||
height *= scale;
|
||||
|
||||
// Minimum output dimension 1x1 prevents creation of textures with 0x0.
|
||||
*dst_width = std::max(1, width);
|
||||
*dst_height = std::max(1, height);
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
#ifndef MEDIAPIPE_CALCULATORS_IMAGE_IMAGE_CROPPING_CALCULATOR_H_
|
||||
#define MEDIAPIPE_CALCULATORS_IMAGE_IMAGE_CROPPING_CALCULATOR_H_
|
||||
|
||||
#include <float.h>
|
||||
|
||||
#include "mediapipe/calculators/image/image_cropping_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
|
||||
@@ -80,6 +82,8 @@ class ImageCroppingCalculator : public CalculatorBase {
|
||||
bool use_gpu_ = false;
|
||||
// Output texture corners (4) after transoformation in normalized coordinates.
|
||||
float transformed_points_[8];
|
||||
float output_max_width_ = FLT_MAX;
|
||||
float output_max_height_ = FLT_MAX;
|
||||
#if !defined(MEDIAPIPE_DISABLE_GPU)
|
||||
bool gpu_initialized_ = false;
|
||||
mediapipe::GlCalculatorHelper gpu_helper_;
|
||||
|
||||
@@ -51,4 +51,10 @@ message ImageCroppingCalculatorOptions {
|
||||
|
||||
// Specifies behaviour for crops that go beyond image borders.
|
||||
optional BorderMode border_mode = 8 [default = BORDER_ZERO];
|
||||
|
||||
// Specifies limits for the size of the output image. It will be scaled down,
|
||||
// preserving ratio, to fit within. These do not change which area of the
|
||||
// input is selected for cropping.
|
||||
optional int32 output_max_width = 9;
|
||||
optional int32 output_max_height = 10;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
// Copyright 2019 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "exif.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/formats/image_file_properties.pb.h"
|
||||
#include "mediapipe/framework/port/canonical_errors.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
namespace {
|
||||
|
||||
// 35 MM sensor has dimensions 36 mm x 24 mm, so diagonal length is
|
||||
// sqrt(36^2 + 24^2).
|
||||
static const double SENSOR_DIAGONAL_35MM = std::sqrt(1872.0);
|
||||
|
||||
::mediapipe::StatusOr<double> ComputeFocalLengthInPixels(
|
||||
int image_width, int image_height, double focal_length_35mm,
|
||||
double focal_length_mm) {
|
||||
// TODO: Allow returning image file properties even when focal length
|
||||
// computation is not possible.
|
||||
if (image_width == 0 || image_height == 0) {
|
||||
return ::mediapipe::InternalError(
|
||||
"Image dimensions should be non-zero to compute focal length in "
|
||||
"pixels.");
|
||||
}
|
||||
if (focal_length_mm == 0) {
|
||||
return ::mediapipe::InternalError(
|
||||
"Focal length in mm should be non-zero to compute focal length in "
|
||||
"pixels.");
|
||||
}
|
||||
if (focal_length_35mm == 0) {
|
||||
return ::mediapipe::InternalError(
|
||||
"Focal length in 35 mm should be non-zero to compute focal length in "
|
||||
"pixels.");
|
||||
}
|
||||
// Derived from
|
||||
// https://en.wikipedia.org/wiki/35_mm_equivalent_focal_length#Calculation.
|
||||
/// Using focal_length_35mm = focal_length_mm * SENSOR_DIAGONAL_35MM /
|
||||
/// sensor_diagonal_mm, we can calculate the diagonal length of the sensor in
|
||||
/// millimeters i.e. sensor_diagonal_mm.
|
||||
double sensor_diagonal_mm =
|
||||
SENSOR_DIAGONAL_35MM / focal_length_35mm * focal_length_mm;
|
||||
// Note that for the following computations, the longer dimension is treated
|
||||
// as image width and the shorter dimension is treated as image height.
|
||||
int width = image_width;
|
||||
int height = image_height;
|
||||
if (image_height > image_width) {
|
||||
width = image_height;
|
||||
height = image_width;
|
||||
}
|
||||
double inv_aspect_ratio = (double)height / width;
|
||||
// Compute sensor width.
|
||||
/// Using Pythagoras theorem, sensor_width^2 + sensor_height^2 =
|
||||
/// sensor_diagonal_mm^2. We can substitute sensor_width / sensor_height with
|
||||
/// the aspect ratio calculated in pixels to compute the sensor width.
|
||||
double sensor_width = std::sqrt((sensor_diagonal_mm * sensor_diagonal_mm) /
|
||||
(1.0 + inv_aspect_ratio * inv_aspect_ratio));
|
||||
|
||||
// Compute focal length in pixels.
|
||||
double focal_length_pixels = width * focal_length_mm / sensor_width;
|
||||
return focal_length_pixels;
|
||||
}
|
||||
|
||||
::mediapipe::StatusOr<ImageFileProperties> GetImageFileProperites(
|
||||
const std::string& image_bytes) {
|
||||
easyexif::EXIFInfo result;
|
||||
int code = result.parseFrom(image_bytes);
|
||||
if (code) {
|
||||
return ::mediapipe::InternalError("Error parsing EXIF, code: " +
|
||||
std::to_string(code));
|
||||
}
|
||||
|
||||
ImageFileProperties properties;
|
||||
properties.set_image_width(result.ImageWidth);
|
||||
properties.set_image_height(result.ImageHeight);
|
||||
properties.set_focal_length_mm(result.FocalLength);
|
||||
properties.set_focal_length_35mm(result.FocalLengthIn35mm);
|
||||
|
||||
ASSIGN_OR_RETURN(auto focal_length_pixels,
|
||||
ComputeFocalLengthInPixels(properties.image_width(),
|
||||
properties.image_height(),
|
||||
properties.focal_length_35mm(),
|
||||
properties.focal_length_mm()));
|
||||
properties.set_focal_length_pixels(focal_length_pixels);
|
||||
|
||||
return properties;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// Calculator to extract EXIF information from an image file. The input is
|
||||
// a std::string containing raw byte data from a file, and the output is an
|
||||
// ImageFileProperties proto object with the relevant fields filled in.
|
||||
// The calculator accepts the input as a stream or a side packet, and can output
|
||||
// the result as a stream or a side packet. The calculator checks that if an
|
||||
// output stream is present, it outputs to that stream, and if not, it checks if
|
||||
// it can output to a side packet.
|
||||
//
|
||||
// Example config with input and output streams:
|
||||
// node {
|
||||
// calculator: "ImageFilePropertiesCalculator"
|
||||
// input_stream: "image_bytes"
|
||||
// output_stream: "image_properties"
|
||||
// }
|
||||
// Example config with input and output side packets:
|
||||
// node {
|
||||
// calculator: "ImageFilePropertiesCalculator"
|
||||
// input_side_packet: "image_bytes"
|
||||
// output_side_packet: "image_properties"
|
||||
// }
|
||||
class ImageFilePropertiesCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
if (cc->Inputs().NumEntries() != 0) {
|
||||
RET_CHECK(cc->Inputs().NumEntries() == 1);
|
||||
cc->Inputs().Index(0).Set<std::string>();
|
||||
} else {
|
||||
RET_CHECK(cc->InputSidePackets().NumEntries() == 1);
|
||||
cc->InputSidePackets().Index(0).Set<std::string>();
|
||||
}
|
||||
if (cc->Outputs().NumEntries() != 0) {
|
||||
RET_CHECK(cc->Outputs().NumEntries() == 1);
|
||||
cc->Outputs().Index(0).Set<::mediapipe::ImageFileProperties>();
|
||||
} else {
|
||||
RET_CHECK(cc->OutputSidePackets().NumEntries() == 1);
|
||||
cc->OutputSidePackets().Index(0).Set<::mediapipe::ImageFileProperties>();
|
||||
}
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) override {
|
||||
cc->SetOffset(TimestampDiff(0));
|
||||
|
||||
if (cc->InputSidePackets().NumEntries() == 1) {
|
||||
const std::string& image_bytes =
|
||||
cc->InputSidePackets().Index(0).Get<std::string>();
|
||||
ASSIGN_OR_RETURN(properties_, GetImageFileProperites(image_bytes));
|
||||
read_properties_ = true;
|
||||
}
|
||||
|
||||
if (read_properties_ && cc->OutputSidePackets().NumEntries() == 1) {
|
||||
cc->OutputSidePackets().Index(0).Set(
|
||||
MakePacket<ImageFileProperties>(properties_));
|
||||
}
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) override {
|
||||
if (cc->Inputs().NumEntries() == 1) {
|
||||
if (cc->Inputs().Index(0).IsEmpty()) {
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
const std::string& image_bytes = cc->Inputs().Index(0).Get<std::string>();
|
||||
ASSIGN_OR_RETURN(properties_, GetImageFileProperites(image_bytes));
|
||||
read_properties_ = true;
|
||||
}
|
||||
if (read_properties_) {
|
||||
if (cc->Outputs().NumEntries() == 1) {
|
||||
cc->Outputs().Index(0).AddPacket(
|
||||
MakePacket<ImageFileProperties>(properties_)
|
||||
.At(cc->InputTimestamp()));
|
||||
} else {
|
||||
cc->OutputSidePackets().Index(0).Set(
|
||||
MakePacket<ImageFileProperties>(properties_)
|
||||
.At(::mediapipe::Timestamp::Unset()));
|
||||
}
|
||||
}
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
private:
|
||||
ImageFileProperties properties_;
|
||||
bool read_properties_ = false;
|
||||
};
|
||||
REGISTER_CALCULATOR(ImageFilePropertiesCalculator);
|
||||
|
||||
} // namespace mediapipe
|
||||
@@ -0,0 +1,134 @@
|
||||
// Copyright 2018 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <math.h>
|
||||
|
||||
#include <cmath>
|
||||
#include <limits>
|
||||
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/deps/file_path.h"
|
||||
#include "mediapipe/framework/formats/image_file_properties.pb.h"
|
||||
#include "mediapipe/framework/port/file_helpers.h"
|
||||
#include "mediapipe/framework/port/gmock.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr char kImageFilePath[] =
|
||||
"/mediapipe/calculators/image/testdata/"
|
||||
"front_camera_pixel2.jpg";
|
||||
constexpr int kExpectedWidth = 2448;
|
||||
constexpr int kExpectedHeight = 3264;
|
||||
constexpr double kExpectedFocalLengthMm = 3.38;
|
||||
constexpr double kExpectedFocalLengthIn35Mm = 25;
|
||||
constexpr double kExpectedFocalLengthPixels = 2357.48;
|
||||
|
||||
double RoundToNDecimals(double value, int n) {
|
||||
return std::round(value * pow(10.0, n)) / pow(10.0, n);
|
||||
}
|
||||
|
||||
TEST(ImageFilePropertiesCalculatorTest, ReadsFocalLengthFromJpegInStreams) {
|
||||
std::string image_filepath = file::JoinPath("./", kImageFilePath);
|
||||
std::string image_contents;
|
||||
MP_ASSERT_OK(file::GetContents(image_filepath, &image_contents));
|
||||
|
||||
CalculatorGraphConfig::Node node_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(R"(
|
||||
calculator: "ImageFilePropertiesCalculator"
|
||||
input_stream: "image_bytes"
|
||||
output_stream: "properties"
|
||||
)");
|
||||
|
||||
CalculatorRunner runner(node_config);
|
||||
runner.MutableInputs()->Index(0).packets.push_back(
|
||||
MakePacket<std::string>(image_contents).At(Timestamp(0)));
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
const auto& outputs = runner.Outputs();
|
||||
ASSERT_EQ(1, outputs.NumEntries());
|
||||
const std::vector<Packet>& packets = outputs.Index(0).packets;
|
||||
ASSERT_EQ(1, packets.size());
|
||||
const auto& result = packets[0].Get<::mediapipe::ImageFileProperties>();
|
||||
EXPECT_EQ(kExpectedWidth, result.image_width());
|
||||
EXPECT_EQ(kExpectedHeight, result.image_height());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthMm, result.focal_length_mm());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthIn35Mm, result.focal_length_35mm());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthPixels,
|
||||
RoundToNDecimals(result.focal_length_pixels(), /*n=*/2));
|
||||
}
|
||||
|
||||
TEST(ImageFilePropertiesCalculatorTest, ReadsFocalLengthFromJpegInSidePackets) {
|
||||
std::string image_filepath = file::JoinPath("./", kImageFilePath);
|
||||
std::string image_contents;
|
||||
MP_ASSERT_OK(file::GetContents(image_filepath, &image_contents));
|
||||
|
||||
CalculatorGraphConfig::Node node_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(R"(
|
||||
calculator: "ImageFilePropertiesCalculator"
|
||||
input_side_packet: "image_bytes"
|
||||
output_side_packet: "properties"
|
||||
)");
|
||||
|
||||
CalculatorRunner runner(node_config);
|
||||
runner.MutableSidePackets()->Index(0) =
|
||||
MakePacket<std::string>(image_contents).At(Timestamp(0));
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
const auto& outputs = runner.OutputSidePackets();
|
||||
EXPECT_EQ(1, outputs.NumEntries());
|
||||
const auto& packet = outputs.Index(0);
|
||||
const auto& result = packet.Get<::mediapipe::ImageFileProperties>();
|
||||
EXPECT_EQ(kExpectedWidth, result.image_width());
|
||||
EXPECT_EQ(kExpectedHeight, result.image_height());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthMm, result.focal_length_mm());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthIn35Mm, result.focal_length_35mm());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthPixels,
|
||||
RoundToNDecimals(result.focal_length_pixels(), /*n=*/2));
|
||||
}
|
||||
|
||||
TEST(ImageFilePropertiesCalculatorTest,
|
||||
ReadsFocalLengthFromJpegStreamToSidePacket) {
|
||||
std::string image_filepath = file::JoinPath("./", kImageFilePath);
|
||||
std::string image_contents;
|
||||
MP_ASSERT_OK(file::GetContents(image_filepath, &image_contents));
|
||||
|
||||
CalculatorGraphConfig::Node node_config =
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(R"(
|
||||
calculator: "ImageFilePropertiesCalculator"
|
||||
input_stream: "image_bytes"
|
||||
output_side_packet: "properties"
|
||||
)");
|
||||
|
||||
CalculatorRunner runner(node_config);
|
||||
runner.MutableInputs()->Index(0).packets.push_back(
|
||||
MakePacket<std::string>(image_contents).At(Timestamp(0)));
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
const auto& outputs = runner.OutputSidePackets();
|
||||
EXPECT_EQ(1, outputs.NumEntries());
|
||||
const auto& packet = outputs.Index(0);
|
||||
const auto& result = packet.Get<::mediapipe::ImageFileProperties>();
|
||||
EXPECT_EQ(kExpectedWidth, result.image_width());
|
||||
EXPECT_EQ(kExpectedHeight, result.image_height());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthMm, result.focal_length_mm());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthIn35Mm, result.focal_length_35mm());
|
||||
EXPECT_DOUBLE_EQ(kExpectedFocalLengthPixels,
|
||||
RoundToNDecimals(result.focal_length_pixels(), /*n=*/2));
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
@@ -449,19 +449,15 @@ REGISTER_CALCULATOR(ImageTransformationCalculator);
|
||||
switch (rotation_) {
|
||||
case mediapipe::RotationMode_Mode_UNKNOWN:
|
||||
case mediapipe::RotationMode_Mode_ROTATION_0:
|
||||
LOG(ERROR) << "Not rotating image.";
|
||||
rotated_mat = input_mat;
|
||||
break;
|
||||
case mediapipe::RotationMode_Mode_ROTATION_90:
|
||||
LOG(ERROR) << "Rotating image by 90 degrees ccw.";
|
||||
cv::rotate(input_mat, rotated_mat, cv::ROTATE_90_COUNTERCLOCKWISE);
|
||||
break;
|
||||
case mediapipe::RotationMode_Mode_ROTATION_180:
|
||||
LOG(ERROR) << "Rotating image by 180 degrees.";
|
||||
cv::rotate(input_mat, rotated_mat, cv::ROTATE_180);
|
||||
break;
|
||||
case mediapipe::RotationMode_Mode_ROTATION_270:
|
||||
LOG(ERROR) << "Rotating image by 90 degrees cw.";
|
||||
cv::rotate(input_mat, rotated_mat, cv::ROTATE_90_CLOCKWISE);
|
||||
break;
|
||||
}
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
filegroup(
|
||||
name = "test_images",
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
|
||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//visibility:private"])
|
||||
|
||||
@@ -57,22 +57,6 @@ proto_library(
|
||||
deps = ["//mediapipe/framework:calculator_proto"],
|
||||
)
|
||||
|
||||
proto_library(
|
||||
name = "tensorflow_session_from_saved_model_generator_proto",
|
||||
srcs = ["tensorflow_session_from_saved_model_generator.proto"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = ["//mediapipe/framework:packet_generator_proto"],
|
||||
)
|
||||
|
||||
proto_library(
|
||||
name = "tensorflow_session_from_saved_model_calculator_proto",
|
||||
srcs = ["tensorflow_session_from_saved_model_calculator.proto"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_proto",
|
||||
],
|
||||
)
|
||||
|
||||
proto_library(
|
||||
name = "tensor_squeeze_dimensions_calculator_proto",
|
||||
srcs = ["tensor_squeeze_dimensions_calculator.proto"],
|
||||
@@ -212,7 +196,10 @@ mediapipe_cc_proto_library(
|
||||
mediapipe_cc_proto_library(
|
||||
name = "tensorflow_session_from_saved_model_generator_cc_proto",
|
||||
srcs = ["tensorflow_session_from_saved_model_generator.proto"],
|
||||
cc_deps = ["//mediapipe/framework:packet_generator_cc_proto"],
|
||||
cc_deps = [
|
||||
"//mediapipe/framework:packet_generator_cc_proto",
|
||||
"@org_tensorflow//tensorflow/core:protos_all_cc",
|
||||
],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [":tensorflow_session_from_saved_model_generator_proto"],
|
||||
)
|
||||
@@ -220,7 +207,10 @@ mediapipe_cc_proto_library(
|
||||
mediapipe_cc_proto_library(
|
||||
name = "tensorflow_session_from_saved_model_calculator_cc_proto",
|
||||
srcs = ["tensorflow_session_from_saved_model_calculator.proto"],
|
||||
cc_deps = ["//mediapipe/framework:calculator_cc_proto"],
|
||||
cc_deps = [
|
||||
"//mediapipe/framework:calculator_cc_proto",
|
||||
"@org_tensorflow//tensorflow/core:protos_all_cc",
|
||||
],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [":tensorflow_session_from_saved_model_calculator_proto"],
|
||||
)
|
||||
@@ -437,6 +427,10 @@ cc_library(
|
||||
deps = [
|
||||
":tensorflow_session",
|
||||
":tensorflow_inference_calculator_cc_proto",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"@com_google_absl//absl/base:core_headers",
|
||||
"@com_google_absl//absl/memory",
|
||||
"//mediapipe/framework:calculator_context",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/tool:status_util",
|
||||
"@com_google_absl//absl/strings",
|
||||
@@ -444,6 +438,8 @@ cc_library(
|
||||
"//mediapipe/framework/deps:clock",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:map_util",
|
||||
"//mediapipe/framework:packet",
|
||||
] + select({
|
||||
"//conditions:default": [
|
||||
"@org_tensorflow//tensorflow/core:framework",
|
||||
@@ -488,6 +484,8 @@ cc_library(
|
||||
"//mediapipe/calculators/tensorflow:tensorflow_session_from_frozen_graph_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/tool:status_util",
|
||||
"//mediapipe/framework/deps:clock",
|
||||
"//mediapipe/framework/port:logging",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
] + select({
|
||||
@@ -518,6 +516,8 @@ cc_library(
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/tool:status_util",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/deps:clock",
|
||||
"//mediapipe/framework/port:logging",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
] + select({
|
||||
"//conditions:default": [
|
||||
@@ -929,6 +929,7 @@ cc_test(
|
||||
"@com_google_absl//absl/strings",
|
||||
"@org_tensorflow//tensorflow/core:all_kernels",
|
||||
"@org_tensorflow//tensorflow/core:direct_session",
|
||||
"@org_tensorflow//tensorflow/core:protos_all_cc",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -954,6 +955,7 @@ cc_test(
|
||||
"@com_google_absl//absl/strings",
|
||||
"@org_tensorflow//tensorflow/core:all_kernels",
|
||||
"@org_tensorflow//tensorflow/core:direct_session",
|
||||
"@org_tensorflow//tensorflow/core:protos_all_cc",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ REGISTER_CALCULATOR(LappedTensorBufferCalculator);
|
||||
cc->Inputs().Index(0).Set<tf::Tensor>(
|
||||
// tensorflow::Tensor stream.
|
||||
);
|
||||
RET_CHECK_EQ(cc->Inputs().NumEntries(), 1)
|
||||
RET_CHECK_EQ(cc->Outputs().NumEntries(), 1)
|
||||
<< "Only one output stream is supported.";
|
||||
|
||||
if (cc->InputSidePackets().HasTag(kBufferSize)) {
|
||||
|
||||
@@ -19,16 +19,22 @@
|
||||
#include <unordered_set>
|
||||
#include <vector>
|
||||
|
||||
#include "absl/base/thread_annotations.h"
|
||||
#include "absl/memory/memory.h"
|
||||
#include "absl/strings/str_split.h"
|
||||
#include "absl/synchronization/mutex.h"
|
||||
#include "mediapipe/calculators/tensorflow/tensorflow_inference_calculator.pb.h"
|
||||
#include "mediapipe/calculators/tensorflow/tensorflow_session.h"
|
||||
#include "mediapipe/framework/calculator_context.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/deps/clock.h"
|
||||
#include "mediapipe/framework/deps/monotonic_clock.h"
|
||||
#include "mediapipe/framework/packet.h"
|
||||
#include "mediapipe/framework/port/map_util.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/port/status_macros.h"
|
||||
#include "mediapipe/framework/timestamp.h"
|
||||
#include "mediapipe/framework/tool/status_util.h"
|
||||
#include "tensorflow/core/framework/tensor.h"
|
||||
#include "tensorflow/core/framework/tensor_shape.h"
|
||||
@@ -77,6 +83,17 @@ class SimpleSemaphore {
|
||||
absl::Mutex mutex_;
|
||||
absl::CondVar cond_;
|
||||
};
|
||||
|
||||
class InferenceState {
|
||||
public:
|
||||
InferenceState() : input_tensor_batches_(), batch_timestamps_() {}
|
||||
// A mapping between stream tags and the tensors we are collecting as a
|
||||
// batch.
|
||||
std::map<std::string, std::vector<tf::Tensor>> input_tensor_batches_;
|
||||
// The timestamps that go into a batch.
|
||||
std::vector<Timestamp> batch_timestamps_;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
// This calculator performs inference on a trained TensorFlow model.
|
||||
@@ -218,11 +235,16 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
const auto& options = cc->Options<TensorFlowInferenceCalculatorOptions>();
|
||||
RET_CHECK(!cc->Inputs().GetTags().empty());
|
||||
for (const std::string& tag : cc->Inputs().GetTags()) {
|
||||
// The tensorflow::Tensor with the tag equal to the graph node. May
|
||||
// have a TimeSeriesHeader if all present TimeSeriesHeaders match.
|
||||
cc->Inputs().Tag(tag).Set<tf::Tensor>();
|
||||
if (!options.batched_input()) {
|
||||
cc->Inputs().Tag(tag).Set<tf::Tensor>();
|
||||
} else {
|
||||
cc->Inputs().Tag(tag).Set<std::vector<mediapipe::Packet>>();
|
||||
}
|
||||
}
|
||||
RET_CHECK(!cc->Outputs().GetTags().empty());
|
||||
for (const std::string& tag : cc->Outputs().GetTags()) {
|
||||
@@ -242,6 +264,22 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
std::unique_ptr<InferenceState> CreateInferenceState(CalculatorContext* cc)
|
||||
ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
|
||||
std::unique_ptr<InferenceState> inference_state =
|
||||
absl::make_unique<InferenceState>();
|
||||
if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS") &&
|
||||
!cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS").IsEmpty()) {
|
||||
std::map<std::string, tf::Tensor>* init_tensor_map;
|
||||
init_tensor_map = GetFromUniquePtr<std::map<std::string, tf::Tensor>>(
|
||||
cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS"));
|
||||
for (const auto& p : *init_tensor_map) {
|
||||
inference_state->input_tensor_batches_[p.first].emplace_back(p.second);
|
||||
}
|
||||
}
|
||||
return inference_state;
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) override {
|
||||
options_ = cc->Options<TensorFlowInferenceCalculatorOptions>();
|
||||
|
||||
@@ -275,15 +313,6 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
recurrent_feed_tags_.insert(tags[0]);
|
||||
recurrent_fetch_tags_to_feed_tags_[tags[1]] = tags[0];
|
||||
}
|
||||
if (cc->InputSidePackets().HasTag("RECURRENT_INIT_TENSORS") &&
|
||||
!cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS").IsEmpty()) {
|
||||
std::map<std::string, tf::Tensor>* init_tensor_map;
|
||||
init_tensor_map = GetFromUniquePtr<std::map<std::string, tf::Tensor>>(
|
||||
cc->InputSidePackets().Tag("RECURRENT_INIT_TENSORS"));
|
||||
for (const auto& p : *init_tensor_map) {
|
||||
input_tensor_batches_[p.first].emplace_back(p.second);
|
||||
}
|
||||
}
|
||||
|
||||
// Check that all tags are present in this signature bound to tensors.
|
||||
for (const std::string& tag : cc->Inputs().GetTags()) {
|
||||
@@ -297,9 +326,15 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
<< options_.signature_name();
|
||||
}
|
||||
|
||||
if (options_.batch_size() == 1) {
|
||||
{
|
||||
absl::WriterMutexLock l(&mutex_);
|
||||
inference_state_ = std::unique_ptr<InferenceState>();
|
||||
}
|
||||
|
||||
if (options_.batch_size() == 1 || options_.batched_input()) {
|
||||
cc->SetOffset(0);
|
||||
}
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
@@ -316,6 +351,24 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status AggregateTensorPacket(
|
||||
const std::string& tag_name, const Packet& packet,
|
||||
std::map<Timestamp, std::map<std::string, tf::Tensor>>*
|
||||
input_tensors_by_tag_by_timestamp,
|
||||
InferenceState* inference_state) ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_) {
|
||||
tf::Tensor input_tensor(packet.Get<tf::Tensor>());
|
||||
RET_CHECK_OK(AddBatchDimension(&input_tensor));
|
||||
if (::mediapipe::ContainsKey(recurrent_feed_tags_, tag_name)) {
|
||||
// If we receive an input on a recurrent tag, override the state.
|
||||
// It's OK to override the global state because there is just one
|
||||
// input stream allowed for recurrent tensors.
|
||||
inference_state_->input_tensor_batches_[tag_name].clear();
|
||||
}
|
||||
(*input_tensors_by_tag_by_timestamp)[packet.Timestamp()].insert(
|
||||
std::make_pair(tag_name, input_tensor));
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
// Removes the batch dimension of the output tensor if specified in the
|
||||
// calculator options.
|
||||
::mediapipe::Status RemoveBatchDimension(tf::Tensor* output_tensor) {
|
||||
@@ -331,48 +384,85 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
::mediapipe::Status Process(CalculatorContext* cc) override {
|
||||
std::map<std::string, tf::Tensor> input_tensors_by_tag;
|
||||
for (const std::string& tag_as_node_name : cc->Inputs().GetTags()) {
|
||||
if (cc->Inputs().Tag(tag_as_node_name).IsEmpty()) {
|
||||
// Recurrent tensors can be empty.
|
||||
if (!::mediapipe::ContainsKey(recurrent_feed_tags_, tag_as_node_name)) {
|
||||
if (options_.skip_on_missing_features()) {
|
||||
return ::mediapipe::OkStatus();
|
||||
} else {
|
||||
return ::mediapipe::InvalidArgumentError(absl::StrCat(
|
||||
"Tag ", tag_as_node_name,
|
||||
" not present at timestamp: ", cc->InputTimestamp().Value()));
|
||||
std::unique_ptr<InferenceState> inference_state_to_process;
|
||||
{
|
||||
absl::WriterMutexLock l(&mutex_);
|
||||
if (inference_state_ == nullptr) {
|
||||
inference_state_ = CreateInferenceState(cc);
|
||||
}
|
||||
std::map<Timestamp, std::map<std::string, tf::Tensor>>
|
||||
input_tensors_by_tag_by_timestamp;
|
||||
for (const std::string& tag_as_node_name : cc->Inputs().GetTags()) {
|
||||
if (cc->Inputs().Tag(tag_as_node_name).IsEmpty()) {
|
||||
// Recurrent tensors can be empty.
|
||||
if (!::mediapipe::ContainsKey(recurrent_feed_tags_,
|
||||
tag_as_node_name)) {
|
||||
if (options_.skip_on_missing_features()) {
|
||||
return ::mediapipe::OkStatus();
|
||||
} else {
|
||||
return ::mediapipe::InvalidArgumentError(absl::StrCat(
|
||||
"Tag ", tag_as_node_name,
|
||||
" not present at timestamp: ", cc->InputTimestamp().Value()));
|
||||
}
|
||||
}
|
||||
} else if (options_.batched_input()) {
|
||||
const auto& tensor_packets =
|
||||
cc->Inputs().Tag(tag_as_node_name).Get<std::vector<Packet>>();
|
||||
if (tensor_packets.size() > options_.batch_size()) {
|
||||
return ::mediapipe::InvalidArgumentError(absl::StrCat(
|
||||
"Batch for tag ", tag_as_node_name,
|
||||
" has more packets than batch capacity. batch_size: ",
|
||||
options_.batch_size(), " packets: ", tensor_packets.size()));
|
||||
}
|
||||
for (const auto& packet : tensor_packets) {
|
||||
RET_CHECK_OK(AggregateTensorPacket(
|
||||
tag_as_node_name, packet, &input_tensors_by_tag_by_timestamp,
|
||||
inference_state_.get()));
|
||||
}
|
||||
} else {
|
||||
RET_CHECK_OK(AggregateTensorPacket(
|
||||
tag_as_node_name, cc->Inputs().Tag(tag_as_node_name).Value(),
|
||||
&input_tensors_by_tag_by_timestamp, inference_state_.get()));
|
||||
}
|
||||
} else {
|
||||
tf::Tensor input_tensor(
|
||||
cc->Inputs().Tag(tag_as_node_name).Get<tf::Tensor>());
|
||||
RET_CHECK_OK(AddBatchDimension(&input_tensor));
|
||||
if (::mediapipe::ContainsKey(recurrent_feed_tags_, tag_as_node_name)) {
|
||||
// If we receive an input on a recurrent tag, override the state.
|
||||
// It's OK to override the global state because there is just one
|
||||
// input stream allowed for recurrent tensors.
|
||||
input_tensor_batches_[tag_as_node_name].clear();
|
||||
}
|
||||
for (const auto& timestamp_and_input_tensors_by_tag :
|
||||
input_tensors_by_tag_by_timestamp) {
|
||||
inference_state_->batch_timestamps_.emplace_back(
|
||||
timestamp_and_input_tensors_by_tag.first);
|
||||
for (const auto& input_tensor_and_tag :
|
||||
timestamp_and_input_tensors_by_tag.second) {
|
||||
inference_state_->input_tensor_batches_[input_tensor_and_tag.first]
|
||||
.emplace_back(input_tensor_and_tag.second);
|
||||
}
|
||||
input_tensors_by_tag.insert(
|
||||
std::make_pair(tag_as_node_name, input_tensor));
|
||||
}
|
||||
if (inference_state_->batch_timestamps_.size() == options_.batch_size() ||
|
||||
options_.batched_input()) {
|
||||
inference_state_to_process = std::move(inference_state_);
|
||||
inference_state_ = std::unique_ptr<InferenceState>();
|
||||
}
|
||||
}
|
||||
batch_timestamps_.emplace_back(cc->InputTimestamp());
|
||||
for (const auto& input_tensor_and_tag : input_tensors_by_tag) {
|
||||
input_tensor_batches_[input_tensor_and_tag.first].emplace_back(
|
||||
input_tensor_and_tag.second);
|
||||
|
||||
if (inference_state_to_process) {
|
||||
MP_RETURN_IF_ERROR(
|
||||
OutputBatch(cc, std::move(inference_state_to_process)));
|
||||
}
|
||||
|
||||
if (batch_timestamps_.size() == options_.batch_size()) {
|
||||
MP_RETURN_IF_ERROR(OutputBatch(cc));
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
::mediapipe::Status Close(CalculatorContext* cc) override {
|
||||
if (!batch_timestamps_.empty()) {
|
||||
MP_RETURN_IF_ERROR(OutputBatch(cc));
|
||||
std::unique_ptr<InferenceState> inference_state_to_process = nullptr;
|
||||
{
|
||||
absl::WriterMutexLock l(&mutex_);
|
||||
if (cc->GraphStatus().ok() && inference_state_ != nullptr &&
|
||||
!inference_state_->batch_timestamps_.empty()) {
|
||||
inference_state_to_process = std::move(inference_state_);
|
||||
inference_state_ = std::unique_ptr<InferenceState>();
|
||||
}
|
||||
}
|
||||
if (inference_state_to_process) {
|
||||
MP_RETURN_IF_ERROR(
|
||||
OutputBatch(cc, std::move(inference_state_to_process)));
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
@@ -385,10 +475,12 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
// memory buffer. Therefore, copies are cheap and should not cause the memory
|
||||
// buffer to fall out of scope. In contrast, concat is only used where
|
||||
// necessary.
|
||||
::mediapipe::Status OutputBatch(CalculatorContext* cc) {
|
||||
::mediapipe::Status OutputBatch(
|
||||
CalculatorContext* cc, std::unique_ptr<InferenceState> inference_state) {
|
||||
const int64 start_time = absl::ToUnixMicros(clock_->TimeNow());
|
||||
std::vector<std::pair<mediapipe::ProtoString, tf::Tensor>> input_tensors;
|
||||
for (auto& keyed_tensors : input_tensor_batches_) {
|
||||
|
||||
for (auto& keyed_tensors : inference_state->input_tensor_batches_) {
|
||||
if (options_.batch_size() == 1) {
|
||||
// Short circuit to avoid the cost of deep copying tensors in concat.
|
||||
if (!keyed_tensors.second.empty()) {
|
||||
@@ -404,7 +496,8 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
} else {
|
||||
// Pad by replicating the first tens or, then ignore the values.
|
||||
keyed_tensors.second.resize(options_.batch_size());
|
||||
std::fill(keyed_tensors.second.begin() + batch_timestamps_.size(),
|
||||
std::fill(keyed_tensors.second.begin() +
|
||||
inference_state->batch_timestamps_.size(),
|
||||
keyed_tensors.second.end(), keyed_tensors.second[0]);
|
||||
tf::Tensor concated;
|
||||
const tf::Status concat_status =
|
||||
@@ -414,7 +507,7 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
concated);
|
||||
}
|
||||
}
|
||||
input_tensor_batches_.clear();
|
||||
inference_state->input_tensor_batches_.clear();
|
||||
std::vector<mediapipe::ProtoString> output_tensor_names;
|
||||
std::vector<std::string> output_name_in_signature;
|
||||
for (const std::string& tag : cc->Outputs().GetTags()) {
|
||||
@@ -466,9 +559,11 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
int pos = std::find(output_name_in_signature.begin(),
|
||||
output_name_in_signature.end(), tag_pair.first) -
|
||||
output_name_in_signature.begin();
|
||||
input_tensor_batches_[tag_pair.second].emplace_back(outputs[pos]);
|
||||
inference_state->input_tensor_batches_[tag_pair.second].emplace_back(
|
||||
outputs[pos]);
|
||||
}
|
||||
|
||||
absl::WriterMutexLock l(&mutex_);
|
||||
// Set that we want to split on each index of the 0th dimension.
|
||||
std::vector<tf::int64> split_vector(options_.batch_size(), 1);
|
||||
for (int i = 0; i < output_tensor_names.size(); ++i) {
|
||||
@@ -478,7 +573,8 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
|
||||
cc->Outputs()
|
||||
.Tag(output_name_in_signature[i])
|
||||
.Add(new tf::Tensor(output_tensor), batch_timestamps_[0]);
|
||||
.Add(new tf::Tensor(output_tensor),
|
||||
inference_state->batch_timestamps_[0]);
|
||||
}
|
||||
} else {
|
||||
std::vector<tf::Tensor> split_tensors;
|
||||
@@ -486,22 +582,30 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
tf::tensor::Split(outputs[i], split_vector, &split_tensors);
|
||||
CHECK(split_status.ok()) << split_status.ToString();
|
||||
// Loop over timestamps so that we don't copy the padding.
|
||||
for (int j = 0; j < batch_timestamps_.size(); ++j) {
|
||||
for (int j = 0; j < inference_state->batch_timestamps_.size(); ++j) {
|
||||
tf::Tensor output_tensor(split_tensors[j]);
|
||||
RET_CHECK_OK(RemoveBatchDimension(&output_tensor));
|
||||
cc->Outputs()
|
||||
.Tag(output_name_in_signature[i])
|
||||
.Add(new tf::Tensor(output_tensor), batch_timestamps_[j]);
|
||||
.Add(new tf::Tensor(output_tensor),
|
||||
inference_state->batch_timestamps_[j]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get end time and report.
|
||||
const int64 end_time = absl::ToUnixMicros(clock_->TimeNow());
|
||||
cc->GetCounter(kTotalUsecsCounterSuffix)
|
||||
->IncrementBy(end_time - start_time);
|
||||
cc->GetCounter(kTotalProcessedTimestampsCounterSuffix)
|
||||
->IncrementBy(batch_timestamps_.size());
|
||||
batch_timestamps_.clear();
|
||||
->IncrementBy(inference_state->batch_timestamps_.size());
|
||||
|
||||
// Make sure we hold on to the recursive state.
|
||||
if (!options_.recurrent_tag_pair().empty()) {
|
||||
inference_state_ = std::move(inference_state);
|
||||
inference_state_->batch_timestamps_.clear();
|
||||
}
|
||||
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
@@ -514,11 +618,8 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
// A mapping between stream tags and the tensor names they are bound to.
|
||||
std::map<std::string, std::string> tag_to_tensor_map_;
|
||||
|
||||
// A mapping between stream tags and the tensors we are collecting as a batch.
|
||||
std::map<std::string, std::vector<tf::Tensor>> input_tensor_batches_;
|
||||
|
||||
// The timestamps that go into a batch.
|
||||
std::vector<Timestamp> batch_timestamps_;
|
||||
absl::Mutex mutex_;
|
||||
std::unique_ptr<InferenceState> inference_state_ ABSL_GUARDED_BY(mutex_);
|
||||
|
||||
// The options for the calculator.
|
||||
TensorFlowInferenceCalculatorOptions options_;
|
||||
|
||||
@@ -76,4 +76,13 @@ message TensorFlowInferenceCalculatorOptions {
|
||||
// only works in the local process, not "globally" across multiple processes
|
||||
// or replicas (if any). Default to 0, i.e. no limit.
|
||||
optional int32 max_concurrent_session_runs = 6 [default = 0];
|
||||
|
||||
// If turned on, the Calculator expects a vector of batched packages as input.
|
||||
// This will make sure that you can turn on max_in_flight for batch_size
|
||||
// greater than 1. Otherwise it results in problems of none-monotonically
|
||||
// increasing timestamps.
|
||||
// Use BatchSequentialCalculator to create the batches. The batch_size
|
||||
// should agree for both calculators. All the data in a batch is processed
|
||||
// together. The BatchSequentialCalculator can't run with max_in_flight.
|
||||
optional bool batched_input = 7;
|
||||
}
|
||||
|
||||
@@ -89,17 +89,31 @@ class TensorflowInferenceCalculatorTest : public ::testing::Test {
|
||||
output_side_packets.Tag("SESSION");
|
||||
}
|
||||
|
||||
// Create tensor from Vector and add as a Packet to the provided tag as input.
|
||||
void AddVectorToInputsAsTensor(const std::vector<int32>& input,
|
||||
const std::string& tag, int64 time) {
|
||||
Packet CreateTensorPacket(const std::vector<int32>& input, int64 time) {
|
||||
tf::TensorShape tensor_shape;
|
||||
tensor_shape.AddDim(input.size());
|
||||
auto tensor = absl::make_unique<tf::Tensor>(tf::DT_INT32, tensor_shape);
|
||||
for (int i = 0; i < input.size(); ++i) {
|
||||
tensor->vec<int32>()(i) = input[i];
|
||||
}
|
||||
return Adopt(tensor.release()).At(Timestamp(time));
|
||||
}
|
||||
|
||||
// Create tensor from Vector and add as a Packet to the provided tag as input.
|
||||
void AddVectorToInputsAsTensor(const std::vector<int32>& input,
|
||||
const std::string& tag, int64 time) {
|
||||
runner_->MutableInputs()->Tag(tag).packets.push_back(
|
||||
Adopt(tensor.release()).At(Timestamp(time)));
|
||||
CreateTensorPacket(input, time));
|
||||
}
|
||||
|
||||
// Create tensor from Vector and add as a Packet to the provided tag as input.
|
||||
void AddVectorToInputsAsPacket(const std::vector<Packet>& packets,
|
||||
const std::string& tag) {
|
||||
CHECK(!packets.empty())
|
||||
<< "Please specify at least some data in the packet";
|
||||
auto packets_ptr = absl::make_unique<std::vector<Packet>>(packets);
|
||||
runner_->MutableInputs()->Tag(tag).packets.push_back(
|
||||
Adopt(packets_ptr.release()).At(packets.begin()->Timestamp()));
|
||||
}
|
||||
|
||||
std::unique_ptr<CalculatorRunner> runner_;
|
||||
@@ -183,6 +197,45 @@ TEST_F(TensorflowInferenceCalculatorTest, GetComputed) {
|
||||
EXPECT_THAT(run_status.ToString(), testing::HasSubstr("Tag B"));
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, GetComputed_MaxInFlight) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
config.add_input_stream("A:tensor_a");
|
||||
config.add_input_stream("B:tensor_b");
|
||||
config.add_output_stream("MULTIPLIED:tensor_o1");
|
||||
config.add_input_side_packet("SESSION:session");
|
||||
config.set_max_in_flight(2);
|
||||
CalculatorOptions options;
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batch_size(1);
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_add_batch_dim_to_tensors(false);
|
||||
*config.mutable_options() = options;
|
||||
|
||||
runner_ = absl::make_unique<CalculatorRunner>(config);
|
||||
AddSessionInputSidePacket();
|
||||
AddVectorToInputsAsTensor({2, 2, 2}, "A", 0);
|
||||
AddVectorToInputsAsTensor({3, 4, 5}, "B", 0);
|
||||
MP_ASSERT_OK(runner_->Run());
|
||||
|
||||
const std::vector<Packet>& output_packets_mult =
|
||||
runner_->Outputs().Tag("MULTIPLIED").packets;
|
||||
ASSERT_EQ(1, output_packets_mult.size());
|
||||
const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
|
||||
tf::TensorShape expected_shape({3});
|
||||
auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10}, expected_shape);
|
||||
tf::test::ExpectTensorEqual<int32>(expected_tensor, tensor_mult);
|
||||
|
||||
// Add only one of the two expected tensors at the next timestamp, expect
|
||||
// useful failure message.
|
||||
AddVectorToInputsAsTensor({1, 2, 3}, "A", 1);
|
||||
auto run_status = runner_->Run();
|
||||
ASSERT_FALSE(run_status.ok());
|
||||
EXPECT_THAT(run_status.ToString(),
|
||||
testing::HasSubstr("TensorFlowInferenceCalculator"));
|
||||
EXPECT_THAT(run_status.ToString(), testing::HasSubstr("Tag B"));
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, BadTag) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
@@ -235,6 +288,86 @@ TEST_F(TensorflowInferenceCalculatorTest, GetMultiBatchComputed) {
|
||||
->Get());
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, GetMultiBatchComputed_MaxInFlight) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
config.add_input_stream("A:tensor_a");
|
||||
config.add_input_stream("B:tensor_b");
|
||||
config.add_output_stream("MULTIPLIED:tensor_o1");
|
||||
config.add_input_side_packet("SESSION:session");
|
||||
config.set_max_in_flight(2);
|
||||
CalculatorOptions options;
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batch_size(1);
|
||||
*config.mutable_options() = options;
|
||||
|
||||
runner_ = absl::make_unique<CalculatorRunner>(config);
|
||||
AddSessionInputSidePacket();
|
||||
AddVectorToInputsAsTensor({2, 2, 2}, "A", 0);
|
||||
AddVectorToInputsAsTensor({3, 4, 5}, "B", 0);
|
||||
AddVectorToInputsAsTensor({3, 3, 3}, "A", 1);
|
||||
AddVectorToInputsAsTensor({3, 4, 5}, "B", 1);
|
||||
MP_ASSERT_OK(runner_->Run());
|
||||
|
||||
const std::vector<Packet>& output_packets_mult =
|
||||
runner_->Outputs().Tag("MULTIPLIED").packets;
|
||||
ASSERT_EQ(2, output_packets_mult.size());
|
||||
const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
|
||||
auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult, expected_tensor);
|
||||
const tf::Tensor& tensor_mult1 = output_packets_mult[1].Get<tf::Tensor>();
|
||||
auto expected_tensor1 = tf::test::AsTensor<int32>({9, 12, 15});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult1, expected_tensor1);
|
||||
|
||||
EXPECT_EQ(2, runner_
|
||||
->GetCounter(
|
||||
"TensorFlowInferenceCalculator-TotalProcessedTimestamps")
|
||||
->Get());
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest,
|
||||
GetMultiBatchComputed_MoreThanMaxInFlight) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
config.add_input_stream("A:tensor_a");
|
||||
config.add_input_stream("B:tensor_b");
|
||||
config.add_output_stream("MULTIPLIED:tensor_o1");
|
||||
config.add_input_side_packet("SESSION:session");
|
||||
config.set_max_in_flight(2);
|
||||
CalculatorOptions options;
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batch_size(1);
|
||||
*config.mutable_options() = options;
|
||||
|
||||
runner_ = absl::make_unique<CalculatorRunner>(config);
|
||||
AddSessionInputSidePacket();
|
||||
AddVectorToInputsAsTensor({2, 2, 2}, "A", 0);
|
||||
AddVectorToInputsAsTensor({3, 4, 5}, "B", 0);
|
||||
AddVectorToInputsAsTensor({3, 3, 3}, "A", 1);
|
||||
AddVectorToInputsAsTensor({3, 4, 5}, "B", 1);
|
||||
AddVectorToInputsAsTensor({4, 4, 4}, "A", 2);
|
||||
AddVectorToInputsAsTensor({3, 4, 5}, "B", 2);
|
||||
MP_ASSERT_OK(runner_->Run());
|
||||
|
||||
const std::vector<Packet>& output_packets_mult =
|
||||
runner_->Outputs().Tag("MULTIPLIED").packets;
|
||||
ASSERT_EQ(3, output_packets_mult.size());
|
||||
const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
|
||||
auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult, expected_tensor);
|
||||
const tf::Tensor& tensor_mult1 = output_packets_mult[1].Get<tf::Tensor>();
|
||||
auto expected_tensor1 = tf::test::AsTensor<int32>({9, 12, 15});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult1, expected_tensor1);
|
||||
const tf::Tensor& tensor_mult2 = output_packets_mult[2].Get<tf::Tensor>();
|
||||
auto expected_tensor2 = tf::test::AsTensor<int32>({12, 16, 20});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult2, expected_tensor2);
|
||||
|
||||
EXPECT_EQ(3, runner_
|
||||
->GetCounter(
|
||||
"TensorFlowInferenceCalculator-TotalProcessedTimestamps")
|
||||
->Get());
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, GetSingleBatchComputed) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
@@ -311,6 +444,66 @@ TEST_F(TensorflowInferenceCalculatorTest, GetCloseBatchComputed) {
|
||||
->Get());
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, GetBatchComputed_MaxInFlight) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
config.add_input_stream("A:tensor_a");
|
||||
config.add_input_stream("B:tensor_b");
|
||||
config.add_output_stream("MULTIPLIED:tensor_o1");
|
||||
config.add_input_side_packet("SESSION:session");
|
||||
config.set_max_in_flight(2);
|
||||
CalculatorOptions options;
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batch_size(2);
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_add_batch_dim_to_tensors(true);
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batched_input(true);
|
||||
*config.mutable_options() = options;
|
||||
|
||||
runner_ = absl::make_unique<CalculatorRunner>(config);
|
||||
AddSessionInputSidePacket();
|
||||
AddVectorToInputsAsPacket(
|
||||
{CreateTensorPacket({2, 2, 2}, 0), CreateTensorPacket({3, 3, 3}, 1)},
|
||||
"A");
|
||||
AddVectorToInputsAsPacket(
|
||||
{CreateTensorPacket({3, 4, 5}, 0), CreateTensorPacket({3, 4, 5}, 1)},
|
||||
"B");
|
||||
AddVectorToInputsAsPacket(
|
||||
{CreateTensorPacket({4, 4, 4}, 2), CreateTensorPacket({5, 5, 5}, 3)},
|
||||
"A");
|
||||
AddVectorToInputsAsPacket(
|
||||
{CreateTensorPacket({3, 4, 5}, 2), CreateTensorPacket({3, 4, 5}, 3)},
|
||||
"B");
|
||||
AddVectorToInputsAsPacket({CreateTensorPacket({6, 6, 6}, 4)}, "A");
|
||||
AddVectorToInputsAsPacket({CreateTensorPacket({3, 4, 5}, 4)}, "B");
|
||||
MP_ASSERT_OK(runner_->Run());
|
||||
|
||||
const std::vector<Packet>& output_packets_mult =
|
||||
runner_->Outputs().Tag("MULTIPLIED").packets;
|
||||
ASSERT_EQ(5, output_packets_mult.size());
|
||||
const tf::Tensor& tensor_mult = output_packets_mult[0].Get<tf::Tensor>();
|
||||
auto expected_tensor = tf::test::AsTensor<int32>({6, 8, 10});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult, expected_tensor);
|
||||
const tf::Tensor& tensor_mult1 = output_packets_mult[1].Get<tf::Tensor>();
|
||||
auto expected_tensor1 = tf::test::AsTensor<int32>({9, 12, 15});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult1, expected_tensor1);
|
||||
const tf::Tensor& tensor_mult2 = output_packets_mult[2].Get<tf::Tensor>();
|
||||
auto expected_tensor2 = tf::test::AsTensor<int32>({12, 16, 20});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult2, expected_tensor2);
|
||||
const tf::Tensor& tensor_mult3 = output_packets_mult[3].Get<tf::Tensor>();
|
||||
auto expected_tensor3 = tf::test::AsTensor<int32>({15, 20, 25});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult3, expected_tensor3);
|
||||
const tf::Tensor& tensor_mult4 = output_packets_mult[4].Get<tf::Tensor>();
|
||||
auto expected_tensor4 = tf::test::AsTensor<int32>({18, 24, 30});
|
||||
tf::test::ExpectTensorEqual<int32>(tensor_mult4, expected_tensor4);
|
||||
|
||||
EXPECT_EQ(5, runner_
|
||||
->GetCounter(
|
||||
"TensorFlowInferenceCalculator-TotalProcessedTimestamps")
|
||||
->Get());
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, TestRecurrentStates) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
@@ -509,4 +702,40 @@ TEST_F(TensorflowInferenceCalculatorTest,
|
||||
->Get());
|
||||
}
|
||||
|
||||
TEST_F(TensorflowInferenceCalculatorTest, BatchedInputTooBigBatch) {
|
||||
CalculatorGraphConfig::Node config;
|
||||
config.set_calculator("TensorFlowInferenceCalculator");
|
||||
config.add_input_stream("A:tensor_a");
|
||||
config.add_input_stream("B:tensor_b");
|
||||
config.add_output_stream("MULTIPLIED:tensor_o1");
|
||||
config.add_input_side_packet("SESSION:session");
|
||||
config.set_max_in_flight(2);
|
||||
CalculatorOptions options;
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batch_size(2);
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_add_batch_dim_to_tensors(true);
|
||||
options.MutableExtension(TensorFlowInferenceCalculatorOptions::ext)
|
||||
->set_batched_input(true);
|
||||
*config.mutable_options() = options;
|
||||
|
||||
runner_ = absl::make_unique<CalculatorRunner>(config);
|
||||
AddSessionInputSidePacket();
|
||||
AddVectorToInputsAsPacket(
|
||||
{CreateTensorPacket({2, 2, 2}, 0), CreateTensorPacket({3, 3, 3}, 1),
|
||||
CreateTensorPacket({4, 4, 4}, 2)},
|
||||
"A");
|
||||
AddVectorToInputsAsPacket(
|
||||
{CreateTensorPacket({3, 4, 5}, 0), CreateTensorPacket({3, 4, 5}, 1),
|
||||
CreateTensorPacket({3, 4, 5}, 2)},
|
||||
"B");
|
||||
|
||||
auto status = runner_->Run();
|
||||
ASSERT_FALSE(status.ok());
|
||||
EXPECT_THAT(
|
||||
status.message(),
|
||||
::testing::HasSubstr(
|
||||
"has more packets than batch capacity. batch_size: 2 packets: 3"));
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -26,9 +26,14 @@
|
||||
#include "mediapipe/calculators/tensorflow/tensorflow_session.h"
|
||||
#include "mediapipe/calculators/tensorflow/tensorflow_session_from_frozen_graph_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/deps/clock.h"
|
||||
#include "mediapipe/framework/deps/monotonic_clock.h"
|
||||
#include "mediapipe/framework/port/logging.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/tool/status_util.h"
|
||||
#include "tensorflow/core/framework/graph.pb.h"
|
||||
#include "tensorflow/core/framework/node_def.pb.h"
|
||||
#include "tensorflow/core/public/session_options.h"
|
||||
|
||||
#if defined(MEDIAPIPE_MOBILE)
|
||||
@@ -41,6 +46,17 @@ namespace mediapipe {
|
||||
|
||||
namespace tf = ::tensorflow;
|
||||
|
||||
namespace {
|
||||
// Updates the graph nodes to use the device as specified by device_id.
|
||||
void SetPreferredDevice(tf::GraphDef* graph_def, absl::string_view device_id) {
|
||||
for (auto& node : *graph_def->mutable_node()) {
|
||||
if (node.device().empty()) {
|
||||
node.set_device(device_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
class TensorFlowSessionFromFrozenGraphCalculator : public CalculatorBase {
|
||||
public:
|
||||
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
|
||||
@@ -77,6 +93,9 @@ class TensorFlowSessionFromFrozenGraphCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
::mediapipe::Status Open(CalculatorContext* cc) override {
|
||||
auto clock = std::unique_ptr<mediapipe::Clock>(
|
||||
mediapipe::MonotonicClock::CreateSynchronizedMonotonicClock());
|
||||
const uint64 start_time = absl::ToUnixMicros(clock->TimeNow());
|
||||
const auto& options =
|
||||
cc->Options<TensorFlowSessionFromFrozenGraphCalculatorOptions>();
|
||||
// Output bundle packet.
|
||||
@@ -108,6 +127,12 @@ class TensorFlowSessionFromFrozenGraphCalculator : public CalculatorBase {
|
||||
tensorflow::GraphDef graph_def;
|
||||
|
||||
RET_CHECK(graph_def.ParseFromString(graph_def_serialized));
|
||||
|
||||
// Update the graph nodes to use the preferred device, if set.
|
||||
if (!options.preferred_device_id().empty()) {
|
||||
SetPreferredDevice(&graph_def, options.preferred_device_id());
|
||||
}
|
||||
|
||||
const tf::Status tf_status = session->session->Create(graph_def);
|
||||
RET_CHECK(tf_status.ok()) << "Create failed: " << tf_status.ToString();
|
||||
|
||||
@@ -123,6 +148,9 @@ class TensorFlowSessionFromFrozenGraphCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
cc->OutputSidePackets().Tag("SESSION").Set(Adopt(session.release()));
|
||||
const uint64 end_time = absl::ToUnixMicros(clock->TimeNow());
|
||||
LOG(INFO) << "Loaded frozen model in: " << end_time - start_time
|
||||
<< " microseconds.";
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
|
||||
|
||||
@@ -69,4 +69,12 @@ message TensorFlowSessionFromFrozenGraphCalculatorOptions {
|
||||
// Graph nodes to run to initialize the model. Any output of these ops is
|
||||
// ignored.
|
||||
repeated string initialization_op_names = 4;
|
||||
|
||||
// The id of the device you would prefer to execute the graph nodes on.
|
||||
// If set, all graph nodes without a previously specified device, will be set
|
||||
// to run on preferred_device_id. Example values include:
|
||||
// ["/device:GPU:0","/device:CPU:0", ...]
|
||||
// NOTE: If config.allow_soft_placement = false, and the device is not found,
|
||||
// an error will be thrown.
|
||||
optional string preferred_device_id = 5;
|
||||
}
|
||||
|
||||
+1
@@ -66,6 +66,7 @@ class TensorFlowSessionFromFrozenGraphCalculatorTest : public ::testing::Test {
|
||||
(*calculator_options_->mutable_tag_to_tensor_names())["B"] = "b:0";
|
||||
calculator_options_->mutable_config()->set_intra_op_parallelism_threads(1);
|
||||
calculator_options_->mutable_config()->set_inter_op_parallelism_threads(2);
|
||||
calculator_options_->set_preferred_device_id("/device:CPU:0");
|
||||
}
|
||||
|
||||
void VerifySignatureMap(const TensorFlowSession& session) {
|
||||
|
||||
@@ -27,16 +27,32 @@
|
||||
#include "mediapipe/calculators/tensorflow/tensorflow_session.h"
|
||||
#include "mediapipe/calculators/tensorflow/tensorflow_session_from_frozen_graph_generator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/deps/clock.h"
|
||||
#include "mediapipe/framework/deps/monotonic_clock.h"
|
||||
#include "mediapipe/framework/port/file_helpers.h"
|
||||
#include "mediapipe/framework/port/logging.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/tool/status_util.h"
|
||||
#include "tensorflow/core/framework/graph.pb.h"
|
||||
#include "tensorflow/core/framework/node_def.pb.h"
|
||||
#include "tensorflow/core/public/session_options.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
namespace tf = ::tensorflow;
|
||||
|
||||
namespace {
|
||||
// Updates the graph nodes to use the device as specified by device_id.
|
||||
void SetPreferredDevice(tf::GraphDef* graph_def, absl::string_view device_id) {
|
||||
for (auto& node : *graph_def->mutable_node()) {
|
||||
if (node.device().empty()) {
|
||||
node.set_device(device_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
class TensorFlowSessionFromFrozenGraphGenerator : public PacketGenerator {
|
||||
public:
|
||||
static ::mediapipe::Status FillExpectations(
|
||||
@@ -77,6 +93,9 @@ class TensorFlowSessionFromFrozenGraphGenerator : public PacketGenerator {
|
||||
static ::mediapipe::Status Generate(
|
||||
const PacketGeneratorOptions& packet_generator_options,
|
||||
const PacketSet& input_side_packets, PacketSet* output_side_packets) {
|
||||
auto clock = std::unique_ptr<mediapipe::Clock>(
|
||||
mediapipe::MonotonicClock::CreateSynchronizedMonotonicClock());
|
||||
const uint64 start_time = absl::ToUnixMicros(clock->TimeNow());
|
||||
const TensorFlowSessionFromFrozenGraphGeneratorOptions& options =
|
||||
packet_generator_options.GetExtension(
|
||||
TensorFlowSessionFromFrozenGraphGeneratorOptions::ext);
|
||||
@@ -108,6 +127,12 @@ class TensorFlowSessionFromFrozenGraphGenerator : public PacketGenerator {
|
||||
tensorflow::GraphDef graph_def;
|
||||
|
||||
RET_CHECK(graph_def.ParseFromString(graph_def_serialized));
|
||||
|
||||
// Update the graph nodes to use the preferred device, if set.
|
||||
if (!options.preferred_device_id().empty()) {
|
||||
SetPreferredDevice(&graph_def, options.preferred_device_id());
|
||||
}
|
||||
|
||||
const tf::Status tf_status = session->session->Create(graph_def);
|
||||
RET_CHECK(tf_status.ok()) << "Create failed: " << tf_status.ToString();
|
||||
|
||||
@@ -123,6 +148,9 @@ class TensorFlowSessionFromFrozenGraphGenerator : public PacketGenerator {
|
||||
}
|
||||
|
||||
output_side_packets->Tag("SESSION") = Adopt(session.release());
|
||||
const uint64 end_time = absl::ToUnixMicros(clock->TimeNow());
|
||||
LOG(INFO) << "Loaded frozen model in: " << end_time - start_time
|
||||
<< " microseconds.";
|
||||
return ::mediapipe::OkStatus();
|
||||
}
|
||||
};
|
||||
|
||||
@@ -69,4 +69,12 @@ message TensorFlowSessionFromFrozenGraphGeneratorOptions {
|
||||
// Graph nodes to run to initialize the model. Any output of these ops is
|
||||
// ignored.
|
||||
repeated string initialization_op_names = 4;
|
||||
|
||||
// The id of the device you would prefer to execute the graph nodes on.
|
||||
// If set, all graph nodes without a previously specified device, will be set
|
||||
// to run on preferred_device_id. Example values include:
|
||||
// ["/device:GPU:0","/device:CPU:0", ...]
|
||||
// NOTE: If config.allow_soft_placement = false, and the device is not found,
|
||||
// an error will be thrown.
|
||||
optional string preferred_device_id = 5;
|
||||
}
|
||||
|
||||
+1
@@ -66,6 +66,7 @@ class TensorFlowSessionFromFrozenGraphGeneratorTest : public ::testing::Test {
|
||||
(*generator_options_->mutable_tag_to_tensor_names())["B"] = "b:0";
|
||||
generator_options_->mutable_config()->set_intra_op_parallelism_threads(1);
|
||||
generator_options_->mutable_config()->set_inter_op_parallelism_threads(2);
|
||||
generator_options_->set_preferred_device_id("/device:CPU:0");
|
||||
}
|
||||
|
||||
void VerifySignatureMap(PacketSet* output_side_packets) {
|
||||
|
||||
@@ -134,8 +134,8 @@ class TensorFlowSessionFromSavedModelCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
tensorflow::RunOptions run_options;
|
||||
// In the future, could construct session options from the options proto.
|
||||
tensorflow::SessionOptions session_options;
|
||||
session_options.config = options.session_config();
|
||||
auto saved_model = absl::make_unique<tensorflow::SavedModelBundle>();
|
||||
::tensorflow::Status status = tensorflow::LoadSavedModel(
|
||||
session_options, run_options, path, tags_set, saved_model.get());
|
||||
|
||||
@@ -17,6 +17,7 @@ syntax = "proto2";
|
||||
package mediapipe;
|
||||
|
||||
import "mediapipe/framework/calculator.proto";
|
||||
import "tensorflow/core/protobuf/config.proto";
|
||||
|
||||
message TensorFlowSessionFromSavedModelCalculatorOptions {
|
||||
extend mediapipe.CalculatorOptions {
|
||||
@@ -55,4 +56,7 @@ message TensorFlowSessionFromSavedModelCalculatorOptions {
|
||||
// If no tag is specified, then use "serve" as the default. Note that in order
|
||||
// to use TPU accelerator hardware, the tag "tpu" needs to be specified.
|
||||
repeated string saved_model_tag = 6;
|
||||
|
||||
// Tensorflow session config options.
|
||||
optional tensorflow.ConfigProto session_config = 7;
|
||||
}
|
||||
|
||||
+27
@@ -26,6 +26,7 @@
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
#include "mediapipe/framework/tool/tag_map_helper.h"
|
||||
#include "mediapipe/framework/tool/validate_type.h"
|
||||
#include "tensorflow/core/framework/device_attributes.pb.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
@@ -204,5 +205,31 @@ TEST_F(TensorFlowSessionFromSavedModelCalculatorTest,
|
||||
ASSERT_NE(session.session, nullptr);
|
||||
}
|
||||
|
||||
TEST_F(TensorFlowSessionFromSavedModelCalculatorTest,
|
||||
ConfiguresSessionGivenConfig) {
|
||||
options_->set_saved_model_path(
|
||||
std::string(file::SplitPath(GetSavedModelDir()).first));
|
||||
options_->set_load_latest_model(true);
|
||||
options_->mutable_session_config()->mutable_device_count()->insert(
|
||||
{"CPU", 10});
|
||||
CalculatorRunner runner(absl::Substitute(R"(
|
||||
calculator: "TensorFlowSessionFromSavedModelCalculator"
|
||||
output_side_packet: "SESSION:tf_model"
|
||||
options {
|
||||
[mediapipe.TensorFlowSessionFromSavedModelCalculatorOptions.ext]: {
|
||||
$0
|
||||
}
|
||||
})",
|
||||
options_->DebugString()));
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
const TensorFlowSession& session =
|
||||
runner.OutputSidePackets().Tag("SESSION").Get<TensorFlowSession>();
|
||||
// Session must be set.
|
||||
ASSERT_NE(session.session, nullptr);
|
||||
std::vector<tensorflow::DeviceAttributes> devices;
|
||||
ASSERT_EQ(session.session->ListDevices(&devices), tensorflow::Status::OK());
|
||||
EXPECT_THAT(devices.size(), 10);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -129,8 +129,8 @@ class TensorFlowSessionFromSavedModelGenerator : public PacketGenerator {
|
||||
}
|
||||
|
||||
tensorflow::RunOptions run_options;
|
||||
// In the future, could construct session options from the options proto.
|
||||
tensorflow::SessionOptions session_options;
|
||||
session_options.config = options.session_config();
|
||||
auto saved_model = absl::make_unique<tensorflow::SavedModelBundle>();
|
||||
::tensorflow::Status status = tensorflow::LoadSavedModel(
|
||||
session_options, run_options, path, tags_set, saved_model.get());
|
||||
|
||||
@@ -17,6 +17,7 @@ syntax = "proto2";
|
||||
package mediapipe;
|
||||
|
||||
import "mediapipe/framework/packet_generator.proto";
|
||||
import "tensorflow/core/protobuf/config.proto";
|
||||
|
||||
message TensorFlowSessionFromSavedModelGeneratorOptions {
|
||||
extend mediapipe.PacketGeneratorOptions {
|
||||
@@ -55,4 +56,7 @@ message TensorFlowSessionFromSavedModelGeneratorOptions {
|
||||
// If no tag is specified, then use "serve" as the default. Note that in order
|
||||
// to use TPU accelerator hardware, the tag "tpu" needs to be specified.
|
||||
repeated string saved_model_tag = 6;
|
||||
|
||||
// Tensorflow session config options.
|
||||
optional tensorflow.ConfigProto session_config = 9;
|
||||
}
|
||||
|
||||
+25
@@ -25,6 +25,7 @@
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
#include "mediapipe/framework/tool/tag_map_helper.h"
|
||||
#include "mediapipe/framework/tool/validate_type.h"
|
||||
#include "tensorflow/core/framework/device_attributes.pb.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
@@ -196,5 +197,29 @@ TEST_F(TensorFlowSessionFromSavedModelGeneratorTest,
|
||||
ASSERT_NE(session.session, nullptr);
|
||||
}
|
||||
|
||||
TEST_F(TensorFlowSessionFromSavedModelGeneratorTest,
|
||||
ConfiguresSessionGivenConfig) {
|
||||
generator_options_->set_saved_model_path(
|
||||
std::string(file::SplitPath(GetSavedModelDir()).first));
|
||||
generator_options_->set_load_latest_model(true);
|
||||
generator_options_->mutable_session_config()->mutable_device_count()->insert(
|
||||
{"CPU", 10});
|
||||
|
||||
PacketSet input_side_packets(tool::CreateTagMap({}).ValueOrDie());
|
||||
PacketSet output_side_packets(
|
||||
tool::CreateTagMap({"SESSION:session"}).ValueOrDie());
|
||||
::mediapipe::Status run_status = tool::RunGenerateAndValidateTypes(
|
||||
"TensorFlowSessionFromSavedModelGenerator", extendable_options_,
|
||||
input_side_packets, &output_side_packets);
|
||||
MP_EXPECT_OK(run_status) << run_status.message();
|
||||
const TensorFlowSession& session =
|
||||
output_side_packets.Tag("SESSION").Get<TensorFlowSession>();
|
||||
// Session must be set.
|
||||
ASSERT_NE(session.session, nullptr);
|
||||
std::vector<tensorflow::DeviceAttributes> devices;
|
||||
ASSERT_EQ(session.session->ListDevices(&devices), tensorflow::Status::OK());
|
||||
EXPECT_THAT(devices.size(), 10);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -29,6 +29,7 @@ namespace mediapipe {
|
||||
// Streams:
|
||||
const char kBBoxTag[] = "BBOX";
|
||||
const char kImageTag[] = "IMAGE";
|
||||
const char kKeypointsTag[] = "KEYPOINTS";
|
||||
const char kFloatFeaturePrefixTag[] = "FLOAT_FEATURE_";
|
||||
const char kForwardFlowImageTag[] = "FORWARD_FLOW_ENCODED";
|
||||
|
||||
@@ -150,7 +151,6 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
|
||||
<< "or" << kAudioDecoderOptions;
|
||||
}
|
||||
|
||||
// Optional streams.
|
||||
if (cc->Outputs().HasTag(kForwardFlowImageTag)) {
|
||||
cc->Outputs().Tag(kForwardFlowImageTag).Set<std::string>();
|
||||
}
|
||||
@@ -244,6 +244,10 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
|
||||
const auto& sequence = cc->InputSidePackets()
|
||||
.Tag(kSequenceExampleTag)
|
||||
.Get<tensorflow::SequenceExample>();
|
||||
if (cc->Outputs().HasTag(kKeypointsTag)) {
|
||||
keypoint_names_ = absl::StrSplit(options.keypoint_names(), ',');
|
||||
default_keypoint_location_ = options.default_keypoint_location();
|
||||
}
|
||||
if (cc->OutputSidePackets().HasTag(kDataPath)) {
|
||||
std::string root_directory = "";
|
||||
if (cc->InputSidePackets().HasTag(kDatasetRootDirTag)) {
|
||||
@@ -357,7 +361,6 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
|
||||
end_timestamp =
|
||||
timestamps_[last_timestamp_key_][current_timestamp_index_ + 1];
|
||||
}
|
||||
|
||||
for (const auto& map_kv : timestamps_) {
|
||||
for (int i = 0; i < map_kv.second.size(); ++i) {
|
||||
if (map_kv.second[i] >= start_timestamp &&
|
||||
@@ -454,6 +457,10 @@ class UnpackMediaSequenceCalculator : public CalculatorBase {
|
||||
int current_timestamp_index_;
|
||||
// Store the very first timestamp, so we output everything on the first frame.
|
||||
int64 first_timestamp_seen_;
|
||||
// List of keypoint names.
|
||||
std::vector<std::string> keypoint_names_;
|
||||
// Default keypoint location when missing.
|
||||
float default_keypoint_location_;
|
||||
};
|
||||
REGISTER_CALCULATOR(UnpackMediaSequenceCalculator);
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -91,11 +91,11 @@ REGISTER_CALCULATOR(VectorFloatToTensorCalculator);
|
||||
cc->Inputs().Index(0).Value().Get<std::vector<std::vector<float>>>();
|
||||
|
||||
const int32 rows = input.size();
|
||||
CHECK_GE(rows, 1);
|
||||
RET_CHECK_GE(rows, 1);
|
||||
const int32 cols = input[0].size();
|
||||
CHECK_GE(cols, 1);
|
||||
RET_CHECK_GE(cols, 1);
|
||||
for (int i = 1; i < rows; ++i) {
|
||||
CHECK_EQ(input[i].size(), cols);
|
||||
RET_CHECK_EQ(input[i].size(), cols);
|
||||
}
|
||||
if (options_.transpose()) {
|
||||
tensor_shape = tf::TensorShape({cols, rows});
|
||||
@@ -116,7 +116,7 @@ REGISTER_CALCULATOR(VectorFloatToTensorCalculator);
|
||||
} else if (options_.input_size() == INPUT_1D) {
|
||||
const std::vector<float>& input =
|
||||
cc->Inputs().Index(0).Value().Get<std::vector<float>>();
|
||||
CHECK_GE(input.size(), 1);
|
||||
RET_CHECK_GE(input.size(), 1);
|
||||
const int32 length = input.size();
|
||||
tensor_shape = tf::TensorShape({length});
|
||||
auto output = ::absl::make_unique<tf::Tensor>(tf::DT_FLOAT, tensor_shape);
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||
load("@bazel_skylib//lib:selects.bzl", "selects")
|
||||
|
||||
licenses(["notice"]) # Apache 2.0
|
||||
licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//visibility:private"])
|
||||
|
||||
@@ -196,13 +196,6 @@ cc_test(
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "util",
|
||||
hdrs = ["util.h"],
|
||||
visibility = ["//visibility:public"],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
selects.config_setting_group(
|
||||
name = "gpu_inference_disabled",
|
||||
match_any = [
|
||||
@@ -229,7 +222,6 @@ cc_library(
|
||||
}),
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":util",
|
||||
":tflite_inference_calculator_cc_proto",
|
||||
"@com_google_absl//absl/memory",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
@@ -265,6 +257,7 @@ cc_library(
|
||||
}) + select({
|
||||
"//conditions:default": [],
|
||||
"//mediapipe:android": [
|
||||
"//mediapipe/util/android/file/base",
|
||||
"@org_tensorflow//tensorflow/lite/delegates/nnapi:nnapi_delegate",
|
||||
],
|
||||
}) + select({
|
||||
@@ -295,7 +288,6 @@ cc_library(
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/util/tflite:config",
|
||||
":util",
|
||||
":tflite_converter_calculator_cc_proto",
|
||||
"//mediapipe/util:resource_util",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
@@ -334,7 +326,6 @@ cc_library(
|
||||
srcs = ["tflite_model_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":util",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:packet",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
@@ -348,7 +339,6 @@ cc_library(
|
||||
srcs = ["tflite_tensors_to_segmentation_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":util",
|
||||
":tflite_tensors_to_segmentation_calculator_cc_proto",
|
||||
"@com_google_absl//absl/strings:str_format",
|
||||
"@com_google_absl//absl/types:span",
|
||||
@@ -418,7 +408,6 @@ cc_library(
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/util/tflite:config",
|
||||
":util",
|
||||
":tflite_tensors_to_detections_calculator_cc_proto",
|
||||
"//mediapipe/framework/formats:detection_cc_proto",
|
||||
"@com_google_absl//absl/strings:str_format",
|
||||
@@ -551,6 +540,7 @@ cc_test(
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/tool:validate_type",
|
||||
"@com_google_absl//absl/memory",
|
||||
"@com_google_absl//absl/strings",
|
||||
"@org_tensorflow//tensorflow/lite:framework",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -16,7 +16,6 @@
|
||||
#include <vector>
|
||||
|
||||
#include "mediapipe/calculators/tflite/tflite_converter_calculator.pb.h"
|
||||
#include "mediapipe/calculators/tflite/util.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/formats/image_frame.h"
|
||||
#include "mediapipe/framework/formats/matrix.h"
|
||||
@@ -146,8 +145,7 @@ class TfLiteConverterCalculator : public CalculatorBase {
|
||||
::mediapipe::Status LoadOptions(CalculatorContext* cc);
|
||||
template <class T>
|
||||
::mediapipe::Status NormalizeImage(const ImageFrame& image_frame,
|
||||
bool zero_center, bool flip_vertically,
|
||||
float* tensor_ptr);
|
||||
bool flip_vertically, float* tensor_ptr);
|
||||
::mediapipe::Status CopyMatrixToTensor(const Matrix& matrix,
|
||||
float* tensor_ptr);
|
||||
::mediapipe::Status ProcessCPU(CalculatorContext* cc);
|
||||
@@ -165,10 +163,7 @@ class TfLiteConverterCalculator : public CalculatorBase {
|
||||
|
||||
bool initialized_ = false;
|
||||
bool use_gpu_ = false;
|
||||
bool zero_center_ = true; // normalize range to [-1,1] | otherwise [0,1]
|
||||
bool use_custom_normalization_ = false;
|
||||
float custom_div_ = -1.0f;
|
||||
float custom_sub_ = -1.0f;
|
||||
absl::optional<std::pair<float, float>> output_range_;
|
||||
bool flip_vertically_ = false;
|
||||
bool row_major_matrix_ = false;
|
||||
bool use_quantized_tensors_ = false;
|
||||
@@ -362,11 +357,11 @@ bool ShouldUseGpu(CC* cc) {
|
||||
float* tensor_buffer = tensor->data.f;
|
||||
RET_CHECK(tensor_buffer);
|
||||
if (image_frame.ByteDepth() == 1) {
|
||||
MP_RETURN_IF_ERROR(NormalizeImage<uint8>(
|
||||
image_frame, zero_center_, flip_vertically_, tensor_buffer));
|
||||
MP_RETURN_IF_ERROR(NormalizeImage<uint8>(image_frame, flip_vertically_,
|
||||
tensor_buffer));
|
||||
} else if (image_frame.ByteDepth() == 4) {
|
||||
MP_RETURN_IF_ERROR(NormalizeImage<float>(
|
||||
image_frame, zero_center_, flip_vertically_, tensor_buffer));
|
||||
MP_RETURN_IF_ERROR(NormalizeImage<float>(image_frame, flip_vertically_,
|
||||
tensor_buffer));
|
||||
} else {
|
||||
return ::mediapipe::InternalError(
|
||||
"Only byte-based (8 bit) and float (32 bit) images supported.");
|
||||
@@ -427,11 +422,11 @@ bool ShouldUseGpu(CC* cc) {
|
||||
auto src = gpu_helper_.CreateSourceTexture(input);
|
||||
glActiveTexture(GL_TEXTURE0 + 0);
|
||||
glBindTexture(GL_TEXTURE_2D, src.name());
|
||||
RET_CHECK_CALL(gpu_data_out_->buffer.BindToIndex(1));
|
||||
MP_RETURN_IF_ERROR(gpu_data_out_->buffer.BindToIndex(1));
|
||||
const tflite::gpu::uint3 workgroups = {
|
||||
NumGroups(input.width(), kWorkgroupSize),
|
||||
NumGroups(input.height(), kWorkgroupSize), 1};
|
||||
RET_CHECK_CALL(gpu_data_out_->program.Dispatch(workgroups));
|
||||
MP_RETURN_IF_ERROR(gpu_data_out_->program.Dispatch(workgroups));
|
||||
glBindBuffer(GL_SHADER_STORAGE_BUFFER, 0);
|
||||
glBindTexture(GL_TEXTURE_2D, 0);
|
||||
src.Release();
|
||||
@@ -445,9 +440,9 @@ bool ShouldUseGpu(CC* cc) {
|
||||
output_tensors->resize(1);
|
||||
{
|
||||
GpuTensor& tensor = output_tensors->at(0);
|
||||
RET_CHECK_CALL(CreateReadWriteShaderStorageBuffer<float>(
|
||||
MP_RETURN_IF_ERROR(CreateReadWriteShaderStorageBuffer<float>(
|
||||
gpu_data_out_->elements, &tensor));
|
||||
RET_CHECK_CALL(CopyBuffer(gpu_data_out_->buffer, tensor));
|
||||
MP_RETURN_IF_ERROR(CopyBuffer(gpu_data_out_->buffer, tensor));
|
||||
}
|
||||
return ::mediapipe::OkStatus();
|
||||
}));
|
||||
@@ -521,7 +516,7 @@ bool ShouldUseGpu(CC* cc) {
|
||||
MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext(
|
||||
[this, &include_alpha, &input, &single_channel]() -> ::mediapipe::Status {
|
||||
// Device memory.
|
||||
RET_CHECK_CALL(
|
||||
MP_RETURN_IF_ERROR(
|
||||
::tflite::gpu::gl::CreateReadWriteShaderStorageBuffer<float>(
|
||||
gpu_data_out_->elements, &gpu_data_out_->buffer));
|
||||
|
||||
@@ -544,7 +539,13 @@ bool ShouldUseGpu(CC* cc) {
|
||||
$6 // alpha channel
|
||||
})",
|
||||
/*$0=*/kWorkgroupSize, /*$1=*/input.width(), /*$2=*/input.height(),
|
||||
/*$3=*/zero_center_ ? "pixel = (pixel - 0.5) * 2.0;" : "",
|
||||
/*$3=*/
|
||||
output_range_.has_value()
|
||||
? absl::Substitute(
|
||||
"pixel = pixel * float($0) + float($1);",
|
||||
(output_range_->second - output_range_->first),
|
||||
output_range_->first)
|
||||
: "",
|
||||
/*$4=*/flip_vertically_ ? "(width_height.y - 1 - gid.y)" : "gid.y",
|
||||
/*$5=*/
|
||||
single_channel
|
||||
@@ -555,10 +556,10 @@ bool ShouldUseGpu(CC* cc) {
|
||||
include_alpha ? "output_data.elements[linear_index + 3] = pixel.w;"
|
||||
: "",
|
||||
/*$7=*/max_num_channels_);
|
||||
RET_CHECK_CALL(GlShader::CompileShader(GL_COMPUTE_SHADER, shader_source,
|
||||
&gpu_data_out_->shader));
|
||||
RET_CHECK_CALL(GlProgram::CreateWithShader(gpu_data_out_->shader,
|
||||
&gpu_data_out_->program));
|
||||
MP_RETURN_IF_ERROR(GlShader::CompileShader(
|
||||
GL_COMPUTE_SHADER, shader_source, &gpu_data_out_->shader));
|
||||
MP_RETURN_IF_ERROR(GlProgram::CreateWithShader(
|
||||
gpu_data_out_->shader, &gpu_data_out_->program));
|
||||
return ::mediapipe::OkStatus();
|
||||
}));
|
||||
|
||||
@@ -599,7 +600,12 @@ bool ShouldUseGpu(CC* cc) {
|
||||
)",
|
||||
/*$0=*/include_alpha ? "float4" : "float3",
|
||||
/*$1=*/include_alpha ? "rgba" : "rgb",
|
||||
/*$2=*/zero_center_ ? "pixel = (pixel - 0.5) * 2.0;" : "",
|
||||
/*$2=*/
|
||||
output_range_.has_value()
|
||||
? absl::Substitute("pixel = pixel * float($0) + float($1);",
|
||||
(output_range_->second - output_range_->first),
|
||||
output_range_->first)
|
||||
: "",
|
||||
/*$3=*/flip_vertically_ ? "(in_tex.get_height() - 1 - gid.y)" : "gid.y",
|
||||
/*$4=*/include_alpha ? 4 : 3,
|
||||
/*$5=*/include_alpha ? "out_buf[linear_index + 3] = pixel.w;" : "");
|
||||
@@ -630,13 +636,27 @@ bool ShouldUseGpu(CC* cc) {
|
||||
const auto& options =
|
||||
cc->Options<::mediapipe::TfLiteConverterCalculatorOptions>();
|
||||
|
||||
// Get data normalization mode.
|
||||
zero_center_ = options.zero_center();
|
||||
// if zero_center, set output float range to match [-1, 1] as specified in
|
||||
// calculator proto.
|
||||
if (options.zero_center()) {
|
||||
output_range_.emplace(std::pair<float, float>(-1.0, 1.0));
|
||||
}
|
||||
|
||||
// Custom output_tensor_float_range values.
|
||||
// If the float range is specified in pb text, use the specified values
|
||||
// instead.
|
||||
if (options.has_output_tensor_float_range()) {
|
||||
output_range_.emplace(options.output_tensor_float_range().min(),
|
||||
options.output_tensor_float_range().max());
|
||||
CHECK_GT(output_range_->second, output_range_->first);
|
||||
}
|
||||
|
||||
// Custom div and sub values.
|
||||
use_custom_normalization_ = options.use_custom_normalization();
|
||||
custom_div_ = options.custom_div();
|
||||
custom_sub_ = options.custom_sub();
|
||||
if (options.use_custom_normalization()) {
|
||||
output_range_.emplace(std::pair<float, float>(
|
||||
-options.custom_sub(),
|
||||
-options.custom_sub() + 255.0 / options.custom_div()));
|
||||
}
|
||||
|
||||
// Get y-flip mode.
|
||||
flip_vertically_ = options.flip_vertically();
|
||||
@@ -664,40 +684,46 @@ bool ShouldUseGpu(CC* cc) {
|
||||
|
||||
template <class T>
|
||||
::mediapipe::Status TfLiteConverterCalculator::NormalizeImage(
|
||||
const ImageFrame& image_frame, bool zero_center, bool flip_vertically,
|
||||
float* tensor_ptr) {
|
||||
const ImageFrame& image_frame, bool flip_vertically, float* tensor_ptr) {
|
||||
const int height = image_frame.Height();
|
||||
const int width = image_frame.Width();
|
||||
const int channels = image_frame.NumberOfChannels();
|
||||
const int channels_preserved = std::min(channels, max_num_channels_);
|
||||
const int channels_ignored = channels - channels_preserved;
|
||||
|
||||
float div, sub;
|
||||
if (output_range_.has_value()) {
|
||||
// If the output float range is set and we are not using custom
|
||||
// normalization, normalize the pixel values from [0, 255] to the specified
|
||||
// output range.
|
||||
RET_CHECK_NE(output_range_->first, output_range_->second);
|
||||
const float scale = (output_range_->second - output_range_->first) / 255.0f;
|
||||
const float bias = output_range_->first;
|
||||
|
||||
if (use_custom_normalization_) {
|
||||
RET_CHECK_GT(custom_div_, 0.0f);
|
||||
RET_CHECK_GE(custom_sub_, 0.0f);
|
||||
div = custom_div_;
|
||||
sub = custom_sub_;
|
||||
} else if (zero_center) {
|
||||
// [-1,1]
|
||||
div = 127.5f;
|
||||
sub = 1.0f;
|
||||
for (int i = 0; i < height; ++i) {
|
||||
const T* image_ptr = reinterpret_cast<const T*>(
|
||||
image_frame.PixelData() +
|
||||
(flip_vertically ? height - 1 - i : i) * image_frame.WidthStep());
|
||||
for (int j = 0; j < width; ++j) {
|
||||
for (int c = 0; c < channels_preserved; ++c) {
|
||||
*tensor_ptr++ = *image_ptr++ * scale + bias;
|
||||
}
|
||||
image_ptr += channels_ignored;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// [0,1]
|
||||
div = 255.0f;
|
||||
sub = 0.0f;
|
||||
}
|
||||
|
||||
for (int i = 0; i < height; ++i) {
|
||||
const T* image_ptr = reinterpret_cast<const T*>(
|
||||
image_frame.PixelData() +
|
||||
(flip_vertically ? height - 1 - i : i) * image_frame.WidthStep());
|
||||
for (int j = 0; j < width; ++j) {
|
||||
for (int c = 0; c < channels_preserved; ++c) {
|
||||
*tensor_ptr++ = *image_ptr++ / div - sub;
|
||||
// [0,1], scale only (bias == 0)
|
||||
// Verified that there are no precision issues with 1.0f / 255.0f expression
|
||||
const float scale = 1.0f / 255.0f;
|
||||
for (int i = 0; i < height; ++i) {
|
||||
const T* image_ptr = reinterpret_cast<const T*>(
|
||||
image_frame.PixelData() +
|
||||
(flip_vertically ? height - 1 - i : i) * image_frame.WidthStep());
|
||||
for (int j = 0; j < width; ++j) {
|
||||
for (int c = 0; c < channels_preserved; ++c) {
|
||||
*tensor_ptr++ = *image_ptr++ * scale;
|
||||
}
|
||||
image_ptr += channels_ignored;
|
||||
}
|
||||
image_ptr += channels_ignored;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -56,4 +56,14 @@ message TfLiteConverterCalculatorOptions {
|
||||
// Quantization option (CPU only).
|
||||
// When true, output kTfLiteUInt8 tensor instead of kTfLiteFloat32.
|
||||
optional bool use_quantized_tensors = 5 [default = false];
|
||||
|
||||
// Normalization option.
|
||||
// Setting normalization_range results in the values normalized to
|
||||
// the range [output_tensor_float_range.min, output_tensor_float_range.max].
|
||||
optional TensorFloatRange output_tensor_float_range = 9;
|
||||
|
||||
message TensorFloatRange {
|
||||
optional float min = 1;
|
||||
optional float max = 2;
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user