Compare commits
@@ -1,27 +0,0 @@
|
||||
---
|
||||
name: "Build/Installation Issue"
|
||||
about: Use this template for build/installation issues
|
||||
labels: type:build/install
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
- OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
||||
- Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
|
||||
- Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
|
||||
- Installed using virtualenv? pip? Conda? (if python):
|
||||
- [MediaPipe version](https://github.com/google/mediapipe/releases):
|
||||
- Bazel version:
|
||||
- XCode and Tulsi versions (if iOS):
|
||||
- Android SDK and NDK versions (if android):
|
||||
- Android [AAR](https://google.github.io/mediapipe/getting_started/android_archive_library.html) ( if android):
|
||||
- OpenCV version (if running on desktop):
|
||||
|
||||
**Describe the problem**:
|
||||
|
||||
|
||||
**[Provide the exact sequence of commands / steps that you executed before running into the problem](https://google.github.io/mediapipe/getting_started/getting_started.html):**
|
||||
|
||||
**Complete Logs:**
|
||||
Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:
|
||||
@@ -1,25 +0,0 @@
|
||||
---
|
||||
name: "Tasks Issue"
|
||||
about: Use this template for assistance with using MediaPipe Tasks (developers.google.com/mediapipe/solutions) to deploy on-device ML solutions (e.g. gesture recognition etc.) on supported platforms.
|
||||
labels: type:support
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a [Tasks](https://developers.google.com/mediapipe/solutions) issue.<em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
||||
- MediaPipe Tasks SDK version:
|
||||
- Task name (e.g. Object detection, Gesture recognition etc.):
|
||||
- Programming Language and version ( e.g. C++, Python, Java):
|
||||
|
||||
**Describe the expected behavior:**
|
||||
|
||||
**Standalone code you may have used to try to get what you need :**
|
||||
|
||||
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem:
|
||||
|
||||
**Other info / Complete Logs :**
|
||||
Include any logs or source code that would be helpful to
|
||||
diagnose the problem. If including tracebacks, please include the full
|
||||
traceback. Large logs and files should be attached:
|
||||
@@ -1,25 +0,0 @@
|
||||
---
|
||||
name: "Model Maker Issue"
|
||||
about: Use this template for assistance with using MediaPipe Model Maker (developers.google.com/mediapipe/solutions) to create custom on-device ML solutions.
|
||||
labels: type:support
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a [Model Maker](https://developers.google.com/mediapipe/solutions) issue.<em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
|
||||
- Python version (e.g. 3.8):
|
||||
- [MediaPipe Model Maker version](https://pypi.org/project/mediapipe-model-maker/):
|
||||
- Task name (e.g. Image classification, Gesture recognition etc.):
|
||||
|
||||
**Describe the expected behavior:**
|
||||
|
||||
**Standalone code you may have used to try to get what you need :**
|
||||
|
||||
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem:
|
||||
|
||||
**Other info / Complete Logs :**
|
||||
Include any logs or source code that would be helpful to
|
||||
diagnose the problem. If including tracebacks, please include the full
|
||||
traceback. Large logs and files should be attached:
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
name: "Solution (legacy) Issue"
|
||||
about: Use this template for assistance with a specific Mediapipe solution (google.github.io/mediapipe/solutions) such as "Pose", including inference model usage/training, solution-specific calculators etc.
|
||||
labels: type:support
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
- Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
||||
- [MediaPipe version](https://github.com/google/mediapipe/releases):
|
||||
- Bazel version:
|
||||
- Solution (e.g. FaceMesh, Pose, Holistic):
|
||||
- Programming Language and version ( e.g. C++, Python, Java):
|
||||
|
||||
**Describe the expected behavior:**
|
||||
|
||||
**Standalone code you may have used to try to get what you need :**
|
||||
|
||||
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
|
||||
|
||||
**Other info / Complete Logs :**
|
||||
Include any logs or source code that would be helpful to
|
||||
diagnose the problem. If including tracebacks, please include the full
|
||||
traceback. Large logs and files should be attached:
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
name: "Studio Issue"
|
||||
about: Use this template for assistance with the MediaPipe Studio application.
|
||||
labels: type:support
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a MediaPipe Studio issue.<em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
||||
- Browser and Version
|
||||
- Any microphone or camera hardware
|
||||
- URL that shows the problem
|
||||
|
||||
**Describe the expected behavior:**
|
||||
|
||||
**Other info / Complete Logs :**
|
||||
Include any js console logs that would be helpful to diagnose the problem.
|
||||
Large logs and files should be attached:
|
||||
@@ -1,51 +0,0 @@
|
||||
---
|
||||
name: "Documentation Issue"
|
||||
about: Use this template for documentation related issues
|
||||
labels: type:docs
|
||||
|
||||
---
|
||||
Thank you for submitting a MediaPipe documentation issue.
|
||||
The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
|
||||
## URL(s) with the issue:
|
||||
|
||||
Please provide a link to the documentation entry, for example: https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
|
||||
|
||||
## Description of issue (what needs changing):
|
||||
|
||||
Kinds of documentation problems:
|
||||
|
||||
### Clear description
|
||||
|
||||
For example, why should someone use this method? How is it useful?
|
||||
|
||||
### Correct links
|
||||
|
||||
Is the link to the source code correct?
|
||||
|
||||
### Parameters defined
|
||||
Are all parameters defined and formatted correctly?
|
||||
|
||||
### Returns defined
|
||||
|
||||
Are return values defined?
|
||||
|
||||
### Raises listed and defined
|
||||
|
||||
Are the errors defined? For example,
|
||||
|
||||
### Usage example
|
||||
|
||||
Is there a usage example?
|
||||
|
||||
See the API guide:
|
||||
on how to write testable usage examples.
|
||||
|
||||
### Request visuals, if applicable
|
||||
|
||||
Are there currently visuals? If not, will it clarify the content?
|
||||
|
||||
### Submit a pull request?
|
||||
|
||||
Are you planning to also submit a pull request to fix the issue? See the docs
|
||||
https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
---
|
||||
name: "Bug Issue"
|
||||
about: Use this template for reporting a bug
|
||||
labels: type:bug
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.</em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
|
||||
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
|
||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
||||
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
|
||||
- Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser:
|
||||
- Programming Language and version ( e.g. C++, Python, Java):
|
||||
- [MediaPipe version](https://github.com/google/mediapipe/releases):
|
||||
- Bazel version (if compiling from source):
|
||||
- Solution ( e.g. FaceMesh, Pose, Holistic ):
|
||||
- Android Studio, NDK, SDK versions (if issue is related to building in Android environment):
|
||||
- Xcode & Tulsi version (if issue is related to building for iOS):
|
||||
|
||||
**Describe the current behavior:**
|
||||
|
||||
**Describe the expected behavior:**
|
||||
|
||||
**Standalone code to reproduce the issue:**
|
||||
Provide a reproducible test case that is the bare minimum necessary to replicate the problem. If possible, please share a link to Colab/repo link /any notebook:
|
||||
|
||||
**Other info / Complete Logs :**
|
||||
Include any logs or source code that would be helpful to
|
||||
diagnose the problem. If including tracebacks, please include the full
|
||||
traceback. Large logs and files should be attached
|
||||
@@ -1,24 +0,0 @@
|
||||
---
|
||||
name: "Feature Request"
|
||||
about: Use this template for raising a feature request
|
||||
labels: type:feature
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a feature request.</em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
|
||||
- MediaPipe Solution (you are using):
|
||||
- Programming language : C++/typescript/Python/Objective C/Android Java
|
||||
- Are you willing to contribute it (Yes/No):
|
||||
|
||||
|
||||
**Describe the feature and the current behavior/state:**
|
||||
|
||||
**Will this change the current api? How?**
|
||||
|
||||
**Who will benefit with this feature?**
|
||||
|
||||
**Please specify the use cases for this feature:**
|
||||
|
||||
**Any Other info:**
|
||||
@@ -0,0 +1,73 @@
|
||||
name: Documentation issue
|
||||
description: Use this template for documentation related issues. If this doesn’t look right, choose a different type.
|
||||
labels: 'type:doc-bug'
|
||||
body:
|
||||
- type: markdown
|
||||
id: link
|
||||
attributes:
|
||||
value: Thank you for submitting a MediaPipe documentation issue. The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
|
||||
- type: markdown
|
||||
id: url
|
||||
attributes:
|
||||
value: URL(s) with the issue Please provide a link to the documentation entry, for example https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
|
||||
- type: input
|
||||
id: description
|
||||
attributes:
|
||||
label: Description of issue (what needs changing)
|
||||
description: Kinds of documentation problems
|
||||
- type: input
|
||||
id: clear_desc
|
||||
attributes:
|
||||
label: Clear description
|
||||
description: For example, why should someone use this method? How is it useful?
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: link
|
||||
attributes:
|
||||
label: Correct links
|
||||
description: Is the link to the source code correct?
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: parameter
|
||||
attributes:
|
||||
label: Parameters defined
|
||||
description: Are all parameters defined and formatted correctly?
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: returns
|
||||
attributes:
|
||||
label: Returns defined
|
||||
description: Are return values defined?
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: raises
|
||||
attributes:
|
||||
label: Raises listed and defined
|
||||
description: Are the errors defined? For example,
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: usage
|
||||
attributes:
|
||||
label: Usage example
|
||||
description: Is there a usage example? See the API guide-on how to write testable usage examples.
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: visual
|
||||
attributes:
|
||||
label: Request visuals, if applicable
|
||||
description: Are there currently visuals? If not, will it clarify the content?
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: pull
|
||||
attributes:
|
||||
label: Submit a pull request?
|
||||
description: Are you planning to also submit a pull request to fix the issue? See the [docs](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md)
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,80 @@
|
||||
name: Solution(Legacy) Issue
|
||||
description: Use this template for assistance with a specific Mediapipe solution (google.github.io/mediapipe/solutions) such as "Pose", including inference model usage/training, solution-specific calculators etc.
|
||||
labels: 'type:support'
|
||||
body:
|
||||
- type: markdown
|
||||
id: linkmodel
|
||||
attributes:
|
||||
value: Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.
|
||||
- type: dropdown
|
||||
id: customcode_model
|
||||
attributes:
|
||||
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||
options:
|
||||
- 'Yes'
|
||||
- 'No'
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: os_model
|
||||
attributes:
|
||||
label: OS Platform and Distribution
|
||||
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: mediapipe_version
|
||||
attributes:
|
||||
label: MediaPipe version
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: bazel_version
|
||||
attributes:
|
||||
label: Bazel version
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: solution
|
||||
attributes:
|
||||
label: Solution
|
||||
placeholder: e.g. FaceMesh, Pose, Holistic
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: programminglang
|
||||
attributes:
|
||||
label: Programming Language and version
|
||||
placeholder: e.g. C++, Python, Java
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: current_model
|
||||
attributes:
|
||||
label: Describe the actual behavior
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: expected_model
|
||||
attributes:
|
||||
label: Describe the expected behaviour
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: what-happened_model
|
||||
attributes:
|
||||
label: Standalone code/steps you may have used to try to get what you need
|
||||
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: other_info
|
||||
attributes:
|
||||
label: Other info / Complete Logs
|
||||
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,112 @@
|
||||
name: Bug Issues
|
||||
description: Use this template for reporting a bug. If this doesn’t look right, choose a different type.
|
||||
labels: 'type:bug'
|
||||
body:
|
||||
- type: markdown
|
||||
id: link
|
||||
attributes:
|
||||
value: Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.
|
||||
- type: dropdown
|
||||
id: customcode_model
|
||||
attributes:
|
||||
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||
options:
|
||||
- 'Yes'
|
||||
- 'No'
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: os
|
||||
attributes:
|
||||
label: OS Platform and Distribution
|
||||
description:
|
||||
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: mobile_device
|
||||
attributes:
|
||||
label: Mobile device if the issue happens on mobile device
|
||||
description:
|
||||
placeholder: e.g. iPhone 8, Pixel 2, Samsung Galaxy
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: browser_version
|
||||
attributes:
|
||||
label: Browser and version if the issue happens on browser
|
||||
placeholder: e.g. Google Chrome 109.0.5414.119, Safari 16.3
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: programminglang
|
||||
attributes:
|
||||
label: Programming Language and version
|
||||
placeholder: e.g. C++, Python, Java
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: mediapipever
|
||||
attributes:
|
||||
label: MediaPipe version
|
||||
description:
|
||||
placeholder: e.g. 0.8.11, 0.9.1
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: bazelver
|
||||
attributes:
|
||||
label: Bazel version
|
||||
description:
|
||||
placeholder: e.g. 5.0, 5.1
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: solution
|
||||
attributes:
|
||||
label: Solution
|
||||
placeholder: e.g. FaceMesh, Pose, Holistic
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: sdkndkversion
|
||||
attributes:
|
||||
label: Android Studio, NDK, SDK versions (if issue is related to building in Android environment)
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: xcode_ver
|
||||
attributes:
|
||||
label: Xcode & Tulsi version (if issue is related to building for iOS)
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: current_model
|
||||
attributes:
|
||||
label: Describe the actual behavior
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected_model
|
||||
attributes:
|
||||
label: Describe the expected behaviour
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: what-happened_model
|
||||
attributes:
|
||||
label: Standalone code/steps you may have used to try to get what you need
|
||||
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: other_info
|
||||
attributes:
|
||||
label: Other info / Complete Logs
|
||||
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,109 @@
|
||||
name: Build/Install Issue
|
||||
description: Use this template to report build/install issue
|
||||
labels: 'type:build/install'
|
||||
body:
|
||||
- type: markdown
|
||||
id: link
|
||||
attributes:
|
||||
value: Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.
|
||||
- type: input
|
||||
id: os
|
||||
attributes:
|
||||
label: OS Platform and Distribution
|
||||
description:
|
||||
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: compilerversion
|
||||
attributes:
|
||||
label: Compiler version
|
||||
description:
|
||||
placeholder: e.g. gcc/g++ 8 /Apple clang version 12.0.0
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: programminglang
|
||||
attributes:
|
||||
label: Programming Language and version
|
||||
description:
|
||||
placeholder: e.g. C++ 14, Python 3.6, Java
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: virtualenv
|
||||
attributes:
|
||||
label: Installed using virtualenv? pip? Conda?(if python)
|
||||
description:
|
||||
placeholder:
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: mediapipever
|
||||
attributes:
|
||||
label: MediaPipe version
|
||||
description:
|
||||
placeholder: e.g. 0.8.11, 0.9.1
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: bazelver
|
||||
attributes:
|
||||
label: Bazel version
|
||||
description:
|
||||
placeholder: e.g. 5.0, 5.1
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: xcodeversion
|
||||
attributes:
|
||||
label: XCode and Tulsi versions (if iOS)
|
||||
description:
|
||||
placeholder:
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: sdkndkversion
|
||||
attributes:
|
||||
label: Android SDK and NDK versions (if android)
|
||||
description:
|
||||
placeholder:
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: androidaar
|
||||
attributes:
|
||||
label: Android AAR (if android)
|
||||
options:
|
||||
- 'Yes'
|
||||
- 'No'
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: opencvversion
|
||||
attributes:
|
||||
label: OpenCV version (if running on desktop)
|
||||
description:
|
||||
placeholder:
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: what-happened
|
||||
attributes:
|
||||
label: Describe the problem
|
||||
description: Provide the exact sequence of commands / steps that you executed before running into the [problem](https://google.github.io/mediapipe/getting_started/getting_started.html)
|
||||
placeholder: Tell us what you see!
|
||||
value: "A bug happened!"
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: code-to-reproduce
|
||||
attributes:
|
||||
label: Complete Logs
|
||||
description: Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||
placeholder: Tell us what you see!
|
||||
value:
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,64 @@
|
||||
name: Feature Request Issues
|
||||
description: Use this template for raising a feature request. If this doesn’t look right, choose a different type.
|
||||
labels: 'type:feature'
|
||||
body:
|
||||
- type: markdown
|
||||
id: linkmodel
|
||||
attributes:
|
||||
value: Please make sure that this is a feature request.
|
||||
- type: input
|
||||
id: solution
|
||||
attributes:
|
||||
label: MediaPipe Solution (you are using)
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: pgmlang
|
||||
attributes:
|
||||
label: Programming language
|
||||
placeholder: C++/typescript/Python/Objective C/Android Java
|
||||
validations:
|
||||
required: false
|
||||
- type: dropdown
|
||||
id: willingcon
|
||||
attributes:
|
||||
label: Are you willing to contribute it
|
||||
options:
|
||||
- 'Yes'
|
||||
- 'No'
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: behaviour
|
||||
attributes:
|
||||
label: Describe the feature and the current behaviour/state
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: api_change
|
||||
attributes:
|
||||
label: Will this change the current API? How?
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: benifit
|
||||
attributes:
|
||||
label: Who will benefit with this feature?
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: use_case
|
||||
attributes:
|
||||
label: Please specify the use cases for this feature
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: info_other
|
||||
attributes:
|
||||
label: Any Other info
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,73 @@
|
||||
name: Model Maker Issues
|
||||
description: Use this template for assistance with using MediaPipe Model Maker (developers.google.com/mediapipe/solutions) to create custom on-device ML solutions.
|
||||
labels: 'type:modelmaker'
|
||||
body:
|
||||
- type: markdown
|
||||
id: linkmodel
|
||||
attributes:
|
||||
value: Please make sure that this is a [Model Maker](https://developers.google.com/mediapipe/solutions) issue
|
||||
- type: dropdown
|
||||
id: customcode_model
|
||||
attributes:
|
||||
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||
options:
|
||||
- 'Yes'
|
||||
- 'No'
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: os_model
|
||||
attributes:
|
||||
label: OS Platform and Distribution
|
||||
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: pythonver
|
||||
attributes:
|
||||
label: Python Version
|
||||
placeholder: e.g. 3.7, 3.8
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: modelmakerver
|
||||
attributes:
|
||||
label: MediaPipe Model Maker version
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: taskname
|
||||
attributes:
|
||||
label: Task name (e.g. Image classification, Gesture recognition etc.)
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: current_model
|
||||
attributes:
|
||||
label: Describe the actual behavior
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected_model
|
||||
attributes:
|
||||
label: Describe the expected behaviour
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: what-happened_model
|
||||
attributes:
|
||||
label: Standalone code/steps you may have used to try to get what you need
|
||||
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: other_info
|
||||
attributes:
|
||||
label: Other info / Complete Logs
|
||||
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,63 @@
|
||||
name: Studio Issues
|
||||
description: Use this template for assistance with the MediaPipe Studio application. If this doesn’t look right, choose a different type.
|
||||
labels: 'type:support'
|
||||
body:
|
||||
- type: markdown
|
||||
id: linkmodel
|
||||
attributes:
|
||||
value: Please make sure that this is a MediaPipe Studio issue.
|
||||
- type: input
|
||||
id: os_model
|
||||
attributes:
|
||||
label: OS Platform and Distribution
|
||||
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: browserver
|
||||
attributes:
|
||||
label: Browser and Version
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: hardware
|
||||
attributes:
|
||||
label: Any microphone or camera hardware
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: url
|
||||
attributes:
|
||||
label: URL that shows the problem
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: current_model
|
||||
attributes:
|
||||
label: Describe the actual behavior
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: expected_model
|
||||
attributes:
|
||||
label: Describe the expected behaviour
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: what-happened_model
|
||||
attributes:
|
||||
label: Standalone code/steps you may have used to try to get what you need
|
||||
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
- type: textarea
|
||||
id: other_info
|
||||
attributes:
|
||||
label: Other info / Complete Logs
|
||||
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,72 @@
|
||||
name: Task Issue
|
||||
description: Use this template for assistance with using MediaPipe Tasks (developers.google.com/mediapipe/solutions) to deploy on-device ML solutions (e.g. gesture recognition etc.) on supported platforms
|
||||
labels: 'type:task'
|
||||
body:
|
||||
- type: markdown
|
||||
id: linkmodel
|
||||
attributes:
|
||||
value: Please make sure that this is a [Tasks](https://developers.google.com/mediapipe/solutions) issue.
|
||||
- type: dropdown
|
||||
id: customcode_model
|
||||
attributes:
|
||||
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||
options:
|
||||
- 'Yes'
|
||||
- 'No'
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: os_model
|
||||
attributes:
|
||||
label: OS Platform and Distribution
|
||||
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: task-sdk-version
|
||||
attributes:
|
||||
label: MediaPipe Tasks SDK version
|
||||
validations:
|
||||
required: false
|
||||
- type: input
|
||||
id: taskname
|
||||
attributes:
|
||||
label: Task name (e.g. Image classification, Gesture recognition etc.)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: programminglang
|
||||
attributes:
|
||||
label: Programming Language and version (e.g. C++, Python, Java)
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: current_model
|
||||
attributes:
|
||||
label: Describe the actual behavior
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected_model
|
||||
attributes:
|
||||
label: Describe the expected behaviour
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: what-happened_model
|
||||
attributes:
|
||||
label: Standalone code/steps you may have used to try to get what you need
|
||||
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: other_info
|
||||
attributes:
|
||||
label: Other info / Complete Logs
|
||||
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||
render: shell
|
||||
validations:
|
||||
required: false
|
||||
@@ -19,6 +19,17 @@ ML solutions for live and streaming media.
|
||||
 | 
|
||||
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
|
||||
|
||||
----
|
||||
|
||||
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||
as the primary developer documentation
|
||||
site for MediaPipe starting April 3, 2023.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## ML solutions in MediaPipe
|
||||
|
||||
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
|
||||
|
||||
@@ -10,33 +10,31 @@ bind(
|
||||
|
||||
http_archive(
|
||||
name = "bazel_skylib",
|
||||
type = "tar.gz",
|
||||
sha256 = "74d544d96f4a5bb630d465ca8bbcfe231e3594e5aae57e1edbf17a6eb3ca2506",
|
||||
urls = [
|
||||
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz",
|
||||
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz",
|
||||
"https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
|
||||
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
|
||||
],
|
||||
sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
|
||||
)
|
||||
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
|
||||
bazel_skylib_workspace()
|
||||
load("@bazel_skylib//lib:versions.bzl", "versions")
|
||||
versions.check(minimum_bazel_version = "3.7.2")
|
||||
|
||||
# ABSL cpp library lts_2021_03_24, patch 2.
|
||||
# ABSL cpp library lts_2023_01_25.
|
||||
http_archive(
|
||||
name = "com_google_absl",
|
||||
urls = [
|
||||
"https://github.com/abseil/abseil-cpp/archive/refs/tags/20220623.1.tar.gz",
|
||||
"https://github.com/abseil/abseil-cpp/archive/refs/tags/20230125.0.tar.gz",
|
||||
],
|
||||
# Remove after https://github.com/abseil/abseil-cpp/issues/326 is solved.
|
||||
patches = [
|
||||
"@//third_party:com_google_absl_f863b622fe13612433fdf43f76547d5edda0c93001.diff"
|
||||
"@//third_party:com_google_absl_windows_patch.diff"
|
||||
],
|
||||
patch_args = [
|
||||
"-p1",
|
||||
],
|
||||
strip_prefix = "abseil-cpp-20220623.1",
|
||||
sha256 = "91ac87d30cc6d79f9ab974c51874a704de9c2647c40f6932597329a282217ba8"
|
||||
strip_prefix = "abseil-cpp-20230125.0",
|
||||
sha256 = "3ea49a7d97421b88a8c48a0de16c16048e17725c7ec0f1d3ea2683a2a75adc21"
|
||||
)
|
||||
|
||||
http_archive(
|
||||
@@ -148,13 +146,14 @@ http_archive(
|
||||
],
|
||||
)
|
||||
|
||||
load("//third_party/flatbuffers:workspace.bzl", flatbuffers = "repo")
|
||||
load("@//third_party/flatbuffers:workspace.bzl", flatbuffers = "repo")
|
||||
flatbuffers()
|
||||
|
||||
http_archive(
|
||||
name = "com_google_audio_tools",
|
||||
strip_prefix = "multichannel-audio-tools-master",
|
||||
urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
|
||||
strip_prefix = "multichannel-audio-tools-1f6b1319f13282eda6ff1317be13de67f4723860",
|
||||
urls = ["https://github.com/google/multichannel-audio-tools/archive/1f6b1319f13282eda6ff1317be13de67f4723860.zip"],
|
||||
sha256 = "fe346e1aee4f5069c4cbccb88706a9a2b2b4cf98aeb91ec1319be77e07dd7435",
|
||||
repo_mapping = {"@com_github_glog_glog" : "@com_github_glog_glog_no_gflags"},
|
||||
)
|
||||
|
||||
@@ -174,7 +173,7 @@ http_archive(
|
||||
"https://github.com/google/sentencepiece/archive/1.0.0.zip",
|
||||
],
|
||||
patches = [
|
||||
"//third_party:com_google_sentencepiece_no_gflag_no_gtest.diff",
|
||||
"@//third_party:com_google_sentencepiece_no_gflag_no_gtest.diff",
|
||||
],
|
||||
patch_args = ["-p1"],
|
||||
repo_mapping = {"@com_google_glog" : "@com_github_glog_glog_no_gflags"},
|
||||
@@ -188,8 +187,8 @@ http_archive(
|
||||
"https://github.com/tensorflow/text/archive/v2.2.0.zip",
|
||||
],
|
||||
patches = [
|
||||
"//third_party:tensorflow_text_remove_tf_deps.diff",
|
||||
"//third_party:tensorflow_text_a0f49e63.diff",
|
||||
"@//third_party:tensorflow_text_remove_tf_deps.diff",
|
||||
"@//third_party:tensorflow_text_a0f49e63.diff",
|
||||
],
|
||||
patch_args = ["-p1"],
|
||||
repo_mapping = {"@com_google_re2": "@com_googlesource_code_re2"},
|
||||
@@ -324,7 +323,7 @@ http_archive(
|
||||
# that the target @zlib//:mini_zlib is available
|
||||
http_archive(
|
||||
name = "zlib",
|
||||
build_file = "//third_party:zlib.BUILD",
|
||||
build_file = "@//third_party:zlib.BUILD",
|
||||
sha256 = "c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1",
|
||||
strip_prefix = "zlib-1.2.11",
|
||||
urls = [
|
||||
@@ -456,9 +455,9 @@ http_archive(
|
||||
)
|
||||
|
||||
# TensorFlow repo should always go after the other external dependencies.
|
||||
# TF on 2022-08-10.
|
||||
_TENSORFLOW_GIT_COMMIT = "af1d5bc4fbb66d9e6cc1cf89503014a99233583b"
|
||||
_TENSORFLOW_SHA256 = "f85a5443264fc58a12d136ca6a30774b5bc25ceaf7d114d97f252351b3c3a2cb"
|
||||
# TF on 2023-03-08.
|
||||
_TENSORFLOW_GIT_COMMIT = "24f7ee636d62e1f8d8330357f8bbd65956dfb84d"
|
||||
_TENSORFLOW_SHA256 = "7f8a96dd99215c0cdc77230d3dbce43e60102b64a89203ad04aa09b0a187a4bd"
|
||||
http_archive(
|
||||
name = "org_tensorflow",
|
||||
urls = [
|
||||
@@ -512,8 +511,8 @@ load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "yarn_install
|
||||
node_repositories()
|
||||
yarn_install(
|
||||
name = "npm",
|
||||
package_json = "//:package.json",
|
||||
yarn_lock = "//:yarn.lock",
|
||||
package_json = "@//:package.json",
|
||||
yarn_lock = "@//:yarn.lock",
|
||||
)
|
||||
|
||||
# Protobuf for Node dependencies
|
||||
@@ -539,8 +538,48 @@ load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_
|
||||
rules_proto_dependencies()
|
||||
rules_proto_toolchains()
|
||||
|
||||
load("//third_party:external_files.bzl", "external_files")
|
||||
load("@//third_party:external_files.bzl", "external_files")
|
||||
external_files()
|
||||
|
||||
load("//third_party:wasm_files.bzl", "wasm_files")
|
||||
load("@//third_party:wasm_files.bzl", "wasm_files")
|
||||
wasm_files()
|
||||
|
||||
# Halide
|
||||
|
||||
new_local_repository(
|
||||
name = "halide",
|
||||
build_file = "@//third_party/halide:BUILD.bazel",
|
||||
path = "third_party/halide"
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "linux_halide",
|
||||
sha256 = "f62b2914823d6e33d18693f5b74484f274523bf5402ce51988e24393d123b375",
|
||||
strip_prefix = "Halide-15.0.0-x86-64-linux",
|
||||
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-x86-64-linux-d7651f4b32f9dbd764f243134001f7554378d62d.tar.gz"],
|
||||
build_file = "@//third_party:halide.BUILD",
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "macos_x86_64_halide",
|
||||
sha256 = "3d832aed942080ea89aa832462c68fbb906f3055c440b7b6d35093d7c52f6aab",
|
||||
strip_prefix = "Halide-15.0.0-x86-64-osx",
|
||||
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-x86-64-osx-d7651f4b32f9dbd764f243134001f7554378d62d.tar.gz"],
|
||||
build_file = "@//third_party:halide.BUILD",
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "macos_arm_64_halide",
|
||||
sha256 = "b1fad3c9810122b187303d7031d9e35fb43761f345d18cc4492c00ed5877f641",
|
||||
strip_prefix = "Halide-15.0.0-arm-64-osx",
|
||||
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-arm-64-osx-d7651f4b32f9dbd764f243134001f7554378d62d.tar.gz"],
|
||||
build_file = "@//third_party:halide.BUILD",
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "windows_halide",
|
||||
sha256 = "5acf6fe161dd375856a2b43f4bb0a32815ba958b0585ee312c44e008aa7b0b64",
|
||||
strip_prefix = "Halide-15.0.0-x86-64-windows",
|
||||
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-x86-64-windows-d7651f4b32f9dbd764f243134001f7554378d62d.zip"],
|
||||
build_file = "@//third_party:halide.BUILD",
|
||||
)
|
||||
|
||||
+1
-3
@@ -4,12 +4,10 @@ py_binary(
|
||||
name = "build_py_api_docs",
|
||||
srcs = ["build_py_api_docs.py"],
|
||||
deps = [
|
||||
"//mediapipe",
|
||||
"//third_party/py/absl:app",
|
||||
"//third_party/py/absl/flags",
|
||||
"//third_party/py/tensorflow_docs",
|
||||
"//third_party/py/mediapipe",
|
||||
"//third_party/py/tensorflow_docs/api_generator:generate_lib",
|
||||
"//third_party/py/tensorflow_docs/api_generator:public_api",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"/>
|
||||
<meta http-equiv="refresh" content="0;url={{ page.target }}"/>
|
||||
<link rel="canonical" href="{{ page.target }}"/>
|
||||
<title>Redirecting</title>
|
||||
</head>
|
||||
<body>
|
||||
<p>This page now lives on https://developers.google.com/mediapipe/. If you aren't automatically
|
||||
redirected, follow this
|
||||
<a href="{{ page.target }}">link</a>.</p>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,697 @@
|
||||
---
|
||||
layout: default
|
||||
title: Building Graphs in C++
|
||||
parent: Graphs
|
||||
nav_order: 1
|
||||
---
|
||||
|
||||
# Building Graphs in C++
|
||||
{: .no_toc }
|
||||
|
||||
1. TOC
|
||||
{:toc}
|
||||
---
|
||||
|
||||
C++ graph builder is a powerful tool for:
|
||||
|
||||
* Building complex graphs
|
||||
* Parametrizing graphs (e.g. setting a delegate on `InferenceCalculator`,
|
||||
enabling/disabling parts of the graph)
|
||||
* Deduplicating graphs (e.g. instead of CPU and GPU dedicated graphs in pbtxt
|
||||
you can have a single code that constructs required graphs, sharing as much
|
||||
as possible)
|
||||
* Supporting optional graph inputs/outputs
|
||||
* Customizing graphs per platform
|
||||
|
||||
## Basic Usage
|
||||
|
||||
Let's see how C++ graph builder can be used for a simple graph:
|
||||
|
||||
```proto
|
||||
# Graph inputs.
|
||||
input_stream: "input_tensors"
|
||||
input_side_packet: "model"
|
||||
|
||||
# Graph outputs.
|
||||
output_stream: "output_tensors"
|
||||
|
||||
node {
|
||||
calculator: "InferenceCalculator"
|
||||
input_stream: "TENSORS:input_tensors"
|
||||
input_side_packet: "MODEL:model"
|
||||
output_stream: "TENSORS:output_tensors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.InferenceCalculatorOptions] {
|
||||
# Requesting GPU delegate.
|
||||
delegate { gpu {} }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Function to build the above `CalculatorGraphConfig` may look like:
|
||||
|
||||
```c++
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Graph inputs.
|
||||
Stream<std::vector<Tensor>> input_tensors =
|
||||
graph.In(0).SetName("input_tensors").Cast<std::vector<Tensor>>();
|
||||
SidePacket<TfLiteModelPtr> model =
|
||||
graph.SideIn(0).SetName("model").Cast<TfLiteModelPtr>();
|
||||
|
||||
auto& inference_node = graph.AddNode("InferenceCalculator");
|
||||
auto& inference_opts =
|
||||
inference_node.GetOptions<InferenceCalculatorOptions>();
|
||||
// Requesting GPU delegate.
|
||||
inference_opts.mutable_delegate()->mutable_gpu();
|
||||
input_tensors.ConnectTo(inference_node.In("TENSORS"));
|
||||
model.ConnectTo(inference_node.SideIn("MODEL"));
|
||||
Stream<std::vector<Tensor>> output_tensors =
|
||||
inference_node.Out("TENSORS").Cast<std::vector<Tensor>>();
|
||||
|
||||
// Graph outputs.
|
||||
output_tensors.SetName("output_tensors").ConnectTo(graph.Out(0));
|
||||
|
||||
// Get `CalculatorGraphConfig` to pass it into `CalculatorGraph`
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
Short summary:
|
||||
|
||||
* Use `Graph::In/SideIn` to get graph inputs as `Stream/SidePacket`
|
||||
* Use `Node::Out/SideOut` to get node outputs as `Stream/SidePacket`
|
||||
* Use `Stream/SidePacket::ConnectTo` to connect streams and side packets to
|
||||
node inputs (`Node::In/SideIn`) and graph outputs (`Graph::Out/SideOut`)
|
||||
* There's a "shortcut" operator `>>` that you can use instead of
|
||||
`ConnectTo` function (E.g. `x >> node.In("IN")`).
|
||||
* `Stream/SidePacket::Cast` is used to cast stream or side packet of `AnyType`
|
||||
(E.g. `Stream<AnyType> in = graph.In(0);`) to a particular type
|
||||
* Using actual types instead of `AnyType` sets you on a better path for
|
||||
unleashing graph builder capabilities and improving your graphs
|
||||
readability.
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Utility Functions
|
||||
|
||||
Let's extract inference construction code into a dedicated utility function to
|
||||
help for readability and code reuse:
|
||||
|
||||
```c++
|
||||
// Updates graph to run inference.
|
||||
Stream<std::vector<Tensor>> RunInference(
|
||||
Stream<std::vector<Tensor>> tensors, SidePacket<TfLiteModelPtr> model,
|
||||
const InferenceCalculatorOptions::Delegate& delegate, Graph& graph) {
|
||||
auto& inference_node = graph.AddNode("InferenceCalculator");
|
||||
auto& inference_opts =
|
||||
inference_node.GetOptions<InferenceCalculatorOptions>();
|
||||
*inference_opts.mutable_delegate() = delegate;
|
||||
tensors.ConnectTo(inference_node.In("TENSORS"));
|
||||
model.ConnectTo(inference_node.SideIn("MODEL"));
|
||||
return inference_node.Out("TENSORS").Cast<std::vector<Tensor>>();
|
||||
}
|
||||
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Graph inputs.
|
||||
Stream<std::vector<Tensor>> input_tensors =
|
||||
graph.In(0).SetName("input_tensors").Cast<std::vector<Tensor>>();
|
||||
SidePacket<TfLiteModelPtr> model =
|
||||
graph.SideIn(0).SetName("model").Cast<TfLiteModelPtr>();
|
||||
|
||||
InferenceCalculatorOptions::Delegate delegate;
|
||||
delegate.mutable_gpu();
|
||||
Stream<std::vector<Tensor>> output_tensors =
|
||||
RunInference(input_tensors, model, delegate, graph);
|
||||
|
||||
// Graph outputs.
|
||||
output_tensors.SetName("output_tensors").ConnectTo(graph.Out(0));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
As a result, `RunInference` provides a clear interface stating what are the
|
||||
inputs/outputs and their types.
|
||||
|
||||
It can be easily reused, e.g. it's only a few lines if you want to run an extra
|
||||
model inference:
|
||||
|
||||
```c++
|
||||
// Run first inference.
|
||||
Stream<std::vector<Tensor>> output_tensors =
|
||||
RunInference(input_tensors, model, delegate, graph);
|
||||
// Run second inference on the output of the first one.
|
||||
Stream<std::vector<Tensor>> extra_output_tensors =
|
||||
RunInference(output_tensors, extra_model, delegate, graph);
|
||||
```
|
||||
|
||||
And you don't need to duplicate names and tags (`InferenceCalculator`,
|
||||
`TENSORS`, `MODEL`) or introduce dedicated constants here and there - those
|
||||
details are localized to `RunInference` function.
|
||||
|
||||
Tip: extracting `RunInference` and similar functions to dedicated modules (e.g.
|
||||
inference.h/cc which depends on the inference calculator) enables reuse in
|
||||
graphs construction code and helps automatically pull in calculator dependencies
|
||||
(e.g. no need to manually add `:inference_calculator` dep, just let your IDE
|
||||
include `inference.h` and build cleaner pull in corresponding dependency).
|
||||
|
||||
### Utility Classes
|
||||
|
||||
And surely, it's not only about functions, in some cases it's beneficial to
|
||||
introduce utility classes which can help making your graph construction code
|
||||
more readable and less error prone.
|
||||
|
||||
MediaPipe offers `PassThroughCalculator` calculator, which is simply passing
|
||||
through its inputs:
|
||||
|
||||
```
|
||||
input_stream: "float_value"
|
||||
input_stream: "int_value"
|
||||
input_stream: "bool_value"
|
||||
|
||||
output_stream: "passed_float_value"
|
||||
output_stream: "passed_int_value"
|
||||
output_stream: "passed_bool_value"
|
||||
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "float_value"
|
||||
input_stream: "int_value"
|
||||
input_stream: "bool_value"
|
||||
# The order must be the same as for inputs (or you can use explicit indexes)
|
||||
output_stream: "passed_float_value"
|
||||
output_stream: "passed_int_value"
|
||||
output_stream: "passed_bool_value"
|
||||
}
|
||||
```
|
||||
|
||||
Let's see the straightforward C++ construction code to create the above graph:
|
||||
|
||||
```c++
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Graph inputs.
|
||||
Stream<float> float_value = graph.In(0).SetName("float_value").Cast<float>();
|
||||
Stream<int> int_value = graph.In(1).SetName("int_value").Cast<int>();
|
||||
Stream<bool> bool_value = graph.In(2).SetName("bool_value").Cast<bool>();
|
||||
|
||||
auto& pass_node = graph.AddNode("PassThroughCalculator");
|
||||
float_value.ConnectTo(pass_node.In("")[0]);
|
||||
int_value.ConnectTo(pass_node.In("")[1]);
|
||||
bool_value.ConnectTo(pass_node.In("")[2]);
|
||||
Stream<float> passed_float_value = pass_node.Out("")[0].Cast<float>();
|
||||
Stream<int> passed_int_value = pass_node.Out("")[1].Cast<int>();
|
||||
Stream<bool> passed_bool_value = pass_node.Out("")[2].Cast<bool>();
|
||||
|
||||
// Graph outputs.
|
||||
passed_float_value.SetName("passed_float_value").ConnectTo(graph.Out(0));
|
||||
passed_int_value.SetName("passed_int_value").ConnectTo(graph.Out(1));
|
||||
passed_bool_value.SetName("passed_bool_value").ConnectTo(graph.Out(2));
|
||||
|
||||
// Get `CalculatorGraphConfig` to pass it into `CalculatorGraph`
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
While `pbtxt` representation maybe error prone (when we have many inputs to pass
|
||||
through), C++ code looks even worse: repeated empty tags and `Cast` calls. Let's
|
||||
see how we can do better by introducing a `PassThroughNodeBuilder`:
|
||||
|
||||
```c++
|
||||
class PassThroughNodeBuilder {
|
||||
public:
|
||||
explicit PassThroughNodeBuilder(Graph& graph)
|
||||
: node_(graph.AddNode("PassThroughCalculator")) {}
|
||||
|
||||
template <typename T>
|
||||
Stream<T> PassThrough(Stream<T> stream) {
|
||||
stream.ConnectTo(node_.In(index_));
|
||||
return node_.Out(index_++).Cast<T>();
|
||||
}
|
||||
|
||||
private:
|
||||
int index_ = 0;
|
||||
GenericNode& node_;
|
||||
};
|
||||
```
|
||||
|
||||
And now graph construction code can look like:
|
||||
|
||||
```c++
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Graph inputs.
|
||||
Stream<float> float_value = graph.In(0).SetName("float_value").Cast<float>();
|
||||
Stream<int> int_value = graph.In(1).SetName("int_value").Cast<int>();
|
||||
Stream<bool> bool_value = graph.In(2).SetName("bool_value").Cast<bool>();
|
||||
|
||||
PassThroughNodeBuilder pass_node_builder(graph);
|
||||
Stream<float> passed_float_value = pass_node_builder.PassThrough(float_value);
|
||||
Stream<int> passed_int_value = pass_node_builder.PassThrough(int_value);
|
||||
Stream<bool> passed_bool_value = pass_node_builder.PassThrough(bool_value);
|
||||
|
||||
// Graph outputs.
|
||||
passed_float_value.SetName("passed_float_value").ConnectTo(graph.Out(0));
|
||||
passed_int_value.SetName("passed_int_value").ConnectTo(graph.Out(1));
|
||||
passed_bool_value.SetName("passed_bool_value").ConnectTo(graph.Out(2));
|
||||
|
||||
// Get `CalculatorGraphConfig` to pass it into `CalculatorGraph`
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
Now you can't have incorrect order or index in your pass through construction
|
||||
code and save some typing by guessing the type for `Cast` from the `PassThrough`
|
||||
input.
|
||||
|
||||
Tip: the same as for the `RunInference` function, extracting
|
||||
`PassThroughNodeBuilder` and similar utility classes into dedicated modules
|
||||
enables reuse in graph construction code and helps to automatically pull in the
|
||||
corresponding calculator dependencies.
|
||||
|
||||
## Dos and Don'ts
|
||||
|
||||
### Define graph inputs at the very beginning if possible
|
||||
|
||||
```c++ {.bad}
|
||||
Stream<D> RunSomething(Stream<A> a, Stream<B> b, Graph& graph) {
|
||||
Stream<C> c = graph.In(2).SetName("c").Cast<C>(); // Bad.
|
||||
// ...
|
||||
}
|
||||
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
Stream<A> a = graph.In(0).SetName("a").Cast<A>();
|
||||
// 10/100/N lines of code.
|
||||
Stream<B> b = graph.In(1).SetName("b").Cast<B>() // Bad.
|
||||
Stream<D> d = RunSomething(a, b, graph);
|
||||
// ...
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
|
||||
```
|
||||
|
||||
In the above code:
|
||||
|
||||
* It can be hard to guess how many inputs you have in the graph.
|
||||
* Can be error prone overall and hard to maintain in future (e.g. is it a
|
||||
correct index? name? what if some inputs are removed or made optional?
|
||||
etc.).
|
||||
* `RunSomething` reuse is limited because other graphs may have different
|
||||
inputs
|
||||
|
||||
Instead, define your graph inputs at the very beginning of your graph builder:
|
||||
|
||||
```c++ {.good}
|
||||
Stream<D> RunSomething(Stream<A> a, Stream<B> b, Stream<C> c, Graph& graph) {
|
||||
// ...
|
||||
}
|
||||
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).SetName("a").Cast<A>();
|
||||
Stream<B> b = graph.In(1).SetName("b").Cast<B>();
|
||||
Stream<C> c = graph.In(2).SetName("c").Cast<C>();
|
||||
|
||||
// 10/100/N lines of code.
|
||||
Stream<D> d = RunSomething(a, b, c, graph);
|
||||
// ...
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
Use `std::optional` if you have an input stream or side packet that is not
|
||||
always defined and put it at the very beginning:
|
||||
|
||||
```c++ {.good}
|
||||
std::optional<Stream<A>> a;
|
||||
if (needs_a) {
|
||||
a = graph.In(0).SetName(a).Cast<A>();
|
||||
}
|
||||
```
|
||||
|
||||
Note: of course, there can be exceptions - for example, there can be a use case
|
||||
where calling `RunSomething1(..., graph)`, ..., `RunSomethingN(..., graph)` is
|
||||
**intended to add new inputs**, so afterwards you can iterate over them and feed
|
||||
only added inputs into the graph. However, in any case, try to make it easy for
|
||||
readers to find out what graph inputs it has or may have.
|
||||
|
||||
### Define graph outputs at the very end
|
||||
|
||||
```c++ {.bad}
|
||||
void RunSomething(Stream<Input> input, Graph& graph) {
|
||||
// ...
|
||||
node.Out("OUTPUT_F")
|
||||
.SetName("output_f").ConnectTo(graph.Out(2)); // Bad.
|
||||
}
|
||||
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// 10/100/N lines of code.
|
||||
node.Out("OUTPUT_D")
|
||||
.SetName("output_d").ConnectTo(graph.Out(0)); // Bad.
|
||||
// 10/100/N lines of code.
|
||||
node.Out("OUTPUT_E")
|
||||
.SetName("output_e").ConnectTo(graph.Out(1)); // Bad.
|
||||
// 10/100/N lines of code.
|
||||
RunSomething(input, graph);
|
||||
// ...
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
In the above code:
|
||||
|
||||
* It can be hard to guess how many outputs you have in the graph.
|
||||
* Can be error prone overall and hard to maintain in future (e.g. is it a
|
||||
correct index? name? what if some outpus are removed or made optional?
|
||||
etc.).
|
||||
* `RunSomething` reuse is limited as other graphs may have different outputs
|
||||
|
||||
Instead, define your graph outputs at the very end of your graph builder:
|
||||
|
||||
```c++ {.good}
|
||||
Stream<F> RunSomething(Stream<Input> input, Graph& graph) {
|
||||
// ...
|
||||
return node.Out("OUTPUT_F").Cast<F>();
|
||||
}
|
||||
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// 10/100/N lines of code.
|
||||
Stream<D> d = node.Out("OUTPUT_D").Cast<D>();
|
||||
// 10/100/N lines of code.
|
||||
Stream<E> e = node.Out("OUTPUT_E").Cast<E>();
|
||||
// 10/100/N lines of code.
|
||||
Stream<F> f = RunSomething(input, graph);
|
||||
// ...
|
||||
|
||||
// Outputs.
|
||||
d.SetName("output_d").ConnectTo(graph.Out(0));
|
||||
e.SetName("output_e").ConnectTo(graph.Out(1));
|
||||
f.SetName("output_f").ConnectTo(graph.Out(2));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
### Keep nodes decoupled from each other
|
||||
|
||||
In MediaPipe, packet streams and side packets are as meaningful as processing
|
||||
nodes. And any node input requirements and output products are expressed clearly
|
||||
and independently in terms of the streams and side packets it consumes and
|
||||
produces.
|
||||
|
||||
```c++ {.bad}
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).Cast<A>();
|
||||
|
||||
auto& node1 = graph.AddNode("Calculator1");
|
||||
a.ConnectTo(node1.In("INPUT"));
|
||||
|
||||
auto& node2 = graph.AddNode("Calculator2");
|
||||
node1.Out("OUTPUT").ConnectTo(node2.In("INPUT")); // Bad.
|
||||
|
||||
auto& node3 = graph.AddNode("Calculator3");
|
||||
node1.Out("OUTPUT").ConnectTo(node3.In("INPUT_B")); // Bad.
|
||||
node2.Out("OUTPUT").ConnectTo(node3.In("INPUT_C")); // Bad.
|
||||
|
||||
auto& node4 = graph.AddNode("Calculator4");
|
||||
node1.Out("OUTPUT").ConnectTo(node4.In("INPUT_B")); // Bad.
|
||||
node2.Out("OUTPUT").ConnectTo(node4.In("INPUT_C")); // Bad.
|
||||
node3.Out("OUTPUT").ConnectTo(node4.In("INPUT_D")); // Bad.
|
||||
|
||||
// Outputs.
|
||||
node1.Out("OUTPUT").SetName("b").ConnectTo(graph.Out(0)); // Bad.
|
||||
node2.Out("OUTPUT").SetName("c").ConnectTo(graph.Out(1)); // Bad.
|
||||
node3.Out("OUTPUT").SetName("d").ConnectTo(graph.Out(2)); // Bad.
|
||||
node4.Out("OUTPUT").SetName("e").ConnectTo(graph.Out(3)); // Bad.
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
In the above code:
|
||||
|
||||
* Nodes are coupled to each other, e.g. `node4` knows where its inputs are
|
||||
coming from (`node1`, `node2`, `node3`) and it complicates refactoring,
|
||||
maintenance and code reuse
|
||||
* Such usage pattern is a downgrade from proto representation, where nodes
|
||||
are decoupled by default.
|
||||
* `node#.Out("OUTPUT")` calls are duplicated and readability suffers as you
|
||||
could use cleaner names instead and also provide an actual type.
|
||||
|
||||
So, to fix the above issues you can write the following graph construction code:
|
||||
|
||||
```c++ {.good}
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).Cast<A>();
|
||||
|
||||
// `node1` usage is limited to 3 lines below.
|
||||
auto& node1 = graph.AddNode("Calculator1");
|
||||
a.ConnectTo(node1.In("INPUT"));
|
||||
Stream<B> b = node1.Out("OUTPUT").Cast<B>();
|
||||
|
||||
// `node2` usage is limited to 3 lines below.
|
||||
auto& node2 = graph.AddNode("Calculator2");
|
||||
b.ConnectTo(node2.In("INPUT"));
|
||||
Stream<C> c = node2.Out("OUTPUT").Cast<C>();
|
||||
|
||||
// `node3` usage is limited to 4 lines below.
|
||||
auto& node3 = graph.AddNode("Calculator3");
|
||||
b.ConnectTo(node3.In("INPUT_B"));
|
||||
c.ConnectTo(node3.In("INPUT_C"));
|
||||
Stream<D> d = node3.Out("OUTPUT").Cast<D>();
|
||||
|
||||
// `node4` usage is limited to 5 lines below.
|
||||
auto& node4 = graph.AddNode("Calculator4");
|
||||
b.ConnectTo(node4.In("INPUT_B"));
|
||||
c.ConnectTo(node4.In("INPUT_C"));
|
||||
d.ConnectTo(node4.In("INPUT_D"));
|
||||
Stream<E> e = node4.Out("OUTPUT").Cast<E>();
|
||||
|
||||
// Outputs.
|
||||
b.SetName("b").ConnectTo(graph.Out(0));
|
||||
c.SetName("c").ConnectTo(graph.Out(1));
|
||||
d.SetName("d").ConnectTo(graph.Out(2));
|
||||
e.SetName("e").ConnectTo(graph.Out(3));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
Now, if needed, you can easily remove `node1` and make `b` a graph input and no
|
||||
updates are needed to `node2`, `node3`, `node4` (same as in proto representation
|
||||
by the way), because they are decoupled from each other.
|
||||
|
||||
Overall, the above code replicates the proto graph more closely:
|
||||
|
||||
```proto
|
||||
input_stream: "a"
|
||||
|
||||
node {
|
||||
calculator: "Calculator1"
|
||||
input_stream: "INPUT:a"
|
||||
output_stream: "OUTPUT:b"
|
||||
}
|
||||
|
||||
node {
|
||||
calculator: "Calculator2"
|
||||
input_stream: "INPUT:b"
|
||||
output_stream: "OUTPUT:C"
|
||||
}
|
||||
|
||||
node {
|
||||
calculator: "Calculator3"
|
||||
input_stream: "INPUT_B:b"
|
||||
input_stream: "INPUT_C:c"
|
||||
output_stream: "OUTPUT:d"
|
||||
}
|
||||
|
||||
node {
|
||||
calculator: "Calculator4"
|
||||
input_stream: "INPUT_B:b"
|
||||
input_stream: "INPUT_C:c"
|
||||
input_stream: "INPUT_D:d"
|
||||
output_stream: "OUTPUT:e"
|
||||
}
|
||||
|
||||
output_stream: "b"
|
||||
output_stream: "c"
|
||||
output_stream: "d"
|
||||
output_stream: "e"
|
||||
```
|
||||
|
||||
On top of that, now you can extract utility functions for further reuse in other graphs:
|
||||
|
||||
```c++ {.good}
|
||||
Stream<B> RunCalculator1(Stream<A> a, Graph& graph) {
|
||||
auto& node = graph.AddNode("Calculator1");
|
||||
a.ConnectTo(node.In("INPUT"));
|
||||
return node.Out("OUTPUT").Cast<B>();
|
||||
}
|
||||
|
||||
Stream<C> RunCalculator2(Stream<B> b, Graph& graph) {
|
||||
auto& node = graph.AddNode("Calculator2");
|
||||
b.ConnectTo(node.In("INPUT"));
|
||||
return node.Out("OUTPUT").Cast<C>();
|
||||
}
|
||||
|
||||
Stream<D> RunCalculator3(Stream<B> b, Stream<C> c, Graph& graph) {
|
||||
auto& node = graph.AddNode("Calculator3");
|
||||
b.ConnectTo(node.In("INPUT_B"));
|
||||
c.ConnectTo(node.In("INPUT_C"));
|
||||
return node.Out("OUTPUT").Cast<D>();
|
||||
}
|
||||
|
||||
Stream<E> RunCalculator4(Stream<B> b, Stream<C> c, Stream<D> d, Graph& graph) {
|
||||
auto& node = graph.AddNode("Calculator4");
|
||||
b.ConnectTo(node.In("INPUT_B"));
|
||||
c.ConnectTo(node.In("INPUT_C"));
|
||||
d.ConnectTo(node.In("INPUT_D"));
|
||||
return node.Out("OUTPUT").Cast<E>();
|
||||
}
|
||||
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).Cast<A>();
|
||||
|
||||
Stream<B> b = RunCalculator1(a, graph);
|
||||
Stream<C> c = RunCalculator2(b, graph);
|
||||
Stream<D> d = RunCalculator3(b, c, graph);
|
||||
Stream<E> e = RunCalculator4(b, c, d, graph);
|
||||
|
||||
// Outputs.
|
||||
b.SetName("b").ConnectTo(graph.Out(0));
|
||||
c.SetName("c").ConnectTo(graph.Out(1));
|
||||
d.SetName("d").ConnectTo(graph.Out(2));
|
||||
e.SetName("e").ConnectTo(graph.Out(3));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
### Separate nodes for better readability
|
||||
|
||||
```c++ {.bad}
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).Cast<A>();
|
||||
auto& node1 = graph.AddNode("Calculator1");
|
||||
a.ConnectTo(node1.In("INPUT"));
|
||||
Stream<B> b = node1.Out("OUTPUT").Cast<B>();
|
||||
auto& node2 = graph.AddNode("Calculator2");
|
||||
b.ConnectTo(node2.In("INPUT"));
|
||||
Stream<C> c = node2.Out("OUTPUT").Cast<C>();
|
||||
auto& node3 = graph.AddNode("Calculator3");
|
||||
b.ConnectTo(node3.In("INPUT_B"));
|
||||
c.ConnectTo(node3.In("INPUT_C"));
|
||||
Stream<D> d = node3.Out("OUTPUT").Cast<D>();
|
||||
auto& node4 = graph.AddNode("Calculator4");
|
||||
b.ConnectTo(node4.In("INPUT_B"));
|
||||
c.ConnectTo(node4.In("INPUT_C"));
|
||||
d.ConnectTo(node4.In("INPUT_D"));
|
||||
Stream<E> e = node4.Out("OUTPUT").Cast<E>();
|
||||
// Outputs.
|
||||
b.SetName("b").ConnectTo(graph.Out(0));
|
||||
c.SetName("c").ConnectTo(graph.Out(1));
|
||||
d.SetName("d").ConnectTo(graph.Out(2));
|
||||
e.SetName("e").ConnectTo(graph.Out(3));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
In the above code, it can be hard to grasp the idea where each node begins and
|
||||
ends. To improve this and help your code readers, you can simply have blank
|
||||
lines before and after each node:
|
||||
|
||||
```c++ {.good}
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).Cast<A>();
|
||||
|
||||
auto& node1 = graph.AddNode("Calculator1");
|
||||
a.ConnectTo(node1.In("INPUT"));
|
||||
Stream<B> b = node1.Out("OUTPUT").Cast<B>();
|
||||
|
||||
auto& node2 = graph.AddNode("Calculator2");
|
||||
b.ConnectTo(node2.In("INPUT"));
|
||||
Stream<C> c = node2.Out("OUTPUT").Cast<C>();
|
||||
|
||||
auto& node3 = graph.AddNode("Calculator3");
|
||||
b.ConnectTo(node3.In("INPUT_B"));
|
||||
c.ConnectTo(node3.In("INPUT_C"));
|
||||
Stream<D> d = node3.Out("OUTPUT").Cast<D>();
|
||||
|
||||
auto& node4 = graph.AddNode("Calculator4");
|
||||
b.ConnectTo(node4.In("INPUT_B"));
|
||||
c.ConnectTo(node4.In("INPUT_C"));
|
||||
d.ConnectTo(node4.In("INPUT_D"));
|
||||
Stream<E> e = node4.Out("OUTPUT").Cast<E>();
|
||||
|
||||
// Outputs.
|
||||
b.SetName("b").ConnectTo(graph.Out(0));
|
||||
c.SetName("c").ConnectTo(graph.Out(1));
|
||||
d.SetName("d").ConnectTo(graph.Out(2));
|
||||
e.SetName("e").ConnectTo(graph.Out(3));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
|
||||
Also, the above representation matches `CalculatorGraphConfig` proto
|
||||
representation better.
|
||||
|
||||
If you extract nodes into utility functions, they are scoped within functions
|
||||
already and it's clear where they begin and end, so it's completely fine to
|
||||
have:
|
||||
|
||||
```c++ {.good}
|
||||
CalculatorGraphConfig BuildGraph() {
|
||||
Graph graph;
|
||||
|
||||
// Inputs.
|
||||
Stream<A> a = graph.In(0).Cast<A>();
|
||||
|
||||
Stream<B> b = RunCalculator1(a, graph);
|
||||
Stream<C> c = RunCalculator2(b, graph);
|
||||
Stream<D> d = RunCalculator3(b, c, graph);
|
||||
Stream<E> e = RunCalculator4(b, c, d, graph);
|
||||
|
||||
// Outputs.
|
||||
b.SetName("b").ConnectTo(graph.Out(0));
|
||||
c.SetName("c").ConnectTo(graph.Out(1));
|
||||
d.SetName("d").ConnectTo(graph.Out(2));
|
||||
e.SetName("e").ConnectTo(graph.Out(3));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/calculators
|
||||
title: Calculators
|
||||
parent: Framework Concepts
|
||||
nav_order: 1
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/overview
|
||||
title: Framework Concepts
|
||||
nav_order: 5
|
||||
has_children: true
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/gpu
|
||||
title: GPU
|
||||
parent: Framework Concepts
|
||||
nav_order: 5
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/graphs
|
||||
title: Graphs
|
||||
parent: Framework Concepts
|
||||
nav_order: 2
|
||||
@@ -12,31 +13,31 @@ nav_order: 2
|
||||
{:toc}
|
||||
---
|
||||
|
||||
## GraphConfig
|
||||
## Graph
|
||||
|
||||
A `GraphConfig` is a specification that describes the topology and functionality
|
||||
of a MediaPipe graph. In the specification, a node in the graph represents an
|
||||
instance of a particular calculator. All the necessary configurations of the
|
||||
node, such its type, inputs and outputs must be described in the specification.
|
||||
Description of the node can also include several optional fields, such as
|
||||
A `CalculatorGraphConfig` proto specifies the topology and functionality of a
|
||||
MediaPipe graph. Each `node` in the graph represents a particular calculator or
|
||||
subgraph, and specifies necessary configurations, such as registered
|
||||
calculator/subgraph type, inputs, outputs and optional fields, such as
|
||||
node-specific options, input policy and executor, discussed in
|
||||
[Synchronization](synchronization.md).
|
||||
|
||||
`GraphConfig` has several other fields to configure the global graph-level
|
||||
settings, eg, graph executor configs, number of threads, and maximum queue size
|
||||
`CalculatorGraphConfig` has several other fields to configure global graph-level
|
||||
settings, e.g. graph executor configs, number of threads, and maximum queue size
|
||||
of input streams. Several graph-level settings are useful for tuning the
|
||||
performance of the graph on different platforms (eg, desktop v.s. mobile). For
|
||||
performance of the graph on different platforms (e.g., desktop v.s. mobile). For
|
||||
instance, on mobile, attaching a heavy model-inference calculator to a separate
|
||||
executor can improve the performance of a real-time application since this
|
||||
enables thread locality.
|
||||
|
||||
Below is a trivial `GraphConfig` example where we have series of passthrough
|
||||
calculators :
|
||||
Below is a trivial `CalculatorGraphConfig` example where we have series of
|
||||
passthrough calculators :
|
||||
|
||||
```proto
|
||||
# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
|
||||
# passthrough calculators.
|
||||
input_stream: "in"
|
||||
output_stream: "out"
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "in"
|
||||
@@ -55,17 +56,46 @@ node {
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "out3"
|
||||
output_stream: "out4"
|
||||
output_stream: "out"
|
||||
}
|
||||
```
|
||||
|
||||
MediaPipe offers an alternative `C++` representation for complex graphs (e.g. ML pipelines, handling model metadata, optional nodes, etc.). The above graph may look like:
|
||||
|
||||
```c++
|
||||
CalculatorGraphConfig BuildGraphConfig() {
|
||||
Graph graph;
|
||||
|
||||
// Graph inputs
|
||||
Stream<AnyType> in = graph.In(0).SetName("in");
|
||||
|
||||
auto pass_through_fn = [](Stream<AnyType> in,
|
||||
Graph& graph) -> Stream<AnyType> {
|
||||
auto& node = graph.AddNode("PassThroughCalculator");
|
||||
in.ConnectTo(node.In(0));
|
||||
return node.Out(0);
|
||||
};
|
||||
|
||||
Stream<AnyType> out1 = pass_through_fn(in, graph);
|
||||
Stream<AnyType> out2 = pass_through_fn(out1, graph);
|
||||
Stream<AnyType> out3 = pass_through_fn(out2, graph);
|
||||
Stream<AnyType> out4 = pass_through_fn(out3, graph);
|
||||
|
||||
// Graph outputs
|
||||
out4.SetName("out").ConnectTo(graph.Out(0));
|
||||
|
||||
return graph.GetConfig();
|
||||
}
|
||||
```
|
||||
See more details in [Building Graphs in C++](building_graphs_cpp.md)
|
||||
|
||||
## Subgraph
|
||||
|
||||
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
|
||||
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
|
||||
public interface of a subgraph consists of a set of input and output streams
|
||||
similar to a calculator's public interface. The subgraph can then be included in
|
||||
an `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is
|
||||
a `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is
|
||||
loaded from a `CalculatorGraphConfig`, each subgraph node is replaced by the
|
||||
corresponding graph of calculators. As a result, the semantics and performance
|
||||
of the subgraph is identical to the corresponding graph of calculators.
|
||||
@@ -151,7 +181,7 @@ protobuf specified for a MediaPipe calculator. These "graph options" can be
|
||||
specified where a graph is invoked, and used to populate calculator options and
|
||||
subgraph options within the graph.
|
||||
|
||||
In a CalculatorGraphConfig, graph options can be specified for a subgraph
|
||||
In a `CalculatorGraphConfig`, graph options can be specified for a subgraph
|
||||
exactly like calculator options, as shown below:
|
||||
|
||||
```
|
||||
@@ -178,7 +208,7 @@ node {
|
||||
}
|
||||
```
|
||||
|
||||
In a CalculatorGraphConfig, graph options can be accepted and used to populate
|
||||
In a `CalculatorGraphConfig`, graph options can be accepted and used to populate
|
||||
calculator options, as shown below:
|
||||
|
||||
```
|
||||
@@ -188,7 +218,7 @@ graph_options: {
|
||||
|
||||
node: {
|
||||
calculator: "ImageToTensorCalculator"
|
||||
input_stream: "IMAGE:multi_backend_image"
|
||||
input_stream: "IMAGE:image"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageToTensorCalculatorOptions] {
|
||||
keep_aspect_ratio: true
|
||||
@@ -247,9 +277,9 @@ NOTE: The current approach is experimental and subject to change. We welcome
|
||||
your feedback.
|
||||
|
||||
Please use the `CalculatorGraphTest.Cycle` unit test in
|
||||
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown
|
||||
below is the cyclic graph in the test. The `sum` output of the adder is the sum
|
||||
of the integers generated by the integer source calculator.
|
||||
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown below is
|
||||
the cyclic graph in the test. The `sum` output of the adder is the sum of the
|
||||
integers generated by the integer source calculator.
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/packets
|
||||
title: Packets
|
||||
parent: Framework Concepts
|
||||
nav_order: 3
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/realtime_streams
|
||||
title: Real-time Streams
|
||||
parent: Framework Concepts
|
||||
nav_order: 6
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/framework_concepts/synchronization
|
||||
title: Synchronization
|
||||
parent: Framework Concepts
|
||||
nav_order: 4
|
||||
@@ -112,14 +113,14 @@ Warning: On the other hand, it is not guaranteed that an input packet will
|
||||
always be available for all streams.
|
||||
|
||||
To explain how it works, we need to introduce the definition of a settled
|
||||
timestamp. We say that a timestamp in a stream is *settled* if it lower than the
|
||||
timestamp bound. In other words, a timestamp is settled for a stream once the
|
||||
state of the input at that timestamp is irrevocably known: either there is a
|
||||
timestamp. We say that a timestamp in a stream is *settled* if it is lower than
|
||||
the timestamp bound. In other words, a timestamp is settled for a stream once
|
||||
the state of the input at that timestamp is irrevocably known: either there is a
|
||||
packet, or there is the certainty that a packet with that timestamp will not
|
||||
arrive.
|
||||
|
||||
Note: For this reason, MediaPipe also allows a stream producer to explicitly
|
||||
advance the timestamp bound farther that what the last packet implies, i.e. to
|
||||
advance the timestamp bound farther than what the last packet implies, i.e. to
|
||||
provide a tighter bound. This can allow the downstream nodes to settle their
|
||||
inputs sooner.
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/android
|
||||
title: MediaPipe on Android
|
||||
parent: Getting Started
|
||||
has_children: true
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/android_archive_library
|
||||
title: MediaPipe Android Archive
|
||||
parent: MediaPipe on Android
|
||||
grand_parent: Getting Started
|
||||
|
||||
@@ -13,6 +13,17 @@ nav_order: 2
|
||||
{:toc}
|
||||
---
|
||||
|
||||
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||
as the primary developer documentation
|
||||
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||
the new site, but will remain available in the source code repository on an
|
||||
as-is basis.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
MediaPipe Android Solution APIs (currently in alpha) are available in:
|
||||
|
||||
* [MediaPipe Face Detection](../solutions/face_detection#android-solution-api)
|
||||
|
||||
@@ -12,6 +12,17 @@ nav_exclude: true
|
||||
{:toc}
|
||||
---
|
||||
|
||||
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||
as the primary developer documentation
|
||||
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||
the new site, but will remain available in the source code repository on an
|
||||
as-is basis.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
### Android
|
||||
|
||||
Please see these [instructions](./android.md).
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/cpp
|
||||
title: MediaPipe in C++
|
||||
parent: Getting Started
|
||||
has_children: true
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/faq
|
||||
title: FAQ
|
||||
parent: Getting Started
|
||||
nav_order: 9
|
||||
@@ -59,7 +60,7 @@ The second approach allows up to [`max_in_flight`] invocations of the
|
||||
packets from [`CalculatorBase::Process`] are automatically ordered by timestamp
|
||||
before they are passed along to downstream calculators.
|
||||
|
||||
With either aproach, you must be aware that the calculator running in parallel
|
||||
With either approach, you must be aware that the calculator running in parallel
|
||||
cannot maintain internal state in the same way as a normal sequential
|
||||
calculator.
|
||||
|
||||
|
||||
@@ -11,3 +11,14 @@ has_children: true
|
||||
1. TOC
|
||||
{:toc}
|
||||
---
|
||||
|
||||
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||
as the primary developer documentation
|
||||
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||
the new site, but will remain available in the source code repository on an
|
||||
as-is basis.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/gpu_support
|
||||
title: GPU Support
|
||||
parent: Getting Started
|
||||
nav_order: 7
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/hello_world_android
|
||||
title: Hello World! on Android
|
||||
parent: MediaPipe on Android
|
||||
grand_parent: Getting Started
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/hello_world_cpp
|
||||
title: Hello World! in C++
|
||||
parent: MediaPipe in C++
|
||||
grand_parent: Getting Started
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/hello_world_ios
|
||||
title: Hello World! on iOS
|
||||
parent: MediaPipe on iOS
|
||||
grand_parent: Getting Started
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/help
|
||||
title: Getting Help
|
||||
parent: Getting Started
|
||||
nav_order: 8
|
||||
@@ -37,8 +38,8 @@ If you open a GitHub issue, here is our policy:
|
||||
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
|
||||
- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
|
||||
- **Bazel version**:
|
||||
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev enviroment)**:
|
||||
- **Xcode & Tulsi version (if issue is related to building in mobile dev enviroment)**:
|
||||
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev environment)**:
|
||||
- **Xcode & Tulsi version (if issue is related to building in mobile dev environment)**:
|
||||
- **Exact steps to reproduce**:
|
||||
|
||||
### Describe the problem
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/install
|
||||
title: Installation
|
||||
parent: Getting Started
|
||||
nav_order: 6
|
||||
@@ -35,7 +36,7 @@ install --user six`.
|
||||
|
||||
```bash
|
||||
$ cd $HOME
|
||||
$ git clone https://github.com/google/mediapipe.git
|
||||
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||
|
||||
# Change directory into MediaPipe root directory
|
||||
$ cd mediapipe
|
||||
@@ -67,6 +68,13 @@ install --user six`.
|
||||
libopencv-video-dev
|
||||
```
|
||||
|
||||
**Note**. On Debian 11/Ubuntu 21.04 where OpenCV 4.5 is installed with
|
||||
`libopencv-video-dev`, `libopencv-contrib-dev` should also be installed.
|
||||
|
||||
```bash
|
||||
$ sudo apt-get install -y libopencv-contrib-dev
|
||||
```
|
||||
|
||||
MediaPipe's [`opencv_linux.BUILD`] and [`WORKSPACE`] are already configured
|
||||
for OpenCV 2/3 and should work correctly on any architecture:
|
||||
|
||||
@@ -287,7 +295,7 @@ build issues.
|
||||
2. Checkout MediaPipe repository.
|
||||
|
||||
```bash
|
||||
$ git clone https://github.com/google/mediapipe.git
|
||||
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||
|
||||
# Change directory into MediaPipe root directory
|
||||
$ cd mediapipe
|
||||
@@ -416,7 +424,7 @@ build issues.
|
||||
3. Checkout MediaPipe repository.
|
||||
|
||||
```bash
|
||||
$ git clone https://github.com/google/mediapipe.git
|
||||
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||
|
||||
$ cd mediapipe
|
||||
```
|
||||
@@ -590,7 +598,7 @@ next section.
|
||||
7. Checkout MediaPipe repository.
|
||||
|
||||
```
|
||||
C:\Users\Username\mediapipe_repo> git clone https://github.com/google/mediapipe.git
|
||||
C:\Users\Username\mediapipe_repo> git clone --depth 1 https://github.com/google/mediapipe.git
|
||||
|
||||
# Change directory into MediaPipe root directory
|
||||
C:\Users\Username\mediapipe_repo> cd mediapipe
|
||||
@@ -651,7 +659,7 @@ cameras. Alternatively, you use a video file as input.
|
||||
|
||||
1. Follow the
|
||||
[instruction](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to
|
||||
install Windows Sysystem for Linux (Ubuntu).
|
||||
install Windows Subsystem for Linux (Ubuntu).
|
||||
|
||||
2. Install Windows ADB and start the ADB server in Windows.
|
||||
|
||||
@@ -680,7 +688,7 @@ cameras. Alternatively, you use a video file as input.
|
||||
6. Checkout MediaPipe repository.
|
||||
|
||||
```bash
|
||||
username@DESKTOP-TMVLBJ1:~$ git clone https://github.com/google/mediapipe.git
|
||||
username@DESKTOP-TMVLBJ1:~$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||
|
||||
username@DESKTOP-TMVLBJ1:~$ cd mediapipe
|
||||
```
|
||||
@@ -771,7 +779,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
|
||||
2. Build a docker image with tag "mediapipe".
|
||||
|
||||
```bash
|
||||
$ git clone https://github.com/google/mediapipe.git
|
||||
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||
$ cd mediapipe
|
||||
$ docker build --tag=mediapipe .
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/ios
|
||||
title: MediaPipe on iOS
|
||||
parent: Getting Started
|
||||
has_children: true
|
||||
@@ -91,6 +92,10 @@ allows you to make use of automatic provisioning (see later section).
|
||||
This will install `Tulsi.app` inside the `Applications` directory in your
|
||||
home directory.
|
||||
|
||||
**Note**: Please ensure the `xcode_version` in the
|
||||
[`build_and_run.sh`](https://github.com/bazelbuild/tulsi/blob/b1d0108e6a93dbe8ab01529b2c607b6b651f0759/build_and_run.sh#L26)
|
||||
file in tulsi repo is the same version as installed in your system.
|
||||
|
||||
2. Open `mediapipe/Mediapipe.tulsiproj` using the Tulsi app.
|
||||
|
||||
Tip: If Tulsi displays an error saying "Bazel could not be found", press the
|
||||
|
||||
@@ -12,6 +12,17 @@ nav_order: 4
|
||||
{:toc}
|
||||
---
|
||||
|
||||
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||
as the primary developer documentation
|
||||
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||
the new site, but will remain available in the source code repository on an
|
||||
as-is basis.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Ready-to-use JavaScript Solutions
|
||||
|
||||
MediaPipe currently offers the following solutions:
|
||||
@@ -29,11 +40,11 @@ Solution | NPM Package | Example
|
||||
Click on a solution link above for more information, including API and code
|
||||
snippets.
|
||||
|
||||
### Supported plaforms:
|
||||
### Supported platforms:
|
||||
|
||||
| Browser | Platform | Notes |
|
||||
| ------- | ----------------------- | -------------------------------------- |
|
||||
| Chrome | Android / Windows / Mac | Pixel 4 and older unsupported. Fuschia |
|
||||
| Chrome | Android / Windows / Mac | Pixel 4 and older unsupported. Fuchsia |
|
||||
| | | unsupported. |
|
||||
| Chrome | iOS | Camera unavailable in Chrome on iOS. |
|
||||
| Safari | iPad/iPhone/Mac | iOS and Safari on iPad / iPhone / |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
title: MediaPipe Python Framework
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/python_framework
|
||||
parent: MediaPipe in Python
|
||||
grand_parent: Getting Started
|
||||
nav_order: 1
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/getting_started/troubleshooting
|
||||
title: Troubleshooting
|
||||
parent: Getting Started
|
||||
nav_order: 10
|
||||
@@ -65,7 +66,7 @@ WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/gith
|
||||
```
|
||||
|
||||
usually indicates that Bazel fails to download necessary dependency repositories
|
||||
that MediaPipe needs. MedaiPipe has several dependency repositories that are
|
||||
that MediaPipe needs. MediaPipe has several dependency repositories that are
|
||||
hosted by Google sites. In some regions, you may need to set up a network proxy
|
||||
or use a VPN to access those resources. You may also need to append
|
||||
`--host_jvm_args "-DsocksProxyHost=<ip address> -DsocksProxyPort=<port number>"`
|
||||
|
||||
@@ -19,6 +19,17 @@ ML solutions for live and streaming media.
|
||||
 | 
|
||||
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
|
||||
|
||||
----
|
||||
|
||||
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||
as the primary developer documentation
|
||||
site for MediaPipe starting April 3, 2023.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## ML solutions in MediaPipe
|
||||
|
||||
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 14
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
AutoFlip is an automatic video cropping pipeline built on top of MediaPipe. This
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 10
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe Box Tracking has been powering real-time tracking in
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 1
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe Face Detection is an ultrafast face detection solution that comes with
|
||||
@@ -54,6 +64,25 @@ used for its improved inference speed. Please refer to the
|
||||
[model cards](./models.md#face_detection) for details. Default to `0` if not
|
||||
specified.
|
||||
|
||||
Note: Not available for JavaScript (use "model" instead).
|
||||
|
||||
#### model
|
||||
|
||||
A string value to indicate which model should be used. Use "short" to
|
||||
select a short-range model that works best for faces within 2 meters from the
|
||||
camera, and "full" for a full-range model best for faces within 5 meters. For
|
||||
the full-range option, a sparse model is used for its improved inference speed.
|
||||
Please refer to the model cards for details. Default to empty string.
|
||||
|
||||
Note: Valid only for JavaScript solution.
|
||||
|
||||
#### selfie_mode
|
||||
|
||||
A boolean value to indicate whether to flip the images/video frames
|
||||
horizontally or not. Default to `false`.
|
||||
|
||||
Note: Valid only for JavaScript solution.
|
||||
|
||||
#### min_detection_confidence
|
||||
|
||||
Minimum confidence value (`[0.0, 1.0]`) from the face detection model for the
|
||||
@@ -146,9 +175,9 @@ Please first see general [introduction](../getting_started/javascript.md) on
|
||||
MediaPipe in JavaScript, then learn more in the companion [web demo](#resources)
|
||||
and the following usage example.
|
||||
|
||||
Supported configuration options:
|
||||
|
||||
* [modelSelection](#model_selection)
|
||||
Supported face detection options:
|
||||
* [selfieMode](#selfie_mode)
|
||||
* [model](#model)
|
||||
* [minDetectionConfidence](#min_detection_confidence)
|
||||
|
||||
```html
|
||||
@@ -176,6 +205,7 @@ Supported configuration options:
|
||||
const videoElement = document.getElementsByClassName('input_video')[0];
|
||||
const canvasElement = document.getElementsByClassName('output_canvas')[0];
|
||||
const canvasCtx = canvasElement.getContext('2d');
|
||||
const drawingUtils = window;
|
||||
|
||||
function onResults(results) {
|
||||
// Draw the overlays.
|
||||
@@ -199,7 +229,7 @@ const faceDetection = new FaceDetection({locateFile: (file) => {
|
||||
return `https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/${file}`;
|
||||
}});
|
||||
faceDetection.setOptions({
|
||||
modelSelection: 0,
|
||||
model: 'short',
|
||||
minDetectionConfidence: 0.5
|
||||
});
|
||||
faceDetection.onResults(onResults);
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 2
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe Face Mesh is a solution that estimates 468 3D face landmarks in
|
||||
@@ -133,7 +143,7 @@ about the model in this [paper](https://arxiv.org/abs/2006.10962).
|
||||
The [Face Landmark Model](#face-landmark-model) performs a single-camera face landmark
|
||||
detection in the screen coordinate space: the X- and Y- coordinates are
|
||||
normalized screen coordinates, while the Z coordinate is relative and is scaled
|
||||
as the X coodinate under the
|
||||
as the X coordinate under the
|
||||
[weak perspective projection camera model](https://en.wikipedia.org/wiki/3D_projection#Weak_perspective_projection).
|
||||
This format is well-suited for some applications, however it does not directly
|
||||
enable the full spectrum of augmented reality (AR) features like aligning a
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 8
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
|
||||
|
||||
## Example Apps
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 4
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
The ability to perceive the shape and motion of hands can be a vital component
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 6
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
Live perception of simultaneous [human pose](./pose.md),
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 11
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
Augmented Reality (AR) technology creates fun, engaging, and immersive user
|
||||
|
||||
+12
-2
@@ -18,6 +18,16 @@ nav_order: 3
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
A wide range of real-world applications, including computational photography
|
||||
@@ -38,7 +48,7 @@ camera, in real-time, without the need for specialized hardware. Through use of
|
||||
iris landmarks, the solution is also able to determine the metric distance
|
||||
between the subject and the camera with relative error less than 10%. Note that
|
||||
iris tracking does not infer the location at which people are looking, nor does
|
||||
it provide any form of identity recognition. With the cross-platfrom capability
|
||||
it provide any form of identity recognition. With the cross-platform capability
|
||||
of the MediaPipe framework, MediaPipe Iris can run on most modern
|
||||
[mobile phones](#mobile), [desktops/laptops](#desktop) and even on the
|
||||
[web](#web).
|
||||
@@ -99,7 +109,7 @@ You can also find more details in this
|
||||
### Iris Landmark Model
|
||||
|
||||
The iris model takes an image patch of the eye region and estimates both the eye
|
||||
landmarks (along the eyelid) and iris landmarks (along ths iris contour). You
|
||||
landmarks (along the eyelid) and iris landmarks (along this iris contour). You
|
||||
can find more details in this [paper](https://arxiv.org/abs/2006.11341).
|
||||
|
||||
 |
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 13
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe KNIFT is a template-based feature matching solution using KNIFT
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 15
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe is a useful and general framework for media processing that can
|
||||
@@ -85,7 +95,7 @@ process new data sets, in the documentation of
|
||||
|
||||
MediaSequence uses SequenceExamples as the format of both inputs and
|
||||
outputs. Annotations are encoded as inputs in a SequenceExample of metadata
|
||||
that defines the labels and the path to the cooresponding video file. This
|
||||
that defines the labels and the path to the corresponding video file. This
|
||||
metadata is passed as input to the C++ `media_sequence_demo` binary, and the
|
||||
output is a SequenceExample filled with images and annotations ready for
|
||||
model training.
|
||||
|
||||
@@ -12,6 +12,20 @@ nav_order: 30
|
||||
{:toc}
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for
|
||||
[these MediaPipe Legacy Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
as of March 1, 2023. All other
|
||||
[MediaPipe Legacy Solutions will be upgraded](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
to a new MediaPipe Solution. The code repository and prebuilt binaries for all
|
||||
MediaPipe Legacy Solutions will continue to be provided on an as-is basis.
|
||||
We encourage you to check out the new MediaPipe Solutions at:
|
||||
[https://developers.google.com/mediapipe/solutions](https://developers.google.com/mediapipe/solutions)*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
||||
|
||||
* Short-range model (best for faces within 2 meters from the camera):
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 9
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
|
||||
|
||||
## Example Apps
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 12
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
MediaPipe Objectron is a mobile real-time 3D object detection solution for
|
||||
@@ -170,7 +180,7 @@ and a
|
||||
The detection subgraph performs ML inference only once every few frames to
|
||||
reduce computation load, and decodes the output tensor to a FrameAnnotation that
|
||||
contains nine keypoints: the 3D bounding box's center and its eight vertices.
|
||||
The tracking subgraph runs every frame, using the box traker in
|
||||
The tracking subgraph runs every frame, using the box tracker in
|
||||
[MediaPipe Box Tracking](./box_tracking.md) to track the 2D box tightly
|
||||
enclosing the projection of the 3D bounding box, and lifts the tracked 2D
|
||||
keypoints to 3D with
|
||||
@@ -613,7 +623,7 @@ z_ndc = 1 / Z
|
||||
|
||||
### Pixel Space
|
||||
|
||||
In this API we set upper-left coner of an image as the origin of pixel
|
||||
In this API we set upper-left corner of an image as the origin of pixel
|
||||
coordinate. One can convert from NDC to pixel space as follows:
|
||||
|
||||
```
|
||||
|
||||
@@ -20,6 +20,16 @@ nav_order: 5
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
Human pose estimation from video plays a critical role in various applications
|
||||
|
||||
@@ -19,6 +19,16 @@ nav_order: 1
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
One of the applications
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 7
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||
Solution. For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
## Overview
|
||||
|
||||
*Fig 1. Example of MediaPipe Selfie Segmentation.* |
|
||||
|
||||
@@ -13,6 +13,22 @@ has_toc: false
|
||||
{:toc}
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions. We have
|
||||
ended support for
|
||||
[these MediaPipe Legacy Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
as of March 1, 2023. All other
|
||||
[MediaPipe Legacy Solutions will be upgraded](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
to a new MediaPipe Solution. The
|
||||
[code repository](https://github.com/google/mediapipe/tree/master/mediapipe)
|
||||
and prebuilt binaries for all MediaPipe Legacy Solutions will continue to
|
||||
be provided on an as-is basis. We encourage you to check out the new MediaPipe
|
||||
Solutions at:
|
||||
[https://developers.google.com/mediapipe/solutions](https://developers.google.com/mediapipe/solutions)*
|
||||
|
||||
*This notice and web page will be removed on June 1, 2023.*
|
||||
|
||||
----
|
||||
|
||||
MediaPipe offers open source cross-platform, customizable ML solutions for live
|
||||
and streaming media.
|
||||
|
||||
|
||||
@@ -18,6 +18,16 @@ nav_order: 16
|
||||
</details>
|
||||
---
|
||||
|
||||
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||
For more information, see the new
|
||||
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||
site.*
|
||||
|
||||
*This notice and web page will be removed on April 3, 2023.*
|
||||
|
||||
----
|
||||
|
||||
MediaPipe is a useful and general framework for media processing that can assist
|
||||
with research, development, and deployment of ML models. This example focuses on
|
||||
model development by demonstrating how to prepare training data and do model
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
---
|
||||
layout: default
|
||||
layout: forward
|
||||
target: https://developers.google.com/mediapipe/framework/tools/visualizer
|
||||
title: Visualizer
|
||||
parent: Tools
|
||||
nav_order: 1
|
||||
|
||||
@@ -117,6 +117,7 @@ mediapipe_proto_library(
|
||||
"//mediapipe/framework:calculator_proto",
|
||||
"//mediapipe/framework/formats:classification_proto",
|
||||
"//mediapipe/framework/formats:landmark_proto",
|
||||
"//mediapipe/framework/formats:time_series_header_proto",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -197,6 +198,7 @@ cc_library(
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/formats:matrix",
|
||||
"//mediapipe/framework/formats:rect_cc_proto",
|
||||
"//mediapipe/framework/formats:tensor",
|
||||
"//mediapipe/framework/port:integral_types",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
@@ -219,6 +221,7 @@ cc_library(
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/formats:matrix",
|
||||
"//mediapipe/framework/formats:rect_cc_proto",
|
||||
"//mediapipe/framework/formats:tensor",
|
||||
"//mediapipe/framework/port:integral_types",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
@@ -235,10 +238,12 @@ cc_test(
|
||||
":begin_loop_calculator",
|
||||
":end_loop_calculator",
|
||||
":gate_calculator",
|
||||
":pass_through_calculator",
|
||||
"//mediapipe/framework:calculator_context",
|
||||
"//mediapipe/framework:calculator_contract",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:packet",
|
||||
"//mediapipe/framework/formats:tensor",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:integral_types",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
@@ -1158,6 +1163,7 @@ cc_library(
|
||||
"//mediapipe/framework:collection_item_id",
|
||||
"//mediapipe/framework/formats:classification_cc_proto",
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/formats:time_series_header_cc_proto",
|
||||
"//mediapipe/framework/port:integral_types",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
@@ -1329,6 +1335,7 @@ cc_library(
|
||||
hdrs = ["merge_to_vector_calculator.h"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:packet",
|
||||
"//mediapipe/framework/api2:node",
|
||||
"//mediapipe/framework/api2:port",
|
||||
"//mediapipe/framework/formats:detection_cc_proto",
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
#include "mediapipe/calculators/core/end_loop_calculator.h"
|
||||
#include "mediapipe/framework/calculator_contract.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/formats/tensor.h"
|
||||
#include "mediapipe/framework/packet.h"
|
||||
#include "mediapipe/framework/port/gmock.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
@@ -444,5 +445,67 @@ TEST_F(BeginEndLoopCalculatorGraphWithClonedInputsTest, MultipleVectors) {
|
||||
PacketOfIntsEq(input_timestamp2, std::vector<int>{6, 9})));
|
||||
}
|
||||
|
||||
absl::Status InitBeginEndTensorLoopTestGraph(
|
||||
CalculatorGraph& graph, std::vector<Packet>& output_packets) {
|
||||
auto graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
R"pb(
|
||||
num_threads: 4
|
||||
input_stream: "tensors"
|
||||
node {
|
||||
calculator: "BeginLoopTensorCalculator"
|
||||
input_stream: "ITERABLE:tensors"
|
||||
output_stream: "ITEM:tensor"
|
||||
output_stream: "BATCH_END:timestamp"
|
||||
}
|
||||
node {
|
||||
calculator: "PassThroughCalculator"
|
||||
input_stream: "tensor"
|
||||
output_stream: "passed_tensor"
|
||||
}
|
||||
node {
|
||||
calculator: "EndLoopTensorCalculator"
|
||||
input_stream: "ITEM:passed_tensor"
|
||||
input_stream: "BATCH_END:timestamp"
|
||||
output_stream: "ITERABLE:output_tensors"
|
||||
}
|
||||
)pb");
|
||||
tool::AddVectorSink("output_tensors", &graph_config, &output_packets);
|
||||
MP_RETURN_IF_ERROR(graph.Initialize(graph_config));
|
||||
return graph.StartRun({});
|
||||
}
|
||||
|
||||
TEST(BeginEndTensorLoopCalculatorGraphTest, SingleNonEmptyVector) {
|
||||
// Initialize the graph.
|
||||
CalculatorGraph graph;
|
||||
std::vector<Packet> output_packets;
|
||||
MP_ASSERT_OK(InitBeginEndTensorLoopTestGraph(graph, output_packets));
|
||||
|
||||
// Prepare the inputs and run.
|
||||
Timestamp input_timestamp = Timestamp(0);
|
||||
std::vector<mediapipe::Tensor> tensors;
|
||||
for (int i = 0; i < 4; i++) {
|
||||
tensors.emplace_back(Tensor::ElementType::kFloat32,
|
||||
Tensor::Shape{4, 3, 2, 1});
|
||||
}
|
||||
Packet vector_packet =
|
||||
MakePacket<std::vector<mediapipe::Tensor>>(std::move(tensors));
|
||||
MP_ASSERT_OK(graph.AddPacketToInputStream(
|
||||
"tensors", std::move(vector_packet).At(input_timestamp)));
|
||||
MP_ASSERT_OK(graph.WaitUntilIdle());
|
||||
|
||||
// Verify the output packet.
|
||||
EXPECT_EQ(output_packets.size(), 1);
|
||||
const std::vector<Tensor>& output_tensors =
|
||||
output_packets[0].Get<std::vector<Tensor>>();
|
||||
EXPECT_EQ(output_tensors.size(), 4);
|
||||
for (int i = 0; i < output_tensors.size(); i++) {
|
||||
EXPECT_THAT(output_tensors[i].shape().dims,
|
||||
testing::ElementsAre(4, 3, 2, 1));
|
||||
}
|
||||
|
||||
MP_ASSERT_OK(graph.CloseAllPacketSources());
|
||||
MP_ASSERT_OK(graph.WaitUntilDone());
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||
#include "mediapipe/framework/formats/matrix.h"
|
||||
#include "mediapipe/framework/formats/rect.pb.h"
|
||||
#include "mediapipe/framework/formats/tensor.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
@@ -55,4 +56,8 @@ REGISTER_CALCULATOR(BeginLoopMatrixVectorCalculator);
|
||||
typedef BeginLoopCalculator<std::vector<uint64_t>> BeginLoopUint64tCalculator;
|
||||
REGISTER_CALCULATOR(BeginLoopUint64tCalculator);
|
||||
|
||||
// A calculator to process std::vector<mediapipe::Tensor>.
|
||||
typedef BeginLoopCalculator<std::vector<Tensor>> BeginLoopTensorCalculator;
|
||||
REGISTER_CALCULATOR(BeginLoopTensorCalculator);
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -15,7 +15,6 @@
|
||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_BEGIN_LOOP_CALCULATOR_H_
|
||||
#define MEDIAPIPE_CALCULATORS_CORE_BEGIN_LOOP_CALCULATOR_H_
|
||||
|
||||
#include "absl/memory/memory.h"
|
||||
#include "mediapipe/framework/calculator_context.h"
|
||||
#include "mediapipe/framework/calculator_contract.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
@@ -24,6 +23,7 @@
|
||||
#include "mediapipe/framework/port/integral_types.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/port/status_macros.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
@@ -49,7 +49,7 @@ namespace mediapipe {
|
||||
// calculator: "EndLoopWithOutputCalculator"
|
||||
// input_stream: "ITEM:output_of_loop_body" # ItemU @loop_internal_ts
|
||||
// input_stream: "BATCH_END:ext_ts" # Timestamp @loop_internal_ts
|
||||
// output_stream: "OUTPUT:aggregated_result" # IterableU @ext_ts
|
||||
// output_stream: "ITERABLE:aggregated_result" # IterableU @ext_ts
|
||||
// }
|
||||
//
|
||||
// Input streams tagged with "CLONE" are cloned to the corresponding output
|
||||
@@ -112,13 +112,38 @@ class BeginLoopCalculator : public CalculatorBase {
|
||||
absl::Status Process(CalculatorContext* cc) final {
|
||||
Timestamp last_timestamp = loop_internal_timestamp_;
|
||||
if (!cc->Inputs().Tag("ITERABLE").IsEmpty()) {
|
||||
const IterableT& collection =
|
||||
cc->Inputs().Tag("ITERABLE").template Get<IterableT>();
|
||||
for (const auto& item : collection) {
|
||||
cc->Outputs().Tag("ITEM").AddPacket(
|
||||
MakePacket<ItemT>(item).At(loop_internal_timestamp_));
|
||||
ForwardClonePackets(cc, loop_internal_timestamp_);
|
||||
++loop_internal_timestamp_;
|
||||
// Try to consume the ITERABLE packet if possible to obtain the ownership
|
||||
// and emit the item packets by moving them.
|
||||
// If the ITERABLE packet is not consumable, then try to copy each item
|
||||
// instead. If the ITEM type is not copy constructible, an error will be
|
||||
// returned.
|
||||
auto iterable_ptr_or =
|
||||
cc->Inputs().Tag("ITERABLE").Value().Consume<IterableT>();
|
||||
if (iterable_ptr_or.ok()) {
|
||||
for (auto& item : *iterable_ptr_or.value()) {
|
||||
Packet item_packet = MakePacket<ItemT>(std::move(item));
|
||||
cc->Outputs().Tag("ITEM").AddPacket(
|
||||
item_packet.At(loop_internal_timestamp_));
|
||||
ForwardClonePackets(cc, loop_internal_timestamp_);
|
||||
++loop_internal_timestamp_;
|
||||
}
|
||||
} else {
|
||||
if constexpr (std::is_copy_constructible<ItemT>()) {
|
||||
const IterableT& collection =
|
||||
cc->Inputs().Tag("ITERABLE").template Get<IterableT>();
|
||||
for (const auto& item : collection) {
|
||||
cc->Outputs().Tag("ITEM").AddPacket(
|
||||
MakePacket<ItemT>(item).At(loop_internal_timestamp_));
|
||||
ForwardClonePackets(cc, loop_internal_timestamp_);
|
||||
++loop_internal_timestamp_;
|
||||
}
|
||||
} else {
|
||||
return absl::InternalError(
|
||||
"The element type is not copiable. Consider making the "
|
||||
"BeginLoopCalculator the sole owner of the input packet so that "
|
||||
"the "
|
||||
"items can be consumed and moved.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -138,7 +163,6 @@ class BeginLoopCalculator : public CalculatorBase {
|
||||
.Tag("BATCH_END")
|
||||
.AddPacket(MakePacket<Timestamp>(cc->InputTimestamp())
|
||||
.At(Timestamp(loop_internal_timestamp_ - 1)));
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
#include "mediapipe/framework/collection_item_id.h"
|
||||
#include "mediapipe/framework/formats/classification.pb.h"
|
||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||
#include "mediapipe/framework/formats/time_series_header.pb.h"
|
||||
#include "mediapipe/framework/port/canonical_errors.h"
|
||||
#include "mediapipe/framework/port/integral_types.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
@@ -84,6 +85,8 @@ class ConstantSidePacketCalculator : public CalculatorBase {
|
||||
packet.Set<LandmarkList>();
|
||||
} else if (packet_options.has_double_value()) {
|
||||
packet.Set<double>();
|
||||
} else if (packet_options.has_time_series_header_value()) {
|
||||
packet.Set<TimeSeriesHeader>();
|
||||
} else {
|
||||
return absl::InvalidArgumentError(
|
||||
"None of supported values were specified in options.");
|
||||
@@ -118,6 +121,9 @@ class ConstantSidePacketCalculator : public CalculatorBase {
|
||||
MakePacket<LandmarkList>(packet_options.landmark_list_value()));
|
||||
} else if (packet_options.has_double_value()) {
|
||||
packet.Set(MakePacket<double>(packet_options.double_value()));
|
||||
} else if (packet_options.has_time_series_header_value()) {
|
||||
packet.Set(MakePacket<TimeSeriesHeader>(
|
||||
packet_options.time_series_header_value()));
|
||||
} else {
|
||||
return absl::InvalidArgumentError(
|
||||
"None of supported values were specified in options.");
|
||||
|
||||
@@ -19,6 +19,7 @@ package mediapipe;
|
||||
import "mediapipe/framework/calculator.proto";
|
||||
import "mediapipe/framework/formats/classification.proto";
|
||||
import "mediapipe/framework/formats/landmark.proto";
|
||||
import "mediapipe/framework/formats/time_series_header.proto";
|
||||
|
||||
message ConstantSidePacketCalculatorOptions {
|
||||
extend CalculatorOptions {
|
||||
@@ -35,6 +36,7 @@ message ConstantSidePacketCalculatorOptions {
|
||||
ClassificationList classification_list_value = 6;
|
||||
LandmarkList landmark_list_value = 7;
|
||||
double double_value = 9;
|
||||
TimeSeriesHeader time_series_header_value = 10;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||
#include "mediapipe/framework/formats/matrix.h"
|
||||
#include "mediapipe/framework/formats/rect.pb.h"
|
||||
#include "mediapipe/framework/formats/tensor.h"
|
||||
#include "mediapipe/util/render_data.pb.h"
|
||||
#include "tensorflow/lite/interpreter.h"
|
||||
|
||||
@@ -52,8 +53,9 @@ typedef EndLoopCalculator<std::vector<::mediapipe::ClassificationList>>
|
||||
EndLoopClassificationListCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopClassificationListCalculator);
|
||||
|
||||
typedef EndLoopCalculator<std::vector<TfLiteTensor>> EndLoopTensorCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopTensorCalculator);
|
||||
typedef EndLoopCalculator<std::vector<TfLiteTensor>>
|
||||
EndLoopTfLiteTensorCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopTfLiteTensorCalculator);
|
||||
|
||||
typedef EndLoopCalculator<std::vector<::mediapipe::Detection>>
|
||||
EndLoopDetectionCalculator;
|
||||
@@ -62,4 +64,7 @@ REGISTER_CALCULATOR(EndLoopDetectionCalculator);
|
||||
typedef EndLoopCalculator<std::vector<Matrix>> EndLoopMatrixCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopMatrixCalculator);
|
||||
|
||||
typedef EndLoopCalculator<std::vector<Tensor>> EndLoopTensorCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopTensorCalculator);
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -15,6 +15,8 @@
|
||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_END_LOOP_CALCULATOR_H_
|
||||
#define MEDIAPIPE_CALCULATORS_CORE_END_LOOP_CALCULATOR_H_
|
||||
|
||||
#include <type_traits>
|
||||
|
||||
#include "mediapipe/framework/calculator_context.h"
|
||||
#include "mediapipe/framework/calculator_contract.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
@@ -75,8 +77,23 @@ class EndLoopCalculator : public CalculatorBase {
|
||||
if (!input_stream_collection_) {
|
||||
input_stream_collection_.reset(new IterableT);
|
||||
}
|
||||
input_stream_collection_->push_back(
|
||||
cc->Inputs().Tag("ITEM").template Get<ItemT>());
|
||||
// Try to consume the item and move it into the collection. If the items
|
||||
// are not consumable, then try to copy them instead. If the items are
|
||||
// not copiable, then an error will be returned.
|
||||
auto item_ptr_or = cc->Inputs().Tag("ITEM").Value().Consume<ItemT>();
|
||||
if (item_ptr_or.ok()) {
|
||||
input_stream_collection_->push_back(std::move(*item_ptr_or.value()));
|
||||
} else {
|
||||
if constexpr (std::is_copy_constructible_v<ItemT>) {
|
||||
input_stream_collection_->push_back(
|
||||
cc->Inputs().Tag("ITEM").template Get<ItemT>());
|
||||
} else {
|
||||
return absl::InternalError(
|
||||
"The item type is not copiable. Consider making the "
|
||||
"EndLoopCalculator the sole owner of the input packets so that "
|
||||
"it can be moved instead of copying.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!cc->Inputs().Tag("BATCH_END").Value().IsEmpty()) { // flush signal
|
||||
|
||||
@@ -35,7 +35,6 @@ message FlowLimiterCalculatorOptions {
|
||||
optional int32 max_in_queue = 2 [default = 0];
|
||||
|
||||
// The maximum time in microseconds to wait for a frame to finish processing.
|
||||
// The default value stops waiting after 1 sec.
|
||||
// The value 0 specifies no timeout.
|
||||
optional int64 in_flight_timeout = 3 [default = 1000000];
|
||||
// The default value 0 specifies no timeout.
|
||||
optional int64 in_flight_timeout = 3 [default = 0];
|
||||
}
|
||||
|
||||
@@ -48,7 +48,6 @@ class MergeToVectorCalculator : public Node {
|
||||
}
|
||||
|
||||
absl::Status Process(CalculatorContext* cc) {
|
||||
const int input_num = kIn(cc).Count();
|
||||
std::vector<T> output_vector;
|
||||
for (auto it = kIn(cc).begin(); it != kIn(cc).end(); it++) {
|
||||
const auto& elem = *it;
|
||||
|
||||
@@ -301,12 +301,18 @@ cc_test(
|
||||
":image_cropping_calculator",
|
||||
":image_cropping_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework/formats:image_frame_opencv",
|
||||
"//mediapipe/framework/formats:rect_cc_proto",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:opencv_core",
|
||||
"//mediapipe/framework/port:opencv_imgproc",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/tool:tag_map",
|
||||
"//mediapipe/framework/tool:tag_map_helper",
|
||||
"//mediapipe/gpu:gpu_buffer_to_image_frame_calculator",
|
||||
"//mediapipe/gpu:image_frame_to_gpu_buffer_calculator",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -223,8 +223,8 @@ absl::Status ImageCroppingCalculator::RenderCpu(CalculatorContext* cc) {
|
||||
cv::Mat input_mat = formats::MatView(&input_img);
|
||||
|
||||
RectSpec specs = GetCropSpecs(cc, input_img.Width(), input_img.Height());
|
||||
int target_width = specs.width, target_height = specs.height,
|
||||
rect_center_x = specs.center_x, rect_center_y = specs.center_y;
|
||||
int target_width = specs.width, target_height = specs.height;
|
||||
float rect_center_x = specs.center_x, rect_center_y = specs.center_y;
|
||||
float rotation = specs.rotation;
|
||||
|
||||
// Get border mode and value for OpenCV.
|
||||
@@ -244,19 +244,29 @@ absl::Status ImageCroppingCalculator::RenderCpu(CalculatorContext* cc) {
|
||||
output_width *= scale;
|
||||
output_height *= scale;
|
||||
|
||||
float dst_corners[8] = {0,
|
||||
output_height - 1,
|
||||
0,
|
||||
0,
|
||||
output_width - 1,
|
||||
0,
|
||||
output_width - 1,
|
||||
output_height - 1};
|
||||
cv::Mat dst_points = cv::Mat(4, 2, CV_32F, dst_corners);
|
||||
cv::Mat projection_matrix =
|
||||
float dst_corners[8] = {
|
||||
0, output_height, 0, 0, output_width, 0, output_width, output_height};
|
||||
const cv::Mat dst_points = cv::Mat(4, 2, CV_32F, dst_corners);
|
||||
// The projection matrix is computed using the corners of rects, not the
|
||||
// centers of corner pixels
|
||||
const cv::Mat projection_matrix =
|
||||
cv::getPerspectiveTransform(src_points, dst_points);
|
||||
// The projection matrix need to be adjusted because `cv::warpPerspective` is
|
||||
// based on integer centers.
|
||||
// clang-format off
|
||||
double shift_src_vec[9] = {1.0, 0.0, 0.5,
|
||||
0.0, 1.0, 0.5,
|
||||
0.0, 0.0, 1.0};
|
||||
double shift_dst_vec[9] = {1.0, 0.0, -0.5,
|
||||
0.0, 1.0, -0.5,
|
||||
0.0, 0.0, 1.0};
|
||||
// clang-format on
|
||||
const cv::Mat shift_src = cv::Mat(3, 3, CV_64F, shift_src_vec);
|
||||
const cv::Mat shift_dst = cv::Mat(3, 3, CV_64F, shift_dst_vec);
|
||||
const cv::Mat adjusted_projection_matrix =
|
||||
shift_dst * projection_matrix * shift_src;
|
||||
cv::Mat cropped_image;
|
||||
cv::warpPerspective(input_mat, cropped_image, projection_matrix,
|
||||
cv::warpPerspective(input_mat, cropped_image, adjusted_projection_matrix,
|
||||
cv::Size(output_width, output_height),
|
||||
/* flags = */ 0,
|
||||
/* borderMode = */ border_mode);
|
||||
@@ -416,8 +426,8 @@ void ImageCroppingCalculator::GetOutputDimensions(CalculatorContext* cc,
|
||||
int* dst_width,
|
||||
int* dst_height) {
|
||||
RectSpec specs = GetCropSpecs(cc, src_width, src_height);
|
||||
int crop_width = specs.width, crop_height = specs.height,
|
||||
x_center = specs.center_x, y_center = specs.center_y;
|
||||
int crop_width = specs.width, crop_height = specs.height;
|
||||
float x_center = specs.center_x, y_center = specs.center_y;
|
||||
float rotation = specs.rotation;
|
||||
|
||||
const float half_width = crop_width / 2.0f;
|
||||
@@ -466,8 +476,8 @@ RectSpec ImageCroppingCalculator::GetCropSpecs(const CalculatorContext* cc,
|
||||
int crop_width = src_width;
|
||||
int crop_height = src_height;
|
||||
// Get the center of cropping box. Default is the at the center.
|
||||
int x_center = src_width / 2;
|
||||
int y_center = src_height / 2;
|
||||
float x_center = src_width / 2.0f;
|
||||
float y_center = src_height / 2.0f;
|
||||
// Get the rotation of the cropping box.
|
||||
float rotation = 0.0f;
|
||||
// Get the normalized width and height if specified by the inputs or options.
|
||||
@@ -494,8 +504,8 @@ RectSpec ImageCroppingCalculator::GetCropSpecs(const CalculatorContext* cc,
|
||||
if (norm_rect.width() > 0.0 && norm_rect.height() > 0.0) {
|
||||
normalized_width = norm_rect.width();
|
||||
normalized_height = norm_rect.height();
|
||||
x_center = std::round(norm_rect.x_center() * src_width);
|
||||
y_center = std::round(norm_rect.y_center() * src_height);
|
||||
x_center = norm_rect.x_center() * src_width;
|
||||
y_center = norm_rect.y_center() * src_height;
|
||||
rotation = norm_rect.rotation();
|
||||
}
|
||||
} else if (cc->Inputs().HasTag(kWidthTag) &&
|
||||
@@ -521,14 +531,30 @@ RectSpec ImageCroppingCalculator::GetCropSpecs(const CalculatorContext* cc,
|
||||
// present from the inputs.
|
||||
if (!cc->Inputs().HasTag(kRectTag) && !cc->Inputs().HasTag(kNormRectTag)) {
|
||||
if (options.has_norm_center_x() && options.has_norm_center_y()) {
|
||||
x_center = std::round(options.norm_center_x() * src_width);
|
||||
y_center = std::round(options.norm_center_y() * src_height);
|
||||
x_center = options.norm_center_x() * src_width;
|
||||
y_center = options.norm_center_y() * src_height;
|
||||
}
|
||||
if (options.has_rotation()) {
|
||||
rotation = options.rotation();
|
||||
}
|
||||
}
|
||||
|
||||
if (rotation == 0.0f) {
|
||||
// Adjust the center to the closest integer when the crop size is
|
||||
// even-number and to the closest half-integer when the crop size is
|
||||
// odd-number.
|
||||
if (crop_width % 2 == 0) {
|
||||
x_center = std::round(x_center);
|
||||
} else {
|
||||
x_center = std::round(x_center + 0.5f) - 0.5f;
|
||||
}
|
||||
if (crop_height % 2 == 0) {
|
||||
y_center = std::round(y_center);
|
||||
} else {
|
||||
y_center = std::round(y_center + 0.5f) - 0.5f;
|
||||
}
|
||||
}
|
||||
|
||||
return {crop_width, crop_height, x_center, y_center, rotation};
|
||||
}
|
||||
|
||||
|
||||
@@ -42,8 +42,8 @@ namespace mediapipe {
|
||||
struct RectSpec {
|
||||
int width;
|
||||
int height;
|
||||
int center_x;
|
||||
int center_y;
|
||||
float center_x;
|
||||
float center_y;
|
||||
float rotation;
|
||||
|
||||
bool operator==(const RectSpec& rect) const {
|
||||
|
||||
@@ -19,8 +19,12 @@
|
||||
|
||||
#include "mediapipe/calculators/image/image_cropping_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/formats/image_frame_opencv.h"
|
||||
#include "mediapipe/framework/formats/rect.pb.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/opencv_core_inc.h"
|
||||
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
#include "mediapipe/framework/tool/tag_map.h"
|
||||
@@ -37,6 +41,166 @@ constexpr char kRectTag[] = "RECT";
|
||||
constexpr char kHeightTag[] = "HEIGHT";
|
||||
constexpr char kWidthTag[] = "WIDTH";
|
||||
|
||||
std::unique_ptr<mediapipe::ImageFrame> GetInputFrame(int width, int height,
|
||||
int channel) {
|
||||
const int total_size = width * height * channel;
|
||||
|
||||
auto image_format = channel == 4 ? mediapipe::ImageFormat::SRGBA
|
||||
: mediapipe::ImageFormat::SRGB;
|
||||
|
||||
auto input_frame = std::make_unique<mediapipe::ImageFrame>(
|
||||
image_format, width, height, /*alignment_boundary =*/1);
|
||||
for (int i = 0; i < total_size; ++i) {
|
||||
input_frame->MutablePixelData()[i] = i % 256;
|
||||
}
|
||||
|
||||
return input_frame;
|
||||
}
|
||||
|
||||
// Test identity function, where cropping size is same as input size
|
||||
TEST(ImageCroppingCalculatorTest, IdentityFunctionCropWithOriginalSize) {
|
||||
auto calculator_node =
|
||||
ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig::Node>(
|
||||
absl::Substitute(
|
||||
R"pb(
|
||||
calculator: "ImageCroppingCalculator"
|
||||
input_stream: "IMAGE:input_frames"
|
||||
output_stream: "IMAGE:cropped_output_frames"
|
||||
options: {
|
||||
[mediapipe.ImageCroppingCalculatorOptions.ext] {
|
||||
width: $0
|
||||
height: $1
|
||||
}
|
||||
}
|
||||
)pb",
|
||||
input_width, input_height));
|
||||
mediapipe::CalculatorRunner runner(calculator_node);
|
||||
|
||||
// Input frame.
|
||||
const auto input_frame = GetInputFrame(input_width, input_height, 3);
|
||||
auto input_frame_packet =
|
||||
mediapipe::MakePacket<mediapipe::ImageFrame>(std::move(*input_frame));
|
||||
runner.MutableInputs()->Tag("IMAGE").packets.push_back(
|
||||
input_frame_packet.At(mediapipe::Timestamp(1)));
|
||||
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const auto& outputs = runner.Outputs();
|
||||
EXPECT_EQ(outputs.NumEntries(), 1);
|
||||
const auto& output_image =
|
||||
outputs.Tag("IMAGE").packets[0].Get<mediapipe::ImageFrame>();
|
||||
|
||||
const auto expected_output = GetInputFrame(input_width, input_height, 3);
|
||||
cv::Mat output_mat = formats::MatView(&output_image);
|
||||
cv::Mat expected_mat = formats::MatView(expected_output.get());
|
||||
double max_diff = cv::norm(expected_mat, output_mat, cv::NORM_INF);
|
||||
EXPECT_EQ(max_diff, 0);
|
||||
} // TEST
|
||||
|
||||
// Test identity function, where cropping size is same as input size.
|
||||
// When an image has an odd number for its size, its center falls on a
|
||||
// fractional pixel. As a result, the values for center_x and center_y need to
|
||||
// be of type float.
|
||||
TEST(ImageCroppingCalculatorTest, IdentityFunctionCropWithOddSize) {
|
||||
const int input_width = 99;
|
||||
const int input_height = 99;
|
||||
|
||||
auto calculator_node =
|
||||
ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig::Node>(
|
||||
absl::Substitute(
|
||||
R"pb(
|
||||
calculator: "ImageCroppingCalculator"
|
||||
input_stream: "IMAGE:input_frames"
|
||||
output_stream: "IMAGE:cropped_output_frames"
|
||||
options: {
|
||||
[mediapipe.ImageCroppingCalculatorOptions.ext] {
|
||||
width: $0
|
||||
height: $1
|
||||
}
|
||||
}
|
||||
)pb",
|
||||
input_width, input_height));
|
||||
mediapipe::CalculatorRunner runner(calculator_node);
|
||||
|
||||
// Input frame.
|
||||
const auto input_frame = GetInputFrame(input_width, input_height, 3);
|
||||
auto input_frame_packet =
|
||||
mediapipe::MakePacket<mediapipe::ImageFrame>(std::move(*input_frame));
|
||||
runner.MutableInputs()->Tag("IMAGE").packets.push_back(
|
||||
input_frame_packet.At(mediapipe::Timestamp(1)));
|
||||
|
||||
MP_ASSERT_OK(runner.Run());
|
||||
|
||||
const auto& outputs = runner.Outputs();
|
||||
EXPECT_EQ(outputs.NumEntries(), 1);
|
||||
const auto& output_image =
|
||||
outputs.Tag("IMAGE").packets[0].Get<mediapipe::ImageFrame>();
|
||||
|
||||
const auto expected_output = GetInputFrame(input_width, input_height, 3);
|
||||
cv::Mat output_mat = formats::MatView(&output_image);
|
||||
cv::Mat expected_mat = formats::MatView(expected_output.get());
|
||||
double max_diff = cv::norm(expected_mat, output_mat, cv::NORM_INF);
|
||||
EXPECT_EQ(max_diff, 0);
|
||||
} // TEST
|
||||
|
||||
// Test identity function on GPU, where cropping size is same as input size.
|
||||
TEST(ImageCroppingCalculatorTest, IdentityFunctionCropWithOriginalSizeGPU) {
|
||||
mediapipe::CalculatorGraphConfig config =
|
||||
ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(absl::Substitute(
|
||||
R"pb(
|
||||
input_stream: "input_frames"
|
||||
node {
|
||||
calculator: "ImageFrameToGpuBufferCalculator"
|
||||
input_stream: "input_frames"
|
||||
output_stream: "input_frames_gpu"
|
||||
}
|
||||
node {
|
||||
calculator: "ImageCroppingCalculator"
|
||||
input_stream: "IMAGE_GPU:input_frames_gpu"
|
||||
output_stream: "IMAGE_GPU:cropped_output_frames_gpu"
|
||||
options: {
|
||||
[mediapipe.ImageCroppingCalculatorOptions.ext] {
|
||||
width: $0
|
||||
height: $1
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
calculator: "GpuBufferToImageFrameCalculator"
|
||||
input_stream: "cropped_output_frames_gpu"
|
||||
output_stream: "cropped_output_frames"
|
||||
}
|
||||
)pb",
|
||||
input_width, input_height));
|
||||
|
||||
std::vector<Packet> output_packets;
|
||||
tool::AddVectorSink("cropped_output_frames", &config, &output_packets);
|
||||
|
||||
CalculatorGraph graph;
|
||||
MP_ASSERT_OK(graph.Initialize(config));
|
||||
|
||||
// Input frame.
|
||||
const auto input_frame = GetInputFrame(input_width, input_height, 4);
|
||||
auto input_frame_packet =
|
||||
mediapipe::MakePacket<mediapipe::ImageFrame>(std::move(*input_frame));
|
||||
|
||||
MP_ASSERT_OK(graph.StartRun({}));
|
||||
MP_ASSERT_OK(graph.AddPacketToInputStream(
|
||||
"input_frames", input_frame_packet.At(mediapipe::Timestamp(1))));
|
||||
MP_ASSERT_OK(graph.WaitUntilIdle());
|
||||
|
||||
// Get and process results.
|
||||
const ImageFrame& output_image = output_packets[0].Get<ImageFrame>();
|
||||
std::cout << output_image.Width();
|
||||
|
||||
const auto expected_output = GetInputFrame(input_width, input_height, 4);
|
||||
cv::Mat output_mat = formats::MatView(&output_image);
|
||||
cv::Mat expected_mat = formats::MatView(expected_output.get());
|
||||
double max_diff = cv::norm(expected_mat, output_mat, cv::NORM_INF);
|
||||
|
||||
EXPECT_EQ(max_diff, 0);
|
||||
} // TEST
|
||||
|
||||
// Test normal case, where norm_width and norm_height in options are set.
|
||||
TEST(ImageCroppingCalculatorTest, GetCroppingDimensionsNormal) {
|
||||
auto calculator_node =
|
||||
|
||||
@@ -207,6 +207,7 @@ class ImageTransformationCalculator : public CalculatorBase {
|
||||
bool flip_vertically_ = false;
|
||||
|
||||
bool use_gpu_ = false;
|
||||
cv::Scalar padding_color_;
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
GlCalculatorHelper gpu_helper_;
|
||||
std::unique_ptr<QuadRenderer> rgb_renderer_;
|
||||
@@ -338,6 +339,9 @@ absl::Status ImageTransformationCalculator::Open(CalculatorContext* cc) {
|
||||
}
|
||||
|
||||
scale_mode_ = ParseScaleMode(options_.scale_mode(), DEFAULT_SCALE_MODE);
|
||||
padding_color_ = cv::Scalar(options_.padding_color().red(),
|
||||
options_.padding_color().green(),
|
||||
options_.padding_color().blue());
|
||||
|
||||
if (use_gpu_) {
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
@@ -480,7 +484,8 @@ absl::Status ImageTransformationCalculator::RenderCpu(CalculatorContext* cc) {
|
||||
cv::copyMakeBorder(intermediate_mat, scaled_mat, top, bottom, left,
|
||||
right,
|
||||
options_.constant_padding() ? cv::BORDER_CONSTANT
|
||||
: cv::BORDER_REPLICATE);
|
||||
: cv::BORDER_REPLICATE,
|
||||
padding_color_);
|
||||
} else {
|
||||
cv::resize(input_mat, scaled_mat, cv::Size(target_width, target_height),
|
||||
0, 0, scale_flag);
|
||||
|
||||
@@ -20,11 +20,21 @@ import "mediapipe/calculators/image/rotation_mode.proto";
|
||||
import "mediapipe/framework/calculator.proto";
|
||||
import "mediapipe/gpu/scale_mode.proto";
|
||||
|
||||
option java_package = "com.google.mediapipe.calculator.proto";
|
||||
option java_outer_classname = "ImageTransformationCalculatorOptionsProto";
|
||||
|
||||
message ImageTransformationCalculatorOptions {
|
||||
extend CalculatorOptions {
|
||||
optional ImageTransformationCalculatorOptions ext = 251952830;
|
||||
}
|
||||
|
||||
// RGB values in range of 0 - 255
|
||||
message Color {
|
||||
optional int32 red = 1 [default = 0];
|
||||
optional int32 green = 2 [default = 0];
|
||||
optional int32 blue = 3 [default = 0];
|
||||
}
|
||||
|
||||
// Output dimensions. Set to 0 if they should be the same as the input.
|
||||
optional int32 output_width = 1 [default = 0];
|
||||
optional int32 output_height = 2 [default = 0];
|
||||
@@ -40,4 +50,8 @@ message ImageTransformationCalculatorOptions {
|
||||
// Default is to use BORDER_CONSTANT. If set to false, it will use
|
||||
// BORDER_REPLICATE instead.
|
||||
optional bool constant_padding = 7 [default = true];
|
||||
|
||||
// The color for the padding. This option is only used when the scale mode is
|
||||
// FIT. Default is black. This is for CPU only.
|
||||
optional Color padding_color = 8;
|
||||
}
|
||||
|
||||
@@ -47,10 +47,11 @@ constexpr int kNumChannelsRGBA = 4;
|
||||
|
||||
enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
|
||||
|
||||
// Combines an RGB cv::Mat and a single-channel alpha cv::Mat of the same
|
||||
// dimensions into an RGBA cv::Mat. Alpha may be read as uint8 or as another
|
||||
// numeric type; in the latter case, it is upscaled to values between 0 and 255
|
||||
// from an assumed input range of [0, 1). RGB and RGBA Mat's must be uchar.
|
||||
// Combines an RGB cv::Mat and an alpha cv::Mat of the same dimensions into an
|
||||
// RGBA cv::Mat. Alpha may be read as uint8 or as another numeric type; in the
|
||||
// latter case, it is upscaled to values between 0 and 255 from an assumed input
|
||||
// range of [0, 1). Only the first channel of Alpha is used. Input & output Mat
|
||||
// must be uchar.
|
||||
template <typename AlphaType>
|
||||
absl::Status MergeRGBA8Image(const cv::Mat input_mat, const cv::Mat& alpha_mat,
|
||||
cv::Mat& output_mat) {
|
||||
@@ -71,9 +72,9 @@ absl::Status MergeRGBA8Image(const cv::Mat input_mat, const cv::Mat& alpha_mat,
|
||||
out_ptr[out_idx + 1] = in_ptr[in_idx + 1];
|
||||
out_ptr[out_idx + 2] = in_ptr[in_idx + 2];
|
||||
if constexpr (std::is_same<AlphaType, uchar>::value) {
|
||||
out_ptr[out_idx + 3] = alpha_ptr[alpha_idx + 0];
|
||||
out_ptr[out_idx + 3] = alpha_ptr[alpha_idx + 0]; // channel 0 of mask
|
||||
} else {
|
||||
const AlphaType alpha = alpha_ptr[alpha_idx + 0];
|
||||
const AlphaType alpha = alpha_ptr[alpha_idx + 0]; // channel 0 of mask
|
||||
out_ptr[out_idx + 3] = static_cast<uchar>(round(alpha * 255.0f));
|
||||
}
|
||||
}
|
||||
@@ -291,8 +292,8 @@ absl::Status SetAlphaCalculator::RenderCpu(CalculatorContext* cc) {
|
||||
const auto& alpha_mask = cc->Inputs().Tag(kInputAlphaTag).Get<ImageFrame>();
|
||||
cv::Mat alpha_mat = mediapipe::formats::MatView(&alpha_mask);
|
||||
|
||||
const bool alpha_is_float = alpha_mat.type() == CV_32FC1;
|
||||
RET_CHECK(alpha_is_float || alpha_mat.type() == CV_8UC1);
|
||||
const bool alpha_is_float = CV_MAT_DEPTH(alpha_mat.type()) == CV_32F;
|
||||
RET_CHECK(alpha_is_float || CV_MAT_DEPTH(alpha_mat.type()) == CV_8U);
|
||||
|
||||
if (alpha_is_float) {
|
||||
MP_RETURN_IF_ERROR(
|
||||
|
||||
@@ -21,6 +21,8 @@ load(
|
||||
)
|
||||
load("//mediapipe/framework:mediapipe_cc_test.bzl", "mediapipe_cc_test")
|
||||
load("//mediapipe/framework:encode_binary_proto.bzl", "encode_binary_proto")
|
||||
load("@org_tensorflow//tensorflow/lite/core/shims:cc_library_with_tflite.bzl", "cc_library_with_tflite")
|
||||
load("//mediapipe/framework:more_selects.bzl", "more_selects")
|
||||
|
||||
licenses(["notice"])
|
||||
|
||||
@@ -34,6 +36,27 @@ exports_files(
|
||||
],
|
||||
)
|
||||
|
||||
config_setting(
|
||||
name = "disable_gpu",
|
||||
define_values = {
|
||||
"MEDIAPIPE_DISABLE_GPU": "1",
|
||||
},
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
|
||||
more_selects.config_setting_negation(
|
||||
name = "not_disable_gpu",
|
||||
negate = ":disable_gpu",
|
||||
)
|
||||
|
||||
selects.config_setting_group(
|
||||
name = "platform_ios_with_gpu",
|
||||
match_all = [
|
||||
":not_disable_gpu",
|
||||
"//mediapipe:ios",
|
||||
],
|
||||
)
|
||||
|
||||
selects.config_setting_group(
|
||||
name = "compute_shader_unavailable",
|
||||
match_any = [
|
||||
@@ -370,10 +393,15 @@ mediapipe_proto_library(
|
||||
# size concerns), depend on those implementations directly, and do not depend on
|
||||
# :inference_calculator.
|
||||
# In all cases, use "InferenceCalulator" in your graphs.
|
||||
cc_library(
|
||||
cc_library_with_tflite(
|
||||
name = "inference_calculator_interface",
|
||||
srcs = ["inference_calculator.cc"],
|
||||
hdrs = ["inference_calculator.h"],
|
||||
tflite_deps = [
|
||||
"//mediapipe/util/tflite:tflite_model_loader",
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:framework_stable",
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:builtin_ops",
|
||||
],
|
||||
deps = [
|
||||
":inference_calculator_cc_proto",
|
||||
":inference_calculator_options_lib",
|
||||
@@ -384,12 +412,9 @@ cc_library(
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/stream_handler:fixed_size_input_stream_handler",
|
||||
"//mediapipe/framework/tool:subgraph_expansion",
|
||||
"//mediapipe/util/tflite:tflite_model_loader",
|
||||
"@com_google_absl//absl/memory",
|
||||
"@com_google_absl//absl/strings",
|
||||
"@org_tensorflow//tensorflow/lite:framework",
|
||||
"@org_tensorflow//tensorflow/lite/core/api:op_resolver",
|
||||
"@org_tensorflow//tensorflow/lite/kernels:builtin_ops",
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
@@ -473,22 +498,33 @@ cc_library(
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
cc_library_with_tflite(
|
||||
name = "tflite_delegate_ptr",
|
||||
hdrs = ["tflite_delegate_ptr.h"],
|
||||
tflite_deps = [
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:c_api_types",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library_with_tflite(
|
||||
name = "inference_interpreter_delegate_runner",
|
||||
srcs = ["inference_interpreter_delegate_runner.cc"],
|
||||
hdrs = ["inference_interpreter_delegate_runner.h"],
|
||||
tflite_deps = [
|
||||
":tflite_delegate_ptr",
|
||||
"//mediapipe/util/tflite:tflite_model_loader",
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:c_api_types",
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:framework_stable",
|
||||
],
|
||||
deps = [
|
||||
":inference_runner",
|
||||
"//mediapipe/framework:mediapipe_profiling",
|
||||
"//mediapipe/framework/api2:packet",
|
||||
"//mediapipe/framework/formats:tensor",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/util/tflite:tflite_model_loader",
|
||||
"@com_google_absl//absl/status",
|
||||
"@com_google_absl//absl/status:statusor",
|
||||
"@org_tensorflow//tensorflow/lite:framework_stable",
|
||||
"@org_tensorflow//tensorflow/lite:string_util",
|
||||
"@org_tensorflow//tensorflow/lite/c:c_api_types",
|
||||
"@org_tensorflow//tensorflow/lite/core/api:op_resolver",
|
||||
],
|
||||
)
|
||||
@@ -506,9 +542,9 @@ cc_library(
|
||||
"@com_google_absl//absl/memory",
|
||||
"@com_google_absl//absl/status",
|
||||
"@com_google_absl//absl/status:statusor",
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:c_api_types",
|
||||
"@org_tensorflow//tensorflow/lite/core/shims:framework_stable",
|
||||
"@org_tensorflow//tensorflow/lite/delegates/xnnpack:xnnpack_delegate",
|
||||
"@org_tensorflow//tensorflow/lite:framework_stable",
|
||||
"@org_tensorflow//tensorflow/lite/c:c_api_types",
|
||||
] + select({
|
||||
"//conditions:default": [],
|
||||
"//mediapipe:android": ["@org_tensorflow//tensorflow/lite/delegates/nnapi:nnapi_delegate"],
|
||||
@@ -571,7 +607,7 @@ cc_library(
|
||||
":inference_calculator_cpu",
|
||||
] + select({
|
||||
"//conditions:default": [":inference_calculator_gl_if_compute_shader_available"],
|
||||
"//mediapipe:ios": [":inference_calculator_metal"],
|
||||
":platform_ios_with_gpu": [":inference_calculator_metal"],
|
||||
}),
|
||||
alwayslink = 1,
|
||||
)
|
||||
@@ -1028,6 +1064,7 @@ cc_test(
|
||||
"@com_google_absl//absl/flags:flag",
|
||||
"@com_google_absl//absl/memory",
|
||||
"@com_google_absl//absl/strings",
|
||||
"@com_google_absl//absl/strings:str_format",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -203,6 +203,7 @@ class AudioToTensorCalculator : public Node {
|
||||
std::unique_ptr<audio_dsp::QResampler<float>> resampler_;
|
||||
Matrix sample_buffer_;
|
||||
int processed_buffer_cols_ = 0;
|
||||
double gain_ = 1.0;
|
||||
|
||||
// The internal state of the FFT library.
|
||||
PFFFT_Setup* fft_state_ = nullptr;
|
||||
@@ -278,7 +279,9 @@ absl::Status AudioToTensorCalculator::Open(CalculatorContext* cc) {
|
||||
padding_samples_after_ = options.padding_samples_after();
|
||||
dft_tensor_format_ = options.dft_tensor_format();
|
||||
flush_mode_ = options.flush_mode();
|
||||
|
||||
if (options.has_volume_gain_db()) {
|
||||
gain_ = pow(10, options.volume_gain_db() / 20.0);
|
||||
}
|
||||
RET_CHECK(kAudioSampleRateIn(cc).IsConnected() ^
|
||||
!kAudioIn(cc).Header().IsEmpty())
|
||||
<< "Must either specify the time series header of the \"AUDIO\" stream "
|
||||
@@ -344,6 +347,10 @@ absl::Status AudioToTensorCalculator::Process(CalculatorContext* cc) {
|
||||
const Matrix& input = channels_match ? input_frame
|
||||
// Mono mixdown.
|
||||
: input_frame.colwise().mean();
|
||||
if (gain_ != 1.0) {
|
||||
return stream_mode_ ? ProcessStreamingData(cc, input * gain_)
|
||||
: ProcessNonStreamingData(cc, input * gain_);
|
||||
}
|
||||
return stream_mode_ ? ProcessStreamingData(cc, input)
|
||||
: ProcessNonStreamingData(cc, input);
|
||||
}
|
||||
|
||||
@@ -81,4 +81,8 @@ message AudioToTensorCalculatorOptions {
|
||||
WITH_DC_AND_NYQUIST = 3;
|
||||
}
|
||||
optional DftTensorFormat dft_tensor_format = 11 [default = WITH_NYQUIST];
|
||||
|
||||
// The volume gain, measured in dB.
|
||||
// Scale the input audio amplitude by 10^(volume_gain_db/20).
|
||||
optional double volume_gain_db = 12;
|
||||
}
|
||||
|
||||
@@ -195,8 +195,9 @@ class ImageToTensorCalculator : public Node {
|
||||
#endif // MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
RotatedRect roi = GetRoi(image->width(), image->height(), norm_rect);
|
||||
ASSIGN_OR_RETURN(auto padding, PadRoi(options_.output_tensor_width(),
|
||||
options_.output_tensor_height(),
|
||||
const int tensor_width = params_.output_width.value_or(image->width());
|
||||
const int tensor_height = params_.output_height.value_or(image->height());
|
||||
ASSIGN_OR_RETURN(auto padding, PadRoi(tensor_width, tensor_height,
|
||||
options_.keep_aspect_ratio(), &roi));
|
||||
if (kOutLetterboxPadding(cc).IsConnected()) {
|
||||
kOutLetterboxPadding(cc).Send(padding);
|
||||
@@ -214,9 +215,8 @@ class ImageToTensorCalculator : public Node {
|
||||
|
||||
Tensor::ElementType output_tensor_type =
|
||||
GetOutputTensorType(image->UsesGpu(), params_);
|
||||
Tensor tensor(output_tensor_type,
|
||||
{1, params_.output_height, params_.output_width,
|
||||
GetNumOutputChannels(*image)});
|
||||
Tensor tensor(output_tensor_type, {1, tensor_height, tensor_width,
|
||||
GetNumOutputChannels(*image)});
|
||||
MP_RETURN_IF_ERROR((image->UsesGpu() ? gpu_converter_ : cpu_converter_)
|
||||
->Convert(*image, roi, params_.range_min,
|
||||
params_.range_max,
|
||||
|
||||
@@ -54,6 +54,8 @@ message ImageToTensorCalculatorOptions {
|
||||
BORDER_REPLICATE = 2;
|
||||
}
|
||||
|
||||
// The width and height of output tensor. The output tensor would have the
|
||||
// input image width/height if not set.
|
||||
optional int32 output_tensor_width = 1;
|
||||
optional int32 output_tensor_height = 2;
|
||||
|
||||
|
||||
@@ -13,10 +13,13 @@
|
||||
// limitations under the License.
|
||||
|
||||
#include <cmath>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "absl/flags/flag.h"
|
||||
#include "absl/memory/memory.h"
|
||||
#include "absl/strings/str_format.h"
|
||||
#include "absl/strings/substitute.h"
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_converter.h"
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_utils.h"
|
||||
@@ -51,13 +54,12 @@ std::string GetFilePath(absl::string_view filename) {
|
||||
|
||||
// Image to tensor test template.
|
||||
// No processing/assertions should be done after the function is invoked.
|
||||
void RunTestWithInputImagePacket(const Packet& input_image_packet,
|
||||
cv::Mat expected_result, float range_min,
|
||||
float range_max, int tensor_width,
|
||||
int tensor_height, bool keep_aspect,
|
||||
absl::optional<BorderMode> border_mode,
|
||||
const mediapipe::NormalizedRect& roi,
|
||||
bool output_int_tensor) {
|
||||
void RunTestWithInputImagePacket(
|
||||
const Packet& input_image_packet, cv::Mat expected_result, float range_min,
|
||||
float range_max, std::optional<int> tensor_width,
|
||||
std::optional<int> tensor_height, bool keep_aspect,
|
||||
absl::optional<BorderMode> border_mode,
|
||||
const mediapipe::NormalizedRect& roi, bool output_int_tensor) {
|
||||
std::string border_mode_str;
|
||||
if (border_mode) {
|
||||
switch (*border_mode) {
|
||||
@@ -93,8 +95,9 @@ void RunTestWithInputImagePacket(const Packet& input_image_packet,
|
||||
})",
|
||||
range_min, range_max);
|
||||
}
|
||||
auto graph_config = mediapipe::ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
absl::Substitute(R"(
|
||||
auto graph_config =
|
||||
mediapipe::ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
|
||||
R"(
|
||||
input_stream: "input_image"
|
||||
input_stream: "roi"
|
||||
node {
|
||||
@@ -104,8 +107,8 @@ void RunTestWithInputImagePacket(const Packet& input_image_packet,
|
||||
output_stream: "TENSORS:tensor"
|
||||
options {
|
||||
[mediapipe.ImageToTensorCalculatorOptions.ext] {
|
||||
output_tensor_width: $0
|
||||
output_tensor_height: $1
|
||||
$0 # output tensor width
|
||||
$1 # output tensor height
|
||||
keep_aspect_ratio: $2
|
||||
$3 # output range
|
||||
$4 # border mode
|
||||
@@ -113,11 +116,16 @@ void RunTestWithInputImagePacket(const Packet& input_image_packet,
|
||||
}
|
||||
}
|
||||
)",
|
||||
/*$0=*/tensor_width,
|
||||
/*$1=*/tensor_height,
|
||||
/*$2=*/keep_aspect ? "true" : "false",
|
||||
/*$3=*/output_tensor_range,
|
||||
/*$4=*/border_mode_str));
|
||||
/*$0=*/tensor_width.has_value()
|
||||
? absl::StrFormat("output_tensor_width: %d", tensor_width.value())
|
||||
: "",
|
||||
/*$1=*/tensor_height.has_value()
|
||||
? absl::StrFormat("output_tensor_height: %d",
|
||||
tensor_height.value())
|
||||
: "",
|
||||
/*$2=*/keep_aspect ? "true" : "false",
|
||||
/*$3=*/output_tensor_range,
|
||||
/*$4=*/border_mode_str));
|
||||
|
||||
std::vector<Packet> output_packets;
|
||||
tool::AddVectorSink("tensor", &graph_config, &output_packets);
|
||||
@@ -149,18 +157,18 @@ void RunTestWithInputImagePacket(const Packet& input_image_packet,
|
||||
if (output_int_tensor) {
|
||||
if (range_min < 0) {
|
||||
EXPECT_EQ(tensor.element_type(), Tensor::ElementType::kInt8);
|
||||
tensor_mat = cv::Mat(tensor_height, tensor_width,
|
||||
tensor_mat = cv::Mat(expected_result.rows, expected_result.cols,
|
||||
channels == 1 ? CV_8SC1 : CV_8SC3,
|
||||
const_cast<int8*>(view.buffer<int8>()));
|
||||
} else {
|
||||
EXPECT_EQ(tensor.element_type(), Tensor::ElementType::kUInt8);
|
||||
tensor_mat = cv::Mat(tensor_height, tensor_width,
|
||||
tensor_mat = cv::Mat(expected_result.rows, expected_result.cols,
|
||||
channels == 1 ? CV_8UC1 : CV_8UC3,
|
||||
const_cast<uint8*>(view.buffer<uint8>()));
|
||||
}
|
||||
} else {
|
||||
EXPECT_EQ(tensor.element_type(), Tensor::ElementType::kFloat32);
|
||||
tensor_mat = cv::Mat(tensor_height, tensor_width,
|
||||
tensor_mat = cv::Mat(expected_result.rows, expected_result.cols,
|
||||
channels == 1 ? CV_32FC1 : CV_32FC3,
|
||||
const_cast<float*>(view.buffer<float>()));
|
||||
}
|
||||
@@ -216,9 +224,9 @@ const std::vector<InputType> kInputTypesToTest = {InputType::kImageFrame,
|
||||
|
||||
void RunTest(cv::Mat input, cv::Mat expected_result,
|
||||
std::vector<std::pair<float, float>> float_ranges,
|
||||
std::vector<std::pair<int, int>> int_ranges, int tensor_width,
|
||||
int tensor_height, bool keep_aspect,
|
||||
absl::optional<BorderMode> border_mode,
|
||||
std::vector<std::pair<int, int>> int_ranges,
|
||||
std::optional<int> tensor_width, std::optional<int> tensor_height,
|
||||
bool keep_aspect, absl::optional<BorderMode> border_mode,
|
||||
const mediapipe::NormalizedRect& roi) {
|
||||
for (auto input_type : kInputTypesToTest) {
|
||||
for (auto float_range : float_ranges) {
|
||||
@@ -486,5 +494,18 @@ TEST(ImageToTensorCalculatorTest, NoOpExceptRangeBorderZero) {
|
||||
BorderMode::kZero, roi);
|
||||
}
|
||||
|
||||
TEST(ImageToTensorCalculatorTest, NoOpExceptRangeAndUseInputImageDims) {
|
||||
mediapipe::NormalizedRect roi;
|
||||
roi.set_x_center(0.5f);
|
||||
roi.set_y_center(0.5f);
|
||||
roi.set_width(1.0f);
|
||||
roi.set_height(1.0f);
|
||||
RunTest(GetRgb(GetFilePath("input.jpg")),
|
||||
GetRgb(GetFilePath("noop_except_range.png")),
|
||||
/*float_ranges=*/{{-1.0f, 1.0f}},
|
||||
/*int_ranges=*/{{0, 255}, {-128, 127}},
|
||||
/*tensor_width=*/std::nullopt, /*tensor_height=*/std::nullopt,
|
||||
/*keep_aspect=*/false, BorderMode::kZero, roi);
|
||||
}
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
#define MEDIAPIPE_CALCULATORS_TENSOR_IMAGE_TO_TENSOR_UTILS_H_
|
||||
|
||||
#include <array>
|
||||
#include <optional>
|
||||
|
||||
#include "absl/types/optional.h"
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_calculator.pb.h"
|
||||
@@ -51,8 +52,8 @@ enum class BorderMode { kZero, kReplicate };
|
||||
// Struct that host commonly accessed parameters used in the
|
||||
// ImageTo[Batch]TensorCalculator.
|
||||
struct OutputTensorParams {
|
||||
int output_height;
|
||||
int output_width;
|
||||
std::optional<int> output_height;
|
||||
std::optional<int> output_width;
|
||||
int output_batch;
|
||||
bool is_float_output;
|
||||
float range_min;
|
||||
@@ -161,10 +162,14 @@ absl::Status ValidateOptionOutputDims(const T& options) {
|
||||
<< "The maximum of the output int tensor range must be less than or "
|
||||
"equal to 127.";
|
||||
}
|
||||
RET_CHECK_GT(options.output_tensor_width(), 0)
|
||||
<< "Valid output tensor width is required.";
|
||||
RET_CHECK_GT(options.output_tensor_height(), 0)
|
||||
<< "Valid output tensor height is required.";
|
||||
if (options.has_output_tensor_width()) {
|
||||
RET_CHECK_GT(options.output_tensor_width(), 0)
|
||||
<< "Valid output tensor width is required.";
|
||||
}
|
||||
if (options.has_output_tensor_height()) {
|
||||
RET_CHECK_GT(options.output_tensor_height(), 0)
|
||||
<< "Valid output tensor height is required.";
|
||||
}
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
@@ -185,8 +190,12 @@ OutputTensorParams GetOutputTensorParams(const T& options) {
|
||||
params.range_min = options.output_tensor_float_range().min();
|
||||
params.range_max = options.output_tensor_float_range().max();
|
||||
}
|
||||
params.output_width = options.output_tensor_width();
|
||||
params.output_height = options.output_tensor_height();
|
||||
if (options.has_output_tensor_width()) {
|
||||
params.output_width = options.output_tensor_width();
|
||||
}
|
||||
if (options.has_output_tensor_height()) {
|
||||
params.output_height = options.output_tensor_height();
|
||||
}
|
||||
params.is_float_output = options.has_output_tensor_float_range();
|
||||
params.output_batch = 1;
|
||||
return params;
|
||||
|
||||
@@ -14,6 +14,8 @@
|
||||
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_utils.h"
|
||||
|
||||
#include <optional>
|
||||
|
||||
#include "mediapipe/framework/formats/rect.pb.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
@@ -172,6 +174,10 @@ constexpr char kValidIntProto[] = R"(
|
||||
output_tensor_height: 200
|
||||
)";
|
||||
|
||||
constexpr char kValidNoTensorDimsProto[] = R"(
|
||||
output_tensor_float_range { min: 0 max: 255 }
|
||||
)";
|
||||
|
||||
TEST(ValidateOptionOutputDims, ImageToTensorCalcOptions) {
|
||||
const auto float_options =
|
||||
mediapipe::ParseTextProtoOrDie<mediapipe::ImageToTensorCalculatorOptions>(
|
||||
@@ -193,13 +199,6 @@ TEST(ValidateOptionOutputDims, EmptyProto) {
|
||||
ValidateOptionOutputDims(options),
|
||||
StatusIs(absl::StatusCode::kInternal,
|
||||
HasSubstr("Valid output float tensor range is required")));
|
||||
|
||||
// Output width/height is not set.
|
||||
options.mutable_output_tensor_float_range()->set_min(0.0);
|
||||
options.mutable_output_tensor_float_range()->set_max(1.0);
|
||||
EXPECT_THAT(ValidateOptionOutputDims(options),
|
||||
StatusIs(absl::StatusCode::kInternal,
|
||||
HasSubstr("Valid output tensor width is required")));
|
||||
}
|
||||
|
||||
TEST(GetOutputTensorParams, ImageToTensorCalcOptionsSetValues) {
|
||||
@@ -215,6 +214,20 @@ TEST(GetOutputTensorParams, ImageToTensorCalcOptionsSetValues) {
|
||||
EXPECT_EQ(params2.output_height, 200);
|
||||
}
|
||||
|
||||
TEST(GetOutputTensorParams, ImageToTensorCalcOptionsNoTensorDims) {
|
||||
// Test valid option for ImageToTensorCalculatorOptions without output
|
||||
// width/height.
|
||||
const auto options =
|
||||
mediapipe::ParseTextProtoOrDie<mediapipe::ImageToTensorCalculatorOptions>(
|
||||
kValidNoTensorDimsProto);
|
||||
const auto params3 = GetOutputTensorParams(options);
|
||||
EXPECT_EQ(params3.range_min, 0.0f);
|
||||
EXPECT_EQ(params3.range_max, 255.0f);
|
||||
EXPECT_EQ(params3.output_batch, 1);
|
||||
EXPECT_EQ(params3.output_width, std::nullopt);
|
||||
EXPECT_EQ(params3.output_height, std::nullopt);
|
||||
}
|
||||
|
||||
TEST(GetBorderMode, GetBorderMode) {
|
||||
// Default to REPLICATE.
|
||||
auto border_mode =
|
||||
|
||||
@@ -63,6 +63,10 @@ class InferenceCalculatorSelectorImpl
|
||||
for (const auto& suffix : impls) {
|
||||
const auto impl = absl::StrCat("InferenceCalculator", suffix);
|
||||
if (!mediapipe::CalculatorBaseRegistry::IsRegistered(impl)) continue;
|
||||
VLOG(1) << "Using " << suffix << " for InferenceCalculator with "
|
||||
<< (options.has_model_path()
|
||||
? "model " + options.model_path()
|
||||
: "output_stream " + subgraph_node.output_stream(0));
|
||||
CalculatorGraphConfig::Node impl_node = subgraph_node;
|
||||
impl_node.set_calculator(impl);
|
||||
return tool::MakeSingleNodeGraph(std::move(impl_node));
|
||||
@@ -90,8 +94,8 @@ InferenceCalculator::GetOpResolverAsPacket(CalculatorContext* cc) {
|
||||
return kSideInCustomOpResolver(cc).As<tflite::OpResolver>();
|
||||
}
|
||||
return PacketAdopting<tflite::OpResolver>(
|
||||
std::make_unique<
|
||||
tflite::ops::builtin::BuiltinOpResolverWithoutDefaultDelegates>());
|
||||
std::make_unique<tflite_shims::ops::builtin::
|
||||
BuiltinOpResolverWithoutDefaultDelegates>());
|
||||
}
|
||||
|
||||
} // namespace api2
|
||||
|
||||
@@ -26,7 +26,7 @@
|
||||
#include "mediapipe/framework/formats/tensor.h"
|
||||
#include "mediapipe/util/tflite/tflite_model_loader.h"
|
||||
#include "tensorflow/lite/core/api/op_resolver.h"
|
||||
#include "tensorflow/lite/kernels/register.h"
|
||||
#include "tensorflow/lite/core/shims/cc/kernels/register.h"
|
||||
|
||||
namespace mediapipe {
|
||||
namespace api2 {
|
||||
@@ -97,8 +97,8 @@ class InferenceCalculator : public NodeIntf {
|
||||
// Deprecated. Prefers to use "OP_RESOLVER" input side packet instead.
|
||||
// TODO: Removes the "CUSTOM_OP_RESOLVER" side input after the
|
||||
// migration.
|
||||
static constexpr SideInput<tflite::ops::builtin::BuiltinOpResolver>::Optional
|
||||
kSideInCustomOpResolver{"CUSTOM_OP_RESOLVER"};
|
||||
static constexpr SideInput<tflite_shims::ops::builtin::BuiltinOpResolver>::
|
||||
Optional kSideInCustomOpResolver{"CUSTOM_OP_RESOLVER"};
|
||||
static constexpr SideInput<tflite::OpResolver>::Optional kSideInOpResolver{
|
||||
"OP_RESOLVER"};
|
||||
static constexpr SideInput<TfLiteModelPtr>::Optional kSideInModel{"MODEL"};
|
||||
@@ -112,7 +112,8 @@ class InferenceCalculator : public NodeIntf {
|
||||
|
||||
protected:
|
||||
using TfLiteDelegatePtr =
|
||||
std::unique_ptr<TfLiteDelegate, std::function<void(TfLiteDelegate*)>>;
|
||||
std::unique_ptr<TfLiteOpaqueDelegate,
|
||||
std::function<void(TfLiteOpaqueDelegate*)>>;
|
||||
|
||||
static absl::StatusOr<Packet<TfLiteModelPtr>> GetModelAsPacket(
|
||||
CalculatorContext* cc);
|
||||
|
||||
@@ -24,7 +24,7 @@
|
||||
#include "mediapipe/calculators/tensor/inference_calculator_utils.h"
|
||||
#include "mediapipe/calculators/tensor/inference_interpreter_delegate_runner.h"
|
||||
#include "mediapipe/calculators/tensor/inference_runner.h"
|
||||
#include "tensorflow/lite/interpreter.h"
|
||||
#include "tensorflow/lite/core/shims/cc/interpreter.h"
|
||||
#if defined(MEDIAPIPE_ANDROID)
|
||||
#include "tensorflow/lite/delegates/nnapi/nnapi_delegate.h"
|
||||
#endif // ANDROID
|
||||
@@ -115,7 +115,7 @@ InferenceCalculatorCpuImpl::MaybeCreateDelegate(CalculatorContext* cc) {
|
||||
const bool opts_has_delegate =
|
||||
calculator_opts.has_delegate() || !kDelegate(cc).IsEmpty();
|
||||
if (opts_has_delegate && opts_delegate.has_tflite()) {
|
||||
// Default tflite inference requeqsted - no need to modify graph.
|
||||
// Default tflite inference requested - no need to modify graph.
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
|
||||
@@ -22,9 +22,9 @@
|
||||
#include "mediapipe/framework/formats/tensor.h"
|
||||
#include "mediapipe/framework/mediapipe_profiling.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "tensorflow/lite/c/c_api_types.h"
|
||||
#include "tensorflow/lite/interpreter.h"
|
||||
#include "tensorflow/lite/interpreter_builder.h"
|
||||
#include "tensorflow/lite/core/shims/c/c_api_types.h"
|
||||
#include "tensorflow/lite/core/shims/cc/interpreter.h"
|
||||
#include "tensorflow/lite/core/shims/cc/interpreter_builder.h"
|
||||
#include "tensorflow/lite/string_util.h"
|
||||
|
||||
#define PERFETTO_TRACK_EVENT_NAMESPACE mediapipe
|
||||
@@ -33,9 +33,12 @@ namespace mediapipe {
|
||||
|
||||
namespace {
|
||||
|
||||
using Interpreter = ::tflite_shims::Interpreter;
|
||||
using InterpreterBuilder = ::tflite_shims::InterpreterBuilder;
|
||||
|
||||
template <typename T>
|
||||
void CopyTensorBufferToInterpreter(const Tensor& input_tensor,
|
||||
tflite::Interpreter* interpreter,
|
||||
Interpreter* interpreter,
|
||||
int input_tensor_index) {
|
||||
auto input_tensor_view = input_tensor.GetCpuReadView();
|
||||
auto input_tensor_buffer = input_tensor_view.buffer<T>();
|
||||
@@ -46,7 +49,7 @@ void CopyTensorBufferToInterpreter(const Tensor& input_tensor,
|
||||
|
||||
template <>
|
||||
void CopyTensorBufferToInterpreter<char>(const Tensor& input_tensor,
|
||||
tflite::Interpreter* interpreter,
|
||||
Interpreter* interpreter,
|
||||
int input_tensor_index) {
|
||||
const char* input_tensor_buffer =
|
||||
input_tensor.GetCpuReadView().buffer<char>();
|
||||
@@ -58,7 +61,7 @@ void CopyTensorBufferToInterpreter<char>(const Tensor& input_tensor,
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
void CopyTensorBufferFromInterpreter(tflite::Interpreter* interpreter,
|
||||
void CopyTensorBufferFromInterpreter(Interpreter* interpreter,
|
||||
int output_tensor_index,
|
||||
Tensor* output_tensor) {
|
||||
auto output_tensor_view = output_tensor->GetCpuWriteView();
|
||||
@@ -73,10 +76,9 @@ void CopyTensorBufferFromInterpreter(tflite::Interpreter* interpreter,
|
||||
|
||||
class InferenceInterpreterDelegateRunner : public InferenceRunner {
|
||||
public:
|
||||
InferenceInterpreterDelegateRunner(
|
||||
api2::Packet<TfLiteModelPtr> model,
|
||||
std::unique_ptr<tflite::Interpreter> interpreter,
|
||||
TfLiteDelegatePtr delegate)
|
||||
InferenceInterpreterDelegateRunner(api2::Packet<TfLiteModelPtr> model,
|
||||
std::unique_ptr<Interpreter> interpreter,
|
||||
TfLiteDelegatePtr delegate)
|
||||
: model_(std::move(model)),
|
||||
interpreter_(std::move(interpreter)),
|
||||
delegate_(std::move(delegate)) {}
|
||||
@@ -86,7 +88,7 @@ class InferenceInterpreterDelegateRunner : public InferenceRunner {
|
||||
|
||||
private:
|
||||
api2::Packet<TfLiteModelPtr> model_;
|
||||
std::unique_ptr<tflite::Interpreter> interpreter_;
|
||||
std::unique_ptr<Interpreter> interpreter_;
|
||||
TfLiteDelegatePtr delegate_;
|
||||
};
|
||||
|
||||
@@ -197,8 +199,7 @@ CreateInferenceInterpreterDelegateRunner(
|
||||
api2::Packet<TfLiteModelPtr> model,
|
||||
api2::Packet<tflite::OpResolver> op_resolver, TfLiteDelegatePtr delegate,
|
||||
int interpreter_num_threads) {
|
||||
tflite::InterpreterBuilder interpreter_builder(*model.Get(),
|
||||
op_resolver.Get());
|
||||
InterpreterBuilder interpreter_builder(*model.Get(), op_resolver.Get());
|
||||
if (delegate) {
|
||||
interpreter_builder.AddDelegate(delegate.get());
|
||||
}
|
||||
@@ -207,7 +208,7 @@ CreateInferenceInterpreterDelegateRunner(
|
||||
#else
|
||||
interpreter_builder.SetNumThreads(interpreter_num_threads);
|
||||
#endif // __EMSCRIPTEN__
|
||||
std::unique_ptr<tflite::Interpreter> interpreter;
|
||||
std::unique_ptr<Interpreter> interpreter;
|
||||
RET_CHECK_EQ(interpreter_builder(&interpreter), kTfLiteOk);
|
||||
RET_CHECK(interpreter);
|
||||
RET_CHECK_EQ(interpreter->AllocateTensors(), kTfLiteOk);
|
||||
|
||||
@@ -20,16 +20,14 @@
|
||||
|
||||
#include "absl/status/statusor.h"
|
||||
#include "mediapipe/calculators/tensor/inference_runner.h"
|
||||
#include "mediapipe/calculators/tensor/tflite_delegate_ptr.h"
|
||||
#include "mediapipe/framework/api2/packet.h"
|
||||
#include "mediapipe/util/tflite/tflite_model_loader.h"
|
||||
#include "tensorflow/lite/core/api/op_resolver.h"
|
||||
#include "tensorflow/lite/interpreter.h"
|
||||
#include "tensorflow/lite/core/shims/c/c_api_types.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
using TfLiteDelegatePtr =
|
||||
std::unique_ptr<TfLiteDelegate, std::function<void(TfLiteDelegate*)>>;
|
||||
|
||||
// Creates inference runner which run inference using newly initialized
|
||||
// interpreter and provided `delegate`.
|
||||
//
|
||||
|
||||
@@ -34,6 +34,8 @@ namespace mediapipe {
|
||||
namespace api2 {
|
||||
namespace {
|
||||
|
||||
using Options = ::mediapipe::TensorsToAudioCalculatorOptions;
|
||||
|
||||
std::vector<float> HannWindow(int window_size, bool sqrt_hann) {
|
||||
std::vector<float> hann_window(window_size);
|
||||
audio_dsp::HannWindow().GetPeriodicSamples(window_size, &hann_window);
|
||||
@@ -135,11 +137,19 @@ class TensorsToAudioCalculator : public Node {
|
||||
// pffft requires memory to work with to avoid using the stack.
|
||||
std::vector<float, Eigen::aligned_allocator<float>> fft_workplace_;
|
||||
std::vector<float, Eigen::aligned_allocator<float>> fft_output_;
|
||||
std::vector<float, Eigen::aligned_allocator<float>> prev_fft_output_;
|
||||
int overlapping_samples_ = -1;
|
||||
int step_samples_ = -1;
|
||||
Options::DftTensorFormat dft_tensor_format_;
|
||||
double gain_ = 1.0;
|
||||
};
|
||||
|
||||
absl::Status TensorsToAudioCalculator::Open(CalculatorContext* cc) {
|
||||
const auto& options =
|
||||
cc->Options<mediapipe::TensorsToAudioCalculatorOptions>();
|
||||
dft_tensor_format_ = options.dft_tensor_format();
|
||||
RET_CHECK(dft_tensor_format_ != Options::DFT_TENSOR_FORMAT_UNKNOWN)
|
||||
<< "dft tensor format must be specified.";
|
||||
RET_CHECK(options.has_fft_size()) << "FFT size must be specified.";
|
||||
RET_CHECK(IsValidFftSize(options.fft_size()))
|
||||
<< "FFT size must be of the form fft_size = (2^a)*(3^b)*(5^c) where b "
|
||||
@@ -153,6 +163,25 @@ absl::Status TensorsToAudioCalculator::Open(CalculatorContext* cc) {
|
||||
fft_input_buffer_.resize(fft_size_);
|
||||
fft_workplace_.resize(fft_size_);
|
||||
fft_output_.resize(fft_size_);
|
||||
if (options.has_num_overlapping_samples()) {
|
||||
RET_CHECK(options.has_num_samples() && options.num_samples() > 0)
|
||||
<< "When `num_overlapping_samples` is set, `num_samples` must also be "
|
||||
"specified.";
|
||||
if (options.num_samples() != fft_size_) {
|
||||
return absl::UnimplementedError(
|
||||
"`num_samples` and `fft_size` must be equivalent.");
|
||||
}
|
||||
RET_CHECK(options.num_overlapping_samples() > 0 &&
|
||||
options.num_overlapping_samples() < options.num_samples())
|
||||
<< "`num_overlapping_samples` must be greater than 0 and less than "
|
||||
"`num_samples.`";
|
||||
overlapping_samples_ = options.num_overlapping_samples();
|
||||
step_samples_ = options.num_samples() - options.num_overlapping_samples();
|
||||
prev_fft_output_.resize(fft_size_);
|
||||
}
|
||||
if (options.has_volume_gain_db()) {
|
||||
gain_ = pow(10, options.volume_gain_db() / 20.0);
|
||||
}
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
@@ -164,14 +193,37 @@ absl::Status TensorsToAudioCalculator::Process(CalculatorContext* cc) {
|
||||
RET_CHECK_EQ(input_tensors.size(), 1);
|
||||
RET_CHECK(input_tensors[0].element_type() == Tensor::ElementType::kFloat32);
|
||||
auto view = input_tensors[0].GetCpuReadView();
|
||||
// DC's real part.
|
||||
input_dft_[0] = kDcAndNyquistIn(cc)->first;
|
||||
// Nyquist's real part is the penultimate element of the tensor buffer.
|
||||
// pffft ignores the Nyquist's imagery part. No need to fetch the last value
|
||||
// from the tensor buffer.
|
||||
input_dft_[1] = *(view.buffer<float>() + (fft_size_ - 2));
|
||||
std::memcpy(input_dft_.data() + 2, view.buffer<float>(),
|
||||
(fft_size_ - 2) * sizeof(float));
|
||||
switch (dft_tensor_format_) {
|
||||
case Options::WITH_NYQUIST: {
|
||||
// DC's real part.
|
||||
input_dft_[0] = kDcAndNyquistIn(cc)->first;
|
||||
// Nyquist's real part is the penultimate element of the tensor buffer.
|
||||
// pffft ignores the Nyquist's imagery part. No need to fetch the last
|
||||
// value from the tensor buffer.
|
||||
input_dft_[1] = *(view.buffer<float>() + (fft_size_ - 2));
|
||||
std::memcpy(input_dft_.data() + 2, view.buffer<float>(),
|
||||
(fft_size_ - 2) * sizeof(float));
|
||||
break;
|
||||
}
|
||||
case Options::WITH_DC_AND_NYQUIST: {
|
||||
// DC's real part is the first element of the tensor buffer.
|
||||
input_dft_[0] = *(view.buffer<float>());
|
||||
// Nyquist's real part is the penultimate element of the tensor buffer.
|
||||
input_dft_[1] = *(view.buffer<float>() + fft_size_);
|
||||
std::memcpy(input_dft_.data() + 2, view.buffer<float>() + 2,
|
||||
(fft_size_ - 2) * sizeof(float));
|
||||
break;
|
||||
}
|
||||
case Options::WITHOUT_DC_AND_NYQUIST: {
|
||||
input_dft_[0] = kDcAndNyquistIn(cc)->first;
|
||||
input_dft_[1] = kDcAndNyquistIn(cc)->second;
|
||||
std::memcpy(input_dft_.data() + 2, view.buffer<float>(),
|
||||
(fft_size_ - 2) * sizeof(float));
|
||||
break;
|
||||
}
|
||||
default:
|
||||
return absl::InvalidArgumentError("Unsupported dft tensor format.");
|
||||
}
|
||||
pffft_transform_ordered(fft_state_, input_dft_.data(), fft_output_.data(),
|
||||
fft_workplace_.data(), PFFFT_BACKWARD);
|
||||
// Applies the inverse window function.
|
||||
@@ -179,7 +231,18 @@ absl::Status TensorsToAudioCalculator::Process(CalculatorContext* cc) {
|
||||
fft_output_.begin(), fft_output_.end(), inv_fft_window_.begin(),
|
||||
fft_output_.begin(),
|
||||
[this](float a, float b) { return a * b * inverse_fft_size_; });
|
||||
Matrix matrix = Eigen::Map<Matrix>(fft_output_.data(), 1, fft_output_.size());
|
||||
Matrix matrix;
|
||||
if (step_samples_ > 0) {
|
||||
matrix = Eigen::Map<Matrix>(fft_output_.data(), 1, step_samples_);
|
||||
matrix.leftCols(overlapping_samples_) += Eigen::Map<Matrix>(
|
||||
prev_fft_output_.data() + step_samples_, 1, overlapping_samples_);
|
||||
prev_fft_output_.swap(fft_output_);
|
||||
} else {
|
||||
matrix = Eigen::Map<Matrix>(fft_output_.data(), 1, fft_output_.size());
|
||||
}
|
||||
if (gain_ != 1.0) {
|
||||
matrix *= gain_;
|
||||
}
|
||||
kAudioOut(cc).Send(std::move(matrix));
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
@@ -26,4 +26,27 @@ message TensorsToAudioCalculatorOptions {
|
||||
// Size of the fft in number of bins. If set, the calculator will do ifft
|
||||
// on the input tensor.
|
||||
optional int64 fft_size = 1;
|
||||
|
||||
// The number of samples per channel the output audio has.
|
||||
optional int64 num_samples = 2;
|
||||
|
||||
// The number of overlapping samples between adjacent windows.
|
||||
optional int64 num_overlapping_samples = 3 [default = 0];
|
||||
|
||||
enum DftTensorFormat {
|
||||
DFT_TENSOR_FORMAT_UNKNOWN = 0;
|
||||
// The input dft tensor without dc and nyquist components.
|
||||
WITHOUT_DC_AND_NYQUIST = 1;
|
||||
// The input dft tensor contains the nyquist component as the last
|
||||
// two values.
|
||||
WITH_NYQUIST = 2;
|
||||
// The input dft tensor contains the dc component as the first two values
|
||||
// and the nyquist component as the last two values.
|
||||
WITH_DC_AND_NYQUIST = 3;
|
||||
}
|
||||
optional DftTensorFormat dft_tensor_format = 11 [default = WITH_NYQUIST];
|
||||
|
||||
// The volume gain, measured in dB.
|
||||
// Scale the output audio amplitude by 10^(volume_gain_db/20).
|
||||
optional double volume_gain_db = 12;
|
||||
}
|
||||
|
||||
@@ -30,6 +30,8 @@
|
||||
namespace mediapipe {
|
||||
namespace {
|
||||
|
||||
using Options = ::mediapipe::TensorsToAudioCalculatorOptions;
|
||||
|
||||
class TensorsToAudioCalculatorFftTest : public ::testing::Test {
|
||||
protected:
|
||||
// Creates an audio matrix containing a single sample of 1.0 at a specified
|
||||
@@ -40,9 +42,10 @@ class TensorsToAudioCalculatorFftTest : public ::testing::Test {
|
||||
return impulse;
|
||||
}
|
||||
|
||||
void ConfigGraph(int num_samples, double sample_rate, int fft_size) {
|
||||
graph_config_ = ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||
absl::Substitute(R"(
|
||||
void ConfigGraph(int num_samples, double sample_rate, int fft_size,
|
||||
Options::DftTensorFormat dft_tensor_format) {
|
||||
graph_config_ = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
|
||||
R"(
|
||||
input_stream: "audio_in"
|
||||
input_stream: "sample_rate"
|
||||
output_stream: "audio_out"
|
||||
@@ -59,6 +62,7 @@ class TensorsToAudioCalculatorFftTest : public ::testing::Test {
|
||||
num_overlapping_samples: 0
|
||||
target_sample_rate: $1
|
||||
fft_size: $2
|
||||
dft_tensor_format: $3
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -70,13 +74,15 @@ class TensorsToAudioCalculatorFftTest : public ::testing::Test {
|
||||
options {
|
||||
[mediapipe.TensorsToAudioCalculatorOptions.ext] {
|
||||
fft_size: $2
|
||||
dft_tensor_format: $3
|
||||
}
|
||||
}
|
||||
}
|
||||
)",
|
||||
/*$0=*/num_samples,
|
||||
/*$1=*/sample_rate,
|
||||
/*$2=*/fft_size));
|
||||
/*$0=*/num_samples,
|
||||
/*$1=*/sample_rate,
|
||||
/*$2=*/fft_size,
|
||||
/*$3=*/Options::DftTensorFormat_Name(dft_tensor_format)));
|
||||
tool::AddVectorSink("audio_out", &graph_config_, &audio_out_packets_);
|
||||
}
|
||||
|
||||
@@ -97,7 +103,7 @@ class TensorsToAudioCalculatorFftTest : public ::testing::Test {
|
||||
};
|
||||
|
||||
TEST_F(TensorsToAudioCalculatorFftTest, TestInvalidFftSize) {
|
||||
ConfigGraph(320, 16000, 103);
|
||||
ConfigGraph(320, 16000, 103, Options::WITH_NYQUIST);
|
||||
MP_ASSERT_OK(graph_.Initialize(graph_config_));
|
||||
MP_ASSERT_OK(graph_.StartRun({}));
|
||||
auto status = graph_.WaitUntilIdle();
|
||||
@@ -109,8 +115,7 @@ TEST_F(TensorsToAudioCalculatorFftTest, TestInvalidFftSize) {
|
||||
TEST_F(TensorsToAudioCalculatorFftTest, TestImpulseSignalAtTheCenter) {
|
||||
constexpr int sample_size = 320;
|
||||
constexpr double sample_rate = 16000;
|
||||
ConfigGraph(sample_size, sample_rate, 320);
|
||||
|
||||
ConfigGraph(sample_size, sample_rate, 320, Options::WITH_NYQUIST);
|
||||
Matrix impulse_data = CreateImpulseSignalData(sample_size, sample_size / 2);
|
||||
RunGraph(impulse_data, sample_rate);
|
||||
ASSERT_EQ(1, audio_out_packets_.size());
|
||||
@@ -122,7 +127,7 @@ TEST_F(TensorsToAudioCalculatorFftTest, TestImpulseSignalAtTheCenter) {
|
||||
TEST_F(TensorsToAudioCalculatorFftTest, TestWindowedImpulseSignal) {
|
||||
constexpr int sample_size = 320;
|
||||
constexpr double sample_rate = 16000;
|
||||
ConfigGraph(sample_size, sample_rate, 320);
|
||||
ConfigGraph(sample_size, sample_rate, 320, Options::WITH_NYQUIST);
|
||||
Matrix impulse_data = CreateImpulseSignalData(sample_size, sample_size / 4);
|
||||
RunGraph(impulse_data, sample_rate);
|
||||
ASSERT_EQ(1, audio_out_packets_.size());
|
||||
@@ -135,7 +140,7 @@ TEST_F(TensorsToAudioCalculatorFftTest, TestWindowedImpulseSignal) {
|
||||
TEST_F(TensorsToAudioCalculatorFftTest, TestImpulseSignalAtBeginning) {
|
||||
constexpr int sample_size = 320;
|
||||
constexpr double sample_rate = 16000;
|
||||
ConfigGraph(sample_size, sample_rate, 320);
|
||||
ConfigGraph(sample_size, sample_rate, 320, Options::WITH_NYQUIST);
|
||||
Matrix impulse_data = CreateImpulseSignalData(sample_size, 0);
|
||||
RunGraph(impulse_data, sample_rate);
|
||||
ASSERT_EQ(1, audio_out_packets_.size());
|
||||
@@ -145,5 +150,31 @@ TEST_F(TensorsToAudioCalculatorFftTest, TestImpulseSignalAtBeginning) {
|
||||
EXPECT_EQ(audio_out_packets_[0].Get<Matrix>(), Matrix::Zero(1, sample_size));
|
||||
}
|
||||
|
||||
TEST_F(TensorsToAudioCalculatorFftTest, TestDftTensorWithDCAndNyquist) {
|
||||
constexpr int sample_size = 320;
|
||||
constexpr double sample_rate = 16000;
|
||||
ConfigGraph(sample_size, sample_rate, 320, Options::WITH_DC_AND_NYQUIST);
|
||||
|
||||
Matrix impulse_data = CreateImpulseSignalData(sample_size, sample_size / 2);
|
||||
RunGraph(impulse_data, sample_rate);
|
||||
ASSERT_EQ(1, audio_out_packets_.size());
|
||||
MP_ASSERT_OK(audio_out_packets_[0].ValidateAsType<Matrix>());
|
||||
// The impulse signal at the center is not affected by the window function.
|
||||
EXPECT_EQ(audio_out_packets_[0].Get<Matrix>(), impulse_data);
|
||||
}
|
||||
|
||||
TEST_F(TensorsToAudioCalculatorFftTest, TestDftTensorWithoutDCAndNyquist) {
|
||||
constexpr int sample_size = 320;
|
||||
constexpr double sample_rate = 16000;
|
||||
ConfigGraph(sample_size, sample_rate, 320, Options::WITHOUT_DC_AND_NYQUIST);
|
||||
|
||||
Matrix impulse_data = CreateImpulseSignalData(sample_size, sample_size / 2);
|
||||
RunGraph(impulse_data, sample_rate);
|
||||
ASSERT_EQ(1, audio_out_packets_.size());
|
||||
MP_ASSERT_OK(audio_out_packets_[0].ValidateAsType<Matrix>());
|
||||
// The impulse signal at the center is not affected by the window function.
|
||||
EXPECT_EQ(audio_out_packets_[0].Get<Matrix>(), impulse_data);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -64,6 +64,8 @@ bool CanUseGpu() {
|
||||
namespace mediapipe {
|
||||
namespace api2 {
|
||||
|
||||
using BoxFormat = ::mediapipe::TensorsToDetectionsCalculatorOptions::BoxFormat;
|
||||
|
||||
namespace {
|
||||
|
||||
void ConvertRawValuesToAnchors(const float* raw_anchors, int num_boxes,
|
||||
@@ -126,6 +128,15 @@ absl::Status CheckCustomTensorMapping(
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
BoxFormat GetBoxFormat(const TensorsToDetectionsCalculatorOptions& options) {
|
||||
if (options.has_box_format()) {
|
||||
return options.box_format();
|
||||
} else if (options.reverse_output_order()) {
|
||||
return mediapipe::TensorsToDetectionsCalculatorOptions::XYWH;
|
||||
}
|
||||
return mediapipe::TensorsToDetectionsCalculatorOptions::YXHW;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// Convert result Tensors from object detection models into MediaPipe
|
||||
@@ -211,6 +222,8 @@ class TensorsToDetectionsCalculator : public Node {
|
||||
int num_boxes_ = 0;
|
||||
int num_coords_ = 0;
|
||||
int max_results_ = -1;
|
||||
BoxFormat box_output_format_ =
|
||||
mediapipe::TensorsToDetectionsCalculatorOptions::YXHW;
|
||||
|
||||
// Set of allowed or ignored class indices.
|
||||
struct ClassIndexSet {
|
||||
@@ -655,6 +668,7 @@ absl::Status TensorsToDetectionsCalculator::LoadOptions(CalculatorContext* cc) {
|
||||
num_classes_ = options_.num_classes();
|
||||
num_boxes_ = options_.num_boxes();
|
||||
num_coords_ = options_.num_coords();
|
||||
box_output_format_ = GetBoxFormat(options_);
|
||||
CHECK_NE(options_.max_results(), 0)
|
||||
<< "The maximum number of the top-scored detection results must be "
|
||||
"non-zero.";
|
||||
@@ -728,17 +742,32 @@ absl::Status TensorsToDetectionsCalculator::DecodeBoxes(
|
||||
for (int i = 0; i < num_boxes_; ++i) {
|
||||
const int box_offset = i * num_coords_ + options_.box_coord_offset();
|
||||
|
||||
float y_center = raw_boxes[box_offset];
|
||||
float x_center = raw_boxes[box_offset + 1];
|
||||
float h = raw_boxes[box_offset + 2];
|
||||
float w = raw_boxes[box_offset + 3];
|
||||
if (options_.reverse_output_order()) {
|
||||
x_center = raw_boxes[box_offset];
|
||||
y_center = raw_boxes[box_offset + 1];
|
||||
w = raw_boxes[box_offset + 2];
|
||||
h = raw_boxes[box_offset + 3];
|
||||
float y_center = 0.0;
|
||||
float x_center = 0.0;
|
||||
float h = 0.0;
|
||||
float w = 0.0;
|
||||
// TODO
|
||||
switch (box_output_format_) {
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::UNSPECIFIED:
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::YXHW:
|
||||
y_center = raw_boxes[box_offset];
|
||||
x_center = raw_boxes[box_offset + 1];
|
||||
h = raw_boxes[box_offset + 2];
|
||||
w = raw_boxes[box_offset + 3];
|
||||
break;
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::XYWH:
|
||||
x_center = raw_boxes[box_offset];
|
||||
y_center = raw_boxes[box_offset + 1];
|
||||
w = raw_boxes[box_offset + 2];
|
||||
h = raw_boxes[box_offset + 3];
|
||||
break;
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::XYXY:
|
||||
x_center = (-raw_boxes[box_offset] + raw_boxes[box_offset + 2]) / 2;
|
||||
y_center = (-raw_boxes[box_offset + 1] + raw_boxes[box_offset + 3]) / 2;
|
||||
w = raw_boxes[box_offset + 2] + raw_boxes[box_offset];
|
||||
h = raw_boxes[box_offset + 3] + raw_boxes[box_offset + 1];
|
||||
break;
|
||||
}
|
||||
|
||||
x_center =
|
||||
x_center / options_.x_scale() * anchors[i].w() + anchors[i].x_center();
|
||||
y_center =
|
||||
@@ -767,11 +796,19 @@ absl::Status TensorsToDetectionsCalculator::DecodeBoxes(
|
||||
const int offset = i * num_coords_ + options_.keypoint_coord_offset() +
|
||||
k * options_.num_values_per_keypoint();
|
||||
|
||||
float keypoint_y = raw_boxes[offset];
|
||||
float keypoint_x = raw_boxes[offset + 1];
|
||||
if (options_.reverse_output_order()) {
|
||||
keypoint_x = raw_boxes[offset];
|
||||
keypoint_y = raw_boxes[offset + 1];
|
||||
float keypoint_y = 0.0;
|
||||
float keypoint_x = 0.0;
|
||||
switch (box_output_format_) {
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::UNSPECIFIED:
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::YXHW:
|
||||
keypoint_y = raw_boxes[offset];
|
||||
keypoint_x = raw_boxes[offset + 1];
|
||||
break;
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::XYWH:
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::XYXY:
|
||||
keypoint_x = raw_boxes[offset];
|
||||
keypoint_y = raw_boxes[offset + 1];
|
||||
break;
|
||||
}
|
||||
|
||||
(*boxes)[offset] = keypoint_x / options_.x_scale() * anchors[i].w() +
|
||||
@@ -856,8 +893,22 @@ Detection TensorsToDetectionsCalculator::ConvertToDetection(
|
||||
}
|
||||
|
||||
absl::Status TensorsToDetectionsCalculator::GpuInit(CalculatorContext* cc) {
|
||||
int output_format_flag = 0;
|
||||
switch (box_output_format_) {
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::UNSPECIFIED:
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::YXHW:
|
||||
output_format_flag = 0;
|
||||
break;
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::XYWH:
|
||||
output_format_flag = 1;
|
||||
break;
|
||||
case mediapipe::TensorsToDetectionsCalculatorOptions::XYXY:
|
||||
output_format_flag = 2;
|
||||
break;
|
||||
}
|
||||
#ifndef MEDIAPIPE_DISABLE_GL_COMPUTE
|
||||
MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this]() -> absl::Status {
|
||||
MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this, output_format_flag]()
|
||||
-> absl::Status {
|
||||
// A shader to decode detection boxes.
|
||||
const std::string decode_src = absl::Substitute(
|
||||
R"( #version 310 es
|
||||
@@ -879,7 +930,7 @@ layout(std430, binding = 2) readonly buffer Input1 {
|
||||
} raw_anchors;
|
||||
|
||||
uint num_coords = uint($0);
|
||||
int reverse_output_order = int($1);
|
||||
int output_format_flag = int($1);
|
||||
int apply_exponential = int($2);
|
||||
int box_coord_offset = int($3);
|
||||
int num_keypoints = int($4);
|
||||
@@ -892,17 +943,25 @@ void main() {
|
||||
uint anchor_offset = g_idx * uint(4); // check kNumCoordsPerBox
|
||||
|
||||
float y_center, x_center, h, w;
|
||||
|
||||
if (reverse_output_order == int(0)) {
|
||||
if (output_format_flag == int(0)) {
|
||||
y_center = raw_boxes.data[box_offset + uint(0)];
|
||||
x_center = raw_boxes.data[box_offset + uint(1)];
|
||||
h = raw_boxes.data[box_offset + uint(2)];
|
||||
w = raw_boxes.data[box_offset + uint(3)];
|
||||
} else {
|
||||
} else if (output_format_flag == int(1)) {
|
||||
x_center = raw_boxes.data[box_offset + uint(0)];
|
||||
y_center = raw_boxes.data[box_offset + uint(1)];
|
||||
w = raw_boxes.data[box_offset + uint(2)];
|
||||
h = raw_boxes.data[box_offset + uint(3)];
|
||||
} else if (output_format_flag == int(2)) {
|
||||
x_center = (-raw_boxes.data[box_offset + uint(0)]
|
||||
+raw_boxes.data[box_offset + uint(2)]) / 2.0;
|
||||
y_center = (-raw_boxes.data[box_offset + uint(1)]
|
||||
+raw_boxes.data[box_offset + uint(3)]) / 2.0;
|
||||
w = raw_boxes.data[box_offset + uint(0)]
|
||||
+ raw_boxes.data[box_offset + uint(2)];
|
||||
h = raw_boxes.data[box_offset + uint(1)]
|
||||
+ raw_boxes.data[box_offset + uint(3)];
|
||||
}
|
||||
|
||||
float anchor_yc = raw_anchors.data[anchor_offset + uint(0)];
|
||||
@@ -936,7 +995,7 @@ void main() {
|
||||
int kp_offset =
|
||||
int(g_idx * num_coords) + keypt_coord_offset + k * num_values_per_keypt;
|
||||
float kp_y, kp_x;
|
||||
if (reverse_output_order == int(0)) {
|
||||
if (output_format_flag == int(0)) {
|
||||
kp_y = raw_boxes.data[kp_offset + int(0)];
|
||||
kp_x = raw_boxes.data[kp_offset + int(1)];
|
||||
} else {
|
||||
@@ -949,8 +1008,7 @@ void main() {
|
||||
}
|
||||
})",
|
||||
options_.num_coords(), // box xywh
|
||||
options_.reverse_output_order() ? 1 : 0,
|
||||
options_.apply_exponential_on_box_size() ? 1 : 0,
|
||||
output_format_flag, options_.apply_exponential_on_box_size() ? 1 : 0,
|
||||
options_.box_coord_offset(), options_.num_keypoints(),
|
||||
options_.keypoint_coord_offset(), options_.num_values_per_keypoint());
|
||||
|
||||
@@ -1104,7 +1162,7 @@ kernel void decodeKernel(
|
||||
uint2 gid [[ thread_position_in_grid ]]) {
|
||||
|
||||
uint num_coords = uint($0);
|
||||
int reverse_output_order = int($1);
|
||||
int output_format_flag = int($1);
|
||||
int apply_exponential = int($2);
|
||||
int box_coord_offset = int($3);
|
||||
int num_keypoints = int($4);
|
||||
@@ -1112,8 +1170,7 @@ kernel void decodeKernel(
|
||||
int num_values_per_keypt = int($6);
|
||||
)",
|
||||
options_.num_coords(), // box xywh
|
||||
options_.reverse_output_order() ? 1 : 0,
|
||||
options_.apply_exponential_on_box_size() ? 1 : 0,
|
||||
output_format_flag, options_.apply_exponential_on_box_size() ? 1 : 0,
|
||||
options_.box_coord_offset(), options_.num_keypoints(),
|
||||
options_.keypoint_coord_offset(), options_.num_values_per_keypoint());
|
||||
decode_src += absl::Substitute(
|
||||
@@ -1129,16 +1186,25 @@ kernel void decodeKernel(
|
||||
|
||||
float y_center, x_center, h, w;
|
||||
|
||||
if (reverse_output_order == int(0)) {
|
||||
if (output_format_flag == int(0)) {
|
||||
y_center = raw_boxes[box_offset + uint(0)];
|
||||
x_center = raw_boxes[box_offset + uint(1)];
|
||||
h = raw_boxes[box_offset + uint(2)];
|
||||
w = raw_boxes[box_offset + uint(3)];
|
||||
} else {
|
||||
} else if (output_format_flag == int(1)) {
|
||||
x_center = raw_boxes[box_offset + uint(0)];
|
||||
y_center = raw_boxes[box_offset + uint(1)];
|
||||
w = raw_boxes[box_offset + uint(2)];
|
||||
h = raw_boxes[box_offset + uint(3)];
|
||||
} else if (output_format_flag == int(2)) {
|
||||
x_center = (-raw_boxes[box_offset + uint(0)]
|
||||
+raw_boxes[box_offset + uint(2)]) / 2.0;
|
||||
y_center = (-raw_boxes[box_offset + uint(1)]
|
||||
+raw_boxes[box_offset + uint(3)]) / 2.0;
|
||||
w = raw_boxes[box_offset + uint(0)]
|
||||
+ raw_boxes[box_offset + uint(2)];
|
||||
h = raw_boxes[box_offset + uint(1)]
|
||||
+ raw_boxes[box_offset + uint(3)];
|
||||
}
|
||||
|
||||
float anchor_yc = raw_anchors[anchor_offset + uint(0)];
|
||||
@@ -1172,7 +1238,7 @@ kernel void decodeKernel(
|
||||
int kp_offset =
|
||||
int(g_idx * num_coords) + keypt_coord_offset + k * num_values_per_keypt;
|
||||
float kp_y, kp_x;
|
||||
if (reverse_output_order == int(0)) {
|
||||
if (output_format_flag == int(0)) {
|
||||
kp_y = raw_boxes[kp_offset + int(0)];
|
||||
kp_x = raw_boxes[kp_offset + int(1)];
|
||||
} else {
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user