Compare commits
611
Commits
@@ -1,27 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Build/Installation Issue"
|
|
||||||
about: Use this template for build/installation issues
|
|
||||||
labels: type:build/install
|
|
||||||
|
|
||||||
---
|
|
||||||
<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
|
|
||||||
|
|
||||||
**System information** (Please provide as much relevant information as possible)
|
|
||||||
- OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
|
||||||
- Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
|
|
||||||
- Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
|
|
||||||
- Installed using virtualenv? pip? Conda? (if python):
|
|
||||||
- [MediaPipe version](https://github.com/google/mediapipe/releases):
|
|
||||||
- Bazel version:
|
|
||||||
- XCode and Tulsi versions (if iOS):
|
|
||||||
- Android SDK and NDK versions (if android):
|
|
||||||
- Android [AAR](https://google.github.io/mediapipe/getting_started/android_archive_library.html) ( if android):
|
|
||||||
- OpenCV version (if running on desktop):
|
|
||||||
|
|
||||||
**Describe the problem**:
|
|
||||||
|
|
||||||
|
|
||||||
**[Provide the exact sequence of commands / steps that you executed before running into the problem](https://google.github.io/mediapipe/getting_started/getting_started.html):**
|
|
||||||
|
|
||||||
**Complete Logs:**
|
|
||||||
Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Solution Issue"
|
|
||||||
about: Use this template for assistance with a specific mediapipe solution, such as "Pose" or "Iris", including inference model usage/training, solution-specific calculators, etc.
|
|
||||||
labels: type:support
|
|
||||||
|
|
||||||
---
|
|
||||||
<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
|
|
||||||
|
|
||||||
**System information** (Please provide as much relevant information as possible)
|
|
||||||
- Have I written custom code (as opposed to using a stock example script provided in Mediapipe):
|
|
||||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
|
||||||
- [MediaPipe version](https://github.com/google/mediapipe/releases):
|
|
||||||
- Bazel version:
|
|
||||||
- Solution (e.g. FaceMesh, Pose, Holistic):
|
|
||||||
- Programming Language and version ( e.g. C++, Python, Java):
|
|
||||||
|
|
||||||
**Describe the expected behavior:**
|
|
||||||
|
|
||||||
**Standalone code you may have used to try to get what you need :**
|
|
||||||
|
|
||||||
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook:
|
|
||||||
|
|
||||||
**Other info / Complete Logs :**
|
|
||||||
Include any logs or source code that would be helpful to
|
|
||||||
diagnose the problem. If including tracebacks, please include the full
|
|
||||||
traceback. Large logs and files should be attached:
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Tasks Issue"
|
|
||||||
about: Use this template for assistance with using MediaPipe Tasks to deploy on-device ML solutions (e.g. gesture recognition etc.) on supported platforms.
|
|
||||||
labels: type:support
|
|
||||||
|
|
||||||
---
|
|
||||||
<em>Please make sure that this is a [Tasks](https://developers.google.com/mediapipe/solutions) issue.<em>
|
|
||||||
|
|
||||||
**System information** (Please provide as much relevant information as possible)
|
|
||||||
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
|
|
||||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
|
||||||
- MediaPipe Tasks SDK version:
|
|
||||||
- Task name (e.g. Object detection, Gesture recognition etc.):
|
|
||||||
- Programming Language and version ( e.g. C++, Python, Java):
|
|
||||||
|
|
||||||
**Describe the expected behavior:**
|
|
||||||
|
|
||||||
**Standalone code you may have used to try to get what you need :**
|
|
||||||
|
|
||||||
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem:
|
|
||||||
|
|
||||||
**Other info / Complete Logs :**
|
|
||||||
Include any logs or source code that would be helpful to
|
|
||||||
diagnose the problem. If including tracebacks, please include the full
|
|
||||||
traceback. Large logs and files should be attached:
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Model Maker Issue"
|
|
||||||
about: Use this template for assistance with using MediaPipe Model Maker to create custom on-device ML solutions.
|
|
||||||
labels: type:support
|
|
||||||
|
|
||||||
---
|
|
||||||
<em>Please make sure that this is a [Model Maker](https://developers.google.com/mediapipe/solutions) issue.<em>
|
|
||||||
|
|
||||||
**System information** (Please provide as much relevant information as possible)
|
|
||||||
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
|
|
||||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
|
|
||||||
- Python version (e.g. 3.8):
|
|
||||||
- [MediaPipe Model Maker version](https://pypi.org/project/mediapipe-model-maker/):
|
|
||||||
- Task name (e.g. Image classification, Gesture recognition etc.):
|
|
||||||
|
|
||||||
**Describe the expected behavior:**
|
|
||||||
|
|
||||||
**Standalone code you may have used to try to get what you need :**
|
|
||||||
|
|
||||||
If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem:
|
|
||||||
|
|
||||||
**Other info / Complete Logs :**
|
|
||||||
Include any logs or source code that would be helpful to
|
|
||||||
diagnose the problem. If including tracebacks, please include the full
|
|
||||||
traceback. Large logs and files should be attached:
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Documentation Issue"
|
|
||||||
about: Use this template for documentation related issues
|
|
||||||
labels: type:docs
|
|
||||||
|
|
||||||
---
|
|
||||||
Thank you for submitting a MediaPipe documentation issue.
|
|
||||||
The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
|
|
||||||
## URL(s) with the issue:
|
|
||||||
|
|
||||||
Please provide a link to the documentation entry, for example: https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
|
|
||||||
|
|
||||||
## Description of issue (what needs changing):
|
|
||||||
|
|
||||||
Kinds of documentation problems:
|
|
||||||
|
|
||||||
### Clear description
|
|
||||||
|
|
||||||
For example, why should someone use this method? How is it useful?
|
|
||||||
|
|
||||||
### Correct links
|
|
||||||
|
|
||||||
Is the link to the source code correct?
|
|
||||||
|
|
||||||
### Parameters defined
|
|
||||||
Are all parameters defined and formatted correctly?
|
|
||||||
|
|
||||||
### Returns defined
|
|
||||||
|
|
||||||
Are return values defined?
|
|
||||||
|
|
||||||
### Raises listed and defined
|
|
||||||
|
|
||||||
Are the errors defined? For example,
|
|
||||||
|
|
||||||
### Usage example
|
|
||||||
|
|
||||||
Is there a usage example?
|
|
||||||
|
|
||||||
See the API guide:
|
|
||||||
on how to write testable usage examples.
|
|
||||||
|
|
||||||
### Request visuals, if applicable
|
|
||||||
|
|
||||||
Are there currently visuals? If not, will it clarify the content?
|
|
||||||
|
|
||||||
### Submit a pull request?
|
|
||||||
|
|
||||||
Are you planning to also submit a pull request to fix the issue? See the docs
|
|
||||||
https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md
|
|
||||||
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Bug Issue"
|
|
||||||
about: Use this template for reporting a bug
|
|
||||||
labels: type:bug
|
|
||||||
|
|
||||||
---
|
|
||||||
<em>Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.</em>
|
|
||||||
|
|
||||||
**System information** (Please provide as much relevant information as possible)
|
|
||||||
|
|
||||||
- Have I written custom code (as opposed to using a stock example script provided in MediaPipe):
|
|
||||||
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):
|
|
||||||
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
|
|
||||||
- Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser:
|
|
||||||
- Programming Language and version ( e.g. C++, Python, Java):
|
|
||||||
- [MediaPipe version](https://github.com/google/mediapipe/releases):
|
|
||||||
- Bazel version (if compiling from source):
|
|
||||||
- Solution ( e.g. FaceMesh, Pose, Holistic ):
|
|
||||||
- Android Studio, NDK, SDK versions (if issue is related to building in Android environment):
|
|
||||||
- Xcode & Tulsi version (if issue is related to building for iOS):
|
|
||||||
|
|
||||||
**Describe the current behavior:**
|
|
||||||
|
|
||||||
**Describe the expected behavior:**
|
|
||||||
|
|
||||||
**Standalone code to reproduce the issue:**
|
|
||||||
Provide a reproducible test case that is the bare minimum necessary to replicate the problem. If possible, please share a link to Colab/repo link /any notebook:
|
|
||||||
|
|
||||||
**Other info / Complete Logs :**
|
|
||||||
Include any logs or source code that would be helpful to
|
|
||||||
diagnose the problem. If including tracebacks, please include the full
|
|
||||||
traceback. Large logs and files should be attached
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
---
|
|
||||||
name: "Feature Request"
|
|
||||||
about: Use this template for raising a feature request
|
|
||||||
labels: type:feature
|
|
||||||
|
|
||||||
---
|
|
||||||
<em>Please make sure that this is a feature request.</em>
|
|
||||||
|
|
||||||
**System information** (Please provide as much relevant information as possible)
|
|
||||||
|
|
||||||
- MediaPipe Solution (you are using):
|
|
||||||
- Programming language : C++/typescript/Python/Objective C/Android Java
|
|
||||||
- Are you willing to contribute it (Yes/No):
|
|
||||||
|
|
||||||
|
|
||||||
**Describe the feature and the current behavior/state:**
|
|
||||||
|
|
||||||
**Will this change the current api? How?**
|
|
||||||
|
|
||||||
**Who will benefit with this feature?**
|
|
||||||
|
|
||||||
**Please specify the use cases for this feature:**
|
|
||||||
|
|
||||||
**Any Other info:**
|
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
name: Documentation issue
|
||||||
|
description: Use this template for documentation related issues. If this doesn’t look right, choose a different type.
|
||||||
|
labels: 'type:doc-bug'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: link
|
||||||
|
attributes:
|
||||||
|
value: Thank you for submitting a MediaPipe documentation issue. The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
|
||||||
|
- type: markdown
|
||||||
|
id: url
|
||||||
|
attributes:
|
||||||
|
value: URL(s) with the issue Please provide a link to the documentation entry, for example https://github.com/google/mediapipe/blob/master/docs/solutions/face_mesh.md#models
|
||||||
|
- type: input
|
||||||
|
id: description
|
||||||
|
attributes:
|
||||||
|
label: Description of issue (what needs changing)
|
||||||
|
description: Kinds of documentation problems
|
||||||
|
- type: input
|
||||||
|
id: clear_desc
|
||||||
|
attributes:
|
||||||
|
label: Clear description
|
||||||
|
description: For example, why should someone use this method? How is it useful?
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: link
|
||||||
|
attributes:
|
||||||
|
label: Correct links
|
||||||
|
description: Is the link to the source code correct?
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: parameter
|
||||||
|
attributes:
|
||||||
|
label: Parameters defined
|
||||||
|
description: Are all parameters defined and formatted correctly?
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: returns
|
||||||
|
attributes:
|
||||||
|
label: Returns defined
|
||||||
|
description: Are return values defined?
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: raises
|
||||||
|
attributes:
|
||||||
|
label: Raises listed and defined
|
||||||
|
description: Are the errors defined? For example,
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: usage
|
||||||
|
attributes:
|
||||||
|
label: Usage example
|
||||||
|
description: Is there a usage example? See the API guide-on how to write testable usage examples.
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: visual
|
||||||
|
attributes:
|
||||||
|
label: Request visuals, if applicable
|
||||||
|
description: Are there currently visuals? If not, will it clarify the content?
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: pull
|
||||||
|
attributes:
|
||||||
|
label: Submit a pull request?
|
||||||
|
description: Are you planning to also submit a pull request to fix the issue? See the [docs](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md)
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
name: Solution(Legacy) Issue
|
||||||
|
description: Use this template for assistance with a specific Mediapipe solution (google.github.io/mediapipe/solutions) such as "Pose", including inference model usage/training, solution-specific calculators etc.
|
||||||
|
labels: 'type:support'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: linkmodel
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.
|
||||||
|
- type: dropdown
|
||||||
|
id: customcode_model
|
||||||
|
attributes:
|
||||||
|
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||||
|
options:
|
||||||
|
- 'Yes'
|
||||||
|
- 'No'
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: os_model
|
||||||
|
attributes:
|
||||||
|
label: OS Platform and Distribution
|
||||||
|
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: mediapipe_version
|
||||||
|
attributes:
|
||||||
|
label: MediaPipe version
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: bazel_version
|
||||||
|
attributes:
|
||||||
|
label: Bazel version
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: solution
|
||||||
|
attributes:
|
||||||
|
label: Solution
|
||||||
|
placeholder: e.g. FaceMesh, Pose, Holistic
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: programminglang
|
||||||
|
attributes:
|
||||||
|
label: Programming Language and version
|
||||||
|
placeholder: e.g. C++, Python, Java
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: current_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the actual behavior
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: expected_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the expected behaviour
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened_model
|
||||||
|
attributes:
|
||||||
|
label: Standalone code/steps you may have used to try to get what you need
|
||||||
|
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: other_info
|
||||||
|
attributes:
|
||||||
|
label: Other info / Complete Logs
|
||||||
|
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
name: Bug Issues
|
||||||
|
description: Use this template for reporting a bug. If this doesn’t look right, choose a different type.
|
||||||
|
labels: 'type:bug'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: link
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.
|
||||||
|
- type: dropdown
|
||||||
|
id: customcode_model
|
||||||
|
attributes:
|
||||||
|
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||||
|
options:
|
||||||
|
- 'Yes'
|
||||||
|
- 'No'
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: os
|
||||||
|
attributes:
|
||||||
|
label: OS Platform and Distribution
|
||||||
|
description:
|
||||||
|
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: mobile_device
|
||||||
|
attributes:
|
||||||
|
label: Mobile device if the issue happens on mobile device
|
||||||
|
description:
|
||||||
|
placeholder: e.g. iPhone 8, Pixel 2, Samsung Galaxy
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: browser_version
|
||||||
|
attributes:
|
||||||
|
label: Browser and version if the issue happens on browser
|
||||||
|
placeholder: e.g. Google Chrome 109.0.5414.119, Safari 16.3
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: programminglang
|
||||||
|
attributes:
|
||||||
|
label: Programming Language and version
|
||||||
|
placeholder: e.g. C++, Python, Java
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: mediapipever
|
||||||
|
attributes:
|
||||||
|
label: MediaPipe version
|
||||||
|
description:
|
||||||
|
placeholder: e.g. 0.8.11, 0.9.1
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: bazelver
|
||||||
|
attributes:
|
||||||
|
label: Bazel version
|
||||||
|
description:
|
||||||
|
placeholder: e.g. 5.0, 5.1
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: solution
|
||||||
|
attributes:
|
||||||
|
label: Solution
|
||||||
|
placeholder: e.g. FaceMesh, Pose, Holistic
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: sdkndkversion
|
||||||
|
attributes:
|
||||||
|
label: Android Studio, NDK, SDK versions (if issue is related to building in Android environment)
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: xcode_ver
|
||||||
|
attributes:
|
||||||
|
label: Xcode & Tulsi version (if issue is related to building for iOS)
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: current_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the actual behavior
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: expected_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the expected behaviour
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened_model
|
||||||
|
attributes:
|
||||||
|
label: Standalone code/steps you may have used to try to get what you need
|
||||||
|
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: other_info
|
||||||
|
attributes:
|
||||||
|
label: Other info / Complete Logs
|
||||||
|
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -0,0 +1,109 @@
|
|||||||
|
name: Build/Install Issue
|
||||||
|
description: Use this template to report build/install issue
|
||||||
|
labels: 'type:build/install'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: link
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.
|
||||||
|
- type: input
|
||||||
|
id: os
|
||||||
|
attributes:
|
||||||
|
label: OS Platform and Distribution
|
||||||
|
description:
|
||||||
|
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: compilerversion
|
||||||
|
attributes:
|
||||||
|
label: Compiler version
|
||||||
|
description:
|
||||||
|
placeholder: e.g. gcc/g++ 8 /Apple clang version 12.0.0
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: programminglang
|
||||||
|
attributes:
|
||||||
|
label: Programming Language and version
|
||||||
|
description:
|
||||||
|
placeholder: e.g. C++ 14, Python 3.6, Java
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: virtualenv
|
||||||
|
attributes:
|
||||||
|
label: Installed using virtualenv? pip? Conda?(if python)
|
||||||
|
description:
|
||||||
|
placeholder:
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: mediapipever
|
||||||
|
attributes:
|
||||||
|
label: MediaPipe version
|
||||||
|
description:
|
||||||
|
placeholder: e.g. 0.8.11, 0.9.1
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: bazelver
|
||||||
|
attributes:
|
||||||
|
label: Bazel version
|
||||||
|
description:
|
||||||
|
placeholder: e.g. 5.0, 5.1
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: xcodeversion
|
||||||
|
attributes:
|
||||||
|
label: XCode and Tulsi versions (if iOS)
|
||||||
|
description:
|
||||||
|
placeholder:
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: sdkndkversion
|
||||||
|
attributes:
|
||||||
|
label: Android SDK and NDK versions (if android)
|
||||||
|
description:
|
||||||
|
placeholder:
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: dropdown
|
||||||
|
id: androidaar
|
||||||
|
attributes:
|
||||||
|
label: Android AAR (if android)
|
||||||
|
options:
|
||||||
|
- 'Yes'
|
||||||
|
- 'No'
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: opencvversion
|
||||||
|
attributes:
|
||||||
|
label: OpenCV version (if running on desktop)
|
||||||
|
description:
|
||||||
|
placeholder:
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened
|
||||||
|
attributes:
|
||||||
|
label: Describe the problem
|
||||||
|
description: Provide the exact sequence of commands / steps that you executed before running into the [problem](https://google.github.io/mediapipe/getting_started/getting_started.html)
|
||||||
|
placeholder: Tell us what you see!
|
||||||
|
value: "A bug happened!"
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: code-to-reproduce
|
||||||
|
attributes:
|
||||||
|
label: Complete Logs
|
||||||
|
description: Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||||
|
placeholder: Tell us what you see!
|
||||||
|
value:
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
name: Feature Request Issues
|
||||||
|
description: Use this template for raising a feature request. If this doesn’t look right, choose a different type.
|
||||||
|
labels: 'type:feature'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: linkmodel
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a feature request.
|
||||||
|
- type: input
|
||||||
|
id: solution
|
||||||
|
attributes:
|
||||||
|
label: MediaPipe Solution (you are using)
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: pgmlang
|
||||||
|
attributes:
|
||||||
|
label: Programming language
|
||||||
|
placeholder: C++/typescript/Python/Objective C/Android Java
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: dropdown
|
||||||
|
id: willingcon
|
||||||
|
attributes:
|
||||||
|
label: Are you willing to contribute it
|
||||||
|
options:
|
||||||
|
- 'Yes'
|
||||||
|
- 'No'
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: behaviour
|
||||||
|
attributes:
|
||||||
|
label: Describe the feature and the current behaviour/state
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: api_change
|
||||||
|
attributes:
|
||||||
|
label: Will this change the current API? How?
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: benifit
|
||||||
|
attributes:
|
||||||
|
label: Who will benefit with this feature?
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: use_case
|
||||||
|
attributes:
|
||||||
|
label: Please specify the use cases for this feature
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: info_other
|
||||||
|
attributes:
|
||||||
|
label: Any Other info
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
name: Model Maker Issues
|
||||||
|
description: Use this template for assistance with using MediaPipe Model Maker (developers.google.com/mediapipe/solutions) to create custom on-device ML solutions.
|
||||||
|
labels: 'type:modelmaker'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: linkmodel
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a [Model Maker](https://developers.google.com/mediapipe/solutions) issue
|
||||||
|
- type: dropdown
|
||||||
|
id: customcode_model
|
||||||
|
attributes:
|
||||||
|
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||||
|
options:
|
||||||
|
- 'Yes'
|
||||||
|
- 'No'
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: os_model
|
||||||
|
attributes:
|
||||||
|
label: OS Platform and Distribution
|
||||||
|
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: pythonver
|
||||||
|
attributes:
|
||||||
|
label: Python Version
|
||||||
|
placeholder: e.g. 3.7, 3.8
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: modelmakerver
|
||||||
|
attributes:
|
||||||
|
label: MediaPipe Model Maker version
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: taskname
|
||||||
|
attributes:
|
||||||
|
label: Task name (e.g. Image classification, Gesture recognition etc.)
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: current_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the actual behavior
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: expected_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the expected behaviour
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened_model
|
||||||
|
attributes:
|
||||||
|
label: Standalone code/steps you may have used to try to get what you need
|
||||||
|
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: other_info
|
||||||
|
attributes:
|
||||||
|
label: Other info / Complete Logs
|
||||||
|
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
name: Studio Issues
|
||||||
|
description: Use this template for assistance with the MediaPipe Studio application. If this doesn’t look right, choose a different type.
|
||||||
|
labels: 'type:support'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: linkmodel
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a MediaPipe Studio issue.
|
||||||
|
- type: input
|
||||||
|
id: os_model
|
||||||
|
attributes:
|
||||||
|
label: OS Platform and Distribution
|
||||||
|
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: browserver
|
||||||
|
attributes:
|
||||||
|
label: Browser and Version
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: hardware
|
||||||
|
attributes:
|
||||||
|
label: Any microphone or camera hardware
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: url
|
||||||
|
attributes:
|
||||||
|
label: URL that shows the problem
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: current_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the actual behavior
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: expected_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the expected behaviour
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened_model
|
||||||
|
attributes:
|
||||||
|
label: Standalone code/steps you may have used to try to get what you need
|
||||||
|
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: textarea
|
||||||
|
id: other_info
|
||||||
|
attributes:
|
||||||
|
label: Other info / Complete Logs
|
||||||
|
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
name: Task Issue
|
||||||
|
description: Use this template for assistance with using MediaPipe Tasks (developers.google.com/mediapipe/solutions) to deploy on-device ML solutions (e.g. gesture recognition etc.) on supported platforms
|
||||||
|
labels: 'type:task'
|
||||||
|
body:
|
||||||
|
- type: markdown
|
||||||
|
id: linkmodel
|
||||||
|
attributes:
|
||||||
|
value: Please make sure that this is a [Tasks](https://developers.google.com/mediapipe/solutions) issue.
|
||||||
|
- type: dropdown
|
||||||
|
id: customcode_model
|
||||||
|
attributes:
|
||||||
|
label: Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
|
||||||
|
options:
|
||||||
|
- 'Yes'
|
||||||
|
- 'No'
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: os_model
|
||||||
|
attributes:
|
||||||
|
label: OS Platform and Distribution
|
||||||
|
placeholder: e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: task-sdk-version
|
||||||
|
attributes:
|
||||||
|
label: MediaPipe Tasks SDK version
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
|
- type: input
|
||||||
|
id: taskname
|
||||||
|
attributes:
|
||||||
|
label: Task name (e.g. Image classification, Gesture recognition etc.)
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: input
|
||||||
|
id: programminglang
|
||||||
|
attributes:
|
||||||
|
label: Programming Language and version (e.g. C++, Python, Java)
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: current_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the actual behavior
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: expected_model
|
||||||
|
attributes:
|
||||||
|
label: Describe the expected behaviour
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: what-happened_model
|
||||||
|
attributes:
|
||||||
|
label: Standalone code/steps you may have used to try to get what you need
|
||||||
|
description: If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab, GitHub repo link or anything that we can use to reproduce the problem
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: true
|
||||||
|
- type: textarea
|
||||||
|
id: other_info
|
||||||
|
attributes:
|
||||||
|
label: Other info / Complete Logs
|
||||||
|
description: Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached
|
||||||
|
render: shell
|
||||||
|
validations:
|
||||||
|
required: false
|
||||||
@@ -19,6 +19,17 @@ ML solutions for live and streaming media.
|
|||||||
 | 
|
 | 
|
||||||
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
|
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||||
|
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||||
|
as the primary developer documentation
|
||||||
|
site for MediaPipe starting April 3, 2023.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## ML solutions in MediaPipe
|
## ML solutions in MediaPipe
|
||||||
|
|
||||||
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
|
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
|
||||||
|
|||||||
@@ -10,33 +10,31 @@ bind(
|
|||||||
|
|
||||||
http_archive(
|
http_archive(
|
||||||
name = "bazel_skylib",
|
name = "bazel_skylib",
|
||||||
type = "tar.gz",
|
sha256 = "74d544d96f4a5bb630d465ca8bbcfe231e3594e5aae57e1edbf17a6eb3ca2506",
|
||||||
urls = [
|
urls = [
|
||||||
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz",
|
"https://storage.googleapis.com/mirror.tensorflow.org/github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
|
||||||
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz",
|
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.3.0/bazel-skylib-1.3.0.tar.gz",
|
||||||
],
|
],
|
||||||
sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
|
|
||||||
)
|
)
|
||||||
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
|
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
|
||||||
bazel_skylib_workspace()
|
bazel_skylib_workspace()
|
||||||
load("@bazel_skylib//lib:versions.bzl", "versions")
|
load("@bazel_skylib//lib:versions.bzl", "versions")
|
||||||
versions.check(minimum_bazel_version = "3.7.2")
|
versions.check(minimum_bazel_version = "3.7.2")
|
||||||
|
|
||||||
# ABSL cpp library lts_2021_03_24, patch 2.
|
# ABSL cpp library lts_2023_01_25.
|
||||||
http_archive(
|
http_archive(
|
||||||
name = "com_google_absl",
|
name = "com_google_absl",
|
||||||
urls = [
|
urls = [
|
||||||
"https://github.com/abseil/abseil-cpp/archive/refs/tags/20220623.1.tar.gz",
|
"https://github.com/abseil/abseil-cpp/archive/refs/tags/20230125.0.tar.gz",
|
||||||
],
|
],
|
||||||
# Remove after https://github.com/abseil/abseil-cpp/issues/326 is solved.
|
|
||||||
patches = [
|
patches = [
|
||||||
"@//third_party:com_google_absl_f863b622fe13612433fdf43f76547d5edda0c93001.diff"
|
"@//third_party:com_google_absl_windows_patch.diff"
|
||||||
],
|
],
|
||||||
patch_args = [
|
patch_args = [
|
||||||
"-p1",
|
"-p1",
|
||||||
],
|
],
|
||||||
strip_prefix = "abseil-cpp-20220623.1",
|
strip_prefix = "abseil-cpp-20230125.0",
|
||||||
sha256 = "91ac87d30cc6d79f9ab974c51874a704de9c2647c40f6932597329a282217ba8"
|
sha256 = "3ea49a7d97421b88a8c48a0de16c16048e17725c7ec0f1d3ea2683a2a75adc21"
|
||||||
)
|
)
|
||||||
|
|
||||||
http_archive(
|
http_archive(
|
||||||
@@ -148,13 +146,14 @@ http_archive(
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
load("//third_party/flatbuffers:workspace.bzl", flatbuffers = "repo")
|
load("@//third_party/flatbuffers:workspace.bzl", flatbuffers = "repo")
|
||||||
flatbuffers()
|
flatbuffers()
|
||||||
|
|
||||||
http_archive(
|
http_archive(
|
||||||
name = "com_google_audio_tools",
|
name = "com_google_audio_tools",
|
||||||
strip_prefix = "multichannel-audio-tools-master",
|
strip_prefix = "multichannel-audio-tools-1f6b1319f13282eda6ff1317be13de67f4723860",
|
||||||
urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
|
urls = ["https://github.com/google/multichannel-audio-tools/archive/1f6b1319f13282eda6ff1317be13de67f4723860.zip"],
|
||||||
|
sha256 = "fe346e1aee4f5069c4cbccb88706a9a2b2b4cf98aeb91ec1319be77e07dd7435",
|
||||||
repo_mapping = {"@com_github_glog_glog" : "@com_github_glog_glog_no_gflags"},
|
repo_mapping = {"@com_github_glog_glog" : "@com_github_glog_glog_no_gflags"},
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -174,7 +173,7 @@ http_archive(
|
|||||||
"https://github.com/google/sentencepiece/archive/1.0.0.zip",
|
"https://github.com/google/sentencepiece/archive/1.0.0.zip",
|
||||||
],
|
],
|
||||||
patches = [
|
patches = [
|
||||||
"//third_party:com_google_sentencepiece_no_gflag_no_gtest.diff",
|
"@//third_party:com_google_sentencepiece_no_gflag_no_gtest.diff",
|
||||||
],
|
],
|
||||||
patch_args = ["-p1"],
|
patch_args = ["-p1"],
|
||||||
repo_mapping = {"@com_google_glog" : "@com_github_glog_glog_no_gflags"},
|
repo_mapping = {"@com_google_glog" : "@com_github_glog_glog_no_gflags"},
|
||||||
@@ -188,8 +187,8 @@ http_archive(
|
|||||||
"https://github.com/tensorflow/text/archive/v2.2.0.zip",
|
"https://github.com/tensorflow/text/archive/v2.2.0.zip",
|
||||||
],
|
],
|
||||||
patches = [
|
patches = [
|
||||||
"//third_party:tensorflow_text_remove_tf_deps.diff",
|
"@//third_party:tensorflow_text_remove_tf_deps.diff",
|
||||||
"//third_party:tensorflow_text_a0f49e63.diff",
|
"@//third_party:tensorflow_text_a0f49e63.diff",
|
||||||
],
|
],
|
||||||
patch_args = ["-p1"],
|
patch_args = ["-p1"],
|
||||||
repo_mapping = {"@com_google_re2": "@com_googlesource_code_re2"},
|
repo_mapping = {"@com_google_re2": "@com_googlesource_code_re2"},
|
||||||
@@ -320,12 +319,30 @@ http_archive(
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
# iOS basic build deps.
|
# Load Zlib before initializing TensorFlow and the iOS build rules to guarantee
|
||||||
|
# that the target @zlib//:mini_zlib is available
|
||||||
|
http_archive(
|
||||||
|
name = "zlib",
|
||||||
|
build_file = "@//third_party:zlib.BUILD",
|
||||||
|
sha256 = "c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1",
|
||||||
|
strip_prefix = "zlib-1.2.11",
|
||||||
|
urls = [
|
||||||
|
"http://mirror.bazel.build/zlib.net/fossils/zlib-1.2.11.tar.gz",
|
||||||
|
"http://zlib.net/fossils/zlib-1.2.11.tar.gz", # 2017-01-15
|
||||||
|
],
|
||||||
|
patches = [
|
||||||
|
"@//third_party:zlib.diff",
|
||||||
|
],
|
||||||
|
patch_args = [
|
||||||
|
"-p1",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
# iOS basic build deps.
|
||||||
http_archive(
|
http_archive(
|
||||||
name = "build_bazel_rules_apple",
|
name = "build_bazel_rules_apple",
|
||||||
sha256 = "77e8bf6fda706f420a55874ae6ee4df0c9d95da6c7838228b26910fc82eea5a2",
|
sha256 = "f94e6dddf74739ef5cb30f000e13a2a613f6ebfa5e63588305a71fce8a8a9911",
|
||||||
url = "https://github.com/bazelbuild/rules_apple/releases/download/0.32.0/rules_apple.0.32.0.tar.gz",
|
url = "https://github.com/bazelbuild/rules_apple/releases/download/1.1.3/rules_apple.1.1.3.tar.gz",
|
||||||
patches = [
|
patches = [
|
||||||
# Bypass checking ios unit test runner when building MP ios applications.
|
# Bypass checking ios unit test runner when building MP ios applications.
|
||||||
"@//third_party:build_bazel_rules_apple_bypass_test_runner_check.diff"
|
"@//third_party:build_bazel_rules_apple_bypass_test_runner_check.diff"
|
||||||
@@ -339,29 +356,24 @@ load(
|
|||||||
"@build_bazel_rules_apple//apple:repositories.bzl",
|
"@build_bazel_rules_apple//apple:repositories.bzl",
|
||||||
"apple_rules_dependencies",
|
"apple_rules_dependencies",
|
||||||
)
|
)
|
||||||
|
|
||||||
apple_rules_dependencies()
|
apple_rules_dependencies()
|
||||||
|
|
||||||
load(
|
load(
|
||||||
"@build_bazel_rules_swift//swift:repositories.bzl",
|
"@build_bazel_rules_swift//swift:repositories.bzl",
|
||||||
"swift_rules_dependencies",
|
"swift_rules_dependencies",
|
||||||
)
|
)
|
||||||
|
|
||||||
swift_rules_dependencies()
|
swift_rules_dependencies()
|
||||||
|
|
||||||
http_archive(
|
load(
|
||||||
name = "build_bazel_apple_support",
|
"@build_bazel_rules_swift//swift:extras.bzl",
|
||||||
sha256 = "741366f79d900c11e11d8efd6cc6c66a31bfb2451178b58e0b5edc6f1db17b35",
|
"swift_rules_extra_dependencies",
|
||||||
urls = [
|
|
||||||
"https://github.com/bazelbuild/apple_support/releases/download/0.10.0/apple_support.0.10.0.tar.gz"
|
|
||||||
],
|
|
||||||
)
|
)
|
||||||
|
swift_rules_extra_dependencies()
|
||||||
|
|
||||||
load(
|
load(
|
||||||
"@build_bazel_apple_support//lib:repositories.bzl",
|
"@build_bazel_apple_support//lib:repositories.bzl",
|
||||||
"apple_support_dependencies",
|
"apple_support_dependencies",
|
||||||
)
|
)
|
||||||
|
|
||||||
apple_support_dependencies()
|
apple_support_dependencies()
|
||||||
|
|
||||||
# More iOS deps.
|
# More iOS deps.
|
||||||
@@ -442,29 +454,10 @@ http_archive(
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
# Load Zlib before initializing TensorFlow to guarantee that the target
|
|
||||||
# @zlib//:mini_zlib is available
|
|
||||||
http_archive(
|
|
||||||
name = "zlib",
|
|
||||||
build_file = "//third_party:zlib.BUILD",
|
|
||||||
sha256 = "c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1",
|
|
||||||
strip_prefix = "zlib-1.2.11",
|
|
||||||
urls = [
|
|
||||||
"http://mirror.bazel.build/zlib.net/fossils/zlib-1.2.11.tar.gz",
|
|
||||||
"http://zlib.net/fossils/zlib-1.2.11.tar.gz", # 2017-01-15
|
|
||||||
],
|
|
||||||
patches = [
|
|
||||||
"@//third_party:zlib.diff",
|
|
||||||
],
|
|
||||||
patch_args = [
|
|
||||||
"-p1",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
|
|
||||||
# TensorFlow repo should always go after the other external dependencies.
|
# TensorFlow repo should always go after the other external dependencies.
|
||||||
# TF on 2022-08-10.
|
# TF on 2023-03-08.
|
||||||
_TENSORFLOW_GIT_COMMIT = "af1d5bc4fbb66d9e6cc1cf89503014a99233583b"
|
_TENSORFLOW_GIT_COMMIT = "24f7ee636d62e1f8d8330357f8bbd65956dfb84d"
|
||||||
_TENSORFLOW_SHA256 = "f85a5443264fc58a12d136ca6a30774b5bc25ceaf7d114d97f252351b3c3a2cb"
|
_TENSORFLOW_SHA256 = "7f8a96dd99215c0cdc77230d3dbce43e60102b64a89203ad04aa09b0a187a4bd"
|
||||||
http_archive(
|
http_archive(
|
||||||
name = "org_tensorflow",
|
name = "org_tensorflow",
|
||||||
urls = [
|
urls = [
|
||||||
@@ -518,8 +511,8 @@ load("@build_bazel_rules_nodejs//:index.bzl", "node_repositories", "yarn_install
|
|||||||
node_repositories()
|
node_repositories()
|
||||||
yarn_install(
|
yarn_install(
|
||||||
name = "npm",
|
name = "npm",
|
||||||
package_json = "//:package.json",
|
package_json = "@//:package.json",
|
||||||
yarn_lock = "//:yarn.lock",
|
yarn_lock = "@//:yarn.lock",
|
||||||
)
|
)
|
||||||
|
|
||||||
# Protobuf for Node dependencies
|
# Protobuf for Node dependencies
|
||||||
@@ -545,8 +538,48 @@ load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_
|
|||||||
rules_proto_dependencies()
|
rules_proto_dependencies()
|
||||||
rules_proto_toolchains()
|
rules_proto_toolchains()
|
||||||
|
|
||||||
load("//third_party:external_files.bzl", "external_files")
|
load("@//third_party:external_files.bzl", "external_files")
|
||||||
external_files()
|
external_files()
|
||||||
|
|
||||||
load("//third_party:wasm_files.bzl", "wasm_files")
|
load("@//third_party:wasm_files.bzl", "wasm_files")
|
||||||
wasm_files()
|
wasm_files()
|
||||||
|
|
||||||
|
# Halide
|
||||||
|
|
||||||
|
new_local_repository(
|
||||||
|
name = "halide",
|
||||||
|
build_file = "@//third_party/halide:BUILD.bazel",
|
||||||
|
path = "third_party/halide"
|
||||||
|
)
|
||||||
|
|
||||||
|
http_archive(
|
||||||
|
name = "linux_halide",
|
||||||
|
sha256 = "f62b2914823d6e33d18693f5b74484f274523bf5402ce51988e24393d123b375",
|
||||||
|
strip_prefix = "Halide-15.0.0-x86-64-linux",
|
||||||
|
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-x86-64-linux-d7651f4b32f9dbd764f243134001f7554378d62d.tar.gz"],
|
||||||
|
build_file = "@//third_party:halide.BUILD",
|
||||||
|
)
|
||||||
|
|
||||||
|
http_archive(
|
||||||
|
name = "macos_x86_64_halide",
|
||||||
|
sha256 = "3d832aed942080ea89aa832462c68fbb906f3055c440b7b6d35093d7c52f6aab",
|
||||||
|
strip_prefix = "Halide-15.0.0-x86-64-osx",
|
||||||
|
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-x86-64-osx-d7651f4b32f9dbd764f243134001f7554378d62d.tar.gz"],
|
||||||
|
build_file = "@//third_party:halide.BUILD",
|
||||||
|
)
|
||||||
|
|
||||||
|
http_archive(
|
||||||
|
name = "macos_arm_64_halide",
|
||||||
|
sha256 = "b1fad3c9810122b187303d7031d9e35fb43761f345d18cc4492c00ed5877f641",
|
||||||
|
strip_prefix = "Halide-15.0.0-arm-64-osx",
|
||||||
|
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-arm-64-osx-d7651f4b32f9dbd764f243134001f7554378d62d.tar.gz"],
|
||||||
|
build_file = "@//third_party:halide.BUILD",
|
||||||
|
)
|
||||||
|
|
||||||
|
http_archive(
|
||||||
|
name = "windows_halide",
|
||||||
|
sha256 = "5acf6fe161dd375856a2b43f4bb0a32815ba958b0585ee312c44e008aa7b0b64",
|
||||||
|
strip_prefix = "Halide-15.0.0-x86-64-windows",
|
||||||
|
urls = ["https://github.com/halide/Halide/releases/download/v15.0.0/Halide-15.0.0-x86-64-windows-d7651f4b32f9dbd764f243134001f7554378d62d.zip"],
|
||||||
|
build_file = "@//third_party:halide.BUILD",
|
||||||
|
)
|
||||||
|
|||||||
+1
-3
@@ -4,12 +4,10 @@ py_binary(
|
|||||||
name = "build_py_api_docs",
|
name = "build_py_api_docs",
|
||||||
srcs = ["build_py_api_docs.py"],
|
srcs = ["build_py_api_docs.py"],
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe",
|
|
||||||
"//third_party/py/absl:app",
|
"//third_party/py/absl:app",
|
||||||
"//third_party/py/absl/flags",
|
"//third_party/py/absl/flags",
|
||||||
"//third_party/py/tensorflow_docs",
|
"//third_party/py/mediapipe",
|
||||||
"//third_party/py/tensorflow_docs/api_generator:generate_lib",
|
"//third_party/py/tensorflow_docs/api_generator:generate_lib",
|
||||||
"//third_party/py/tensorflow_docs/api_generator:public_api",
|
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,13 @@
|
|||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8"/>
|
||||||
|
<meta http-equiv="refresh" content="0;url={{ page.target }}"/>
|
||||||
|
<link rel="canonical" href="{{ page.target }}"/>
|
||||||
|
<title>Redirecting</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<p>This page now lives on https://developers.google.com/mediapipe/. If you aren't automatically
|
||||||
|
redirected, follow this
|
||||||
|
<a href="{{ page.target }}">link</a>.</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -44,14 +44,14 @@ _OUTPUT_DIR = flags.DEFINE_string(
|
|||||||
|
|
||||||
_URL_PREFIX = flags.DEFINE_string(
|
_URL_PREFIX = flags.DEFINE_string(
|
||||||
'code_url_prefix',
|
'code_url_prefix',
|
||||||
'https://github.com/google/mediapipe/tree/master/mediapipe',
|
'https://github.com/google/mediapipe/blob/master/mediapipe',
|
||||||
'The url prefix for links to code.')
|
'The url prefix for links to code.')
|
||||||
|
|
||||||
_SEARCH_HINTS = flags.DEFINE_bool(
|
_SEARCH_HINTS = flags.DEFINE_bool(
|
||||||
'search_hints', True,
|
'search_hints', True,
|
||||||
'Include metadata search hints in the generated files')
|
'Include metadata search hints in the generated files')
|
||||||
|
|
||||||
_SITE_PATH = flags.DEFINE_string('site_path', '/mediapipe/api_docs/python',
|
_SITE_PATH = flags.DEFINE_string('site_path', '/mediapipe/api/solutions/python',
|
||||||
'Path prefix in the _toc.yaml')
|
'Path prefix in the _toc.yaml')
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,697 @@
|
|||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Building Graphs in C++
|
||||||
|
parent: Graphs
|
||||||
|
nav_order: 1
|
||||||
|
---
|
||||||
|
|
||||||
|
# Building Graphs in C++
|
||||||
|
{: .no_toc }
|
||||||
|
|
||||||
|
1. TOC
|
||||||
|
{:toc}
|
||||||
|
---
|
||||||
|
|
||||||
|
C++ graph builder is a powerful tool for:
|
||||||
|
|
||||||
|
* Building complex graphs
|
||||||
|
* Parametrizing graphs (e.g. setting a delegate on `InferenceCalculator`,
|
||||||
|
enabling/disabling parts of the graph)
|
||||||
|
* Deduplicating graphs (e.g. instead of CPU and GPU dedicated graphs in pbtxt
|
||||||
|
you can have a single code that constructs required graphs, sharing as much
|
||||||
|
as possible)
|
||||||
|
* Supporting optional graph inputs/outputs
|
||||||
|
* Customizing graphs per platform
|
||||||
|
|
||||||
|
## Basic Usage
|
||||||
|
|
||||||
|
Let's see how C++ graph builder can be used for a simple graph:
|
||||||
|
|
||||||
|
```proto
|
||||||
|
# Graph inputs.
|
||||||
|
input_stream: "input_tensors"
|
||||||
|
input_side_packet: "model"
|
||||||
|
|
||||||
|
# Graph outputs.
|
||||||
|
output_stream: "output_tensors"
|
||||||
|
|
||||||
|
node {
|
||||||
|
calculator: "InferenceCalculator"
|
||||||
|
input_stream: "TENSORS:input_tensors"
|
||||||
|
input_side_packet: "MODEL:model"
|
||||||
|
output_stream: "TENSORS:output_tensors"
|
||||||
|
node_options: {
|
||||||
|
[type.googleapis.com/mediapipe.InferenceCalculatorOptions] {
|
||||||
|
# Requesting GPU delegate.
|
||||||
|
delegate { gpu {} }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Function to build the above `CalculatorGraphConfig` may look like:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Graph inputs.
|
||||||
|
Stream<std::vector<Tensor>> input_tensors =
|
||||||
|
graph.In(0).SetName("input_tensors").Cast<std::vector<Tensor>>();
|
||||||
|
SidePacket<TfLiteModelPtr> model =
|
||||||
|
graph.SideIn(0).SetName("model").Cast<TfLiteModelPtr>();
|
||||||
|
|
||||||
|
auto& inference_node = graph.AddNode("InferenceCalculator");
|
||||||
|
auto& inference_opts =
|
||||||
|
inference_node.GetOptions<InferenceCalculatorOptions>();
|
||||||
|
// Requesting GPU delegate.
|
||||||
|
inference_opts.mutable_delegate()->mutable_gpu();
|
||||||
|
input_tensors.ConnectTo(inference_node.In("TENSORS"));
|
||||||
|
model.ConnectTo(inference_node.SideIn("MODEL"));
|
||||||
|
Stream<std::vector<Tensor>> output_tensors =
|
||||||
|
inference_node.Out("TENSORS").Cast<std::vector<Tensor>>();
|
||||||
|
|
||||||
|
// Graph outputs.
|
||||||
|
output_tensors.SetName("output_tensors").ConnectTo(graph.Out(0));
|
||||||
|
|
||||||
|
// Get `CalculatorGraphConfig` to pass it into `CalculatorGraph`
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Short summary:
|
||||||
|
|
||||||
|
* Use `Graph::In/SideIn` to get graph inputs as `Stream/SidePacket`
|
||||||
|
* Use `Node::Out/SideOut` to get node outputs as `Stream/SidePacket`
|
||||||
|
* Use `Stream/SidePacket::ConnectTo` to connect streams and side packets to
|
||||||
|
node inputs (`Node::In/SideIn`) and graph outputs (`Graph::Out/SideOut`)
|
||||||
|
* There's a "shortcut" operator `>>` that you can use instead of
|
||||||
|
`ConnectTo` function (E.g. `x >> node.In("IN")`).
|
||||||
|
* `Stream/SidePacket::Cast` is used to cast stream or side packet of `AnyType`
|
||||||
|
(E.g. `Stream<AnyType> in = graph.In(0);`) to a particular type
|
||||||
|
* Using actual types instead of `AnyType` sets you on a better path for
|
||||||
|
unleashing graph builder capabilities and improving your graphs
|
||||||
|
readability.
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Utility Functions
|
||||||
|
|
||||||
|
Let's extract inference construction code into a dedicated utility function to
|
||||||
|
help for readability and code reuse:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
// Updates graph to run inference.
|
||||||
|
Stream<std::vector<Tensor>> RunInference(
|
||||||
|
Stream<std::vector<Tensor>> tensors, SidePacket<TfLiteModelPtr> model,
|
||||||
|
const InferenceCalculatorOptions::Delegate& delegate, Graph& graph) {
|
||||||
|
auto& inference_node = graph.AddNode("InferenceCalculator");
|
||||||
|
auto& inference_opts =
|
||||||
|
inference_node.GetOptions<InferenceCalculatorOptions>();
|
||||||
|
*inference_opts.mutable_delegate() = delegate;
|
||||||
|
tensors.ConnectTo(inference_node.In("TENSORS"));
|
||||||
|
model.ConnectTo(inference_node.SideIn("MODEL"));
|
||||||
|
return inference_node.Out("TENSORS").Cast<std::vector<Tensor>>();
|
||||||
|
}
|
||||||
|
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Graph inputs.
|
||||||
|
Stream<std::vector<Tensor>> input_tensors =
|
||||||
|
graph.In(0).SetName("input_tensors").Cast<std::vector<Tensor>>();
|
||||||
|
SidePacket<TfLiteModelPtr> model =
|
||||||
|
graph.SideIn(0).SetName("model").Cast<TfLiteModelPtr>();
|
||||||
|
|
||||||
|
InferenceCalculatorOptions::Delegate delegate;
|
||||||
|
delegate.mutable_gpu();
|
||||||
|
Stream<std::vector<Tensor>> output_tensors =
|
||||||
|
RunInference(input_tensors, model, delegate, graph);
|
||||||
|
|
||||||
|
// Graph outputs.
|
||||||
|
output_tensors.SetName("output_tensors").ConnectTo(graph.Out(0));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
As a result, `RunInference` provides a clear interface stating what are the
|
||||||
|
inputs/outputs and their types.
|
||||||
|
|
||||||
|
It can be easily reused, e.g. it's only a few lines if you want to run an extra
|
||||||
|
model inference:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
// Run first inference.
|
||||||
|
Stream<std::vector<Tensor>> output_tensors =
|
||||||
|
RunInference(input_tensors, model, delegate, graph);
|
||||||
|
// Run second inference on the output of the first one.
|
||||||
|
Stream<std::vector<Tensor>> extra_output_tensors =
|
||||||
|
RunInference(output_tensors, extra_model, delegate, graph);
|
||||||
|
```
|
||||||
|
|
||||||
|
And you don't need to duplicate names and tags (`InferenceCalculator`,
|
||||||
|
`TENSORS`, `MODEL`) or introduce dedicated constants here and there - those
|
||||||
|
details are localized to `RunInference` function.
|
||||||
|
|
||||||
|
Tip: extracting `RunInference` and similar functions to dedicated modules (e.g.
|
||||||
|
inference.h/cc which depends on the inference calculator) enables reuse in
|
||||||
|
graphs construction code and helps automatically pull in calculator dependencies
|
||||||
|
(e.g. no need to manually add `:inference_calculator` dep, just let your IDE
|
||||||
|
include `inference.h` and build cleaner pull in corresponding dependency).
|
||||||
|
|
||||||
|
### Utility Classes
|
||||||
|
|
||||||
|
And surely, it's not only about functions, in some cases it's beneficial to
|
||||||
|
introduce utility classes which can help making your graph construction code
|
||||||
|
more readable and less error prone.
|
||||||
|
|
||||||
|
MediaPipe offers `PassThroughCalculator` calculator, which is simply passing
|
||||||
|
through its inputs:
|
||||||
|
|
||||||
|
```
|
||||||
|
input_stream: "float_value"
|
||||||
|
input_stream: "int_value"
|
||||||
|
input_stream: "bool_value"
|
||||||
|
|
||||||
|
output_stream: "passed_float_value"
|
||||||
|
output_stream: "passed_int_value"
|
||||||
|
output_stream: "passed_bool_value"
|
||||||
|
|
||||||
|
node {
|
||||||
|
calculator: "PassThroughCalculator"
|
||||||
|
input_stream: "float_value"
|
||||||
|
input_stream: "int_value"
|
||||||
|
input_stream: "bool_value"
|
||||||
|
# The order must be the same as for inputs (or you can use explicit indexes)
|
||||||
|
output_stream: "passed_float_value"
|
||||||
|
output_stream: "passed_int_value"
|
||||||
|
output_stream: "passed_bool_value"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Let's see the straightforward C++ construction code to create the above graph:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Graph inputs.
|
||||||
|
Stream<float> float_value = graph.In(0).SetName("float_value").Cast<float>();
|
||||||
|
Stream<int> int_value = graph.In(1).SetName("int_value").Cast<int>();
|
||||||
|
Stream<bool> bool_value = graph.In(2).SetName("bool_value").Cast<bool>();
|
||||||
|
|
||||||
|
auto& pass_node = graph.AddNode("PassThroughCalculator");
|
||||||
|
float_value.ConnectTo(pass_node.In("")[0]);
|
||||||
|
int_value.ConnectTo(pass_node.In("")[1]);
|
||||||
|
bool_value.ConnectTo(pass_node.In("")[2]);
|
||||||
|
Stream<float> passed_float_value = pass_node.Out("")[0].Cast<float>();
|
||||||
|
Stream<int> passed_int_value = pass_node.Out("")[1].Cast<int>();
|
||||||
|
Stream<bool> passed_bool_value = pass_node.Out("")[2].Cast<bool>();
|
||||||
|
|
||||||
|
// Graph outputs.
|
||||||
|
passed_float_value.SetName("passed_float_value").ConnectTo(graph.Out(0));
|
||||||
|
passed_int_value.SetName("passed_int_value").ConnectTo(graph.Out(1));
|
||||||
|
passed_bool_value.SetName("passed_bool_value").ConnectTo(graph.Out(2));
|
||||||
|
|
||||||
|
// Get `CalculatorGraphConfig` to pass it into `CalculatorGraph`
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
While `pbtxt` representation maybe error prone (when we have many inputs to pass
|
||||||
|
through), C++ code looks even worse: repeated empty tags and `Cast` calls. Let's
|
||||||
|
see how we can do better by introducing a `PassThroughNodeBuilder`:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
class PassThroughNodeBuilder {
|
||||||
|
public:
|
||||||
|
explicit PassThroughNodeBuilder(Graph& graph)
|
||||||
|
: node_(graph.AddNode("PassThroughCalculator")) {}
|
||||||
|
|
||||||
|
template <typename T>
|
||||||
|
Stream<T> PassThrough(Stream<T> stream) {
|
||||||
|
stream.ConnectTo(node_.In(index_));
|
||||||
|
return node_.Out(index_++).Cast<T>();
|
||||||
|
}
|
||||||
|
|
||||||
|
private:
|
||||||
|
int index_ = 0;
|
||||||
|
GenericNode& node_;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
And now graph construction code can look like:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Graph inputs.
|
||||||
|
Stream<float> float_value = graph.In(0).SetName("float_value").Cast<float>();
|
||||||
|
Stream<int> int_value = graph.In(1).SetName("int_value").Cast<int>();
|
||||||
|
Stream<bool> bool_value = graph.In(2).SetName("bool_value").Cast<bool>();
|
||||||
|
|
||||||
|
PassThroughNodeBuilder pass_node_builder(graph);
|
||||||
|
Stream<float> passed_float_value = pass_node_builder.PassThrough(float_value);
|
||||||
|
Stream<int> passed_int_value = pass_node_builder.PassThrough(int_value);
|
||||||
|
Stream<bool> passed_bool_value = pass_node_builder.PassThrough(bool_value);
|
||||||
|
|
||||||
|
// Graph outputs.
|
||||||
|
passed_float_value.SetName("passed_float_value").ConnectTo(graph.Out(0));
|
||||||
|
passed_int_value.SetName("passed_int_value").ConnectTo(graph.Out(1));
|
||||||
|
passed_bool_value.SetName("passed_bool_value").ConnectTo(graph.Out(2));
|
||||||
|
|
||||||
|
// Get `CalculatorGraphConfig` to pass it into `CalculatorGraph`
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Now you can't have incorrect order or index in your pass through construction
|
||||||
|
code and save some typing by guessing the type for `Cast` from the `PassThrough`
|
||||||
|
input.
|
||||||
|
|
||||||
|
Tip: the same as for the `RunInference` function, extracting
|
||||||
|
`PassThroughNodeBuilder` and similar utility classes into dedicated modules
|
||||||
|
enables reuse in graph construction code and helps to automatically pull in the
|
||||||
|
corresponding calculator dependencies.
|
||||||
|
|
||||||
|
## Dos and Don'ts
|
||||||
|
|
||||||
|
### Define graph inputs at the very beginning if possible
|
||||||
|
|
||||||
|
```c++ {.bad}
|
||||||
|
Stream<D> RunSomething(Stream<A> a, Stream<B> b, Graph& graph) {
|
||||||
|
Stream<C> c = graph.In(2).SetName("c").Cast<C>(); // Bad.
|
||||||
|
// ...
|
||||||
|
}
|
||||||
|
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
Stream<A> a = graph.In(0).SetName("a").Cast<A>();
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
Stream<B> b = graph.In(1).SetName("b").Cast<B>() // Bad.
|
||||||
|
Stream<D> d = RunSomething(a, b, graph);
|
||||||
|
// ...
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
In the above code:
|
||||||
|
|
||||||
|
* It can be hard to guess how many inputs you have in the graph.
|
||||||
|
* Can be error prone overall and hard to maintain in future (e.g. is it a
|
||||||
|
correct index? name? what if some inputs are removed or made optional?
|
||||||
|
etc.).
|
||||||
|
* `RunSomething` reuse is limited because other graphs may have different
|
||||||
|
inputs
|
||||||
|
|
||||||
|
Instead, define your graph inputs at the very beginning of your graph builder:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
Stream<D> RunSomething(Stream<A> a, Stream<B> b, Stream<C> c, Graph& graph) {
|
||||||
|
// ...
|
||||||
|
}
|
||||||
|
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).SetName("a").Cast<A>();
|
||||||
|
Stream<B> b = graph.In(1).SetName("b").Cast<B>();
|
||||||
|
Stream<C> c = graph.In(2).SetName("c").Cast<C>();
|
||||||
|
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
Stream<D> d = RunSomething(a, b, c, graph);
|
||||||
|
// ...
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `std::optional` if you have an input stream or side packet that is not
|
||||||
|
always defined and put it at the very beginning:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
std::optional<Stream<A>> a;
|
||||||
|
if (needs_a) {
|
||||||
|
a = graph.In(0).SetName(a).Cast<A>();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: of course, there can be exceptions - for example, there can be a use case
|
||||||
|
where calling `RunSomething1(..., graph)`, ..., `RunSomethingN(..., graph)` is
|
||||||
|
**intended to add new inputs**, so afterwards you can iterate over them and feed
|
||||||
|
only added inputs into the graph. However, in any case, try to make it easy for
|
||||||
|
readers to find out what graph inputs it has or may have.
|
||||||
|
|
||||||
|
### Define graph outputs at the very end
|
||||||
|
|
||||||
|
```c++ {.bad}
|
||||||
|
void RunSomething(Stream<Input> input, Graph& graph) {
|
||||||
|
// ...
|
||||||
|
node.Out("OUTPUT_F")
|
||||||
|
.SetName("output_f").ConnectTo(graph.Out(2)); // Bad.
|
||||||
|
}
|
||||||
|
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
node.Out("OUTPUT_D")
|
||||||
|
.SetName("output_d").ConnectTo(graph.Out(0)); // Bad.
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
node.Out("OUTPUT_E")
|
||||||
|
.SetName("output_e").ConnectTo(graph.Out(1)); // Bad.
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
RunSomething(input, graph);
|
||||||
|
// ...
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
In the above code:
|
||||||
|
|
||||||
|
* It can be hard to guess how many outputs you have in the graph.
|
||||||
|
* Can be error prone overall and hard to maintain in future (e.g. is it a
|
||||||
|
correct index? name? what if some outpus are removed or made optional?
|
||||||
|
etc.).
|
||||||
|
* `RunSomething` reuse is limited as other graphs may have different outputs
|
||||||
|
|
||||||
|
Instead, define your graph outputs at the very end of your graph builder:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
Stream<F> RunSomething(Stream<Input> input, Graph& graph) {
|
||||||
|
// ...
|
||||||
|
return node.Out("OUTPUT_F").Cast<F>();
|
||||||
|
}
|
||||||
|
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
Stream<D> d = node.Out("OUTPUT_D").Cast<D>();
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
Stream<E> e = node.Out("OUTPUT_E").Cast<E>();
|
||||||
|
// 10/100/N lines of code.
|
||||||
|
Stream<F> f = RunSomething(input, graph);
|
||||||
|
// ...
|
||||||
|
|
||||||
|
// Outputs.
|
||||||
|
d.SetName("output_d").ConnectTo(graph.Out(0));
|
||||||
|
e.SetName("output_e").ConnectTo(graph.Out(1));
|
||||||
|
f.SetName("output_f").ConnectTo(graph.Out(2));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Keep nodes decoupled from each other
|
||||||
|
|
||||||
|
In MediaPipe, packet streams and side packets are as meaningful as processing
|
||||||
|
nodes. And any node input requirements and output products are expressed clearly
|
||||||
|
and independently in terms of the streams and side packets it consumes and
|
||||||
|
produces.
|
||||||
|
|
||||||
|
```c++ {.bad}
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).Cast<A>();
|
||||||
|
|
||||||
|
auto& node1 = graph.AddNode("Calculator1");
|
||||||
|
a.ConnectTo(node1.In("INPUT"));
|
||||||
|
|
||||||
|
auto& node2 = graph.AddNode("Calculator2");
|
||||||
|
node1.Out("OUTPUT").ConnectTo(node2.In("INPUT")); // Bad.
|
||||||
|
|
||||||
|
auto& node3 = graph.AddNode("Calculator3");
|
||||||
|
node1.Out("OUTPUT").ConnectTo(node3.In("INPUT_B")); // Bad.
|
||||||
|
node2.Out("OUTPUT").ConnectTo(node3.In("INPUT_C")); // Bad.
|
||||||
|
|
||||||
|
auto& node4 = graph.AddNode("Calculator4");
|
||||||
|
node1.Out("OUTPUT").ConnectTo(node4.In("INPUT_B")); // Bad.
|
||||||
|
node2.Out("OUTPUT").ConnectTo(node4.In("INPUT_C")); // Bad.
|
||||||
|
node3.Out("OUTPUT").ConnectTo(node4.In("INPUT_D")); // Bad.
|
||||||
|
|
||||||
|
// Outputs.
|
||||||
|
node1.Out("OUTPUT").SetName("b").ConnectTo(graph.Out(0)); // Bad.
|
||||||
|
node2.Out("OUTPUT").SetName("c").ConnectTo(graph.Out(1)); // Bad.
|
||||||
|
node3.Out("OUTPUT").SetName("d").ConnectTo(graph.Out(2)); // Bad.
|
||||||
|
node4.Out("OUTPUT").SetName("e").ConnectTo(graph.Out(3)); // Bad.
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
In the above code:
|
||||||
|
|
||||||
|
* Nodes are coupled to each other, e.g. `node4` knows where its inputs are
|
||||||
|
coming from (`node1`, `node2`, `node3`) and it complicates refactoring,
|
||||||
|
maintenance and code reuse
|
||||||
|
* Such usage pattern is a downgrade from proto representation, where nodes
|
||||||
|
are decoupled by default.
|
||||||
|
* `node#.Out("OUTPUT")` calls are duplicated and readability suffers as you
|
||||||
|
could use cleaner names instead and also provide an actual type.
|
||||||
|
|
||||||
|
So, to fix the above issues you can write the following graph construction code:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).Cast<A>();
|
||||||
|
|
||||||
|
// `node1` usage is limited to 3 lines below.
|
||||||
|
auto& node1 = graph.AddNode("Calculator1");
|
||||||
|
a.ConnectTo(node1.In("INPUT"));
|
||||||
|
Stream<B> b = node1.Out("OUTPUT").Cast<B>();
|
||||||
|
|
||||||
|
// `node2` usage is limited to 3 lines below.
|
||||||
|
auto& node2 = graph.AddNode("Calculator2");
|
||||||
|
b.ConnectTo(node2.In("INPUT"));
|
||||||
|
Stream<C> c = node2.Out("OUTPUT").Cast<C>();
|
||||||
|
|
||||||
|
// `node3` usage is limited to 4 lines below.
|
||||||
|
auto& node3 = graph.AddNode("Calculator3");
|
||||||
|
b.ConnectTo(node3.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node3.In("INPUT_C"));
|
||||||
|
Stream<D> d = node3.Out("OUTPUT").Cast<D>();
|
||||||
|
|
||||||
|
// `node4` usage is limited to 5 lines below.
|
||||||
|
auto& node4 = graph.AddNode("Calculator4");
|
||||||
|
b.ConnectTo(node4.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node4.In("INPUT_C"));
|
||||||
|
d.ConnectTo(node4.In("INPUT_D"));
|
||||||
|
Stream<E> e = node4.Out("OUTPUT").Cast<E>();
|
||||||
|
|
||||||
|
// Outputs.
|
||||||
|
b.SetName("b").ConnectTo(graph.Out(0));
|
||||||
|
c.SetName("c").ConnectTo(graph.Out(1));
|
||||||
|
d.SetName("d").ConnectTo(graph.Out(2));
|
||||||
|
e.SetName("e").ConnectTo(graph.Out(3));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Now, if needed, you can easily remove `node1` and make `b` a graph input and no
|
||||||
|
updates are needed to `node2`, `node3`, `node4` (same as in proto representation
|
||||||
|
by the way), because they are decoupled from each other.
|
||||||
|
|
||||||
|
Overall, the above code replicates the proto graph more closely:
|
||||||
|
|
||||||
|
```proto
|
||||||
|
input_stream: "a"
|
||||||
|
|
||||||
|
node {
|
||||||
|
calculator: "Calculator1"
|
||||||
|
input_stream: "INPUT:a"
|
||||||
|
output_stream: "OUTPUT:b"
|
||||||
|
}
|
||||||
|
|
||||||
|
node {
|
||||||
|
calculator: "Calculator2"
|
||||||
|
input_stream: "INPUT:b"
|
||||||
|
output_stream: "OUTPUT:C"
|
||||||
|
}
|
||||||
|
|
||||||
|
node {
|
||||||
|
calculator: "Calculator3"
|
||||||
|
input_stream: "INPUT_B:b"
|
||||||
|
input_stream: "INPUT_C:c"
|
||||||
|
output_stream: "OUTPUT:d"
|
||||||
|
}
|
||||||
|
|
||||||
|
node {
|
||||||
|
calculator: "Calculator4"
|
||||||
|
input_stream: "INPUT_B:b"
|
||||||
|
input_stream: "INPUT_C:c"
|
||||||
|
input_stream: "INPUT_D:d"
|
||||||
|
output_stream: "OUTPUT:e"
|
||||||
|
}
|
||||||
|
|
||||||
|
output_stream: "b"
|
||||||
|
output_stream: "c"
|
||||||
|
output_stream: "d"
|
||||||
|
output_stream: "e"
|
||||||
|
```
|
||||||
|
|
||||||
|
On top of that, now you can extract utility functions for further reuse in other graphs:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
Stream<B> RunCalculator1(Stream<A> a, Graph& graph) {
|
||||||
|
auto& node = graph.AddNode("Calculator1");
|
||||||
|
a.ConnectTo(node.In("INPUT"));
|
||||||
|
return node.Out("OUTPUT").Cast<B>();
|
||||||
|
}
|
||||||
|
|
||||||
|
Stream<C> RunCalculator2(Stream<B> b, Graph& graph) {
|
||||||
|
auto& node = graph.AddNode("Calculator2");
|
||||||
|
b.ConnectTo(node.In("INPUT"));
|
||||||
|
return node.Out("OUTPUT").Cast<C>();
|
||||||
|
}
|
||||||
|
|
||||||
|
Stream<D> RunCalculator3(Stream<B> b, Stream<C> c, Graph& graph) {
|
||||||
|
auto& node = graph.AddNode("Calculator3");
|
||||||
|
b.ConnectTo(node.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node.In("INPUT_C"));
|
||||||
|
return node.Out("OUTPUT").Cast<D>();
|
||||||
|
}
|
||||||
|
|
||||||
|
Stream<E> RunCalculator4(Stream<B> b, Stream<C> c, Stream<D> d, Graph& graph) {
|
||||||
|
auto& node = graph.AddNode("Calculator4");
|
||||||
|
b.ConnectTo(node.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node.In("INPUT_C"));
|
||||||
|
d.ConnectTo(node.In("INPUT_D"));
|
||||||
|
return node.Out("OUTPUT").Cast<E>();
|
||||||
|
}
|
||||||
|
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).Cast<A>();
|
||||||
|
|
||||||
|
Stream<B> b = RunCalculator1(a, graph);
|
||||||
|
Stream<C> c = RunCalculator2(b, graph);
|
||||||
|
Stream<D> d = RunCalculator3(b, c, graph);
|
||||||
|
Stream<E> e = RunCalculator4(b, c, d, graph);
|
||||||
|
|
||||||
|
// Outputs.
|
||||||
|
b.SetName("b").ConnectTo(graph.Out(0));
|
||||||
|
c.SetName("c").ConnectTo(graph.Out(1));
|
||||||
|
d.SetName("d").ConnectTo(graph.Out(2));
|
||||||
|
e.SetName("e").ConnectTo(graph.Out(3));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Separate nodes for better readability
|
||||||
|
|
||||||
|
```c++ {.bad}
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).Cast<A>();
|
||||||
|
auto& node1 = graph.AddNode("Calculator1");
|
||||||
|
a.ConnectTo(node1.In("INPUT"));
|
||||||
|
Stream<B> b = node1.Out("OUTPUT").Cast<B>();
|
||||||
|
auto& node2 = graph.AddNode("Calculator2");
|
||||||
|
b.ConnectTo(node2.In("INPUT"));
|
||||||
|
Stream<C> c = node2.Out("OUTPUT").Cast<C>();
|
||||||
|
auto& node3 = graph.AddNode("Calculator3");
|
||||||
|
b.ConnectTo(node3.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node3.In("INPUT_C"));
|
||||||
|
Stream<D> d = node3.Out("OUTPUT").Cast<D>();
|
||||||
|
auto& node4 = graph.AddNode("Calculator4");
|
||||||
|
b.ConnectTo(node4.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node4.In("INPUT_C"));
|
||||||
|
d.ConnectTo(node4.In("INPUT_D"));
|
||||||
|
Stream<E> e = node4.Out("OUTPUT").Cast<E>();
|
||||||
|
// Outputs.
|
||||||
|
b.SetName("b").ConnectTo(graph.Out(0));
|
||||||
|
c.SetName("c").ConnectTo(graph.Out(1));
|
||||||
|
d.SetName("d").ConnectTo(graph.Out(2));
|
||||||
|
e.SetName("e").ConnectTo(graph.Out(3));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
In the above code, it can be hard to grasp the idea where each node begins and
|
||||||
|
ends. To improve this and help your code readers, you can simply have blank
|
||||||
|
lines before and after each node:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).Cast<A>();
|
||||||
|
|
||||||
|
auto& node1 = graph.AddNode("Calculator1");
|
||||||
|
a.ConnectTo(node1.In("INPUT"));
|
||||||
|
Stream<B> b = node1.Out("OUTPUT").Cast<B>();
|
||||||
|
|
||||||
|
auto& node2 = graph.AddNode("Calculator2");
|
||||||
|
b.ConnectTo(node2.In("INPUT"));
|
||||||
|
Stream<C> c = node2.Out("OUTPUT").Cast<C>();
|
||||||
|
|
||||||
|
auto& node3 = graph.AddNode("Calculator3");
|
||||||
|
b.ConnectTo(node3.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node3.In("INPUT_C"));
|
||||||
|
Stream<D> d = node3.Out("OUTPUT").Cast<D>();
|
||||||
|
|
||||||
|
auto& node4 = graph.AddNode("Calculator4");
|
||||||
|
b.ConnectTo(node4.In("INPUT_B"));
|
||||||
|
c.ConnectTo(node4.In("INPUT_C"));
|
||||||
|
d.ConnectTo(node4.In("INPUT_D"));
|
||||||
|
Stream<E> e = node4.Out("OUTPUT").Cast<E>();
|
||||||
|
|
||||||
|
// Outputs.
|
||||||
|
b.SetName("b").ConnectTo(graph.Out(0));
|
||||||
|
c.SetName("c").ConnectTo(graph.Out(1));
|
||||||
|
d.SetName("d").ConnectTo(graph.Out(2));
|
||||||
|
e.SetName("e").ConnectTo(graph.Out(3));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Also, the above representation matches `CalculatorGraphConfig` proto
|
||||||
|
representation better.
|
||||||
|
|
||||||
|
If you extract nodes into utility functions, they are scoped within functions
|
||||||
|
already and it's clear where they begin and end, so it's completely fine to
|
||||||
|
have:
|
||||||
|
|
||||||
|
```c++ {.good}
|
||||||
|
CalculatorGraphConfig BuildGraph() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Inputs.
|
||||||
|
Stream<A> a = graph.In(0).Cast<A>();
|
||||||
|
|
||||||
|
Stream<B> b = RunCalculator1(a, graph);
|
||||||
|
Stream<C> c = RunCalculator2(b, graph);
|
||||||
|
Stream<D> d = RunCalculator3(b, c, graph);
|
||||||
|
Stream<E> e = RunCalculator4(b, c, d, graph);
|
||||||
|
|
||||||
|
// Outputs.
|
||||||
|
b.SetName("b").ConnectTo(graph.Out(0));
|
||||||
|
c.SetName("c").ConnectTo(graph.Out(1));
|
||||||
|
d.SetName("d").ConnectTo(graph.Out(2));
|
||||||
|
e.SetName("e").ConnectTo(graph.Out(3));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/calculators
|
||||||
title: Calculators
|
title: Calculators
|
||||||
parent: Framework Concepts
|
parent: Framework Concepts
|
||||||
nav_order: 1
|
nav_order: 1
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/overview
|
||||||
title: Framework Concepts
|
title: Framework Concepts
|
||||||
nav_order: 5
|
nav_order: 5
|
||||||
has_children: true
|
has_children: true
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/gpu
|
||||||
title: GPU
|
title: GPU
|
||||||
parent: Framework Concepts
|
parent: Framework Concepts
|
||||||
nav_order: 5
|
nav_order: 5
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/graphs
|
||||||
title: Graphs
|
title: Graphs
|
||||||
parent: Framework Concepts
|
parent: Framework Concepts
|
||||||
nav_order: 2
|
nav_order: 2
|
||||||
@@ -12,31 +13,31 @@ nav_order: 2
|
|||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
## GraphConfig
|
## Graph
|
||||||
|
|
||||||
A `GraphConfig` is a specification that describes the topology and functionality
|
A `CalculatorGraphConfig` proto specifies the topology and functionality of a
|
||||||
of a MediaPipe graph. In the specification, a node in the graph represents an
|
MediaPipe graph. Each `node` in the graph represents a particular calculator or
|
||||||
instance of a particular calculator. All the necessary configurations of the
|
subgraph, and specifies necessary configurations, such as registered
|
||||||
node, such its type, inputs and outputs must be described in the specification.
|
calculator/subgraph type, inputs, outputs and optional fields, such as
|
||||||
Description of the node can also include several optional fields, such as
|
|
||||||
node-specific options, input policy and executor, discussed in
|
node-specific options, input policy and executor, discussed in
|
||||||
[Synchronization](synchronization.md).
|
[Synchronization](synchronization.md).
|
||||||
|
|
||||||
`GraphConfig` has several other fields to configure the global graph-level
|
`CalculatorGraphConfig` has several other fields to configure global graph-level
|
||||||
settings, eg, graph executor configs, number of threads, and maximum queue size
|
settings, e.g. graph executor configs, number of threads, and maximum queue size
|
||||||
of input streams. Several graph-level settings are useful for tuning the
|
of input streams. Several graph-level settings are useful for tuning the
|
||||||
performance of the graph on different platforms (eg, desktop v.s. mobile). For
|
performance of the graph on different platforms (e.g., desktop v.s. mobile). For
|
||||||
instance, on mobile, attaching a heavy model-inference calculator to a separate
|
instance, on mobile, attaching a heavy model-inference calculator to a separate
|
||||||
executor can improve the performance of a real-time application since this
|
executor can improve the performance of a real-time application since this
|
||||||
enables thread locality.
|
enables thread locality.
|
||||||
|
|
||||||
Below is a trivial `GraphConfig` example where we have series of passthrough
|
Below is a trivial `CalculatorGraphConfig` example where we have series of
|
||||||
calculators :
|
passthrough calculators :
|
||||||
|
|
||||||
```proto
|
```proto
|
||||||
# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
|
# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
|
||||||
# passthrough calculators.
|
# passthrough calculators.
|
||||||
input_stream: "in"
|
input_stream: "in"
|
||||||
|
output_stream: "out"
|
||||||
node {
|
node {
|
||||||
calculator: "PassThroughCalculator"
|
calculator: "PassThroughCalculator"
|
||||||
input_stream: "in"
|
input_stream: "in"
|
||||||
@@ -55,17 +56,46 @@ node {
|
|||||||
node {
|
node {
|
||||||
calculator: "PassThroughCalculator"
|
calculator: "PassThroughCalculator"
|
||||||
input_stream: "out3"
|
input_stream: "out3"
|
||||||
output_stream: "out4"
|
output_stream: "out"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
MediaPipe offers an alternative `C++` representation for complex graphs (e.g. ML pipelines, handling model metadata, optional nodes, etc.). The above graph may look like:
|
||||||
|
|
||||||
|
```c++
|
||||||
|
CalculatorGraphConfig BuildGraphConfig() {
|
||||||
|
Graph graph;
|
||||||
|
|
||||||
|
// Graph inputs
|
||||||
|
Stream<AnyType> in = graph.In(0).SetName("in");
|
||||||
|
|
||||||
|
auto pass_through_fn = [](Stream<AnyType> in,
|
||||||
|
Graph& graph) -> Stream<AnyType> {
|
||||||
|
auto& node = graph.AddNode("PassThroughCalculator");
|
||||||
|
in.ConnectTo(node.In(0));
|
||||||
|
return node.Out(0);
|
||||||
|
};
|
||||||
|
|
||||||
|
Stream<AnyType> out1 = pass_through_fn(in, graph);
|
||||||
|
Stream<AnyType> out2 = pass_through_fn(out1, graph);
|
||||||
|
Stream<AnyType> out3 = pass_through_fn(out2, graph);
|
||||||
|
Stream<AnyType> out4 = pass_through_fn(out3, graph);
|
||||||
|
|
||||||
|
// Graph outputs
|
||||||
|
out4.SetName("out").ConnectTo(graph.Out(0));
|
||||||
|
|
||||||
|
return graph.GetConfig();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
See more details in [Building Graphs in C++](building_graphs_cpp.md)
|
||||||
|
|
||||||
## Subgraph
|
## Subgraph
|
||||||
|
|
||||||
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
|
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
|
||||||
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
|
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
|
||||||
public interface of a subgraph consists of a set of input and output streams
|
public interface of a subgraph consists of a set of input and output streams
|
||||||
similar to a calculator's public interface. The subgraph can then be included in
|
similar to a calculator's public interface. The subgraph can then be included in
|
||||||
an `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is
|
a `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is
|
||||||
loaded from a `CalculatorGraphConfig`, each subgraph node is replaced by the
|
loaded from a `CalculatorGraphConfig`, each subgraph node is replaced by the
|
||||||
corresponding graph of calculators. As a result, the semantics and performance
|
corresponding graph of calculators. As a result, the semantics and performance
|
||||||
of the subgraph is identical to the corresponding graph of calculators.
|
of the subgraph is identical to the corresponding graph of calculators.
|
||||||
@@ -151,7 +181,7 @@ protobuf specified for a MediaPipe calculator. These "graph options" can be
|
|||||||
specified where a graph is invoked, and used to populate calculator options and
|
specified where a graph is invoked, and used to populate calculator options and
|
||||||
subgraph options within the graph.
|
subgraph options within the graph.
|
||||||
|
|
||||||
In a CalculatorGraphConfig, graph options can be specified for a subgraph
|
In a `CalculatorGraphConfig`, graph options can be specified for a subgraph
|
||||||
exactly like calculator options, as shown below:
|
exactly like calculator options, as shown below:
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -178,7 +208,7 @@ node {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
In a CalculatorGraphConfig, graph options can be accepted and used to populate
|
In a `CalculatorGraphConfig`, graph options can be accepted and used to populate
|
||||||
calculator options, as shown below:
|
calculator options, as shown below:
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -188,7 +218,7 @@ graph_options: {
|
|||||||
|
|
||||||
node: {
|
node: {
|
||||||
calculator: "ImageToTensorCalculator"
|
calculator: "ImageToTensorCalculator"
|
||||||
input_stream: "IMAGE:multi_backend_image"
|
input_stream: "IMAGE:image"
|
||||||
node_options: {
|
node_options: {
|
||||||
[type.googleapis.com/mediapipe.ImageToTensorCalculatorOptions] {
|
[type.googleapis.com/mediapipe.ImageToTensorCalculatorOptions] {
|
||||||
keep_aspect_ratio: true
|
keep_aspect_ratio: true
|
||||||
@@ -247,9 +277,9 @@ NOTE: The current approach is experimental and subject to change. We welcome
|
|||||||
your feedback.
|
your feedback.
|
||||||
|
|
||||||
Please use the `CalculatorGraphTest.Cycle` unit test in
|
Please use the `CalculatorGraphTest.Cycle` unit test in
|
||||||
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown
|
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown below is
|
||||||
below is the cyclic graph in the test. The `sum` output of the adder is the sum
|
the cyclic graph in the test. The `sum` output of the adder is the sum of the
|
||||||
of the integers generated by the integer source calculator.
|
integers generated by the integer source calculator.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/packets
|
||||||
title: Packets
|
title: Packets
|
||||||
parent: Framework Concepts
|
parent: Framework Concepts
|
||||||
nav_order: 3
|
nav_order: 3
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/realtime_streams
|
||||||
title: Real-time Streams
|
title: Real-time Streams
|
||||||
parent: Framework Concepts
|
parent: Framework Concepts
|
||||||
nav_order: 6
|
nav_order: 6
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/framework_concepts/synchronization
|
||||||
title: Synchronization
|
title: Synchronization
|
||||||
parent: Framework Concepts
|
parent: Framework Concepts
|
||||||
nav_order: 4
|
nav_order: 4
|
||||||
@@ -112,14 +113,14 @@ Warning: On the other hand, it is not guaranteed that an input packet will
|
|||||||
always be available for all streams.
|
always be available for all streams.
|
||||||
|
|
||||||
To explain how it works, we need to introduce the definition of a settled
|
To explain how it works, we need to introduce the definition of a settled
|
||||||
timestamp. We say that a timestamp in a stream is *settled* if it lower than the
|
timestamp. We say that a timestamp in a stream is *settled* if it is lower than
|
||||||
timestamp bound. In other words, a timestamp is settled for a stream once the
|
the timestamp bound. In other words, a timestamp is settled for a stream once
|
||||||
state of the input at that timestamp is irrevocably known: either there is a
|
the state of the input at that timestamp is irrevocably known: either there is a
|
||||||
packet, or there is the certainty that a packet with that timestamp will not
|
packet, or there is the certainty that a packet with that timestamp will not
|
||||||
arrive.
|
arrive.
|
||||||
|
|
||||||
Note: For this reason, MediaPipe also allows a stream producer to explicitly
|
Note: For this reason, MediaPipe also allows a stream producer to explicitly
|
||||||
advance the timestamp bound farther that what the last packet implies, i.e. to
|
advance the timestamp bound farther than what the last packet implies, i.e. to
|
||||||
provide a tighter bound. This can allow the downstream nodes to settle their
|
provide a tighter bound. This can allow the downstream nodes to settle their
|
||||||
inputs sooner.
|
inputs sooner.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/android
|
||||||
title: MediaPipe on Android
|
title: MediaPipe on Android
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
has_children: true
|
has_children: true
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/android_archive_library
|
||||||
title: MediaPipe Android Archive
|
title: MediaPipe Android Archive
|
||||||
parent: MediaPipe on Android
|
parent: MediaPipe on Android
|
||||||
grand_parent: Getting Started
|
grand_parent: Getting Started
|
||||||
|
|||||||
@@ -13,6 +13,17 @@ nav_order: 2
|
|||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||||
|
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||||
|
as the primary developer documentation
|
||||||
|
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||||
|
the new site, but will remain available in the source code repository on an
|
||||||
|
as-is basis.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
MediaPipe Android Solution APIs (currently in alpha) are available in:
|
MediaPipe Android Solution APIs (currently in alpha) are available in:
|
||||||
|
|
||||||
* [MediaPipe Face Detection](../solutions/face_detection#android-solution-api)
|
* [MediaPipe Face Detection](../solutions/face_detection#android-solution-api)
|
||||||
|
|||||||
@@ -12,6 +12,17 @@ nav_exclude: true
|
|||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||||
|
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||||
|
as the primary developer documentation
|
||||||
|
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||||
|
the new site, but will remain available in the source code repository on an
|
||||||
|
as-is basis.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
### Android
|
### Android
|
||||||
|
|
||||||
Please see these [instructions](./android.md).
|
Please see these [instructions](./android.md).
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/cpp
|
||||||
title: MediaPipe in C++
|
title: MediaPipe in C++
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
has_children: true
|
has_children: true
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/faq
|
||||||
title: FAQ
|
title: FAQ
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
nav_order: 9
|
nav_order: 9
|
||||||
@@ -59,7 +60,7 @@ The second approach allows up to [`max_in_flight`] invocations of the
|
|||||||
packets from [`CalculatorBase::Process`] are automatically ordered by timestamp
|
packets from [`CalculatorBase::Process`] are automatically ordered by timestamp
|
||||||
before they are passed along to downstream calculators.
|
before they are passed along to downstream calculators.
|
||||||
|
|
||||||
With either aproach, you must be aware that the calculator running in parallel
|
With either approach, you must be aware that the calculator running in parallel
|
||||||
cannot maintain internal state in the same way as a normal sequential
|
cannot maintain internal state in the same way as a normal sequential
|
||||||
calculator.
|
calculator.
|
||||||
|
|
||||||
|
|||||||
@@ -11,3 +11,14 @@ has_children: true
|
|||||||
1. TOC
|
1. TOC
|
||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||||
|
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||||
|
as the primary developer documentation
|
||||||
|
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||||
|
the new site, but will remain available in the source code repository on an
|
||||||
|
as-is basis.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/gpu_support
|
||||||
title: GPU Support
|
title: GPU Support
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
nav_order: 7
|
nav_order: 7
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/hello_world_android
|
||||||
title: Hello World! on Android
|
title: Hello World! on Android
|
||||||
parent: MediaPipe on Android
|
parent: MediaPipe on Android
|
||||||
grand_parent: Getting Started
|
grand_parent: Getting Started
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/hello_world_cpp
|
||||||
title: Hello World! in C++
|
title: Hello World! in C++
|
||||||
parent: MediaPipe in C++
|
parent: MediaPipe in C++
|
||||||
grand_parent: Getting Started
|
grand_parent: Getting Started
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/hello_world_ios
|
||||||
title: Hello World! on iOS
|
title: Hello World! on iOS
|
||||||
parent: MediaPipe on iOS
|
parent: MediaPipe on iOS
|
||||||
grand_parent: Getting Started
|
grand_parent: Getting Started
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/help
|
||||||
title: Getting Help
|
title: Getting Help
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
nav_order: 8
|
nav_order: 8
|
||||||
@@ -37,8 +38,8 @@ If you open a GitHub issue, here is our policy:
|
|||||||
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
|
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
|
||||||
- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
|
- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
|
||||||
- **Bazel version**:
|
- **Bazel version**:
|
||||||
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev enviroment)**:
|
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev environment)**:
|
||||||
- **Xcode & Tulsi version (if issue is related to building in mobile dev enviroment)**:
|
- **Xcode & Tulsi version (if issue is related to building in mobile dev environment)**:
|
||||||
- **Exact steps to reproduce**:
|
- **Exact steps to reproduce**:
|
||||||
|
|
||||||
### Describe the problem
|
### Describe the problem
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/install
|
||||||
title: Installation
|
title: Installation
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
nav_order: 6
|
nav_order: 6
|
||||||
@@ -35,7 +36,7 @@ install --user six`.
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
$ cd $HOME
|
$ cd $HOME
|
||||||
$ git clone https://github.com/google/mediapipe.git
|
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||||
|
|
||||||
# Change directory into MediaPipe root directory
|
# Change directory into MediaPipe root directory
|
||||||
$ cd mediapipe
|
$ cd mediapipe
|
||||||
@@ -67,6 +68,13 @@ install --user six`.
|
|||||||
libopencv-video-dev
|
libopencv-video-dev
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Note**. On Debian 11/Ubuntu 21.04 where OpenCV 4.5 is installed with
|
||||||
|
`libopencv-video-dev`, `libopencv-contrib-dev` should also be installed.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ sudo apt-get install -y libopencv-contrib-dev
|
||||||
|
```
|
||||||
|
|
||||||
MediaPipe's [`opencv_linux.BUILD`] and [`WORKSPACE`] are already configured
|
MediaPipe's [`opencv_linux.BUILD`] and [`WORKSPACE`] are already configured
|
||||||
for OpenCV 2/3 and should work correctly on any architecture:
|
for OpenCV 2/3 and should work correctly on any architecture:
|
||||||
|
|
||||||
@@ -287,7 +295,7 @@ build issues.
|
|||||||
2. Checkout MediaPipe repository.
|
2. Checkout MediaPipe repository.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$ git clone https://github.com/google/mediapipe.git
|
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||||
|
|
||||||
# Change directory into MediaPipe root directory
|
# Change directory into MediaPipe root directory
|
||||||
$ cd mediapipe
|
$ cd mediapipe
|
||||||
@@ -416,7 +424,7 @@ build issues.
|
|||||||
3. Checkout MediaPipe repository.
|
3. Checkout MediaPipe repository.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$ git clone https://github.com/google/mediapipe.git
|
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||||
|
|
||||||
$ cd mediapipe
|
$ cd mediapipe
|
||||||
```
|
```
|
||||||
@@ -590,7 +598,7 @@ next section.
|
|||||||
7. Checkout MediaPipe repository.
|
7. Checkout MediaPipe repository.
|
||||||
|
|
||||||
```
|
```
|
||||||
C:\Users\Username\mediapipe_repo> git clone https://github.com/google/mediapipe.git
|
C:\Users\Username\mediapipe_repo> git clone --depth 1 https://github.com/google/mediapipe.git
|
||||||
|
|
||||||
# Change directory into MediaPipe root directory
|
# Change directory into MediaPipe root directory
|
||||||
C:\Users\Username\mediapipe_repo> cd mediapipe
|
C:\Users\Username\mediapipe_repo> cd mediapipe
|
||||||
@@ -651,7 +659,7 @@ cameras. Alternatively, you use a video file as input.
|
|||||||
|
|
||||||
1. Follow the
|
1. Follow the
|
||||||
[instruction](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to
|
[instruction](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to
|
||||||
install Windows Sysystem for Linux (Ubuntu).
|
install Windows Subsystem for Linux (Ubuntu).
|
||||||
|
|
||||||
2. Install Windows ADB and start the ADB server in Windows.
|
2. Install Windows ADB and start the ADB server in Windows.
|
||||||
|
|
||||||
@@ -680,7 +688,7 @@ cameras. Alternatively, you use a video file as input.
|
|||||||
6. Checkout MediaPipe repository.
|
6. Checkout MediaPipe repository.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
username@DESKTOP-TMVLBJ1:~$ git clone https://github.com/google/mediapipe.git
|
username@DESKTOP-TMVLBJ1:~$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||||
|
|
||||||
username@DESKTOP-TMVLBJ1:~$ cd mediapipe
|
username@DESKTOP-TMVLBJ1:~$ cd mediapipe
|
||||||
```
|
```
|
||||||
@@ -771,7 +779,7 @@ This will use a Docker image that will isolate mediapipe's installation from the
|
|||||||
2. Build a docker image with tag "mediapipe".
|
2. Build a docker image with tag "mediapipe".
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$ git clone https://github.com/google/mediapipe.git
|
$ git clone --depth 1 https://github.com/google/mediapipe.git
|
||||||
$ cd mediapipe
|
$ cd mediapipe
|
||||||
$ docker build --tag=mediapipe .
|
$ docker build --tag=mediapipe .
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/ios
|
||||||
title: MediaPipe on iOS
|
title: MediaPipe on iOS
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
has_children: true
|
has_children: true
|
||||||
@@ -91,6 +92,10 @@ allows you to make use of automatic provisioning (see later section).
|
|||||||
This will install `Tulsi.app` inside the `Applications` directory in your
|
This will install `Tulsi.app` inside the `Applications` directory in your
|
||||||
home directory.
|
home directory.
|
||||||
|
|
||||||
|
**Note**: Please ensure the `xcode_version` in the
|
||||||
|
[`build_and_run.sh`](https://github.com/bazelbuild/tulsi/blob/b1d0108e6a93dbe8ab01529b2c607b6b651f0759/build_and_run.sh#L26)
|
||||||
|
file in tulsi repo is the same version as installed in your system.
|
||||||
|
|
||||||
2. Open `mediapipe/Mediapipe.tulsiproj` using the Tulsi app.
|
2. Open `mediapipe/Mediapipe.tulsiproj` using the Tulsi app.
|
||||||
|
|
||||||
Tip: If Tulsi displays an error saying "Bazel could not be found", press the
|
Tip: If Tulsi displays an error saying "Bazel could not be found", press the
|
||||||
|
|||||||
@@ -12,6 +12,17 @@ nav_order: 4
|
|||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||||
|
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||||
|
as the primary developer documentation
|
||||||
|
site for MediaPipe starting April 3, 2023. This content will not be moved to
|
||||||
|
the new site, but will remain available in the source code repository on an
|
||||||
|
as-is basis.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Ready-to-use JavaScript Solutions
|
## Ready-to-use JavaScript Solutions
|
||||||
|
|
||||||
MediaPipe currently offers the following solutions:
|
MediaPipe currently offers the following solutions:
|
||||||
@@ -29,11 +40,11 @@ Solution | NPM Package | Example
|
|||||||
Click on a solution link above for more information, including API and code
|
Click on a solution link above for more information, including API and code
|
||||||
snippets.
|
snippets.
|
||||||
|
|
||||||
### Supported plaforms:
|
### Supported platforms:
|
||||||
|
|
||||||
| Browser | Platform | Notes |
|
| Browser | Platform | Notes |
|
||||||
| ------- | ----------------------- | -------------------------------------- |
|
| ------- | ----------------------- | -------------------------------------- |
|
||||||
| Chrome | Android / Windows / Mac | Pixel 4 and older unsupported. Fuschia |
|
| Chrome | Android / Windows / Mac | Pixel 4 and older unsupported. Fuchsia |
|
||||||
| | | unsupported. |
|
| | | unsupported. |
|
||||||
| Chrome | iOS | Camera unavailable in Chrome on iOS. |
|
| Chrome | iOS | Camera unavailable in Chrome on iOS. |
|
||||||
| Safari | iPad/iPhone/Mac | iOS and Safari on iPad / iPhone / |
|
| Safari | iPad/iPhone/Mac | iOS and Safari on iPad / iPhone / |
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
title: MediaPipe Python Framework
|
target: https://developers.google.com/mediapipe/framework/getting_started/python_framework
|
||||||
parent: MediaPipe in Python
|
parent: MediaPipe in Python
|
||||||
grand_parent: Getting Started
|
grand_parent: Getting Started
|
||||||
nav_order: 1
|
nav_order: 1
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/getting_started/troubleshooting
|
||||||
title: Troubleshooting
|
title: Troubleshooting
|
||||||
parent: Getting Started
|
parent: Getting Started
|
||||||
nav_order: 10
|
nav_order: 10
|
||||||
@@ -65,7 +66,7 @@ WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/gith
|
|||||||
```
|
```
|
||||||
|
|
||||||
usually indicates that Bazel fails to download necessary dependency repositories
|
usually indicates that Bazel fails to download necessary dependency repositories
|
||||||
that MediaPipe needs. MedaiPipe has several dependency repositories that are
|
that MediaPipe needs. MediaPipe has several dependency repositories that are
|
||||||
hosted by Google sites. In some regions, you may need to set up a network proxy
|
hosted by Google sites. In some regions, you may need to set up a network proxy
|
||||||
or use a VPN to access those resources. You may also need to append
|
or use a VPN to access those resources. You may also need to append
|
||||||
`--host_jvm_args "-DsocksProxyHost=<ip address> -DsocksProxyPort=<port number>"`
|
`--host_jvm_args "-DsocksProxyHost=<ip address> -DsocksProxyPort=<port number>"`
|
||||||
|
|||||||
@@ -19,6 +19,17 @@ ML solutions for live and streaming media.
|
|||||||
 | 
|
 | 
|
||||||
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
|
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
|
**Attention:** *Thanks for your interest in MediaPipe! We are moving to
|
||||||
|
[https://developers.google.com/mediapipe](https://developers.google.com/mediapipe)
|
||||||
|
as the primary developer documentation
|
||||||
|
site for MediaPipe starting April 3, 2023.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## ML solutions in MediaPipe
|
## ML solutions in MediaPipe
|
||||||
|
|
||||||
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
|
Face Detection | Face Mesh | Iris | Hands | Pose | Holistic
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 14
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
AutoFlip is an automatic video cropping pipeline built on top of MediaPipe. This
|
AutoFlip is an automatic video cropping pipeline built on top of MediaPipe. This
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 10
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
MediaPipe Box Tracking has been powering real-time tracking in
|
MediaPipe Box Tracking has been powering real-time tracking in
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 1
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
MediaPipe Face Detection is an ultrafast face detection solution that comes with
|
MediaPipe Face Detection is an ultrafast face detection solution that comes with
|
||||||
@@ -54,6 +64,25 @@ used for its improved inference speed. Please refer to the
|
|||||||
[model cards](./models.md#face_detection) for details. Default to `0` if not
|
[model cards](./models.md#face_detection) for details. Default to `0` if not
|
||||||
specified.
|
specified.
|
||||||
|
|
||||||
|
Note: Not available for JavaScript (use "model" instead).
|
||||||
|
|
||||||
|
#### model
|
||||||
|
|
||||||
|
A string value to indicate which model should be used. Use "short" to
|
||||||
|
select a short-range model that works best for faces within 2 meters from the
|
||||||
|
camera, and "full" for a full-range model best for faces within 5 meters. For
|
||||||
|
the full-range option, a sparse model is used for its improved inference speed.
|
||||||
|
Please refer to the model cards for details. Default to empty string.
|
||||||
|
|
||||||
|
Note: Valid only for JavaScript solution.
|
||||||
|
|
||||||
|
#### selfie_mode
|
||||||
|
|
||||||
|
A boolean value to indicate whether to flip the images/video frames
|
||||||
|
horizontally or not. Default to `false`.
|
||||||
|
|
||||||
|
Note: Valid only for JavaScript solution.
|
||||||
|
|
||||||
#### min_detection_confidence
|
#### min_detection_confidence
|
||||||
|
|
||||||
Minimum confidence value (`[0.0, 1.0]`) from the face detection model for the
|
Minimum confidence value (`[0.0, 1.0]`) from the face detection model for the
|
||||||
@@ -146,9 +175,9 @@ Please first see general [introduction](../getting_started/javascript.md) on
|
|||||||
MediaPipe in JavaScript, then learn more in the companion [web demo](#resources)
|
MediaPipe in JavaScript, then learn more in the companion [web demo](#resources)
|
||||||
and the following usage example.
|
and the following usage example.
|
||||||
|
|
||||||
Supported configuration options:
|
Supported face detection options:
|
||||||
|
* [selfieMode](#selfie_mode)
|
||||||
* [modelSelection](#model_selection)
|
* [model](#model)
|
||||||
* [minDetectionConfidence](#min_detection_confidence)
|
* [minDetectionConfidence](#min_detection_confidence)
|
||||||
|
|
||||||
```html
|
```html
|
||||||
@@ -176,6 +205,7 @@ Supported configuration options:
|
|||||||
const videoElement = document.getElementsByClassName('input_video')[0];
|
const videoElement = document.getElementsByClassName('input_video')[0];
|
||||||
const canvasElement = document.getElementsByClassName('output_canvas')[0];
|
const canvasElement = document.getElementsByClassName('output_canvas')[0];
|
||||||
const canvasCtx = canvasElement.getContext('2d');
|
const canvasCtx = canvasElement.getContext('2d');
|
||||||
|
const drawingUtils = window;
|
||||||
|
|
||||||
function onResults(results) {
|
function onResults(results) {
|
||||||
// Draw the overlays.
|
// Draw the overlays.
|
||||||
@@ -199,7 +229,7 @@ const faceDetection = new FaceDetection({locateFile: (file) => {
|
|||||||
return `https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/${file}`;
|
return `https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/${file}`;
|
||||||
}});
|
}});
|
||||||
faceDetection.setOptions({
|
faceDetection.setOptions({
|
||||||
modelSelection: 0,
|
model: 'short',
|
||||||
minDetectionConfidence: 0.5
|
minDetectionConfidence: 0.5
|
||||||
});
|
});
|
||||||
faceDetection.onResults(onResults);
|
faceDetection.onResults(onResults);
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 2
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
MediaPipe Face Mesh is a solution that estimates 468 3D face landmarks in
|
MediaPipe Face Mesh is a solution that estimates 468 3D face landmarks in
|
||||||
@@ -133,7 +143,7 @@ about the model in this [paper](https://arxiv.org/abs/2006.10962).
|
|||||||
The [Face Landmark Model](#face-landmark-model) performs a single-camera face landmark
|
The [Face Landmark Model](#face-landmark-model) performs a single-camera face landmark
|
||||||
detection in the screen coordinate space: the X- and Y- coordinates are
|
detection in the screen coordinate space: the X- and Y- coordinates are
|
||||||
normalized screen coordinates, while the Z coordinate is relative and is scaled
|
normalized screen coordinates, while the Z coordinate is relative and is scaled
|
||||||
as the X coodinate under the
|
as the X coordinate under the
|
||||||
[weak perspective projection camera model](https://en.wikipedia.org/wiki/3D_projection#Weak_perspective_projection).
|
[weak perspective projection camera model](https://en.wikipedia.org/wiki/3D_projection#Weak_perspective_projection).
|
||||||
This format is well-suited for some applications, however it does not directly
|
This format is well-suited for some applications, however it does not directly
|
||||||
enable the full spectrum of augmented reality (AR) features like aligning a
|
enable the full spectrum of augmented reality (AR) features like aligning a
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 8
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
## Example Apps
|
## Example Apps
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 4
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
The ability to perceive the shape and motion of hands can be a vital component
|
The ability to perceive the shape and motion of hands can be a vital component
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 6
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Live perception of simultaneous [human pose](./pose.md),
|
Live perception of simultaneous [human pose](./pose.md),
|
||||||
@@ -259,6 +269,7 @@ mp_holistic = mp.solutions.holistic
|
|||||||
|
|
||||||
# For static images:
|
# For static images:
|
||||||
IMAGE_FILES = []
|
IMAGE_FILES = []
|
||||||
|
BG_COLOR = (192, 192, 192) # gray
|
||||||
with mp_holistic.Holistic(
|
with mp_holistic.Holistic(
|
||||||
static_image_mode=True,
|
static_image_mode=True,
|
||||||
model_complexity=2,
|
model_complexity=2,
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 11
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Augmented Reality (AR) technology creates fun, engaging, and immersive user
|
Augmented Reality (AR) technology creates fun, engaging, and immersive user
|
||||||
|
|||||||
+12
-2
@@ -18,6 +18,16 @@ nav_order: 3
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
A wide range of real-world applications, including computational photography
|
A wide range of real-world applications, including computational photography
|
||||||
@@ -38,7 +48,7 @@ camera, in real-time, without the need for specialized hardware. Through use of
|
|||||||
iris landmarks, the solution is also able to determine the metric distance
|
iris landmarks, the solution is also able to determine the metric distance
|
||||||
between the subject and the camera with relative error less than 10%. Note that
|
between the subject and the camera with relative error less than 10%. Note that
|
||||||
iris tracking does not infer the location at which people are looking, nor does
|
iris tracking does not infer the location at which people are looking, nor does
|
||||||
it provide any form of identity recognition. With the cross-platfrom capability
|
it provide any form of identity recognition. With the cross-platform capability
|
||||||
of the MediaPipe framework, MediaPipe Iris can run on most modern
|
of the MediaPipe framework, MediaPipe Iris can run on most modern
|
||||||
[mobile phones](#mobile), [desktops/laptops](#desktop) and even on the
|
[mobile phones](#mobile), [desktops/laptops](#desktop) and even on the
|
||||||
[web](#web).
|
[web](#web).
|
||||||
@@ -99,7 +109,7 @@ You can also find more details in this
|
|||||||
### Iris Landmark Model
|
### Iris Landmark Model
|
||||||
|
|
||||||
The iris model takes an image patch of the eye region and estimates both the eye
|
The iris model takes an image patch of the eye region and estimates both the eye
|
||||||
landmarks (along the eyelid) and iris landmarks (along ths iris contour). You
|
landmarks (along the eyelid) and iris landmarks (along this iris contour). You
|
||||||
can find more details in this [paper](https://arxiv.org/abs/2006.11341).
|
can find more details in this [paper](https://arxiv.org/abs/2006.11341).
|
||||||
|
|
||||||
 |
|
 |
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 13
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
MediaPipe KNIFT is a template-based feature matching solution using KNIFT
|
MediaPipe KNIFT is a template-based feature matching solution using KNIFT
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 15
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
MediaPipe is a useful and general framework for media processing that can
|
MediaPipe is a useful and general framework for media processing that can
|
||||||
@@ -85,7 +95,7 @@ process new data sets, in the documentation of
|
|||||||
|
|
||||||
MediaSequence uses SequenceExamples as the format of both inputs and
|
MediaSequence uses SequenceExamples as the format of both inputs and
|
||||||
outputs. Annotations are encoded as inputs in a SequenceExample of metadata
|
outputs. Annotations are encoded as inputs in a SequenceExample of metadata
|
||||||
that defines the labels and the path to the cooresponding video file. This
|
that defines the labels and the path to the corresponding video file. This
|
||||||
metadata is passed as input to the C++ `media_sequence_demo` binary, and the
|
metadata is passed as input to the C++ `media_sequence_demo` binary, and the
|
||||||
output is a SequenceExample filled with images and annotations ready for
|
output is a SequenceExample filled with images and annotations ready for
|
||||||
model training.
|
model training.
|
||||||
|
|||||||
@@ -12,6 +12,20 @@ nav_order: 30
|
|||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for
|
||||||
|
[these MediaPipe Legacy Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
as of March 1, 2023. All other
|
||||||
|
[MediaPipe Legacy Solutions will be upgraded](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
to a new MediaPipe Solution. The code repository and prebuilt binaries for all
|
||||||
|
MediaPipe Legacy Solutions will continue to be provided on an as-is basis.
|
||||||
|
We encourage you to check out the new MediaPipe Solutions at:
|
||||||
|
[https://developers.google.com/mediapipe/solutions](https://developers.google.com/mediapipe/solutions)*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
||||||
|
|
||||||
* Short-range model (best for faces within 2 meters from the camera):
|
* Short-range model (best for faces within 2 meters from the camera):
|
||||||
@@ -94,8 +108,6 @@ one over the other.
|
|||||||
|
|
||||||
* [TFLite model](https://storage.googleapis.com/mediapipe-assets/ssdlite_object_detection.tflite)
|
* [TFLite model](https://storage.googleapis.com/mediapipe-assets/ssdlite_object_detection.tflite)
|
||||||
* [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/object-detector-quantized_edgetpu.tflite)
|
* [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/object-detector-quantized_edgetpu.tflite)
|
||||||
* [TensorFlow model](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model)
|
|
||||||
* [Model information](https://github.com/google/mediapipe/tree/master/mediapipe/models/object_detection_saved_model/README.md)
|
|
||||||
|
|
||||||
### [Objectron](https://google.github.io/mediapipe/solutions/objectron)
|
### [Objectron](https://google.github.io/mediapipe/solutions/objectron)
|
||||||
|
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 9
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
## Example Apps
|
## Example Apps
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 12
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
MediaPipe Objectron is a mobile real-time 3D object detection solution for
|
MediaPipe Objectron is a mobile real-time 3D object detection solution for
|
||||||
@@ -170,7 +180,7 @@ and a
|
|||||||
The detection subgraph performs ML inference only once every few frames to
|
The detection subgraph performs ML inference only once every few frames to
|
||||||
reduce computation load, and decodes the output tensor to a FrameAnnotation that
|
reduce computation load, and decodes the output tensor to a FrameAnnotation that
|
||||||
contains nine keypoints: the 3D bounding box's center and its eight vertices.
|
contains nine keypoints: the 3D bounding box's center and its eight vertices.
|
||||||
The tracking subgraph runs every frame, using the box traker in
|
The tracking subgraph runs every frame, using the box tracker in
|
||||||
[MediaPipe Box Tracking](./box_tracking.md) to track the 2D box tightly
|
[MediaPipe Box Tracking](./box_tracking.md) to track the 2D box tightly
|
||||||
enclosing the projection of the 3D bounding box, and lifts the tracked 2D
|
enclosing the projection of the 3D bounding box, and lifts the tracked 2D
|
||||||
keypoints to 3D with
|
keypoints to 3D with
|
||||||
@@ -613,7 +623,7 @@ z_ndc = 1 / Z
|
|||||||
|
|
||||||
### Pixel Space
|
### Pixel Space
|
||||||
|
|
||||||
In this API we set upper-left coner of an image as the origin of pixel
|
In this API we set upper-left corner of an image as the origin of pixel
|
||||||
coordinate. One can convert from NDC to pixel space as follows:
|
coordinate. One can convert from NDC to pixel space as follows:
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -20,6 +20,16 @@ nav_order: 5
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
Human pose estimation from video plays a critical role in various applications
|
Human pose estimation from video plays a critical role in various applications
|
||||||
|
|||||||
@@ -19,6 +19,16 @@ nav_order: 1
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
One of the applications
|
One of the applications
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 7
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
As of March 1, 2023, this solution is planned to be upgraded to a new MediaPipe
|
||||||
|
Solution. For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
*Fig 1. Example of MediaPipe Selfie Segmentation.* |
|
*Fig 1. Example of MediaPipe Selfie Segmentation.* |
|
||||||
|
|||||||
@@ -13,6 +13,22 @@ has_toc: false
|
|||||||
{:toc}
|
{:toc}
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions. We have
|
||||||
|
ended support for
|
||||||
|
[these MediaPipe Legacy Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
as of March 1, 2023. All other
|
||||||
|
[MediaPipe Legacy Solutions will be upgraded](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
to a new MediaPipe Solution. The
|
||||||
|
[code repository](https://github.com/google/mediapipe/tree/master/mediapipe)
|
||||||
|
and prebuilt binaries for all MediaPipe Legacy Solutions will continue to
|
||||||
|
be provided on an as-is basis. We encourage you to check out the new MediaPipe
|
||||||
|
Solutions at:
|
||||||
|
[https://developers.google.com/mediapipe/solutions](https://developers.google.com/mediapipe/solutions)*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on June 1, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
MediaPipe offers open source cross-platform, customizable ML solutions for live
|
MediaPipe offers open source cross-platform, customizable ML solutions for live
|
||||||
and streaming media.
|
and streaming media.
|
||||||
|
|
||||||
|
|||||||
@@ -18,6 +18,16 @@ nav_order: 16
|
|||||||
</details>
|
</details>
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**Attention:** *Thank you for your interest in MediaPipe Solutions.
|
||||||
|
We have ended support for this MediaPipe Legacy Solution as of March 1, 2023.
|
||||||
|
For more information, see the new
|
||||||
|
[MediaPipe Solutions](https://developers.google.com/mediapipe/solutions/guide#legacy)
|
||||||
|
site.*
|
||||||
|
|
||||||
|
*This notice and web page will be removed on April 3, 2023.*
|
||||||
|
|
||||||
|
----
|
||||||
|
|
||||||
MediaPipe is a useful and general framework for media processing that can assist
|
MediaPipe is a useful and general framework for media processing that can assist
|
||||||
with research, development, and deployment of ML models. This example focuses on
|
with research, development, and deployment of ML models. This example focuses on
|
||||||
model development by demonstrating how to prepare training data and do model
|
model development by demonstrating how to prepare training data and do model
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
---
|
---
|
||||||
layout: default
|
layout: forward
|
||||||
|
target: https://developers.google.com/mediapipe/framework/tools/visualizer
|
||||||
title: Visualizer
|
title: Visualizer
|
||||||
parent: Tools
|
parent: Tools
|
||||||
nav_order: 1
|
nav_order: 1
|
||||||
|
|||||||
@@ -12,12 +12,12 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
|
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||||
|
|
||||||
licenses(["notice"])
|
licenses(["notice"])
|
||||||
|
|
||||||
package(default_visibility = ["//visibility:private"])
|
package(default_visibility = ["//visibility:private"])
|
||||||
|
|
||||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
|
||||||
|
|
||||||
proto_library(
|
proto_library(
|
||||||
name = "mfcc_mel_calculators_proto",
|
name = "mfcc_mel_calculators_proto",
|
||||||
srcs = ["mfcc_mel_calculators.proto"],
|
srcs = ["mfcc_mel_calculators.proto"],
|
||||||
|
|||||||
@@ -280,6 +280,13 @@ absl::Status SpectrogramCalculator::Open(CalculatorContext* cc) {
|
|||||||
audio_dsp::HammingWindow().GetPeriodicSamples(frame_duration_samples_,
|
audio_dsp::HammingWindow().GetPeriodicSamples(frame_duration_samples_,
|
||||||
&window);
|
&window);
|
||||||
break;
|
break;
|
||||||
|
case SpectrogramCalculatorOptions::SQRT_HANN: {
|
||||||
|
audio_dsp::HannWindow().GetPeriodicSamples(frame_duration_samples_,
|
||||||
|
&window);
|
||||||
|
absl::c_transform(window, window.begin(),
|
||||||
|
[](double x) { return std::sqrt(x); });
|
||||||
|
break;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Propagate settings down to the actual Spectrogram object.
|
// Propagate settings down to the actual Spectrogram object.
|
||||||
|
|||||||
@@ -68,6 +68,7 @@ message SpectrogramCalculatorOptions {
|
|||||||
HANN = 0;
|
HANN = 0;
|
||||||
HAMMING = 1;
|
HAMMING = 1;
|
||||||
COSINE = 2;
|
COSINE = 2;
|
||||||
|
SQRT_HANN = 4;
|
||||||
}
|
}
|
||||||
optional WindowType window_type = 6 [default = HANN];
|
optional WindowType window_type = 6 [default = HANN];
|
||||||
|
|
||||||
|
|||||||
@@ -13,12 +13,21 @@
|
|||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
#
|
#
|
||||||
|
|
||||||
|
load("@bazel_skylib//lib:selects.bzl", "selects")
|
||||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_proto_library")
|
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_proto_library")
|
||||||
|
|
||||||
licenses(["notice"])
|
licenses(["notice"])
|
||||||
|
|
||||||
package(default_visibility = ["//visibility:public"])
|
package(default_visibility = ["//visibility:public"])
|
||||||
|
|
||||||
|
selects.config_setting_group(
|
||||||
|
name = "ios_or_disable_gpu",
|
||||||
|
match_any = [
|
||||||
|
"//mediapipe/gpu:disable_gpu",
|
||||||
|
"//mediapipe:ios",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "concatenate_vector_calculator_proto",
|
name = "concatenate_vector_calculator_proto",
|
||||||
srcs = ["concatenate_vector_calculator.proto"],
|
srcs = ["concatenate_vector_calculator.proto"],
|
||||||
@@ -108,6 +117,7 @@ mediapipe_proto_library(
|
|||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
"//mediapipe/framework/formats:classification_proto",
|
"//mediapipe/framework/formats:classification_proto",
|
||||||
"//mediapipe/framework/formats:landmark_proto",
|
"//mediapipe/framework/formats:landmark_proto",
|
||||||
|
"//mediapipe/framework/formats:time_series_header_proto",
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -188,6 +198,7 @@ cc_library(
|
|||||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||||
"//mediapipe/framework/formats:matrix",
|
"//mediapipe/framework/formats:matrix",
|
||||||
"//mediapipe/framework/formats:rect_cc_proto",
|
"//mediapipe/framework/formats:rect_cc_proto",
|
||||||
|
"//mediapipe/framework/formats:tensor",
|
||||||
"//mediapipe/framework/port:integral_types",
|
"//mediapipe/framework/port:integral_types",
|
||||||
"//mediapipe/framework/port:ret_check",
|
"//mediapipe/framework/port:ret_check",
|
||||||
"//mediapipe/framework/port:status",
|
"//mediapipe/framework/port:status",
|
||||||
@@ -210,6 +221,7 @@ cc_library(
|
|||||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||||
"//mediapipe/framework/formats:matrix",
|
"//mediapipe/framework/formats:matrix",
|
||||||
"//mediapipe/framework/formats:rect_cc_proto",
|
"//mediapipe/framework/formats:rect_cc_proto",
|
||||||
|
"//mediapipe/framework/formats:tensor",
|
||||||
"//mediapipe/framework/port:integral_types",
|
"//mediapipe/framework/port:integral_types",
|
||||||
"//mediapipe/framework/port:ret_check",
|
"//mediapipe/framework/port:ret_check",
|
||||||
"//mediapipe/framework/port:status",
|
"//mediapipe/framework/port:status",
|
||||||
@@ -226,10 +238,12 @@ cc_test(
|
|||||||
":begin_loop_calculator",
|
":begin_loop_calculator",
|
||||||
":end_loop_calculator",
|
":end_loop_calculator",
|
||||||
":gate_calculator",
|
":gate_calculator",
|
||||||
|
":pass_through_calculator",
|
||||||
"//mediapipe/framework:calculator_context",
|
"//mediapipe/framework:calculator_context",
|
||||||
"//mediapipe/framework:calculator_contract",
|
"//mediapipe/framework:calculator_contract",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework:packet",
|
"//mediapipe/framework:packet",
|
||||||
|
"//mediapipe/framework/formats:tensor",
|
||||||
"//mediapipe/framework/port:gtest_main",
|
"//mediapipe/framework/port:gtest_main",
|
||||||
"//mediapipe/framework/port:integral_types",
|
"//mediapipe/framework/port:integral_types",
|
||||||
"//mediapipe/framework/port:parse_text_proto",
|
"//mediapipe/framework/port:parse_text_proto",
|
||||||
@@ -567,7 +581,7 @@ cc_library(
|
|||||||
name = "packet_thinner_calculator",
|
name = "packet_thinner_calculator",
|
||||||
srcs = ["packet_thinner_calculator.cc"],
|
srcs = ["packet_thinner_calculator.cc"],
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/calculators/core:packet_thinner_calculator_cc_proto",
|
":packet_thinner_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_context",
|
"//mediapipe/framework:calculator_context",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework/formats:video_stream_header",
|
"//mediapipe/framework/formats:video_stream_header",
|
||||||
@@ -584,7 +598,7 @@ cc_test(
|
|||||||
srcs = ["packet_thinner_calculator_test.cc"],
|
srcs = ["packet_thinner_calculator_test.cc"],
|
||||||
deps = [
|
deps = [
|
||||||
":packet_thinner_calculator",
|
":packet_thinner_calculator",
|
||||||
"//mediapipe/calculators/core:packet_thinner_calculator_cc_proto",
|
":packet_thinner_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework:calculator_runner",
|
"//mediapipe/framework:calculator_runner",
|
||||||
"//mediapipe/framework/formats:video_stream_header",
|
"//mediapipe/framework/formats:video_stream_header",
|
||||||
@@ -762,7 +776,7 @@ cc_library(
|
|||||||
srcs = ["packet_resampler_calculator.cc"],
|
srcs = ["packet_resampler_calculator.cc"],
|
||||||
hdrs = ["packet_resampler_calculator.h"],
|
hdrs = ["packet_resampler_calculator.h"],
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/calculators/core:packet_resampler_calculator_cc_proto",
|
":packet_resampler_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework:collection_item_id",
|
"//mediapipe/framework:collection_item_id",
|
||||||
"//mediapipe/framework/deps:mathutil",
|
"//mediapipe/framework/deps:mathutil",
|
||||||
@@ -786,7 +800,7 @@ cc_test(
|
|||||||
],
|
],
|
||||||
deps = [
|
deps = [
|
||||||
":packet_resampler_calculator",
|
":packet_resampler_calculator",
|
||||||
"//mediapipe/calculators/core:packet_resampler_calculator_cc_proto",
|
":packet_resampler_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework:calculator_runner",
|
"//mediapipe/framework:calculator_runner",
|
||||||
"//mediapipe/framework/formats:video_stream_header",
|
"//mediapipe/framework/formats:video_stream_header",
|
||||||
@@ -852,10 +866,10 @@ cc_test(
|
|||||||
name = "flow_limiter_calculator_test",
|
name = "flow_limiter_calculator_test",
|
||||||
srcs = ["flow_limiter_calculator_test.cc"],
|
srcs = ["flow_limiter_calculator_test.cc"],
|
||||||
deps = [
|
deps = [
|
||||||
|
":counting_source_calculator",
|
||||||
":flow_limiter_calculator",
|
":flow_limiter_calculator",
|
||||||
":flow_limiter_calculator_cc_proto",
|
":flow_limiter_calculator_cc_proto",
|
||||||
"//mediapipe/calculators/core:counting_source_calculator",
|
":pass_through_calculator",
|
||||||
"//mediapipe/calculators/core:pass_through_calculator",
|
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework:calculator_runner",
|
"//mediapipe/framework:calculator_runner",
|
||||||
"//mediapipe/framework:test_calculators",
|
"//mediapipe/framework:test_calculators",
|
||||||
@@ -899,8 +913,7 @@ cc_library(
|
|||||||
"@org_tensorflow//tensorflow/lite:framework",
|
"@org_tensorflow//tensorflow/lite:framework",
|
||||||
"@org_tensorflow//tensorflow/lite/kernels:builtin_ops",
|
"@org_tensorflow//tensorflow/lite/kernels:builtin_ops",
|
||||||
] + select({
|
] + select({
|
||||||
"//mediapipe/gpu:disable_gpu": [],
|
":ios_or_disable_gpu": [],
|
||||||
"//mediapipe:ios": [],
|
|
||||||
"//conditions:default": [
|
"//conditions:default": [
|
||||||
"@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_buffer",
|
"@org_tensorflow//tensorflow/lite/delegates/gpu/gl:gl_buffer",
|
||||||
],
|
],
|
||||||
@@ -1150,6 +1163,7 @@ cc_library(
|
|||||||
"//mediapipe/framework:collection_item_id",
|
"//mediapipe/framework:collection_item_id",
|
||||||
"//mediapipe/framework/formats:classification_cc_proto",
|
"//mediapipe/framework/formats:classification_cc_proto",
|
||||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||||
|
"//mediapipe/framework/formats:time_series_header_cc_proto",
|
||||||
"//mediapipe/framework/port:integral_types",
|
"//mediapipe/framework/port:integral_types",
|
||||||
"//mediapipe/framework/port:ret_check",
|
"//mediapipe/framework/port:ret_check",
|
||||||
"//mediapipe/framework/port:status",
|
"//mediapipe/framework/port:status",
|
||||||
@@ -1302,7 +1316,7 @@ cc_test(
|
|||||||
srcs = ["packet_sequencer_calculator_test.cc"],
|
srcs = ["packet_sequencer_calculator_test.cc"],
|
||||||
deps = [
|
deps = [
|
||||||
":packet_sequencer_calculator",
|
":packet_sequencer_calculator",
|
||||||
"//mediapipe/calculators/core:pass_through_calculator",
|
":pass_through_calculator",
|
||||||
"//mediapipe/framework:calculator_cc_proto",
|
"//mediapipe/framework:calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework:subgraph",
|
"//mediapipe/framework:subgraph",
|
||||||
@@ -1321,8 +1335,10 @@ cc_library(
|
|||||||
hdrs = ["merge_to_vector_calculator.h"],
|
hdrs = ["merge_to_vector_calculator.h"],
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
|
"//mediapipe/framework:packet",
|
||||||
"//mediapipe/framework/api2:node",
|
"//mediapipe/framework/api2:node",
|
||||||
"//mediapipe/framework/api2:port",
|
"//mediapipe/framework/api2:port",
|
||||||
|
"//mediapipe/framework/formats:detection_cc_proto",
|
||||||
"//mediapipe/framework/formats:image",
|
"//mediapipe/framework/formats:image",
|
||||||
"@com_google_absl//absl/status",
|
"@com_google_absl//absl/status",
|
||||||
],
|
],
|
||||||
|
|||||||
@@ -20,6 +20,7 @@
|
|||||||
#include "mediapipe/calculators/core/end_loop_calculator.h"
|
#include "mediapipe/calculators/core/end_loop_calculator.h"
|
||||||
#include "mediapipe/framework/calculator_contract.h"
|
#include "mediapipe/framework/calculator_contract.h"
|
||||||
#include "mediapipe/framework/calculator_framework.h"
|
#include "mediapipe/framework/calculator_framework.h"
|
||||||
|
#include "mediapipe/framework/formats/tensor.h"
|
||||||
#include "mediapipe/framework/packet.h"
|
#include "mediapipe/framework/packet.h"
|
||||||
#include "mediapipe/framework/port/gmock.h"
|
#include "mediapipe/framework/port/gmock.h"
|
||||||
#include "mediapipe/framework/port/gtest.h"
|
#include "mediapipe/framework/port/gtest.h"
|
||||||
@@ -444,5 +445,67 @@ TEST_F(BeginEndLoopCalculatorGraphWithClonedInputsTest, MultipleVectors) {
|
|||||||
PacketOfIntsEq(input_timestamp2, std::vector<int>{6, 9})));
|
PacketOfIntsEq(input_timestamp2, std::vector<int>{6, 9})));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
absl::Status InitBeginEndTensorLoopTestGraph(
|
||||||
|
CalculatorGraph& graph, std::vector<Packet>& output_packets) {
|
||||||
|
auto graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(
|
||||||
|
R"pb(
|
||||||
|
num_threads: 4
|
||||||
|
input_stream: "tensors"
|
||||||
|
node {
|
||||||
|
calculator: "BeginLoopTensorCalculator"
|
||||||
|
input_stream: "ITERABLE:tensors"
|
||||||
|
output_stream: "ITEM:tensor"
|
||||||
|
output_stream: "BATCH_END:timestamp"
|
||||||
|
}
|
||||||
|
node {
|
||||||
|
calculator: "PassThroughCalculator"
|
||||||
|
input_stream: "tensor"
|
||||||
|
output_stream: "passed_tensor"
|
||||||
|
}
|
||||||
|
node {
|
||||||
|
calculator: "EndLoopTensorCalculator"
|
||||||
|
input_stream: "ITEM:passed_tensor"
|
||||||
|
input_stream: "BATCH_END:timestamp"
|
||||||
|
output_stream: "ITERABLE:output_tensors"
|
||||||
|
}
|
||||||
|
)pb");
|
||||||
|
tool::AddVectorSink("output_tensors", &graph_config, &output_packets);
|
||||||
|
MP_RETURN_IF_ERROR(graph.Initialize(graph_config));
|
||||||
|
return graph.StartRun({});
|
||||||
|
}
|
||||||
|
|
||||||
|
TEST(BeginEndTensorLoopCalculatorGraphTest, SingleNonEmptyVector) {
|
||||||
|
// Initialize the graph.
|
||||||
|
CalculatorGraph graph;
|
||||||
|
std::vector<Packet> output_packets;
|
||||||
|
MP_ASSERT_OK(InitBeginEndTensorLoopTestGraph(graph, output_packets));
|
||||||
|
|
||||||
|
// Prepare the inputs and run.
|
||||||
|
Timestamp input_timestamp = Timestamp(0);
|
||||||
|
std::vector<mediapipe::Tensor> tensors;
|
||||||
|
for (int i = 0; i < 4; i++) {
|
||||||
|
tensors.emplace_back(Tensor::ElementType::kFloat32,
|
||||||
|
Tensor::Shape{4, 3, 2, 1});
|
||||||
|
}
|
||||||
|
Packet vector_packet =
|
||||||
|
MakePacket<std::vector<mediapipe::Tensor>>(std::move(tensors));
|
||||||
|
MP_ASSERT_OK(graph.AddPacketToInputStream(
|
||||||
|
"tensors", std::move(vector_packet).At(input_timestamp)));
|
||||||
|
MP_ASSERT_OK(graph.WaitUntilIdle());
|
||||||
|
|
||||||
|
// Verify the output packet.
|
||||||
|
EXPECT_EQ(output_packets.size(), 1);
|
||||||
|
const std::vector<Tensor>& output_tensors =
|
||||||
|
output_packets[0].Get<std::vector<Tensor>>();
|
||||||
|
EXPECT_EQ(output_tensors.size(), 4);
|
||||||
|
for (int i = 0; i < output_tensors.size(); i++) {
|
||||||
|
EXPECT_THAT(output_tensors[i].shape().dims,
|
||||||
|
testing::ElementsAre(4, 3, 2, 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
MP_ASSERT_OK(graph.CloseAllPacketSources());
|
||||||
|
MP_ASSERT_OK(graph.WaitUntilDone());
|
||||||
|
}
|
||||||
|
|
||||||
} // namespace
|
} // namespace
|
||||||
} // namespace mediapipe
|
} // namespace mediapipe
|
||||||
|
|||||||
@@ -20,6 +20,7 @@
|
|||||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||||
#include "mediapipe/framework/formats/matrix.h"
|
#include "mediapipe/framework/formats/matrix.h"
|
||||||
#include "mediapipe/framework/formats/rect.pb.h"
|
#include "mediapipe/framework/formats/rect.pb.h"
|
||||||
|
#include "mediapipe/framework/formats/tensor.h"
|
||||||
|
|
||||||
namespace mediapipe {
|
namespace mediapipe {
|
||||||
|
|
||||||
@@ -55,4 +56,8 @@ REGISTER_CALCULATOR(BeginLoopMatrixVectorCalculator);
|
|||||||
typedef BeginLoopCalculator<std::vector<uint64_t>> BeginLoopUint64tCalculator;
|
typedef BeginLoopCalculator<std::vector<uint64_t>> BeginLoopUint64tCalculator;
|
||||||
REGISTER_CALCULATOR(BeginLoopUint64tCalculator);
|
REGISTER_CALCULATOR(BeginLoopUint64tCalculator);
|
||||||
|
|
||||||
|
// A calculator to process std::vector<mediapipe::Tensor>.
|
||||||
|
typedef BeginLoopCalculator<std::vector<Tensor>> BeginLoopTensorCalculator;
|
||||||
|
REGISTER_CALCULATOR(BeginLoopTensorCalculator);
|
||||||
|
|
||||||
} // namespace mediapipe
|
} // namespace mediapipe
|
||||||
|
|||||||
@@ -15,7 +15,6 @@
|
|||||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_BEGIN_LOOP_CALCULATOR_H_
|
#ifndef MEDIAPIPE_CALCULATORS_CORE_BEGIN_LOOP_CALCULATOR_H_
|
||||||
#define MEDIAPIPE_CALCULATORS_CORE_BEGIN_LOOP_CALCULATOR_H_
|
#define MEDIAPIPE_CALCULATORS_CORE_BEGIN_LOOP_CALCULATOR_H_
|
||||||
|
|
||||||
#include "absl/memory/memory.h"
|
|
||||||
#include "mediapipe/framework/calculator_context.h"
|
#include "mediapipe/framework/calculator_context.h"
|
||||||
#include "mediapipe/framework/calculator_contract.h"
|
#include "mediapipe/framework/calculator_contract.h"
|
||||||
#include "mediapipe/framework/calculator_framework.h"
|
#include "mediapipe/framework/calculator_framework.h"
|
||||||
@@ -24,6 +23,7 @@
|
|||||||
#include "mediapipe/framework/port/integral_types.h"
|
#include "mediapipe/framework/port/integral_types.h"
|
||||||
#include "mediapipe/framework/port/ret_check.h"
|
#include "mediapipe/framework/port/ret_check.h"
|
||||||
#include "mediapipe/framework/port/status.h"
|
#include "mediapipe/framework/port/status.h"
|
||||||
|
#include "mediapipe/framework/port/status_macros.h"
|
||||||
|
|
||||||
namespace mediapipe {
|
namespace mediapipe {
|
||||||
|
|
||||||
@@ -49,7 +49,7 @@ namespace mediapipe {
|
|||||||
// calculator: "EndLoopWithOutputCalculator"
|
// calculator: "EndLoopWithOutputCalculator"
|
||||||
// input_stream: "ITEM:output_of_loop_body" # ItemU @loop_internal_ts
|
// input_stream: "ITEM:output_of_loop_body" # ItemU @loop_internal_ts
|
||||||
// input_stream: "BATCH_END:ext_ts" # Timestamp @loop_internal_ts
|
// input_stream: "BATCH_END:ext_ts" # Timestamp @loop_internal_ts
|
||||||
// output_stream: "OUTPUT:aggregated_result" # IterableU @ext_ts
|
// output_stream: "ITERABLE:aggregated_result" # IterableU @ext_ts
|
||||||
// }
|
// }
|
||||||
//
|
//
|
||||||
// Input streams tagged with "CLONE" are cloned to the corresponding output
|
// Input streams tagged with "CLONE" are cloned to the corresponding output
|
||||||
@@ -112,13 +112,38 @@ class BeginLoopCalculator : public CalculatorBase {
|
|||||||
absl::Status Process(CalculatorContext* cc) final {
|
absl::Status Process(CalculatorContext* cc) final {
|
||||||
Timestamp last_timestamp = loop_internal_timestamp_;
|
Timestamp last_timestamp = loop_internal_timestamp_;
|
||||||
if (!cc->Inputs().Tag("ITERABLE").IsEmpty()) {
|
if (!cc->Inputs().Tag("ITERABLE").IsEmpty()) {
|
||||||
const IterableT& collection =
|
// Try to consume the ITERABLE packet if possible to obtain the ownership
|
||||||
cc->Inputs().Tag("ITERABLE").template Get<IterableT>();
|
// and emit the item packets by moving them.
|
||||||
for (const auto& item : collection) {
|
// If the ITERABLE packet is not consumable, then try to copy each item
|
||||||
cc->Outputs().Tag("ITEM").AddPacket(
|
// instead. If the ITEM type is not copy constructible, an error will be
|
||||||
MakePacket<ItemT>(item).At(loop_internal_timestamp_));
|
// returned.
|
||||||
ForwardClonePackets(cc, loop_internal_timestamp_);
|
auto iterable_ptr_or =
|
||||||
++loop_internal_timestamp_;
|
cc->Inputs().Tag("ITERABLE").Value().Consume<IterableT>();
|
||||||
|
if (iterable_ptr_or.ok()) {
|
||||||
|
for (auto& item : *iterable_ptr_or.value()) {
|
||||||
|
Packet item_packet = MakePacket<ItemT>(std::move(item));
|
||||||
|
cc->Outputs().Tag("ITEM").AddPacket(
|
||||||
|
item_packet.At(loop_internal_timestamp_));
|
||||||
|
ForwardClonePackets(cc, loop_internal_timestamp_);
|
||||||
|
++loop_internal_timestamp_;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if constexpr (std::is_copy_constructible<ItemT>()) {
|
||||||
|
const IterableT& collection =
|
||||||
|
cc->Inputs().Tag("ITERABLE").template Get<IterableT>();
|
||||||
|
for (const auto& item : collection) {
|
||||||
|
cc->Outputs().Tag("ITEM").AddPacket(
|
||||||
|
MakePacket<ItemT>(item).At(loop_internal_timestamp_));
|
||||||
|
ForwardClonePackets(cc, loop_internal_timestamp_);
|
||||||
|
++loop_internal_timestamp_;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
return absl::InternalError(
|
||||||
|
"The element type is not copiable. Consider making the "
|
||||||
|
"BeginLoopCalculator the sole owner of the input packet so that "
|
||||||
|
"the "
|
||||||
|
"items can be consumed and moved.");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -138,7 +163,6 @@ class BeginLoopCalculator : public CalculatorBase {
|
|||||||
.Tag("BATCH_END")
|
.Tag("BATCH_END")
|
||||||
.AddPacket(MakePacket<Timestamp>(cc->InputTimestamp())
|
.AddPacket(MakePacket<Timestamp>(cc->InputTimestamp())
|
||||||
.At(Timestamp(loop_internal_timestamp_ - 1)));
|
.At(Timestamp(loop_internal_timestamp_ - 1)));
|
||||||
|
|
||||||
return absl::OkStatus();
|
return absl::OkStatus();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -19,6 +19,7 @@
|
|||||||
#include "mediapipe/framework/collection_item_id.h"
|
#include "mediapipe/framework/collection_item_id.h"
|
||||||
#include "mediapipe/framework/formats/classification.pb.h"
|
#include "mediapipe/framework/formats/classification.pb.h"
|
||||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||||
|
#include "mediapipe/framework/formats/time_series_header.pb.h"
|
||||||
#include "mediapipe/framework/port/canonical_errors.h"
|
#include "mediapipe/framework/port/canonical_errors.h"
|
||||||
#include "mediapipe/framework/port/integral_types.h"
|
#include "mediapipe/framework/port/integral_types.h"
|
||||||
#include "mediapipe/framework/port/ret_check.h"
|
#include "mediapipe/framework/port/ret_check.h"
|
||||||
@@ -84,6 +85,8 @@ class ConstantSidePacketCalculator : public CalculatorBase {
|
|||||||
packet.Set<LandmarkList>();
|
packet.Set<LandmarkList>();
|
||||||
} else if (packet_options.has_double_value()) {
|
} else if (packet_options.has_double_value()) {
|
||||||
packet.Set<double>();
|
packet.Set<double>();
|
||||||
|
} else if (packet_options.has_time_series_header_value()) {
|
||||||
|
packet.Set<TimeSeriesHeader>();
|
||||||
} else {
|
} else {
|
||||||
return absl::InvalidArgumentError(
|
return absl::InvalidArgumentError(
|
||||||
"None of supported values were specified in options.");
|
"None of supported values were specified in options.");
|
||||||
@@ -118,6 +121,9 @@ class ConstantSidePacketCalculator : public CalculatorBase {
|
|||||||
MakePacket<LandmarkList>(packet_options.landmark_list_value()));
|
MakePacket<LandmarkList>(packet_options.landmark_list_value()));
|
||||||
} else if (packet_options.has_double_value()) {
|
} else if (packet_options.has_double_value()) {
|
||||||
packet.Set(MakePacket<double>(packet_options.double_value()));
|
packet.Set(MakePacket<double>(packet_options.double_value()));
|
||||||
|
} else if (packet_options.has_time_series_header_value()) {
|
||||||
|
packet.Set(MakePacket<TimeSeriesHeader>(
|
||||||
|
packet_options.time_series_header_value()));
|
||||||
} else {
|
} else {
|
||||||
return absl::InvalidArgumentError(
|
return absl::InvalidArgumentError(
|
||||||
"None of supported values were specified in options.");
|
"None of supported values were specified in options.");
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ package mediapipe;
|
|||||||
import "mediapipe/framework/calculator.proto";
|
import "mediapipe/framework/calculator.proto";
|
||||||
import "mediapipe/framework/formats/classification.proto";
|
import "mediapipe/framework/formats/classification.proto";
|
||||||
import "mediapipe/framework/formats/landmark.proto";
|
import "mediapipe/framework/formats/landmark.proto";
|
||||||
|
import "mediapipe/framework/formats/time_series_header.proto";
|
||||||
|
|
||||||
message ConstantSidePacketCalculatorOptions {
|
message ConstantSidePacketCalculatorOptions {
|
||||||
extend CalculatorOptions {
|
extend CalculatorOptions {
|
||||||
@@ -35,6 +36,7 @@ message ConstantSidePacketCalculatorOptions {
|
|||||||
ClassificationList classification_list_value = 6;
|
ClassificationList classification_list_value = 6;
|
||||||
LandmarkList landmark_list_value = 7;
|
LandmarkList landmark_list_value = 7;
|
||||||
double double_value = 9;
|
double double_value = 9;
|
||||||
|
TimeSeriesHeader time_series_header_value = 10;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -21,6 +21,7 @@
|
|||||||
#include "mediapipe/framework/formats/landmark.pb.h"
|
#include "mediapipe/framework/formats/landmark.pb.h"
|
||||||
#include "mediapipe/framework/formats/matrix.h"
|
#include "mediapipe/framework/formats/matrix.h"
|
||||||
#include "mediapipe/framework/formats/rect.pb.h"
|
#include "mediapipe/framework/formats/rect.pb.h"
|
||||||
|
#include "mediapipe/framework/formats/tensor.h"
|
||||||
#include "mediapipe/util/render_data.pb.h"
|
#include "mediapipe/util/render_data.pb.h"
|
||||||
#include "tensorflow/lite/interpreter.h"
|
#include "tensorflow/lite/interpreter.h"
|
||||||
|
|
||||||
@@ -52,8 +53,9 @@ typedef EndLoopCalculator<std::vector<::mediapipe::ClassificationList>>
|
|||||||
EndLoopClassificationListCalculator;
|
EndLoopClassificationListCalculator;
|
||||||
REGISTER_CALCULATOR(EndLoopClassificationListCalculator);
|
REGISTER_CALCULATOR(EndLoopClassificationListCalculator);
|
||||||
|
|
||||||
typedef EndLoopCalculator<std::vector<TfLiteTensor>> EndLoopTensorCalculator;
|
typedef EndLoopCalculator<std::vector<TfLiteTensor>>
|
||||||
REGISTER_CALCULATOR(EndLoopTensorCalculator);
|
EndLoopTfLiteTensorCalculator;
|
||||||
|
REGISTER_CALCULATOR(EndLoopTfLiteTensorCalculator);
|
||||||
|
|
||||||
typedef EndLoopCalculator<std::vector<::mediapipe::Detection>>
|
typedef EndLoopCalculator<std::vector<::mediapipe::Detection>>
|
||||||
EndLoopDetectionCalculator;
|
EndLoopDetectionCalculator;
|
||||||
@@ -62,4 +64,7 @@ REGISTER_CALCULATOR(EndLoopDetectionCalculator);
|
|||||||
typedef EndLoopCalculator<std::vector<Matrix>> EndLoopMatrixCalculator;
|
typedef EndLoopCalculator<std::vector<Matrix>> EndLoopMatrixCalculator;
|
||||||
REGISTER_CALCULATOR(EndLoopMatrixCalculator);
|
REGISTER_CALCULATOR(EndLoopMatrixCalculator);
|
||||||
|
|
||||||
|
typedef EndLoopCalculator<std::vector<Tensor>> EndLoopTensorCalculator;
|
||||||
|
REGISTER_CALCULATOR(EndLoopTensorCalculator);
|
||||||
|
|
||||||
} // namespace mediapipe
|
} // namespace mediapipe
|
||||||
|
|||||||
@@ -15,6 +15,8 @@
|
|||||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_END_LOOP_CALCULATOR_H_
|
#ifndef MEDIAPIPE_CALCULATORS_CORE_END_LOOP_CALCULATOR_H_
|
||||||
#define MEDIAPIPE_CALCULATORS_CORE_END_LOOP_CALCULATOR_H_
|
#define MEDIAPIPE_CALCULATORS_CORE_END_LOOP_CALCULATOR_H_
|
||||||
|
|
||||||
|
#include <type_traits>
|
||||||
|
|
||||||
#include "mediapipe/framework/calculator_context.h"
|
#include "mediapipe/framework/calculator_context.h"
|
||||||
#include "mediapipe/framework/calculator_contract.h"
|
#include "mediapipe/framework/calculator_contract.h"
|
||||||
#include "mediapipe/framework/calculator_framework.h"
|
#include "mediapipe/framework/calculator_framework.h"
|
||||||
@@ -75,8 +77,23 @@ class EndLoopCalculator : public CalculatorBase {
|
|||||||
if (!input_stream_collection_) {
|
if (!input_stream_collection_) {
|
||||||
input_stream_collection_.reset(new IterableT);
|
input_stream_collection_.reset(new IterableT);
|
||||||
}
|
}
|
||||||
input_stream_collection_->push_back(
|
// Try to consume the item and move it into the collection. If the items
|
||||||
cc->Inputs().Tag("ITEM").template Get<ItemT>());
|
// are not consumable, then try to copy them instead. If the items are
|
||||||
|
// not copiable, then an error will be returned.
|
||||||
|
auto item_ptr_or = cc->Inputs().Tag("ITEM").Value().Consume<ItemT>();
|
||||||
|
if (item_ptr_or.ok()) {
|
||||||
|
input_stream_collection_->push_back(std::move(*item_ptr_or.value()));
|
||||||
|
} else {
|
||||||
|
if constexpr (std::is_copy_constructible_v<ItemT>) {
|
||||||
|
input_stream_collection_->push_back(
|
||||||
|
cc->Inputs().Tag("ITEM").template Get<ItemT>());
|
||||||
|
} else {
|
||||||
|
return absl::InternalError(
|
||||||
|
"The item type is not copiable. Consider making the "
|
||||||
|
"EndLoopCalculator the sole owner of the input packets so that "
|
||||||
|
"it can be moved instead of copying.");
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!cc->Inputs().Tag("BATCH_END").Value().IsEmpty()) { // flush signal
|
if (!cc->Inputs().Tag("BATCH_END").Value().IsEmpty()) { // flush signal
|
||||||
|
|||||||
@@ -35,7 +35,6 @@ message FlowLimiterCalculatorOptions {
|
|||||||
optional int32 max_in_queue = 2 [default = 0];
|
optional int32 max_in_queue = 2 [default = 0];
|
||||||
|
|
||||||
// The maximum time in microseconds to wait for a frame to finish processing.
|
// The maximum time in microseconds to wait for a frame to finish processing.
|
||||||
// The default value stops waiting after 1 sec.
|
// The default value 0 specifies no timeout.
|
||||||
// The value 0 specifies no timeout.
|
optional int64 in_flight_timeout = 3 [default = 0];
|
||||||
optional int64 in_flight_timeout = 3 [default = 1000000];
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ namespace api2 {
|
|||||||
// calculator: "Get{SpecificType}VectorItemCalculator"
|
// calculator: "Get{SpecificType}VectorItemCalculator"
|
||||||
// input_stream: "VECTOR:vector"
|
// input_stream: "VECTOR:vector"
|
||||||
// input_stream: "INDEX:index"
|
// input_stream: "INDEX:index"
|
||||||
// input_stream: "ITEM:item"
|
// output_stream: "ITEM:item"
|
||||||
// options {
|
// options {
|
||||||
// [mediapipe.GetVectorItemCalculatorOptions.ext] {
|
// [mediapipe.GetVectorItemCalculatorOptions.ext] {
|
||||||
// item_index: 5
|
// item_index: 5
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ limitations under the License.
|
|||||||
|
|
||||||
#include "mediapipe/calculators/core/merge_to_vector_calculator.h"
|
#include "mediapipe/calculators/core/merge_to_vector_calculator.h"
|
||||||
|
|
||||||
|
#include "mediapipe/framework/formats/detection.pb.h"
|
||||||
#include "mediapipe/framework/formats/image.h"
|
#include "mediapipe/framework/formats/image.h"
|
||||||
|
|
||||||
namespace mediapipe {
|
namespace mediapipe {
|
||||||
@@ -27,5 +28,9 @@ typedef MergeToVectorCalculator<mediapipe::GpuBuffer>
|
|||||||
MergeGpuBuffersToVectorCalculator;
|
MergeGpuBuffersToVectorCalculator;
|
||||||
MEDIAPIPE_REGISTER_NODE(MergeGpuBuffersToVectorCalculator);
|
MEDIAPIPE_REGISTER_NODE(MergeGpuBuffersToVectorCalculator);
|
||||||
|
|
||||||
|
typedef MergeToVectorCalculator<mediapipe::Detection>
|
||||||
|
MergeDetectionsToVectorCalculator;
|
||||||
|
MEDIAPIPE_REGISTER_NODE(MergeDetectionsToVectorCalculator);
|
||||||
|
|
||||||
} // namespace api2
|
} // namespace api2
|
||||||
} // namespace mediapipe
|
} // namespace mediapipe
|
||||||
|
|||||||
@@ -48,7 +48,6 @@ class MergeToVectorCalculator : public Node {
|
|||||||
}
|
}
|
||||||
|
|
||||||
absl::Status Process(CalculatorContext* cc) {
|
absl::Status Process(CalculatorContext* cc) {
|
||||||
const int input_num = kIn(cc).Count();
|
|
||||||
std::vector<T> output_vector;
|
std::vector<T> output_vector;
|
||||||
for (auto it = kIn(cc).begin(); it != kIn(cc).end(); it++) {
|
for (auto it = kIn(cc).begin(); it != kIn(cc).end(); it++) {
|
||||||
const auto& elem = *it;
|
const auto& elem = *it;
|
||||||
|
|||||||
@@ -76,7 +76,11 @@ constexpr char kMaxInFlightTag[] = "MAX_IN_FLIGHT";
|
|||||||
// }
|
// }
|
||||||
// output_stream: "gated_frames"
|
// output_stream: "gated_frames"
|
||||||
// }
|
// }
|
||||||
class RealTimeFlowLimiterCalculator : public CalculatorBase {
|
//
|
||||||
|
// Please use FlowLimiterCalculator, which replaces this calculator and
|
||||||
|
// defines a few additional configuration options.
|
||||||
|
class ABSL_DEPRECATED("Use FlowLimiterCalculator instead.")
|
||||||
|
RealTimeFlowLimiterCalculator : public CalculatorBase {
|
||||||
public:
|
public:
|
||||||
static absl::Status GetContract(CalculatorContract* cc) {
|
static absl::Status GetContract(CalculatorContract* cc) {
|
||||||
int num_data_streams = cc->Inputs().NumEntries("");
|
int num_data_streams = cc->Inputs().NumEntries("");
|
||||||
|
|||||||
@@ -66,12 +66,16 @@ class SequenceShiftCalculator : public Node {
|
|||||||
// The number of packets or timestamps we need to store to output packet[i] at
|
// The number of packets or timestamps we need to store to output packet[i] at
|
||||||
// the timestamp of packet[i + packet_offset]; equal to abs(packet_offset).
|
// the timestamp of packet[i + packet_offset]; equal to abs(packet_offset).
|
||||||
int cache_size_;
|
int cache_size_;
|
||||||
|
bool emit_empty_packets_before_first_packet_ = false;
|
||||||
};
|
};
|
||||||
MEDIAPIPE_REGISTER_NODE(SequenceShiftCalculator);
|
MEDIAPIPE_REGISTER_NODE(SequenceShiftCalculator);
|
||||||
|
|
||||||
absl::Status SequenceShiftCalculator::Open(CalculatorContext* cc) {
|
absl::Status SequenceShiftCalculator::Open(CalculatorContext* cc) {
|
||||||
packet_offset_ = kOffset(cc).GetOr(
|
packet_offset_ = kOffset(cc).GetOr(
|
||||||
cc->Options<mediapipe::SequenceShiftCalculatorOptions>().packet_offset());
|
cc->Options<mediapipe::SequenceShiftCalculatorOptions>().packet_offset());
|
||||||
|
emit_empty_packets_before_first_packet_ =
|
||||||
|
cc->Options<mediapipe::SequenceShiftCalculatorOptions>()
|
||||||
|
.emit_empty_packets_before_first_packet();
|
||||||
cache_size_ = abs(packet_offset_);
|
cache_size_ = abs(packet_offset_);
|
||||||
// An offset of zero is a no-op, but someone might still request it.
|
// An offset of zero is a no-op, but someone might still request it.
|
||||||
if (packet_offset_ == 0) {
|
if (packet_offset_ == 0) {
|
||||||
@@ -96,6 +100,8 @@ void SequenceShiftCalculator::ProcessPositiveOffset(CalculatorContext* cc) {
|
|||||||
// Ready to output oldest packet with current timestamp.
|
// Ready to output oldest packet with current timestamp.
|
||||||
kOut(cc).Send(packet_cache_.front().At(cc->InputTimestamp()));
|
kOut(cc).Send(packet_cache_.front().At(cc->InputTimestamp()));
|
||||||
packet_cache_.pop_front();
|
packet_cache_.pop_front();
|
||||||
|
} else if (emit_empty_packets_before_first_packet_) {
|
||||||
|
LOG(FATAL) << "Not supported yet";
|
||||||
}
|
}
|
||||||
// Store current packet for later output.
|
// Store current packet for later output.
|
||||||
packet_cache_.push_back(kIn(cc).packet());
|
packet_cache_.push_back(kIn(cc).packet());
|
||||||
|
|||||||
@@ -23,4 +23,8 @@ message SequenceShiftCalculatorOptions {
|
|||||||
optional SequenceShiftCalculatorOptions ext = 107633927;
|
optional SequenceShiftCalculatorOptions ext = 107633927;
|
||||||
}
|
}
|
||||||
optional int32 packet_offset = 1 [default = -1];
|
optional int32 packet_offset = 1 [default = -1];
|
||||||
|
|
||||||
|
// Emits empty packets before the first delayed packet is emitted. Takes
|
||||||
|
// effect only when packet offset is set to positive.
|
||||||
|
optional bool emit_empty_packets_before_first_packet = 2 [default = false];
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -301,12 +301,18 @@ cc_test(
|
|||||||
":image_cropping_calculator",
|
":image_cropping_calculator",
|
||||||
":image_cropping_calculator_cc_proto",
|
":image_cropping_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
|
"//mediapipe/framework:calculator_runner",
|
||||||
|
"//mediapipe/framework/formats:image_frame_opencv",
|
||||||
"//mediapipe/framework/formats:rect_cc_proto",
|
"//mediapipe/framework/formats:rect_cc_proto",
|
||||||
"//mediapipe/framework/port:gtest_main",
|
"//mediapipe/framework/port:gtest_main",
|
||||||
|
"//mediapipe/framework/port:opencv_core",
|
||||||
|
"//mediapipe/framework/port:opencv_imgproc",
|
||||||
"//mediapipe/framework/port:parse_text_proto",
|
"//mediapipe/framework/port:parse_text_proto",
|
||||||
"//mediapipe/framework/port:status",
|
"//mediapipe/framework/port:status",
|
||||||
"//mediapipe/framework/tool:tag_map",
|
"//mediapipe/framework/tool:tag_map",
|
||||||
"//mediapipe/framework/tool:tag_map_helper",
|
"//mediapipe/framework/tool:tag_map_helper",
|
||||||
|
"//mediapipe/gpu:gpu_buffer_to_image_frame_calculator",
|
||||||
|
"//mediapipe/gpu:image_frame_to_gpu_buffer_calculator",
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -378,8 +384,8 @@ cc_library(
|
|||||||
name = "scale_image_calculator",
|
name = "scale_image_calculator",
|
||||||
srcs = ["scale_image_calculator.cc"],
|
srcs = ["scale_image_calculator.cc"],
|
||||||
deps = [
|
deps = [
|
||||||
|
":scale_image_calculator_cc_proto",
|
||||||
":scale_image_utils",
|
":scale_image_utils",
|
||||||
"//mediapipe/calculators/image:scale_image_calculator_cc_proto",
|
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
"//mediapipe/framework/formats:image_format_cc_proto",
|
"//mediapipe/framework/formats:image_format_cc_proto",
|
||||||
"//mediapipe/framework/formats:image_frame",
|
"//mediapipe/framework/formats:image_frame",
|
||||||
@@ -747,8 +753,8 @@ cc_test(
|
|||||||
tags = ["desktop_only_test"],
|
tags = ["desktop_only_test"],
|
||||||
deps = [
|
deps = [
|
||||||
":affine_transformation",
|
":affine_transformation",
|
||||||
|
":image_transformation_calculator",
|
||||||
":warp_affine_calculator",
|
":warp_affine_calculator",
|
||||||
"//mediapipe/calculators/image:image_transformation_calculator",
|
|
||||||
"//mediapipe/calculators/tensor:image_to_tensor_converter",
|
"//mediapipe/calculators/tensor:image_to_tensor_converter",
|
||||||
"//mediapipe/calculators/tensor:image_to_tensor_utils",
|
"//mediapipe/calculators/tensor:image_to_tensor_utils",
|
||||||
"//mediapipe/calculators/util:from_image_calculator",
|
"//mediapipe/calculators/util:from_image_calculator",
|
||||||
|
|||||||
@@ -92,8 +92,8 @@ class GlTextureWarpAffineRunner
|
|||||||
|
|
||||||
constexpr GLchar kVertShader[] = R"(
|
constexpr GLchar kVertShader[] = R"(
|
||||||
in vec4 position;
|
in vec4 position;
|
||||||
in mediump vec4 texture_coordinate;
|
in highp vec4 texture_coordinate;
|
||||||
out mediump vec2 sample_coordinate;
|
out highp vec2 sample_coordinate;
|
||||||
uniform mat4 transform_matrix;
|
uniform mat4 transform_matrix;
|
||||||
|
|
||||||
void main() {
|
void main() {
|
||||||
@@ -104,7 +104,7 @@ class GlTextureWarpAffineRunner
|
|||||||
)";
|
)";
|
||||||
|
|
||||||
constexpr GLchar kFragShader[] = R"(
|
constexpr GLchar kFragShader[] = R"(
|
||||||
DEFAULT_PRECISION(mediump, float)
|
DEFAULT_PRECISION(highp, float)
|
||||||
in vec2 sample_coordinate;
|
in vec2 sample_coordinate;
|
||||||
uniform sampler2D input_texture;
|
uniform sampler2D input_texture;
|
||||||
|
|
||||||
|
|||||||
@@ -38,6 +38,7 @@ void SetColorChannel(int channel, uint8 value, cv::Mat* mat) {
|
|||||||
|
|
||||||
constexpr char kRgbaInTag[] = "RGBA_IN";
|
constexpr char kRgbaInTag[] = "RGBA_IN";
|
||||||
constexpr char kRgbInTag[] = "RGB_IN";
|
constexpr char kRgbInTag[] = "RGB_IN";
|
||||||
|
constexpr char kBgrInTag[] = "BGR_IN";
|
||||||
constexpr char kBgraInTag[] = "BGRA_IN";
|
constexpr char kBgraInTag[] = "BGRA_IN";
|
||||||
constexpr char kGrayInTag[] = "GRAY_IN";
|
constexpr char kGrayInTag[] = "GRAY_IN";
|
||||||
constexpr char kRgbaOutTag[] = "RGBA_OUT";
|
constexpr char kRgbaOutTag[] = "RGBA_OUT";
|
||||||
@@ -57,6 +58,7 @@ constexpr char kGrayOutTag[] = "GRAY_OUT";
|
|||||||
// RGB -> RGBA
|
// RGB -> RGBA
|
||||||
// RGBA -> BGRA
|
// RGBA -> BGRA
|
||||||
// BGRA -> RGBA
|
// BGRA -> RGBA
|
||||||
|
// BGR -> RGB
|
||||||
//
|
//
|
||||||
// This calculator only supports a single input stream and output stream at a
|
// This calculator only supports a single input stream and output stream at a
|
||||||
// time. If more than one input stream or output stream is present, the
|
// time. If more than one input stream or output stream is present, the
|
||||||
@@ -69,6 +71,7 @@ constexpr char kGrayOutTag[] = "GRAY_OUT";
|
|||||||
// RGB_IN: The input video stream (ImageFrame, SRGB).
|
// RGB_IN: The input video stream (ImageFrame, SRGB).
|
||||||
// BGRA_IN: The input video stream (ImageFrame, SBGRA).
|
// BGRA_IN: The input video stream (ImageFrame, SBGRA).
|
||||||
// GRAY_IN: The input video stream (ImageFrame, GRAY8).
|
// GRAY_IN: The input video stream (ImageFrame, GRAY8).
|
||||||
|
// BGR_IN: The input video stream (ImageFrame, SBGR).
|
||||||
//
|
//
|
||||||
// Output streams:
|
// Output streams:
|
||||||
// RGBA_OUT: The output video stream (ImageFrame, SRGBA).
|
// RGBA_OUT: The output video stream (ImageFrame, SRGBA).
|
||||||
@@ -122,6 +125,10 @@ absl::Status ColorConvertCalculator::GetContract(CalculatorContract* cc) {
|
|||||||
cc->Inputs().Tag(kBgraInTag).Set<ImageFrame>();
|
cc->Inputs().Tag(kBgraInTag).Set<ImageFrame>();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (cc->Inputs().HasTag(kBgrInTag)) {
|
||||||
|
cc->Inputs().Tag(kBgrInTag).Set<ImageFrame>();
|
||||||
|
}
|
||||||
|
|
||||||
if (cc->Outputs().HasTag(kRgbOutTag)) {
|
if (cc->Outputs().HasTag(kRgbOutTag)) {
|
||||||
cc->Outputs().Tag(kRgbOutTag).Set<ImageFrame>();
|
cc->Outputs().Tag(kRgbOutTag).Set<ImageFrame>();
|
||||||
}
|
}
|
||||||
@@ -194,6 +201,11 @@ absl::Status ColorConvertCalculator::Process(CalculatorContext* cc) {
|
|||||||
return ConvertAndOutput(kRgbaInTag, kBgraOutTag, ImageFormat::SBGRA,
|
return ConvertAndOutput(kRgbaInTag, kBgraOutTag, ImageFormat::SBGRA,
|
||||||
cv::COLOR_RGBA2BGRA, cc);
|
cv::COLOR_RGBA2BGRA, cc);
|
||||||
}
|
}
|
||||||
|
// BGR -> RGB
|
||||||
|
if (cc->Inputs().HasTag(kBgrInTag) && cc->Outputs().HasTag(kRgbOutTag)) {
|
||||||
|
return ConvertAndOutput(kBgrInTag, kRgbOutTag, ImageFormat::SRGB,
|
||||||
|
cv::COLOR_BGR2RGB, cc);
|
||||||
|
}
|
||||||
|
|
||||||
return mediapipe::InvalidArgumentErrorBuilder(MEDIAPIPE_LOC)
|
return mediapipe::InvalidArgumentErrorBuilder(MEDIAPIPE_LOC)
|
||||||
<< "Unsupported image format conversion.";
|
<< "Unsupported image format conversion.";
|
||||||
|
|||||||
@@ -37,7 +37,8 @@ enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
|
|||||||
namespace mediapipe {
|
namespace mediapipe {
|
||||||
|
|
||||||
namespace {
|
namespace {
|
||||||
|
using ::mediapipe::NormalizedRect;
|
||||||
|
using ::mediapipe::Rect;
|
||||||
#if !MEDIAPIPE_DISABLE_GPU
|
#if !MEDIAPIPE_DISABLE_GPU
|
||||||
|
|
||||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||||
@@ -222,8 +223,8 @@ absl::Status ImageCroppingCalculator::RenderCpu(CalculatorContext* cc) {
|
|||||||
cv::Mat input_mat = formats::MatView(&input_img);
|
cv::Mat input_mat = formats::MatView(&input_img);
|
||||||
|
|
||||||
RectSpec specs = GetCropSpecs(cc, input_img.Width(), input_img.Height());
|
RectSpec specs = GetCropSpecs(cc, input_img.Width(), input_img.Height());
|
||||||
int target_width = specs.width, target_height = specs.height,
|
int target_width = specs.width, target_height = specs.height;
|
||||||
rect_center_x = specs.center_x, rect_center_y = specs.center_y;
|
float rect_center_x = specs.center_x, rect_center_y = specs.center_y;
|
||||||
float rotation = specs.rotation;
|
float rotation = specs.rotation;
|
||||||
|
|
||||||
// Get border mode and value for OpenCV.
|
// Get border mode and value for OpenCV.
|
||||||
@@ -243,19 +244,29 @@ absl::Status ImageCroppingCalculator::RenderCpu(CalculatorContext* cc) {
|
|||||||
output_width *= scale;
|
output_width *= scale;
|
||||||
output_height *= scale;
|
output_height *= scale;
|
||||||
|
|
||||||
float dst_corners[8] = {0,
|
float dst_corners[8] = {
|
||||||
output_height - 1,
|
0, output_height, 0, 0, output_width, 0, output_width, output_height};
|
||||||
0,
|
const cv::Mat dst_points = cv::Mat(4, 2, CV_32F, dst_corners);
|
||||||
0,
|
// The projection matrix is computed using the corners of rects, not the
|
||||||
output_width - 1,
|
// centers of corner pixels
|
||||||
0,
|
const cv::Mat projection_matrix =
|
||||||
output_width - 1,
|
|
||||||
output_height - 1};
|
|
||||||
cv::Mat dst_points = cv::Mat(4, 2, CV_32F, dst_corners);
|
|
||||||
cv::Mat projection_matrix =
|
|
||||||
cv::getPerspectiveTransform(src_points, dst_points);
|
cv::getPerspectiveTransform(src_points, dst_points);
|
||||||
|
// The projection matrix need to be adjusted because `cv::warpPerspective` is
|
||||||
|
// based on integer centers.
|
||||||
|
// clang-format off
|
||||||
|
double shift_src_vec[9] = {1.0, 0.0, 0.5,
|
||||||
|
0.0, 1.0, 0.5,
|
||||||
|
0.0, 0.0, 1.0};
|
||||||
|
double shift_dst_vec[9] = {1.0, 0.0, -0.5,
|
||||||
|
0.0, 1.0, -0.5,
|
||||||
|
0.0, 0.0, 1.0};
|
||||||
|
// clang-format on
|
||||||
|
const cv::Mat shift_src = cv::Mat(3, 3, CV_64F, shift_src_vec);
|
||||||
|
const cv::Mat shift_dst = cv::Mat(3, 3, CV_64F, shift_dst_vec);
|
||||||
|
const cv::Mat adjusted_projection_matrix =
|
||||||
|
shift_dst * projection_matrix * shift_src;
|
||||||
cv::Mat cropped_image;
|
cv::Mat cropped_image;
|
||||||
cv::warpPerspective(input_mat, cropped_image, projection_matrix,
|
cv::warpPerspective(input_mat, cropped_image, adjusted_projection_matrix,
|
||||||
cv::Size(output_width, output_height),
|
cv::Size(output_width, output_height),
|
||||||
/* flags = */ 0,
|
/* flags = */ 0,
|
||||||
/* borderMode = */ border_mode);
|
/* borderMode = */ border_mode);
|
||||||
@@ -415,8 +426,8 @@ void ImageCroppingCalculator::GetOutputDimensions(CalculatorContext* cc,
|
|||||||
int* dst_width,
|
int* dst_width,
|
||||||
int* dst_height) {
|
int* dst_height) {
|
||||||
RectSpec specs = GetCropSpecs(cc, src_width, src_height);
|
RectSpec specs = GetCropSpecs(cc, src_width, src_height);
|
||||||
int crop_width = specs.width, crop_height = specs.height,
|
int crop_width = specs.width, crop_height = specs.height;
|
||||||
x_center = specs.center_x, y_center = specs.center_y;
|
float x_center = specs.center_x, y_center = specs.center_y;
|
||||||
float rotation = specs.rotation;
|
float rotation = specs.rotation;
|
||||||
|
|
||||||
const float half_width = crop_width / 2.0f;
|
const float half_width = crop_width / 2.0f;
|
||||||
@@ -465,8 +476,8 @@ RectSpec ImageCroppingCalculator::GetCropSpecs(const CalculatorContext* cc,
|
|||||||
int crop_width = src_width;
|
int crop_width = src_width;
|
||||||
int crop_height = src_height;
|
int crop_height = src_height;
|
||||||
// Get the center of cropping box. Default is the at the center.
|
// Get the center of cropping box. Default is the at the center.
|
||||||
int x_center = src_width / 2;
|
float x_center = src_width / 2.0f;
|
||||||
int y_center = src_height / 2;
|
float y_center = src_height / 2.0f;
|
||||||
// Get the rotation of the cropping box.
|
// Get the rotation of the cropping box.
|
||||||
float rotation = 0.0f;
|
float rotation = 0.0f;
|
||||||
// Get the normalized width and height if specified by the inputs or options.
|
// Get the normalized width and height if specified by the inputs or options.
|
||||||
@@ -493,8 +504,8 @@ RectSpec ImageCroppingCalculator::GetCropSpecs(const CalculatorContext* cc,
|
|||||||
if (norm_rect.width() > 0.0 && norm_rect.height() > 0.0) {
|
if (norm_rect.width() > 0.0 && norm_rect.height() > 0.0) {
|
||||||
normalized_width = norm_rect.width();
|
normalized_width = norm_rect.width();
|
||||||
normalized_height = norm_rect.height();
|
normalized_height = norm_rect.height();
|
||||||
x_center = std::round(norm_rect.x_center() * src_width);
|
x_center = norm_rect.x_center() * src_width;
|
||||||
y_center = std::round(norm_rect.y_center() * src_height);
|
y_center = norm_rect.y_center() * src_height;
|
||||||
rotation = norm_rect.rotation();
|
rotation = norm_rect.rotation();
|
||||||
}
|
}
|
||||||
} else if (cc->Inputs().HasTag(kWidthTag) &&
|
} else if (cc->Inputs().HasTag(kWidthTag) &&
|
||||||
@@ -520,14 +531,30 @@ RectSpec ImageCroppingCalculator::GetCropSpecs(const CalculatorContext* cc,
|
|||||||
// present from the inputs.
|
// present from the inputs.
|
||||||
if (!cc->Inputs().HasTag(kRectTag) && !cc->Inputs().HasTag(kNormRectTag)) {
|
if (!cc->Inputs().HasTag(kRectTag) && !cc->Inputs().HasTag(kNormRectTag)) {
|
||||||
if (options.has_norm_center_x() && options.has_norm_center_y()) {
|
if (options.has_norm_center_x() && options.has_norm_center_y()) {
|
||||||
x_center = std::round(options.norm_center_x() * src_width);
|
x_center = options.norm_center_x() * src_width;
|
||||||
y_center = std::round(options.norm_center_y() * src_height);
|
y_center = options.norm_center_y() * src_height;
|
||||||
}
|
}
|
||||||
if (options.has_rotation()) {
|
if (options.has_rotation()) {
|
||||||
rotation = options.rotation();
|
rotation = options.rotation();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (rotation == 0.0f) {
|
||||||
|
// Adjust the center to the closest integer when the crop size is
|
||||||
|
// even-number and to the closest half-integer when the crop size is
|
||||||
|
// odd-number.
|
||||||
|
if (crop_width % 2 == 0) {
|
||||||
|
x_center = std::round(x_center);
|
||||||
|
} else {
|
||||||
|
x_center = std::round(x_center + 0.5f) - 0.5f;
|
||||||
|
}
|
||||||
|
if (crop_height % 2 == 0) {
|
||||||
|
y_center = std::round(y_center);
|
||||||
|
} else {
|
||||||
|
y_center = std::round(y_center + 0.5f) - 0.5f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return {crop_width, crop_height, x_center, y_center, rotation};
|
return {crop_width, crop_height, x_center, y_center, rotation};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -42,8 +42,8 @@ namespace mediapipe {
|
|||||||
struct RectSpec {
|
struct RectSpec {
|
||||||
int width;
|
int width;
|
||||||
int height;
|
int height;
|
||||||
int center_x;
|
float center_x;
|
||||||
int center_y;
|
float center_y;
|
||||||
float rotation;
|
float rotation;
|
||||||
|
|
||||||
bool operator==(const RectSpec& rect) const {
|
bool operator==(const RectSpec& rect) const {
|
||||||
|
|||||||
@@ -19,8 +19,12 @@
|
|||||||
|
|
||||||
#include "mediapipe/calculators/image/image_cropping_calculator.pb.h"
|
#include "mediapipe/calculators/image/image_cropping_calculator.pb.h"
|
||||||
#include "mediapipe/framework/calculator_framework.h"
|
#include "mediapipe/framework/calculator_framework.h"
|
||||||
|
#include "mediapipe/framework/calculator_runner.h"
|
||||||
|
#include "mediapipe/framework/formats/image_frame_opencv.h"
|
||||||
#include "mediapipe/framework/formats/rect.pb.h"
|
#include "mediapipe/framework/formats/rect.pb.h"
|
||||||
#include "mediapipe/framework/port/gtest.h"
|
#include "mediapipe/framework/port/gtest.h"
|
||||||
|
#include "mediapipe/framework/port/opencv_core_inc.h"
|
||||||
|
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
|
||||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||||
#include "mediapipe/framework/port/status_matchers.h"
|
#include "mediapipe/framework/port/status_matchers.h"
|
||||||
#include "mediapipe/framework/tool/tag_map.h"
|
#include "mediapipe/framework/tool/tag_map.h"
|
||||||
@@ -37,6 +41,166 @@ constexpr char kRectTag[] = "RECT";
|
|||||||
constexpr char kHeightTag[] = "HEIGHT";
|
constexpr char kHeightTag[] = "HEIGHT";
|
||||||
constexpr char kWidthTag[] = "WIDTH";
|
constexpr char kWidthTag[] = "WIDTH";
|
||||||
|
|
||||||
|
std::unique_ptr<mediapipe::ImageFrame> GetInputFrame(int width, int height,
|
||||||
|
int channel) {
|
||||||
|
const int total_size = width * height * channel;
|
||||||
|
|
||||||
|
auto image_format = channel == 4 ? mediapipe::ImageFormat::SRGBA
|
||||||
|
: mediapipe::ImageFormat::SRGB;
|
||||||
|
|
||||||
|
auto input_frame = std::make_unique<mediapipe::ImageFrame>(
|
||||||
|
image_format, width, height, /*alignment_boundary =*/1);
|
||||||
|
for (int i = 0; i < total_size; ++i) {
|
||||||
|
input_frame->MutablePixelData()[i] = i % 256;
|
||||||
|
}
|
||||||
|
|
||||||
|
return input_frame;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Test identity function, where cropping size is same as input size
|
||||||
|
TEST(ImageCroppingCalculatorTest, IdentityFunctionCropWithOriginalSize) {
|
||||||
|
auto calculator_node =
|
||||||
|
ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig::Node>(
|
||||||
|
absl::Substitute(
|
||||||
|
R"pb(
|
||||||
|
calculator: "ImageCroppingCalculator"
|
||||||
|
input_stream: "IMAGE:input_frames"
|
||||||
|
output_stream: "IMAGE:cropped_output_frames"
|
||||||
|
options: {
|
||||||
|
[mediapipe.ImageCroppingCalculatorOptions.ext] {
|
||||||
|
width: $0
|
||||||
|
height: $1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)pb",
|
||||||
|
input_width, input_height));
|
||||||
|
mediapipe::CalculatorRunner runner(calculator_node);
|
||||||
|
|
||||||
|
// Input frame.
|
||||||
|
const auto input_frame = GetInputFrame(input_width, input_height, 3);
|
||||||
|
auto input_frame_packet =
|
||||||
|
mediapipe::MakePacket<mediapipe::ImageFrame>(std::move(*input_frame));
|
||||||
|
runner.MutableInputs()->Tag("IMAGE").packets.push_back(
|
||||||
|
input_frame_packet.At(mediapipe::Timestamp(1)));
|
||||||
|
|
||||||
|
MP_ASSERT_OK(runner.Run());
|
||||||
|
|
||||||
|
const auto& outputs = runner.Outputs();
|
||||||
|
EXPECT_EQ(outputs.NumEntries(), 1);
|
||||||
|
const auto& output_image =
|
||||||
|
outputs.Tag("IMAGE").packets[0].Get<mediapipe::ImageFrame>();
|
||||||
|
|
||||||
|
const auto expected_output = GetInputFrame(input_width, input_height, 3);
|
||||||
|
cv::Mat output_mat = formats::MatView(&output_image);
|
||||||
|
cv::Mat expected_mat = formats::MatView(expected_output.get());
|
||||||
|
double max_diff = cv::norm(expected_mat, output_mat, cv::NORM_INF);
|
||||||
|
EXPECT_EQ(max_diff, 0);
|
||||||
|
} // TEST
|
||||||
|
|
||||||
|
// Test identity function, where cropping size is same as input size.
|
||||||
|
// When an image has an odd number for its size, its center falls on a
|
||||||
|
// fractional pixel. As a result, the values for center_x and center_y need to
|
||||||
|
// be of type float.
|
||||||
|
TEST(ImageCroppingCalculatorTest, IdentityFunctionCropWithOddSize) {
|
||||||
|
const int input_width = 99;
|
||||||
|
const int input_height = 99;
|
||||||
|
|
||||||
|
auto calculator_node =
|
||||||
|
ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig::Node>(
|
||||||
|
absl::Substitute(
|
||||||
|
R"pb(
|
||||||
|
calculator: "ImageCroppingCalculator"
|
||||||
|
input_stream: "IMAGE:input_frames"
|
||||||
|
output_stream: "IMAGE:cropped_output_frames"
|
||||||
|
options: {
|
||||||
|
[mediapipe.ImageCroppingCalculatorOptions.ext] {
|
||||||
|
width: $0
|
||||||
|
height: $1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)pb",
|
||||||
|
input_width, input_height));
|
||||||
|
mediapipe::CalculatorRunner runner(calculator_node);
|
||||||
|
|
||||||
|
// Input frame.
|
||||||
|
const auto input_frame = GetInputFrame(input_width, input_height, 3);
|
||||||
|
auto input_frame_packet =
|
||||||
|
mediapipe::MakePacket<mediapipe::ImageFrame>(std::move(*input_frame));
|
||||||
|
runner.MutableInputs()->Tag("IMAGE").packets.push_back(
|
||||||
|
input_frame_packet.At(mediapipe::Timestamp(1)));
|
||||||
|
|
||||||
|
MP_ASSERT_OK(runner.Run());
|
||||||
|
|
||||||
|
const auto& outputs = runner.Outputs();
|
||||||
|
EXPECT_EQ(outputs.NumEntries(), 1);
|
||||||
|
const auto& output_image =
|
||||||
|
outputs.Tag("IMAGE").packets[0].Get<mediapipe::ImageFrame>();
|
||||||
|
|
||||||
|
const auto expected_output = GetInputFrame(input_width, input_height, 3);
|
||||||
|
cv::Mat output_mat = formats::MatView(&output_image);
|
||||||
|
cv::Mat expected_mat = formats::MatView(expected_output.get());
|
||||||
|
double max_diff = cv::norm(expected_mat, output_mat, cv::NORM_INF);
|
||||||
|
EXPECT_EQ(max_diff, 0);
|
||||||
|
} // TEST
|
||||||
|
|
||||||
|
// Test identity function on GPU, where cropping size is same as input size.
|
||||||
|
TEST(ImageCroppingCalculatorTest, IdentityFunctionCropWithOriginalSizeGPU) {
|
||||||
|
mediapipe::CalculatorGraphConfig config =
|
||||||
|
ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(absl::Substitute(
|
||||||
|
R"pb(
|
||||||
|
input_stream: "input_frames"
|
||||||
|
node {
|
||||||
|
calculator: "ImageFrameToGpuBufferCalculator"
|
||||||
|
input_stream: "input_frames"
|
||||||
|
output_stream: "input_frames_gpu"
|
||||||
|
}
|
||||||
|
node {
|
||||||
|
calculator: "ImageCroppingCalculator"
|
||||||
|
input_stream: "IMAGE_GPU:input_frames_gpu"
|
||||||
|
output_stream: "IMAGE_GPU:cropped_output_frames_gpu"
|
||||||
|
options: {
|
||||||
|
[mediapipe.ImageCroppingCalculatorOptions.ext] {
|
||||||
|
width: $0
|
||||||
|
height: $1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
node {
|
||||||
|
calculator: "GpuBufferToImageFrameCalculator"
|
||||||
|
input_stream: "cropped_output_frames_gpu"
|
||||||
|
output_stream: "cropped_output_frames"
|
||||||
|
}
|
||||||
|
)pb",
|
||||||
|
input_width, input_height));
|
||||||
|
|
||||||
|
std::vector<Packet> output_packets;
|
||||||
|
tool::AddVectorSink("cropped_output_frames", &config, &output_packets);
|
||||||
|
|
||||||
|
CalculatorGraph graph;
|
||||||
|
MP_ASSERT_OK(graph.Initialize(config));
|
||||||
|
|
||||||
|
// Input frame.
|
||||||
|
const auto input_frame = GetInputFrame(input_width, input_height, 4);
|
||||||
|
auto input_frame_packet =
|
||||||
|
mediapipe::MakePacket<mediapipe::ImageFrame>(std::move(*input_frame));
|
||||||
|
|
||||||
|
MP_ASSERT_OK(graph.StartRun({}));
|
||||||
|
MP_ASSERT_OK(graph.AddPacketToInputStream(
|
||||||
|
"input_frames", input_frame_packet.At(mediapipe::Timestamp(1))));
|
||||||
|
MP_ASSERT_OK(graph.WaitUntilIdle());
|
||||||
|
|
||||||
|
// Get and process results.
|
||||||
|
const ImageFrame& output_image = output_packets[0].Get<ImageFrame>();
|
||||||
|
std::cout << output_image.Width();
|
||||||
|
|
||||||
|
const auto expected_output = GetInputFrame(input_width, input_height, 4);
|
||||||
|
cv::Mat output_mat = formats::MatView(&output_image);
|
||||||
|
cv::Mat expected_mat = formats::MatView(expected_output.get());
|
||||||
|
double max_diff = cv::norm(expected_mat, output_mat, cv::NORM_INF);
|
||||||
|
|
||||||
|
EXPECT_EQ(max_diff, 0);
|
||||||
|
} // TEST
|
||||||
|
|
||||||
// Test normal case, where norm_width and norm_height in options are set.
|
// Test normal case, where norm_width and norm_height in options are set.
|
||||||
TEST(ImageCroppingCalculatorTest, GetCroppingDimensionsNormal) {
|
TEST(ImageCroppingCalculatorTest, GetCroppingDimensionsNormal) {
|
||||||
auto calculator_node =
|
auto calculator_node =
|
||||||
@@ -195,11 +359,11 @@ TEST(ImageCroppingCalculatorTest, RedundantSpecWithInputStream) {
|
|||||||
auto cc = absl::make_unique<CalculatorContext>(
|
auto cc = absl::make_unique<CalculatorContext>(
|
||||||
calculator_state.get(), inputTags, tool::CreateTagMap({}).value());
|
calculator_state.get(), inputTags, tool::CreateTagMap({}).value());
|
||||||
auto& inputs = cc->Inputs();
|
auto& inputs = cc->Inputs();
|
||||||
mediapipe::Rect rect = ParseTextProtoOrDie<mediapipe::Rect>(
|
Rect rect = ParseTextProtoOrDie<Rect>(
|
||||||
R"pb(
|
R"pb(
|
||||||
width: 1 height: 1 x_center: 40 y_center: 40 rotation: 0.5
|
width: 1 height: 1 x_center: 40 y_center: 40 rotation: 0.5
|
||||||
)pb");
|
)pb");
|
||||||
inputs.Tag(kRectTag).Value() = MakePacket<mediapipe::Rect>(rect);
|
inputs.Tag(kRectTag).Value() = MakePacket<Rect>(rect);
|
||||||
RectSpec expectRect = {
|
RectSpec expectRect = {
|
||||||
.width = 1,
|
.width = 1,
|
||||||
.height = 1,
|
.height = 1,
|
||||||
|
|||||||
@@ -207,6 +207,7 @@ class ImageTransformationCalculator : public CalculatorBase {
|
|||||||
bool flip_vertically_ = false;
|
bool flip_vertically_ = false;
|
||||||
|
|
||||||
bool use_gpu_ = false;
|
bool use_gpu_ = false;
|
||||||
|
cv::Scalar padding_color_;
|
||||||
#if !MEDIAPIPE_DISABLE_GPU
|
#if !MEDIAPIPE_DISABLE_GPU
|
||||||
GlCalculatorHelper gpu_helper_;
|
GlCalculatorHelper gpu_helper_;
|
||||||
std::unique_ptr<QuadRenderer> rgb_renderer_;
|
std::unique_ptr<QuadRenderer> rgb_renderer_;
|
||||||
@@ -338,6 +339,9 @@ absl::Status ImageTransformationCalculator::Open(CalculatorContext* cc) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
scale_mode_ = ParseScaleMode(options_.scale_mode(), DEFAULT_SCALE_MODE);
|
scale_mode_ = ParseScaleMode(options_.scale_mode(), DEFAULT_SCALE_MODE);
|
||||||
|
padding_color_ = cv::Scalar(options_.padding_color().red(),
|
||||||
|
options_.padding_color().green(),
|
||||||
|
options_.padding_color().blue());
|
||||||
|
|
||||||
if (use_gpu_) {
|
if (use_gpu_) {
|
||||||
#if !MEDIAPIPE_DISABLE_GPU
|
#if !MEDIAPIPE_DISABLE_GPU
|
||||||
@@ -480,7 +484,8 @@ absl::Status ImageTransformationCalculator::RenderCpu(CalculatorContext* cc) {
|
|||||||
cv::copyMakeBorder(intermediate_mat, scaled_mat, top, bottom, left,
|
cv::copyMakeBorder(intermediate_mat, scaled_mat, top, bottom, left,
|
||||||
right,
|
right,
|
||||||
options_.constant_padding() ? cv::BORDER_CONSTANT
|
options_.constant_padding() ? cv::BORDER_CONSTANT
|
||||||
: cv::BORDER_REPLICATE);
|
: cv::BORDER_REPLICATE,
|
||||||
|
padding_color_);
|
||||||
} else {
|
} else {
|
||||||
cv::resize(input_mat, scaled_mat, cv::Size(target_width, target_height),
|
cv::resize(input_mat, scaled_mat, cv::Size(target_width, target_height),
|
||||||
0, 0, scale_flag);
|
0, 0, scale_flag);
|
||||||
|
|||||||
@@ -20,11 +20,21 @@ import "mediapipe/calculators/image/rotation_mode.proto";
|
|||||||
import "mediapipe/framework/calculator.proto";
|
import "mediapipe/framework/calculator.proto";
|
||||||
import "mediapipe/gpu/scale_mode.proto";
|
import "mediapipe/gpu/scale_mode.proto";
|
||||||
|
|
||||||
|
option java_package = "com.google.mediapipe.calculator.proto";
|
||||||
|
option java_outer_classname = "ImageTransformationCalculatorOptionsProto";
|
||||||
|
|
||||||
message ImageTransformationCalculatorOptions {
|
message ImageTransformationCalculatorOptions {
|
||||||
extend CalculatorOptions {
|
extend CalculatorOptions {
|
||||||
optional ImageTransformationCalculatorOptions ext = 251952830;
|
optional ImageTransformationCalculatorOptions ext = 251952830;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RGB values in range of 0 - 255
|
||||||
|
message Color {
|
||||||
|
optional int32 red = 1 [default = 0];
|
||||||
|
optional int32 green = 2 [default = 0];
|
||||||
|
optional int32 blue = 3 [default = 0];
|
||||||
|
}
|
||||||
|
|
||||||
// Output dimensions. Set to 0 if they should be the same as the input.
|
// Output dimensions. Set to 0 if they should be the same as the input.
|
||||||
optional int32 output_width = 1 [default = 0];
|
optional int32 output_width = 1 [default = 0];
|
||||||
optional int32 output_height = 2 [default = 0];
|
optional int32 output_height = 2 [default = 0];
|
||||||
@@ -40,4 +50,8 @@ message ImageTransformationCalculatorOptions {
|
|||||||
// Default is to use BORDER_CONSTANT. If set to false, it will use
|
// Default is to use BORDER_CONSTANT. If set to false, it will use
|
||||||
// BORDER_REPLICATE instead.
|
// BORDER_REPLICATE instead.
|
||||||
optional bool constant_padding = 7 [default = true];
|
optional bool constant_padding = 7 [default = true];
|
||||||
|
|
||||||
|
// The color for the padding. This option is only used when the scale mode is
|
||||||
|
// FIT. Default is black. This is for CPU only.
|
||||||
|
optional Color padding_color = 8;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -142,6 +142,9 @@ absl::Status FindOutputDimensions(int input_width, //
|
|||||||
static_cast<double>(input_height));
|
static_cast<double>(input_height));
|
||||||
try_width = (try_width / 2) * 2;
|
try_width = (try_width / 2) * 2;
|
||||||
try_height = (try_height / 2) * 2;
|
try_height = (try_height / 2) * 2;
|
||||||
|
// The output width/height should be greater than 0.
|
||||||
|
try_width = std::max(try_width, 1);
|
||||||
|
try_height = std::max(try_height, 1);
|
||||||
|
|
||||||
if (target_height <= 0 || try_height <= target_height) {
|
if (target_height <= 0 || try_height <= target_height) {
|
||||||
// The resulting height based on the target width and aspect ratio
|
// The resulting height based on the target width and aspect ratio
|
||||||
@@ -160,6 +163,9 @@ absl::Status FindOutputDimensions(int input_width, //
|
|||||||
static_cast<double>(input_width));
|
static_cast<double>(input_width));
|
||||||
try_width = (try_width / 2) * 2;
|
try_width = (try_width / 2) * 2;
|
||||||
try_height = (try_height / 2) * 2;
|
try_height = (try_height / 2) * 2;
|
||||||
|
// The output width/height should be greater than 0.
|
||||||
|
try_width = std::max(try_width, 1);
|
||||||
|
try_height = std::max(try_height, 1);
|
||||||
|
|
||||||
if (target_width <= 0 || try_width <= target_width) {
|
if (target_width <= 0 || try_width <= target_width) {
|
||||||
// The resulting width based on the target width and aspect ratio
|
// The resulting width based on the target width and aspect ratio
|
||||||
|
|||||||
@@ -124,6 +124,16 @@ TEST(ScaleImageUtilsTest, FindOutputDimensionsPreserveRatio) {
|
|||||||
&output_width, &output_height));
|
&output_width, &output_height));
|
||||||
EXPECT_EQ(151, output_width);
|
EXPECT_EQ(151, output_width);
|
||||||
EXPECT_EQ(101, output_height);
|
EXPECT_EQ(101, output_height);
|
||||||
|
// Scale to height 1.
|
||||||
|
MP_ASSERT_OK(FindOutputDimensions(10000, 10, 100, 0, 0, true, 2,
|
||||||
|
&output_width, &output_height));
|
||||||
|
EXPECT_EQ(100, output_width);
|
||||||
|
EXPECT_EQ(1, output_height);
|
||||||
|
// Scale to width 1.
|
||||||
|
MP_ASSERT_OK(FindOutputDimensions(10, 10000, 0, 100, 0, true, 2,
|
||||||
|
&output_width, &output_height));
|
||||||
|
EXPECT_EQ(1, output_width);
|
||||||
|
EXPECT_EQ(100, output_height);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Tests scaling without keeping the aspect ratio fixed.
|
// Tests scaling without keeping the aspect ratio fixed.
|
||||||
|
|||||||
@@ -47,10 +47,11 @@ constexpr int kNumChannelsRGBA = 4;
|
|||||||
|
|
||||||
enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
|
enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
|
||||||
|
|
||||||
// Combines an RGB cv::Mat and a single-channel alpha cv::Mat of the same
|
// Combines an RGB cv::Mat and an alpha cv::Mat of the same dimensions into an
|
||||||
// dimensions into an RGBA cv::Mat. Alpha may be read as uint8 or as another
|
// RGBA cv::Mat. Alpha may be read as uint8 or as another numeric type; in the
|
||||||
// numeric type; in the latter case, it is upscaled to values between 0 and 255
|
// latter case, it is upscaled to values between 0 and 255 from an assumed input
|
||||||
// from an assumed input range of [0, 1). RGB and RGBA Mat's must be uchar.
|
// range of [0, 1). Only the first channel of Alpha is used. Input & output Mat
|
||||||
|
// must be uchar.
|
||||||
template <typename AlphaType>
|
template <typename AlphaType>
|
||||||
absl::Status MergeRGBA8Image(const cv::Mat input_mat, const cv::Mat& alpha_mat,
|
absl::Status MergeRGBA8Image(const cv::Mat input_mat, const cv::Mat& alpha_mat,
|
||||||
cv::Mat& output_mat) {
|
cv::Mat& output_mat) {
|
||||||
@@ -71,9 +72,9 @@ absl::Status MergeRGBA8Image(const cv::Mat input_mat, const cv::Mat& alpha_mat,
|
|||||||
out_ptr[out_idx + 1] = in_ptr[in_idx + 1];
|
out_ptr[out_idx + 1] = in_ptr[in_idx + 1];
|
||||||
out_ptr[out_idx + 2] = in_ptr[in_idx + 2];
|
out_ptr[out_idx + 2] = in_ptr[in_idx + 2];
|
||||||
if constexpr (std::is_same<AlphaType, uchar>::value) {
|
if constexpr (std::is_same<AlphaType, uchar>::value) {
|
||||||
out_ptr[out_idx + 3] = alpha_ptr[alpha_idx + 0];
|
out_ptr[out_idx + 3] = alpha_ptr[alpha_idx + 0]; // channel 0 of mask
|
||||||
} else {
|
} else {
|
||||||
const AlphaType alpha = alpha_ptr[alpha_idx + 0];
|
const AlphaType alpha = alpha_ptr[alpha_idx + 0]; // channel 0 of mask
|
||||||
out_ptr[out_idx + 3] = static_cast<uchar>(round(alpha * 255.0f));
|
out_ptr[out_idx + 3] = static_cast<uchar>(round(alpha * 255.0f));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -291,8 +292,8 @@ absl::Status SetAlphaCalculator::RenderCpu(CalculatorContext* cc) {
|
|||||||
const auto& alpha_mask = cc->Inputs().Tag(kInputAlphaTag).Get<ImageFrame>();
|
const auto& alpha_mask = cc->Inputs().Tag(kInputAlphaTag).Get<ImageFrame>();
|
||||||
cv::Mat alpha_mat = mediapipe::formats::MatView(&alpha_mask);
|
cv::Mat alpha_mat = mediapipe::formats::MatView(&alpha_mask);
|
||||||
|
|
||||||
const bool alpha_is_float = alpha_mat.type() == CV_32FC1;
|
const bool alpha_is_float = CV_MAT_DEPTH(alpha_mat.type()) == CV_32F;
|
||||||
RET_CHECK(alpha_is_float || alpha_mat.type() == CV_8UC1);
|
RET_CHECK(alpha_is_float || CV_MAT_DEPTH(alpha_mat.type()) == CV_8U);
|
||||||
|
|
||||||
if (alpha_is_float) {
|
if (alpha_is_float) {
|
||||||
MP_RETURN_IF_ERROR(
|
MP_RETURN_IF_ERROR(
|
||||||
|
|||||||
@@ -12,10 +12,10 @@
|
|||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
licenses(["notice"])
|
|
||||||
|
|
||||||
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
|
||||||
|
|
||||||
|
licenses(["notice"])
|
||||||
|
|
||||||
package(default_visibility = ["//visibility:private"])
|
package(default_visibility = ["//visibility:private"])
|
||||||
|
|
||||||
proto_library(
|
proto_library(
|
||||||
|
|||||||
@@ -21,10 +21,12 @@ load(
|
|||||||
)
|
)
|
||||||
load("//mediapipe/framework:mediapipe_cc_test.bzl", "mediapipe_cc_test")
|
load("//mediapipe/framework:mediapipe_cc_test.bzl", "mediapipe_cc_test")
|
||||||
load("//mediapipe/framework:encode_binary_proto.bzl", "encode_binary_proto")
|
load("//mediapipe/framework:encode_binary_proto.bzl", "encode_binary_proto")
|
||||||
|
load("@org_tensorflow//tensorflow/lite/core/shims:cc_library_with_tflite.bzl", "cc_library_with_tflite")
|
||||||
|
load("//mediapipe/framework:more_selects.bzl", "more_selects")
|
||||||
|
|
||||||
licenses(["notice"])
|
licenses(["notice"])
|
||||||
|
|
||||||
package(default_visibility = ["//visibility:private"])
|
package(default_visibility = ["//visibility:public"])
|
||||||
|
|
||||||
exports_files(
|
exports_files(
|
||||||
glob(["testdata/image_to_tensor/*"]),
|
glob(["testdata/image_to_tensor/*"]),
|
||||||
@@ -34,6 +36,27 @@ exports_files(
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
config_setting(
|
||||||
|
name = "disable_gpu",
|
||||||
|
define_values = {
|
||||||
|
"MEDIAPIPE_DISABLE_GPU": "1",
|
||||||
|
},
|
||||||
|
visibility = ["//visibility:public"],
|
||||||
|
)
|
||||||
|
|
||||||
|
more_selects.config_setting_negation(
|
||||||
|
name = "not_disable_gpu",
|
||||||
|
negate = ":disable_gpu",
|
||||||
|
)
|
||||||
|
|
||||||
|
selects.config_setting_group(
|
||||||
|
name = "platform_ios_with_gpu",
|
||||||
|
match_all = [
|
||||||
|
":not_disable_gpu",
|
||||||
|
"//mediapipe:ios",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
selects.config_setting_group(
|
selects.config_setting_group(
|
||||||
name = "compute_shader_unavailable",
|
name = "compute_shader_unavailable",
|
||||||
match_any = [
|
match_any = [
|
||||||
@@ -44,9 +67,6 @@ selects.config_setting_group(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "audio_to_tensor_calculator_proto",
|
name = "audio_to_tensor_calculator_proto",
|
||||||
srcs = ["audio_to_tensor_calculator.proto"],
|
srcs = ["audio_to_tensor_calculator.proto"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -56,17 +76,6 @@ mediapipe_proto_library(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "audio_to_tensor_calculator",
|
name = "audio_to_tensor_calculator",
|
||||||
srcs = ["audio_to_tensor_calculator.cc"],
|
srcs = ["audio_to_tensor_calculator.cc"],
|
||||||
copts = select({
|
|
||||||
# b/215212850
|
|
||||||
"//mediapipe:apple": [
|
|
||||||
"-x objective-c++",
|
|
||||||
"-fobjc-arc",
|
|
||||||
],
|
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
":audio_to_tensor_calculator_cc_proto",
|
":audio_to_tensor_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -113,9 +122,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensors_to_audio_calculator_proto",
|
name = "tensors_to_audio_calculator_proto",
|
||||||
srcs = ["tensors_to_audio_calculator.proto"],
|
srcs = ["tensors_to_audio_calculator.proto"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -125,9 +131,6 @@ mediapipe_proto_library(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "tensors_to_audio_calculator",
|
name = "tensors_to_audio_calculator",
|
||||||
srcs = ["tensors_to_audio_calculator.cc"],
|
srcs = ["tensors_to_audio_calculator.cc"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensors_to_audio_calculator_cc_proto",
|
":tensors_to_audio_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -164,9 +167,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "feedback_tensors_calculator_proto",
|
name = "feedback_tensors_calculator_proto",
|
||||||
srcs = ["feedback_tensors_calculator.proto"],
|
srcs = ["feedback_tensors_calculator.proto"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -176,17 +176,6 @@ mediapipe_proto_library(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "feedback_tensors_calculator",
|
name = "feedback_tensors_calculator",
|
||||||
srcs = ["feedback_tensors_calculator.cc"],
|
srcs = ["feedback_tensors_calculator.cc"],
|
||||||
copts = select({
|
|
||||||
# b/215212850
|
|
||||||
"//mediapipe:apple": [
|
|
||||||
"-x objective-c++",
|
|
||||||
"-fobjc-arc",
|
|
||||||
],
|
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
":feedback_tensors_calculator_cc_proto",
|
":feedback_tensors_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -216,9 +205,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "bert_preprocessor_calculator_proto",
|
name = "bert_preprocessor_calculator_proto",
|
||||||
srcs = ["bert_preprocessor_calculator.proto"],
|
srcs = ["bert_preprocessor_calculator.proto"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -228,9 +214,6 @@ mediapipe_proto_library(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "bert_preprocessor_calculator",
|
name = "bert_preprocessor_calculator",
|
||||||
srcs = ["bert_preprocessor_calculator.cc"],
|
srcs = ["bert_preprocessor_calculator.cc"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
":bert_preprocessor_calculator_cc_proto",
|
":bert_preprocessor_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -274,9 +257,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "regex_preprocessor_calculator_proto",
|
name = "regex_preprocessor_calculator_proto",
|
||||||
srcs = ["regex_preprocessor_calculator.proto"],
|
srcs = ["regex_preprocessor_calculator.proto"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -286,9 +266,6 @@ mediapipe_proto_library(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "regex_preprocessor_calculator",
|
name = "regex_preprocessor_calculator",
|
||||||
srcs = ["regex_preprocessor_calculator.cc"],
|
srcs = ["regex_preprocessor_calculator.cc"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
":regex_preprocessor_calculator_cc_proto",
|
":regex_preprocessor_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -330,9 +307,6 @@ cc_test(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "text_to_tensor_calculator",
|
name = "text_to_tensor_calculator",
|
||||||
srcs = ["text_to_tensor_calculator.cc"],
|
srcs = ["text_to_tensor_calculator.cc"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_context",
|
"//mediapipe/framework:calculator_context",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -366,9 +340,6 @@ cc_test(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "universal_sentence_encoder_preprocessor_calculator",
|
name = "universal_sentence_encoder_preprocessor_calculator",
|
||||||
srcs = ["universal_sentence_encoder_preprocessor_calculator.cc"],
|
srcs = ["universal_sentence_encoder_preprocessor_calculator.cc"],
|
||||||
visibility = [
|
|
||||||
"//mediapipe/framework:mediapipe_internal",
|
|
||||||
],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_context",
|
"//mediapipe/framework:calculator_context",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -408,7 +379,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "inference_calculator_proto",
|
name = "inference_calculator_proto",
|
||||||
srcs = ["inference_calculator.proto"],
|
srcs = ["inference_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -423,19 +393,15 @@ mediapipe_proto_library(
|
|||||||
# size concerns), depend on those implementations directly, and do not depend on
|
# size concerns), depend on those implementations directly, and do not depend on
|
||||||
# :inference_calculator.
|
# :inference_calculator.
|
||||||
# In all cases, use "InferenceCalulator" in your graphs.
|
# In all cases, use "InferenceCalulator" in your graphs.
|
||||||
cc_library(
|
cc_library_with_tflite(
|
||||||
name = "inference_calculator_interface",
|
name = "inference_calculator_interface",
|
||||||
srcs = ["inference_calculator.cc"],
|
srcs = ["inference_calculator.cc"],
|
||||||
hdrs = ["inference_calculator.h"],
|
hdrs = ["inference_calculator.h"],
|
||||||
copts = select({
|
tflite_deps = [
|
||||||
# TODO: fix tensor.h not to require this, if possible
|
"//mediapipe/util/tflite:tflite_model_loader",
|
||||||
"//mediapipe:apple": [
|
"@org_tensorflow//tensorflow/lite/core/shims:framework_stable",
|
||||||
"-x objective-c++",
|
"@org_tensorflow//tensorflow/lite/core/shims:builtin_ops",
|
||||||
"-fobjc-arc", # enable reference-counting
|
],
|
||||||
],
|
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_calculator_cc_proto",
|
":inference_calculator_cc_proto",
|
||||||
":inference_calculator_options_lib",
|
":inference_calculator_options_lib",
|
||||||
@@ -446,12 +412,9 @@ cc_library(
|
|||||||
"//mediapipe/framework/port:ret_check",
|
"//mediapipe/framework/port:ret_check",
|
||||||
"//mediapipe/framework/stream_handler:fixed_size_input_stream_handler",
|
"//mediapipe/framework/stream_handler:fixed_size_input_stream_handler",
|
||||||
"//mediapipe/framework/tool:subgraph_expansion",
|
"//mediapipe/framework/tool:subgraph_expansion",
|
||||||
"//mediapipe/util/tflite:tflite_model_loader",
|
|
||||||
"@com_google_absl//absl/memory",
|
"@com_google_absl//absl/memory",
|
||||||
"@com_google_absl//absl/strings",
|
"@com_google_absl//absl/strings",
|
||||||
"@org_tensorflow//tensorflow/lite:framework",
|
|
||||||
"@org_tensorflow//tensorflow/lite/core/api:op_resolver",
|
"@org_tensorflow//tensorflow/lite/core/api:op_resolver",
|
||||||
"@org_tensorflow//tensorflow/lite/kernels:builtin_ops",
|
|
||||||
],
|
],
|
||||||
alwayslink = 1,
|
alwayslink = 1,
|
||||||
)
|
)
|
||||||
@@ -460,7 +423,6 @@ cc_library(
|
|||||||
name = "inference_calculator_gl",
|
name = "inference_calculator_gl",
|
||||||
srcs = ["inference_calculator_gl.cc"],
|
srcs = ["inference_calculator_gl.cc"],
|
||||||
tags = ["nomac"], # config problem with cpuinfo via TF
|
tags = ["nomac"], # config problem with cpuinfo via TF
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_calculator_cc_proto",
|
":inference_calculator_cc_proto",
|
||||||
":inference_calculator_interface",
|
":inference_calculator_interface",
|
||||||
@@ -478,7 +440,6 @@ cc_library(
|
|||||||
name = "inference_calculator_gl_advanced",
|
name = "inference_calculator_gl_advanced",
|
||||||
srcs = ["inference_calculator_gl_advanced.cc"],
|
srcs = ["inference_calculator_gl_advanced.cc"],
|
||||||
tags = ["nomac"],
|
tags = ["nomac"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_calculator_interface",
|
":inference_calculator_interface",
|
||||||
"@com_google_absl//absl/memory",
|
"@com_google_absl//absl/memory",
|
||||||
@@ -509,9 +470,9 @@ cc_library(
|
|||||||
"-framework MetalKit",
|
"-framework MetalKit",
|
||||||
],
|
],
|
||||||
tags = ["ios"],
|
tags = ["ios"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"inference_calculator_interface",
|
"inference_calculator_interface",
|
||||||
|
"//mediapipe/framework/formats:tensor",
|
||||||
"//mediapipe/gpu:MPPMetalHelper",
|
"//mediapipe/gpu:MPPMetalHelper",
|
||||||
"//mediapipe/gpu:MPPMetalUtil",
|
"//mediapipe/gpu:MPPMetalUtil",
|
||||||
"//mediapipe/gpu:gpu_buffer",
|
"//mediapipe/gpu:gpu_buffer",
|
||||||
@@ -530,15 +491,6 @@ cc_library(
|
|||||||
cc_library(
|
cc_library(
|
||||||
name = "inference_runner",
|
name = "inference_runner",
|
||||||
hdrs = ["inference_runner.h"],
|
hdrs = ["inference_runner.h"],
|
||||||
copts = select({
|
|
||||||
# TODO: fix tensor.h not to require this, if possible
|
|
||||||
"//mediapipe:apple": [
|
|
||||||
"-x objective-c++",
|
|
||||||
"-fobjc-arc", # enable reference-counting
|
|
||||||
],
|
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_context",
|
"//mediapipe/framework:calculator_context",
|
||||||
"//mediapipe/framework/formats:tensor",
|
"//mediapipe/framework/formats:tensor",
|
||||||
@@ -546,31 +498,33 @@ cc_library(
|
|||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
cc_library(
|
cc_library_with_tflite(
|
||||||
|
name = "tflite_delegate_ptr",
|
||||||
|
hdrs = ["tflite_delegate_ptr.h"],
|
||||||
|
tflite_deps = [
|
||||||
|
"@org_tensorflow//tensorflow/lite/core/shims:c_api_types",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
cc_library_with_tflite(
|
||||||
name = "inference_interpreter_delegate_runner",
|
name = "inference_interpreter_delegate_runner",
|
||||||
srcs = ["inference_interpreter_delegate_runner.cc"],
|
srcs = ["inference_interpreter_delegate_runner.cc"],
|
||||||
hdrs = ["inference_interpreter_delegate_runner.h"],
|
hdrs = ["inference_interpreter_delegate_runner.h"],
|
||||||
copts = select({
|
tflite_deps = [
|
||||||
# TODO: fix tensor.h not to require this, if possible
|
":tflite_delegate_ptr",
|
||||||
"//mediapipe:apple": [
|
"//mediapipe/util/tflite:tflite_model_loader",
|
||||||
"-x objective-c++",
|
"@org_tensorflow//tensorflow/lite/core/shims:c_api_types",
|
||||||
"-fobjc-arc", # enable reference-counting
|
"@org_tensorflow//tensorflow/lite/core/shims:framework_stable",
|
||||||
],
|
],
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_runner",
|
":inference_runner",
|
||||||
"//mediapipe/framework:mediapipe_profiling",
|
"//mediapipe/framework:mediapipe_profiling",
|
||||||
"//mediapipe/framework/api2:packet",
|
"//mediapipe/framework/api2:packet",
|
||||||
"//mediapipe/framework/formats:tensor",
|
"//mediapipe/framework/formats:tensor",
|
||||||
"//mediapipe/framework/port:ret_check",
|
"//mediapipe/framework/port:ret_check",
|
||||||
"//mediapipe/util/tflite:tflite_model_loader",
|
|
||||||
"@com_google_absl//absl/status",
|
"@com_google_absl//absl/status",
|
||||||
"@com_google_absl//absl/status:statusor",
|
"@com_google_absl//absl/status:statusor",
|
||||||
"@org_tensorflow//tensorflow/lite:framework_stable",
|
|
||||||
"@org_tensorflow//tensorflow/lite:string_util",
|
"@org_tensorflow//tensorflow/lite:string_util",
|
||||||
"@org_tensorflow//tensorflow/lite/c:c_api_types",
|
|
||||||
"@org_tensorflow//tensorflow/lite/core/api:op_resolver",
|
"@org_tensorflow//tensorflow/lite/core/api:op_resolver",
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
@@ -580,15 +534,6 @@ cc_library(
|
|||||||
srcs = [
|
srcs = [
|
||||||
"inference_calculator_cpu.cc",
|
"inference_calculator_cpu.cc",
|
||||||
],
|
],
|
||||||
copts = select({
|
|
||||||
# TODO: fix tensor.h not to require this, if possible
|
|
||||||
"//mediapipe:apple": [
|
|
||||||
"-x objective-c++",
|
|
||||||
"-fobjc-arc", # enable reference-counting
|
|
||||||
],
|
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_calculator_interface",
|
":inference_calculator_interface",
|
||||||
":inference_calculator_utils",
|
":inference_calculator_utils",
|
||||||
@@ -597,9 +542,9 @@ cc_library(
|
|||||||
"@com_google_absl//absl/memory",
|
"@com_google_absl//absl/memory",
|
||||||
"@com_google_absl//absl/status",
|
"@com_google_absl//absl/status",
|
||||||
"@com_google_absl//absl/status:statusor",
|
"@com_google_absl//absl/status:statusor",
|
||||||
|
"@org_tensorflow//tensorflow/lite/core/shims:c_api_types",
|
||||||
|
"@org_tensorflow//tensorflow/lite/core/shims:framework_stable",
|
||||||
"@org_tensorflow//tensorflow/lite/delegates/xnnpack:xnnpack_delegate",
|
"@org_tensorflow//tensorflow/lite/delegates/xnnpack:xnnpack_delegate",
|
||||||
"@org_tensorflow//tensorflow/lite:framework_stable",
|
|
||||||
"@org_tensorflow//tensorflow/lite/c:c_api_types",
|
|
||||||
] + select({
|
] + select({
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
"//mediapipe:android": ["@org_tensorflow//tensorflow/lite/delegates/nnapi:nnapi_delegate"],
|
"//mediapipe:android": ["@org_tensorflow//tensorflow/lite/delegates/nnapi:nnapi_delegate"],
|
||||||
@@ -627,15 +572,6 @@ cc_library(
|
|||||||
srcs = [
|
srcs = [
|
||||||
"inference_calculator_xnnpack.cc",
|
"inference_calculator_xnnpack.cc",
|
||||||
],
|
],
|
||||||
copts = select({
|
|
||||||
# TODO: fix tensor.h not to require this, if possible
|
|
||||||
"//mediapipe:apple": [
|
|
||||||
"-x objective-c++",
|
|
||||||
"-fobjc-arc", # enable reference-counting
|
|
||||||
],
|
|
||||||
"//conditions:default": [],
|
|
||||||
}),
|
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_calculator_interface",
|
":inference_calculator_interface",
|
||||||
":inference_calculator_utils",
|
":inference_calculator_utils",
|
||||||
@@ -651,7 +587,6 @@ cc_library(
|
|||||||
|
|
||||||
cc_library(
|
cc_library(
|
||||||
name = "inference_calculator_gl_if_compute_shader_available",
|
name = "inference_calculator_gl_if_compute_shader_available",
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = selects.with_or({
|
deps = selects.with_or({
|
||||||
":compute_shader_unavailable": [],
|
":compute_shader_unavailable": [],
|
||||||
"//conditions:default": [
|
"//conditions:default": [
|
||||||
@@ -667,13 +602,12 @@ cc_library(
|
|||||||
# inference_calculator_interface.
|
# inference_calculator_interface.
|
||||||
cc_library(
|
cc_library(
|
||||||
name = "inference_calculator",
|
name = "inference_calculator",
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":inference_calculator_interface",
|
":inference_calculator_interface",
|
||||||
":inference_calculator_cpu",
|
":inference_calculator_cpu",
|
||||||
] + select({
|
] + select({
|
||||||
"//conditions:default": [":inference_calculator_gl_if_compute_shader_available"],
|
"//conditions:default": [":inference_calculator_gl_if_compute_shader_available"],
|
||||||
"//mediapipe:ios": [":inference_calculator_metal"],
|
":platform_ios_with_gpu": [":inference_calculator_metal"],
|
||||||
}),
|
}),
|
||||||
alwayslink = 1,
|
alwayslink = 1,
|
||||||
)
|
)
|
||||||
@@ -681,7 +615,6 @@ cc_library(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensor_converter_calculator_proto",
|
name = "tensor_converter_calculator_proto",
|
||||||
srcs = ["tensor_converter_calculator.proto"],
|
srcs = ["tensor_converter_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -706,7 +639,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensor_converter_calculator_cc_proto",
|
":tensor_converter_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -725,6 +657,7 @@ cc_library(
|
|||||||
|
|
||||||
cc_library(
|
cc_library(
|
||||||
name = "tensor_converter_calculator_gpu_deps",
|
name = "tensor_converter_calculator_gpu_deps",
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = select({
|
deps = select({
|
||||||
"//mediapipe:android": [
|
"//mediapipe:android": [
|
||||||
"//mediapipe/gpu:gl_calculator_helper",
|
"//mediapipe/gpu:gl_calculator_helper",
|
||||||
@@ -769,7 +702,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensors_to_detections_calculator_proto",
|
name = "tensors_to_detections_calculator_proto",
|
||||||
srcs = ["tensors_to_detections_calculator.proto"],
|
srcs = ["tensors_to_detections_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -794,7 +726,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensors_to_detections_calculator_cc_proto",
|
":tensors_to_detections_calculator_cc_proto",
|
||||||
"//mediapipe/framework/formats:detection_cc_proto",
|
"//mediapipe/framework/formats:detection_cc_proto",
|
||||||
@@ -817,6 +748,7 @@ cc_library(
|
|||||||
|
|
||||||
cc_library(
|
cc_library(
|
||||||
name = "tensors_to_detections_calculator_gpu_deps",
|
name = "tensors_to_detections_calculator_gpu_deps",
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = select({
|
deps = select({
|
||||||
"//mediapipe:ios": [
|
"//mediapipe:ios": [
|
||||||
"//mediapipe/gpu:MPPMetalUtil",
|
"//mediapipe/gpu:MPPMetalUtil",
|
||||||
@@ -832,7 +764,6 @@ cc_library(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensors_to_landmarks_calculator_proto",
|
name = "tensors_to_landmarks_calculator_proto",
|
||||||
srcs = ["tensors_to_landmarks_calculator.proto"],
|
srcs = ["tensors_to_landmarks_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -849,7 +780,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensors_to_landmarks_calculator_cc_proto",
|
":tensors_to_landmarks_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -864,7 +794,6 @@ cc_library(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "landmarks_to_tensor_calculator_proto",
|
name = "landmarks_to_tensor_calculator_proto",
|
||||||
srcs = ["landmarks_to_tensor_calculator.proto"],
|
srcs = ["landmarks_to_tensor_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -882,7 +811,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":landmarks_to_tensor_calculator_cc_proto",
|
":landmarks_to_tensor_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -915,7 +843,6 @@ cc_test(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensors_to_floats_calculator_proto",
|
name = "tensors_to_floats_calculator_proto",
|
||||||
srcs = ["tensors_to_floats_calculator.proto"],
|
srcs = ["tensors_to_floats_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -932,7 +859,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensors_to_floats_calculator_cc_proto",
|
":tensors_to_floats_calculator_cc_proto",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
@@ -970,7 +896,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensors_to_classification_calculator_cc_proto",
|
":tensors_to_classification_calculator_cc_proto",
|
||||||
"@com_google_absl//absl/container:node_hash_map",
|
"@com_google_absl//absl/container:node_hash_map",
|
||||||
@@ -1001,7 +926,6 @@ cc_library(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensors_to_classification_calculator_proto",
|
name = "tensors_to_classification_calculator_proto",
|
||||||
srcs = ["tensors_to_classification_calculator.proto"],
|
srcs = ["tensors_to_classification_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -1039,7 +963,6 @@ cc_library(
|
|||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
features = ["-layering_check"], # allow depending on image_to_tensor_calculator_gpu_deps
|
features = ["-layering_check"], # allow depending on image_to_tensor_calculator_gpu_deps
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":image_to_tensor_calculator_cc_proto",
|
":image_to_tensor_calculator_cc_proto",
|
||||||
":image_to_tensor_converter",
|
":image_to_tensor_converter",
|
||||||
@@ -1068,6 +991,7 @@ cc_library(
|
|||||||
|
|
||||||
cc_library(
|
cc_library(
|
||||||
name = "image_to_tensor_calculator_gpu_deps",
|
name = "image_to_tensor_calculator_gpu_deps",
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = selects.with_or({
|
deps = selects.with_or({
|
||||||
"//mediapipe:android": [
|
"//mediapipe:android": [
|
||||||
":image_to_tensor_converter_gl_buffer",
|
":image_to_tensor_converter_gl_buffer",
|
||||||
@@ -1091,7 +1015,6 @@ cc_library(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "image_to_tensor_calculator_proto",
|
name = "image_to_tensor_calculator_proto",
|
||||||
srcs = ["image_to_tensor_calculator.proto"],
|
srcs = ["image_to_tensor_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -1141,6 +1064,7 @@ cc_test(
|
|||||||
"@com_google_absl//absl/flags:flag",
|
"@com_google_absl//absl/flags:flag",
|
||||||
"@com_google_absl//absl/memory",
|
"@com_google_absl//absl/memory",
|
||||||
"@com_google_absl//absl/strings",
|
"@com_google_absl//absl/strings",
|
||||||
|
"@com_google_absl//absl/strings:str_format",
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -1154,7 +1078,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":image_to_tensor_utils",
|
":image_to_tensor_utils",
|
||||||
"//mediapipe/framework/formats:image",
|
"//mediapipe/framework/formats:image",
|
||||||
@@ -1174,7 +1097,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":image_to_tensor_converter",
|
":image_to_tensor_converter",
|
||||||
":image_to_tensor_utils",
|
":image_to_tensor_utils",
|
||||||
@@ -1194,6 +1116,7 @@ cc_library(
|
|||||||
name = "image_to_tensor_converter_gl_buffer",
|
name = "image_to_tensor_converter_gl_buffer",
|
||||||
srcs = ["image_to_tensor_converter_gl_buffer.cc"],
|
srcs = ["image_to_tensor_converter_gl_buffer.cc"],
|
||||||
hdrs = ["image_to_tensor_converter_gl_buffer.h"],
|
hdrs = ["image_to_tensor_converter_gl_buffer.h"],
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = ["//mediapipe/framework:port"] + selects.with_or({
|
deps = ["//mediapipe/framework:port"] + selects.with_or({
|
||||||
"//mediapipe:apple": [],
|
"//mediapipe:apple": [],
|
||||||
"//conditions:default": [
|
"//conditions:default": [
|
||||||
@@ -1227,6 +1150,7 @@ cc_library(
|
|||||||
name = "image_to_tensor_converter_gl_texture",
|
name = "image_to_tensor_converter_gl_texture",
|
||||||
srcs = ["image_to_tensor_converter_gl_texture.cc"],
|
srcs = ["image_to_tensor_converter_gl_texture.cc"],
|
||||||
hdrs = ["image_to_tensor_converter_gl_texture.h"],
|
hdrs = ["image_to_tensor_converter_gl_texture.h"],
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = ["//mediapipe/framework:port"] + select({
|
deps = ["//mediapipe/framework:port"] + select({
|
||||||
"//mediapipe/gpu:disable_gpu": [],
|
"//mediapipe/gpu:disable_gpu": [],
|
||||||
"//conditions:default": [
|
"//conditions:default": [
|
||||||
@@ -1251,6 +1175,7 @@ cc_library(
|
|||||||
name = "image_to_tensor_converter_gl_utils",
|
name = "image_to_tensor_converter_gl_utils",
|
||||||
srcs = ["image_to_tensor_converter_gl_utils.cc"],
|
srcs = ["image_to_tensor_converter_gl_utils.cc"],
|
||||||
hdrs = ["image_to_tensor_converter_gl_utils.h"],
|
hdrs = ["image_to_tensor_converter_gl_utils.h"],
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = ["//mediapipe/framework:port"] + select({
|
deps = ["//mediapipe/framework:port"] + select({
|
||||||
"//mediapipe/gpu:disable_gpu": [],
|
"//mediapipe/gpu:disable_gpu": [],
|
||||||
"//conditions:default": [
|
"//conditions:default": [
|
||||||
@@ -1280,6 +1205,7 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
|
visibility = ["//visibility:private"],
|
||||||
deps = ["//mediapipe/framework:port"] + select({
|
deps = ["//mediapipe/framework:port"] + select({
|
||||||
"//mediapipe:apple": [
|
"//mediapipe:apple": [
|
||||||
":image_to_tensor_converter",
|
":image_to_tensor_converter",
|
||||||
@@ -1311,7 +1237,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":image_to_tensor_calculator_cc_proto",
|
":image_to_tensor_calculator_cc_proto",
|
||||||
"@com_google_absl//absl/status",
|
"@com_google_absl//absl/status",
|
||||||
@@ -1354,7 +1279,6 @@ selects.config_setting_group(
|
|||||||
mediapipe_proto_library(
|
mediapipe_proto_library(
|
||||||
name = "tensors_to_segmentation_calculator_proto",
|
name = "tensors_to_segmentation_calculator_proto",
|
||||||
srcs = ["tensors_to_segmentation_calculator.proto"],
|
srcs = ["tensors_to_segmentation_calculator.proto"],
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_options_proto",
|
"//mediapipe/framework:calculator_options_proto",
|
||||||
"//mediapipe/framework:calculator_proto",
|
"//mediapipe/framework:calculator_proto",
|
||||||
@@ -1372,7 +1296,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
":tensors_to_segmentation_calculator_cc_proto",
|
":tensors_to_segmentation_calculator_cc_proto",
|
||||||
"@com_google_absl//absl/strings:str_format",
|
"@com_google_absl//absl/strings:str_format",
|
||||||
@@ -1430,7 +1353,6 @@ cc_library(
|
|||||||
],
|
],
|
||||||
"//conditions:default": [],
|
"//conditions:default": [],
|
||||||
}),
|
}),
|
||||||
visibility = ["//visibility:public"],
|
|
||||||
deps = [
|
deps = [
|
||||||
"//mediapipe/framework:calculator_context",
|
"//mediapipe/framework:calculator_context",
|
||||||
"//mediapipe/framework:calculator_framework",
|
"//mediapipe/framework:calculator_framework",
|
||||||
|
|||||||
@@ -203,6 +203,7 @@ class AudioToTensorCalculator : public Node {
|
|||||||
std::unique_ptr<audio_dsp::QResampler<float>> resampler_;
|
std::unique_ptr<audio_dsp::QResampler<float>> resampler_;
|
||||||
Matrix sample_buffer_;
|
Matrix sample_buffer_;
|
||||||
int processed_buffer_cols_ = 0;
|
int processed_buffer_cols_ = 0;
|
||||||
|
double gain_ = 1.0;
|
||||||
|
|
||||||
// The internal state of the FFT library.
|
// The internal state of the FFT library.
|
||||||
PFFFT_Setup* fft_state_ = nullptr;
|
PFFFT_Setup* fft_state_ = nullptr;
|
||||||
@@ -278,7 +279,9 @@ absl::Status AudioToTensorCalculator::Open(CalculatorContext* cc) {
|
|||||||
padding_samples_after_ = options.padding_samples_after();
|
padding_samples_after_ = options.padding_samples_after();
|
||||||
dft_tensor_format_ = options.dft_tensor_format();
|
dft_tensor_format_ = options.dft_tensor_format();
|
||||||
flush_mode_ = options.flush_mode();
|
flush_mode_ = options.flush_mode();
|
||||||
|
if (options.has_volume_gain_db()) {
|
||||||
|
gain_ = pow(10, options.volume_gain_db() / 20.0);
|
||||||
|
}
|
||||||
RET_CHECK(kAudioSampleRateIn(cc).IsConnected() ^
|
RET_CHECK(kAudioSampleRateIn(cc).IsConnected() ^
|
||||||
!kAudioIn(cc).Header().IsEmpty())
|
!kAudioIn(cc).Header().IsEmpty())
|
||||||
<< "Must either specify the time series header of the \"AUDIO\" stream "
|
<< "Must either specify the time series header of the \"AUDIO\" stream "
|
||||||
@@ -344,6 +347,10 @@ absl::Status AudioToTensorCalculator::Process(CalculatorContext* cc) {
|
|||||||
const Matrix& input = channels_match ? input_frame
|
const Matrix& input = channels_match ? input_frame
|
||||||
// Mono mixdown.
|
// Mono mixdown.
|
||||||
: input_frame.colwise().mean();
|
: input_frame.colwise().mean();
|
||||||
|
if (gain_ != 1.0) {
|
||||||
|
return stream_mode_ ? ProcessStreamingData(cc, input * gain_)
|
||||||
|
: ProcessNonStreamingData(cc, input * gain_);
|
||||||
|
}
|
||||||
return stream_mode_ ? ProcessStreamingData(cc, input)
|
return stream_mode_ ? ProcessStreamingData(cc, input)
|
||||||
: ProcessNonStreamingData(cc, input);
|
: ProcessNonStreamingData(cc, input);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -81,4 +81,8 @@ message AudioToTensorCalculatorOptions {
|
|||||||
WITH_DC_AND_NYQUIST = 3;
|
WITH_DC_AND_NYQUIST = 3;
|
||||||
}
|
}
|
||||||
optional DftTensorFormat dft_tensor_format = 11 [default = WITH_NYQUIST];
|
optional DftTensorFormat dft_tensor_format = 11 [default = WITH_NYQUIST];
|
||||||
|
|
||||||
|
// The volume gain, measured in dB.
|
||||||
|
// Scale the input audio amplitude by 10^(volume_gain_db/20).
|
||||||
|
optional double volume_gain_db = 12;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -195,8 +195,9 @@ class ImageToTensorCalculator : public Node {
|
|||||||
#endif // MEDIAPIPE_DISABLE_GPU
|
#endif // MEDIAPIPE_DISABLE_GPU
|
||||||
|
|
||||||
RotatedRect roi = GetRoi(image->width(), image->height(), norm_rect);
|
RotatedRect roi = GetRoi(image->width(), image->height(), norm_rect);
|
||||||
ASSIGN_OR_RETURN(auto padding, PadRoi(options_.output_tensor_width(),
|
const int tensor_width = params_.output_width.value_or(image->width());
|
||||||
options_.output_tensor_height(),
|
const int tensor_height = params_.output_height.value_or(image->height());
|
||||||
|
ASSIGN_OR_RETURN(auto padding, PadRoi(tensor_width, tensor_height,
|
||||||
options_.keep_aspect_ratio(), &roi));
|
options_.keep_aspect_ratio(), &roi));
|
||||||
if (kOutLetterboxPadding(cc).IsConnected()) {
|
if (kOutLetterboxPadding(cc).IsConnected()) {
|
||||||
kOutLetterboxPadding(cc).Send(padding);
|
kOutLetterboxPadding(cc).Send(padding);
|
||||||
@@ -214,9 +215,8 @@ class ImageToTensorCalculator : public Node {
|
|||||||
|
|
||||||
Tensor::ElementType output_tensor_type =
|
Tensor::ElementType output_tensor_type =
|
||||||
GetOutputTensorType(image->UsesGpu(), params_);
|
GetOutputTensorType(image->UsesGpu(), params_);
|
||||||
Tensor tensor(output_tensor_type,
|
Tensor tensor(output_tensor_type, {1, tensor_height, tensor_width,
|
||||||
{1, params_.output_height, params_.output_width,
|
GetNumOutputChannels(*image)});
|
||||||
GetNumOutputChannels(*image)});
|
|
||||||
MP_RETURN_IF_ERROR((image->UsesGpu() ? gpu_converter_ : cpu_converter_)
|
MP_RETURN_IF_ERROR((image->UsesGpu() ? gpu_converter_ : cpu_converter_)
|
||||||
->Convert(*image, roi, params_.range_min,
|
->Convert(*image, roi, params_.range_min,
|
||||||
params_.range_max,
|
params_.range_max,
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user