112 lines
3.0 KiB
Markdown
112 lines
3.0 KiB
Markdown
# Experimental new APIs
|
|
|
|
This directory defines new APIs for MediaPipe:
|
|
|
|
- Node API, an update to the Calculator API for defining MediaPipe components.
|
|
- Builder API, for assembling CalculatorGraphConfigs with C++, as an alternative
|
|
to using the proto API directly.
|
|
|
|
The code is working, and the new APIs interoperate fully with the existing
|
|
framework code. They are considered a work in progress, but are being released
|
|
now so we can begin adopting them in our calculators.
|
|
|
|
Developers are welcome to try out these APIs as early adopters, but should
|
|
expect breaking changes. The placement of this code under the `mediapipe::api2`
|
|
namespace is not final.
|
|
|
|
## Node API
|
|
|
|
This API can be used to define calculators. It is designed to be more type-safe
|
|
and less verbose than the original API.
|
|
|
|
Input/output ports (streams and side packets) can now be declared as typed
|
|
constants, instead of using plain strings for access.
|
|
|
|
For example, instead of
|
|
|
|
```
|
|
constexpr char kSelectTag[] = "SELECT";
|
|
if (cc->Inputs().HasTag(kSelectTag)) {
|
|
cc->Inputs().Tag(kSelectTag).Set<int>();
|
|
}
|
|
```
|
|
|
|
you can write
|
|
|
|
```
|
|
static constexpr Input<int>::Optional kSelect{"SELECT"};
|
|
```
|
|
|
|
Instead of setting up the contract procedurally in `GetContract`, add ports to
|
|
the contract declaratively, as follows:
|
|
|
|
```
|
|
MEDIAPIPE_NODE_CONTRACT(kInput, kOutput);
|
|
```
|
|
|
|
To access an input in Process, instead of
|
|
|
|
```
|
|
int select = cc->Inputs().Tag(kSelectTag).Get<int>();
|
|
```
|
|
|
|
write
|
|
|
|
```
|
|
int select = kSelectTag(cc).Get(); // alternative: *kSelectTag(cc)
|
|
```
|
|
|
|
Sets of multiple ports can be declared with `::Multiple`. Note, also, that a tag
|
|
string must always be provided when declaring a port; use `""` for untagged
|
|
ports. For example:
|
|
|
|
|
|
```
|
|
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
|
|
cc->Inputs().Index(i).SetAny();
|
|
}
|
|
```
|
|
|
|
becomes
|
|
|
|
```
|
|
static constexpr Input<AnyType>::Multiple kIn{""};
|
|
```
|
|
|
|
For output ports, the payload can be passed directly to the `Send` method. For
|
|
example, instead of
|
|
|
|
```
|
|
cc->Outputs().Index(0).Add(
|
|
new std::pair<Packet, Packet>(cc->Inputs().Index(0).Value(),
|
|
cc->Inputs().Index(1).Value()),
|
|
cc->InputTimestamp());
|
|
```
|
|
|
|
you can write
|
|
|
|
```
|
|
kPair(cc).Send({kIn(cc)[0].packet(), kIn(cc)[1].packet()});
|
|
```
|
|
|
|
The input timestamp is propagated to the outputs by default. If your calculator
|
|
wants to alter timestamps, it must add a `TimestampChange` entry to its contract
|
|
declaration. For example:
|
|
|
|
```
|
|
MEDIAPIPE_NODE_CONTRACT(kMain, kLoop, kPrevLoop,
|
|
StreamHandler("ImmediateInputStreamHandler"),
|
|
TimestampChange::Arbitrary());
|
|
```
|
|
|
|
Several calculators in
|
|
[`calculators/core`](https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core) and
|
|
[`calculators/tensor`](https://github.com/google/mediapipe/tree/master/mediapipe/calculators/tensor)
|
|
have been updated to use this API. Reference them for more examples.
|
|
|
|
More complete documentation will be provided in the future.
|
|
|
|
## Builder API
|
|
|
|
Documentation will be provided in the future.
|