NIFI-16204 Add KafkaProvenanceReportingTask

[thomews]

Summary

NIFI-16204

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for proposing this new feature @thomews.

Before providing evaluating the code itself, please review the pull request requirements and push an update to the branch with signed commits.

[exceptionfactory]

Thanks for making the initial updates @thomews.

On a general review, I'm not sure that this is an optimal candidate for inclusion in the project. The properties provide a large number of configuration options on one hand, which can be helpful, but also highlight the complex nature of the implementation. There are also a number of opinionated selections in the implementation, such as field names and JSON formatting. These may be optimal for a number of use cases, but they also highlight the challenge of supporting a generic capability.

There are other issues, like the use of the Jakarta JSON API, versus Jackson, and other more minor implementation concerns.

[thomews]

Thanks for the feedback @exceptionfactory. We have already addressed the JSON API concern and switched from Jakarta JSON to Jackson.

Regarding the configuration complexity, one possible direction would be to move the Kafka connection and Record Writer configuration into a separate KafkaRecordSink Controller Service and have the Reporting Task consume a RecordSinkService. This would simplify the Reporting Task configuration, while also separating the provenance reporting concerns from the Kafka-specific configuration.

The question would be whether the Reporting Task should become destination-agnostic, since any RecordSinkService could then be selected. Do you think this would be a better fit for the concerns you raised around configuration complexity?