Skip to content

[#17939] Implement FFT table function for table model#18006

Open
DaZuiZui wants to merge 3 commits into
apache:masterfrom
DaZuiZui:feat/fft-table-v1-17939
Open

[#17939] Implement FFT table function for table model#18006
DaZuiZui wants to merge 3 commits into
apache:masterfrom
DaZuiZui:feat/fft-table-v1-17939

Conversation

@DaZuiZui

Copy link
Copy Markdown
Contributor

Description

FFT table-valued function

This PR implements the v1 built-in FFT / fft table-valued function for the table model.

The function accepts a required DATA table argument with set semantics and ORDER BY time, plus optional SAMPLE_INTERVAL, N, and NORM arguments. It transforms all numeric input columns except time and partition columns, emits the full complex spectrum, and returns frequency_index, frequency, and <column>_real / <column>_imag columns. When partition columns exist, they are kept at the front of the output schema.

The FFT implementation uses JTransforms DoubleFFT_1D. N supports default partition length, truncation, and zero padding. NORM supports backward, forward, and ortho. SAMPLE_INTERVAL can be supplied as a duration literal; otherwise the interval is inferred from the partition time range.

Analyzer, planner, and validation

The function is registered as a built-in table function and gets analyzer/planner handling similar to M4 where needed so the output column order matches the table-function contract.

Validation covers:

  • DATA must have ORDER BY time in ascending order.
  • SAMPLE_INTERVAL, if provided, must be positive.
  • N, if provided, must be positive.
  • NORM must be one of backward, forward, or ortho.
  • At least one numeric transform column is required.
  • Runtime data must be strictly increasing by time and numeric transform values must not be null.
  • If SAMPLE_INTERVAL is omitted, at least two rows are required to infer it.

Tests

This PR adds focused unit coverage for the FFT processor, analyzer/planner coverage for built-in table-function recognition and schema handling, and a table-model IT covering normal execution and failure cases.

Tested locally with:

JAVA_HOME=$(/usr/libexec/java_home -v 21) ./mvnw test -pl iotdb-core/node-commons -am -Dtest=FFTTableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
JAVA_HOME=$(/usr/libexec/java_home -v 21) ./mvnw test -pl iotdb-core/datanode -am -Dtest=TableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
JAVA_HOME=$(/usr/libexec/java_home -v 21) ./mvnw verify -DskipUTs -Dit.test=IoTDBFFTTableFunctionIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -PTableSimpleIT -P with-integration-tests

This PR has:

  • been self-reviewed.
  • added Javadocs for most classes and all non-trivial methods.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.commons.udf.builtin.relational.tvf.FFTTableFunction
  • org.apache.iotdb.commons.queryengine.plan.relational.function.TableBuiltinTableFunction
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.StatementAnalyzer
  • org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlanner
  • org.apache.iotdb.commons.udf.builtin.relational.tvf.FFTTableFunctionTest
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.TableFunctionTest
  • org.apache.iotdb.relational.it.db.it.IoTDBFFTTableFunctionIT

Related to #17939.

@DaZuiZui DaZuiZui force-pushed the feat/fft-table-v1-17939 branch from dc535ca to 2faa54b Compare June 23, 2026 06:28
@DaZuiZui

Copy link
Copy Markdown
Contributor Author

Updated this PR for the latest review comments:

  • Capped FFT transform length at 65,536 and added a spectrum buffer-size guard for multi-column input.
  • Validate evenly spaced timestamps when SAMPLE_INTERVAL is omitted, and validate input gaps against SAMPLE_INTERVAL when it is provided.
  • Added UT/IT coverage for oversized N, irregular timestamps, and explicit interval mismatch.

Tests run locally with JDK 21:

  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode,integration-test -P with-integration-tests
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw test -pl iotdb-core/node-commons -am -Dtest=FFTTableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw test -pl iotdb-core/datanode -am -Dtest=TableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw verify -DskipUTs -Dit.test=IoTDBFFTTableFunctionIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -PTableSimpleIT -P with-integration-tests
@DaZuiZui

Copy link
Copy Markdown
Contributor Author

Follow-up update:

  • Added node-commons unit coverage for actual FFT spectrum output, zero padding, N truncation, and the explicit failure behavior for invalid rows even when they are beyond the truncated N range.
  • Added a release-note level entry documenting SAMPLE_INTERVAL, default/capped N, null rejection, and timestamp spacing behavior.

Pushed as a normal follow-up commit (no force push).

Tests run locally with JDK 21:

  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw spotless:apply -pl iotdb-core/node-commons
  • JAVA_HOME=$( /usr/libexec/java_home -v 21 ) ./mvnw test -pl iotdb-core/node-commons -am -Dtest=FFTTableFunctionTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant