How to specify "output_format" when create ingest pipeline with DateProcessor using opensearch-java client

Question

Our use case is like this: client will send json containing a field:

{
  "start_time": "2025-03-12T01:33:45.377Z",
  ...
}

We need to convert or parse this start_time field to epoch millis like this:

  "start_time": 1741743225377,

It was easy with just 1 http request using curl:

curl -XPUT "$url/_ingest/pipeline/date-string-to-millis" -ku admin:$pw -H 'Content-Type: application/json' -d'
{
  "description": "Convert date string to epoch millis",
  "processors": [
    {
      "date": {
        "field": "start_time",
        "formats": ["date_time"],
        "output_format": "epoch_millis",
        "target_field": "start_time"
      }
    },
    {
        "convert": {
            "field": "start_time",
            "type": "long"
        }
    }
  ]
}'

However when we try to create this ingest pipeline on opensearch-java client, there is no output_format property. I also check elasticsearch-java client and there no output_format as well, is this intended? How can I create this pipeline on opensearch-java client? If not possible, is there any other way we can parse that date string to epoch_millis?

Answer 1

Creating an ingest pipeline is a one time process per pipeline. So simply create the pipeline, and use it during ingestion. You have two options for that.

1. Define ingest pipeline on indexing request header

2. Define ingest pipeline as a default pipeline on index settings.

Define ingest pipeline on indexing request header
POST my-data-stream/_doc?pipeline=my-pipeline
{
  "@timestamp": "2099-03-07T11:04:05.000Z",
  "my-keyword-field": "foo"
}

PUT your_index_name/_settings
{
  "index.default_pipeline": "my-pipeline"
}

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html

is there any other way we can parse that date string to epoch_millis?

Yes, the easier way is to define the field type like the following.

PUT your_new_index_name
{
  "mappings": {
    "properties": {
      "start_time": {
        "type": "date",
        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ||epoch_millis"
      }
    }
  }
}