Fluentbit logging based on string in filename [closed]

Question

I'm currently sending logs from Fluentbit in my Kubernetes cluster to Opensearch to one pipeline and this is working fine. However, I'm trying to send logs to Opensearch but depending on a string within the log file name it will output to a different pipeline.

For example, my logfile names contain the words "dev" and "tst" and depending on whether it's "tst" or "dev" I want it to OUTPUT to the Opensearch pipeline that contains the word tst or dev.

Is there a way in fluentbit that this can be done. I'm aware that Tag_Regex can grab fields from the logfile name using regex which I've created the regex for this and should be able to grab this out but wasn't sure how to then apply this in the OUTPUT so that it essentially says if the Tag_Regex matches "dev" use in this OUTPUT or if it matches "tst" using the other OUTPUT.

Answer 1

Yes, you can use MATCH with regex, somehow like that

[INPUT]
    Name              tail
    Path              /var/log/myapp-*.log
    Tag               myapp.*
    Tag_Regex         ^myapp-(?P<env>.*(dev|tst).*)

[FILTER]
    Name              record_modifier
    Match             *
    Record            env ${env}

[OUTPUT]
    Name            opensearch
    Match           myapp.*dev*
    Host            opensearch.dev.local
    Port            9200
    Pipeline        dev-pipeline
    Index           dev-logs

[OUTPUT]
    Name            opensearch
    Match           myapp.*tst*
    Host            opensearch.tst.local
    Port            9200
    Pipeline        tst-pipeline
    Index           tst-logs