Return to Answer

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

logrotate on the other hand, checks the logfiles when it is run, and usually systems are setup to run logrotate (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/logrotate.d and yes, it usually runs as root.

Edit: @mike rodent

logrotate is run (by cron or by hand) and then checks the logfile statistics as seen in the file-system, it then triggers a rotation depending on it's configuration.

rotatelogs reads it's standard input continuously appending to a logfile, then when the time or the size of logfile reaches a trigger-point, it closes the current logfile, creates a new logfile (appropriately named), then continues to append to the new logfile.

This has a bearing on how you start the program which generates the output, it needs to write to standard output and then you pipe this into rotatelogs.

eg

while true ; do date ; sleep 30 ; done | rotatelogs -n 10 /home/mike/tmp/qqq.log 60

Will give you a circular rotation through 10 files, cut every 60 seconds

Using rclone, I would expect --log-file "/dev/stdout" would make it write to standard output.

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

logrotate on the other hand, checks the logfiles when it is run, and usually systems are setup to run logrotate (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/logrotate.d and yes, it usually runs as root.

Edit: @mike rodent

logrotate is run (by cron or by hand) and then checks the logfile statistics as seen in the file-system, it then triggers a rotation depending on it's configuration.

rotatelogs reads it's standard input continuously appending to a logfile, then when the time or the size of logfile reaches a trigger-point, it closes the current logfile, creates a new logfile (appropriately named), then continues to append to the new logfile.

This has a bearing on how you start the program which generates the output, it needs to write to standard output and then you pipe this into rotatelogs.

eg

while true ; do date ; sleep 30 ; done | rotatelogs -n 10 /home/mike/tmp/qqq.log 60

Will give you a circular rotation through 10 files, cut every 60 seconds

Using rclone, I would expect --log-file "/dev/stdout" would make it write to standard output.

[edit 2] @mike rodent

I'm not familiar with rclone, whether it writes to stdout or stderr and what if anything it writes to either, but there are ways around this, even if the --log-file=/dev/stdout didn't work.

This StackOverflow Q/Answer, amongst others explains redirection and piping. But to summarise, it's possible to redirect stderr only or merged with stdout into a pipe.

Bash example:

Merge stderr with stdout into pipe

FirstCommand 2>&1 | OtherCommand

Only stderr into pipe, stdout to otherfile (which could be /dev/null or even just a hyphen - which closes stdout).

FirstCommand 2>&1 1>otherfile.log | otherCommand

These is examples use the older bash syntax, bash v4 has a modern less verbose variant.

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

rotatelogs on the other hand, checks the logfiles when it is run, and usually systems are setup to run rotatelogs (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/rotatelogs.d and yes it runs as root.

Edit: @mike rodent

logrotate is run (by cron or by hand) and then checks the logfile statistics as seen in the file-system, it then triggers a rotation depending on it's configuration.

rotatelogs reads it's standard input continuously appending to a logfile, then when the time or the size of logfile reaches a trigger-point, it closes the current logfile, creates a new logfile (appropriately named), then continues to append to the new logfile.

This has a bearing on how you start the program which generates the output, it needs to write to standard output and then you pipe this into rotatelogs.

eg

while true ; do date ; sleep 30 ; done | rotatelogs -n 10 /home/mike/tmp/qqq.log 60

Will give you a circular rotation through 10 files, cut every 60 seconds

Using rclone, I would expect --log-file "/dev/stdout" would make it write to standard output.

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

logrotate on the other hand, checks the logfiles when it is run, and usually systems are setup to run logrotate (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/logrotate.d and yes, it usually runs as root.

Edit: @mike rodent

logrotate is run (by cron or by hand) and then checks the logfile statistics as seen in the file-system, it then triggers a rotation depending on it's configuration.

rotatelogs reads it's standard input continuously appending to a logfile, then when the time or the size of logfile reaches a trigger-point, it closes the current logfile, creates a new logfile (appropriately named), then continues to append to the new logfile.

This has a bearing on how you start the program which generates the output, it needs to write to standard output and then you pipe this into rotatelogs.

eg

while true ; do date ; sleep 30 ; done | rotatelogs -n 10 /home/mike/tmp/qqq.log 60

Will give you a circular rotation through 10 files, cut every 60 seconds

Using rclone, I would expect --log-file "/dev/stdout" would make it write to standard output.

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

rotatelogs on the other hand, checks the logfiles when it is run, and usually systems are setup to run rotatelogs (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/rotatelogs.d and yes it runs as root.

As you are producing your own log files and require a more real-time size chopping, you should take a look at the Apache utility rotatelogs here.

It works by reading stdin, and chops up the logfile based on command line arguments.

eg.

program-writing-to-stdout|/bin/rotatelogs /home/mike/tmp/qqq.log 1M"

rotatelogs on the other hand, checks the logfiles when it is run, and usually systems are setup to run rotatelogs (via cron) once per day. The configuration of sub-systems is usually done by dropping the configuration file into /etc/rotatelogs.d and yes it runs as root.

Edit: @mike rodent

logrotate is run (by cron or by hand) and then checks the logfile statistics as seen in the file-system, it then triggers a rotation depending on it's configuration.

rotatelogs reads it's standard input continuously appending to a logfile, then when the time or the size of logfile reaches a trigger-point, it closes the current logfile, creates a new logfile (appropriately named), then continues to append to the new logfile.

This has a bearing on how you start the program which generates the output, it needs to write to standard output and then you pipe this into rotatelogs.

eg

while true ; do date ; sleep 30 ; done | rotatelogs -n 10 /home/mike/tmp/qqq.log 60

Will give you a circular rotation through 10 files, cut every 60 seconds

Using rclone, I would expect --log-file "/dev/stdout" would make it write to standard output.