Bash script to copy *.log files into a new directory

Question

I have some files like this:

2012-12-17_083213_1.log  
2012-12-17_083213_1.log_stats  
2012-12-17_083213_1.logaccount_ptr  
2012-12-17_083213_1.loginitial_ptr  
2012-12-17_083213_1.logptr  
2012-12-17_093049_2.log  
2012-12-17_093049_2.log_stats  
2012-12-17_093049_2.logaccount_ptr  
2012-12-17_093049_2.loginitial_ptr  
2012-12-17_093049_2.logptr

In here every 1G of data, a new file is created with the name in Year-Month-Day_HourMinuteSecond_number.log format (as you can see above).

I want to

1. Copy all *.log files to a new folder named with the format Year-Month-Day, so for these it would be named: 2012-12-17
2. tgz that folder
3. Send it to a server IP.

I don't know how exactly. I want it as a bash script so that I can run it every day for the newly added files.

Answer 1

I will copy you a script to do it, but next time do some research first and try to write it by yourself. That is the only way to learn. And, please, don't use it without understanding it.

#!/bin/bash
tosend=""
for i in *.log; do
    dir=$(echo $i | awk '{split($1,a,"_"); print a[1]}');
    if [ ! -d $dir ]; then
        mkdir $dir;
    fi;
    cp $i $dir;
    tosend="$tosend\n$dir"
done
for i in $(echo -e $tosend | uniq);do
    echo Compressing $i
    tar -zcvf $i.tar.gz $i
    scp $i.tar.gz user@server
done

The key part is using awk to split the file name, and then keeping a list of the folders you have modified. It is not the most elegant but it works.

Note that if you copy the files, next time you run it they will be copied again. The best thing would be to move them.

Btw, I added the uniq part instead of adding the name to the "array" within the if because it's possible that the folders already exist when you run the script.

However, if you only wish to send the new files and you run it at midnight, this is probably the best approach:

#!/bin/bash
tosend=""
for i in *.log; do
    dir=$(echo $i | awk '{split($1,a,"_"); print a[1]}');
    if [ ! -d $dir ]; then
        mkdir $dir;
        tosend="$tosend\n$dir"
    fi;
    mv $i $dir;
done
for i in $tosend;do
    echo Compressing $i
    tar -zcvf $i.tar.gz $i
    scp $i.tar.gz user@server
done

This would only send the new folders (even though some of the old ones may have new files). And with the mv the main folder will be kept clean.

For more sophisticated/bulletproof approaches, I think rsync is the best shot.