Which Scheduling algorithm is used in Linux?

The current Linux task scheduler is called Completely Fair Scheduler (CFS). You should have a look at http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt for more details. The design is quite complex and in my view not suitable for RTOS.

A common technique in realtime systems is rate-monotonic scheduling, because it has strong guarantees if certain assumptions hold (e.g. static task priorities and fixed execution time and rate). There are a whole lot other algorithms and there has been a lot of research. So it's basically all about the properties you need and what you know about your task and what is fixed.

The Round Robin algorithm is generally used in time sharing environments.

I'm not quite sure, whether you are taking about the I/O scheduling of the Kernel. In case you are not: Ignore this answer.

Wikipedia states, that the CFG (completely Fair Queuing) is default since Kernel 2.6.18.

On my openSUSE (running Kernel 2.6.37) I can switch between the CFG, NOOP and Deadline.

The algorithm used by Linux scheduler is a complex scheme with combination of preemptive priority and biased time slicing. It assigns longer time quantum to higher priority tasks and shorter time quantum to lower priority tasks.

It identifies each process either as real time process or a normal (other) process. Real-time tasks are assigned static priorities in the range [0,99], where lower number indicates higher priority.

All other tasks have dynamic priorities in the range [100,139], based on the interactivity of a task that are based on their nice values plus or minus the value 5. Tasks that are more interactive typically have longer sleep times and therefore are more likely to have adjustments closer to −5, as the scheduler favors interactive tasks. (A task’s interactivity is determined by how long it has been sleeping while waiting for I/O.) The interactivity of a task determines whether the value 5 will be added to or subtracted from the nice value. The result of such adjustments will be higher priorities for these tasks. Conversely, tasks with shorter sleep times are often more CPU-bound and thus will have their priorities lowered.

The kernel maintains a list of all runnable tasks in a runqueue data structure. A runqueue contains two priority arrays: active and expired. The active array contains all tasks with time remaining in their time slices, and the expired array contains all expired tasks. Each of these priority arrays contains a list of tasks indexed according to priority. The scheduler chooses the task with the highest priority from the active array for execution on the CPU. When all tasks have exhausted their time slices (that is, the active array is empty), the two priority arrays are exchanged: the expired array becomes the active array, and vice versa.

A task’s dynamic priority is recalculated when the task has exhausted its time quantum and is to be moved to the expired array. Thus, when the two arrays are exchanged, all tasks in the new active array have been assigned new priorities and corresponding time slices. (Note: This is an excerpt from the book on Operating System Concepts (9th edition) by Abraham Silberschatz, et. al. For details kindly refer section 5.6.3 of this book)

This is an answer to your another question. Real time systems(RTS) are of two types, hard and soft. CPU scheduling algorithm for hard RTS is priority based preemptive algorithm and that for soft RTS is non-preemptive priority.