You can look at /proc/<pid>/maps, /proc/<pid>/smaps (or pmap -x <pid> if your OS supports) of interested process ID's and compare outputs to determine shared memory regions. That includes shared memory segments via shmget calls, as well as any shared libraries, files.
Edit: As mr.spuratic pointed out his answer here has more details on kernel side
You can look at a process RSS using ps, however it doesn't take into consideration all the shared pages. To see RSS for specific process, see below
cv@thunder:~$ ps -o rss,pid,comm -p $$,7023
RSS PID COMMAND
22060 7023 xfwm4
6876 18094 bash
smem tool provides more detailed information, taking into consideration of shared pages. See below output for the same above process
cv@thunder:~$ smem -t |egrep "RSS|$$|7023"
PID User Command Swap USS PSS RSS
9852 cv grep -E RSS|18094|7023 0 340 367 2220
18094 cv bash 0 3472 4043 6876
7023 cv xfwm4 --display :0.0 --sm-c 0 5176 7027 22192
From man smem:
smem reports physical memory usage, taking shared memory pages into account. Unshared memory is reported as the USS (Unique Set Size). Shared
memory is divided evenly among the processes sharing that memory. The unshared memory (USS) plus a process's proportion of shared memory is
reported as the PSS (Proportional Set Size). The USS and PSS only include physical memory usage. They do not include memory that has been
swapped out to disk.