What you exactly want is not possible as it would require statefull process (to know the state previous of switching to by layout).
However, along with the ISO_Prev_Group and ISO_Next_Group symbols allowing to decrement/increment in the layouts ring, there is also those two: ISO_First_Group and ISO_Last_Group.
With this ring: us,by,ru you can always access any of the three layouts in a single toggle.
For example, define Shift-CapsLock to send ISO_Last_Group when in us layout, ISO_First_Group when in ru or by;
define Ctrl-RightShift to send ISO_Next_Group when in us ISO_Prev_Group when in ru and ISO_Next_Group (or Last) when in by.
The difference with what you want is that you have to manually choose (with different shortcuts) the layout you want to go, when in by.
Also, if you need by layout for just a few belarussian specific cyrillic letters; another option would be to use a modified ru layout, with extra symbols in some keys, accessible with an AltGr or similar key, like in most latin keyboards.
As in the by file there are only 3 different keys from ru it seems that would be the right approach.
In such cas I would stack the layouts as: us,ru,by
Use ISO_Next_Group in "us", ISO_Prev_Group in "ru" for Shift-Capslock, and in "ru" define a key with a latching ISO_Next_Group.
To do that:
Create a ~/.xkb/keymap/mykbd where you put the output of setxkbmap, it will be your base keyboard definition; eg:
setxkbmap "us,ru,by" ; setxkbmap -print > ~/.xkb/keymap/mykbd
then create a file ~/.xkb/symbols/mysymbols with:
partial modifier_keys
xkb_symbols "shift_caps_us_ru_by" {
key <CAPS> {
symbols[Group1] = [ Caps_Lock, ISO_Next_Group ],
symbols[Group2] = [ Caps_Lock, ISO_Prev_Group ],
symbols[Group3] = [ Caps_Lock, ISO_First_Group ]
};
key <RALT> {
symbols[Group2] = [ ISO_Group_Latch ]
};
};
edit the ~/.xkb/keymap/mykbd file, and change the xkb_symbols line to add +mysymbols(shift_caps_us_ru_by)
finally, you can load it with xkbcomp -I$HOME/.xkb ~/.xkb/keymap/mykbd and now Shift-CapsLock switch between latin (us) and cyrillic (ru) layouts; and while in cyrillic, you can access belarusian-specific letters holding AltGr (right Alt) key.