Lack of trust

You write that your empire is a "semi-totalitarian" one, and that the number of soldiers is limited. This means some recruits could be untrustworthy (maybe there is a mandatory conscription system?).
It could be dangerous to arm those soldiers with very dangerous weapons, which are overkill anyway when it comes to conquering low-tech civilizations, so the empire keeps the high-tech weapons for the elite platoons.

1) Never hand your enemies advanced technologies - especially if they aliens

There are certain concepts which are nearly impossible for less advanced societies to figure out. A renaissance level civilization couldn't reverse engineer nuclear technology if you left them a running reactor, a functional MRI scanner and a nuke with a "press here" button glued on it's side. If a single soldier had to drop an automatic rifle while doing a routine cull inspection however, then the next time your fleet visited that medieval society, every militia would be packing heat.

Its such a problem that I expect that even in the real world, if we ever figure out crazy space age tech, it would never be allowed on ground vehicles or outposts - only stealth fighters which should never be shot down and ships large enough to carry nukes to self detonate should their advanced tech ever be at risk of capture.

2) Horses self-replicate, nano-tech doesn't

We have a few regional conflicts going on at the moment in real life and the entire military industrial complexes of the world are struggling to keep up with the equipment demands. We like to show off the few fancy vehicles we build during times of peace but actual wars are beasts of mass consumption. If you send a military fleet into space, there is no adequate way to resupply that force with munitions or replacement vehicles. So while you definitely going to keep a stock of space age tanks and support vehicles, you not letting those things touch any planet until you've catalogued every counter-measure that planet has proven able to throw at horses for a few months - and the horses can self-replicate on any planet that has edible vegetation.

3) Military Cultures are military cultures

You can't keep order amongst countless cultures and planets unless you have dedicated high quality soldiers. Mercenaries have conflicting interests and local conscripts will default to the base behavior of their culture the moment you look away. You need home grown troops to pull off interstellar subjugation and the only way your planets will produce men with the temperance and willingness to serve is if you can maintain strong military traditions across your worlds. That means your troops are likely going to be game hunting with rifles when they boys and game hunting on alien worlds with bolt-locks when they men (decent target practice training as well). While even the most honorable military cultures will land up with troops that laugh while they mow down fleeing civilians from the passenger seats of Apache helicopters, each moral tradition still puts a profound limit on how far down the track of savagery your troops will go - and sometimes that means making them perform duties with primitive "honorable" weaponry.

4) Bayonets are a vital competence test for the officer core

While bayonets should be a standard inclusion in military provisions, and a recommended tool in the general tactics guide, they are only there to identify officers due for termination. Any officer that thinks bayonets are a viable piece of equipment are due for execution on the grounds of treasonous incompetence. Remember to make their deaths look like accidents to preserve the integrity of the test though. Bayonets degrade the performance of the gun unless they are manually attached after firing - and your soldiers should never stop firing unless they have to personally push the gunpowder down the gun nozzle with a stick between rounds. If melee capabilities are required, give your soldiers daggers or short swords. Any alien species that requires more then a bolt-rifle and a short sword will necessitate you to provide your troops with actual weaponry.

Because not all species encountered will be as technologically advanced as the ones the Empire is at war with.

As it explores the Galaxy the Empire will encounter worlds whose inhabitants are not technically advanced. In fact there will likely be many more primitive societies than technically advanced ones at any point in galactic history. This being the case the empires most advanced weaponry isn't needed to pacify such societies and primitive weapons are more than sufficient to force the local populous into submission.

Stone aged cultures will fall under the heel of steel swords and armor, crossbows and lances, renaissance level societies will fall before the power of bolt action rifles, cannons and grenades etc etc. So what the Empire does is train specialist units in the use of historic weapons and tactics while other units are trained for the Empires high tech wars. This saves time money and equipment.

The usual deal would be to land troops at a chosen point on the world to conquer the locals and establish a base of operations from which you start expanding out across the rest of the planet. Time isn't an issue because even if it takes a few generations the planet's population are always going to be outgunned and 'out teched'. (A stone age culture will fall long before it figures out how to cast bronze let alone steel and a medieval one long before it can make breach loading rifles.) This is especially the case since while the troops are equipped with primitive weapons nothing says some can't be moved around quickly by aircraft or ground vehicle when the need arises. The locals basically stand no chance.

And the best bit? While the descendant's of the original military units go on to become the local aristocracy on the world their ancestors landed on (with high tech benefits for them of course like electricity & plumbing). After the planet is pacified you can start recruiting soldiers for other 'reenactment regiments' from these locals.

And while each conquered world is slowly being brought up to an approved "Imperial Standard" tech level in a process that takes generations along the way you get to recruit new soldiers for whom a cross bow or rifle is the latest in high tech gear for units whose job will be to conquer other primitives worlds.

The goal is not to kill aliens, it is to coerce them.

An assault rifle can kill a protester, or several of them. But can it make them back down? Depending to the psychology, cold steel (a bayonet on top of a rifle) could be able to intimidate protesters and so make them back down. The longer the bayonet, the longer the rifle, the scarier it looks. It would be difficult to explain why this is a bolt-action rather than a semi-automatic or selective fire battle rifle. But for a long time, the Brits used semi-automatics rather than selective fire.

The same explanation may be able to explain some use of cavalry. They're actually mounted police, not mounted infantry. Especially for riot control, those are still in use in many countries today.

Why, then, are your ground forces glorified riot police?

• Starships have the means to destroy any planetary surface which is not defended by other starships. They can "nuke them from orbit" or drop near-C rocks.
• Infantry is used to accept the surrender of the old government and then conduct counterinsurgency warfare to make the new collaboration government stick.

Vernor Vinge's Zones of Thought series addresses this problem with the titular Zones - the laws of physics in the universe are not homogenous, but exist on a gradient. In the upper zones, all kinds of technological wizardry is possible, but that technology begins to fail as you dive deeper. Go deep enough, and intelligence of any kind is impossible.

Adapting this idea fixes your problem. It makes sense to give soldiers horses and swords if they are going to be fighting in a place without the physics that make guns work. A bolt-action rifle with a bayonet is a versatile weapon for units operating on the boundary of such a zone, because they are able to defend themselves on either side of that line (as a bonus, the longer range of a rifle vs an SMG means that you can fire across the boundary and hit units that can't fire back, and are too far to rush you).

Lots of Guns, Lots of Reasons:

There a re a lot of reasons an empire of this sort would have units primarily equipped with "primitive" weaponry. Let's run through a few:

• Primitive colonies: Your empire is colonizing worlds in a panspermian universe where most worlds are compatible with human life. Colonies consist of groups of humans from "third world" nations being dropped off with limited supplies. Their industrial bases are unable to build vehicles or advanced weapons, but there are billions of humans being dropped off this way. All these colonies would be early in development by your timeline, and much of the industry is devoted to simply building more industry, roads, harbors and the like. Horses don't need a factory. Bolt-action rifles are inexpensive, being made for hunting/defense etc., and surprisingly efficient in the hands of trained troops. So these colonies equip units locally on the cheap because tools are expensive and human life is cheap.
• Technology Dominance: Your empire deals with a lot of aliens that aren't that militarily inclined. They don't NEED sophisticated weapons to have an overwhelming advantage. But some of these aliens are REALLY good at copying anything they get a hold of - they aren't stupid. So advanced units destroy critical targets, seize invaluable resources, and then get replaced by yokel troops armed with simple weapons. The aliens can't copy what they don't have access to, and you really don't want to let them get access.
• Rules of engagement: A staple of scifi is societies where wars are only allowed to be fought with primitive weapons. Advanced aliens may only intervene in wars where the two sides are significantly different in capabilities. Or perhaps they CAN make advanced weapons but will only do so when faced with enemies also using such weapons.
• Alien Recruitment: Humans are conquering the universe, but some of the races they have conquered are okay with that. With billions of potential alien troops available to recruit, it's cheap to throw waves of alien cannon fodder into battle, but expensive to give them advanced weapons. There may be huge stockpiles of bolt-action weapons in arsenals held from previous wars on Earth that are rusting away. You may not trust those aliens with sophisticated weapons out of concerns of rebellion. Aliens may already technologically be able to make simple weapons but not advanced ones with their existing industry.

Already existing tactics

If all other civilizations are less advanced, then for every civilization we already have a proven tried and true counter.

The enemy relies on battleships? Bring out the aircraft carriers. The enemy relies on cavalry? Bring out the machine guns. The enemy relies on mountains to dodge your mechanized troops? Bring the mountaineers with their mules.

The point is... crafting new doctrines all the time costs valuable "staff officer time". And it's risky. How do you apply your military doctrine of nano-submarine hyper-wolfpacks against the enemies horse archers? Who knows. But there are for sure books over books in the library, how people beat the Huns or Mongols in the past. We already know, because it already happened.

If you know what works and you can just hand a historical book on those tactics and equipment to your army to fight that civilization, what could go wrong? You just send a force exactly advanced enough to apply exactly the tactics that were successful before.

Call it surgical historical precision and make it the intergalactic armies global doctrine to apply exactly the technology they know is effective to beat the foe.

There is literally just one reason to do this: mandatory conscription to reduce the population of Earth, or to remove "undesirable elements" from that population.

Any other application is basically insane, with a number of reasons - of which three are military doctrine and at least two are socio-politico-economical - for me to think that:

1. Modern soldier, especially of the first-tier military forces (not talking about tier-1 unit grading, just evaluating whole militaries), is a highly trained and equipped individual. Just the training costs a lot, and this is basically just to create minimally skilled infantryman (or tanker, artilleryman, airmen etc.). The equipment is designed to give greatest possible advantage over the opposition, which by definition means higher lethality and survivability,
2. Military equipment does not amount to an uniform, basic gear, weapon and personal effects. By basic gear it's meant not the field-strip-and-clean toolkit for the weapon, but it's - for example - advanced communication equipment (which needs to be reliable and secure), night-time fighting equipment, ballistic armor, protective gear... Then we move to more specialized ones: satellite communication, target-marking, electronic surveillance, aerial surveillance, battlefield awareness and tactical net equipment (i.e. IFF, integrated net display devices, showing disposition of forces). And that's just infantry. Same applies to reconaissance (with emphasis), armor, artillery, air force, navy...
3. Every level of tactics - from basic to tactical unit groups (i.e. army or a corps) - is designed based on decades of experience, available equipment and above all performance level. In other words: everyone is expected to perform to some minimum standard, because that expectation shapes the tactical and strategic planning. But since armies do have something called an institutionalized memory and they do have a bureaucracy to record everything, they are also used to increase performance levels. For example: militaries know what type of training is better for that type of warfare (because it was done, tested, analyzed and conclusions have been drawn from it - see WW2 77th Infantry Division story), further maximizing the effectiveness of the whole military. So the more serious the military, the higher is that "minimum standard".
4. Prolonged war will always produce some kind of a fatigue of it in the society, which will always have detrimental effect on both society and military. Morale, recruit quality, equipment quality - take your pick. Most important factor of that fatigue is the casualties - and not the KIAs, rather the WIAs and MIAs. And less tech advanced force tends to produce much more of the latter two than the former. In the past, militaries suffered more casualties from disease and food-related sickness, than from combat and following it inadequate medical treatment and care of the wounded. Modern medicine basically got rid of these non-combat losses permanently, and at least part of the modern military equipment is there to make sure it stays that way (water purification, modern drugs and treatment techniques combined together, etc.),
5. Military expenditures are designed to be long-term, first of all because of the cost (it has to be spread over time), but also production capacity. There's literally no way to produce best modern equipment and arms in quantities needed in short term (by which I mean inside a year). So because of that long-term necessity, equipment needs to be designed, tested, approved for use over certain period, and then financed and only then, finally, produced. There is no way to make that short-term because no one has that kind of money, and just a familiarization with that new equiment is not enough time to fully realize it's potential, strong and weak points, and best way to use it. Those can come with experience only, which by definition can be long term only.

In other words: military force evolves with experience and both societal and technological advancements. You can't somehow disentangle them. There is a reason armies do not use horses anymore, even if they eat less than a truck towing a howitzer (on a pound-for-pound basis)...

And that's just for starters, because we didn't even touch the most important bit: logistics. It is true that in some ways that will be simplified with less technologically advanced military force - just as other answers indicate, you can't field-produce modern ammo, or make field-expedient repairs, or replacement parts for the equipment that broke. But you can't apply same to food, medical, quartering, especially ON OTHER PLANETS. If you want to maintain the readiness level at the maximum available you have to stick to modern medicine and associated equipment, modern sleep gear/equipment and waste disposal, just to name the most important ones. If you have a sizable force in the field - say an infantry brigade - JUST water purification vehicle systems (i.e 3000 gph system), water storage and distribution system weighs about 33% of the 30-day ammunition supply for for an infantry brigade (see p.107).

And now food. It is especially critical - the chances that there will be anything edible on a different world are minuscule. On Earth everything in the living sphere is designed to interact, which can be simplified to amino-acids and capability to break down food into energy by organism. Here it all - with the possible exception of rice and corn - can be broken down to fats, sugars or carbohydrates, minerals and fiber - in other words, for my purposes: "left" stereo isomers (I'm super simplifying here). There are a number of ways how isomers combine, and chances for the correct, compatible with human biology, way to happen on another planet are not high.

All that is just the tip of an iceberg, which other answers neglect completely. Where is the rest of the supply train, besides food and medicines: living quarters (i.e. tents, beds), non-field-manufacturable spare parts, tools, fuel, vehicles?

Because foraging for food and ammo may literally be out of the question (for both food and ammo - black powder requires biological components, too), any kind of saving on the supply is simply a hallucination.

Unless you don't want the soldiers you send out to return. So if you want to use them cannon-fodder-style in combat, there's no point to train them, too...

So, unless you plan to have two kinds of military forces - the capable one for defense and the expendable one for conquest (and the latter not aware of the former, of course), with large enough population to support the latter variant indefinitely, I don't see any point of reverting to 200-year old way of conducting warfare in interstellar conflict.

Arms-Race Blind Spots

I read one SF series where it turned out chemically-propelled firearms, while suboptimal, were not as bad as one might think.

The predominant weaponry civilizations used was energy-based, but there were also lots of counter-measures which space-faring militaries had developed over the centuries to deal with them. Those counter-measures were not built to stop old-fashioned physical projectiles, because they never faced those anymore. That meant that, while an army with high-velocity firearms wasn't as effective against modern soldier gear as one with modern energy weapons, they weren't as ineffective as say an army with 2 generations out-of-date energy weapons would be (which is what would be available to backwater rebels otherwise).

I think this is somewhat analogous to how billion \$ ships, tanks, and aircraft have recently seemed to have been rendered obsolete by cheap drones.

It would be difficult to explain the lack of SMGs in this era since in the USA, before the passage of a law in 1934 put an end to this being done legally, it was something of a hobby among professional and amateur gunsmiths to convert bolt action and lever action rifles to machine guns. The technology is not advanced, not if there's access to machining to produce bolt action rifles and springs.

If we are going to allow for some creative physics then one idea I have is that the FTL drive somehow impacts the physical properties of metals on the ships that makes spring tension shift unpredictably. A perfectly functional and safe SMG could become something on the spectrum of jam-o-matic, which would be no better than a bolt action rifle, to a hand grenade with a pistol grip and iron sights.

This shift in metallic properties could also explain why there's no internal combustion engines or electric motors for vehicles, the best they'd have would be beasts of burden like horses, elephants, and whatever else suits the situation.

I'll pose a potential workaround to this problem, auto-revolvers. An auto-revolver that works off the force of gasses from the fired bullet could rotate the cylinder to quickly put another round inline with the barrel and ready to fire. Since the cylinder rotates in only one direction a design could be developed that does not rely on springs but instead a kind of clockwork. Depending on how elaborate those involved want to get with the design the number of rounds in a cylinder could be a low as six to as many as 50 or 100 before the size becomes unwieldy. With some added complexity the cylinder could be replaced with some kind of belt fed mechanism, which could make a machine gun that could hold hundreds of rounds. Belt fed machine guns are more of a crew served weapon than something a single soldier would carry. In either case a reloading of the weapon would involve replacing the cylinder, magazine, or belt with a new one.

The shifting properties of metals under the influence of FTL could make a lot of other items useless. Semiconductors could fail to operate unless overbuilt to compensate, which would rule out guided missiles, portable radios, and so much else. Electronic computers and radio communications could logically be limited to use on the ships due to their mass. Mechanical computing devices might be preferred in many cases. There's a lot of examples through history of adding machines, pocket calculators, and special purpose computing devices made to operate with gears and powered with a hand crank.

Overbuilding rifles, bladed weapons, certain kinds of armor, and so much else, to compensate for shifting metallic properties could be explained as practical since they are relatively small and simple.

It may be possible to restore the properties of these metals but only with reforging/recycling, application of some kind of radiation, just waiting it out for whatever caused the problem to dissipate or decay, or some other process that takes too much time and effort to perform between the use of FTL and the battle.

Because soldiers are clones or conscripts and we had the stuff laying around.

This only makes sense under one condition, the military can replace soldiers for practically nothing, and they don't really care how well they are armed. There is only a narrow set of conditions where this will happen but it did happen in real history often enough.

practically free soldiers is tricky, either they are mind implant clones or they are local conscripts. In both cased replacing soldiers is very low cost. This also means the public can't care much about these soldiers so clones would be better. Of course an interstellar empire might be able to lock down communcation welll enough to keep the number of conscript losses under wraps.

If soldiers are basically just fodder then inferior weapons makes sense, you just give them whatever outdated junk you have lying around, or stuff that can be made locally. you give out your mothballed stuff just to recoup some of the cost of making it.

We see this in history a lot, conscripts are often given whatever the oldest weapons you have. Since you have to arm them to make them somewhat effective but you don't actually care about the troops you just give them the oldest stuff you haven't gotten rid of yet. Or if they are conscripts often they have to outfit themselves with whatever they can or whatever their lower tech worlds could afford to make. This makes a lot of sense for an empire which will have lots of weapons lying around and lots of marginal populations.

Alien Auxilia

In Early Imperial Rome, the Legionary armies were equipped and trained with Roman arms, armaments, and tactics; however, nearly 1/2 of Rome's forces were various forms of Auxiallia who were not Romans at all. The Auxilia were all competent soldiers in their own right, but did not speak the same language or have any experience with Roman weapons and tactics and had completely different military doctrines; so, even if Roman weapons and tactics were arguably better, trying to integrate them with the Roman army was more trouble than it was worth. So, these unintegrated foreign soldiers fought with their own native weapons and tactics in detached formations adjacent to Roman's legions, but without being a part of the legionary system itself.

Because humanity is conquering other species, you will run into a similar problem. As your empire rapidly expands, you will need to start relying on non-human forces to bolster your army, but these forces will not speak the Empire's common language which will make training them in human weapons and tactics and implementing them in human combined arms exercises cause more problems than not recruiting them at all. So instead you use them as unintegrated forces that do things like local peacekeeping, guarding low priority targets, force recon, or diversionary assaults. They are already trained, armed, and on par with the local forces that they are drawn from; so, they are incredibly expendable, even if they are not as effective as human forces.

So, your alien cavaliers do not know how to drive, or about suppressive fire, or target discipline, or any of that, but they do already know how to ride alien horses and use alien weapons and are masters of the tactics that make the most out of those alien weapons. So, you let your alien auxilia do what they know how to do so that you don't need to bother with rearming and retraining them in human styles of warfare.

Even though there have been some awesome answers, and the owner has accepted one already, I'd still like to add this because it hasn't been mentioned yet I believe:

A quasi-religious Code of Honor.

Maybe these units within the Grand Liberation Army are an elite, the best of the best. Maybe they have elaborate rituals, behavioural codes, etc., and they take pride in using weapons and modes of battle that other parts of the army consider archaic. Maybe there's a whole esoteric framework around the use of these weapons, where they are revered as objects possessed by spirits...

Or something along those lines. :)

There's literally no reason to do any of that. It's utterly pointless and nonsensical to produce inferior weapons when one has noticeably superior ones for the same/less cost. Assuming the society in question even has the tools and facilities to produce them(like horse-cavalry) at scale anymore, and haven't just discarded them.

Modern/Near-Future assault rifles, machines guns, and UAVs are more cost-effective than pre-1945 bolt-action rifles and horses. And that would hold true regardless of what enemies you would encounter.

If said enemies are primitive pre-industrial civilizations, then the Empire's equivalent of an F-35, Carrier Strike-Group, or electronic warfare units aren't cost-effective/useful. But infantry and UAVs with some token heavy artillery support would undoubtedly be highly cost-effective in such a situation.

A society capable of interstellar travel isn't going to encounter a situation where horses would be an optimal(or the only practical) form of transportation, regardless of local infrastructure so that's out.

Why waste resources building carved-wood and milled-steel bolt-action rifles when less costly aluminium/polymer/stamped steel-alloy assaults rifles and SMGs are the standard? So that's out as well.

You could get away with bayonets if you want to use them as protester control like o.m. suggested. But you're straddling a half-assed line between non-lethal and lethal forms of crowd-control so it's still sub-optimal.

You kind of wrote yourself into a corner. You want certain tools to be used, but your set-up makes it an illogical outcome if one were to follow real-world logic.

Why do you even want soldiers of an interstellar civilization based on a speculative alternate future to be fighting with essentially pre-1918 equipment?