How would this work with compiled languages though?
Your wording is wrong. A programming language is a specification written in some technical report (for a good example, see R5RS). Actually you are refering to some specific language implementation (which is a software).
My understanding is that once the compiler has compiled the source code to the target code - specifically native machine code - it is done. Its job is finished.
I strongly recommend reading the GC handbook. An entire book is needed to answer. Before that, read the Garbage Collection wikipage (which I assume you have read before reading below).
The runtime system of the compiled language implementation contains the garbage collector, and the compiler is generating code which is fit to that particular runtime system. In particular, allocation primitives (are compiled to machine code that) will (or may) call the runtime system.
So how could the compiled program be garbage collected as well?
Just by emitting machine code which uses (and is "friendly" and "compatible with") the runtime system.
Notice that you can find several garbage collection libraries, in particular Boehm GC, Ravenbrook's MPS, or even my (unmaintained) Qish. And coding a simple GC is not very difficult (however, debugging it is harder, and coding a competitive GC is difficult).
In some cases, the compiler would use a conservative GC (like Boehm GC). Then, there is no much to code. The conservative GC would (when the compiler calls its allocation routine, or the entire GC routine) sometimes scan the entire call stack, and assume that any memory zone (indirectly) reachable from the call stack is live. This is called a conservative GC because typing information is lost: if an integer on the call stack happens to look like some address, it would be followed, etc.
In other (more difficult) cases, the runtime provides a generational copying garbage collection (a typical example is the Ocaml compiler, which compiles Ocaml code to machine code using such a GC). Then the issue is to find precisely on the call stacks all the pointers, and some of them are moved by the GC. Then the compiler generates meta-data describing call stack frames, which the runtime uses. So the calling conventions and ABI are becoming specific to that implementation (i.e. compiler) & runtime system.
In some cases, machine code generated by the compiler (actually even closures pointing to it) is itself garbage collected. This is notably the case for SBCL (a good Common Lisp implementation) which generates machine code for every REPL interaction. This also requires some meta-data describing the code and the call frames used inside it.
Does the compiler store a copy of some garbage collection program and pastes it into each executable it generates?
Sort-of. However, the runtime system could be a shared library, etc.
BTW, most compilers for language with garbage collection (and their runtime system) are today free software. So download the source code and study it.