You're absolutely right about needing to link against other libraries. However, you don't necessarily need to worry about all of the details yourself, unless you want/need to do something unusual. Several alternative IDEs already support Arduino, typically via plugins which handle a lot of the details automatically.
I normally recommend Eclipse as it's a very widely used IDE for various purposes. However, there are several other possibilities. See this question for more information: