Java - In terms of performance when handling Swing component views. Is it better to hide the component or just remove the component altogether?

Question

I am implementing a simple Java Desktop application that has a main panel where any of the views chosen by the user from the menu will be loaded. Currently, I am accomplishing this by removing the current components on the panel by invoking:

viewPanel.removeAll();
viewPanel.add(chosen view);
viewPanel.repaint();

I am wondering if it is better to create some sort of ViewManager that goes something like:

private final Component mainView;
private Component activeComponent;
public class ViewManager(Component main){
   mainView = main;
}

public void register(Component subview){
   mainView.add(subview, "span, grow"); // I am using MigLayout as my LayoutManager
}

public void show(Component subview){
   if(activeComponent != null){
      activeComponent.setVisible(false);
   }
   activeComponent = subview;
   activeComponent.setVisible(true);  
}

In this way I could somehow minimize objects that are being created and re-created over and over again, I figured that's a bit of an overhead.

Although, the second approach is probably good only for applications that contain a minimum amount of views and where switching of views are frequent because this approach could also take up a lot of memory as well, at that point maybe the idea of implementing an Object pool to limit the number of object reference can be managed is worth exploring.

Am I on the right track? Or am I thinking about it the wrong way? Or maybe, loading the views in that panel is not a very good idea in the first place, but how else could implement it?