It is not “whenever I want”
First, “Governor in Council” is a term of art in Commonwealth parliamentary democracies. It means that the decision is made by the relevant Minister (here, the Immigration Minister) and presented for the Governor’s agreement “in Council” which means with at least one other Minister present. This is not just a “rubber stamp” process, the Governor can and does question the Minister’s recommendation, as can the other Minister present, to satisfy themselves that the proposed action really is in the “public interest”. However, it is not a reserve power of the Governor, so, if the Minister insists, the Governor must acquiesce.
Second, the courts can intervene. A person affected by this decision can sue and the government would need to prove in court why the decision was in the “public interest”. That is, they need to show that cancellation of the visa is in the interest of the citizens, residents or general domestic or international objectives of Canada in general; that’s what “public interest” means.
So, not being in the “public interest” is fact dependent. The test is not whether the court or the Governor would conclude that cancellation was in the “public interest”, but whether a reasonable person in the Minister’s position could, on the information available, reasonably decide that cancellation is in the “public interest”. Or, to put that from the opposite point of view, the decision is valid unless no reasonable person could, on the evidence, reach that decision. Typically this would include circumstances where the decision was made arbitrarily, capriciously, corruptly, for reasons unrelated to the actual case (e.g. blanket “tough on immigration” political reasons), or by not actually considering the specific facts of the case.