Different language; different syntax.
There is no one thing that necessarily corresponds to an infinitive as found in Indo-European languages.
For example, if you wanted to say, "I want to go to Tokyo", in Japanese you just say
東京に行きたいです。
たい is an inflectional ending appended to the base of the verb indicating the want to do something. There is no infinitive (or anything remotely like it) in this sentence.
If you want to say, "I want you to go to Tokyo", you say
東京に行ってほしいです。
In English we get the same infinitive but in Japanese the constructions are quite different. This time ほしい expresses the idea of "want" and the verb takes a different form 行って as opposed to 行き when you express "I want to go".
If you want to say something like, "I forgot to ..." there are a number of ways to express this idea. But let's stick with your example, "I forgot to say"
いいわすれました。
where いい is the inflected form of いう. In English, we call this the continuative form. In Japanese it's called [連用形]{れんようけい} (renyoukei). It is the form off of which many constructions/inflections of the verb are built.
The renyoukei is not an infinitive.
The idea of infinitive comes around from the description of certain inflected forms of verbs, such as amo or amas. These forms are called finite. Forms of the verb not inflected for person are called infinite, ie., not finite. Hence we get the descriptor infinitive.
No verbs in Japanese inflect for person or number. So this distinction between finite verb forms and infinite verb forms is non-existent.
I said above that there are a number of ways to express these things. My sense is this expression "I forgot to ... I'm sorry" is common in English. In Japanese, I feel it's more natural to say something like
昨日お弁当を買ってくれたのに『ありがとう』と言っていませんでした。申し訳ございません。本当にありがとうございました。
Yesterday, even though you bought me bento [for lunch], I did not say thank you. I apologize. Thank you very much.
Perhaps the Japanese here is a bit more formal than you need. But, my point is that these sorts of ideas can be conveyed in very different ways. It's not that a Japanese person would never say something like "I forgot to say...", but they're more likely to focus on the actions of the person who did something for you than on your own actions (or lack there of). The idea of gratitude comes through twice in fact: once via くれた (you did something for me) and again through ありがとう.
Some other great points and expressions are brought up in the comments.
