Google AppEngine Error "The API call search.IndexDocument() required more quota than is available"

Question

A few hours ago, indexing documents on AppEngine search started failing. Been working for years.

The following error occurs when saving a single document:

Caused by: com.google.appengine.api.search.PutException: The API call search.IndexDocument() required more quota than is available.

    at com.google.appengine.api.search.IndexImpl$3.wrap(IndexImpl.java:335)

    at com.google.appengine.api.search.IndexImpl$3.wrap(IndexImpl.java:314)

    at com.google.appengine.api.utils.FutureWrapper.wrapAndCache(FutureWrapper.java:74)

    at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:115)

    at com.google.appengine.api.search.FutureHelper.getInternal(FutureHelper.java:88)

    at com.google.appengine.api.search.FutureHelper.quietGet(FutureHelper.java:47)

    at com.google.appengine.api.search.IndexImpl.put(IndexImpl.java:499)

    ... 76 more

I have checked the quotas page, and no quota seems to have been exceeded (screenshot attached). Also billing is enabled and settled.

Any ideas on what happened? Thank you.

Answer 1

It turns out it was no fault of our code, non-reproducible, some bad luck.

Index-level quotas for a couple of indexes were reset to 0GB. It is important to note that quota for an index is by default 10GB, and it is not possible to for a user to set it to 0GB.

Google Cloud support was helpful. After an investigation by Product Engineering Team, the explanation was that "during a recent upgrade of our quota infrastructure, there was a transient error that may have temporarily reset your quota bucket.".

If this happens to you, writes on the index are suddenly and effectively disabled. Quota increase request take days to be processed. No workaround. Therefore, your best bet may be to create a new index, copy the documents over to the new index and begin using the new index.

After your request for quota increase has been granted, you may re-use the index. However, we noticed possible signs of data corruption, so it might be safer to delete all documents, delete the index, and start the index anew if need be.