Firebase Optimistic Locking Duplicate Processing

Question

I have two Firebase functions that run hourly and process the same list of users. They share a batch document(this.batchId: MM-dd-yyyy-HH so it's unique for each hour) and use transactions to coordinate processing. Each instance gets a batch of users after a specific ID (lastProcessedId) and updates the batch document with:

• A new lastProcessedId
• A global totalCount increment
• Instance-specific counts and processed IDs

The global totalCount is always correct, however, sometimes the instance-specific fields show that both instances processed some of the same IDs (A batch of two usually). This conflicts with how i understand optimistic locking to work - why aren't the transactions preventing this overlap in processing? Any advice or insight would be apprechiated.

private async getNextBatchTransaction(): Promise<{ userDocs: QueryDocumentSnapshot<DocumentData>[] | null, needsCleanup: boolean }> {
  return this.firestore.runTransaction(async (transaction) => {
    const batchRef = this.firestore.collection("batch_sequence").doc(this.batchId);
    const batchDoc = await transaction.get(batchRef);

    const data = (batchDoc.exists ? batchDoc.data() : {
      lastProcessedId: null,
      complete: false,
    }) as BatchDocument;

    if (data.complete) {
      return { userDocs: null };
    }

    let query = this.firestore
      .collection("users")
      .orderBy("__name__")
      .limit(this.batchSize);

    if (data.lastProcessedId) {
      query = query.startAfter(data.lastProcessedId);
    }

    const userSnapshot = await transaction.get(query);

    if (userSnapshot.empty) {
      transaction.set(
        batchRef,
        { complete: true },
        { merge: true }
      );
      return { userDocs: null };
    }

    const batchLength = userSnapshot.docs.length;
    const lastDoc = userSnapshot.docs[batchLength - 1];
    const processedIds = userSnapshot.docs.map(doc => doc.id);
    
    transaction.set(
      batchRef,
      {
        lastProcessedId: lastDoc.id,
        totalCount: FieldValue.increment(batchLength),
        [`instance.${this.instanceId}`]: FieldValue.increment(batchLength),
        [`processedIds.${this.instanceId}`]: FieldValue.arrayUnion(...processedIds),
      },
      { merge: true }
    );

    return { userDocs: userSnapshot.docs};
  });
}

I would expect Thread 1 to commit the transaction, updating 'lastProcessedId'. Thread 2, which started simultaneously, would see that lastProcessedId has been updated (from firestore.collection("batch_sequence").doc(this.batchId)) and consequently fail. It would then retry, this time grabbing the 'lastProcessedId' that Thread 1 set, and process the next batch of users. This cycle would repeat until all users are processed.

Considering the following notes on runTransaction, I'm surprised it doesn't act like the above.

    /**
     * Executes the given updateFunction and commits the changes applied within
     * the transaction.
     *
     * You can use the transaction object passed to 'updateFunction' to read and
     * modify Firestore documents under lock. You have to perform all reads
     * before you perform any write.
     *
     * Transactions can be performed as read-only or read-write transactions. By
     * default, transactions are executed in read-write mode.
     *
     * A read-write transaction obtains a pessimistic lock on all documents that
     * are read during the transaction. These locks block other transactions,
     * batched writes, and other non-transactional writes from changing that
     * document. Any writes in a read-write transactions are committed once
     * 'updateFunction' resolves, which also releases all locks.
     *
     * If a read-write transaction fails with contention, the transaction is
     * retried up to five times. The `updateFunction` is invoked once for each
     * attempt.
     *
     * Read-only transactions do not lock documents. They can be used to read
     * documents at a consistent snapshot in time, which may be up to 60 seconds
     * in the past. Read-only transactions are not retried.
     *
     * Transactions time out after 60 seconds if no documents are read.
     * Transactions that are not committed within than 270 seconds are also
     * aborted. Any remaining locks are released when a transaction times out.
     *
     * @param updateFunction The function to execute within the transaction
     * context.
     * @param transactionOptions Transaction options.
     * @return If the transaction completed successfully or was explicitly
     * aborted (by the updateFunction returning a failed Promise), the Promise
     * returned by the updateFunction will be returned here. Else if the
     * transaction failed, a rejected Promise with the corresponding failure
     * error will be returned.
     */

Answer 1

I was using firebase-admin version 11.8.0, which is two major releases behind the newest 13.0.2 version. After upgrading to version 13.0.2, the code works as expected.

After test running it 100 times, each instance processed unique ids each time.