Why are cache uploads slow?

[razvanz]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

Actions Cache

Discussion Details

Hi! Why have cache uploads become so slow over the last few weeks? What used to take ~1m at the beginning of Jan now takes over 5 mins. I tried clearing the cache to rule out slow evictions, but it didn't make a difference.

Are they being throttled?

Sent 0 of 1015506426 (0.0%), 0.0 MBs/sec
Sent 0 of 1015506426 (0.0%), 0.0 MBs/sec
Sent 469762048 of 1015506426 (46.3%), 149.2 MBs/sec
Sent 805306368 of 1015506426 (79.3%), 191.9 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 180.8 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 150.6 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 129.1 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 113.0 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 100.4 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 90.4 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 82.2 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 69.5 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 50.2 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 41.1 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 30.1 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 20.1 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 15.1 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 10.0 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 8.0 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 5.9 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 5.0 MBs/sec
Sent 948397562 of 1015506426 (93.4%), 3.2 MBs/sec
Sent 1015506426 of 1015506426 (100.0%), 3.4 MBs/sec

[DavitEgoian]

Hi razvanz,

That definitely looks frustrating. A jump from 1 minute to 5 minutes is significant, especially when you can see the speed dropping off like that in the logs.

Based on the output you shared, here is what is likely happening:

1. Network Retries: The upload stalling at 93.4% and the speed dropping usually means a specific "chunk" of your cache failed to upload and the action is trying to send it again. The "speed" you see is an average, so while it retries, the average speed keeps going down.
2. Backend Throttling: Sometimes the storage backend (Azure Blob Storage) throttles requests if there is heavy load in that specific region, even if your specific repository isn't hitting a rate limit.

Here are a few things you can try to fix it:

• Update the Action: Ensure you are using actions/cache@v4. The newer version has improvements in how it handles compression and uploads compared to v3.
• Enable Debug Logs: Set the secret ACTIONS_STEP_DEBUG to true in your repository. Rerun the workflow, and you might see specific HTTP errors (like 503 or 504) that confirm if it is a server-side issue.
• Check Status: Since this started happening recently, it might be a regional degradation. If it persists for another day or two, you might want to open a ticket with GitHub Support.

Hopefully, it's just a temporary hiccup!