Using StackExchange.StringSetAsync to set multiple keys with the same expiration

Question

We are currently using the StackExchange Redis library to handle our Redis-database but came across a little snag. Please keep in mind that we are new to Redis, so there might be a better way to do this entirely.

Our requirement is to store about 10000 keys as quickly as possible, but all with the same expiration time. The expiration is not really a requirement, only that the database should not grow over time, and the next update might be 10100 keys, but only 90% of the keys will be same, or maybe the next update is only 9000 keys. So in effect some of the keys might never be visited again, or overridden.

I could use a diff to find all the keys from last time and remove them manually, but that seems unnecessary complicated when we have the concept of expiration as well as a redis actively removing expired keys.

When using the StringSetAsync I can send all keys at once and the operation is completed very quickly, but I cannot set the expiration-time, only parameters for When and CommandFlag is possible.

If I use StringSetAsync to add the keys one-by-one, I get the expiration parameter, but the process takes a magnitude more time.

Is there an good way to do this?

Update your question to show your current code.

mjwills
– mjwills

2025-09-13 05:55:09 +00:00
Commented Sep 13 at 5:55 — mjwills
– mjwills, Commented Sep 13 at 5:55

Michał Turczyn · Accepted Answer · 2025-09-12 07:49:32Z

0

You could try setting keys with expiration in parallel.

You just need to keep in mind throttling, to not send all requests at the same time.

Sample implementation:

var keys = new string[] { "key1", };

await Parallel.ForEachAsync(
    keys,
    // To control throttling
    new ParallelOptions { MaxDegreeOfParallelism = 4 },
    async (k, cancellationToken) => await _redis.StringSetAsync(k, ..., cancellationToken));

answered Sep 12 at 7:49

Michał Turczyn

41.2k18 gold badges58 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Marc Gravell · Accepted Answer · 2025-09-15 10:50:11Z

There are some Redis limitations in play here. The SET command takes a single key/value pair and allows expiration. The MSET command takes multiple key/value pairs, but does not allow expiration.

I suspect the main problem at the moment is that you're observing performance while paying latency per pair, but Redis is remarkably good at pipelining operations, i.e. firing multiple commands simultaneously without waiting for each reply. There are 3 common ways of doing this:

"fire and forget" is trivial to implement via CommandFlags (on every SE.Redis command); this means the library doesn't track any part of the reply - which is usually fine, but it does mean that if an unexpected error occurs: you won't see it. This is usually fine if you can validate the data and re-issue it in the unlikely case of a problem
batching (CreateBatch) allows multiple commands to be sent as a large chunk; in current v2 SE.Redis library versions this works ok-ish, but it gets much, much more effective when I finish the v3 overhaul
you can use a queue of incomplete operations, only awaiting when you've reached the limit of what you want in flight

The basic gist of the last two options is shown here, noting that the API usage is slightly atypical because this bit of code is abstracted to work with 3 different library implementations.

Collectives™ on Stack Overflow

Using StackExchange.StringSetAsync to set multiple keys with the same expiration

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related