Redis is an in-memory key-value store. Redis can be operated as a simple store for managing (notably) volatile persistent data. Redis can operate in serveral modes, ranging from a single server to clusters organised in several different ways to provide high availability, resilience and replicating the data close to the involved servers. In addition to being a key-value store, Redis enables additional communication between clients such as publish/subscribe to message channels, streams, etc.
These features can be used to connect micro services, both for sharing state, notifications and distributing tasks.
The connection between the redis client and server uses a stream pair. Although SWI-Prolog stream I/O is thread-safe, having multiple threads using this same connection will mixup writes and their replies.
At the moment, the following locking is in place.
- Connections created using redis_connect/3 are not locked. This implies the connection handle may be used from a single thread only, or redis/3 requests must be protected using with_mutex/2.
- Redis/3 request using a server name established using redis_server/3 are locked using a mutex with the same name as the server name.
If SWI-Prolog includes the
ssl library, the Redis client
can connect to the server using TLS (SSL). Connecting requires the same
three files as
redis-cli requires: the root certificate
file, a client certificate and the private key of the client
certificate. Below is an example call to redis_server/3:
:- redis_server(swish, localhost:6379, [ user(bob), password("topsecret"), version(3), tls(true), cacert('ca.crt'), key('client.key'), cert('client.cert') ]).
Redis sentinels is one of the two options to create a high
availability service. It consists of minimally three Redis servers and
mininally three sentinel servers. The sentinel servers monitor the Redis
servers and will initiate a fail-over when the master becomes
disfunctional and certain safety constraints are satisfied. A client
needs to be aware of this setup. It is given an initial list with (a
subset of) the known sentinels. The client attempts to connect to one of
the sentinels and ask it for the current Redis master server. Details
are described in Sentinel
client spec. The SWI-Prolog client maintains the actual list of
sentinels dynamically after successful discovery of the first sentinel.
Below is an example
redis_server/3 to connect to
a sentinel network. The Address specification
sentinel(swish) tells the library we want to connect to a
sentinel network that is monitored under the name
:- redis_server(swish_sentinel, sentinel(swish), [ user(janbob), password("topsecret"), version(3), sentinels([ host1:26379, host2:26379, host3:26379 ]) ]).
The current stable version of Redis is 6. Many Linux distros still ship with version 5. Both talk protocol version 2. Version 6 also supports protocol version 3. The main differences are:
- The version 3 protocol has several improvements that notably improvement passing large objects using a streaming protocol.
- Hashes (maps) in the version 3 protocol are exchanged as lists of pairs
Name-Value), while version 2 exchanges hashes as a list of alternating names and values. This is visible to the user. High level predicates such as redis_get_hash/3 deal with both representations.
- The version 3 protocol supports push messages to deal with monitor and subscribe events on the same connection as used for handling normal requests.
New projects are encouraged to use Redis version 6 with the version 3 protocol. See redis_server/3.
Starting with Redis 5, redis supports streams. A stream is a
list of messages. Streams can be used as a reliable alternative to the
older Redis PUB/SUB (Publish Subscribe) mechanism that has no memory,
i.e., if a node is down when a message arrives the message is missed. In
addition, they can be used to have each message processed by a
consumer that belongs to a consumer group. Both facilities
are supported by
Redis streams provide all the low-level primitives to realise message brokering. Putting it all together is non-trivial though. Notably:
- We must take care of messages that have been sent to some consumer but the consumer fails to process the message and (thus) ACK it is processed. This is handled by xlisten_group/5 using several options. Good defaults for these options are hard to give as it depends on the required processing time for a message, how common failures are and an acceptable delay time in case of a failure, what to do in case of a persistent failure, etc.
- Streams are independent from consumer groups and acknowledged messages remain in the stream. xstream_set/3 can be used to limit the length of the stream, discarding the oldest messages. However, it is hard to give a sensible default. The required queue length depends on the the size of the messages, whether messages come in more or less randomly or in bursts (that cause the stream to grow for a while), available memory, how bad it is if some messages get lost, etc.
doc/packages/examples/redis in the
installation provides an example using streams and consumer groups to
realise one or more clients connected to one or more compute nodes.
This module is based on the
gpredis.pl by Sean Charles
for GNU-Prolog. This file greatly helped me understanding what had to be
done, although, eventually, not much of the original interface is left.
The main difference to the original client are:
- Replies are not wrapped by type in a compound term.
- String replies use the SWI-Prolog string type.
- Values can be specified as
Value as prolog, after which they are returns as a (copy of) Value. This prefixes the value using "
- Strings are in UTF-8 encoding to support full Unicode.
- Using redis_server/3, actual connections are established lazily and when a connection is lost it is automatically restarted.
- This library allows for using the Redis publish/subscribe interface. Messages are propagated using broadcast/1.