In this series, I’ve been sharing about my experience building my first Python application — an implementation of the KillrVideo microservice tier. In the previous posts, I’ve shared my motivations for this project and how I started things out by building GRPC service stubs. This time, I’ll dig into the next step in developing the application — advertising the service endpoints in etcd.
What is etcd?
If you’re not familiar, etcd describes itself as a “distributed reliable key-value store for the most critical data of a distributed system.” I like the precision of this definition, but confess that I typically refer to it as a “service registry” since that is the role in which I’ve seen it used most often.
The documentation of the project is currently in flux as the project is in the process of transitioning to the control of the Cloud Native Computing Foundation (CNCF), but for the short term I’d recommend https://etcd.readthedocs.io/en/latest/.
Etcd usage in KillrVideo — current state
Before explaining how we use etcd in killrvideo-python, it will help to give some background. The KillrVideo system uses etcd to share the location of microservices as well as also database connection information for our Apache Cassandra (DataStax Enterprise) cluster.
There are two main ways which we deploy KillrVideo, both of which make use of Docker, but in slightly different ways. There’s a deployment that you can download and run on your desktop and a cloud deployment which we use for development and production. We use etcd in both cases.
Docker configuration and Registrator
The desktop configuration is described in thedocker-compose.yaml file found in the killrvideo-docker-common repository. We start etcd like this:
etcd:
image: quay.io/coreos/etcd:v2.3.6
command: [ -advertise-client-urls, "http://${KILLRVIDEO_DOCKER_IP}:2379", -
listen-client-urls, "http://0.0.0.0:2379" ]
ports:
# The client port - "2379:2379" environment:
SERVICE_2379_NAME: etcd
If you look closely, you’ll also see that we are using a utility called registrator, which reads environment variables in the the docker-compose.yaml file and registers services on behalf of the service:
# Registrator to register containers with Etcd registrator:
image: gliderlabs/registrator:latest
# Tell registrator where the etcd HTTP API is and to use
# the docker VM's IP
command: [ -ip, "$KILLRVIDEO_DOCKER_IP", "etcd://etcd:2379/killrvideo/services" ]
volumes:
# So registrator can use the docker API to inspect containers - "/var/run/docker.sock:/tmp/docker.sock" depends_on:
- etcd
For example, the following snip of code from one of our Docker Compose files shows how we use registrator to register the KillrVideo web app with etcd:
# Start the KillrVideo web UI on port 3000
web:
image: killrvideo/killrvideo-web:2.1.0
ports:
- "3000:3000"
depends_on:
- dse
- etcd
environment:
SERVICE_3000_NAME: web
By exposing port 3000 and setting SERVICE_3000_NAME, we’re telling the registrator to register that service with etcd
Using a custom utility for service registration
This registrator is a very useful utility, but we don’t actually use it for registering our DSE cluster in etcd.
Instead, we created a custom utility called killrvideo-dse-config which is included in our docker-compose.yaml file. This utility has two main responsibilities. The first is to perform a one-time load of the schema into DSE, and the second is to register services provided by DSE into etcd.
The killrvideo-dse-config utility allows us to have more fine-grained control over the startup sequence — since DSE can take a bit to start up, we wait to register its services (Cassandra, Search, and Graph endpoints) with etcd until we are sure they are available. This helps keep the connection logic simple for clients that are looking up the database connection information.
Another thing to note about this utility is that it works for both the desktop deployment where we’re starting DSE within Docker, and the cloud deployment where we’re pointing at an existing cluster.
One thing that is not very sophisticated about our etcd integration in KillrVideo is that we make no provision for cleaning up its contents — we don’t ever delete any of the records or configure expiration. This is a shortcoming that we should address at some point.
Service naming conventions
All of the services that we register with etcd use a killrvideo/services namespace. The various services that are registered can be seen by using a browser to navigate to an HTTP interface exposed by the etcd server. For example, if you are running KillrVideo on your desktop, accessing the URL http://10.0.75.1:2379/v2/keys/killrvideo/services/ will show a list of all of the registered services.
Using etcd in KillrVideo Python
Now that you understand the overall picture of how etcd is used in KillrVideo, Let’s look at some of the implementation details involved using etcd to lookup and to advertise in the Python application. First, we’ll need the python-etcd library:
pip install python-etcd
Then we can do an import etcd in our application code in order to access the library. You can find this code in the main application file at the root of the module (__init__.py).
Finding database connection information
The first interaction that our application code has with etcd is to locate the information it will need to connect to DSE. In order to make the code more robust to the startup order of various components, we use a loop.
# Wait for Cassandra (DSE) to be up, aka registered in etcd while True: try:
etcd_client.read('/killrvideo/services/cassandra')
break # if we get here, Cassandra is available
except etcd.EtcdKeyNotFound:
logging.info('Waiting for Cassandra to be registered in etcd, sleeping 10s')
time.sleep(10)
A more elegant approach might be to use a callback, but this is sufficient for our purposes.
Registering services with etcd in Python
Once DSE is available, we first instantiate the DataStax Python Driver (which we’ll cover in a future post) and then our GRPC services. Once the service endpoints have been registered with the GRPC server, we’re ready to advertise the endpoints in etcd. The code do do this is pretty straightforward.
# Register Services with etcd
etcd_client.write('/killrvideo/services/CommentsService/killrvideo-python', service_address)
etcd_client.write('/killrvideo/services/RatingsService/killrvideo-python', service_address)
etcd_client.write('/killrvideo/services/SearchService/killrvideo-python', service_address)
etcd_client.write('/killrvideo/services/StatisticsService/killrvideo-python', service_address) etcd_client.write('/killrvideo/services/SuggestedVideoService/killrvideo-python', service_address) etcd_client.write('/killrvideo/services/UserManagementService/killrvideo-python', service_address)
etcd_client.write('/killrvideo/services/VideoCatalogService/killrvideo-python', service_address)
Notice that the path for each service endpoint follows the convention described above. You’ve probably also noted that the same service_address is used for each service (as discussed previously, it is fine to expose multiple endpoints on a single GRPC server). The service_address is defined by the hostname and port as follows:
service_address = _SERVICE_HOST + ":" + _SERVICE_PORT
For a simple Docker deployment of KillrVideo, this address will be 10.0.75.1:8899.
Why everything I just wrote may become moot
One of my fellow advocates, Aleks Volochnev, has been doing quite a bit of work on improving the initial developer experience of using KillrVideo. He’s rightly argued that the current experience is too oriented around getting set up to start coding, when most developers would rather just run the application first before diving into code.
As part of these efforts Aleks has been experimenting with removing the dependence on etcd and relying on DNS name resolution instead. Early results are good, but we’re still testing how well this approach scales to multiple service instances and adapts to different deployments before converting the entire project over.
We’ve also had other options on our roadmap to consider, including use of a service mesh. There are a number of competing service meshes emerging, but one common trait of these implementations is the usage of a service registry. So who knows, we may come full circle. Stay tuned…
What’s next
In this series of posts, I’ve been following the development of killrvideo-python somewhat chronologically. So far, I’ve described creating GRPC service stubs and registering the service endpoints in etcd. The next step: implementing the business logic of the services. In the next post I’ll share my approach to developing and testing this business logic, read part 4 of the KillrVideo series, Who Needs Unit Tests? I’m Building Microservices! here.
This article is cross-posted from Jeff's personal blog on Medium.
