Jeffrey Carpenter

In this series, I’ve been sharing about my experience building my first Python application — an implementation of the KillrVideo microservice tier. In the previous posts, I’ve shared my <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">motivations for this project</a> and how I started things out by <a href="https://medium.com/@jscarp/python-app-dev-with-protobuf-and-grpc-e5bff779783d">building GRPC service stubs</a>. This time, I’ll dig into the next step in developing the application — advertising the service endpoints in <code>etcd</code>.

<img alt="KillrVideo Python GRPC etcd logos" data-entity-type="file" data-entity-uuid="ce3f8e68-1cfd-436d-9f05-b0c834d6b8ea" src="https://www.datastax.com/sites/default/files/inline-images/1_0F8byTpGAhRXHn75RX2cKA.png" />

<h2>What is <code>etcd</code>?</h2>

If you’re not familiar, <code>etcd</code> describes itself as a “distributed reliable key-value store for the most critical data of a distributed system.” I like the precision of this definition, but confess that I typically refer to it as a “service registry” since that is the role in which I’ve seen it used most often.

The documentation of the project is currently in flux as the project is in the process of <a href="https://www.cncf.io/blog/2018/12/11/cncf-to-host-etcd/">transitioning to the control of the Cloud Native Computing Foundation</a> (CNCF), but for the short term I’d recommend <a href="https://etcd.readthedocs.io/en/latest/">https://etcd.readthedocs.io/en/latest/</a>.

<h2>Etcd usage in KillrVideo — current state</h2>

Before explaining how we use <code>etcd</code> in <a href="https://github.com/KillrVideo/killrvideo-python"><code>killrvideo-python</code></a>, it will help to give some background. The KillrVideo system uses <code>etcd</code> to share the location of microservices as well as also database connection information for our Apache Cassandra (DataStax Enterprise) cluster.

There are two main ways which we deploy KillrVideo, both of which make use of Docker, but in slightly different ways. There’s a deployment that you can download and run on your desktop and a cloud deployment which we use for development and production. We use <code>etcd</code> in both cases.

<h2>Docker configuration and Registrator</h2>

The desktop configuration is described in thedocker-compose.yaml file found in the <a href="https://github.com/KillrVideo/killrvideo-docker-common">killrvideo-docker-common</a> repository. We start&nbsp;<code>etcd</code> like this:

<code>etcd: 
&nbsp; &nbsp;image: quay.io/coreos/etcd:v2.3.6 
&nbsp; &nbsp;command: [ -advertise-client-urls, "http://${KILLRVIDEO_DOCKER_IP}:2379", - 
listen-client-urls, "http://0.0.0.0:2379" ] 
&nbsp;&nbsp;ports: 
# The client port - "2379:2379" environment: 
&nbsp; SERVICE_2379_NAME: etcd </code>

If you look closely, you’ll also see that we are using a utility called <a href="https://github.com/gliderlabs/registrator"><code>registrator</code></a>, which reads environment variables in the the <code>docker-compose.yaml</code> file and registers services on behalf of the service:

<code># Registrator to register containers with Etcd registrator: 
&nbsp; &nbsp;image: gliderlabs/registrator:latest 
&nbsp; &nbsp;# Tell registrator where the etcd HTTP API is and to use 
&nbsp; &nbsp;# the docker VM's IP 
command: [ -ip, "$KILLRVIDEO_DOCKER_IP", "etcd://etcd:2379/killrvideo/services" ] 
&nbsp; &nbsp;volumes: 
&nbsp; &nbsp;# So registrator can use the docker API to inspect containers - "/var/run/docker.sock:/tmp/docker.sock" depends_on: 
&nbsp; &nbsp;- etcd </code>

For example, the following snip of code from one of our Docker Compose files shows how we use <code>registrator</code> to register the KillrVideo web app with <code>etcd</code>:

<code># Start the KillrVideo web UI on port 3000 &nbsp; 
web: &nbsp; &nbsp; 
&nbsp; image: killrvideo/killrvideo-web:2.1.0 
&nbsp; &nbsp; ports: 
&nbsp; &nbsp; - "3000:3000" &nbsp; &nbsp; 
&nbsp; &nbsp; depends_on: 
&nbsp; &nbsp; - dse 
&nbsp; &nbsp; - etcd 
&nbsp; &nbsp; environment: 
&nbsp; &nbsp; &nbsp; SERVICE_3000_NAME: web </code>

By exposing port 3000 and setting SERVICE_3000_NAME, we’re telling the <code>registrator</code>&nbsp;to register that service with&nbsp;<code>etcd</code>

<h2>Using a custom utility for service registration</h2>

This <code>registrator</code> is a very useful utility, but we don’t actually use it for registering our DSE cluster in etcd.

Instead, we created a custom utility called <code>killrvideo-dse-config</code> which is included in our <code>docker-compose.yaml</code> file. This utility has two main responsibilities. The first is to perform a one-time load of the schema into DSE, and the second is to register services provided by DSE into <code>etcd</code>.

The <code>killrvideo-dse-config</code> utility allows us to have more fine-grained control over the startup sequence — since DSE can take a bit to start up, we wait to register its services (Cassandra, Search, and Graph endpoints) with <code>etcd</code> until we are sure they are available. This helps keep the connection logic simple for clients that are looking up the database connection information.

Another thing to note about this utility is that it works for both the desktop deployment where we’re starting DSE within Docker, and the cloud deployment where we’re pointing at an existing cluster.

One thing that is not very sophisticated about our <code>etcd</code> integration in KillrVideo is that we make no provision for cleaning up its contents — we don’t ever delete any of the records or configure expiration. This is a shortcoming that we should address at some point.

<h2>Service naming conventions</h2>

All of the services that we register with <code>etcd</code> use a killrvideo/services namespace. The various services that are registered can be seen by using a browser to navigate to an HTTP interface exposed by the <code>etcd</code> server. For example, if you are running KillrVideo on your desktop, accessing the URL <a href="http://10.0.75.1:2379/v2/keys/killrvideo/services/">http://10.0.75.1:2379/v2/keys/killrvideo/services/</a> will show a list of all of the registered services.

<h2>Using etcd in KillrVideo Python</h2>

Now that you understand the overall picture of how <code>etcd</code> is used in KillrVideo, Let’s look at some of the implementation details involved using <code>etcd</code> to lookup and to advertise in the Python application. First, we’ll need the <code>python-etcd</code> library:

<code>pip install python-etcd </code>

Then we can do an <code>import etcd</code> in our application code in order to access the library. You can find this code in the main application file at the root of the module <code>(<a href="https://github.com/KillrVideo/killrvideo-python/blob/master/killrvideo/__init__.py">__init__.py</a>)</code>.

<h2>Finding database connection information</h2>

The first interaction that our application code has with <code>etcd</code> is to locate the information it will need to connect to DSE. In order to make the code more robust to the startup order of various components, we use a loop.

<code># Wait for Cassandra (DSE) to be up, aka registered in etcd while True:&nbsp; &nbsp; try: 
&nbsp; &nbsp; &nbsp;etcd_client.read('/killrvideo/services/cassandra') 
&nbsp; &nbsp; &nbsp;break # if we get here, Cassandra is available 
except etcd.EtcdKeyNotFound: 
&nbsp; &nbsp; &nbsp;logging.info('Waiting for Cassandra to be registered in etcd, sleeping 10s') 
&nbsp; &nbsp; &nbsp;time.sleep(10) </code>

A more elegant approach might be to use a callback, but this is sufficient for our purposes.

<h2>Registering services with etcd in Python</h2>

Once DSE is available, we first instantiate the <a href="https://docs.datastax.com/en/developer/python-driver-dse/2.7/">DataStax Python Driver</a> (which we’ll cover in a future post) and then our GRPC services. Once the service endpoints have been registered with the GRPC server, we’re ready to advertise the endpoints in <code>etcd</code>. The code do do this is pretty straightforward.

<code># Register Services with etcd 
etcd_client.write('/killrvideo/services/CommentsService/killrvideo-python', service_address) 
etcd_client.write('/killrvideo/services/RatingsService/killrvideo-python', service_address) 
etcd_client.write('/killrvideo/services/SearchService/killrvideo-python', service_address) 
etcd_client.write('/killrvideo/services/StatisticsService/killrvideo-python', service_address) etcd_client.write('/killrvideo/services/SuggestedVideoService/killrvideo-python', service_address) etcd_client.write('/killrvideo/services/UserManagementService/killrvideo-python', service_address) 
etcd_client.write('/killrvideo/services/VideoCatalogService/killrvideo-python', service_address)</code>

Notice that the path for each service endpoint follows the convention described above. You’ve probably also noted that the same&nbsp; <code>service_address</code>&nbsp;is used for each service (as discussed previously, it is fine to expose multiple endpoints on a single GRPC server). The&nbsp; <code>service_address&nbsp;</code>is defined by the hostname and port as follows:

<code>service_address = _SERVICE_HOST + ":" + _SERVICE_PORT </code>

For a simple Docker deployment of KillrVideo, this address will be <code>10.0.75.1:8899</code>.

<h2>Why everything I just wrote may become moot</h2>

One of my fellow advocates, <a href="https://twitter.com/HadesArchitect">Aleks Volochnev</a>, has been doing quite a bit of work on improving the initial developer experience of using KillrVideo. He’s rightly argued that the current experience is too oriented around getting set up to start coding, when most developers would rather just run the application first before diving into code.

As part of these efforts Aleks has been experimenting with removing the dependence on <code>etcd</code> and relying on DNS name resolution instead. Early results are good, but we’re still testing how well this approach scales to multiple service instances and adapts to different deployments before converting the entire project over.

We’ve also had other options on our roadmap to consider, including use of a service mesh. There are a number of competing service meshes emerging, but one common trait of these implementations is the usage of a service registry. So who knows, we may come full circle. Stay tuned…

<h2>What’s next</h2>

In this series of posts, I’ve been following the development of <code><a href="https://github.com/KillrVideo/killrvideo-python">killrvideo-python</a></code> somewhat chronologically. So far, I’ve described creating GRPC service stubs and registering the service endpoints in <code>etcd</code>. The next step: implementing the business logic of the services. In the next post I’ll share my approach to developing and testing this business logic, read part 4 of the KillrVideo series, Who Needs Unit Tests?&nbsp;I’m Building Microservices! <a href="https://www.datastax.com/blog/2019/04/killrvideo-python-pt-4-who-needs-unit-tests-im-building-microservices">here</a>.

This article is cross-posted from <a href="https://medium.com/datadriveninvestor/killrvideo-python-pt-1-the-backstory-5c38191fb330">Jeff's personal blog on Medium</a>.

KillrVideo Python Pt. 3 — Advertising Python Services via etcd

Jeffrey CarpenterSoftware Engineer - Stargate

Discover more

Share

More Technology

Open-Source,
Scale-Out, Cloud-Native
NoSQL Database

Jeffrey CarpenterSoftware Engineer - Stargate

Discover more

Share

More Technology

Open-Source,Scale-Out, Cloud-NativeNoSQL Database

Open-Source,
Scale-Out, Cloud-Native
NoSQL Database