The Huginn and Kubernetes logos.
The Huginn and Kubernetes logos.

Hello! My name is Nick and I'm going to show you how to self-host your own Huginn server using Kubernetes and Amazon’s Simple Email Service.

What Is Huginn?

Huginn is kind of like an open source version of IFTTT. It allows you to create agents/robots which act on your behalf to periodically perform any number of tasks--from notifying you of increased Twitter chatter about your favorite half-baked politician, to watching for airline ticket deals which airlines are still in business. Huginn is a fun and useful little utility.

What Is the Point of This Article?

In this post, I’m going to show you how to use Huggin to periodically search for “cheap” apartments in San Francisco.

In the last article, we covered concepts like deployments, services, pods, secrets, volumes, PVCs, ingresses, Nextcloud, and Minikube. It was intended to give you a crash course in Kubernetes.

In this article, we're going to cover some additional topics like Kubernetes ConfigMaps, Amazon Simple Email Service, Huginn, and name-based virtual hosting.

Source Code (Optional)

If you want to get your hands on the code, click on the Github link below. Be sure to git checkout the huginn tag to view the finished code. Note that in future posts, I will add  additional projects to the repo and making changes that aren't reflected  in this post.

nick-true-dev/usable-k8s-projects
Three projects that showcase Kubernetes. Project 1 uses K8s to host a Nextcloud server. Project 2 uses K8s to host a Huginn server. And project 3 uses K8s to host a MonicaHQ server. - nick-true-dev...

The Huginn Database Config File

Here we have a Kubernetes deployment named huginn-db.

huginn-db-1

This deployment makes sure that exactly one huginn-db pod is running at all times. The pod contains a single container created from the postgres:12 Docker image. That’s the DB which backs our Huginn server.

In order to ensure that the DB data will survive a container crash, we need to mount a volume onto the directory where PostgreSQL stores the data.

huginn-db-1a

The Persistent Volume Claim

Now we need to declare a persistent volume claim with a name that matches the value of the claimName field in the DB deployment.

huginn-db-1b-2

This PVC requests 256Mi of persistent storage--e.g. hard disks or SSD--from the Kubernetes cluster. This is the storage that backs our database volume.

In the last article, the PVC was used by both the database pod and the server pod. Thus, it seemed appropriate to put the PVC in its own file.

But this time we are going to put the PVC into the Huginn database config file because the Huginn server won’t be sharing the volume.

Environment Variables

Just like the last post, our DB container is missing some important environment variables that are used to configure PostgreSQL.

huginn-db-1c

The environment variable POSTGRES_DB=huginn sets the name of the DB. And envFrom… extracts the PostgreSQL root user’s credentials from the Kubernetes secret object named huginn-db-secret. (We'll create that secret in a minute.)

DB Service

This is the huginn-db service configuration code.

huginn-db-1d

All packets sent to port 5432 on the service will get forwarded to port 5432 on the huginn-db-pod. Thanks to the service, clients can now reliably communicate with the DB pod--whether or not the pod gets replaced.

Create the Deployment, Secret, Service, and PVC

We are going to test everything out using Minikube, the single-node development cluster. If you don't already have it installed, go here: https://kubernetes.io/docs/tasks/tools/install-minikube.

Now, start up Minikube:

minikube start

Then create the secret:

kubectl create secret generic huginn-db-secret \
    --from-literal=POSTGRES_USER=root \
    --from-literal=POSTGRES_PASSWORD=rootPassw0rd1

Then create the PVC, deployment, and service for the PostgreSQL database:

kubectl apply -f huginn-db.yaml

It will take a few minutes the first time you do this because your cluster has to download the postgre:12 image from Docker Hub.


Huginn Server Config File

Okay, here is the deployment for the Huginn server. It creates a single pod which runs a container created from the huginn/huginn:b915466f432a275fa1fd4f599650ea7f3009fd8fimage.

huginn-server-1

Environment Variables and ConfigMaps

The purpose of ConfigMaps is to provide a place where you can store non-secret properties and environment variables. While we could define those environment variables directly on the target container, doing so would make the deployment less readable. So we're going to store all non-sensitive environment variables in a configMap instead.

Here's the configMap:
huginn-server-1a

This configMap stores four environment variables:
huginn-server-1b

Notice that the database port number is surrounded by double quotes. This is necessary because all configMap properties must have string values and YAML would interpret 5432 without quotes as a number.

Back in the server container code, the envFrom.configMapRef field tells the huginn container to extract the configMap properties and turn them into environment variables.

huginn-server-1c

Amazon Simple Email Service

One of the things that Huginn can do is notify us via email when one of its agents finds something on the Internet or finishes a task. However, it can't send emails on its own. We need to configure it to use the email provider of our choice to enable this feature.

We could use Gmail, but there are some problems with that plan. Google has been known to abruptly suspend people’s accounts for using Gmail to send too many transactional emails. Therefore, we are going to use Amazon SES instead.

I picked Amazon SES over other transactional email providers for two reasons:

  1. it's cheap
  2. and it's a popular tool which makes it worth learning.

Amazon SES Setup

Let me show you how to setup Amazon SES to send transactional emails from one of your email addresses.

Step 1: Signup/login

First, we need to sign in to the AWS Management Console.

  1. Go to the Amazon SES home page.
  2. Click the Get started with Amazon SES button.
  3. Create a new account OR log in with an existing AWS account.

Step 2: Create SMTP credentials

To send emails through AWS Simple Email Service's SMTP interface, we need to create some SMTP credentials.

  1. Click on the SMTP Settings link.

    amazon-ses-1

  2. Click on the Create My SMTP Credentials button.

    amazon-ses-1a

  3. Save the credentials because Amazon won't display them again!

Step 3: Verify your email address

Next, we need to give Amazon the email address from which we want it to send emails. Then we need to verify that we actually own that email address. Only then will Amazon SES send our emails.

  1. Click on the Email Addresses button under Identity Management.

    amazon-ses-1b-1
    (Note that email-smtp.us-west-2.amazonaws.com is the server that I use. Your server might be different if you're in a different AWS region.)

  2. Click on the Verify a New Email Address button.

    amazon-ses-1c-2

  3. Fill in the email address and click the Verify This Email Address button.

    amazon-ses-1d-1

  4. Go to the email inbox for that address, and click on the link to verify that you are the owner.

Step 4: Remove Amazon SES's "Send" Restrictions

New SES accounts are locked down by default. They can't send emails to unverified email addresses, and they're limited to a maximum of 200 messages in a 24-hour period. This is done to stop fly-by-night spammers.

To remove the restrictions, you will need to create a support ticket that requests their removal. Here are the official instructions to do that.

Note: For this project, Huginn will only use Amazon SES to send us notification emails. So long as we don't anticipate sending 200+ messages a day, we can ignore Step 4. Just make sure that you verify the email address that will receive the Huginn notifications.

Configure Huginn to Use Amazon SES

In the previous section, we prepared an SES account for Huginn. Now we need to configure the huginn-server deployment to use that account.

Add the following five environment variables to the Huginn server configMap.

  SMTP_DOMAIN: amazonaws.com
  SMTP_PORT: "587"
  SMTP_AUTHENTICATION: plain
  SMTP_ENABLE_STARTTLS_AUTO: "true"
  SEND_EMAIL_IN_DEVELOPMENT: "true"

You'll also need to add the following two environment variables. (Be sure to use your own values.)

  SMTP_SERVER: <your Amazon SMTP server>
  EMAIL_FROM_ADDRESS: <your sending address>

Note that you can get the address of your Amazon SES server on the Management Console.

amazon-ses-1e-2

Here's what the server configMap looks like now.

huginn-server-1d

Note that we had to double quote "587" and "true" to force them to be treated as YAML strings, since all configMap values must be strings.

The Huginn Server Secret Object

While the configMap contains all of the server's non-sensitive environment variables, we shouldn't use it to store things like database credentials and Amazon SMTP credentials. That's what Kubernetes Secret objects are for.

Here's the kubectl command to create that secret object named huginn-server-secret.

kubectl create secret generic huginn-server-secret \
    --from-literal=DATABASE_USERNAME=root \
    --from-literal=DATABASE_PASSWORD=rootPassw0rd1 \
    --from-literal=SMTP_USER_NAME=BobLaBlah \
    --from-literal=SMTP_PASSWORD=blahblahblah/blah

Finally, we need to configure the huginn-server container to extract the sensitive environment variables from huginn-server-secret. That's what the secretRef section does.

huginn-server-1e

That's it. The Huginn server has been configured to use Amazon SES to send notification emails.

Give the Huginn Server Pods a Kubernetes Service

Like usual, we need to have a service which sits in front of the Huginn server pods, and acts as a communications proxy. Therefore, let's append the service's YAML config document to the end of the file.

huginn-server-1f

Note that the Huginn listens on port 3000 for messages forwarded by the huginn-server service.

Create the Secret, ConfigMap, Deployment, and Service for the Huginn Server

We've created the config files for the database and the server. Now let's use them to create the DB and server in our Minikube cluster.

  1. If you haven't already done so, create the Huginn server secret. (Be sure to replace the SMTP username and password below!)
    kubectl create secret generic huginn-server-secret \
        --from-literal=DATABASE_USERNAME=root \
        --from-literal=DATABASE_PASSWORD=rootPassw0rd1 \
        --from-literal=SMTP_USER_NAME=your-username \
        --from-literal=SMTP_PASSWORD=your-password
    
  2. Then create all the other server components.
    kubectl apply -f huginn-server.yaml
    

Part 1 Summary

In this article, we used Kubernetes to create the Huginn server and its database. In the next article, I'm going to show you two more things:

  1. How to use Huginn to search for apartments on Craigslist.
  2. And how to host Huginn and Nextcloud (from the previous article) on the same cluster using an Ingress object.

Thanks for reading this far. I hope you're excited to read Part 2 where we're gonna use all this hard work to go apartment hunting!