What Is Huginn?
Huginn is kind of like an open source version of IFTTT. It allows you to create agents/robots which act on your behalf to periodically perform any number of tasks--from notifying you of increased Twitter chatter about your favorite half-baked politician, to watching for
airline ticket deals which airlines are still in business. Huginn is a fun and useful little utility.
What Is the Point of This Article?
In this post, I’m going to show you how to use Huggin to periodically search for “cheap” apartments in San Francisco.
In the last article, we covered concepts like deployments, services, pods, secrets, volumes, PVCs, ingresses, Nextcloud, and Minikube. It was intended to give you a crash course in Kubernetes.
In this article, we're going to cover some additional topics like Kubernetes ConfigMaps, Amazon Simple Email Service, Huginn, and name-based virtual hosting.
Source Code (Optional)
If you want to get your hands on the code, click on the Github link below. Be sure to
git checkout the
huginn tag to view the finished code. Note that in future posts, I will add additional projects to the repo and making changes that aren't reflected in this post.
The Huginn Database Config File
Here we have a Kubernetes deployment named
This deployment makes sure that exactly one
huginn-db pod is running at all times. The pod contains a single container created from the
postgres:12 Docker image. That’s the DB which backs our Huginn server.
In order to ensure that the DB data will survive a container crash, we need to mount a volume onto the directory where PostgreSQL stores the data.
The Persistent Volume Claim
Now we need to declare a persistent volume claim with a name that matches the value of the
claimName field in the DB deployment.
This PVC requests 256Mi of persistent storage--e.g. hard disks or SSD--from the Kubernetes cluster. This is the storage that backs our database volume.
In the last article, the PVC was used by both the database pod and the server pod. Thus, it seemed appropriate to put the PVC in its own file.
But this time we are going to put the PVC into the Huginn database config file because the Huginn server won’t be sharing the volume.
Just like the last post, our DB container is missing some important environment variables that are used to configure PostgreSQL.
The environment variable
POSTGRES_DB=huginn sets the name of the DB. And
envFrom… extracts the PostgreSQL root user’s credentials from the Kubernetes secret object named
huginn-db-secret. (We'll create that secret in a minute.)
This is the
huginn-db service configuration code.
All packets sent to port
5432 on the service will get forwarded to port
5432 on the
huginn-db-pod. Thanks to the service, clients can now reliably communicate with the DB pod--whether or not the pod gets replaced.
Create the Deployment, Secret, Service, and PVC
We are going to test everything out using Minikube, the single-node development cluster. If you don't already have it installed, go here: https://kubernetes.io/docs/tasks/tools/install-minikube.
Now, start up Minikube:
Then create the secret:
kubectl create secret generic huginn-db-secret \ --from-literal=POSTGRES_USER=root \ --from-literal=POSTGRES_PASSWORD=rootPassw0rd1
Then create the PVC, deployment, and service for the PostgreSQL database:
kubectl apply -f huginn-db.yaml
It will take a few minutes the first time you do this because your cluster has to download the
postgre:12 image from Docker Hub.
Huginn Server Config File
Okay, here is the deployment for the Huginn server. It creates a single pod which runs a container created from the
Environment Variables and ConfigMaps
The purpose of ConfigMaps is to provide a place where you can store non-secret properties and environment variables. While we could define those environment variables directly on the target container, doing so would make the deployment less readable. So we're going to store all non-sensitive environment variables in a configMap instead.
Here's the configMap:
This configMap stores four environment variables:
Notice that the database port number is surrounded by double quotes. This is necessary because all configMap properties must have string values and YAML would interpret
5432 without quotes as a number.
Back in the server container code, the
envFrom.configMapRef field tells the
huginn container to extract the configMap properties and turn them into environment variables.
Amazon Simple Email Service
One of the things that Huginn can do is notify us via email when one of its agents finds something on the Internet or finishes a task. However, it can't send emails on its own. We need to configure it to use the email provider of our choice to enable this feature.
We could use Gmail, but there are some problems with that plan. Google has been known to abruptly suspend people’s accounts for using Gmail to send too many transactional emails. Therefore, we are going to use Amazon SES instead.
I picked Amazon SES over other transactional email providers for two reasons:
- it's cheap
- and it's a popular tool which makes it worth learning.
Amazon SES Setup
Let me show you how to setup Amazon SES to send transactional emails from one of your email addresses.
Step 1: Signup/login
First, we need to sign in to the AWS Management Console.
- Go to the Amazon SES home page.
- Click the Get started with Amazon SES button.
- Create a new account OR log in with an existing AWS account.
Step 2: Create SMTP credentials
To send emails through AWS Simple Email Service's SMTP interface, we need to create some SMTP credentials.
Click on the SMTP Settings link.
Click on the Create My SMTP Credentials button.
Save the credentials because Amazon won't display them again!
Step 3: Verify your email address
Next, we need to give Amazon the email address from which we want it to send emails. Then we need to verify that we actually own that email address. Only then will Amazon SES send our emails.
Click on the Email Addresses button under Identity Management.
(Note that email-smtp.us-west-2.amazonaws.com is the server that I use. Your server might be different if you're in a different AWS region.)
Click on the Verify a New Email Address button.
Fill in the email address and click the Verify This Email Address button.
Go to the email inbox for that address, and click on the link to verify that you are the owner.
Step 4: Remove Amazon SES's "Send" Restrictions
New SES accounts are locked down by default. They can't send emails to unverified email addresses, and they're limited to a maximum of 200 messages in a 24-hour period. This is done to stop fly-by-night spammers.
To remove the restrictions, you will need to create a support ticket that requests their removal. Here are the official instructions to do that.
Note: For this project, Huginn will only use Amazon SES to send us notification emails. So long as we don't anticipate sending 200+ messages a day, we can ignore Step 4. Just make sure that you verify the email address that will receive the Huginn notifications.
Configure Huginn to Use Amazon SES
In the previous section, we prepared an SES account for Huginn. Now we need to configure the
huginn-server deployment to use that account.
Add the following five environment variables to the Huginn server configMap.
SMTP_DOMAIN: amazonaws.com SMTP_PORT: "587" SMTP_AUTHENTICATION: plain SMTP_ENABLE_STARTTLS_AUTO: "true" SEND_EMAIL_IN_DEVELOPMENT: "true"
You'll also need to add the following two environment variables. (Be sure to use your own values.)
SMTP_SERVER: <your Amazon SMTP server> EMAIL_FROM_ADDRESS: <your sending address>
Note that you can get the address of your Amazon SES server on the Management Console.
Here's what the server configMap looks like now.
Note that we had to double quote
"true" to force them to be treated as YAML strings, since all configMap values must be strings.
The Huginn Server Secret Object
While the configMap contains all of the server's non-sensitive environment variables, we shouldn't use it to store things like database credentials and Amazon SMTP credentials. That's what Kubernetes Secret objects are for.
kubectl command to create that secret object named
kubectl create secret generic huginn-server-secret \ --from-literal=DATABASE_USERNAME=root \ --from-literal=DATABASE_PASSWORD=rootPassw0rd1 \ --from-literal=SMTP_USER_NAME=BobLaBlah \ --from-literal=SMTP_PASSWORD=blahblahblah/blah
Finally, we need to configure the
huginn-server container to extract the sensitive environment variables from
huginn-server-secret. That's what the
secretRef section does.
That's it. The Huginn server has been configured to use Amazon SES to send notification emails.
Give the Huginn Server Pods a Kubernetes Service
Like usual, we need to have a service which sits in front of the Huginn server pods, and acts as a communications proxy. Therefore, let's append the service's YAML config document to the end of the file.
Note that the Huginn listens on port
3000 for messages forwarded by the
Create the Secret, ConfigMap, Deployment, and Service for the Huginn Server
We've created the config files for the database and the server. Now let's use them to create the DB and server in our Minikube cluster.
- If you haven't already done so, create the Huginn server secret. (Be sure to replace the SMTP username and password below!)
kubectl create secret generic huginn-server-secret \ --from-literal=DATABASE_USERNAME=root \ --from-literal=DATABASE_PASSWORD=rootPassw0rd1 \ --from-literal=SMTP_USER_NAME=your-username \ --from-literal=SMTP_PASSWORD=your-password
- Then create all the other server components.
kubectl apply -f huginn-server.yaml
Part 1 Summary
In this article, we used Kubernetes to create the Huginn server and its database. In the next article, I'm going to show you two more things:
- How to use Huginn to search for apartments on Craigslist.
- And how to host Huginn and Nextcloud (from the previous article) on the same cluster using an Ingress object.
Thanks for reading this far. I hope you're excited to read Part 2 where we're gonna use all this hard work to go apartment hunting!