30 Jun 2014
Late last year, octohost was created as a system to host websites:
- With no or minimal levels of manual intervention.
- With very little regard to underlying technology or framework.
- As a personal mini-PaaS modeled after Heroku with a
git push interface to deploy these sites.
- Using disposable, immutable and rebuildable containers of source code.
What have we found?
- Docker is and incredible tool to take containers on Linux to the next level.
- If you keep your containers simple and ruthlessly purge unnecessary features, they can run uninterrupted for long periods of time.
- Having the ability to install anything in a disposable container is awesome.
- You can utilize your server resources much more efficiently using containers to host individual websites.
As we've been using it, we've also been thinking about ways to make it better:
- How can we make it faster?
- How can we make it simpler and more reliable?
- How big can we make it? How many sites can we put on a single server?
- How can we combine multiple octohosts together as a distributed cluster that's bigger and more fault-tolerant than a single one?
- How can we run the same container on different octohosts for fault-tolerance and additional scalability for a particular website?
- How can we persist configuration data beyond the lifecycle of the disposable container?
- How can we distribute and make this configuration data available around the system?
- How can we integrate remote data stores so that we can still keep the system itself relatively disposable?
- How can we trace an HTTP request through the entire chain from the proxy, to container and back?
- How can we lower the barrier to entry so that it can be built/spun up easier?
A number of these have been 'accomplished', but we've done a number of large changes to help to enable the next phases of octohost's lifecycle.
- We replaced the Hipache proxy with Openresty which immediately sped everything up and allowed us to use Lua to extend the proxy's capabilities.
- We moved from etcd to Consul to store and distribute our persistent configuration data. That change allowed us to make use of Consul's Services and Health Check features.
- We removed the tentacles container which used Ruby, Sinatra and Redis to store a website's endpoint. Due to how it was hooked up to nginx, it was queried for every hit so that it knew which endpoing to route the request to. The data model was also limited to a single endpoint and required a number of moving parts. I like less moving parts - removing it was a win in many ways.
- We refactored the
octo command and the gitreceive script which enabled the launching of multiple containers for a single site.
- We added a configuration flag to use a private registry, so that an image only has to be built once and can be pulled onto other members of the cluster quickly and easily.
- We added a plugin architecture for the
octo command, and the first plugin was for MySQL user and database creation.
- We replaced tentacles with the octoconfig gem that pulls the Service and configuration data out of Consul and writes an nginx config file. The gem should be extensible enough that we can re-use it for other daemons as needed.
So what are we working on going forward?
- Getting octohost clustered easily and reliably. At a small enough size and workload, each system should be able to proxy for any container in the cluster.
- Working on the movement, co-ordination and duplication of containers from octohost to octohost.
- Improving the consistency and efficiency of octohost's current set of base images. We will be starting from Ubuntu 14.04LTS and rebuilding from there.
- Continuing to improve the traceability of HTTP requests through the proxy, to the container and back.
- Improving the performance wherever bottlenecks are found.
- Improving the documentation and setup process.
What are some pain points that you've found? What do you think of our plans?
Send any comments to Darron or hit us up on Twitter.
13 Jun 2014
I'm converting a number of old websites that were using mod_auth_mysql - which doesn't work anymore - and was having a hard time finding clear, concise and working information.
First off - DO NOT INSTALL libapache2-mod-auth-mysql - it doesn't work. I'm not even sure why it's in Ubuntu anymore, it doesn't even work with Apache 2.4.
Here's how to do get Apache 2.4 / mod_authn_dbd and MySQL to play nice together:
apt-get install apache2 apache2-utils
apt-get install mysql-server-5.6
apt-get install libaprutil1-dbd-mysql
Create a MySQL user that you can query your databases with.
Once that's done, let's setup the global dbd_mysql configuration in this file
DBDParams "host=127.0.0.1 port=3306 user=username_here pass=password_here"
Now you need to enable a number of modules and this new configuration file:
Now configure the virtualhost where you need the Basic authentication - add something like this:
AuthName "You Must Login"
AuthDBDUserPWQuery "SELECT encrypt(password) AS password FROM password WHERE username = %s"
NOTE: The 'encrypt(password)' in the SQL statement is because the legacy information I'm moving over is in plaintext. If you've got your passwords encrypted, then you can use one of the options here and skip the encrypt call.
I am using a password table that looks like this:
CREATE TABLE `password` (
`id` int(11) unsigned NOT NULL auto_increment,
`username` varchar(255) default NULL,
`password` varchar(255) default NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Insert a user and password into the table, then
service apache2 restart and you're ready to go.
Hopefully this helps - I know I was pretty frustrated this afternoon with all the misinformation I found online.
15 May 2014
Two days ago, Jeff Lindsay released logspout - a Docker container that is:
A log router for Docker container output that runs entirely inside Docker. It attaches to all containers on a host, then routes their logs wherever you want.
As soon as I saw it, I knew that I had to see how I could get the logs out of my Docker containers and into something like Papertrail.
With our current Docker setup, we see the logs come into the HTTP proxy server and then out - but there wasn't a great way to see the logs from inside each Docker container. We have servers with a few dozen containers - we were really missing the visibility that comes with being able to see the logs easily.
nginx plus allows you to log to a remote syslog destination, but we're not using it as it would be cost prohibitive with our setup. Some online posts that talk about "how" to do it, either want you to run another daemon or log tailing utility. That seems a little kludgy - I don't want to manage more running processes.
The post that finally helped me solve it was here:
error_log /dev/stdout info;
I tried to create those devices in my Dockerfile:
RUN cd /dev && MAKEDEV fd
They installed when I built the image, but they didn't actually show up when I launched the container. Then I noticed that they were just links to /proc:
root@bd9e6c27ddce:/dev# MAKEDEV fd
root@bd9e6c27ddce:/dev# ls -l
crw------- 1 root root 136, 3 May 15 00:31 console
lrwxrwxrwx 1 root root 13 May 15 00:31 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 May 15 00:31 full
crw-rw-rw- 1 root root 1, 3 May 15 00:31 null
lrwxrwxrwx 1 root root 8 May 15 00:31 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 May 15 00:31 pts
crw-rw-rw- 1 root root 1, 8 May 15 00:31 random
drwxrwxrwt 2 root root 40 May 15 00:31 shm
lrwxrwxrwx 1 root root 4 May 15 00:31 stderr -> fd/2
lrwxrwxrwx 1 root root 4 May 15 00:31 stdin -> fd/0
lrwxrwxrwx 1 root root 4 May 15 00:31 stdout -> fd/1
crw-rw-rw- 1 root root 5, 0 May 15 00:31 tty
crw-rw-rw- 1 root root 1, 9 May 15 00:31 urandom
crw-rw-rw- 1 root root 1, 5 May 15 00:31 zero
So - one simple change to our nginx config - and we have all of our nginx logs from a Docker instance aggregated in one place:
This works with Apache too - should work with almost anything.
I have some work ahead of me to make the logs more useful and have better information in them - but at least now I can see what's happening inside each container without having to type
docker logs over and over and over again.
Thanks Jeff - like I said the other night - you write some bad-ass tools:
06 May 2014
Last week, I was in a hotel with incredibly brutal internet and I really wanted to update my Consul cookbook.
I tried for a while, but when it looked pretty hopeless I started to try some remote cloud providers to see if I could speed things up. Rackspace and AWS were no go since they use Xen behind the scenes - which doesn't work with VirtualBox - so I tried Digital Ocean - and it got farther than anybody else.
After an Vagrant ssh login failure, a hint from @michaelpgoetz and a comment from @GermanDZ I tried the kitchen-docker driver for Test Kitchen. It worked, and it seemed to work really well.
I now had a VM to work on my cookbook - and I wasn't limited to 25K / second download speeds anymore. Yay!
Here's the cookbook that I built to create that Test Kitchen development VM - tkdevenv-cookbook.
You can build your own image on Digital Ocean by, installing Packer, exporting a few environment variables:
# Digital Ocean - get these here: https://cloud.digitalocean.com/api_access
git clone https://github.com/darron/tkdevenv-cookbook.git
# Edit attributes/default.rb to put in your own name and email address
Once that's built, any time you need it, you can create an environment to use very easily. I will use the tugboat gem in this example:
[master] darron@~/Dropbox/src/tkdevenv-cookbook: tugboat sizes
512MB (id: 66)
1GB (id: 63)
2GB (id: 62)
4GB (id: 64)
8GB (id: 65)
16GB (id: 61)
32GB (id: 60)
48GB (id: 70)
64GB (id: 69)
[master] darron@~/Dropbox/src/tkdevenv-cookbook: tugboat images
tkdevenv (May 5, 2014) (id: 3550834, distro: Ubuntu)
[master] darron@~/Dropbox/src/tkdevenv-cookbook: tugboat keys
darron (id: 52104)
[master] darron@~/Dropbox/src/tkdevenv-cookbook: tugboat create testkitchen -s 62 -r 1 -i 3550834 -k 52104
Queueing creation of droplet 'testkitchen'...done
Now you have your very own Test Kitchen development environment "as a Service" (TKDEaaS), available in about 60 seconds. Perfect for those really terrible internet connection days.
NOTE: I only have 1 unresolved minor problem so far - Serverspec seems to fail for running processes - probably because of Docker. Haven't looked into it in detail yet - I just ignored that failure for now.
05 May 2014
Last year, after a lot of soul searching, I realized that it was time to move on from nonfiction - it was time for a change. However when you've founded and built a company over 12 years, moving on isn't so simple, so my business partner and I have been working a for the last few months to simplify and streamline my exit from day-to-day operations.
nonfiction is a great company that I'm proud to have built, it will be in my business partner's capable hands and I couldn't be prouder of the team that we built together.
As of May 31, 2014 I will no longer be a full time employee of nonfiction, however I will be sticking around for at least one month (on contract) to finish up a few projects and tie up some loose ends.
In July, I am looking for a new challenge - somewhere where I can learn and build new things, improve existing systems and keep my brain stretched.
I've been working a lot with Docker, Chef, "the cloud" (Amazon, Digital Ocean, Rackspace, etc), Packer, Vagrant, Cloud Foundry, Heroku and other similar tools over the last few years - so I'm looking towards challenges that involve some of those technologies, but I'm pretty open to anything involving the web and distributed systems.
I'm also very excited about emerging technologies like Consul, Serf, etcd and CoreOS - I see lots of potential and interesting work around them.
Much of what I've worked on and experimented with is posted on github - but some of my best work to date has been around octohost - nonfiction's Docker based mini-PaaS that's powering an unannounced and almost released product.
Another project I'm extremely proud of is the Varnish based CDN that I built in early 2012. It was scaling up in production and handled 100M hits over 3 months very well, but when I found Fastly - we switched over and abandoned it. (Pro tip: If you're not using Fastly yet, you should.)
My skillset allows me to create internet systems with a focus on empowering a team to build and run on top of those systems.
If you're looking for somebody like me, please send me a note and let's chat.