blag

See you in hell, Nginx

2021-07-06T00:01:00-04:00

I'm mostly kidding, I'll always love me some Nginx.

But I did decide to do some needless engineering and migrate to a new Reverse Proxy on my public webserver.

I had a few needs that I wanted to address and I felt it was time. I wanted to take advantage of more automatic configuration through Docker that some modern reverse proxies use (it's always good to get rid of extra config files), and I also wanted to take advantage of automatic Let's Encrypt renewals (because I've been doing it mostly by hand for too long). I'd also like to do all of this in Docker Compose as well (my set-up is pretty basic and I don't need to deal with the eight million ton gorilla on my off-time).

I decided to move to Traefik since it seems to handle my (limited) requirements the best.

Traefik is a Cloud Native Ingress/Reverse Proxy that has built-in support for ACME/Let's Encrypt and is pretty much completely configurable through Labels (for Docker compose) or Annotations (for Kubernetes). The extent of my configuration outside of my compose file is mounting a volume to store the certificates that are requested.

Is what I'm doing correct? Probably not. Does it work? As far as I can tell. I don't have any issues with things.

Core Config

    traefik:  
      image: "traefik:v2.4"
      container_name: "traefik"
      command:
        - "--log.level=DEBUG"
        - "--api.insecure=false"
        - "--api.dashboard=false"
        - "--providers.docker=true"
        - "--providers.docker.exposedbydefault=false"
        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--certificatesresolvers.xyz.acme.httpchallenge=true"
        - "--certificatesresolvers.xyz.acme.httpchallenge.entrypoint=web"
        - "--certificatesresolvers.xyz.acme.email=my email addy"
        - "--certificatesresolvers.xyz.acme.storage=/letsencrypt/xyz.json"
        - "--certificatesresolvers.tde.acme.httpchallenge=true"
        - "--certificatesresolvers.tde.acme.httpchallenge.entrypoint=web"
        - "--certificatesresolvers.tde.acme.email=my email again"
        - "--certificatesresolvers.tde.acme.storage=/letsencrypt/tde.json"

I left out a bunch of this configuration since it wasn't important to this post.

Here are some of the more important directives:

    - "--providers.docker.exposedbydefault=false"

I don't want every service exposed, I want to control things on a service-by-service basis. I'll cover that in the next section.

     - "--certificatesresolvers.xyz.acme.httpchallenge=true"
     - "--certificatesresolvers.xyz.acme.httpchallenge.entrypoint=web"
     - "--certificatesresolvers.xyz.acme.email=my email addy"
     - "--certificatesresolvers.tde.acme.storage=/letsencrypt/xyz.json"

These are for ACME/Let's Encrypt. I'm a fan of the HTTP challenge, it seems like the easiest thing to set up and maintain in my opinion (it's literally just these commands for each domain I want to secure).

Since the HTTP challenge is over HTTP, we want the entrypoint to be my configured web entrypoint (as opposed to websecure, which is HTTPS).

The email command is just my email address so when the renewal comes up, I get the notification.

The storage command is just a mounted folder from the host that Docker is running on in the event something bad happens to the host, like I delete it or something - I can quickly get up and running again. /letsencrypt is a mounted NFS share frrom Digital Ocean that's backed up.

Service Config

    blog:
      image: "ghost:3.27-alpine"
      restart: always
      volumes:
        - /docker/ghost/:/var/lib/ghost/content
      environment:
        - url=http://www.roylarsen.xyz
      ports:
        - "2368:2368"
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.blog-http.entrypoints=web"
        - "traefik.http.routers.blog-http.rule=Host(`roylarsen.xyz`) || Host(`www.roylarsen.xyz`)"
        - "traefik.http.routers.blog-http.middlewares=blog-https"
        - "traefik.http.middlewares.blog-https.redirectscheme.scheme=https"
        - "traefik.http.routers.blog.entrypoints=websecure"
        - "traefik.http.routers.blog.rule=Host(`roylarsen.xyz`) || Host(`www.roylarsen.xyz`)"
        - "traefik.http.services.blog.loadbalancer.server.port=2368"
        - "traefik.http.routers.blog.tls.certresolver=xyz"

Above is how I'm exposing the container hosting my Blog.

  "traefik.enable=true"

This is how to enable/expose each service as mentioned above.

    - "traefik.http.routers.blog-http.entrypoints=web"
    - "traefik.http.routers.blog-http.rule=Host(`roylarsen.xyz`) || Host(`www.roylarsen.xyz`)"
    - "traefik.http.routers.blog-http.middlewares=blog-https"
    - "traefik.http.middlewares.blog-https.redirectscheme.scheme=https"

These lines are how to enable the HTTP listener for the domains I want my blog to answer on, and then redirect to HTTPS (since things should only be served over HTTPS, but I don't want errors over HTTP).

    - "traefik.http.routers.blog.entrypoints=websecure"
    - "traefik.http.routers.blog.rule=Host(`roylarsen.xyz`) || Host(`www.roylarsen.xyz`)"
    - "traefik.http.services.blog.loadbalancer.server.port=2368"
    - "traefik.http.routers.blog.tls.certresolver=xyz"

After we accept the request from the HTTP listener, we need a place to redirect to (also just straight just to HTTPS).

The biggest differences between the two are the loadbalancer forward and the certresolver.

The loadbalancer is just where requests are sent on the backend, and then the certresolver is just what resolver to pull the certificate from in the Traefik config from the previous section.

That's It

Seriously. It was pretty easy overall to be able to swap from Nginx to Traefik (except when I forgot the url for the admin panel to write this, heh), and now I don't have to think about my cert renewals for the rest of the life of this server. It's all very exciting.

Jenkins Build Agents as Docker Swarm Services

2019-09-25T20:07:00-04:00

Lately I've been working on (re-)learning Jenkins and learning how to use their Pipelines feature.

We had set up a Jenkins server at MHB to off-load some of the potential work from Bamboo, and also just because we're trying as a DevOps team to move away from Bamboo since we're not too happy with it.

Docker Swarm

First, let's address the ten thousand pound elephant in the room: Docker Swarm is less cool than Kubernetes.

For small, easy to set up/manage Docker clusters on-prem, Docker Swarm is fine and perfectly serviceable. The automatic load balancing of published ports to all nodes/manager by default is really slick.

That being said, not everything is great. For this to work, you need to expose the Docker Swarm API and to do that is fairly ugly. I'm not going to get into securing it, since this is a lab set-up.

Exposing the swarm in an insecure manner is fairly straight-forward. Find the systemd unit file and add this to the end of the ExecStart line and then restart the service: -H tcp://0.0.0.0:2375

Jenkins Service

Here's my docker-compose.yml I used to deploy this:

    version: '3'
    services:  jenkins:
    image: jenkins/jenkins
    ports:
      - 8080:8080
      - 50000:50000
    restart: always
    volumes:
      - /usr/bin/docker:/usr/bin/docker
    environment:
      - "JENKINS_SLAVE_AGENT_PORT=50000"

Beyond the Docker/Swarm stuff, I'm not getting into Jenkins configuration. There's plenty of documentation that exists for general Jenkins configuration and usage.

I can't stress this isn't a production service. Nothing is persistant.

Jenkins Docker Swarm Plug-In

The next two sections are the most annoying. The documentation is definitely lacking when it exists.

To enable Docker Swarm access in Jenkins, you need to install 'Docker Swarm Plugin' and then reload Jenkins.

That adds a couple things to Jenkins. First and foremost, the Docker Swarm Dashboard:

Then the more important thing, in Manage Jenkins -> Configure System, at the bottom there's an option for Adding a Cloud:

The configuration for that is pretty straight forward:

The 'Test' button is a little flaky from what I've seen, you might need to save your configuration before it will succeed.

Jenkins JNLP Agents

This was the most interesting part. And by interesting, I mean frustrating. This is where the documetation was really lacking and I happened to find a blog post with a quick explanation of how this all works.

When you configure your cluster above, you get a button to add templates:

Clicking that button gives you this:

The label is what you can use for targetting, the image is the Docker Container to pull from. They actually provide a bunch of containers to use: https://github.com/jenkinsci/jnlp-agents

The docker image is different from the other containers and works as documented/is fine with the example configuration. The general idea on how it's supposed to work is that when the container starts, it does a curl to the Jenkins master and downloads the JNLP agent to ensure there's no version mis-matches.

As far as I can tell, none of the other agent images work that way. They're all based on their languages/toolsets alpine image and don't have tools like curl or git installed, but in the Dockerfile, Jenkins copies the agent .jar from the base JNLP agent image.

jnlp-docker command:

    sh -cx curl --connect-timeout 20 --max-time 60 -o agent.jar $DOCKER_SWARM_PLUGIN_JENKINS_AGENT_JAR_URL && java -jar agent.jar -jnlpUrl $DOCKER_SWARM_PLUGIN_JENKINS_AGENT_JNLP_URL -secret $DOCKER_SWARM_PLUGIN_JENKINS_AGENT_SECRET -noReconnect -workDir /tmp

jnlp-python command:

    sh -cx apk -u add git && java -jar /usr/share/jenkins/agent.jar -jnlpUrl $DOCKER_SWARM_PLUGIN_JENKINS_AGENT_JNLP_URL -secret $DOCKER_SWARM_PLUGIN_JENKINS_AGENT_SECRET -noReconnect -workDir /tmp

It took a bit of debugging and digging to figure out what was different between the two container images and figure out why my Docker container was working but my Python container wasn't.

The other big 'gotchas' I ran into is that the container needs to be run as root, or it will fail to launch. For Docker, you also need to make sure you mount /var/run/docker.sock from the host into the container so the Docker client in the container has a Docker daemon to work against.

Jenkinsfile

    pipeline{
      agent {
        label 'jenkins-docker'
      }
      stages {
        stage('Print Test'){
          agent {
            label 'jenkins-docker'
          }
          steps {
            sh "uname -a"
            sh "docker run hello-world"
          }
        }
        stage('Schedule Tests'){
          agent {
            label 'jenkins-python'
          }
          steps{
            sh 'python --version'
            sh 'ls .'
            sh 'printenv'
          }
        }
      }
    }

The above Jenkinsfile is a working example where I have the agents for the Pipeline level, and each stage (the job spawns as the first agent, then that spawns the agents for each of the stages).

I'm not going to get into setting up builds on API calls, again, that gets into Jenkins documentation that is good and there's plenty of.

Hopefully this helps. It'll be nice to have this reference for myself in the future.

So I decided to containerize our application stack... Pt 2

2019-06-12T04:01:00-04:00

Here's the story so far:

Part 1

Beginning to work through my list

In the last post I came up with a list of tasks that I need to work through so that we can do things in Docker "correctly".

("Correctly" meaning, as close to "dev clones the repo, fills in their .env, then runs docker-compose")

Getting SQL Server to work in Docker on Linux

It's funny, this was actually pretty straight-forward. Faults aside over the years, Microsoft has been putting a lot of effort into building compatibility from their flagship products and also transitioning from a traditional software company to a platform company, and transitioning a lot of their major software to either being a SaaS offering (Outlook/O365) or making it run on Linux (SQL Server, Powershell) as well.

Running SQL Server is pretty easy, MS has good guides on SQL Server running on Linux on technet: https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-docker?view=sql-server-2017&pivots=cs1-bash

It's super easy and all you really need to do is provide the password for the SA account as an environment variable.

Running Flyway migrations

Everything's easier when the vendor makes docker images easily available and usable.

Now, this is still a little tricky, because while the repo that has all our application code also has all our migrations, it's more than one migration and the flyway container comes expecting to be run via docker run and just providing arguments to the migrate command.

That doesn't really work for me.

There's a couple of issues that I needed to overcome to get this service in my compose file working:

I need the two databases the application expects
I need to run the two migrations for our databases

By the time this service starts/runs, all I have is a bare SQL Server instance with no databases or accounts other than the default ones.

Normally, to create the two databases that our application needs, I just use sqlcmd or Invoke-SQLCmd, but this container doesn't have the first command and these are Linux containers and not Windows - so no Invoke-SQLCmd.

I started thinking about how (since I'm already using it in this container) I could abuse Flyway - since all it does is run specially named .sql files.

So, the first thing I did was create a new migration to do the initial Database creation. It was pretty straightforward since in the repo, there was already a script to create the base databases, I just needed to name it correctly (eg, VO__createDB.sql, and write the config file for this migration)

However, since this adds to the complexity of the problem, I need to update the above list:

I need the two databases the application expects
I need to run the ~~two~~ three migrations for our databases

Luckily, point two are why shell scripts exist.

I threw together a really basic shell script to run three flyway migrations. It's nothing fancy, but at the end of the day, it didn't need to be.

Now, the fun thing was trying to figure out how to overwrite the container's command and inject my shell script.

The nice thing is that Compose gives you exactly that.

entrypoint: /flyway/mh-scripts/mh-run-migrations.sh

The 'entrypoint' option just overrides the CMD of the container. In my case, giving me a one off container that runs the migrations.

Building and running the Tomcat Application

First and foremost, shoutout to Sammy for telling me about this.

My goal was to be able to have one Dockerfile to handle doing both the build, and also running the application.

Luckily, these days Dockerfiles have multi-stage builds (thanks for pointing that out, Sammy).

Dockerfile!

The important parts of this are the following:

FROM gradle:5.4-jdk8 as build

The first FROM line sets up the stage to copy artifacts from. In my instance, I can refer to it as 'build'.

The next important thing is the change to the COPY command.

COPY --from=build /home/gradle/project/heartbeatWeb/build/libs/heartbeat.war /usr/local/tomcat/webapps

The important part of this is the --from= argument. Normally, you use COPY to copy from the host to the container. When using the from argument, you can specify the first step and copy artifacts from that container instead.

Wiring everything up in Docker

I'm not up to doing anything particularly fancy yet (separate netoworks for separation of concerns or anything, so a basic network configuration in compose is more than enough.

networks:    default:

That's more than enough to get all the containers into a state where they can talk.

Example:

web:        build: .        links:            - db        ports:            - "80:8080"        environment:            - JAVA_OPTS=$JAVA_OPTS -Ddbserver=db        depends_on:            - flyway_migrate

This is the last container. From here, we see the links to the DB service, the the dependency on running the database migrations.

The default network is enough that you can address containers by their service name (see the -Ddbserver arg for the environment above).

Now, I can't stress this enough, this set up isn't for production. Everything is accessible. In the future, one of my items will be to separate internal services and external services, and to put things behind a proper proxy service (probably something like HAProxy, maybe Traefik if that can handle non-http traffic).

Task List Update

I did everything in my old task list, now I need a new one!

Our Event Management Service
Ejabberd
Maybe clustered Ejabberd
ActiveMQ
Redis

So I decided to containerize our application stack...

2019-05-12T17:28:00-04:00

This is going to be the first part of a series of posts where I'm going to document my journey containerizing our Platform.

Here's the story so far:

Part 2

First Forays

The beginning of this journey was extremely haphazard (aka, I was bored one night and wanted something to work on while watching TV). For fun I spun up a Server 2016 box with our software installed on it and set up Docker on Windows since (due to our Customers) our Platform is Windows first.

Once that was all done, I copied the .wars that make up our admin application and the distribution of tomcat that we use and threw all of that into a Dockerfile and spun it up to talk to the SQL Server instance installed on that box.

It was pretty cool to type docker run and connect to that tomcat instance and see it work. Server 2016 Dockerfile

It's all fun to get this to work on Windows, but lets face it, Windows is a limiting factor when all the new developments in DevOps treats Windows as a Second Class Citizen at best.

Moving Forward

Those first steps were done back in November of 2018 and I eventually picked this back up in May of 2019.

This is when I started to want to get more serious about my personal projects and wanting to learn more about containers and take learning Docker more seriously.

Deploying our wars to linux

This is where things get really fun.

With the help of my awesome co-worker Valerie answering my really basic questions, I got things working.

The nice thing about Java is the WORA (write once, run anywhere) principal it follows. For the most part the initial getting things working was a case of pulling the image for Tomcat 8.5.40 and building my container by writing another pretty basic Dockerfile. Linux Dockerfile

After some trial and error and tracking down dependencies (jar with the SQL Server driver, a mail library) to get things working, I got a connection to an existing database on another server, and authentication via LDAP that was already pre-configured in that database.

Getting our webapps to work on Linux has been fun, because I'm getting to learn more about Tomcat. For the most part, the server.xml is a static entity but there are items that I want to be able to pass to the container as it starts up. Mainly the DB connection string and information.

That's where I learned how to use the \$JAVA_OPTS to populate variables in the server.xml.

docker run -it -p 80:8080 -e "JAVA_OPTS=$JAVA_OPTS -Ddbserver=sql1" --net test mhb/heartbeat:1.0

Is what I've been using to launch the tomcat container with the environment variable to set the JAVA_OPTS environment variable in the container.

With -e "JAVA_OPTS=$JAVA_OPTS -Ddbserver=sql1" option, that sets the \$JAVA_OPTS environment variable in the container. Within the server.xml I have access to the dbserver variable that I can set with \${dbserver}.

The next iteration of this is going to involve learning the setenv.sh script so I can use environment files in my compose file so I can more securly set the variables.

Coming up with a task list

Once I saw that working, I came up with a list of tasks to go from the most basic of POC to an actual MVP that I can show off and see what people think.

Get SQL Server on Linux working in Docker
Figure out how to run our Flyway migrations in Docker
Wire everything up in docker
Perform builds and deploy to Docker
Docker Compose

Now I have a task list to work through

Microsoft OpenHack - Day 2

2019-05-09T04:11:00-04:00

Where did we leave off yesterday? Oh yeah, Challenge 3.

Challenge 3 is where we start really seeing the pipelines in Azure DevOps (ADO) come together.

Side note: Days 2 and 3 I wasn't feeling super great, so starting this day, we have less screenshots.

Since we got one of the test pipelines working the end of the day before, that gave us a good starting point for getting all our other testing working.

The above is where I ended up for Challenge 3 with the node.js API.

The pipeline started with getting the sources, building the container, then pushing it to ACR.

We needed to add running tests, verifying the tests, and then acting based on the outcome of the tests.

The nice thing about modern languages, is that they provide all the tools you need to do things like running tests in the same binary as what you use to run your scripts.

Now that's out of the way, lets start pulling that screen shot apart, job by job.

Get Sources
- This is pretty straightforward. This preps the build environment for the current code base (git pull)
- npm install
- This is also pretty straightforward. This will install the npm tooling and project dependencies
- npm custom
- This is where the testing happens. ADO only gives two two jobs options (this is what I'm rembering seeing a few days later) and those are the npm install command and a 'custom' command to let you use the other npm tools
- The custom command in this intance is just for running npm test in the root of the node.js API
- Publish Test Results
- This was me going a little above and beyond
- I was having some issues running my npm test command (because I don't know npm or node.js) and wasn't getting the debugging output I needed to figure out the problem
- This just takes the .xml output of the test results and puts it in the pipeline output so you can see the results with more output
- Build an Image
- This is the same as it was before, it just builds the docker image from the accompanying Dockerfile
- Push an Image
- Again, unchanged from before. This just pushes the container to the ACR on a successful build
- Publish Artifact helm
- This is new and unneeded (for this challenge). All this does is take the /helm/ folder and publush it as a artifact to be shared in downstream pipelines

Now, the next question should be: What about the second part of the challenge posted in the previous article? Creating a work ticket based on problems in this pipeline?

It was literally a toggle in the options for build pipelines. Between that and making sure that jobs are set to "run only if the previous job succeeded", that will keep the pipelines from moving bad code down the line.

Challenge 4)

Create a release pipeline!

(this was after lunch where I started feeling way worse than before lunch and I started forgetting to take screenshots)

So, a lot of this is a lot like setting up the build pipelines. When you have your pipelines defaulted to the drag and drop it's pretty straight-forward.

When you're building pipelines in ADO, it's mostly dragging and dropping the tools you need to where you need them. For Release Pipelines, the main difference is that they go horizontal as opposed to vertical.

Our release pipeline was roughly:

|Artifact| -> | Stage 1 |

(I know, my ASCII drawing is too good)

-Long time ago post-

So, during this day I started feeling ill and I kinda dropped this (oops). There's not much for Day 3 since it was mostly trying to get these tasks done.

I was also still not feeling well, so I actually don't remember much beyond closing things out/

Microsoft OpenHack - Day 1

2018-11-28T06:02:00-05:00

Lions, tigers, and Azure DevOps. Oh my!

Where to begin?

Recently my boss emailed myself and a couple other engineers at Mobile Heartbeat about an event Microsoft was running about learning Azure, Azure Kubernetes (k8s) Service, and Azure DevOps (silly name for an actually pretty cool project). Naturally, I took him up on the offer because it was free and was a training on stuff I'm interested in/not well versed in.

Alright, now the fun bit.

The task:

That Architecture, but allowing for automatic rolling/zero downtime deployments of the 4 API microservices within the k8s cluster.

The event is setup in a pretty interesting way. The way that Microsoft set it up is as a series of challenges that build on the successes of the previous challenges.

When we got there this morning, they assigned everyone into different teams and after some moving me around, I settled into Team 11 with my first cup of coffee.

Challenge 1)

"In order to have a successful DevOps strategy, your team needs to have a plan."

The first challenge of the day, we needed to come up with a plan of how we want to run our team. Pretty basic, but it gave us a chance to work out how we like to work.

For the most part we decided that between tickets and the Wiki within Azure DevOps and the fact that we're sitting face to face, we don't need to worry about also adding some chat software into the mix.

Challenge 2)

Show to your coach that you can link a work item with the associated code changes and the artifacts that have been generated.
Demonstrate to your coach that you have implemented a policy to prevent any unreviewed code changes from being committed to master.
On a successful build, demonstrate that the updated version of the container is pushed to a container registry.

From here we created stories for each of these tasks, and then sub-divided them into actionable tasks.

The first thing we did was import the git repository from Github (https://github.com/Azure-Samples/openhack-devops-team), into Azure DevOps.

That allowed our team member working on task 2 to create the appropriate policies to institute mandatory code reviews where we need two reviewers that aren't the person who created the PR.

The rest of us tackled task 3. We broke that into feature tickets for each of the API services.

We were able to create build pipelines for each of the APIs.

Here's the Java Pipeline I worked on for example (there's an extra task, but I'll cover that in 3)

On approved PR, the pipeline is kicked off for API folders that there's changes.

Once these items were completed and we had containers building and being pushed to the correct repositories (there was a little confusion that was quickly cleared up), we then quickly took care of task 1 by putting everything together.

We created a couple tasks to run end to end tests of our new pipelines.

Create a ticket and associated branch from master
Make a change in our branch and create a PR to master
Get the required amount of approvals (2 minimum, can't be the person who created the PR) which then kicked off the pipeline
As soon as Azure DevOps saw the PR had the correct approvals, the build was automatically started

Challenge 3)

You are expected to improve the pipeline that you have started to build in the previous challenge by validating the functionality of at least one part of the code for the APIs of your application that you have selected.

-

In the event that a test fail, a bug shall be created automatically and added in the backlog and the submitted changes are not integrated in your code.

When the second challenge was accepted by our coach, we got to work on 3. Overall, it's not that complicated of a challenge - but when you're sitting at a table of non-developers, unit tests become tricky.

So, first thing I did was create stories for the two main tasks and start splitting those into features and issues.

Luckily, the APIs all already had their test suites written and had jobs in their various build systems ready to go, so all we had to do was add a task before the docker image build.

The hardest one to get going was the C# project, because each of the APIs had documentation in the README.md about how tests are run - that project didn't.

After some trial and error, we had that pipeline ready to go and it was 4:30 and time to go.

Tomorrow is Day 2 and we're starting off strong and getting task 3 finished.

DevOpsDays Boston 2018: Day 2

2018-09-25T23:41:00-04:00

Today, a lot like yesterday. I got to sit in talks and listen to smart people talk about things they care a lot about. Back to regular life tomorrow, but I'll at least be going back with some new ideas.

James Meickle- KEYNOTE: Beyond Burnout

This was awesome. It was about making your engineering teams more inclusive for our friends/collegues dealing with mental illness as a way to better serve your team(s) as a whole.

Tiffany Longworth - Change Management for Humans

We need to build for specific people. Our change is going to fail if we dont' have a specific target goal. Stakeholders are paramount.

Create your own evangelists!

Flynn - The rise of Layer 7, microservices, and the proxy war with Envoy, NGINX, and HAProxy

This was one of the more technical talks I got to go to. This talk was about how we moved from Monoliths to Decentralized Monoliths to Monoliths (again) and now to Decentralized microservices.

We've been able to vetter decentralize our services by using things like HAPRoxy and Zookeeper, but then all-in-one tools like Envoy came out to simplify comms and increase observability.

Matty Stratton - Everything Is A Product - How To Apply Product Management Practices to Technology Services

It was fun seeing a host of a podcast I actively like and enjoy listening to talk at the conference.

There was a lot of good information in this about how everything we do should be treated as a product with owners and stakeholders. Just because an environment is labelled as Dev or QA or that a tool we've written for internal use, people still rely on it and it needs to be up as much as we can garantee it.

Yegor Bugayenko - Expertise vs Experts

There were things in this talk I didn't agree with, but I think he made a lot of points that were good.

Software is better as manufacturing than art. Art makes things magic and special. Magic and special is unmaintainable. A good engineer is one that can easily be replaced.

Paul Bruce - Progressive Testing to Meet the Performance Imperative

This talk was the biggest bummer. Paul was having A/V issues throughout his entire talk.

Mike Rothmann - Quick and Dirty DevSecOps

Long story short: There is no such thing as Quick and Dirty DevSecOps. We're too often sacrificing security in the name of speed and agility. Mike argued that if we build our software and pipelines with security at the forefront, we don't have to sacrifice anything.

Overall thoughts

Overall, one of my favorite things about this conference was how much emphasis on inclusivity and diversity there was. Whether it was the breakfast they ran for underrepresented groups, or the Keynote this morning where James framed accomodating for people with mental illnesses as a way to make things better for you organization as a whole.

Catering to underrepresented groups is a strength, not a weakness. Homogeneous teams and companies are missing out on valuable input from individuals that you wouldn't normally get.

DevOpsDays Boston 2018: Day 1

2018-09-24T20:53:00-04:00

So, work paid for me and my boss to go to a conference

Yes, it's only in Boston, but it's still not work, so that rules. Here are my big takeaways from the first day:

Katy Farmer - KEYNOTE: How to be Happy Being Average

It's ok to not be a rockstar! We're all getting shit done and able to stand on the shoulders of giants to do cool things.

Anton Gurov - Anatomy of an AWS Account Cryptojack

hahaha, holy shit we're bad at security. Don't ever post AWS/GCE/Azure/DO/etc... creds to a public git repository. Not only will you get owned in a hilariously short amount of time, but you'll probably be on the hook for thousands to your IaaS provider.

Quintessence Anx - Sensory Friendly Monitoring: Keeping the Noise Down

Notifications are insanely distracting. Even if it's a welcomed notification, it's going to take time to deal with it and get back to work. Take a good long look at your notifications and figure out how to make them more manageable and how to make them better for you.

Jane Pruskakova - Can You Keep a Secret? Privacy in the Digital Age

Long story short: No. No we can't. Also, everyone wants every bit of data out of us that they can get.

Matt Surabian - Vault at Scale

Vault is actually pretty easy to run at scale, it's the backend and dataload that's hard to scale (insert fun anectdote of Zipcar's 6k container fleet clogging up token lease aggregation)

Pete Cheslock - Pick any three: Good, Fast, and Safe. DevOps from Scratch

We're still bad at DevOps, but we're getting better. Now that Dev and Ops are a happy family, we need to start being better at talking to Security.

Alicia Davis - Why we provisioned 37 Jenkins Servers: How to architect your CICD pipeline to account for organizational maturity

Apparently a company with a couple hundred dev teams requires a lot of space to work effectively.

Angel Rivera - CICD 101 with CircleCI

Modern CICD pipelines are fun! git commit -a to publish a new container to the Docker Hub.

Ok, that's day 1. All that's left is closing and drinking at Banyon!

Docker on Windows

2018-09-12T17:31:00-04:00

This is a thing I've beeen doing at work. Partly because I've been learning docker on my own time, and partly because it'll make deploying our chatbot code easier.

A lot of this is going to be copy and pasted from the wiki article I wrote about Docker on Windows.

I'm not going to get into what are containers and what Docker is, so, a certain level of knowledge is assumed. Smarter people who are better writers have already done that and those articles are easy to find.

Background

So, there's a few things that we need to address first.

Docker on Server 2016 is very different than Docker on Windows 10
Everything in this doc is assumed Server 2016
When did MS perform the dark ritual to get Docker working on Windows?

Docker on Server 2016 is a native feature that doesn't rely on running Hyper-V or anything, u like Windows 10. You install it just like you install any other Windows Feature.

Once the feature is installed, you're pretty much off to the races. There's no configuration or anything to do.

The only thing that you'll probably want to do once the features are installed is installing compose. Don't worry, it's not hard and I have the instructions for that.

The Set-Up

The version of Server 2016 doesn't matter that much. I've done this with both core installs, and full desktop environments.

To prep the server for Docker for Windows:

Install-Module -Name DockerMsftProvider -Repository PSGallery -ForceInstall-Package -Name docker -ProviderName DockerMsftProviderRestart-Computer -force

Once those lines are run, you can test your installation now:

docker run microsoft/dotnet-samples:dotnetapp-nanoserver

At this point, you should have a working Docker install.

Docker is great, but you also probably want docker-compose so that you can have something useful to run things with:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12Invoke-WebRequest "https://github.com/docker/compose/releases/download/1.22.0/docker-compose-Windows-x86_64.exe" -UseBasicParsing -OutFile $Env:ProgramFiles\docker\docker-compose.exe

Powershell is kinda annoying, and you have to force it to use TLS 1.2 in order to interact with sites that have a proper HTTPS set-up.

The the download is pretty straight forward. Substitute the version of compose for the version that you want. The -usebasicparsing flag just makes it go a bit quicker.

Usage

Now that docker and compose are installed on your system, you can pretty much use them as you would on Linux systems.

The big things to remember when using Docker is that you can only run containers of the same OS on the host. For Windows, there are two base containers that you can use:

https://hub.docker.com/r/microsoft/windowsservercore/ - This is about \~4GB for a container

https://hub.docker.com/r/microsoft/nanoserver/ - This is about \~2GB for a container, but you can't use insallers

Docker on Windows is cool and makes life easier, but for all of the sacrifices to the Dark Lord, the reality of life in Windows is that the network stack is garbage.

The workflow that I've been working with for a bit is having the Dockerfile and docker-compose.yml in my git repo for our project, then run the compose for deployments.

I haven't gotten around to using container registries yet, but that's the next improvement I plan on making. Have the container build process be part of a CI pipeline and automatically build and push the container when done. Currently I'm just cloning the git repo and building and running the container from our host.

Ghost + NGINX + Docker, OH MY

2018-07-10T04:47:00-04:00

So, I recently decided to redo my website and this is a active work in progress.

Enter:

DOCKER

~WHAAAAAA~

I've been using this as an excuse to finally get my shit together and actually learn Docker.

Up until now I've done basic docker work (docker run hello-world, etc...), but I haven't done anything particularly serious with it.

I'm not going to get into details and explain the internals and how docker works. I'm not super great at that, plus, there ware way smarter people who've beaten that horse to death.

Putting it together

So, after some furious research (aka, typing the words "docker", "ghost", and "blog" into google) I found their official docker image.

I then went and set up a new VM in DigitalOcean to act as my docker host with a external volume to hold all the persistent data (blog DB, config info for nginx, etc...) so that all can be easily backed up and I don't have to worry about it.

At first, my set-up was kinda clunky. I was building containers and running those with all the static content (tastefuldinosaurerotica.com, keybase challenge) pre-baked into the container. For simple set-ups that don't change - this is fine for the most part.

Here was the initial Dockerfile I was using to run nginx:

FROM nginxMAINTAINER Roy LarsenCOPY nginx.conf /etc/nginx/nginx.confCOPY keybase.txt /usr/share/nginx/htmlCOPY tde/* /usr/share/nginx/html/tde/

See? Not very exciting, not very portable/maintainable since now I need to maintain my own container that only exists on this one server.

Part of the portability, maintainability problem was was made this blog possible: a container running ghost.

So, now I have two containers that are essentially running the default software and I'm re-building at least one of them.

Enter Compose:

So, the first big change that I made was create a symlink on the host to be what the containers expect for their persistent storage instead of building the containers with the static content inside of them.

Here's how I'm now defining my containers:

version: "2.2"services:        nginx:               image: "nginx:latest"               ports:                        - "80:80"                        - "443:443"               volumes:                       - /docker/nginx/conf/nginx.conf:/etc/nginx/nginx.conf                       - /docker/nginx/www:/usr/share/nginx/html               networks:                        - default               links:                       - "blog"        blog:                image: "ghost:1-alpine"                volumes:                        - /docker/ghost/:/var/lib/ghost/content                environment:                        - URL=http://www.roylarsen.xyz                networks:                        - defaultnetworks:        default:                external:                        name: web

When that file is in place, all you need to do is sudo docker-compose up -d from that directory and you're good to go.

So, the nice thing about this is that when I want to tackle that first bullet point in Future, I can just have the container hosting certbot place the certs into the volume mapped to /docker/nginx/conf/certs and that will be that.

For portability, as long as the appropriate drive is mapped and symlink is created - I can always spin up these containers the same way every time. It's pretty sweet.

Future:

~~Let's Encrypt for doing SSL.~~ Ha, notice how this is served over HTTPS? Need to write that article
~~https://mastodon.social/~~ Ha, https://social.tastefuldinosaurerotica.com. Need to write that up
Separating internal and external container networks.
Ansible to prep the host server
~~Get more serious about Docker on Windows (I know: boo, hiss, etc...)~~ Ha, I wrote this

first

2018-04-02T03:38:00-04:00

Ayy, lets try blogging on Ghost again.

saltstack gitfs

2018-01-09T04:04:00-05:00

This is mostly notes for myself, but I wanted to write up an easy reference.

basics

This is to get a basic setup with a map from saltenv to git branch.

This requires some basic knowlege of saltstack concepts like pillar, salt environments.

My setup is pretty simplistic, I have two main branches: master and dev. The branches are

present in the state and pillar repos. Top files are managed per-repo, per-branch (there are

probably better ways of handling it, but this is fine for right now).

Here's the link to the saltstack docs on gitfs: https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html

For my notes/usage, I use pygit2 and ssh key auth.

First things first, you need to add your git host key to your master's known hosts:

https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#adding-the-ssh-host-key-to-the-known-hosts-file

At this point you've created your repos and the branches that you want ot work with. My mapping is pretty simplistic since

I just use two branches:

branch -> saltenv

master -> base

dev -> dev

actual configuration stuff

So, there are three things that you really need to configure before we're off to the races.

These are all in your /etc/salt/master

Configuring your backend:

fileserver_backend:  - git  - roots

Configuring your state repo:

gitfs_remotes:  - ssh://<state git repo address>.git:    - pubkey: /root/.ssh/id_rsa.pub    - privkey: /root/.ssh/id_rsa    - saltenv:      - dev:        - ref: dev      - base:        - ref: master

In the above config, you configure your saltenv's per remote. I'm assuming you you do this over multiple

backend definitions and use saltstack's merging function. I haven't played around with it, so, don't

quote me on any of this.

Pillar repo config:

ext_pillar:  - git:    - master ssh://<pillar git repo address>.git:      - env: base      - privkey: /root/.ssh/id_rsa      - pubkey: /root/.ssh/id_rsa.pub    - dev ssh://<pillar repo address>.git:      - privkey: /root/.ssh/id_rsa      - pubkey: /root/.ssh/id_rsa.pub

If you've noticed, the pillar definition is a tad different from the state definition. Where in the state, you could define the remote once and then map the

env to the branch/ref.

For the pillar definition, you need to define the branch/ref and the repo in the same lone, then map the saltenv under it (if you want to map

saltenvs to branches that are named differently - master/base for me)

conclusion

That's pretty much it. Restart the salt-master and you should have it all set up:

sudo salt '*' state.show_top<id>:    ----------    base:       - state       - state<id2>:    ----------    dev:       - state       - state

(I removed my company specific info from that output)

I think my next post will be about my mhqa tool and the Saltstack API

devops

2017-12-17T03:59:00-05:00

Well, not really. I have a pretty decent idea of what's going on.. Trust me, I've got this (mostly)

I've also figured out how to do images in Hugo, #dealwtithit

mhqa

I'm currently working on authoring a tool in python, and that's been fun to work

on. It's currently pretty limited in scope, but soon I should be able to start

building things out and really start doing some fun Continuous Deployment/Delivery

stuff for our QA people.

Long story short, this tool uses the saltstack API (probably something I'll write a post about in the future)

to interact with the vSphere API to revert back to "clean" snapshots and reinstall our backend

software with a named version on the CLI. In the future, I'll probably kill off the snapshot part of

the tool (because you're not really supposed to use snapshots like that) and just do fresh VMs for installs

(and upgrade installs).

It'll be exciting to see this tool when I finish it. It's the first real bit of programming I've ever

done at work (I think I'm up to \~200 lines of code, probably closer to 300 with the salt runner I wrote

to revert snapshots).

moar saltstack

A big reason I was hired into this role was because I had experience doing using Saltstack. Today (20171215)

I spent a bunch of time setting up saltenvs with gitfs branches. Now I can safely/easily work on things in my

dev branch and not have to worry about messing things up for people doing real work (not that there are other

people doing things yet, but now when people do start working - I can experiment in peace).

Soon I'll be doing work with salt-cloud, just 100% less Proxmox and 100% more vSphere.

Once I start getting the keys to that kingdom, I'll be writing overwrought autounattend.xml files (I don't think

I'll ever be able to adequately express how much I hate XML - it's the worst). But the nice thing about all of that,

I'll be able to spin up Windows Server 2012r2 boxes easily.

I've got a new job!

2017-10-09T03:57:00-04:00

So, as of 20171009, I'm in my last week of work at Hachette Book Group. Next week I start my new role as DevOps Engineer at Mobile Heartbeat!

I'm an equal mix of excited, scared, and a little bummed.

Excited because this is something I've been woring towards for a couple years now (the huge pay bump doesn't hurt)
Scared because it'll be one of my first more senior roles I've had
Bummed because I'm leaving Downtown Boston and I'll be leaving my friends at HBG

This job will be a great platform to really start developing my skills in the DevOps space. I'm stoked

to work with some amazing people to build and deliver a (tbh) awesome tool in the healthcare space.

Proxmox + salt-cloud =

2017-06-17T03:53:00-04:00

So, I've been working on getting salt-cloud playing nicely with Proxmox. The biggest issue I've seen so far is within salt-cloud's proxmox driver is more of a limitation within proxmox itself.

Unless you statically assign you IP address, you're not going to see your IP address in the GUI or in the standard salt-cloud query to your hypervisor. The proxmox driver also allows your to query DNS for

the IP address of your machine.

Neither of those really work for me at home, since I like to have my VMs get an address from DHCP, then convert that to a static reservation when I'm sure that VM will stick around.

My prefered solution would be to pull the IP address straight from the hypervisor, which would make my life so much easier. That's doable when you're using the VMWare/ESXi driver when the agent is

installed, I figure that there should be a way of handling that within Proxmox.

Enter the QEMU Guest Agent

As it turns out, installing this is pretty easy. You install the service within your target VM, set the Agent option under hardware in the PVE web GUI to 'On', and then you reboot the VM.

Boom. Easy.

At this point you have better access to the guest through the API (either through the web or pvesh command on the host).

Now I'm not going to do into details about how long it took me to dig through the PVE API, but I found what I want.

PVE APIv2 Docs!

Dig down to nodes/{node}/qemu/{vmid}/agent - That's the API commands for querying the guest agent directly. Not a lot is implemented, however, network-get-interfaces looks promising.

(spoiler: that's what we want)

So playing around with pvesh on my VM host gets me to this command:

pvesh create nodes/technodrome/qemu/101/agent -command network-get-interfaces(yes, my main node name is a reference to TMNT)

That returns a JSON response with all the network information on the interfaces on your guest:

pvesh create nodes/technodrome/qemu/101/agent -command network-get-interfaces200 OK{  "result" : [    {      "hardware-address" : "00:00:00:00:00:00",      "ip-addresses" : [        {          "ip-address" : "127.0.0.1",          "ip-address-type" : "ipv4",          "prefix" : 8        },        {          "ip-address" : "::1",          "ip-address-type" : "ipv6",          "prefix" : 128        }      ],      "name" : "lo"    },    {      "hardware-address" : "<MAC address>",      "ip-addresses" : [        {          "ip-address" : "<ipv4 address",          "ip-address-type" : "ipv4",          "prefix" : 24        },        {          "ip-address" : "<ipv6 address>",          "ip-address-type" : "ipv6",          "prefix" : 64        }      ],      "name" : "eth0"    }  ]}sanitised so you can't see my internal numbering scheme

OH SHIT.

So, now the biggest obstacles I have are as follow:

Getting what I found into: https://github.com/saltstack/salt/blob/develop/salt/cloud/clouds/proxmox.py
- Need to create a function wait_for_ip_address()
- Need to add some options to the Proxmox profile to handle being able to use DHCP
- Support having multiple interfaces that aren't lo - Get patch accepted into salt-cloud
- (Or I just keep it for myself)

UPDATE, NERDS

So, as I've been working on the code for my patch to the proxmox provider, I was running some other solutions to my problem in my head.

Enter: cloud-init

What I've been realizing that I can do is use cloud-init to run a python scipt to set the hostname of the box.

Get the VMID of the VM
- possibly use the MAC address as my unique identifier?
- Use VMID to call /qemu/{VMID}/status/current - Potentially use a shell script instead of a Python script to lower the amount of dependencies used

Update 2, nerds

I think at this point I am going to eventually end up with the cloud-init solution. I think it's the easiest to support and it's really interesting what other

clouds do.

I'm going to write my patch for the Proxmox provider, but probably just keep that fork as an example in my github account as something to show off. It'll be a fun exercise, but I don't see it as supportable

beyond a stopgap until I get cloud-init working.

Plus, I'm pretty sure cloud-init will be a better solution for when I want to experiment with building LXC containers.

Update 3, nerds

So, I've been neglecting my blog because I'm a dummy.

I've actually written my patch for the proxmox provider and have been using it in my homelab for a while at this point.

It was pretty cool, I got to learn how to do list comprehensions and now I can easily spin up new VMs in my lab!