Author: chuckyz

Ansible: Nginx server blocks from Swagger YAML

I’ve recently been working on a project with a swagger specification.

(If you wanna just get right into it, here’s the goods)

A few things lead me to try experimenting creating valid nginx server blocks from a swagger.yaml:

For setup, I used a swagger.yaml with paths “/pets”, and “/pets/{id}” with basePath of “/myswaggerapi.”  I used this sample as my template.

This also lead me to realize that Jinja2 includes no way to really do a real regex replace, as for correctness sake, any path with {...} should be replaced with .* and have ~ prepended to location in nginx, to denote regex matching.

(Note: ~* should not be required, as your path’s should be lowercase.  (Thought: maybe I should just use .lower() on the path…) )

So I created a regex_replace filter after reading this StackOverflow post.

This leads to the meat and potatoes of the path part of the template:

location {% if "{" in path %}~ {% endif %}{% if basePath is defined %}{{ basePath }}{% endif %}{{ path | regex_replace("{.*}",".*") }}

This makes location /myswaggerapi/pets {... for the /pets path and location ~ /myswaggerapi/pets/.* {... for the /pets/{id} path.  This means all requests  to swagger.tld/myswaggerapi/pets will be forwarded to the first chunk, and any valid request containing anything will be passed to the second bit, to be validated by the actual swagger API.

(Note: This could be done by nginx, but for now it’s left up to the API)

Then, for security reasons and as it should be included in the paths, I limit requests by request type to each path using the following line:

limit_except {% for method in paths[path] %}{% if method != "parameters" %}{{ method.upper() }} {% endif %}{% endfor %}

This produces limit_except GET POST {... for the /pets path, and limit_except GET {... for the /pets/{id} path.  Is it actually more secure?  Well…  maybe?  It sure feels like it though!

I believe this could be used for CI using a job to Ansible to push your updated paths to an include file included in the correct server block to CI your API paths, allowing for dead simple updates to API end points without ever having to touch your nginx config (an nginx reload should still be required, but those are completely atomic).

For a working example and complete role of the above, here’s my repo.

Posted on by chuckyz 0

CentOS 7: Disabling OOMKiller for a process

In the latest version of the proc filesystem the OOMKiller has had some adjustments.  The valid range is now -1000 to +1000; previously it was -16 to +15 with a special score of -17 to outright disable it.  It also now uses /proc/<pid>/oom_score_adj instead of /proc/<pid>/oom_adj.  You can read the finer details here.

Given that, systemd now includes OOMScoreAdjust specifically for altering this.  To fully disable OOMKiller on a service simply add OOMScoreAdjust=-1000 directly underneath a [Service] definition, as follows.

...
[Service]
OOMScoreAdjust=-1000
...

This score can be adjusted if you want to ensure the parent PID lives, but children processes can be safely reaped by setting it to something like -999, then if “/bin/parent”, has “/bin/parent –memory-hungry-child,” it will be killed first.

If you have a third-party daemon (like Datadog, used in this example below) which manages itself and uses a sysvinit script you can still calm the OOMKiller.  A good way I’ve found this is, at whatever regular interval you choose, adjust the oom_score_adj, manually.

As a raw example, using all datadog processes, I’ve done the following (as root):

pgrep -f "/opt/datadog-agent/embedded/bin/python" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done;

In an example Ansible playbook that would allow you to exclude more than one group of processes:

default.yml
---
oomkiller_exclusions:
  - "/opt/datadog-agent/embedded/bin/python"
  - "/opt/my-process/bin/foo"
main.yml
---
- name: Exclude processes from oomkiller
  cron: 
    name: "Exclude {{ item }} from oomkiller"
    job: "pgrep -f \"{{ item }}\" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done"
    minute: "*/5"
    state: present
  with_items: "{{ oomkiller_exclusions }}"
  tags:
    - oomkiller

Note:

It’s not directly in any of the docs that I linked, but some commenters mentioned that children processes inherit parent processes oom_score_adj.  I confirmed this with some quick testing.  In the below, 11745 is a python CLI, and 12203 is subprocess.call(["sleep", "60"]) called by 11745.

$ cat /proc/11745/oom_score_adj
0
$ echo -1000 > /proc/11745/oom_score_adj
$ cat /proc/12203/oom_score_adj
-1000
Posted on by chuckyz 5

Advancing in Ansible: ternary (Jinja2 filter)

Ansible version 1.9 introduced the ternary filter.  This filter is one of the most useful filters, in my opinion.  It can be overused, but typically what ends up happening is instead of using | ternary() a huge amount of logic gets baked into other stuff (with_items, set_fact) instead; so I prefer it.

The filter is tougher to explain than to simply provide examples, so see the following (A snippet from my playbook to manage ec2 instances):

- name: "AWS EC2 INSTANCE(S) | Managing instance(s)"
  ec2:
...
    vpc_subnet_id: "{{ (ec2_subnet_type == 'public') | ternary(ec2_vpc_public_subnet_id,ec2_vpc_private_subnet_id) }}"
...

What this is doing is I under defaults/main.yml I set ec2_subnet_type: "private" , so if an instance needs to be deployed in a public subnet in my AWS VPC, I just set the type to public in a variable loaded through vars_files; then the subnet id itself is gathered later on.  This allows me to gather all subnets early on in a single task, and then deploy it to the correct subnet in this task.

This is most useful when you find yourself thinking something along the following lines…

  • “set_fact and when might work here…”
  • “I’ll just run this module more than once with when“
  • “I could use jinja2 inside this anything…” (Yes, this works, but avoid this at all costs.)

A second example, when LetsEncrypt very first came out, is I needed to setup multiple ‘stubs’ serving port 80 to “bootstrap” new domains in nginx, here’s the snippet that created these stubs:

- name: "NGINX | Setup LetsEncrypt http stubs"
  template:
    src: letsencrypt.stub.j2
    dest: "/etc/nginx/conf.d/{{ item.name }}.le_stub.conf"
  with_items: "{{ ('production' not in ansible_fqdn) | ternary(vhosts,nonprod_vhosts)  }}"
  when: "{{ item.letsencrypt }} is defined"
  tags: letsencrypt

Hopefully these two examples help illustrate some uses of this filter.  I find I don’t use it very often, but it’s very much a…

Ah, this is exactly what I was looking for!

Kind of moment!

 

Update:

Another helpful use of this that I forgot to include here is overloading set_fact, like as follows…

 - set_fact:
     instance_list: "{{ (ec2_instances.tagged_instances is defined) | ternary(ec2_instances.tagged_instances,ec2_instances.instances) }}"

The ec2 module will output “tagged_instances” if you include tags, and ec2_remote_facts outputs “instances,” so you can re-use one role by passing instances explicitly, or passing newly created instances straight to the role, and always get the right list.

Leave a comment if you can provide any other useful examples of this filter.

Posted on by chuckyz 4

Advancing in Ansible: Using a Bastion (Jumphost)

Ansible 2.0 introduced a great set of features (ansible_*_args), and glances over using it in the documentation.

However, my personal preference is to do it directly in the inventory.  This means you can do things like bake it into a dynamic inventory, or use [all:vars] to keep your inventory nice and DRY.

To do this directly in an inventory:

[all:vars]
ansible_ssh_common_args=' -o ProxyCommand="ssh -W %h:%p bastion.host.tld"'

[proxy]
10.0.0.2

[application]
10.0.0.3

[database]
10.0.0.4

If you only have, say, one bastion per host:

[proxy]
10.0.0.2 ansible_ssh_common_args=' -o ProxyCommand="ssh -W %h:%p proxy-bastion.host.tld"'

[application]
10.0.0.3 ansible_ssh_common_args=' -o ProxyCommand="ssh -W %h:%p app-bastion.host.tld"'

[database]
10.0.0.4 

# Also acceptable
[database:vars]
ansible_ssh_common_args=' -o ProxyCommand="ssh -W %h:%p db-bastion.host.tld"'

However, ansible_ssh_common_args is not the only useful thing here.  If you’re using sftp or scp (say, for jailed users) you can use it in exactly the same way.  Looking at the most recent manpages for sftp and scp all above examples will work, right down to the same -o line.

Doing the above I locate my bastion and put it into my dynamic inventory.

To do this with a dynamic inventory you’d want something like the following:

$./inventory.py --list
{
  ...

  "proxy": {
    "hosts": ["10.0.0.2"]
    "vars": {
      "ansible_ssh_common_args":"-o ProxyCommand='ssh -W %h:%p bastion.host.tld'"
    }
  },

  ...

}

Doing this makes accessing hosts on private networks through Ansible an absolute snap.

 

Posted on by chuckyz 0

Advancing in Ansible: combine (Jinja2 Filter)

In Ansible 2.0 the combine filter was added.  At a glance it appears useful, but the examples they provide in the documentation are missing some grunt.  A bigger and more useful example would be “I need to manage 2 similar but different named databases per environment across 100 environments.”  Manage in this article will be scoped to only users/passwords.

You could do this with a massive set of variables files, or you could do it with, say, 1 set of inventory variables and "{{ ... | combine({'key':'value'}) }}".

To start with, I’m going to make the following assumptions about our fake environment:

  • One environment named “prod”
  • One environment named “staging”
  • Every other environment has a variable attached to it named “env_name”
  • The databases outside of production/staging are always (using Jinja2) "db_{{ env_name }}"

Given the following defaults/main.yml and vars/main.yml:

vars/main.yml:

---
base_dbs:
  main:
    prod: main
    staging: main_staging
  events:
    prod: events
    staging: events_staging

defaults/main.yml:

---
env_name: default
db_users:
 - name: writer
   password:
     prod: aaa
     staging: bbb
     default: ccc

 - name: reader
   password:
     prod: aaa
     staging: bbb
     default: ccc

default_dbs:
  main:
    default: main_development
  events:
    default: main_development

### The good stuff!
dbs: "{{ default_dbs | combine({'main':{env_name: 'main_'+env_name}},{'events':{env_name: 'events_'+env_name}},base_dbs,recursive=True) }}" 

main_perms: "SELECT,INSERT,UPDATE,DELETE,CREATE TEMPORARY TABLES,TRIGGER,SHOW VIEW,EXECUTE"
read_perms: "SELECT,SHOW VIEW"

permissions:
  writer: "{{ dbs.main[env_name] }}.*:{{ main_perms }}/{{ dbs.events[env_name].*:{{ main_perms }}"
  reader: "{{ dbs.main[env_name] }}.*:{{ read_perms }}/{{ dbs.events[env_name].*:{{ read_perms }}"

 

We can now use the following task, say, tasks/configure_users.yml to configure these users under MySQL:

---
- name: MYSQL USERS | Create MySQL Users
  mysql_user:
    name: "{{ item.name }}"
    password: "{{ item.password[env_name] | default(item.password['default'])] }}"
    # Don't use this outside of an example!
    host: "%"
    priv: "{{ permissions[item.name] }}" 
  with_items: db_users

 

Now a breakdown of what’s going on here…

The combine filter takes in a dictionary (default_dbs), it then takes a,b,c,…,n dictionaries ({'main':{env_name: 'main_'+env_name}},{'events':{env_name: 'events_'+env_name}}, and base_dbs).  An important part of this process is that out of a,b,c,…,n — a overrides the input, b overrides a (and the input), c overrides b (and a, and the input), and the final dictionary “n” (note: it’s perfectly acceptable to have a single dictionary here) overrides everything.   recursive=True is explained in the documentation:

The filter also accepts an optional recursive=True parameter to not only override keys in the first hash, but also recurse into nested hashes and merge their keys too:

{{ {'a':{'foo':1, 'bar':2}, 'b':2}|combine({'a':{'bar':3, 'baz':4}}, recursive=True) }}

This would result in:

{'a':{'foo':1, 'bar':3, 'baz':4}, 'b':2}

This isn’t crystal clear though, if you leave out recursive=True I’ve found that it simply overrides the hash, so in the above example you’d end up with a dictionary of {'a':{'bar':3, 'baz':4}, 'b':2}, as 'a' would be over-ridden entirely by the combine filter.

 

A side-note on why “base_dbs” lives in vars/main.yml instead of defaults/main.yml.  The vars directory is one of the highest precedence values, and typically something like a production database name is something that one would not want to override.  Because of this, it is better suited to be stored as a variable instead of a default.

 

Posted on by chuckyz 0

Advancing in Ansible: Custom Libraries (Modules)

Ansible 2.2 was just released, and all documentation is based on the development branch.   At the time of this writing I’m still using Ansible 2.1.  I wanted to include a library from 2.2 into one of my playbooks (specifically the AWS NAT Gateway module).  However, the docs that mention your own libraries are somewhat sparse. (Developing Modules and Best Practices)

Heading into the IRC channel brought me to the advice given in the Best Practices document.  That the ‘library/‘ directory, when in the same directory as the playbook, allows you to use any non-included library.  This means no need to set something like ANSIBLE_LIBRARY as mentioned in this Quora post (https://www.quora.com/How-do-you-add-your-own-module-to-the-Ansible-library-path).  It certainly is still an option though.

This means if you have playbook.yml and want to use a non-included module, even something as simple as a newer module you can simply plop it straight into library/ , name it what you want to use as a module name and it’ll “just work.”  Given a playbook, and directory structure like the following, and Ansible 2.1 you can use the ec2_vpc_nat_gateway 2.2 module easily.

Directory:

playbook.yml                
library/                    
  ec2_vpc_nat_gateway.py

Playbook:

---
- hosts: localhost
  connection: local
  gather_facts: false
  vars:
    vpc_public_subnet_id: "my-vpc-public-subnet-id"
    vpc_region: "my-region"
  tasks:
    ec2_vpc_nat_gateway:
       if_exist_do_not_create: yes
       state: present
       subnet_id: "{{ vpc_public_subnet_id }}"
       wait: yes
       region: "{{ vpc_region }}"

This would then call library/ec2_vpc_nat_gateway.py when ran.

A benefit to this is it’s very simple to work on fixing bugs or developing on current modules, simply pull an existing module out of site-packages/ansible/modules and plop it into library/ under another name.  I recently submitted a pull-request for fixing a bug in ini_file about unexpected behavior, and simply renamed ini_file to library/ini_file_fix.py for testing, then changed all instances of ini_file to ini_file_fix in my playbook, and the “new” module was immediately used.

Pretty handy!

Posted on by chuckyz 0