Docker for Mac fix for max files, transparent hugepages warnings

To avoid Redis and other apps’ “max files limit” and “transparent hugepages” etc errors in Docker containers on Docker for Mac, you can set sysctls options in your docker-compose file by using compose format 2.1 or greater, which should be supported in docker-compose >= 1.10.

in docker-compose.yml file:

version: '2.1'
services:
  redis:
    image: redis:3.2-alpine
    command: redis-server /usr/local/etc/redis/redis.conf
    sysctls:
      - net.core.somaxconn=1024

To update transparent hugepages settings in Docker for Mac’s Alpine hypervisor host system, you can use this hack:

docker run --rm --privileged -ti alpine /bin/sh -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

In theory you wouldn’t need/want to do this on a production host as you would configure the host properly. If you update Docker, you will need to re-run this.

Sysctl optimization updates for Redis, MongoDB, etc.

Sysctl Performance tips for Redis, MongoDB, etc.

# backup:
$ sysctl -a > /home/sysctl_$(date +%Y%m%d).bak
#check:
$ sysctl -a
 
# make changes to sysctl.conf:
$ vi/nano /etc/sysctl.conf
 
# web - nginx, redis, mongo, etc:
vm.swappiness = 10
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10
# for larger servers:
net.ipv4.tcp_rmem = 4096 12582912 33554432
net.ipv4.tcp_wmem = 4096 12582912 33554432
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
#
net.ipv4.tcp_max_syn_backlog = 65536
kernel.keys.root_maxkeys = 1000000
#Raise somaxconn (above 511)
net.core.somaxconn = 4096
 
# vm.overcommit_memory (optional):
# vm.overcommit_memory = 1
####
 
# set in /etc/rc.local (test reboot stickiness)
# ensure changes to sysctl configuration persist across reboots:
$ sysctl -p
 
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

more info:

  • https://wiki.mikejung.biz/Sysctl_tweaks
  • https://www.techandme.se/performance-tips-for-redis-cache-server/
  • https://unix.stackexchange.com/questions/99154/disable-transparent-hugepages
  • Increase nginx open-files limit on systemd

    $ cat /proc/$(cat /run/nginx.pid)/limits
     
    $ nano /lib/systemd/system/nginx.service
    # --> [Service]
    LimitNOFILE=16384
     
    $ systemctl daemon-reload
    $ systemctl restart nginx && systemctl status nginx

    Docker-in-docker Docker Compose with sshd

    Dockerfile:

    ##
    # Docker client with docker-compose && sshd
    #
    # use on a Docker host to allow you to ssh and access Docker and Compose remotely
    # e.g., as part of CI/CD on a private network.
    # ** Not for production use on publicly-exposed server **
    #
    # mount for docker host socket:
    #    -v /var/run/docker.sock:/var/run/docker.sock:ro
    # mount for docker-compose access (optional):
    #    -v /host/compose/root:/opt/compose/alias
    # cd or reference -f /alias/to/docker-compose.yml file when using docker-compose ...
    #####
     
    FROM docker:17
    # (uses Alpine)
     
    RUN apk add --update py-pip
    RUN pip install docker-compose
     
    # RUN apk add ca-certificates curl openssl nano
    RUN apk add openssh
     
    # use fresh keys: (could also do on startup)
    RUN rm -rf /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_dsa_key
    RUN /usr/bin/ssh-keygen -A
     
    # install (or append) to authorized_keys: (optional)
    COPY certs/my-deploy.pub /root/.ssh/authorized_keys
     
    # cleanup install:
    RUN rm  -rf /tmp/* /var/cache/apk/*
     
    EXPOSE 22
     
    # remove prior entrypoint if there is one:
    ENTRYPOINT []
    CMD ["/usr/sbin/sshd","-D"]

    Build ID from Git (updated)

    A simple build_id can be generated from Git via:

    git log --pretty=oneline | wc -l

    which will give you something like 9682

    And while this will likely vary by branch, you really want to base a build id only from the branch you are deploying to production (in our case, release) anyway. It won’t let you name a work-in-progress, so that might be an issue for some people but also might enforce good practice by not letting you name a release until it’s “done”. As long as the naming is consistent, it should work. Of course in this scenario if you hot/bug-fix release that actual value will go up, but that’s not really what we care about, seems like our criterion would be:

  • something we can generate in an automated build
  • something sequential so it’s easy to tell the order to other builds
  • something unique (potential edge cases it may not be, but would be very edge-case)

    If you want a sequential number to give ordinal context but hash info for relevancy can use:

    echo $(git log --pretty=oneline | wc -l)-$(git log -1 '--pretty=format:%h')

    which will give you something like 9682-36a51c8

    log count + “-” + truncated last commit hash. You could add zero-padding or similar if desired.

    This seems particularly useful if you aren’t tagging at the time you create the id. Otherwise, you would be able to get the hash from the tag.

  • « Previous Entries