Docker for Mac fix for max files, transparent hugepages warnings

To avoid Redis and other apps’ “max files limit” and “transparent hugepages” etc errors in Docker containers on Docker for Mac, you can set sysctls options in your docker-compose file by using compose format 2.1 or greater, which should be supported in docker-compose >= 1.10.

in docker-compose.yml file:

version: '2.1'
services:
  redis:
    image: redis:3.2-alpine
    command: redis-server /usr/local/etc/redis/redis.conf
    sysctls:
      - net.core.somaxconn=1024

To update transparent hugepages settings in Docker for Mac’s Alpine hypervisor host system, you can use this hack:

docker run --rm --privileged -ti alpine /bin/sh -c "echo never > /sys/kernel/mm/transparent_hugepage/enabled && echo never > /sys/kernel/mm/transparent_hugepage/defrag"

In theory you wouldn’t need/want to do this on a production host as you would configure the host properly. If you update Docker, you will need to re-run this.

Sysctl optimization updates for Redis, MongoDB, etc.

Sysctl Performance tips for Redis, MongoDB, etc.

# backup:
$ sysctl -a > /home/sysctl_$(date +%Y%m%d).bak
#check:
$ sysctl -a
 
# make changes to sysctl.conf:
$ vi/nano /etc/sysctl.conf
 
# web - nginx, redis, mongo, etc:
vm.swappiness = 10
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10
# for larger servers:
net.ipv4.tcp_rmem = 4096 12582912 33554432
net.ipv4.tcp_wmem = 4096 12582912 33554432
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
#
net.ipv4.tcp_max_syn_backlog = 65536
kernel.keys.root_maxkeys = 1000000
#Raise somaxconn (above 511)
net.core.somaxconn = 4096
 
# vm.overcommit_memory (optional):
# vm.overcommit_memory = 1
####
 
# set in /etc/rc.local (test reboot stickiness)
# ensure changes to sysctl configuration persist across reboots:
$ sysctl -p
 
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

more info:

  • https://wiki.mikejung.biz/Sysctl_tweaks
  • https://www.techandme.se/performance-tips-for-redis-cache-server/
  • https://unix.stackexchange.com/questions/99154/disable-transparent-hugepages
  • Redis Docker container

    Easy Docker set-up using pre-built image:

    (Ubuntu host in this example, but others should be similar)

    # see https://hub.docker.com/_/redis/

    Docker run command:

    docker run -d -v $(pwd)/redis.conf:/usr/local/etc/redis/redis.conf \
     -v /data/redis:/data -p 6379:6379 \
     --restart="always" \
     --name myredis redis:3.0.3 redis-server /usr/local/etc/redis/redis.conf
  • On docker host, create data volume so we aren’t reading/writing on the container:
    make sure this path is in the redis.conf file
  • sudo mkdir /data/redis && sudo chown ubuntu:docker $_ && sudo chmod g+w $_

  • if you get this warning:
    WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
    > update Docker host:
    echo never > /sys/kernel/mm/transparent_hugepage/enabled

  • redis force save data file to disk command to test external volume:
    BGSAVE

    security issues:

  • add authentication if needed and check it on remote clients
  • set firewall: (ubuntu example where eth1 is LAN)
    ufw allow in on eth1 to any port 6379 proto tcp

    If you want to bake-in your config file or set other customizations, create Dockerfile:

    FROM redis:3.0.3
    COPY ./redis.conf /usr/local/etc/redis/redis.conf
  • PHP Redis with socket connection

    if you are using Redis on a single server and/or want the web server at the same location to use sockets instead of TCP, update your redis config (redis.conf) to include:

    unixsocket /var/run/redis/redis.sock

    (you may be able to just uncomment that line to use the default)

    You will need to make sure the socket connection has the correct permissions for the apache/php service. You can set the “unixsocketperm” attribute in redis.conf and make sure the socket location is appropriate for your apache/php set-up, or you can add a line in your redis start-up script to update the permissions, something like this:

    chown :apache $SOCK && chmod g+w $SOCK

    I’m not sure what the best practice is here, but I needed to do one the above to get it to work.

    Redis page created

    internal link > Redis Page

    « Previous Entries