14 January 2018

Redmine, Passenger, and Nginx on Ubuntu 16.04

Redmine with Passenger and Nginx on Ubuntu 16.04

Just in time for Ubuntu 18, here are Ubuntu 16 instructions for getting Redmine up and running. Though better than on Ubuntu 14, this is still a maze of twisty little passages, all alike. 

Here are flexible instructions to get an up to date, robust, secure installation going.
  • Ubuntu 16.04 LTS - a mature Ubuntu version with long term support 
  • MySQL - the database 
  • Ruby - the technology on which Redmine runs, installed using RVM to manage the ruby version and have access to up to date components 
  • Phusion Passenger- the application server in which to run Redmine 
  • Nginx - the web server within which Redmine runs   


MySQL Configuration

Install MySQL. For this step, relying on the Ubuntu packages is fine.
sudo apt-get install -y mysql-server libmysqlclient-dev
 Connect to the mysql service (mysql -p) and create the database and provide access to the redmine user:
mysql -p -u root
CREATE USER 'redmine'@'localhost' IDENTIFIED BY 'yourpassword';
GRANT ALL PRIVILEGES ON redmine.* TO 'redmine'@'localhost';
If this is just a test system, move onto the Ruby Installation section. 

For production systems, a separate data disk should be used rather that storing data on the same partition as the root system. Stop the service and then migrate the data directory.
 service mysql stop
Modify the data directory configuration in MySQL:
sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf
#datadir = /var/lib/mysql #old location
datadir = /data/mysql # new location
On install, MySQL 5.7 automatically initiates the data directory. So this must be moved to the new location.
sudo mkdir /data
sudo mv /var/lib/mysql/ /data/mysql/
Update AppArmor, otherwise the mysql process won't start. The MySQL 5.7 and Ubuntu combination have an AppArmor bug that shows when moving the data directory. The error will show in syslog like this:
Jan 12 10:03:12 ubuntu16 kernel: [ 1289.012262] audit: type=1400 audit(1515780192.392:116): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8834/status" pid=8834 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Jan 12 10:03:12 ubuntu16 kernel: [ 1289.012897] audit: type=1400 audit(1515780192.396:117): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=8834 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
To really geek out, this bug has one of the more sad but funny threads I've read in a long time. They eventually get to the right conclusion and open two bugs, but getting there is a journey. We will patch it ourselves, because the bugs haven't been actually fixed in the Ubuntu release.

To update AppArmor:
sudo vim /etc/apparmor.d/usr.sbin.mysqld
In the "Allow system resource access" section add the following to fix the bug(s): 
/sys/devices/system/node/ r,
/sys/devices/system/node/** r,
/proc/** r,
And under the "Allow data dir access" section change the entries from /var/lib/mysql/ to the new directory, /data/mysql/.
# Allow data dir access
#  /var/lib/mysql/ r,
#  /var/lib/mysql/** rwk,
/data/mysql/ r,
/data/mysql/** rwk,
Reload AppArmor:
sudo systemctl reload apparmor
Restart mysql:
 service mysql start
The system should start without error if everything has been done correctly. Run the following command to ensure everything looks okay:
mysql -p -u redmine

Ruby Installation

Many methods exist to install Ruby. Unfortunately, the most convenient using apt-get will leave the system many versions behind, which means many plugins for redmine, and parts of redmine itself, will be unsupported. RVM is a tried and true method to provide clean management of ruby versions. 

For whatever reason, these steps must be run as root:
sudo su - #get into root
apt-add-repository -y ppa:rael-gc/rvm
apt-get update
apt-get install rvm
Once done, logout and log back in as root.
sudo su - #log out and log back in
command curl -sSL https://rvm.io/mpapis.asc | gpg --import -
rvmsudo rvm get stable
rvm requirements
rvm install 2.5.0
rvm use 2.5.0 --default
Test that the install looks okay. 
ruby -v
ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux] 
  Note: once setup, each user of rvm needs to be added to the rvm group. 
sudo usermod -a -G rvm username

Phusion Passenger Installation

Again, many options exist for application containers. Phusion has a Passenger-Nginx combo that is straight-forward to install and configure. It does not need to be done as root.

Details of this installation can be found on the Phusion site.
sudo apt-get install -y dirmngr gnupg
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7
sudo apt-get install -y apt-transport-https ca-certificates
sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger xenial main > /etc/apt/sources.list.d/passenger.list'
sudo apt-get update
sudo apt-get install -y nginx-extras passenger

Nginx Configuration 

Passenger configuration with nginx has simplified since previous versions. 

Enable passenger in the nginx:
sudo vim /etc/nginx/nginx.conf
Uncomment the following line:
include /etc/nginx/passenger.conf;
 Restart nginx:
sudo service nginx restart
 Confirm everything looks correct
sudo /usr/bin/passenger-config validate-install
sudo /usr/sbin/passenger-memory-stats
The result will be something like this:
Also, confirm the correct location of the correct ruby.
which passenger-config 
/usr/bin/passenger-config #use this result to perform the next command
/usr/bin/passenger-config --ruby-command
The result will be something like this:
Build out the site - assuming http for now. More configuration will be needed to SSL-ize the system and lock it down. But for now, this will get things correct and running
sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/default.orig
sudo vim /etc/nginx/sites-available/default
Update the root location and add passenger configuration. The assumption is your redmine root location is also on the /data partition. This directory will be created when we add the redmine account.
root /data/redmine/redmine/public/; #installation location
passenger_enabled on; #turn on application container
client_max_body_size 10m; # Max attachement size allowed
Then to prevent a mess of 404 errors, comment out the location entry. Missing this step results in a special level of redmine 404 hell.
#location / {
    # First attempt to serve request as file, then
    # as directory, then fall back to displaying a 404.
    #try_files $uri $uri/ =404;
No need to restart nginx just yet. The system is almost ready for redmine installation.


Create the redmine account

Before doing too much with ruby, create a redmine service account. Note: Ideally the home directory is located where redmine is going to be installed - for production systems this should be on a separate partition. 
sudo adduser --system --shell /bin/bash --gecos 'Redmine Administrator' --group --disabled-password --home /data/redmine redmine; sudo usermod -a -G rvm redmine 
 Give the account sudo privileges (temporarily).
sudo visudo 
redmine    ALL=(ALL)      NOPASSWD:ALL

Welcome to dependency-o-rama 

The ruby add-on dependencies next depends (ha ha get it?) on various ruby pieces needed to install ruby components. A minimal list will look something like this: 
sudo apt-get install -y build-essential imagemagick libmagickwand-dev

Redmine, remember this was the main point of the article?

Whew. Like a hero that doesn't show up until the third reel of a movie, redmine is finally on the scene. Yes, this is just like Batman vs Superman. There is a lot of build up to the main event, and when you get there it is anticlimactic.

Install the latest redmine

Switch to the redmine user and pull down the latest stable release.
sudo su - redmine # should result in being in the redmine installation directory
wget http://www.redmine.org/releases/redmine-3.4.4.tar.gz
tar xvfz redmine-3.4.4.tar.gz
ln -s redmine-3.4.4 redmine
rm redmine-3.4.4.tar.gz

Configure the mysql connection 

Update the production entry with the account connection information.
cd redmine 
cp -pR config/database.yml.example config/database.yml
vim config/database.yml

Adding gems - truly outrageous!

Configure the gems - and avoid doing this as root. As with everything in redmine administration, this should be as that fancy redmine service account:
gem install bundler
bundle install --without development test
If versions and such match these instructions, the bundle install should go clean:
Bundle complete! 31 Gemfile dependencies, 55 gems now installed.
Gems in the groups development and test were not installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.

If versions are different or new features are needed, some iteration may be needed to build and install the gems. This seems to be a "normal" task for ruby administrators. Apply google-fu and iterate. 

Rake magic

Next run the magic rake commands. All magic comes in three. 
bundle exec rake generate_secret_token
RAILS_ENV=production bundle exec rake db:migrate
RAILS_ENV=production bundle exec rake redmine:load_default_data
Now everything should be ready to start. This is done by restarting nginx. Monitor the following logs to ensure things start clean: 
tail -f /var/log/nginx/error.log
tail -f  /data/redmine/redmine/log/production.log

Clean up

Congratulations, the redmine site should now be up and available. 

Remove sudo privileges from the redmine account. Move the site to HTTPS, ideally using the excellent Let's Encrypt service. These instructions can be found here.

Links of interest 

General Redmine installation 
Always a good place to review the latest information on generic Redmine installations.
Using Google Authentication
If using Google Apps or Google Auth is of interest with Redmine, a longstanding plugin has been brought back to life with a patch. 

The plugin can be found here:

But in modern ruby and google land, a patch will be needed. The plugin will install just fine, but it will not save any configuration information, nor will it put the google button on the login page without the patch below.

Though out of date, this article has some pointers on setting up the google side:

Installing RVM 

Instead of MySQL, consider Aurora 
For some of my more recent Redmine installations, I've been using Amazon Aurora. It is cheaper than MySQL on AWS, compatible with MySQL 5.6, backups are taken care of, and even has regional failover built in. Much better than researching AppArmor patches because you want to simply move a data directory. 

Setting up redmine is the same as above, but instead of doing MySQL, connect to an AWS Aurora instance. (You will still need the MySQL client - libmysqlclient-dev):


  1. Hello I am having problem by installation for the production system. I tried to move the data directory and have allowed data dir access but cant start mysql. when i try to start mysql It shows the problem like :
    Job for mysql.service failed because the control process exited with error code.
    Can you give me some idea what i can do?

    Thank you!!

    1. This is likely because you need to update AppArmor configuration, otherwise the mysql process won't start with the error you are describing. Please see the section about mysql, which details this issue. A work around is to symlink the data directory instead of moving it and reconfiguring.

  2. Thank you for this nice guide!

    One comment and one question :D

    On AWS you need to "sudo apt-get update" before doing the intial apt-get's, or it will not do it properly

    On AWS, how do I get to the site from "the outside"? Ive verified that the install works with lynx (text browser) on fist, then the "internal IP" listed on the AWS-site, but neither the external IP or the long-ass DNS-address work.

    I've tried to open port 80 via
    sudo ufw allow 80

    Ant tips or tricks?

    1. Getting to the system from outside depends on your AWS setup:
      * The AWS security group must allow for traffic on port 80 (and 443).
      * If using a NATed VPC, the VPC must have a NAT micro running to allow traffic in and out.

      Allowing ping through to the system will also tell you if it is a network thing or an application thing. But it is likely a network issue.

  3. During the Ruby Installation, when I run

    rvmsudo rvm get stable

    I get

    rvmsudo: command not found

    Googling the issue, does not resolve the problem. How can I fix this?

    1. I have found that if rvm isn't installed as the root user, you can run into odd issues including pathing. When installing rvm for users other than root, the pathing can get crunked and other users can't access the rvm commands, like rvmsudo.

      Try reinstalling rvm as root and your setup should look like this:

      root@redmine:~# which rvmsudo

    2. I've found that i might need to reboot the system after the rvm installing, to get it to recognize ruby 2.5, but there might be other ways, but that worked like a charm (on AWS)

    3. looks like a spacing issues in the instructions "rvmsudo" is not a command. maybe 'rvm' is part of the previous command. then 'sudo' is the start of the next command.

    4. install RVM steps

      #install rvm
      sudo apt-get install -y software-properties-common
      sudo apt-add-repository -y ppa:rael-gc/rvm
      sudo apt-get -y update
      sudo apt-get -y install rvm
      sudo su -
      rvm get stable
      rvm requirements
      rvm install 2.5.0
      rvm use 2.5.0 --default

      #also i had to do this after adding redmine user to run gem
      sudo usermod -a -G rvm redmine

      #had to do this at the end after noticing curl localhost returned nginx baase page
      sudo service nginx restart

      #then to test again
      curl localhost

    5. @james - rvmsudo is indeed a command. But you are correct in that as root it is not really required for installation.

      For sure, the redmine user must be a member of rvm and redmine (this is in the user creation command in the blog).

      Thanks for the clarifications.

  4. Hello. In ruby installation when I type rvm requirements, something is wrong:
    root@redmineubuntu:~# rvm requirements
    Checking requirements for ubuntu.
    Installing requirements for ubuntu.
    Updating system..
    Installing required packages: libgmp-dev, libgmp-dev..
    Error running 'requirements_debian_libs_install libgmp-dev libgmp-dev',
    please read /usr/share/rvm/log/1524135824/package_install_libgmp-dev_libgmp-dev.log
    Requirements installation failed with status: 100.

    I'm trying to install libgmp-dev, without success.
    Sorry for the Spanish message.

    root@redmineubuntu:~# apt-get install libgmp-dev
    Leyendo lista de paquetes... Hecho
    Creando árbol de dependencias
    Leyendo la información de estado... Hecho
    El paquete libgmp-dev no está disponible, pero algún otro paquete hace referencia
    a él. Esto puede significar que el paquete falta, está obsoleto o sólo se
    encuentra disponible desde alguna otra fuente

    E: El paquete «libgmp-dev» no tiene un candidato para la instalación

    Can anybody help me? Thank you very much.

    1. Looking at the log may help. These kind of errors usually point to a problem with apt, your package manager.

      Try running "sudo apt-get update" and see if there is an error. It may be there are broken sources in your /etc/apt/sources.list.d.

      If that command runs without error, try "sudo apt-get install build-essential"

      This link may help: