Introduction

When making changes to The Bug Genie, first step is to set-up a local development environment. This involves installing a number of different servers, pre-requisites, and making a number of configuration changes.

To make things easier, The Bug Genie ships with dedicated Vagrantfile and Ansible playbooks that can be used for quickly bringing up a small development server with everything you need.

If you have not used them before, Vagrant makes it possible to quickly provide a local, isolated virtual machine for development purposes, and can call a number of provisioners (shell scripts, Ansible playbooks etc) to install additional software and configure the server. It should be noted that Vagrant out-of-the-box uses VirtualBox for providing the virtual machines.

WARNING: While Vagrant can be used on a whole range of operating systems, these pages mainly concentrate on use under GNU/Linux distributions. If you encounter issues, feel free to get in touch through Zulip. Take note that in order to access the stream on Zulip, you must have a valid account for it. Contributions via Github pull requests are welcome as well.

Pre-requisites

Before proceeding, make sure that you have the following pre-requisites have been met:

• You have created a Github account.
• You are familiar how to use with Github.
• You are familiar how to use Git.
• Git has been installed and configured on your local machine.
• You are not running any virtual machine hypervisor (KVM, Xen) except for VirtualBox.
• VirtualBox has been installed. This can be done using your favourite package manager. Older versions may not work correctly in combination with Vagrant, so if you find yourself in position where distribution ships with outdated package, have a look at VirtualBox offical download pages.
• Vagrant has been installed. Although a lot of distributions ship it in their repositories, those are usually heavily outdated versions which are almost certain to cause issues. It is recommended to install Vagrant directly from the official website.

The following combinations of VirtualBox and Vagrant are known to be working at the time of this writing:

• VirtualBox 6.1.18 + Vagrant 2.2.14 on Debian 9 Stretch (64-bit).

Assumptions

For the purpose of this guide it will be assumed that The Bug Genie git repository is cloned under directory ~/projects/. Make sure to adjust any commands listed for your own needs.

Another assumption will be that your username on Github is yourusername - which is of course incorrect, but a value needs to be provided in below commands. Make sure to update commands appropriatelly below.

Getting the sources

Do the following in order to obtain the sources:

1. Fork The Bug Genie repository on Github.

2. Make sure you have created the projects directory under your home directory:

   mkdir -p ~/projects/

3. Clone The Bug Genie repository:

   cd ~/projects/
   git clone https://github.com/YOURUSERNAME/thebuggenie/

4. Register the upstream repository. This is useful for keeping your local repository and Github fork in sync with original Github repository:

   cd ~/projects/thebuggenie/
   git remote add upstream https://github.com/thebuggenie/thebuggenie/

5. Create a local branch you be working with:

   cd ~/projects/thebuggenie/
   git checkout -b mywork

Basic operations

In order to check the current status of Vagrant virtual machines, you can run:

cd ~/projects/thebuggenie/
vagrant status

This will produce small output listing the status of all virtual machines registered in the Vagrantfile (there should be just the default one). The status is commonly one of not created, running, or powered off.

In order to start a powered-off machine or to create a new one (if it has not been created already), run:

cd ~/projects/thebuggenie/
vagrant up

If this is the first run (the machine status was not created), this command will download the base box from the Internet (this box is cached under ~/.vagrant.d/ directory for future runs), import it into VirtualBox, start it, and run provisioning (Ansible playbook) on it. This process can take 5-10 minutes or more, depending on your Internet connection, computer speed etc. Feeling impatient? You have enough time to grab coffee or tea :)

A virtual machine that is running can be easily halted (shut-down) with command:

cd ~/projects/thebuggenie/
vagrant halt

If at any point you want to destroy the virtual machine (you're done with the work, or maybe you want to recreate it with fresh vagrant up), you can easily do so with:

cd ~/projects/thebuggenie/
vagrant destroy

Before the machine is destroy, you will be prompted to confirm the action.

If at any point you feel a need to rerun the provisioning on a running virtual machine (maybe initial provisioning has failed due to Internet connectivity issues, or you have been messing with configuration too much), you can do this with:

cd ~/projects/thebuggenie/
vagrant provision

Installing The Bug Genie

With your virtual machine finally up and running, the next thing you will be interested in is how to access The Bug Genie.

Before using it for the first time, you need to complete the installation process using the web installer. Do the following:

1. In your favourite browser, open URL http://localhost:8080/. You should get presented with first step in the installation process.

2. Click on the Agree and continue button to accept the license.

3. A pre-installation checks page will be shown. Unless there are some errors reported, all requirements should have already been met during the provisioning, so just click on the Continue button to move on.

4. Now you will get presented with database configuration page. The values should have been pre-filled for you. Just click on the Continue button.

5. At this point you should be presented with the server/URL information page. Defaults should be fine, so just click on the Continue button.

6. Wait for default user information page to be shown. Feel free to enter your own name and e-mail address. It is recommended not to change the administrator username. As for password, you can leave it as-is (default is admin) or change it as you wish. Once done, click on the Continue button.

7. Finalization page will be shown, with administrator's username/password (default is administrator/admin). Click on the Finalize installation button.

8. You should finally be done now, with installation confirmation page being shown. Click on the Got it! button.

9. Due to the way configuration is written out by The Bug Genie, debugging configuration will be overwritten by the installer. To fix this, and re-enable debugging, run the following commands:

   cd ~/projects/thebuggenie/
   vagrant provision
   rm cache/*.cache

Accessing The Bug Genie

In order to provide access to The Bug Genie, Vagrant will set-up two ports on the host machine that will in turn be forwarded to the virtual machine:

• Plain HTTP, on port 8080 (http://localhost:8080/).
• HTTPS, with locally-issued private key/certificate pair, on port 8443 (https://localhost:8443/). For a recommended way to access via HTTPS, see section Avoiding HTTPS exceptions for accessing The Bug Genie below.

If you ever find yourself in need of using The Bug Genie CLI, you can easily SSH into the server with:

cd ~/projects/thebuggenie/
vagrant ssh

This will log you in into the virtual machine as user vagrant. This same user is used for running the web application as well, therefore you can use it for running the CLI commands too. The Bug Genie top-level directory can be found under directory /vagrant on the virtual machine.

For example, in order to clear the caches, all you need to do is run (once you have logged-in via vagrant ssh):

cd /vagrant/
./tbg_cli cc

Deploying code changes

Up until now, everything has been revolving around some basics on how to get the environment up and running. Chances are that you are here in order to make code changes, though, so let's get to it.

When Vagrant brings up the virtual machine, it sets up a shared directory between the virtual machine and host (using the guest additions). Incidentally, the directory that is being shared is the one which contains the Vagrantfile itself. If you have followed all the instructions up until now, this should be the ~/projects/thebuggenie/ directory.

In essence this means that all you need to do is make changes in your local directory, where The Bug Genie has been cloned at, and simply reload the page in a browser. Simple as that!

WARNING: This also means that any change made on the virtual machine is propagated to your directory on the host! In other words, you can easily wipe out all of your work on the host, so be careful when removing files directly on the virtual machine!

Log files

In order to make developer's life easier, both The Bug Genie and web server are set-up to write the log files into shared directory. The file are as follows:

• ~/projects/thebuggenie/apache-access.log, access log for the Apache web server.
• ~/projects/thebuggenie/apache-error.log, error log for the Apache web server.
• ~/projects/thebuggenie/thebuggenie.log, log file created by The Bug Genie itself.

If you access the virtual machine directly, all three log files should be visible under directory /vagrant/.

This set-up makes it easy to access the log files quickly on both host and virtual machine.

Debugging via Xdebug

If you ever find yourself wishing to use Xdebug to step through the code and figure out what is going during certain calls, the provisioning playbook will set-up Xdebug for you as well.

Xdebug is set-up to connect to your host machine via port 9000. All you have to do is make sure that your IDE is listening.

To attach debugging process while accessing the web pages, simply append XDEBUG_SESSION_START=netbeans-xdebug get parameter to your URL.

For example, in order to debug the landing page, open URL http://localhost:8080/?XDEBUG_SESSION_START=your_session_key, replacing your_session_key with whatever session key your IDE uses (for Netbeans, this is usually netbeans-xdebug).

Similar to the web interface, you can perform debugging on the CLI as well. For this you need to be logged-in to the virtual machine (via vagrant ssh command), and you must export an environment variable XDEBUG_CONFIG to value idekey=your_session_key.

For example:

cd /vagrant/
export XDEBUG_CONFIG="idekey=your_session_key"
./tbg_cli cc

Services and accounts on virtual machine

Operating system users and SSH

The operating system deployed on Vagrant virtual machine at time of this writing is Debian 8.6 Jesssie (amd64).

If you find yourself in need to access the operating system on the virtual machine directly to make some changes to the OS, or to access local-only services, you can do so with command:

cd ~/projects/thebuggenie/
vagrant ssh

This will get you logged-in as user vagrant. The user is allowed to run any sudo command without password, which should let you easily managed the entire machine. If you ever happen to need it, the user's password is set to vagrant as well.

Web server

The web server deployed on virtual machine is Apache httpd. Two important extensions are enabled for it:

• mod_rewrite (for nicer URLs)
• mod_ssl (for HTTPS)

Virtual host configuration is set-up under /etc/apache2/sites-available/000-default - for both HTTP and HTTPS. The file /etc/apache2/sites-available/default-ssl.conf is left unused.

Private key used by Apache web server for HTTPS can be found under /etc/ssl/private/apache2.key.pem, while the corresponding certificate is stored at /etc/ssl/certs/apache2.cert.pem.

Apache has been configured to store its error and access logs in files /vagrant/apache-error.log and /vagrant/apache-access.log.

Database server

MariaDB is used as database server. Once you have logged-in via SSH, you can connect to the database server as either of the two following accounts:

• root (no password)
• tbg (password set to `tbg)

A single database is created for use with the application - tbg. user tbg has full privileges to this database.

LDAP server

If you ever find yourself wanting to test the LDAP directory integration in The Bug Genie, the virtual machine comes with pre-configured LDAP server (slapd).

Directory structure is as follows:

• dc=tbg,dc=local (root/base)
• ou=people,dc=tbg,dc=local (for defining regular user accounts)
• ou=groups,dc=tbg,dc=local (for defining groups)
• ou=services,dc=tbg,dc=local (for defining services accounts)

A single service account is provided that can be used by The Bug Genie for accessing the directory:

• cn=tbg,ou=services,dc=tbg,dc=local, password tbg.

There are in total 6 regular user accounts present:

• uid=administrator,ou=people,dc=tbg,dc=local, password administrator. If you keep the default administrator username during installation process, this will help not to get you locked-out of the system once the LDAP authentication is enabled.
• uid=user1,ou=people,dc=tbg,dc=local, password user1.
• uid=user2,ou=people,dc=tbg,dc=local, password user2.
• uid=user3,ou=people,dc=tbg,dc=local, password user3.
• uid=user4,ou=people,dc=tbg,dc=local, password user4.
• uid=user5,ou=people,dc=tbg,dc=local, password user5.
• uid=usernoaccess,ou=people,dc=tbg,dc=local, password usernoaccess. This user is meant mainly as means to test if The Bug Genie correctly checks group membership.

A single group is provided:

• cn=tbg,ou=groups,dc=tbg,dc=local, with members being:
  • uid=user1,ou=people,dc=tbg,dc=local
  • uid=user2,ou=people,dc=tbg,dc=local
  • uid=user3,ou=people,dc=tbg,dc=local
  • uid=user4,ou=people,dc=tbg,dc=local
  • uid=user5,ou=people,dc=tbg,dc=local

Both the LDAP server and LDAP client configuration on server are set-up so you can easily run queries against the directory.

To access the directory as administrator, you can run:

sudo ldapsearch -Y EXTERNAL

If you want to try logging-in using one of the user or service accounts:

ldapsearch -x -W -D uid=user1,ou=people,dc=tbg,dc=local
ldapsearch -x -W -D cn=tbg,ou=services,dc=tbg,dc=local

In addition to local access, you can connect to the LDAP server from host maching through two ports forwarded on localhost, 8389 for plaintext connection (supports STARTTLS) or 8636 for TLS connection. Just don't forget to specify the base DN for running queries as well. For example:

ldapsearch -x -W -D uid=user1,ou=people,dc=tbg,dc=local -b dc=tbg,dc=local -H ldap://localhost:8389/
ldapsearch -x -W -D uid=user1,ou=people,dc=tbg,dc=local -b dc=tbg,dc=local -H ldaps://localhost:8636/

Setting-up LDAP authentication

In order to make The Bug Genie authenticate users agains the LDAP directory, perform the following steps:

1. Access The Bug Genie via http://localhost:8080/.

2. Log-in as administrator user.

3. Open the configuration page.

4. Select Manage Modules option.

5. Click on the Installable local modules tab.

6. Click on the Install button for the LDAP Authentication module.

7. Open the LDAP Authentication tab that should have appeared on the left side. Set the configuration as follows:

   1. LDAP connection URI: ldaps://localhost/

   2. Base DN: dc=tbg,dc=local

   3. Control user DN: cn=tbg,ou=services,dc=tbg,dc=local

   4. Control user password: tbg

   5. Use HTTP Integrated Authentication: Off

   6. HTTP header field: KEEP_IT_EMPTY

   7. Object DN attribute: entrydn

   8. User object class: inetOrgPerson

   9. User username attribute: uid

   10. User full name attribute: cn

   11. User buddy name attribute: gn

   12. User e-mail address attribute: mail

   13. Allowed groups: tbg

   14. Group object class: groupOfUniqueNames

   15. Group member attribute: uniqueMember

8. Click on the Save button.

9. Click on the Test connection button.

10. Verify message at top states Connection test successful.

11. Click on the Import users button.

12. Verify the message states Import successful! Imported 5 users and updated 1 users out of total 6 valid users found in LDAP.

13. Open tab Authenticaton.

14. Set Authentication backend to LDAP Authentication.

15. Click on the Save button.

16. Now log-in again using username administrator and password administrator.

17. You should also be able to login as user1 through user5, with password being equal to username, i.e. also user through user5.

X.509 certificates

As mentioned before, a small local CA hierarchy with two server certificates is set-up the virtual machine itself. Generated files can be found under directory ~/projects/thebuggenie/artefacts/x509 (on host machine) or /vagrant/artefacts/x509 (on virtual machine).

Keys and certificates are generated using the GnuTLS CLI tools.

CA private key is stored in a file called ca.key.pem, while CA certificate is stored in a file called ca.cert.pem.

The CA is mainly used for issuing server certificates for web server and LDAP server. Both of those include a number of DNS names and IP address as subject alternative name:

• localhost
• tbg-dev
• tbg-dev.local
• 127.0.0.1
• INTERNAL_VM_IP

Avoiding HTTPS exceptions for accessing The Bug Genie

When working on The Bug Genie, it can sometimes be necessary to validate the application behaves correctly when access via HTTPS. I.e. that URLs generated are using HTTPS, or that there are no errors being reported by browsre about mixed content.

As mentioned in one of the section above, The Bug Genie installation can be reached through port 8443 for HTTPS.

Since the CA hierarchy used for issuing the web server certificate is not trusted by browsers, you will be presented with a big fat warning no matter what browser you use. For example, Mozilla Firefox will report something similar to:

Your connection is not secure

The owner of localhost has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

Commonly users just click their way through such an error, and add a permanent exception without even checking the certificate. This is a terrible security practice, and you should never, ever do this, unless you really, really, really know what you are doing. And by knowing what you are doing, I mean you understand what X.509 is, what is a certificate extension, what is issuer DN, what is subject DN. If you do not? Well, you probably should not do it :)

However, since the CA certificate used for issuing the web server certificate is available locally, it should be easy to import it and use it for testing the HTTPS connection properly. While this can be done in your main browser, it is recommended against because:

• This is just development environment. If by mistake you manage to give the CA private key to someone (or even publish it to git), anyone will be able to misuse it and issue fake website certificates in order to intercept your communication with them (think compromised passwords, user accounts etc).
• Since this is development environment, the CA key/certificate might get destroyed. So you probably do not want to mess with certificate settings in your browser all the time and risk messing something up.

The best approach is described within this section, with the added benefit of gaining an easy way to test The Bug Genie during development with multiple users at the same time.

For this to work, we will need to use Mozila Firefox, so make sure you have it installed before proceeding. The steps outlined here will allow you to run multple, separate instances of Mozilla Firefox.

1. Start off by launching Mozilla Firefox from terminal. If you already have Mozilla Firefox running, you do not need to shut it down.

   firefox --ProfileManager --no-remote

2. You will be presented with small dialogue for creating new profiles. If you haven't touched this before, you should have just one profile named Default. Make sure not to destroy it.

3. Click on the Create Profile button. This should open up creation wizard.

4. Go through the wizard, and make sure to change the profile name to something meaningful, for example tbgdev.

5. Once the profile has been created, do not start, but simply click on the Exit button to close the profile manager.

6. Again from terminal, start new instance of Mozilla Firefox using the profile you just created:

   firefox -P tbgdev --no-remote

7. You should be presented with a new Mozilla Firefox window. Keep in mind this is your development instance, so don't use it for regular browsing. You may even want to enable the bookmarks toolbar and put a bookmark with some clear name in it in order to distinguish it visually.

8. Now onto import the CA certificate. Open page Edit -> Preferences.

9. Select tab Advanced.

10. Within the tab, select sub-tab Certificates.

11. Click on the View Certificates. A Certificate Manager should open up.

12. Click on the Authorities tab.

13. Click on the Import button and select the CA certificate file ~/projects/thebuggenie/artefacts/x509/ca.cert.pem via presented file browser.

14. Click on the Open button.

15. A dialogue will show-up with a number of options. Tick the Trust this CA to identify websites, and click on the OK button.

16. Now close the preferences window, and visit https://localhost:8443/ - keep in mind that since this is not the main Firefox instance, you will need to copy/paste the link into it.

17. If all went well, you should see The Bug Genie home page, and no errors. From now on, any time you want to start this Firefox instance, simply run:

    firefox -P tbgdev --no-remote

WARNING: In order to avoid issues with links being opened in wrong Firefox instance, I suggest that you make sure your main Firefox is up and running before you start the development instance. Very often Firefox will direct new links to whatever instances was started first.

Using Netbeans for debugging

Although primarily a Java IDE, Netbeans provides surprisingly useful variant that is tailored towards PHP development. This section will guide you through set-up of Netbeans for working with The Bug Genie development, concentrating on Xdebug integration.

WARNING: Prior to following these instructions, make sure that you have already started and provisioned the VM using vagrant up command. You probably also want to make sure you have installed The Bug Genie via installation wizard.

1. Start off by downloading and unpacking Netbeans from project download page. Make sure to get the PHP download bundle. At time of this writing the Netbeans stable version is 8.2.

2. Once Netbeans has been unpacked and installed, run it.

3. You should be presented with a start page. Select File -> Open Project menu item.

4. Navigate to directory ~/projects/ and select thebuggenie directory from the list. At this point you should see thebuggenie in list of the projects on the left.

5. Right-click the project, and select option Properties.

6. Open category Run Configuration.

7. Set Project URL to http://localhost:8080/.

8. Click on the Advanced button.

9. Add a new Path Mapping. Server path should be /vagrant. Project path should be ~/projects/thebuggenie.

10. Click on the OK button to confirm changes.

11. Click on the OK button to confirm changes again.

12. Now, with Netbeans window in focus, press Ctrl + F5. This should open URL http://localhost:8080/?XDEBUG_SESSION_START=netbeans-xdebug in your browser.

13. The browser page will be stuck in loading state. This is fine, because default Netbeans setting is to stop execution on first line.

14. Switch back to Netbeans window. You should see that it has opened index.php in a new tab, and that the first non-commented line of script is highlighted.

15. Now that we have confirmed that Xdebug has managed to connect to Netbeans, press Shift + F5 to stop the debug process. This should open a new tab in your browser with URL http://localhost:8080/?XDEBUG_SESSION_STOP_NO_EXEC=netbeans-xdebug.

16. Now, most likely you want Netbeans to stop execution on very specific breakpoints instead of on first line of the script being run. So let's fix that next.

17. Switch back to Netbeans window and open menu item Tools -> Options.

18. Click on the PHP tab.

19. Click on the Debugging sub-tab.

20. Untick the Stop at First Line box.

21. Click on button OK to apply settings.

22. From now on you should be able to set manual line breakpoints through Netbeans IDE, and when accessing The Bug Genie execution should stop on those when execution reaches them.

Troubleshooting

This troubleshooting section documents some of the common pitfalls when using the provided development server.

After running vagrant up and trying to access The Bug Genie, I get Could not load scope error

The full message should be:

An error occurred in The Bug Genie

Could not load scope. This is usually because the scopes table doesn't have a scope for this hostname

An unhandled exception occurred:

If you also have debugging enabled, there should be a linie similar to the following down the page:

fatal [main] 21.19 ms  Table 'tbg.tbg_scopes' doesn't exist

This usually happens if you have destroyed your Vagrant environment and recreated it without removing the installed file. Simply remove the installed file from The Bug Genie root directory and try accessing the application again.

The Bug Genie does not log anything in thebuggenie.log

The most likely case is that installer has overwritten debugging settings. To fix this, simply run:

cd ~/projects/thebuggenie
vagrant provision
rm cache/*.cache

Extending provisioning

This section is dedicated to working on the provisioning tools themselves. It provides some hints and information about file structure, caveats etc.

Vagrant

A single Vagrantfile is provided the root directory. The file is fairly simple, defining just a single machine.

At the time of this writing, it uses the bento/debian-8.6 box. Although Debian project has its own, semi-official boxes on HashiCorp Atlas service, the bento box was chosen instead because it ships with VirtualBox guest additions (which makes it possible to have VirtualBox shared folders). The semi-offical boxes, for example debian/jessie64, currently do not have guest additions built-in, which means that directory synchronisation requires rsync, which is not instantenous. I.e. it was a convenience pick.

As for why a Debian box was picked instead of some others, this is simply the original's author preference. It might be possible to run equally well the provisioning on an Ubuntu box. The only issues that pop to the mind when it comes down to using Ubuntu box is that (at time of this writing) the Ubuntu boxes do not follow Vagrant recommendations on standard user, and they do not come with Python 2.7 pre-installed (required for use with Ansible).

Another important thing to note about Vagrant use is that the Vagrantfile utilises ansible_local provisioner instead of the ansible one. This is helpful in order to reduce the amount of work/software to be installed on the host machine. I.e. all the necessary sofware for running Ansbile is getting installed on the virtual machine, thus reducing the cruft on host. This also has the benfit of possibly running Vagrant on Windows.

Ansible is, in turn, deployed using pip instead of package manager in order to get an up-to-date version of it. Debian by default ships with a really old version of Ansible that lacks a lot of useful features. To make the deployment more predictable, version is (as of this writing) fixed to 2.2.0.0.

Ansible

All files related to Ansible can be found in sub-directory (relative to TBG root) ansible. This helps keep the directory structure clean.

Additional directory, where any sort of artefacts (X.509 keys/certificates) are stored at is called artefacts. So, if you decided to create any kind of local files, you may want to put them in there if needed - again the idea is to not pollute the filesystem. At this moment only exceptions to this rule are log files, which are found directly under project's root. This is mainly for convenience sake (so devs can access them quickly).

As a side-note, there are two hacks deployed using command and shell modules to set-up the LDAP server. One is for setting-up permissions and configuration, and the other one for creating the base structure and a couple of users. At time of this writing Ansible (version 2.2.0.0) does not contain any native modules for handling LDAP entries. However, two new modules will probably become available as of Ansible 2.3.0.0 (ldap_entry and ldap_attr). Once those land in, it would make sense to update the Vagrantfile and provisioning playbook to take advantage of latest Ansible and new modules.

As for actual playbooks, there is just a single playbook available at the moment. While normally Ansible tasks are best split amongst roles, the original author thought it would be better to have a single playbook for simplicity.

The playbook itself should be pretty straighforward, possibly the biggest challenge being the LDAP server set-up part and issuance of private keys and corresponding X.509 certificates.

A couple of notes regarding the playbook:

• The easiest way to have the vagrant user run the web application was using mod_ruid2 module for Apache. The way the module is configured for default virtual host is to run the script as owner of the file. This helps resolve a lot of management issues (user can just log-in as vagrant user and run CLI without permission issues etc).
• Make sure to notify Restart apache2 handler for any kind of configuration change or PHP package installation. This is important in order to make PHP extensions available to the application.
• The trick of setting executable bit on /etc/ssl/private/ directory allows any user to read a file under this directory provided they are allowed to read the file. Users not allowed to read the file would get denied the right, although they could theoretically discover if file exists or not.
• Make sure to run any The Bug Genie CLI commands as the vagrant user in the playbook (i.e. via become_user).