Deploying JIRA in the Enterprise – Part 3 – Configuration in Subversion and Deployment using Bamboo

This is the 3rd part in a series of posts on Deploying JIRA in the Enterprise. The last part covered Packaging using the RPM Maven Plugin. This post will focus on how to store the environment configuration in Subversion and using Bamboo for deployment.

As I have gone over in my last post, the reasons why we use a technology are often based on many factors. The same can be said for the choice to use Subversion for storing environment configuration. At my previous company, Red Hat Satellite was standard for configuration management and deploying RPM packages. There are many other tools that can provide the same functionality but I am using Subversion for two main reasons:

• it is a supported production environment
• it was fast to implement

What are we storing in Subversion? My rule is simple – only store environment specific files. For JIRA this could include some of the following files depending on how your environment is setup:

• dbconfig.xml – the database configuration
• server.xml – the tomcat configuration
• jira-config.properties – jira config options
• logo.png – logo for different environment
• setenv.sh – different JVM requirements

The structure in Subversion is a folder per environment with a path of folders to the files from /apps/foo-jira. For example, dbconfig.xml for Production would be located at Production/jira-home/dbconfig.xml which would map to /apps/foo-jira/jira-home/dbconfig.xml. Ofcourse, the path could start at / or any other path if that makes sense for your deployment environment.

This method does mean that any changes to a configuration file need to be updated in each environment. I would prefer to have a more elegant solution that substitutes variables into template files like in a Chef cookbook or similar but for the limited changes required to these files, Subversion works fine.

So, we have our RPM package and we have our environment configuration in Subversion. Let’s have a look at how Bamboo can tie this all together.

Firstly, lets have a look at Building the RPM. It will actually be a very simple plan – there will just be one Stage and one Job.

The first task for the Job will checkout the package source from subversion – this is a standard Bamboo task. The second will run the “mvn package” command. I tried to use a Maven Task but I couldn’t find a way to configure the task to run via sudo as I need privileged access for Maven to call rpmbuild. Instead I used the Custom Command Task and I run “sudo mvn package”. This task builds the Maven project and generates the RPM.

The Shared Artifact functionality defines where the RPM will be picked up from for use in other tasks – in our case the deployment.

Bamboo has a neat new feature called Deployment Projects. Although you could previously use Build Projects for deployment tasks, the new Deployment Projects enhances the management of deployments by introducing specific functionality suited to deployments such as defining environments and releases. For my example, we are going to use the Integration environment.

Once a successful build has been made, we setup Bamboo to automatically trigger a deployment to our Integration environment. It also creates a release for this deployment using a specific naming convention.

Lets move onto the Tasks of the Integration environment deployment plan. The first task cleans the working directory, the second picks up the RPM via the Shared Artifact configuration generated from the Build Plan.

Next we stop JIRA and uninstall the existing rpm. I am using some basic shell scripting to list the existing rpm packages and find the one containing “jira” to uninstall.

We need to cleanup some remaining application directories and temporary files. Then its time to install the new rpm package.

We export the configuration from Subversion and update some permissions on the fly as the export command in Subversion doesn’t seem to support the executable flags.

Start JIRA and we run some tests of basic functionality.

And that is it – the deployment is complete in the Integration environment!

The build takes 10-15 minutes and the deployment around 4-5 minutes. Timing will obviously vary depending on environments and the amount of testing that you perform. Once the build is completed and released, we can deploy the release easily to any other environment.

That completes our deployment process.

Ofcourse this is not perfect and there are many different ways to implement deployments but it does work quite smoothly. Here are a few points of review to note:

• You may be wondering, why use RPMs at all? Why not check out the package contents and use scripts to copy and permission the files where you need them – its basically the same as using an RPM. I must admit that it will make the whole process faster but RPMs do have an advantage in that they provide a backup in case Bamboo and/or Subversion is not available. Plus packages are more final – they can be signed and secured to ensure integrity as well.
• It would be nice if it was possible to easily template or share tasks between environments in Bamboo. I think Atlassian are working on this already and  definitely would help to ensure consistency of deployments.
• When I am actually building a new package, I usually prototype and test my changes on my Development environment without updating the package first, then I update the package with the changes build and deploy to Development and Integration, test and promote to Staging and then Production. It doesn’t make sense to continually package for smaller changes.

Ok, I think thats all for this series. Hope you found it interesting and if you have any questions, please let me know.

the jira guy

Deploying JIRA in the Enterprise – Part 2 – Packaging using the RPM Maven Plugin

Following on from my last post, this is Part 2, focusing on Packaging using the RPM Maven Plugin.

Firstly, why am I using RPM packages? There are many solutions for packaging and deploying applications using other tools. Every environment is going to be different and a decision will be based on the OS platform, standards and policies within your organisation, the infrastructure team, the knowledge and use of particular tools and processes. RPM is just one solution and it is your environment that will dictate the best choice.

The main driver for me has been the infrastructure environment. This is not just about using Red Hat Linux – it is also driven by what the Linux infrastructure teams are using or what they want you to use or are happy for you to use. In my case at my previous company, RPM’s were the standard for deploying applications on Linux infrastructure. We were encouraged to comply to the standard as best we could to ensure the Linux infrastructure team could better support our application. Once I had experience using this method, it was easier to reuse in other companies who were happy to support the method.

If you haven’t had any experience with RPM packages, there are plenty of resources online that will guide you through the basics. The Maximum RPM online book provides a good overview. Building RPM packages can seem daunting and many application teams at my previous company were using the RPM Maven Plugin to generate the RPM packages. It does simplify the configuration of the RPM build and integrate nicely into the release process if you are already using Maven.

Lets have a look at what the package will actually be doing:

• Create the directory structure
• Copy files to the directory structure
• Set file and directory permissions
• Add any symbolic links
• Run any pre/post/clean/verify installation scripts.

The RPM Maven Plugin can do much more but I will just go over the general setup in this post and some small enhancements that I have added.

What are we going to put into the package? In our case, we package everything that we need to deploy the application except for the environment-specific configuration.  This includes:

• The application package from Atlassian (standalone tar.gz version)
• The edited files to the application package (email templates, additional libraries, configuration)
• Plugin JAR files
• Control scripts (cluster and init.d)
• Optional dependencies like JDK and Apache (could also be managed via the dependency configuration)

These files and directories are placed in the build directory and stored in Subversion. You can use any other configuration management or source code management tool to store the files or directories as well. You could also use a file share or any other storage but it is better to have each package version controlled.

For those who are not familiar with Maven, the pom.xml (POM) file contains the configuration details to build a project and in our case the RPM package. Each of the above packaging details are specified in the POM file.

Firstly, we setup the standard POM variables and custom properties for use within the configuration definition.

...
<groupId>company.jira.packages</groupId>
<artifactId>company-jira-foo</artifactId>
<version>6.1.3</version>
<packaging>pom</packaging>
<name>company-jira-foo</name>
<url>http://www.company.com/company-jira-foo</url>
<properties>
<project.shortname>jira-foo</project.shortname>
</properties>
...

Next we need to add the start of the RPM Maven Plugin plugin configuration.

...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>rpm-maven-plugin</artifactId>
<version>2.1-alpha-1</version>
<executions>
<execution>
<goals>
<goal>rpm</goal>
</goals>
</execution>
</executions>
<configuration>
<version>${project.version}</version></pre>
<group>Application/System</group>
<copyright>COMPANY</copyright>
...

We have the RPM Maven Plugin initialised and we can first create the directory structure for the deployment. Here is an example of some directories that I create but note this is following a standard directory layout for the company where I work – it will vary depending on the standards that apply to your environment:

• /apps/jira-foo
• /apps/jira-foo/company-jira-foo (for the webapp)
• /apps/jira-foo/scripts
• /apps/jira-foo/logs
• /apps/jira-foo/jira-home
• /apps/jira-foo/plugins/installed-plugins

If the directory already exists then the package will not overwrite the directory. This is important, for example, the log files where you want to keep history in a production environment or jira-home where you want to keep your indexes, data and other files. We will cover cleaning the environment before installation in a later post.

We can then create the directories in the POM file.

...
<!-- Apps Root Directory -->
<mapping>
<directory>/apps/${project.shortname}</directory>
<filemode>755</filemode>
<username>apsadm</username>
<groupname>apsadm</groupname>
</mapping>
<!-- Web Application Directory -->
<mapping>
<directory>/apps/${project.shortname}/${project.artifactId}</directory>
<filemode>755</filemode>
<username>jirauser</username>
<groupname>jirauser</groupname>
</mapping>
...

We then define mapping the folders in the package to the structure we created.

• edit-webapp -> /apps/jira-foo/company-jira-foo
• webapp -> /apps/jira-foo/company-jira-foo
• scripts -> /apps/jira-foo/scripts
• plugins -> /apps/jira-foo/jira-home/plugins/installed-plugins

Note, that the edit-webapp is defined before the webapp but that mapping takes priority and overwrites the webapp mapping.

...
<!-- Mapping edit-webapp Files -->
<mapping>
<directory>/apps/${project.shortname}/${project.artifactId}</directory>
<directoryIncluded>false</directoryIncluded>
<recurseDirectories>true</recurseDirectories>
<filemode>755</filemode>
<username>jirauser</username>
<groupname>jirauser</groupname>
<sources>
<source>
<location>edit-webapp</location>
</source>
</sources>
</mapping>
<!-- Mapping webapp Files -->
<mapping>
<directory>/apps/${project.shortname}/${project.artifactId}</directory>
<directoryIncluded>false</directoryIncluded>
<recurseDirectories>true</recurseDirectories>
<filemode>755</filemode>
<username>jirauser</username>
<groupname>jirauser</groupname>
<sources>
<source>
<location>webapp</location>
</source>
</sources>
</mapping>
...

Finally, we add our symbolic links. One example is that we have a symbolic link called webapp to our webapp directory. This enables us to easily create scripts that work for all our applications and don’t need to be updated per application.

• webapp -> /apps/jira-foo/company-jira-foo

...
<!-- Symbolic Links -->
<mapping>
<directory>/apps/${project.shortname}/live</directory>
<username>jirauser</username>
<groupname>jirauser</groupname>
<sources>
<softlinkSource>
<location>/apps/${project.shortname}/${project.artifactId}</location>
</softlinkSource>
</sources>
</mapping>
...

This is just the basic features of the RPM Maven Plugin to build your package. I have uploaded an example POM file here with all the features that I covered.

There are many features of the RPM Maven Plugin that you can use to customise your package. There are further details available here. Also I use other maven plugins such as the Build Number Maven Plugin to add build numbers to the end of the rpms and the Exec Maven Plugin to run some cleaning scripts on the build directory. The full process can be easily customised to meet your requirements.

Once you have your POM file setup and added your package contents to the build directory, you are ready to build your package. Ofcourse Maven is a prerequisite – I am not going to go into the details in this post. And another important prerequisite is the rpm-build package. It should be installed and configured so that the user executing the build has permission to execute the rpmbuild command (usually requires privileged/sudo access). There are usage details for the RPM Maven Plugin here but in general when the RPM is the primary artifact for the project such as in the example, you can use the command:

mvn package

In my current environment, the build takes around 5-10 minutes. It might take a little longer if you add additional dependencies or files in your package. You will notice that the plugin creates a target directory for all the build files. The rpm file itself will be placed in this directory structure if the build was successful – usually somewhere under target/rpm/company-jira-foo/RPMS/ .

There can be many things that can go wrong with the rpmbuild and/or with Maven. Again, I wont go into detail here but if you want to give this a try and come across some issues, please feel free to ask, I may have resolved a similar issue in the past.

For Part 3, I will go into the Configuration stored in Subversion and how to bring the whole process together with the Deployment Method using Bamboo.

the jira guy

Deploying JIRA in the Enterprise – Part 1 – Setting the Scene

Last week I presented at the London Atlassian User Group. The title of the presentation was “Deploying Atlassian Tools in the Enterprise”. You can view the slide set here. I tried to keep the presentation on the general side but following on, I thought I would dive a little deeper into the detailed implementation for those who are interested.

I am not going to go over the whole introduction part of the presentation in detail but to quickly review…

• In my experience, the challenge of deploying a large number of application instances drove my teams to look for better ways to manage deployments.
• Its not just about large numbers of instances, it can help any environment with having a controlled way to manage application environments.
• The approach that you take will depend on the environment that you work in. Utilise the infrastructure and knowledge that you have within your team or organisation.
• This process is not just applicable to JIRA or Atlassian tools – it can be used for other application deployments.

A real world example of the number of instances of Atlassian application instances managed by my team

For most Atlassian applications, there are two main parts of the deployment – the application (binaries, configuration, etc) and the data (database, attachments, temporary files, etc) . I am not going to go into the data part in detail during this set of posts but focus on the application deployment. For the application deployment, there are three components that we will break down:

• Packaging using the RPM Maven Plugin
• Configuration stored in Subversion
• Deployment Method using Bamboo

A summary of the application deployment process

So, watch out for Part 2 – I will start on the detail into the Packaging with the RPM Maven Plugin.

the jira guy

Launching a new blog for “the JIRA guy”

When I first started managing Atlassian tools, I was thrown into the deep end in a fast paced environment. My role was to tame JIRA and Confluence. They were (and still are!) being used in anger and an integral part of that company’s infrastructure. It wasn’t long before I was known as “the JIRA guy” and I am sure there are many others who have a similar experience.

Managing JIRA has always had its challenges. In earlier times JIRA definitely had its quirks and was at times difficult to configure or missing functionality especially for larger organisations. JIRA has come a long way and Atlassian continues to improve their tools to the level that I can confidently recommend them to large organisations. But that doesn’t mean the challenges have gone – they are just different!

I see JIRA as more than just an issue tracking tool. To me it is a platform. I have seen it used effectively for so many processes. I have always admired Atlassian’s open source approach, plugin and integration support. It is forward thinking and has largely contributed to the popularity of the tools and the ability to extend their use for wider purposes.

What is this blog going to be about? To be honest, I don’t know exactly but it will have a wide remit – from new features, plugins and news, to technical deep dive, tips, tricks, scripts and everything else I have in my arsenal to help manage JIRA effectively. I am also interested in how to effectively promote JIRA within organisations. More and more, JIRA is seen as a critical service to organisations and it is important to manage stakeholders effectively.

So, if you are passionate about JIRA and Atlassian tools, I hope you enjoy following my posts.

the jira guy