Note:  This has been updated from its original post in 2015.

I will be upgrading some Ubuntu 12 servers in the near future and will be deploying Redmine 3 (upgrading from 2.2).  In order to prepare for that, and because I needed a Redmine instance for a project I will be undertaking (an store management app to integrate with Square), I decided to do a full deploy of Redmine to an Ubuntu 14 server I had setup to use in a course I was teaching at UCSD.  It had Apache, MySQL and PHP installed to support a BPM solution, so I would need to add RoR support to the server, and get Redmine running.  Although this wouldn’t be fully parallel to what I will be doing in a couple of weeks, it would serve as a refresher for me, and alert me to things I need to be aware of.  I was mostly able to follow the Redmine install documents, but there were a few exceptions that I wanted to document, as well as some additional pieces that the Redmine docs don’t really address.

The first step is to get Passenger running.  The instructions are simple and straightforward, and this installation worked just fine.  Once installed, you need to install Redmine.  The first question is about Ruby in your environment.  I attempted to use RBEnv as that seems to be growing in popularity.  After hitting a number of snags, I opted to uninstall RBEnv and switch back to RVM which I’ve had success with.  I suspect that had I wanted to try for a bit longer, I could have ironed out the RBEnv issues, but I decided to move on.  Basically what was happening was that I had first installed it in my home account, then realized that this would be problematic and located instructions for doing a multiuser install, as I want to run my app under a service account, not as my user.  I should point out that I did all of this work with a “standard” user account that was also in the sudoers list.  I believe that I will be able to effectively manage the upgrades over time with this approach.

I followed the multiuser RVM installation instructions located here.  Again, this is straightforward, just follow the instructions and make sure you are working in the multiuser part.  After completing the install, create a Redmine user (this is after the whole Redmine install is completed) to act as a service account, and grant it the appropriate permissions.  The account can be locked and will work just fine, so people cannot attempt to log on to your system if they guess the service account name.  Once RVM is installed, you can install whatever Ruby you wish (or rubies).  I installed 2.2.3, then made it the system default (RVM use 2.2.3 –default).  I give the redmine user ownership of the files, tmp and log directories recursively as that user needs to write to those directories.

Once that is complete, create a database for Redmine and select a user for this.  As a matter of safety, I create a specific user to access the database, and limit connections to localhost only, for the Redmine user only.  If you have PHPMyAdmin running this is a simple enough task.  In general, and MySQL defaults to this, you shouldn’t be directly connecting to it from another machine.  If such a thing is necessary, you should do it on an IP by IP basis, so that you are allowing just a specific IP.  I have an implementation at a client where there is an integration between MySQL and another tool that requires a direct external connection for read only.  I created a read only user, and allow it to authenticate only from that one IP address.

Install both SVN and ImageMagick.  The OpenID libraries are dependent on whether you want OpenID.  ImageMagick is optional as well, but the default with Redmine is to use RMagick to handle images, and it will not install without ImageMagick.  You’ll want images, so just go for it.  There is one thing that won’t install with the RMagick gem, if you just install ImageMagick (sudo apt-get install imagemagick).  You will also need to install a dev library by using sudo apt-get install libmagickwand-dev.

At this point, you will want to install the MySQL libraries and drivers so that the MySQL2 gem will install correctly.  Run the following.

sudo apt-get install ruby-mysql  libmysqlclient-dev (the old libmysql-ruby was a transitional package).

At this point download Redmine.  I used SVN and downloaded it to a directory in usr/local.  Then I created a symlink to the directory in /var/www.  From here, configure the database.yml file as directed in the installation directions, then install bundler (gem install bundler) then install the required gems using bundle install –without development test (you can exclude rmagick at this point, as well, if you aren’t going to use it).

Next, create the application secret by executing, from within the Redmine directory (also applies to any app) rake secret.  This generates a secret key and then captures it in an environment variable.  Rails 4 requires that you create a config/secrets.yml file and place the following two lines, in lieu of using the config/initializers/secret_token.rb file:

production:

secret_key_base: ‘secret generated above’

A key configuration item, as it turns out is:

PassengerResolveSymlinksInDocumentRoot on

Without this, you will simply receive a list of items in the public folder.  This appears to be Rails related as I have an older version of Redmine running the same version of Passenger, but on Rails 3 that doesn’t require this directive.  Redmine 3, using Rails 4+ appears to have this requirement.

This secures your application by not placing your secret key in any of the application files (it exists only at the OS level).  Alternatively you can set secret_key_base: <%= ENV[“SECRET_KEY_BASE”] %> and load it from the profile of the redmine user (assuming that this is the user which will be running the application).

Finish by creating the database schema, default data set, and setting the file permissions as indicated.

The last step is to let Apache know about the new application.  In general, I prefer, like many, to create a redmine.conf file in the /etc/apache2/sites-available directory.  Within it, create the appropriate virtual host entry pointing to the public directory inside of the redline application (e.g. /var/www/redmine/public) for both the document root and the directory.  You may want to specify the following lines to ensure that Passenger runs correctly, but in my experience they are not entirely necessary:

PassengerRuby (path to ruby, see below)

PassengerDefaultUser <service account created earlier>

If you are upgrading from an older Ubuntu version, vs. performing a clean install, make sure that you don’t have any passenger configuration items in your apache2.conf file (typically in /etc/apache2/).  The passenger binary itself should be loaded as a module in Apache, and the PassengerRuby and Default user, although they can be defined in the apache2.conf should probably be specified within the virtual host definition.  It wouldn’t hurt to keep things in the apache2.conf file, but by keeping it in the sites definition, the option exists to run different rails apps using different rubies and users.  If, as I have had in the past, you have a legacy app, you can run it under a legacy ruby at the same time a newer app is running on the latest ruby.

To determine the path to ruby, run the following command:  passenger-config about ruby-command, then copy the path to the redmine.conf file as indicated.

Enable the redmine site (a2ensite redmine), restart apache, and all should work.

As the world has become overwhelmed with spam, marketers are increasingly looking to tailor there message more to the recipient. It began years ago with including your target’s name, so that it appears you are receiving a personal message, when you are simply getting spam. Now marketers are becoming more creative, trolling news services to look for more specific information about the company you work for in order to craft a message that appears as if their company is truly interested in yours. Unfortunately, such automation can lead to abject failure, and accomplish the exact opposite (to be fair, I don’t know for a fact that automation is involved in the story I’m about to share, I can only assume that an actual person would have been more careful).

The company I currently work for had an unfortunately negative result in a key clinical trial recently. Such things are not unusual. In fact, clinical trial failures are more the rule than the exception in the pharmaceutical business. We all wish it were otherwise, but it isn’t. So, a consulting firm that provides a variety of services around the pharma industry apparently has some robots that troll Biospace.com looking for interesting headlines. One of our recent press releases had the following phrase “announces top-line data from” followed by the type of trial in question. From the title of the release, there would be no way to know whether the data was positive or negative. Anyone familiar with the industry would assume it was negative, simply because you are wrong more often than right in drug development. However, the individual who constructed this robot, apparently didn’t know that, so I received a congratulatory email about the announcement. The email started with “Congratulations on your recent success! We read about on biospace.com. Continued success to you all on this project.” The email then went on to tell me about the company, which purports to be a life sciences solution provider. They claim to have expertise in our industry. Given that they don’t understand the success rate of clinical trials, that is clearly not the case.

This blog cites a study that looked at failure rates from drugs transitioning from phase 2 to phase 3. Per the embedded diagram, that number sits at about 68%. A company that understands our industry would know that, and would not do anything as foolish as assuming that a pivotal phase II announcement is going to be positive. As a result of this email, I will remember the company’s name, and won’t waste even 30 seconds talking to them about an engagement. If they can’t get something as trivial as this correct, I’m not going to look to them for understanding in any of the more complex aspects of our business. I’m being kind and not including their name in this blog posting. Perhaps I should rethink that.

At any rate, this gets back to a theme that I have been thinking about a lot, which is success recipes for IT leadership. I’ll be teaching a course at UCSD Extension this summer that is focused, at some level, on these recipes. Unfortunately, like drug development, there is no magic bullet that guarantees success, so we’ll be exploring approaches that can help, not dictating a solution. What we can do, and will do, is look at approaches that can guarantee failure. The most common one that has plagued IT and IT related initiatives since the dawn of the computer era is a failure to understand the business you support. This particular email is yet another version of that story, where an IT solution was apparently crafted (the automated robot) that had a decision algorithm that was clearly not written by someone with an appropriate level of drug development understanding. Interestingly, and sadly, this means they have no chance at an engagement with me. I don’t have the luxury of that sort of risk taking. If they do this to other companies (or other people learn of their lack of expertise), what will this mean to the long term success of their organization?

Several months ago, UCSD extension assembled a group of senior IT professionals as an advisory board for their certificate programs in the IT space. One of the discussion threads was that software engineers (read programmers, business analysts, whatever) were coming out of school with inadequate knowledge of the business to successfully provide services to the businesses they were to work in. The other weak point seemed to be in the knowledge of the importance of systems integration. All of this led to a new class to be required as part of the Software Engineering Management certificate program. This class is focusing on integration from three perspectives. Integrating IT with the business, integrating IT with itself, and integrating IT with the regulatory environment of the business.

This isn’t a sales pitch for the class, but rather a bit of thinking out loud as I continue to struggle with exactly what I’m going to teach in this class. Yes, I’m the person who volunteered for this first effort. Of course, I’m also the person who kicked off the aforementioned discussion, so I definitely deserve this. What was I after, exactly when I initiated that discussion? And what is the solution to the underlying problem?

I started writing this post in early December. It is now the day of the first session, and I guess I’ve come up with at least a preliminary answer. I have a feeling it will morph a bit along the way.

For our first session we are going to look at both software failures – both familiar and a few I’ve been involved with, as well as successes. One area of focus will be a period of time when myself and a small group of others developed multiple systems that were broadly enabling, relatively popular, and were used for almost a decade. Initial attempts to replace these systems with commercial products were a failure. I refer to this point in time as my Shangri-La. It was an almost mythical IT world. Once I left, however, I’ve never been able to make it back.

When I look at Shangri-La, as well as other successes, and most notably failures, I find only one common theme. Tools varied, methodologies varied, but the one common theme was the degree to which developers understood what the users needed. Not what they wanted, but what they needed. In order to return to Shangri-La, we need to figure out how to develop this understanding.

To that end, I’m going to be focusing on the discipline of enterprise architecture as the most likely candidate for providing a general approach to get us, if not to Shangri-La itself, to a place where IT systems are more successful more frequently. My thesis, which is all that it really is, is that software projects are most successful when the developers, through whatever mechanism, really understand the business, its drivers, and its needs.

I’ll try to post back about this topic as we proceed.

Enterprise architecture(EA), as a specific discipline, dates back to the late 80‘s. However, it has been very difficult for EA programs to gain much traction. Very often, the architecture program simply becomes focused on creating as-builts of existing applications and infrastructure. Indeed, unless you are starting a completely new endeavor, some amount of retrospective work will become necessary. Prospectively, EA programs fall by the wayside. My own experience with an enterprise architecture program supports that. I engaged with the group as I thought the modeling tools might be useful. It never occurred to me that the architecture group might be of use in engineering new systems.

So, EA groups need to prepare and market themselves internally as resources to help facilitate new information systems engineering efforts. That would be a start, but it will not be enough. Two other things are necessary. The first is that the CIO needs to mandate that the architecture group isn’t simply a resource. Their tools and methodology must be the only path to travel. I’ll explore this topic in more detail down the road, as I prepare for a course I will be teaching early next year. The other thing needed is a significant change in the perspective of the typical CEO. Many CEOs look at IT as simply a provider of widgets, and as the group that makes sure computers are on desks and that email is on. That is a fundamentally flawed perspective.

As I sit here typing this on my iPad, and with an iPhone in my pocket, I’m led to reflect on the degree to which information technology has become ingrained in our daily lives. My devices help me find where I’m going, track my nutrition and exercise, allow me to communicate with others in a variety of ways, and help me stay plugged in to what is happening in the world. The devices have become integrated into my life. They improve it, and, yes, sometimes detract from it. But either way it has become integrated. Engineers and app developers all think about life processes and how IT might make them better. More and more frequently they re-imagine the processes themselves.

The modern CEO needs to take this perspective and expect their IT function to engage at the level of the business processes. Not only enhancing them, but even rethinking them. There are two different models that come to mind to effect this. One is an organization where IT is centralized. All information systems are built and managed by this group and they provide resources for business design to each line function. The other is a more distributed design, where IT is responsible for processes and standards, but execution is embedded within individual business units. I will explore both of these models in the future, but in either one we see that IT is not primarily a technology implementer, they are a business process designer. Failing to make this philosophical change will always limit the possibilities for an organization. The CEO shouldn’t just visit his head of IT when his phone is no longer receiving email. He should visit him every time he is pondering the way his organization runs.

Ever since the dawn of the computer age, we have been seeking a mythical computer utopia.  A place where computers are truly an integral part of our lives and truly bring benefit to them.  I’m reminded, to some degree, of the car in the Robert Heinlein novel “Number of the Beast” that was in virtual constant communications with its owners, was able to write its own programs, etc (It may have been more of a computer that had an interface in the car, I haven’t read the book in probably 30 years).  Steve Jobs certainly had this vision, as is discussed in the book “Insanely Great”.  However, there is always the sense that we haven’t quite gotten there yet.  The ring is just out of reach.  It is always dangerous to say that thus and such a technology will allow us to grasp the ring, so I won’t go there.  However, I will say that Cloud technology has allowed us to move closer to the ring.

We all think of the cloud as a way to store data or deploy applications outside of our own data centers, and that is still its primary objective.  There is a necessarily positive side effect of doing this.  That is, I can now access these applications and data from anywhere.  When I started moving more of our functions into the cloud, it was, in part, so that our small team could function anywhere.  It dawned on me this morning, that this, still, was only the tip of the iceberg.  In fact, it is the ability to work anywhere, and on any device, that is key.  I am working in my home office right now, with no devices that belong to my company.  Yet, I needed to see what time a couple of meetings were today.  I looked on my iPad (because it was the closest device to me and was open at the time).  As I opened the collection where my calendar was, I was actually looking for a planner app I had been using a while back, but it wasn’t there anymore.  I then remembered why.  I had run across several planner apps that were something like the Covey planner system.  I like the way they handled tasks and presented everything holistically.  However, the bad ones were almost unusable, but the good ones used an independent system of tasks and reminders from the iOS ones.  Suddenly, any tasks I created were only available on my iPad.  I stopped using them.

My calendar, however, is available on every device I own.  From my office laptop, to my personal laptop, iPhone, iPad, you name it.  So are my reminders.  So are my emails.  So these apps are actually useful because they allow me to have my data at the ready wherever I am.  I’ll admit the most important device in this regard is my phone.  Not because I use it for that much, I don’t.  It is because it is the one device that is always with me.

In order for this sharing of data between devices to work, the data must exist in the cloud somewhere.  The alternative is to perform an inter computer sync, which was quite the rage for a while with different applications, but is beginning to go the way of the dinosaur.  Calendar, email, tasks, etc., are all in the cloud for a lot of us.  A growing number of applications will use a cloud file storage solution (Box, DropBox, Skydrive, etc.) as a backend.  I use DropBox extensively in my personal life simply because it permits me to have access to a lot of information no matter where I am.  Evernote is increasingly the poster child for this sort of functionality.  As I’ve looked at applications for things like handwritten note capture, I’ve dropped a great number of them because they either don’t provide a seamless multi device utilization model.

You see, the cloud allows our computers to be more integrated with our lives, and truly become servants.  Although a legitimate argument can be made that computers have taken over our lives, I think that is looking at the situation incorrectly.  When we had to go sit at a desk and fire up a machine, wait for the boot cycle, then dig for information in order to achieve our goals, we were slaves to the machine.  Somewhat like Oliver Twist begging for more, we would approach the machines like they were our masters.  Now, I expect my information to be at my finger tips when I want it, on my terms.  A bit more like Downton Abbey, where I can pull a cord from whatever room I am in, and a servant will arrive ready to provide whatever I need.  Applications that cannot perform this task are of no use to me, and must be banished.

When I first began thinking about using the cloud, it was all about risk mitigation and cost management.  Now, I see it more as how to enable computers to truly be integrated in our lives.  The next step, is to begin to educate my user base on this, so that they can realize these benefits.

After a week troubleshooting Solr and getting it running on Ubuntu for a test version of the Limspec app, I think I am allowed the bad humor.

I recently posted a summary of how I deployed a Rails application on a Debian VPS using Nginx.  The steps are fairly similar to what you would do with Ubuntu.  For the Limspec project I’m working on, we’re using Apache – mostly because that is what I started with, but there is no practical reason to not use Nginx.  The instructions in that post should work well, but you would need to install the mod_rails module for Apache and create the appropriate sites-available files.

The application that I was deploying is a fairly simple application for my Church’s folk dance director to use for managing the dance program, and at this point doesn’t utilize any search, let alone full text, so the instructions didn’t include what is a fairly critical element to some of my other applications, and that is Solr for full text searching.

Solr is from the Apache Lucene project, and is a very powerful enterprise search platform.  I had implemented it for Limspec quite some time ago.  However, we had a VM meltdown a few months ago, and the replacement VM only seemed to have pieces and parts from the previous VM.  This was a huge problem getting Limspec deployed again (in addition to this, I no longer had root access, which is probably good as I was forced to set things up in a more secure fashion).  When all was said and done, however, I had forgotten to check Solr out.  It was working fine on my dev machine, so all search related tests passed with flying colors.  Important safety tip with TDD.  Even if you test exensively on your dev machine, you need to be very aware of those things that are deployed quite differently in production.  Solr is one of those.  Although it appears that you might be able to use the sunspot_solr gem in production, the developers who created it indicate they only intend it for use in development.  After hours spent trying to make it work, I tend to agree.  I could never quite get it running, so I finally gave up.

So, not having taken notes on how I installed Solr the first time (well over a year ago), I set out to do it again.  Of course, there is a newer version of Solr, and a newer version of the sunspot_rails gem.  When I was rebuilding the actual Limspec server, I created an Ubuntu VM on my desktop to try everything on first.  So, I continued to use that VM to figure out Solr.  The following instructions are based on adding Solr to my local VM, which is running Precise Pangolin (Ubuntu 12.04 LTS).

Jetty

Solr is a Java servlet, and so needs a servlet container of some sort.  Previously, I had used Tomcat.  However, Tomcat is fairly memory intensive and is really only necessary for more complex Solr installs (such as multiple instance).  Of course, if you are already using Tomcat for other purposes, it would probably make more sense to deploy Solr with Tomcat than to run yet another web server.  If not, Solr comes complete with its own copy of Jetty.

From here on out, everything is fairly straightforward, but it took me a while to figure it all out.

The first step is to download the latest version of Solr, or at least the version you are interested in.  I opted for 4.6, which is the most recent version at the time of me writing this. Once you’ve downloaded the tar file, you can untar it wherever you’d like, as you’ll be copying a subdirectory out to another location.  There are a number of locations you can use as your Solr home.  I recommend that you take a look ahead to the startup script from the Solr Wiki and choose one of the standard locations in the script.  Remember that I’m looking to use Solr in support of another goal, so I want to minimize anything that makes my installation non-standard.  Taking this approach, as well, makes maintenance of the application and installation by other users, much easier.  I chose opt/solr as my home, so I executed a mv solr-4.6.0/example /opt/solr.

The next thing to do is decide whether you are going to run single or multicore.  I tend to have staging instances on the same server as production, so I want multicore.  To deploy for multicore, within opt/solr, delete the solr directory (i.e. rm -r /opt/solr/solr), then mv /opt/solr/multicore /opt/solr/solr.  This then gives you a multicore deploy.  By default, you have two cores in place, core0 and core1.  You can certainly stick with those names, but I wanted names that would tell me what those cores are being used for.  If you want to change the names, first execute a mv core0 <newCoreName>, then update the solr.xml file within /opt/solr/solr to indicate the new names and paths.  That is, change

<cores adminPath=”/admin/cores” host=”${host:}” hostPort=”${jetty.port:8983}” hostContext=”${hostContext:solr}”>
<core name=”core0″ instanceDir=”core0″ />
<core name=”core1″ instanceDir=”core1″ />
</cores>

to:

<cores adminPath=”/admin/cores” host=”${host:}” hostPort=”${jetty.port:8983}” hostContext=”${hostContext:solr}”>
<core name=“newCore” instanceDir=“newCore” />
<core name=“newCore2″ instanceDir=“newCore2″ />
</cores>

or whatever names you chose.  I tried to use names with spaces in them, and Jetty didn’t like that.  I’m not sure if it was because of the spaces, or the fact that the name wasn’t the same as the instanceDir, so I just made them both the same and the problem went away.

If you want to test your installation at this point, you can fire up jetty by running java -jar /opt/solr/start.jar.  Then go to http://yourserver:8983/solr, and you should see both of your cores in the coreadmin screen.  Note that you can change the port for Solr within the solr.xml file if you would like.

Starting Automatically

The next thing you’ll want is for Solr to launch on startup in the background.  This proves to be easy to do.  First, download the jetty.sh script linked to from the SolrJetty page.  If you looked ahead and parked Solr in one of the standard locations, the script will work fine as is.  Place the script in the /etc/init.d directory and make it executable.

Next, follow the instructions for creating the /etc/default/jetty file for the various parameters jetty will need on launch, setting the Jetty home Java home, jetty user, etc. as appropriate.  If you opt to run it under a non-privileged user, such as solr (always a good idea), then follow the instructions on this page for creating the user and changing ownership of the solr directory.  Also set the user name correctly in the jetty configuration file (/etc/default/jetty).  Finally set the run levels.  I just used the defaults (update-rc.d jetty.sh defaults).  I should note that every linux command you see on this page, I preface with sudo, as I’m not operating as root.  More than likely this will be your situation, or should be.

I always prefer to bounce my server after making a lot of these changes, to make sure that everything will start as it should.  So, I recommend doing that, then visit the solr admin page again to make sure everything is loaded.

Configuring for Rails

As I stated before, this is for my rails application, so I need to do a few things to make that work.  I’m assuming you’ve followed something like this to install sunspot_rails in your application.  If not, then do that.  Once completed, you will have a schema.xml file in your <rails_project>/solr/conf folder.  This needs to be copied into the conf folder for each core you are going to be using with your rails application (i.e., cp <rails_project>/solr/conf/schema.xml /opt/solr/solr/core0/conf/schema.xml).  If you have an old schema.xml as I did, that predated Solr 4, you will be missing a key field definition, that needs to be added back.  Sunspot has been patched so that if you just installed it, you shouldn’t have a problem.  If you get an error message about field _version_, then add this line in the fields definition section of schema.xml:

<field name=”_version_” type=”string” indexed=”true” stored=”true” multiValued=”false” />

Next, make sure your sunspot.yml file located in <rails_project>/config is accurate with regard to port and path.  One thing that wasn’t obvious to me, and sure is problematic is that the path is relative to the Solr directory.  That is, if your Solr directory is /opt/solr, and your core is /opt/solr/solr/core0, then the path in the yaml file should be /solr/core0.  The leading / is important as you will get an error otherwise.

Finally, you will want to run a rake sunspot:reindex from within the your app directory.  If you get one of those great rake errors about having the wrong rake running, do a bundle exec rake sunspot:reindex, and all should be well.  I typically run a reindex on every deploy, just to make sure everything is good.  Sunspot will only index new and modified database rows, so if you want pre-existing rows to be searchable, then you need to reindex.

My next step is to run all of this on our production server.  I’ll post back an update on how that goes.

UPDATE:  Ran this on the production server, and all worked as it should (provided you follow the directions, which I didn’t at first, but that’s another story).