Day 237 – Just Like a Run

It was always going to be a full day today. Weeks with Melbourne travel in them are always a little more hectic than any others to begin with.

One of the last meetings of my day was with a counterpart in the business. I’ve extended the concept of employee one-on-ones out to the business as well. We are after all trying to meet business needs, and the best way to keep in-the-loop in that regard is to have regular conversations.

It’s always good to catch up.

When I got back to my desk there was a message indicating the Wiki system was having some issues. Quick click. No, no dice for me either.

Login to the VM. Restart service. No dice.

Upgrade OS. Upgrade Wiki. Restart service. No dice.

This was really not the last thing I wanted to deal with for the day. One hour before my gym class starts. Login, cancel, sulk.

Have you ever tried to read and comprehend a Tomcat/Java log file full of warnings?
No, neither had I. I had no idea what I was looking for.
I looked at it two times without much luck.

Third time was luckier though.
I noticed that the most verbose messages were detailing database connection issues. Surely not? Why now? What has changed?

To keep a long story short. It turned out that the database the system was relying on had never been properly noted. The database had been handed from one team to another. Other team was consolidating, and didn’t realise the database server was being used for anything else any more. Database server gets turned off.

A quick fix.

Alas, 15 minutes too late to get me to the gym in time.

Saturday will be my next exercise now.

Day 85 – Almost a Disaster

I planned the day somewhat hopefully, and I really should have known better. An internal Confluence system was moving between servers, and I wanted to take the opportunity to do a software upgrade.

Calendar/Plan before faced with reality:

  • 4:00pm – Networks guy takes system down, does last backup and then restores on new hardware
  • 4:40pm – I run an upgrade to the latest version, and everything is sunshine and rainbows
  • 4:50pm – Worst-case-scenario, I tell networks guy to re-do the restore and we’ll live with the earlier version
  • 5:30pm – Done and dusted

Overlaid on this plan, I had booked a Pilates Reformer class at 5:30pm, so that if everything goes well I can leave the office at 5pm and make my class. And if not, I just cancel it at 5pm when things go pear-shaped.

At first, everything exceeds expectations. At 4pm the migration starts, and it completes in only 25 minutes! Awesome!

So, my turn arrives to upgrade the software. I run the installer… done in 2 minutes. Awesomer! And more-over, when I browse to the new install, the pages load and everything looks happy. Until I try to escalate to Admin privileges.

Oh dear, I broke authentication.

The system spits out lots of messages at me, and I decide there is just no time to fix this on-the-fly, so I make the call to restore the backup and just live with the earlier version of the software.

No big deal. It’s a shame not to be on the latest version, but on the upside, it is now 5:35pm and after another restore it will still be shy of 5pm, so at least I will make my gym class.

The restore completes.

As part of the software upgrade, the installer upgraded the database server which resides on a separate host that is now incompatible with the earlier version of the software.

Okay, so the only way is forward. I cancel my gym class, since that’s just not going to happen now.

I make a backup copy of the software folders, and re-run the installer.

There was a message about something to do with “cacerts”… that sounds like it might be relevant. Another Google search points me to the location of this file in the installation.

It turns out that the certificate file got replaced as part of the update. The version of this file in the backup I made earlier has a different size from the one that’s there now.

So, I make a backup of the new version, overwrite with the old version, and see what happens…

And the Software Gods Smiled Upon Me, and Behold! The System Now Worked. And It Was Magnificent. In that way in which everything is magnificent when it works right after you thought you were going to be there all night.

I guess I’ll have to go on a long walk after dinner to make up for the missed exercise tonight. I could do with the soothing evening air.