My Colocation Deployment

How this Started

Hosting this site along with a few others, I have come to realize that 5mbps of upload bandwidth is not enough for web hosting.  This site’s home page sometimes took as much as 23 seconds to fully load.  This was completely unacceptable to me, as I am also hosting a website for my DJ business, and need that one to load as quickly as possible.  So back in late September or early October, I made a post on r/homelab asking for people’s recommendations for cheap 1u colocation.  I had a Poweredge R330 that I wanted to run with more WAN bandwidth, and highly available WAN and power.  I figured this would get me the faster loading times and better uptime I was looking for.  Someone reached out and told me that they had a few rack spaces available in their colocation rack that they weren’t using, and were willing to rent out a space to me  for a reasonable price.  We discussed logistics and agreed that when my hardware was properly configured, I would send the system out to him to install.

Configuring the Hardware

The first thing I did was grab the latest release of ProxMox VE and install it to a flash drive inside the R330.  I thought this was the best way to run it because it would limit wear on the single SSD I was going to install for VM storage.  This soon became an issue because they system didn’t like booting off of the flash drive.  I had issues where the volume wouldn’t mount in time and the drive mounting timed out during boot.  I temporarily solved this by setting a higher rootdelay in the GRUB configuration, but the issue continued after a week or so.  Eventually I gave up and reinstalled PVE on the SSD.  I test booted the system about 25 times, and it came up every time with no reconfiguration of the GRUB file.  This ended up being for the best because my R710 that booted PVE from an internal flash drive started having issues related to the age of the flash drive (about two years old) around this time.

I configured my networks and internal pfSense with the WAN information provided to me and the LAN settings required to make it a part of my existing network infrastructure, including adding an OpenVPN client to bridge the colocation LAN to my existing home LAN.  I also setup a Ubuntu desktop VM that I could use to access the pfSense web UI if I screwed up the network settings at some point  (I have a separate management LAN that I use for the ProxMox VE management interface and the iDRAC interface.  My pfSense router at home maintains a site-to-site VPN so I can always access these management interfaces).


When I was done with all of the configuration and testing (making sure everything came up correctly on reboots, verifying network connectivity, making sure the drive was healthy…) I wrapped the server in anti-static bubble wrap.  After using nearly the entire roll, I boxed the system up with the power and network cables and rails.  I used almost an entire roll of packing tape to make sure the box didn’t open in transit, and took it to the UPS store for shipping.  For two weeks, I checked the shipping tracker daily waiting to see when my machine had safely arrived.  When it did, I was finally able to let out a sigh of relief.  The system was installed the day it arrived, and I was ecstatic to see the virtual pfSense pop up in the OpenVPN connected clients log.

Getting Setup at the New Datacenter

I tried pinging the system, the router, and accessing the management interfaces to verify everything was working,  and it was.  Well, almost everything was working.  I was unable to log into my iDrac, requiring a KVM module to be connected to the server so I could reset it. Once that was taken care of, I was able to migrate my WordPress installations and relevant MySQL databases over, along with my authoritative DNS server.  A new Traefik server proxies all incoming HTTP and HTTPS requests.  I was able to setup a couple new services, like GitLab (self-hosted GitHub, a system for collaborative software development), and a backup FreePBX server for my home telephone system (if the main PBX goes down, the phones should switch to the backup PBX for inbound and outbound calls).


So far, I could not be happier with the performance and reliability of this new Colocation setup.  I highly recommend it if you are trying to manage your own hosting but require uptime and bandwidth that you just can’t get at home.

Delaware Valley Regional High School Staff Talent Show 2018

December 20th was a busy day for me, I had a Calculus final and a talent show to play in all in just a couple of hours.  I wouldn’t have been able to do it without the help of my  (now ex girlfriend), who drove us from Montclair back to my house so I could grab my instruments in time to fly back to Del Val to get setup.


Ricky and Justin performing Metallica’s cover of Bob Seger’s “Turn The Page” at Delaware Valley Regional High School’s 2018 Staff Talent Show

The show went well, but we didn’t win (again).  We don’t care much as we were just looking for a place to rock for a few minutes.  I’m sure we’ll try again next time.

Justin sings the line “What She Said” during the 2018 Staff Talent Show with Josh Paul and Ben McPherson. Photo Credit – Delaware Valley Regional High School

We barely got a soundcheck, but I think we sounded decently good.

Bringing In The New Year With a Cluster of Network Problems

I always seem to have issues, particularly with my network, in clusters.

I caught the flu New Year’s Eve, and was out of service for about a week. During the first day or two of being sick, my R710 running Proxmox and all of my VMs had an issue and kernel panicked. I didn’t get any details as to what the exact issue was because I was too out of it.  I did try to restart it, but had no luck.  It never managed to fully boot.  While I was sick, I was able to re-install Proxmox to a new RAID 1 array (PVE was previously installed on a a flash drive, and I think that had something to do with the problem) and restore all of my backed up VMs. I was still pretty out of it while I did this, but everything worked fine after and I was relieved that everything was working again – home-assistant was controlling all of the outside lights, the telephone system was working, and the websites I host were back up.

Around this time the server I shipped off to colocation was installed and I was looking forward to getting services moved over before I had another issue with my infrastructure at home. This couldn’t happen soon enough. The next day (Saturday) I was feeling better, and the universe decided to test just how much better I was feeling. Sometime around one pm, the power went out. I got the generator up and running within a couple of minutes, but found out that three of my four UPS units do not run on generator power. After about twenty minutes, I had to power down the newly-rescued Proxmox server, and the file server with over 180 days of uptime. I was not happy about this.  My plan was to work on migrating services over to the newly installed colocation server, but I couldn’t do this if the primary server was down.  With the power out and most of the network down, I worked on cleaning up my cabling. I worked on the cabling in the back for an hour or so and when the power came back on, the back looked a bit better. I watched as everything came back online for the second time in two days, and once everything was working, I thought that I wouldn’t have to deal with this issue again for a while, as the R710 used to be very stable.

Everything was stable Sunday, so I thought I was in the clear.

The following day (Monday), I decided to spend another hour or so in the lab and work on cleaning up the cabling for the client network. I didn’t take a before picture, but it definitively looked a mess. I’m pretty happy with the way it came out. Again everything seemed stable so I thought I was in the clear – the cluster of issues was over.

Rack photo - January 2019
My rack as of January 2019

Nope. I woke up on Tuesday with devices having a hard time connecting to wifi, or not connecting at all, and my IP phones were showing as unregistered. I went to the lab and saw that the R710 was completely off. Looking down, the UPS that powers it was completely off. I have no idea what could have caused this. The cats can’t turn it off because they can’t hold the power button, but something weird must have happened. I don’t see what would have caused this. Regardless, I turned it back on and watched all of my services come back online for the third time in a week. Now on to the WiFi issue. Devices either taking a long time to connect, or not connecting at all. I looked at the UniFi dashboard and saw that one of my APs was showing as disconnected from the UniFi controller. I disconnected this one and the WiFi issues seemed to stop. A bit later I thought to try connecting the offending AP to a different switch port and the issue went away, so I must have connected it to a port configured for something weird when I cleaned up the client network cabling the previous day.  Fortunatly the cluster seems to be over now and everything is running smoothly.  Fingers crossed it stays that way.

Lab January 2019 Update

Rack photo - January 2019
My rack as of January 2019

I came down with the flu last week, so towards the end when I was getting better, I had some time to working on cabling in the rack.


Most of the differences here from before are just cleaning up the cabling, adding a couple of UPSs, and the addition of a zwave stick for home assistant.


The R710 runs:

  • WordPress (not for long)
  • Tekkit
  • BIND Authoritative
  • Accounting (Custom written)
  • Radius (Not working yet)
  • Unifi
  • Lime Survey
  • Apt Cache
  • Bitwarden
  • Transmission
  • Simple Invoices
  • Nginx Reverse Proxy
  • Apache Server with various applications including Nextcloud
  • FreeIPA
  • email
  • FreePBX
  • UPS Monitor
  • MySQL
  • Home Assistant

The 2950 Runs:

  • Plex
  • Samba
  • Netatalk
  • NFS

The Dimension E310 Runs:

  • pfSense

Private Party with the Midnight Ramblers

I am writing this in the car on my way home from this one…

I was contacted a month or two ago by a friend of my father to provide lighting for a private event. The band is called Midnight Ramblers, and they were a pleasure to work with. Their setlist was packed with older classics; from Santana to The Who, this was probably the best setlist I’ve ever programmed light shows for. My lighting prep started way later than I would have liked, as I did not get a setlist until about two weeks prior to the event. As such, programming the shows was a little more crammed and I didn’t get to get as intimate with the songs as I prefer. The show, however, did not reflect this. The crowd was very well receptive to the various effects used for different songs, and the band appreciated the connections the lighting made with the songs. Setup was easy in that I was only running lighting, and no PA. Setup was difficult, however, because I was trying to maneuver around the backline which had been setup before I arrived. I have made a mental note to myself to at least try to get the major pieces in place before the backline goes up.

No technical issues arose during the show, other than me forgetting to make sure the VNC server on my fog controller computer was enabled. (I have one computer connected to the fog controller, and remote into it with my new tablet) Another mental note was taken there to verify that all computers are configured as they would be needed for the show before leaving the house. I was able to fix that issue during intermission, and by the time the sun set and the second set started, my fog was pumping, my position palettes were set, and I was ready to go. The second set went off very well, the lights were effective, and the audience was much more responsive than than they were in the first set. (Because the sun had set). All in all, the show was a great success, and gave me a couple of important takeaways.

  • Always verify show computers are configured before arriving at the event
  • Request to get in before the band to get lighting up before backline.

Del Val Dreams For Kids Basketball Game 2018

We were tired after the Friday night Knight’s Challenge, but pushed forward on through the next night.  I met up with Ophelie outside Del Val at around 10 in the morning, and we got started getting equipment into the building.  Once everything was inside, we had the arduous task of untangling the cables from the night before.  This took a while, but we had arrived so early, I really wasn’t worried about anything time wise.  The system was a little more in depth than I normally have, as the acoustics of this space are particularly challenging.  The first step was to setup and tune the Main PA.  This was my standard Yamaha Club V rig  (two S115vs and one CW118v).  It sounded okay when running through the automatic tuning program in the DriveRack PA2, but when it started to turn the volume up to be loud enough for the measurements at the back of the gym, the reverberations started to become overwhelming.  The generated EQ curve of the gym sounded alright once it was applied.  Once the automatic tuning was done, I made a couple of adjustments per my preferences, particularly a pretty decent boost at 75 hertz to get a little bit of the kick drums in songs to carry throughout the gym.

After the Yamaha rig was tuned and ready to go, I turned the volume down quite a bit; to the point where announcements and music were at a comfortable listen level for a sporting event about half way down the bleachers.  Then we setup the delay speakers.  I have a pair of Behringer Eurolive B1520 Pros that were given to me with blown compression drivers.  I was able to repair them by replacing the diaphragms and fuses so I now have a second set of PA, which is very convenient in situations like this.  I had the Behringer speakers setup half way down the court on the sidelines to cover the back half of the gym.  This way I could keep the volume lower than if I had just one set of speakers covering the whole gym and keep the energy on the court as low as possible, to minimize reflections from the gym floor.  I don’t have a second DriveRack, but I do have a Behringer DCX 2496, and I was able to punch in all of the required adjustments I felt were necessary.  I think I came out with a pretty good sounding rig.  I spent about 45 minutes  aiming the four speakers and getting my gains set just so that the coverage across the front and back half of both sets of bleachers was as uniform as possible.  When I was done, the sound was better than I had ever heard in that gym.  A few people even commented on how much better our rig sounded than the gym’s installed PA.

In between setup and the start of the event, there were a couple of people shooting baskets on the far side of the gym, so Ophelie and I had to stay to make sure no rogue balls hit any of our equipment before the show.  We were finished after a couple of hours, and still had a few hours to kill before everything got started.  I forget what time people were allowed in to start getting seated, but we started playing our walk in music when they did, and wee ready as soon as the event was set to start.  There were no major issues other than the audience complaining that the delay speakers were blocking their line of sight.  I eventually caved and had Michael and Ophelie take them off the stands, but was mad that I had put all of that time getting the system to sound perfect and uniform for nothing.  It’s like moving in next to an airport and complaining that the planes make too much noise.  There were plenty of seats that weren’t blocked, but those apparently are less desirable than picking a spot with a bad sight line and then complaining.

Otherwise, the event went well, and we had no major issues.  This was a good cause and I am glad that Mystic Rhythms was a part of it.

Knight’s Challenge 2018

This is going to be the fifth year that Mystic Rhythms has DJ’d Kingwood Township School’s Knight’s Challenge event.  For those of you who are not familiar with the event, it is a series of team based relay events to raise money for the local volunteer rescue squad and fire department and the 8th grade trip to Washington DC.  It’s always an enjoyable night, and rather casual on the DJ end.  If we make a mistake, it’s really not a big deal, making it an excellent warm up gig.  Of course, it it optimal to not have any issues at all, but when there are large gaps in time between gigs, mistakes and other problems are inevitable.

On the day of the event, I picked Ophelie from Del Val after checking in with the status of the jukebox, which had just had the vinyl tubing applied to the lighting channels.  We arrived at Kingwood at 4pm, and started setup.  Michael arrived a bit later and helped to finish the setup.  Everything went smoothly until we realized that I had forgotten to bring speaker stands.  Fortunately, this was the biggest issue we had, and that was easily mitigated by my father bringing them on his way to the event.

Michael and I handed off control of the console throughout the night.  Feedback is a problem at this event, so we are constantly moving the mic fader up and down all night.  There’s only one mic for announcements, so it’s not too bad.  We played music throughout the entire night, and had no issues during the event, which makes it rather dull to read or write about, but that was great for us.  Ophelie picked up some experience with the equipment and workflow for the next day, and I knew my equipment was all in working condition, so all in all, it was a good night, and we were ready for the Saturday gig.


Continue Reading