You’ve seen the release notes for our last few builds and it’s obvious we were focusing on bug fixes and feature completion. We’re waiting on two big features to go through our testing process; Web hooks and Option 66 Provisioning. In the meantime, we’re focusing on optimizing all of Powercode for the next couple weeks. With a large, organized push, we should be able to tackle some known performance issues throughout the software and increase responsiveness and scalability. There are some low hanging fruit for us to focus on that should make a big dent in any performance issues.
Remove duplicate processing of database items
Yes… on just about every page load of Powercode, the same information is being pulled from and processed by the database server multiple times. Systematically going through and removing these will help quite a bit, as each query locks the tables it touches. Are we embarrassed? A little, but the developers that thought it was a good idea are no longer here and we’re going to fix it.
Converting to a new database driver for PHP
It’s no real secret what Powercode’s technology stack looks like. The MySQL driver for PHP that Powercode uses is a little outdated. Again, we are making an organized push to replace it all.
Most information about the customer isn’t going to change. We shouldn’t need to process all that info every time you load a customer overview. We’re going to intelligently cache different items throughout Powercode and purge them when information is updated. Unfortunately, this piece isn’t entirely a part of our 2 week push, but is in progress.
Explore the replacement of MySQL with Percona
What is interesting about this is that our exploration alone is forcing us to make necessary optimizations as they are required by Percona. Percona is fantastic for increasing throughput over MySQL given it is configured properly, and even if we don’t make the switch, Percona is a little less forgiving which is helping us optimize.
We have gone through and documented over 90% of pages in Powercode and their load times, query times, and other request profile attributes for more long-term optimization targeting.
Beyond the MySQL database optimizations, we’re auditing the server as a whole and changing some things on the install/update process. Before I started working here, while doing some research about Powercode I came across a post by one of our customers comically questioning the addition of Ruby to PHP and tcl. Being here for several months now, I’m questioning the same.
As amazing as Powercode is for our customer base, there are a ton of obvious things that need just a little love. I’ll keep you guys updated on what we’re working on.
This particular update may come more from me personally, rather than from Powercode as a whole, as it relates to one of the first things I was thrust into when I started at Powercode in November 2016. In case you don’t know, and for those that may have doubt, there are many major things on the board that Powercode needs to implement, re-implement, fix, or enhance. However, there are two things that truly overshadow just about everything else on that list; BMU and IPv6. So where are we on those?
The BMU is one of the strongest features Powercode offers. Having a device like this cohesively integrated with your billing system offers automation on delinquency, repayment, and traffic enforcement. In fact, when the BMU was originally designed, it was genuinely novel in this market.
Over time as new features were added and new developers were brought on for different tasks, the Powercode firmware that ships with the BMU has gone through some rich history. Some of this development over the past several years has led to a small percentage of our users having issues.
Having a network device like that positioned where it is in your topology requires rock solid reliability, and the BMU fell short of that for some users.
It became clear after a couple weeks of developing on the BMU that it would probably require the same amount of time to hunt down and fix the issues as it would take to simply start from scratch.
Starting from scratch allows us to utilize new technologies, firmware and drivers, and implement a true development methodology that I feel this product has been missing for years.
Over the past 4 months we have worked with a number of people who specialize in this field to help us develop a new, optimized firmware for this device with the core of Powercode functionality in mind.
We are getting very close to making this release publicly available. You might have noticed a few weeks ago, your Powercode instances reporting the BMU firmware as being “out of date.” That means we’re getting ready to release this soon.
So, how does that tie in with IPv6?
Unfortunately, IPv6 management in Powercode requires the BMU to be able to handle the sync process with IPv6 components. When I inherited the project there was progress with IPv6, but unfortunately, it needed to be redone with our larger vision in mind. Once the new BMU firmware is made generally available, we’ll be able to finally complete development on IPv6.
However, the difficulties with IPv6 don’t end with the software development. There are real deployment decisions that need to be taken into account, and we have to be able to support a variety of deployment paths for our customers.
The BMU is close and IPv6 isn’t far behind. Once the new BMU firmware is released, I will update you with our IPv6 implementation milestones and release a hard timeline.
One thing lacking on Powercode’s side is updates on feature development. We are working hard to ramp up this blog as a space for us to show you our progress on future features. There will no longer be silence on these matters. We will develop these larger items in the public eye and shape things using your feedback.
Beginning April 25th, 2017, our phone support hours on Tuesdays will open at 10AM instead of 9AM. All other days in the week will remain the same.
We are taking this hour for internal meetings between the development and support teams. There is a unique benefit from having our entire staff together that we will take advantage of.
Our meetings will be focused on transferring knowledge between the departments to re-prioritize tasks, update demands, and digest new data to shape our development of Powercode.
We apologize for any inconvenience this may cause. If you have an emergency during this time, please call in and leave a voice message. On call charges will be waived during this time.
As you’ve probably noticed by now, we’ve posted a couple of release notes over the past 2 weeks. You’re probably wondering, “How do I update?” The reality is, we’ve tweaked our update platform to amplify testing and gain additional real world use prior to making new builds available to all ISPs using Powercode. Let’s dive into some of the upcoming changes.
There are 2 main elements to this, but let’s first back up and briefly explain what our current internal testing looks like.
Internal Lifecycle of New/Changed Code
When we develop Powercode and move our code base forward, the author first tests each and every change they’ve made. Next, that change is submitted to the entire development team and is reviewed before it is accepted as “part of Powercode.”
As different changes from all our developers get submitted, reviewed, and accepted into Powercode, it accumulates in a new build. We decide when to cut off new changes from entering the “next build” by evaluating how much “risk” is created by all the changes in the build.
Now, when we use the word “risk” here, I don’t want anyone to think Powercode gambles with these releases (not since I’ve taken over at least).
What we’re talking about is simply:
- What is the scope of the changes?
- If something were to break, where could it potentially happen?
- If something were to break, what is the worst case impact of that break?
- How will the amount of changes impact our support burden?
Though there a few more factors we think about, the above about covers it.
So now we have our build and we decide this is our release. The next step is to make sure our support and sales departments understand the changes and then have them use their real world experience with our customers to try to break things, offer new perspectives and generally do some more targeted testing.
Support has a fully built-out lab for them to test against, with various hardware pieces, topologies, etc.
After another round of tweaks and changes based on the support team’s perspective, we have a build ready.
New Update Tracks
Now, this is where the bigger public facing changes come in. We recognize that Powercode is used in very different ways across many ISPs. There are tons of options that create many forks in the road. Quite truthfully, some scenarios and situations are not only impossible for us to test, but some are inconceivable to us. There are number of creative option/setting combinations out there that our customers employ to achieve some… let’s just say… “secret features”.
Stable vs Beta
With that said, it’s very important to us to get some diverse, real-world use cases tested before we unleash these builds out into the world. We’re going to accomplish this with two main tracks. One is a customer facing track (you). The other is an internal track for us.
In the system settings of Powercode of the latest release and onwards, you’ll notice a new setting that allows you to upgrade from either the “Stable” track or the “Beta” track. It’s a simple dropdown you use to choose and save.
You’ll always be able to jump back and forth without any consequences. The only limitation is that if you choose “Beta,” and have a version that is ahead of the “Stable” track, you cannot switch back to “Stable” and expect to be downgraded. You’ll need to wait until the “Stable” track gets ahead of you before you see any more updates.
In combination with choosing which update track to follow, we’ve also implemented update groups, an internal setting accessible to us at Powercode. As we track different support issues, use cases, and other variations between our ISP customers, we build groups to get a sampling of different environments we want Powercode to be released against. For example, we have a number of ISPs that run a test instance of Powercode for themselves (This is free by the way. Just call and ask.) to use for training staff, API testing, and more. Internally we will put all of those test servers into “Group 1.” So if you operate a test server, and set it to track the Beta releases, you’ll get all of our releases immediately.
We’ll take any and all feedback from our “Beta Group 1” servers and fix, tweak, or add anything that is necessary then release to a wider audience, “Beta Group 2.”
Only once we’ve gone through this amount of rigorous, real-world testing, will we push the build out to “Stable Group 1” and subsequently, “Stable Group 2.”
Why Multiple Groups For Stable?
You might think, “Well, if you’re calling it ‘Stable,’ why split it into groups?” Splitting it into groups during the “Stable” release isn’t about testing, but rather about making sure the service and support we provide is efficient. Not only has Powercode grown significantly recently, we have some fairly ambitious features in the pipeline. We will be implementing another phase of these update changes which will require these stable groups to be differentiated. The biggest piece of that next phase is attended upgrades, where you will have a Powercode representative on standby during all of your upgrades. This is not “just in case something goes wrong,” but because we want to make sure you implement new features to their maximum return.
Some of this is still up in the air as we iron out the internal processes. However, our goal is to make the pain of updating as low as possible so we can enable everyone to take advantage of the new features, fixes, and enhancements Powercode is about to unleash.