CentOS Stream

A couple of days ago, I heard about Redhat changing the future direction of CentOS. While there have been a range of sentiments floating around online; For me, the only thing surprising about this is that it took this long. So let’s take a moment to look at where things came from, where it’s going, what’s next, and how I think we should handle that.

• Where did it come from?
• What is the change?
• Should you migrate to something other than CentOS Stream?
• How much work is involved in migrating to the next solution?
• CentOS Stream
• RHEL
• Fedora
• Other RPM based distributions
• Non-RPM based distributions
• What is the time frame?
• Why would RedHat/CentOS make this change?
• What will happen next?
• What’s next for those of us using it?
• Use-cases vs Licensing
• Testing/VMs/dev
• Charities
• Small businesses starting out
• Other distros
• Ubuntu server
• SuSE
• Debian
• Other, other distributions
• Doing it yourself
• Are you upset?
• You directly contributed (ie financial or expertise) to CentOS.
• You used it, but did not contribute.
• Thank you

In my opinion, CentOS began life as a middle finger to RedHat Enterprise Linux (RHEL). The open source community was still largely trying to figure out how to make a living by giving away it’s work, and still having enough resources to work on it, without getting burnt out by burning the candle at both ends. RedHat had recently moved from having just one, free distribution (RedHat), to having multiple options, that eventually boiled down to Fedora (free) and RHEL (subscription). The controversy came with the RHEL subscription. You weren’t supposed to be charging for Free and Open Source Software (FOSS), so what was the money for? These days, that is better understood, but it wasn’t at the time.

A consequence of many of the open source licenses in the wild, is that the source code and any modifications must be accessible to users so they can modify the code as they see fit. For a long time, RedHat’s mechanism for achieving this has been to provide SRPM packages, which are a source version of their matching RPM packages. This is an excellent mechanism to be able to reproduce the same package that RedHat has provided, and then make anything from minor tweaks, onwards.

This was an excellent opening for CentOS to be born. RedHat would release a new version of RHEL behind a paywall, and provide paid services around it. CentOS would then release the same thing, but with the logos, and branding replaced. While a significant portion of this work would be automated, and that automation would be continuously improved on over time to make things quicker and easier, it should not be understated just how much work this is. Maintaining a distribution takes a lot of effort, from a lot of people.

So this brings us back to

In my opinion, CentOS began life as a middle finger to RedHat Enterprise Linux (RHEL).

This may not have been the original author’s intention when starting CentOS and the projects that came before it. But based on the sentiments floating around the community at the time; I think this is how people treated it, and I think this played a huge part in it gaining as much traction and success as it did.

For a long time, I was always surprised when I’d see, or hear of a company relying on CentOS. It seemed like a risky move to rely on something that was so blatently under-cutting one of the big-dogs that was successfully making money. But RedHat behaved well; There are many reasons why this was a good idea, but this is old ground, so let’s move on. As time went on the future of CentOS seemed to get more and more stable.

Also around this time, the SCO patent trolling was in full swing. It’s mostly in the past now, but at the time, this really looked like a serious threat to Linux. Novell/SuSE and RedHat retaliated with their own versions of indemnification/liability insurance to protect at least their own customers (although I vaguely remember them stepping up to the plate to protect non-customers as well). This move gave people and companies confidence to continue using Linux, and undoubtedly made a massive possitive impact on where we are now. We should always remain grateful for this. It wouldn’t have been an easy business decision to stand up to that, and they didn’t have to do it. But they did it anyway.

In the beginning of 2014, RedHat brought CentOS under their umberella. I’m intentionally being vague in my wording here, because I don’t fully understand how this aggreement worked. Wikipedia uses the word “sponsoring”, but from what I’m reading about the interactions around ownership of branding etc, “sponsoring” seems to me like an understatement. In any case, this arrangement likely provided essential funding to keep the project going. Nearly 7 years have passed since then.

As announced on their blog

The future of the CentOS Project is CentOS Stream, and over the next year we’ll be shifting focus from CentOS Linux, the rebuild of Red Hat Enterprise Linux (RHEL), to CentOS Stream, which tracks just ahead of a current RHEL release.

Normally I down-play announcements like this. But this one is actually much bigger than it sounds:

• The target audience of CentOS has always been where tried-and-true, enterprise-grade, production-ready builds are needed, but they don’t want to pay for support or liability insurance because they want to do it themselves, or don’t feel they need it. Typically these are small businesses, charities, and hobby/self improvement setups.
• The target audience of CentOS stream is a little less clear. But it looks like it is for developers/companies that want to make sure that their latest widget works on up-coming versions of RHEL.

When CentOS Linux 8 (the rebuild of RHEL8) ends, your best option will be to migrate to CentOS Stream 8, which is a small delta from CentOS Linux 8, and has regular updates like traditional CentOS Linux releases.

In the short term, this is great, because it provides people/entities an avenue to keep security updates fresh until they are ready to migrate to their next production system. It should not, however, be interpreted as a continuation of CentOS. In the short term, the delta will likely (but not definitely) be small. But over time this will grow, and assumptions around QA and production-readiness will become less valid.

A key assumption that will be implied throughout this article is that, since the target audiences are so different between CentOS and CentOS Stream, this move is effectively killing off CentOS because the new direction will not be the right tool for the job for existing users.

The nature of the target audience for CentOS is such that if CentOS was the right choice in tool for your tasks, then CentOS Stream will almost definitely not be the right tool for those tasks. However, CentOS Stream will likely buy you some time in the form of security updates, so you can migrate to your chosen solution at a more controlled pace. Having said that, the effective latency from when a security patch is made available upstream, to when it becomes available in the distro could be almost immediate, or that updates just show up when they want to test the next bit of functionality. I suspect the reality will be a mixture of the two, but not in between.

From their press-release

and has regular updates like traditional CentOS Linux releases.

I think this will be true during the peace-making transition period. I don’t expect it to be true long term. But like everything else in this article, I have no inside knowledge of their plans, and I could be completely off-base.

It sounds like this will be a trivial, in-place, upgrade:

your best option will be to migrate to CentOS Stream 8, which is a small delta from CentOS Linux 8

This could change between now and then, so should be assessed and tested before being relied on.

It might be possible to migrate to this in-place. However, I’d strongly recommend doing a re-install whether you need to or not.

Your existing automation will likely have few, if any, breakages.

A lot of your automation will probably work fine, and package names will likely be fine. The sorts of breakages you’ll likely see the most of are where technology choices, and default configurations are different. You may also need to mess with kernel parameters to get the performance you need in the right places for your tasks (eg desktop systems are usually prioritised for responsiveness over throughput). You should expect to invest a fair bit of time in the “I’m 99% finished” stage.

PS See commentary about distro choices further down.

Assumptions around package names, technology choices, default configurations, and package mangement will break. You should expect to invest a fair bit of time in the “I’m 80% finished” stage.

Most of your automation will need to be re-written. Budget enough time to write it from scratch.

As of this writing, CentOS 7 is going to see through its full lifecycle, which will mean security updates until 2024-06-30. This gives us lots of time to work out what we will be doing about existing deployments, and get on with it. I am grateful for this.

CentOS 8 will end at 2021-12-31. This is considerably tighter, and surprised me. I was expecting to read that it would end at the same time as CentOS 7, or perhaps a little later. I have some suspicions as to why this is, but the level of speculation is getting pretty high at this point, so I’ll stop here. In any case, how much of a hassle this is will depend on the use-case of a given deployment. At a company, it will be anywhere from just a nuisance, to a major stress point. At a charity, it could mean hiring someone to come in and migrate stuff over to a new platform. Even if they manage to score free RHEL licenses, getting someone in to do the migration would eat a significant portion of their coming budget. Depending how the big the charity is, and how well it is doing, this could mean the end of the charity.

Like the rest of this article, I have no inside knowledge. But there are several likely reasons that they have have had for making this decision:

• CentOS is dividing and diluting RedHat’s brand.
• RedHat is paying/working twice for the same output.
• RedHat is doubling their testing surface area.
• RedHat is doubling their vulnerability surface area. (Which is not necessarily identical.)
• RedHat can achieve the same thing with licensing.
• Changing the focus in this way, gives them an avenue to solve all of the above points by killing off the project in a two step process:
  1. “It’s not dead; it’s just got a new focus.”
  2. “No one is using it now, so it’s silly to keep pouring money in to it.”

Whether IBM buying RedHat has changed the attitude to any of these points, I don’t know. I’ve seen confident speculation on this point, but I think it really could go either way.

Conversely, there are reasons not to do it, which have likely contributed to why CentOS has lasted so long:

• It will piss off a lot of
  • talent that has/is actively contributing to the ecosystem.
  • advocates/persuaders.
  • users that might have considered subscribing to RHEL.
• It will push away mindshare from the RedHat ecosystem to other distributions, or possibly competing OSes.

With the recent announcement, my original prediction was that Fedora and CentOS Stream would be competing, and therefore one of them would have to become obsolete at some point. CentOS Stream would seem like the natural one to deprecate, since I don’t expect a big uptake in new user-base, while Fedora is well established as a desktop distribution. But after thinking about it some more, I think there’s a bit more to it:

Fedora is primarily aimed at the desktop. CentOS Stream is likely to remain aimed at the server. This might be enough distinction to justify keeping it around. More likely, I think they’ll both stick around for a while. And then when the time is right, they’ll declare that not enough people are using it to justify the effort/resources in maintaining it, and they’ll ditch it entirely.

If however, they do keep it around, I predict that it will be renamed to something like “RedHat Stream” to more accurately reflect what it has become and how it fits into the RedHat port folio.

TL;DR RedHat is in a position to be able to solve almost every problem with licenses. Chat with your RedHat sales representative to find out what is possible.

If your company uses RHEL in production, and CentOS in non-production, I think this comment will be relevant to you:

Get yourself a Red Hat Developer subscription which is free for non-production work.

You will get a self-supported full RHEL version! For all your virtual machines!

So no issues with compatibility at all.

Go to https://developers.redhat.com/ to register.

If you don’t have success here, chat with your RedHat sales representative who will likely be keen to find a solution.

Chat with a RedHat sales representative. I can’t speak for them, but I suspect that they would love to bump up the statistics on how many charities they are supporting, and how much they are giving back. If you are willing to do a case study with them, I’m sure they would love to add you to their success stories, which would give you even more (most of the) leverage.

Chat with a RedHat sales representative. You are “greenfield” something something (I’m sure I should be using the word “synergy” in here…), and an excellent opportunity for future growth.

Ubuntu is probably the one that most people think of, and talk about, when discussing what to use instead of CentOS. I’ve used it a lot myself, and highly recommend it.

SuSE is often overlooked at the moment, and I’m not sure why. I think there was fear that when Novell bought SuSE, they’d do to SuSE what RedHat appears to be doing to CentOS. To the best of my knowledge, after many years, this hasn’t eventuated.

Like CentOS, SuSE uses RPM for package management. So you might feel slightly more at home than with Ubuntu.

It has been a long time since I last used SuSE deeply, but every time I use it, I’m filled with nostalgia of a beautilfully thought out distribution that provided lots of choice. Hmmm, I’m due for a re-install on my laptop…

Debian is a foundation from which many distributions have been based. I’ve only used it a little, but my experiences were positive.

There are many other distributions that would be worth considering depending on your requirements. But I need to stop here, otherwise this is going to turn into a distribution comparison, which is not what this article is for. Hmmm, but I need a reason to mention how awesome Gentoo is…

As far as I can tell; all the previous mechanisms that existed that enabled CentOS to happen the first time, still exist, and will do so for the foreseeable future. If CentOS is still the right choice for you, it may be worth re-inventing it. The build systems that took the RHEL sources, and replaced the branding will likely be open source, and the configuration for making it happen is probably accessible also. For now.

Keep in mind that this will be quite an undertaking. - There’s a reason they needed external funding. But I think there will be enough people/entities that have been shaken by change that you might be able to find the support that you need now.

I have seen a lot of passion floating around regarding this decision. There’s even a petition asking that CentOS continue as they have been all of these years. I can’t imagine that this is going to change anything. Ie The people that made the decision:

• will already be well aware of the outrage that will follow.
• will still have the same business problems to solve.
• will already have reasonable visibility into how many people/entities are using the distribution.
• will already expect that the petition will get a lot of signatures.

I just can’t imagine that they will learn anything from the petition, even if they are open to it. But I could be wrong. In any case, it will be interesting to watch that space and see how it evolves.

So let’s leave that to simmer for now, and move along. Instead, I’d like to focus on two possibilties:

• You directly contributed (ie financial or expertise) to CentOS.
• You used it, but did not contribute.

Thank you very much for your effort. I’m sorry that it probably hasn’t gone in the way that you’d like.

You (we) are part of the problem. We are why it is in this situation now. FOSS (Free and Open Source Software) thives from participation. It’s fantastic that there is so much high quality FOSS software available for anyone to download and use. But the time, resources and expertise required to make it happen has to come from somewhere. Some of it is companies wanting to give back. A lot of it is people like us scratching an itch, and making the solution available to the world.

If a FOSS project you care about is not going in the direction you want, that is a sign that you are overdue for contributing to that project. The fact that there are so many projects that we care about, means that we have to be careful in budgeting where our time and effort goes to. We will not always get it right. We will tend to be drawn to the projects that are trendy and shiney. But we also need to give love to the projects that form the foundations of what we do, and to the people/companies that make those projects happen as well.

CentOS didn’t surprise me. Docker surprised me, because it is shiney, trendy, highly adopted, and utterly essential to what we do every day. But CentOS, did not surprise me.

Every time a project you love bites the dust. That is a sign that we are doing something wrong. Petitioning the evil corporation to keep putting money into a duplicate product, or getting aggressive about decisions that have been made, are not going to make the situation better. If anything, they are going reduce the chances that those people want to help you.

We need to support the projects that we love and rely on. There are lots of ways you can do this. Here are a few in roughly the order I think they are most effective (most effective at the top), although different projects will have different priorities depending on what they are short on:

1. If you work at a company that uses open source software, advocate for contributing back to some of the projects you use.
2. If the project offers advice for how you can best contribute, read it.
3. Pull requests with solutions to known problems/tickets.
4. Pull requests with solutions to problems you have.
5. Pull requests with new features.
6. Financial:
   • Direct contributions.
   • If they have a subscription service, use that… Déjà vu. If you don’t have time or relevant expertise to give to the project, this is often one of the best ways you can contribute.
7. Help other users using the software.
8. Document gaps in knowledge.
9. Ask questions in the forums.
   1. Post back with
      • feedback with what worked.
      • any understanding you gained in the process that might be useful for someone else.
      • links to documentation that you found helpful.
   2. Be courteous, and assume that it’s you doing something silly rather than the developer doing something malicious.
      • Signs that you need to step back from the keyboard, and take some time to process things before continuing:
        • You need to use curse words (whether censored or not).
        • You write in an aggressive tone.
        • You feel pissed off. (It’s probably coming through in your writing even if it’s not intentional.)
        • You need to assign/deflect blame.
   3. Follow-up with a bug report if it’s relevant.
      • Be strictly courteous. See above.
   4. Contribute amendments to the documentation.
   5. If it’s a rare use-case, blog about it and the solution. Make sure it’s timestamped, because this information will not age well.
10. Submit bug reports after applying updates, then searching and discovering that there is no bug report for the bug you are experiencing.
    • Be strictly courteous. See above.
11. Be grateful. Remind the people running the project of what an awesome job they are doing and that you appreciate them.

If we don’t support the projects we love or rely on, they will either have to end, or make deals that make come with strings that will influence their direction, or cause them to end in the future.

I’d like to give a big thank you to everyone who has contributed to CentOS over the years. Your work has made the world a better place.

I’d also like to thank RedHat for

• stepping up when SCO was on the patent trolling rampage.
• keeping CentOS going for as long as you did.
• and for all the contributions you continue to make into the FOSS ecosystem.

It’s easy to loose sight of these things during a transition like this, but we shouldn’t.