Last week, we hosted another episode of our Continuous Discussion (#c9d9) video podcast, featuring expert panelists discussing the benefits of using open source tools and how DevOps can mitigate risks in quality and security when incorporating open source into your application code, environments and tool chain.
Our expert panel included: Chris Stump, full stack Chicago Ruby on Rails developer who’s big on Docker, Linux, and DevOps and currently working at Airspace Technologies;Eduardo Piairo, database administrator at Celfinet; Moritz Lenz, software engineer, architect, and contributor to the Perl 6 language working at noris network, where he set up their Continuous Delivery pipeline; and, our very own Anders Wallgren and Sam Fell.
During the episode, panelists told viewers where they use open source in their code and processes, and discussed the quality, security and legal implications associated with using open source tools, and how DevOps can help.
Open Source – Free as in Beer or Free as in Puppy?
Stump says open source can be both free as in beer and as in puppy: “I think it’s both. If you are new to open source it’s definitely going to feel like it’s new as in puppy because you are in an unfamiliar territory and for any one task that you want to do it’s going to feel like there is a million different projects. Once you have your tooling, you know what everyone is using and what is well supported, then it becomes pretty easy and it becomes more like free as in beer.”
Piario explains that open source isn’t completely free: “In open source you always have a cost, it depends on the size of your team, the complexity of your task and the frequency of change, and every change has a cost. The good thing about open source is you can contribute to the change and take it in your direction.”
Open source allows for a free flow of ideas, explains Lenz: “Some big companies like Google and Facebook open source their own stuff, and they get additional ideas for what to do with their tools and how to improve them. They also get patterns. But I think the ideas are the main thing, so open source also allows us a free flow of ideas which you can then use in commercial products.”
Fell expands more on the flow of ideas in open source: “The idea of competing with the potential innovation from a cloud of people is a very difficult thing to do. You will find lots of outside/in ideas and lots of enthusiasm for those ideas.”
Culture is a big part of successful open source, says Wallgren: “As with companies, there are communities that have good open source culture and communities that have bad open source culture – because it’s people. There are open source communities that are open, that are welcoming, that are aware of their own shortcomings and strengths. Then there’s other open source communities that are like ‘Yeah, sorry we don’t really want your contribution,’ and it’s difficult to get things going.”
Where Do You Use Open Source?
Some companies work with dozens of different open source tools, explains Fell: “When we talk to customers or prospects, most of them have about 60 tools in their pipeline – 60 different combinations of things just to move something out of Source Code Repository Land and into Production Land, to help with the various configurations or monitoring that needs to be done.”
There isn’t anywhere Stump can’t use open source in his pipeline: “As a Ruby on Rails developer which is an open source stack, I pretty much use open source through and through from the front-end using Angular or React JavaScript frameworks all the way down to the back-end, to a Postgres database with all the Ruby Gems that lie in between them and make our projects run. For servers we definitely use some Debian variants, usually on Ubuntu server containerization stick with Docker.”
Wallgren advises to question what it takes to work in an open source tool: “The thing you have to be concerned about is, what is the cost of ownership for this thing? Is it something that has to grow with me; is it something where if it’s broken it’s a really big problem; or, do I have alternatives? There’s things you have to worry about, but for the most part I use open source just about anywhere. It’s just another tool.”
Lenz uses open source tools for essentially every part of the pipeline: “There are areas where we use it because it’s just the best fit, but wherever open source excels we use it and that’s basically 90% of everything that we do. We use Puppet, we use Ansible, we use all the different test frameworks for Perl and Python and for automating the browser, Selenium, all this good stuff, both inside the product as libraries, and as backing services, for authentication, and then in the pipeline, in the build tool chain, test, deployment, everything… statistics, monitoring, you name it.”
Piario gives his advice on picking the right open source tools: “As a startup we started with a lot of open source tools, and with the evolution and complexity we started to migrate to commercial tools. We say ‘Try it before you buy it.’ We started to try different combinations, Jenkins, TFS, TFS build, TFS Release – for testing we use tSQLt a framework for testing databases.”
Quality Concerns?
There are three main areas to look for when assessing quality in open source tools, per Stump: “You have to know how to find the quality in open source tooling and usually it boils down to: how active is the community, how widely used is that software, and is there a strong leadership team behind that particular project (and do they have good quality control practices). Once you identify that a tool that has all those attributes for the thing you are trying to accomplish, I think quality is just on par if not better than a lot of proprietary solutions.”
Open source quality requires individual responsibility, according to Fell: “When you are using open source components as part of your product what exposure do you have from a quality perspective? Not that they are any more or less quality than what you would have if you did it yourself, but it doesn’t abdicate you from taking responsibility for it when it’s there. If there is a quality problem, if it’s open source you can go in and try to fix it, but if they don’t accept your changes then you are stuck.”
Take extra quality precautions if open source is baked into your product, says Piario: “If the tools support your pipeline, you can better manage the exposure to errors, but if the tool is included in your product then you have to assure that quality is there. Your client will talk to you if some problem happens, not the maker of the tool.”
Even though you have the freedom to change the source code in open source, it’s not an easy task to do, says Wallgren: “Even if you have the source code you still may be kind of screwed because you may not be able to build it, you may not understand it, you may not be able to document it – you now have to go solve a problem that would be nice to have somebody else solve for you.”
Security Concerns?
Having the right toolchain is important in ensuring open source security, says Lenz: “Last year there was a study by HP Security that half of the breaches they investigated were vulnerabilities that were known for two years or longer. Whether the batch comes out this week or maybe in two weeks is not as relevant, often it is a question of, do I have the toolchain to notice that I have to build a new product and ideally automatically upgrade, build, integrate, test and then release the product.”
Having code easily visible to any and all means flaws can actually be fixed more quickly in open source, says Stump: “I would say that with open source it’s like a double edged sword, because the code is open so there are more eyeballs on the code to find security flaws, but there are also more eye balls on the code that can exploit those flaws. But most of the time I think it leads to them getting discovered and patched quickly.”
Transparency is key to addressing security concerns in open source, saysFell: “Transparency is very important. Apple had a security case where the FBI tried to hack into the phone. Up until now Apple had never released the source code for the Kernel of the iPhone, and just this last time around when they did their SDK for the developers for iOS 10 they apparently released the source code un-obfuscated so that people could really start to dig in. You ask yourself, will people find exploits? Probably. Will there be transparency around those exploits? Probably.”
More testing needs to be done to ensure security of open source code, advises Wallgren: “Just because somebody finds a problem doesn’t mean it’s automatically going to be updated and patched in all the applications that use the open source platform because updating is a big deal. The lack of unit testing in open source code is pretty deplorable. It’s stunning how easy some of these things get by and how long bugs can sit there and the mean time to discovery for bugs is pretty long. Every assumes someone else is reading the code, finding the bugs and fixing it, and is that really true?”
Legal Concerns?
Be prepared to have to share what open source is in your product or code, says Fell: “When you are trying to acquire a company or a solution, a lot of the times they’ll say ‘Tell me what open source components that isn’t your code is within your code. It’s not necessarily a black mark against you, but it’s something you need to be aware of.”
Even Electric Cloud clients ask to know what open source or third part tools we use in our products, explains Wallgren: “This is something a decent portion of our customer community cares about is what third-party tools, not just open source tools, are we using, what are the relevant licenses, are we allowed to ship it. All of those things are concerns of anybody who even buys our software.”
Stump recommends spending more time learning the different open source licenses to help you pick the right tool for your needs: “Open source, I like to think, maybe demands a little more respect rather than just clicking through on the 48 pages of the user license agreement that we are all used to seeing, simply because it’s people’s free time and people aren’t getting paid for it generally and there are a lot of different open source licenses out there. If you work with and deal with open source you need to be aware of the difference between the MIT license, BSD license, GPL, that can affect your tooling choices.”
Lenz reminds us that there are legal implications for all types of software: “Proprietary software can also come at a high legal risk, for example there is software that is licensed by the CPU, by the core count, so when you are in a virtualization environment, can you safely use that software? If yes, for which cores do you pay? Do you pay for the cores that are assigned for the virtual machine? To the whole cluster? There are actually very valid legal reasons not to use some types of proprietary source software.”
