How GitLab’s Single Application for the Entire DevOps Lifecycle Simplified Optoro’s Mish-Mash of DevOps Tools
The recent coronavirus pandemic is responsible for a significant jump in online retail shopping as people were required to stay at home under shelter-in-place orders and non-essential businesses shuttered their storefronts. But this increase in online shopping has also resulted in more waste in the reverse supply chain (in normal circumstances, retail returns can be as much as 15-20% higher than in-person purchases, depending on the industry).
Optoro, a returns technology company, seeks to eliminate this retail waste. With a technology solution like Optoro, a retailer’s goods can be managed at the first touch and scanned into a sophisticated system that automates the process of getting a product to its next best home as quickly and efficiently as possible. Cutting down on transportation waste, labor costs, and reducing the total process time helps retailers gain back a significant amount of the wholesale cost of goods and helps to eliminate product waste.
But in order to provide its customers and their customers with a seamless return experience, Optoro needs a seamless operation workflow within as well. Although the company was founded in 2010, the major shift from brick-and-mortar to online retail in the early aughts required the company to focus its efforts on delivering a more efficient process for businesses managing their customers’ returns. Accomplishing this target required more emphasis on DevOps within the company.
Initially, the team began checking-in code within GitHub, as many other DevOps teams were wont to do during this time. And while it helped Optoro to maintain some organization for its DevOps workflow, a team member suggested that the code be tested as well.
To host the test, Optoro relied on an AWS server. Then, in true, scrappy start-up form, tests were run through Jenkins during a screen-share session on an employee’s laptop.
“It was very handcrafted,” said Zach Dunn, senior director of platform operations at Optoro. “When we got serious about our set-up in AWS, we had to migrate it. That was a nightmare because, of course, it actually hadn’t been version-controlled or anything like that.”
After a few iterations of rebuilding “that same sort of Frankenstein over and over again,” said Dunn, his team spun up a Jenkins server managed by Chef. But even then, the solution wasn’t ideal.
“If anyone’s ever managed a Jenkins server, it’s not designed to be managed,” said Dunn “It’s a UI-based infrastructure.” To mitigate this, the team again took an “artisanal” approach of tools specifically joined together for Optoro’s DevOps workflow needs, but the team continuously found they were correcting breaks in the workflow rather than creating a solution to help improve its customer experience.
To add insult to injury, Optoro found itself maxing its repository limits in GitHub.
When it became a cost burden to structure Optoro’s code as individual repos, Dunn decided to run GitLab on his own.
“We started using the Docker executors,” Dunn said. “So now at this point, we had GitHub, a CI Jenkins, an infrastructure Jenkins, we had executors for both of those, static executors… I think at the end we had something like 24, maybe 32 gig VM sitting there executing jobs for Jenkins. It was just super complicated.”
Moving to the open source version of GitLab helped somewhat, but Dunn knew his team was still having to work harder than necessary. They took to blaming CI.
“It was never that our tests were wrong or we’re testing the wrong thing or we’re not maybe thinking through all the business cases right. It’s ‘CI sucks’,” said Dunn.
Although Dunn would love to say he had an “aha” moment around GitLab’s CI, the reality is the company needed to start taking things like compliance seriously.
“We looked around and we realized, ‘We need to grow up,’” he said. That, and a nudge from a potential auditor, turned their attention to GitLab Silver, a SaaS version of GitLab. It worked out better than Dunn expected.
“We ended up liking GitLab CI enough that we wrote a tool called GitGlue, which would watch GitHub repos in our organization and then run CI on our internal CI system, which was later a feature that GitLab added,” he said. “The interesting thing here was we actually liked what [GitLab’s] CI was doing for us… And so we were very much committed already to GitLab’s CI. So no matter what we were doing, we were always kind of like, ‘Well, I don’t want to rewrite CI because we think this pattern works.’”
The only hiccup involved moving from Docker to Kubernetes during the migration to GitLab, Dunn said. His advice: remember to avoid “Dockerisms” because Kubernetes “doesn’t know what that is.”
Within three months, Optoro managed to leverage GitLab mirroring, consolidated off of its CI into GitLab, and archived its original repo and removed mirroring. They also managed to complete 4,500 pipelines.
Today, Optoro is hosted in GitLab with Kubernetes runners. Dunn’s team now has access to a templated set of consistent best practices for CI with caching enabled (no more recompiling of Nokogiri!), Kaniko Docker builds, and application security tools, all within one platform.
“I got the joy of turning off about a terabyte of crap,” said Dunn. “It was beautiful.”
Want to learn practical DevOps strategies from developers, ops pros, engineers, managers, and leaders leading the charge? Join us for our first-ever virtual Commit event on August 26. Learn more and get your tickets here.