Snapdeal Engagement

About Snapdeal

With millions of users and more than 300,000 sellers, Snapdeal is one of the biggest online Marketplace in India, delivering to 6000+ cities. While operating at such scale you need to have a world class CI and CD system in place to build, test and release faster.

We collaborated with Snapdeal for around 3 months to work on:

Streamlining the mobile client releases
Centralising Mobile Test and Release Infrastructure
Training QA team

Streamlining the mobile client release

Problem Statement

Though Snapdeal had a two week release cycle but there was plenty of room for improvement as a lot of release related tasks were manual such as

Uploading builds manually to App Store and Google Playstore. Distributing alpha and beta builds via emails and other primitive distribution medium.
Release Managers were spending a lot of time in manual chase ups for sign-offs and all of this was done verbally.
No usage of alpha and beta channel to get quick feedback before hitting the market

Solution Approach

The very first area we acted upon were the release steps that can be automated with less effort and greater impact. We introduced an automated Continuous Delivery pipeline for both Android and iOS that would automatically create the artefacts and upload them to distribution channels.

Additionally, we worked on building an automated Alert and Timeout based signoff system. The idea was to integrate the sign-offs from the various stakeholders with the Continuous Delivery pipeline and also keep a record of it. This helped consolidate the release actions and records in one place and gave the opportunity to Release Manager to do more important work.

Our next target was to bring up the adoption of alpha and beta builds internally and externally. With over 1000 employees Snapdeal had a considerable alpha base that could help with daily dogfooding of the pre-release candidates and early reads on critical metrics. Due to the distribution happening over emails, there was a lot of friction and steps involved for users to keep the app updated.

We leveraged the Google PlayStore and App Store alpha and beta channels to move the distribution from emails to seamless App Store auto upgrades. This was all being done in an automated way as we integrated these channels with the Continuous Delivery pipeline. At the end of each day we had a deployable artifact which could be rolled out to alpha and beta customers for getting early user feedback before hitting 100% production rollout.

Outcome

While the above efforts were small improvements but they had a tremendous impact in terms of

Removing the friction between various stakeholders and reduced need for someone to chase people with automated sign-offs. QA Managers & Release Managers were now able to do more meaningful work
With seamless alpha and beta distribution, more users were now upgrading to the build and Snapdeal team was able to release alpha app multiple times a day to all internal employees and have a production ready build artifacts which can be released to beta base for quick data collection from live users and get device coverage and used firebase analytics to get data for P0 metrics before hitting 100% production rollout.

Centralised Mobile test and Release Infrastructure

Problem Statement

Each team in Snapdeal had a separate GUI test framework. The team used to trigger it manually on need basis on their local jenkins box.

As the above distributed test framework scaled,it resulted into following challenges:

Silos: Since all the teams were building their own test framework and tests, it gave rise to a lot of test code duplicacy and also restricting sharing of test code / design across organisation. This was not ideal for a Test Org testing for scale.
Delayed Test Execution cycle: The tests built by the teams were being executed very late in the development cycle and were intended more towards catching breakages rather than preventing them from getting merged to master.

Device Infra Scalability: Because of the inefficiencies in the test infra due to de-centralised framework, GUI tests were taking a lot of time for execution and needed a large pool of devices to cope up with the required test execution time. Moreover, the devices coverage was limited to only 2 devices, inadequate for a comprehensive test coverage.

Outcome

Common test framework propagated an implicit sense of test code sharing among the tests teams which in turn induced reusability of code and decreased duplicacy. This played a key role in automating P0 functional test cases within sprint and hence reducing the manual test cycle to half.
With comprehensive Continuous Integration each code pushed to Gitlab was being sanitised and helped with preventing bugs landing onto the main line. This helped the test and release team to keep the master green and alpha/beta ready at all times.

● Brought transparency to the development process and helped improve review process since the reviewer had all the checks results available in PR.

Solution Approach

Single Appium framework: We worked towards laying the foundation for a common

Appium framework that was implemented across various applications in Snapdeal and the tests were integrated as a part of Continuous Integration pipeline. This helped with a. Easy test code maintenance

Consolidating the test processes and best practices to be consistent across board c. Tooling around Application test teams to bring in efficiencies
Centralised Device Farm: For building a reliable and scalable device infra, we set up a centralised device farm in house using Smartphone test farm. This not only helped us to match up with the Test Execution requirements but also maximise the device utilisation. Additionally, automation scope increased from top 2 devices to top 20 devices used by snapdeal user base.
Shift left : A good test system is always intended to prevent bugs rather than catching them. And prevention of bug happens at the place where it is introduced i.e during the development. We started looking at how we can provide quick and reliable feedback to developers on the changes they are pushing so that we can prevent bugs to land onto master base

To address above, we built a Continuous Integration pipeline for all Android and IOS mobile applications that helped them in detecting bugs early in development cycle by integrating Unit tests, Integration tests, Static Code analysis, End to End tests and implementing code review policies. This is explained further in details in the following diagram

Multiple training sessions with QA team

Our collaboration with Snapdeal was not only limited to building processes and systems but it encompassed training the test teams on how to become a high performing Quality Engineering functions. Few of the many topics covered are listed below

Isolating bugs: Well! Finding the root cause of a bug takes more time than actually fixing it and one of the keys is to identify the source of bugs server or client. This was done in order to increase developer productivity by putting enough information on bug summary and to isolate bugs. We worked with team on leveraging various tools like leak canary, logcat, Charles etc
Appium workshops: We conducted few hands-on Appium workshops to enable the test teams so that they are well equipped to take the common Appium framework to next level. This was also intended on training the existing manual team on contributing towards automation and by the end of our engagement, we were able to ramp up automation team from 2 to 7 test engineers.
Workshops on Automation mindset: We ran a couple of interesting and engaging sessions with the team on how product and process automation can play a role in streamlining releases and reducing lead time and failure rate. Few industry case studies and few from our own engagements were discussed that gave the team interesting insights on how automation can enable them to do more meaningful work and do more in less time.