Blogs

Buy devices or use services for testing on various devices

by Shilpa Bhavsar | Mar 26, 2023 | Knowledge | 1 Comments

This article mainly relates to testers who have to test their apps on multiple devices and face challenges as there are so many devices used in the market. 

Testers doing mobile application testing have to ensure that their app, performs consistently on all devices. We have to make sure the app is tested on the various ranges of devices possible.  

Challenge

To buy the devices OR rent a cloud subscription of the testing platform?

This is an age-old dilemma for the testers testing the web, desktop or mobile applications. This situation arises as testing needs to be performed on different devices (and screen sizes). 

Nowadays, development platforms provide very effective simulators to test functionality, but that is not enough as it is not the actual device but just the software that provides similar functionality. Some limitations that arise while testing on simulators or emulators are –

• They cannot simulate the battery or performance issues. 
• Receiving notifications is not possible 
• Testing camera-related functionality is not possible etc.  
• Dealing with actual geo-location functionality.
• Bluetooth and network connectivity-related test cases

Some companies can provide a separate budget for setting up the devices and allow testers to test on devices.  

But there are more side effects to it –  

• Buying devices would increase the company’s overhead as they will have to buy both iOS and Android devices. So if you consider only iOS devices, buying it, includes buying more than 50 iPhone devices and more than 25 iPad devices.
• Inventory of devices has to be managed.
• Challenge of tracking devices given out for testing and checking if it is returned. 
• Keeping a track of who holds the device for testing? 
• What if a new version of the device (hardware) comes to market? this is religiously true in the case of mobile devices. Certain manufacturers like apple even impose force updates of software and/or hardware after certain updates. 
• Even after all the inventory companies have, one may not be sure if testing has covered all or focused the devices.

Solution

In order to address this problem at YHWorks, we took a hybrid approach.

First – own the inventory of devices that are a must for testing (only curated ones) and use testing platform services to use for automation and other regression tests.

We have kept our own inventory of few devices and relied on the Testing Platform to do testing. This helped us at YHWorks in many ways. Some of the benefits are listed below. 

• No need to do huge investments on buying new devices 
• No longer have to keep track of big inventory 
• No security issues 
• Available big range (almost 3000+ real devices/hardware) for testing
• Subscription based so no upfront cost, and also subscription is relatively cheaper compared to spending on maintaining the inventory. 
• Always (Round the clock) availability of target OS and hardware configuration to the functionality. 
• Easy integration with existing development, testing, management and communication frameworks like Git, Jenkins, Selenium, Appium, Slack etc. 

Our experience

At YHWorks, as we used a hybrid approach to testing, we have broken down the test cases targeted for testing platforms and devices that we own. With this, we are able to cover a large area of testing across multiple devices and have improved quality reports. The client also gets detailed reports of testing results that lead to confidence of the quality of the product.

Conclusion

We vouch for a hybrid approach based on the experience we have. Completely depending on either approach is not recommended. Having a hybrid approach, we get the benefits of both approaches (having own inventory and relying on testing platforms) by this way – we can negate the limitation of both approaches. For example, sound and video functionality testing can be best suited with own inventory, whereas API, walk-through, and other browsing user experiences are best meant and covered most effectively on testing platforms. 

Now the question is – which platform to select? There are many such platforms available, you can pick by comparing the services they provide and the requirements you have. 

The author of this article Shilpa Bhavsar is having 12+ years of experience, is QA Head at YHWorks Tech Private Limited.