First of all, you need to build a set of the most critical/important scenarios. So, create a smoke tests with the critical basic features and divide them into phases. Also, remember to add the most frequent scenarios, those that are used in a daily basis.
Implement your continuous integration since the beginning so you can follow when a scenario has broken and if you have false positives. The reason why is you need to trust on your automation, for this reason in the beginning you will need to pair the manual tests with the automation until you have confidence on your tests.
It is impossible to make your tests run on all the existent devices in the world. So, what you need to do is getting the information about what are the most used devices for your app. Exactly, this needs to follow your app, your users. If you don’t have and there is no possibility to get this data, then you can follow the most used devices in general. Focus on your app and your client in the first place.
In this category we can include the different OS’s, screen resolutions, etc.
Mobiles are trick because you need to test the network, so you will need to have specific scenarios to simulate the 3G, 4G, WiFi. Remember to have the expected behaviour with poor connection or if the connection drops down and back again.
Language (Localisation Testings)
If you have a multiple language app, you also need to worry with the translation.
- You can add the language after all the smoke tests are done, since this is easier and faster to test manually.
- You can add a specific scenario to go through all the pages and check the translation against the database.
- You can specify on your automation that you will run each time with a different language and add the checks along the scenarios.
My suggestion is go for a specific scenario going through all the main pages and checking the translations (2). If you go with option 3 remember, your automation will take longer since it is performing all the scenarios again but with different languages, when a simple assertion on the page without any functionality check would be enough.
As for mobile, you can have portrait or landscape, so you need to remember to add scenarios related to the orientation. You can start the tests including both of the orientations. You will need to set this in the beginning of the automation or you can have specific scenario to test the orientation for the main screens.
Emulators vs Real Devices
Another aspect for which “balance” is a good mantra is testing on real devices vs. emulators. Emulators can’t give you the pixel-perfect resolution you might need for some testing or allow you to see how your app functions in conjunction with the quirks of real-life phone hardware. But they do allow you to do cost-efficient testing at scale and are a powerful tool to have in your mobile testing arsenal. A healthy mix of real device and simulator/emulator testing can give you the test coverage you need at a reasonable price.
Be sure you are leaving room for growth, both of the marketplace and your own needs. You need to always choose the best tools and practices that fit your needs, but at the same time you need to think about what is coming in the future. So, expand your automation thinking about what could come next, and minimize the threat of having to spend more time and resources redoing or revising systems that are out of date. Choose always flexibility: Cross-platform testing tools and scalable third party infrastructure are a good example of how to keep it.
Hi guys, today I have another Webinar that I participate about Appium, how to choose the automation mobile tool, how should be the structure, etc. You can watch online or download, just click in this link to start.
You can jump the advertisement moving to 00:10:51 of the video. Also, you can jump to 00:23:12 and watch the demo. To jump the demo go to 00:40:00.
Difference between hybrid native and html
– HTML: web applications which runs in web browser.
– Native: just native elements, apps which doesn’t have html elements.
– Hybrid: mix of native elements and html elements.
Mobile QA Solutions
– How choose the automation tool ?
– Verify how you inspect the elements.
– Scalability, How many devices supports (IOS, windows phone, tablet, kindle, ipad, etc).
– Integration with other tools like Jenkins, JIRA, etc.
– Cloud, If this tool supports cloud tests (Amazon cloud, Sauce Labs, etc).
– Automation Support, How long do you take to find the solution for some problem ? (Particularly, this is the most important for me )
Open source automation test automation framework for native, hybrid and mobile apps.
Environment: Appium, Android Studio (Sdks) and Eclipse.
Setting parameters: App device, launch device, android setting (path of sdks).
Make sure that you are doing right
– Do some proof of concepts (The configuration it is the hard and most boring part for me).
– Share the right quantity of tests between emulators and devices.
– Distribute the right level of tests between unit (70%), integration (20%) and manual tests (10%).
– Remember that if you are using BDD, it is just 3 layers.
– Use CI since the beginning.
– Decide what must to have in the report.
– Considerate the language which the developers are using (You never know when you will need help).
Thank you guys ! See you next week 🙂
Hey guys, today I will post a webinar that I joined last week about Mobile CI.
It is very interesting, so you will see briefly when you need to use, why, how it works, etc. It lasts 43 minutes so, make sure that you have time to watch. Jump to 00:27:35 to watch the demo of Sauce Labs.
To download or watch you need follow the instructions in the link below:
What is CI for mobile ?
Automatically – every commit
How does it works?
– Reduce the need for human-based testing (Save money and time)
– Faster Feedback
– Automate Everything (Very optimistic point of view)
– No touch configuration
– Automated OTA Distribution
– Code Validation: Automated Builds and Tests
How choose the right process ?
Depend the size of the
Mobile is harder
– Infrastructure (If you are building in iOS – You will need XCode, a MacMini, etc.)
– Compilation/Code Signing (You have to configure your build machine, so you will need your key store file, mobile provisioning…)
– Testing (What is the right time to use, challenge to choose what is the better for the project- real devices, simulators…)
Simulators x Emulators x Devices
– Used by iOS, included w/ XCode
– Execute i386 instruction set, not ARM
– Upside: Very fast compared to Emulators
– Downside : You don’t have access hardware functions (GPS,Bluetooth Radio, Accelerometers,etc)
– Used by Android, included with Android sdks
– Execute ARM (or native device instruction set)
– Upside: You can find free tools
– Downside: Very slow compared to Simulators
– Do not have access to certain hardware functions (GPS, Bluetooth Radio, Accelerometers, etc)
– Upside: Reproduces the actual performance experienced by your users
– Upside: The only way to catch manufacturer/OEM bugs
– Downside: Very expensive to configure and maintain compared
– Upside: Full access to hardware functions (GPS, Bluetooth Radio, Accelerometers, etc)
When to use devices vs simulators/emulators ?
– Emulators and Simulators are an excellent solution for running automated tests during development. They are inexpensive and will reliably catch most problems.
– Physical Devices can be used on a lower frequency (i.e. pre-release, weekly, daily, etc.). They are the only way to catch performance problems, test certain hardware features and find OEM issues. In the least, devices should be used before every release.
– 77% of developers said app quality is “very important or mission critical”
– 80% of mobile developers test their apps manually
– Only 13% of mobile developers use automated testing
Unit vs Functional
– Unit: Testing small pieces of code
– Functional: Testing button clicks and UI interaction
– Instant Gratification !
– Can automatically test every commit
– Unit Tests are not users
– Lot of work to write and maintain them
Which framework to use ?
– You need to remember:
– What is the language/framework do your developers know
– Open Source/Community Support
– 3rd party framework requirements
CI Tools/Services for mobile
Open Source, Lots of plugins, iOS and Android
Self-Hosted, DYI solution (Setup all the environment, it isn’t easy this part)
– Travis CI:
Hosted Solution, OS X support, lots of plugins, iOS and Android
Tedious setup process
Hosted solution, Designed specifically for mobile, Easy setup, iOS and Android
Less flexible than other solutions
– Xcode CI:
Integrated w/ Xcode, Apple-Supported
Self-hosted, iOS Only
Thank you guys, thank you Sauce Labs ! Hope you like it !