Blog We love to talk!

Apr Tue

Cross Browser Testing Solutions – The Ultimate Guide

  •  4-21-2015
cross browser testing

If you build it, they will come. Unfortunately, the method they’ll be using to get there is a little more uncertain. Cross browser testing solutions are an essential web development tool, allowing designer and developers to develop and test their application on a variety of platforms at once. There are a multitude of products available, depending on a developer or designer’s individual needs and preferences. When comparing cross browser solutions, you should be looking for solutions that offer features depending on your needs and that support the all necessary software and operating system options.

Browserling

This affordable, light, HTML5 cross-browser testing service offers live interactive sessions for Internet Explorer, Chrome, Firefox,  Safari and Opera. 

browserling.com

For developers that just want to run a quick test, Browserling is an extremely attractive option. Browserling offers a free service (which is limited to one version of Internet Explorer, though it has a complete listing of Chrome, Firefox, Safari and Opera versions) and its basic service is extremely affordable (under $20 / month). Launching Browserling, which is an HTML5-based solution, takes less than a second; you just type in the URL and go. Tools offered through the program include local testing (through SSH), resolution changing and both screenshots and video.

Unfortunately, Browserling is a little light on advanced features. Only resolution can be controlled — so while you could theoretically test a mobile device simply by changing your resolution, it will not show you any quirks of the actual mobile platform, just of the relevant browser (i.e. Safari). But Browserling does now offer a so called Live-API which offers embedding and automation support through HTML 5 and JavaScript.

Browserling is an excellent solution for desktop browser testing. Users who need an affordable solution to test out specific browsers in a desktop environment may find this service quite attractive. Users who need to test on a single specific platform for free will also find this an easy service to load up and test out.

Why should you choose Browserling? 

“We let you embed and automate browsers in your own applications through Live API. Users should choose us because we’re a small and fast company. We adopt to customer needs quickly and we’ve fast and enthusiastic customer support. We love our users and our users love us!”

Peteris Krumins, Browserling

Pros: Extremely fast and simple, free service available, very affordable.

Cons: No mobile support.

BrowseEmAll

BrowseEmAll is a comprehensive mobile and desktop browser testing application, allowing users to compare up to four different browsers and mobile simulators side-by-side.

browseemall

BrowseEmAll is a fast and clean cross browser testing solution. With a drop down list of literally dozens of browsers and mobile devices, developers can quickly flip through their site on a multitude of platforms and correct their layout as needed. Simulations render quickly and accurately, and a useful side-by-side feature lets the developer compare their site on up to four platforms at once. For more than four comparisons, developers can open each simulator and browser separately, allowing for easy viewing. BrowseEmAll is an all-in-one tool for designers who want something simple and easy to use.

Unlike many other cross browser solutions, BrowseEmAll is a local, desktop application — not cloud-based or web-based software. It launches quickly and loads cleanly, with the additional advantage of being able to work in a staging environment rather than on a live site without any SSH or Java connection tunneling. There are subscription and one off licensing options available, all being fairly reasonable pricing. It’s an excellent all-around solution.

The standard version of BrowseEmAll does not come with support for mobile devices — which is fairly essential in a modern development environment. Most users will require at least the professional version, which is $159/license or $12/month. However, this is still lower than all tested competitors.

BrowseEmAll does offer an automatic compatibility checking feature alongside a screenshot functionality. Users have complete control over the resolution of their displays and can view their HTML and CSS within the application itself. As with most cross browser applications, the application focuses primarily on Internet Explorer, Firefox, Chrome, Safari and Opera. Users can save their preferences for the application so that they are restored each time.

Why should you choose BrowseEmAll? 

“Because of our desktop only architecture BrowseEmAll is a great tool if you need to cross browser test security sensitive or locally hosted websites. Many customers also prefer our one-time fee rather than the ongoing subscription of nearly all other solutions. “

Daniel Herken, BrowseEmAll

Pros: Side-by-side testing function, comprehensive mobile device support (iOS, Android and BlackBerry).

Cons: No mobile browser in the standard version, Windows only.

Full Disclosure: Please note that you are currently on the website of the vendor of BrowseEmAll

CrossBrowserTesting.com

Cross Browser Testing invites developers to choose their operating system, their browser and then their website, allowing developers to view their website on over 900 combinations of platforms.

CrossBrowserTesting.com

With CrossBrowserTesting.com, developers can create a live testing environment based on the selection of an operating system, browser type and their website. Cross Browser testing even supports some of the most popular plugins, for developers who want to have an accurate picture of what their users are looking at. Snapshots and videos can both be recorded through the web-based solution and the it supports basic authentication, for programmers that need to test login-gated websites. Local support is also available through Java. The solution includes Flash and other necessary plugins to ensure that the web experience is reproduced with as much fidelity as possible.

Some developers may be turned off by the subscription-based service; though extremely affordable, it does mean that this service will be a recurring expense. For most professional designers, it won’t be a problem. Plans are available for solo users, professionals, teams, businesses and enterprise clients, and the pricing for all of these options is quite low. All subscriptions are able to complete an unlimited amount of manual testing and a specific amount of automated testing, starting at 150 minutes for the most basic plan.

As noted, Cross Browser Testing offers “over 900” combinations. This also counts plugins, which are useful for developers who want to test out more specific environments. However, their array of operating systems and browsers are also extremely competitive. Cross Browser Testing also recently released their own plugin for Selenium, allowing users to capture their tests and then play them back later to isolate and analyze any issues. Where Cross Browser Testing has the clearest advantage is in its user experience. Even beginners will be able to easily load their website in a variety of formats, even if it isn’t necessarily the most expedient way for more advanced users.

Why should you choose CrossBrowserTesting.com? 

“CrossBrowserTesting.com provides access to over 900 OS and browser combinations. Unlike their competitors, they use real operating systems and mobile devices, not just simulators.  Live testing, automated screenshots, and selenium are all included in every plan.  Their simple pricing means you get all their tools and access to all their browsers and physical devices for one low price.”

Ken Hamric, CrossBrowserTesting.com

Pros: Simple, straightforward and easy to use. Low introductory pricing and automated testing.

Cons: Subscription-based service.

BrowserStack

Web-based virtual environment browser and platform testing, with automated testing environments and screenshot generation. 

browserstack.com

BrowserStack boasts 100% reliability through their Rapid Selenium WebDriver cloud testing, with over 700 platforms to choose from. Users simply load up the website, click a few selections and are immediately treated to their website in the chosen environment. Regardless of which subscription service is purchased, each account is allowed unlimited testing time. Instead, plans are differentiated between how many parallel tests or sessions can be run. The basic accounts for live testing allows one session at a time while the basic accounts for automated testing allow two parallel tests at the same time. Enterprise accounts are available for larger companies and the services are tiered based on company size.

The testing platform that BrowserStack offers is simple and clean, with a lot development tools already loaded. As it’s a web-based solution, it can be used on any platform and in any major browser. Screenshots can also be generated in a variety of platforms so that designers and developers can look at their layout across multiple platforms quickly — this enables them to focus on major issues first and test out any minor changes to their website as necessary.

BrowserStack also encountered some consumer faith issues in late 2014 after the announcement that it had been compromised — and that its enterprise-level customers had been affected — so customers that are concerned about their security may want to consider a different (not web based) testing solution.

Why should you choose BrowserStack? 

“If you decide to go ahead with us, you will see continuous innovation – with respect to speed, UI and features – all with the view to make browser testing easier and faster. We invest thousands of hours, so you can test in minutes.”

BrowserStack

Pros: Unlimited testing time, affordable for solo testers, automated testing environments.

Cons: Rendered via Flash, local testing only available through a Java.

Browsershots

Get screenshots of your website in over 120 browsers in under an hour with this free screenshot service. 

browsershots.com

The queue for Browsershots appears to range between 15 minutes to 30 minutes. Once you’ve finished waiting, your screenshots are generated and you can download them. The download itself expires within fifteen minutes (sixty minutes for priority users), so you need to work quickly — but the system itself is relatively clear, fast and accurate. Priority processing is $29.95 a month. You can extend the amount of time that the download will last manually, but you need to click a button while remaining on the screen.

Browsershots does not offer mobile screenshots, so that limits its usefulness as an all-around testing tool. It does, however, offer an impressive variety of browser and operating system combinations. Browsershots also cannot be used to test local documents, which again could be problematic to users that don’t yet have live environments. Browsershots is one of the oldest solutions available and it’s a bit difficult to use for some users.

Browsershots does not offer live testing; in fact, Browsershots actually refers users to Cross Browser Testing for live action testing. Browsershots is a unique cross platform development tool that simply does one thing very well: create screenshots of websites in a variety of platforms. While most cross browser testing solutions do include automated screenshots, many of them have a limited amount of automation processing time, and they require a subscription fee. By using Browsershots, developers can acquire screenshots of their website at any time without having to pay an additional fee or use up their minutes.

Ultimately, Browsershots is probably better for a hobbyist than a professional developer because of its limitations. But it’s still something that a developer might want to keep in mind as an alternative tool should they simply need a large amount of screenshots generated for free. For single screenshots, it may actually be faster to just download the relevant browser and test than wait for the queue. It’s likely though that Browsershots probably offers the most screenshots available in a wide variety of settings.

A caveat is that you need to set the resolution before you actually load the page. One major advantage to live testing is that you can see how your page reacts when you resize the window. With Browsershots, you would need to run multiple requests for every single resolution you are thinking of supporting and you still wouldn’t be able to identify any glitches that occur when you’re actually resizing or switching between resolutions. That can be incredibly time-intensive and is not likely to be a valuable use of a developer’s time. On mobile devices, this may not matter; on desktops, where users continuously position their windows, it may be more of a problem.

Pros: Free automated screenshot testing service on over 120 browsers.

Cons: No live testing, long wait queues.

Sauce Labs

Test your site on the cloud with over 450 browser and operating system combinations. 

sourcelabs

Sauce Labs is an extremely robust cross browser testing solution that offers a wide variety of platforms and an affordable subscription service. Uniquely, Sauce Labs lets you connect with other users so that websites can be debugged simultaneously in real-time — the perfect thing for larger teams. With a subscription service that begins at $12 a month (on an annual basis; $16 a month for month-to-month subscriptions), Sauce Labs is also one of the most affordable solutions available. Sauce Labs offers real-time virtual machines, a wide selection of debugging tools, video and screenshot capture and automated testing. The basic account offers unlimited manual testing for a single user and two hours of automated testing. Local testing is offered through a secured local tunneling.

Sauce Labs is a fairly new startup, launched with the goal of making development time shorter and the testing process much faster. This developer-driven business offers solid customer support at a very low cost (though it can be intermittently accessible), and the products used are almost all open source. In addition to offering accurate virtual machines as live testing environments, tests can also be automated and triggered through the use of a variety of languages, including Java and Ruby. Development tools are included on the virtual machines themselves.

Sauce Labs also distinguishes itself from the competition by having an extremely robust set of online documentation, which can be reviewed by anyone interested in the product and how it works, even before purchase. The documentation outlines details not only about their product but also about the issues that developers might run into while testing their system incompatibilities.

Sauce Labs is a good all-around solution for the developer on the budget, offering many extremely robust features in a streamlined, easy-to-use package. As Sauce Labs continues to grow and develop, it’s likely that it will become even more powerful and accessible.

Pros: Live share functionality to work directly with coworkers in real-time, a large amount of combinations.

Cons: No scheduling service

Multi-Browser Viewer

With 67 standalone browsers, 16 mobile emulators and 80 screen capture browsers, Multi-Browser Viewer can create a complete testing environment for a developer.

multi browser viewer

Multi-Browser Viewer creates a full testing environment on your local computer. The program includes 67 virtualized browsers, which can be used to perfectly replicate any website — local or live — in a variety of browsers on Windows. Mobile emulators can be used to test out Android, iOS and BlackBerry devices while a total of 80 screen capture browsers (hosted by MultiBrowserViewer on the web) let you generate screenshots showing your website in high resolution. Users looking for a desktop application that will allow them to test out their website quickly and in a controlled environment may find Multi-Browser Viewer the perfect solution. Multi-Browser Viewer includes a variety of development tools, including HTML 4 validation which will look through the HTML code of websites and isolate and identify any potential problems.

At $139.95 per license, Multi-Browser Viewer may be a little pricey for what it is. Though it is licensed once, support runs out at the end of the year. Maintenance plans are also available which include free upgrades, and discounts can be acquired through package deals. The program itself is a little dated and the fact that it is a desktop application may be a negative to some. The program does not include as many browsers, operating systems or mobile device features as many of the other leading cross-browser testing solutions, but it’s also cheaper than many of them even on an annual basis.

Automated testing can be run through Selenium in Multi-Browser Viewer, including a series of browsing steps that can be run against any of the included browser. Multi-Browser Viewer also has sandboxed environments for testing. When running the automated screenshot generation service, the solution will automatically save HTML captures and screenshot captures to the computer, which can make the process of testing much easier than in a standard web or cloud environment.

Perhaps the biggest problem with Multi-Browser Viewer is that it is a desktop application but still sends its queries to a virtual machine on the web. There are still web-related delays involved, along with the requirement to install a local desktop program; it’s basically the worst of both worlds. While Multi-Browser Viewer is generally fast, this could still represent some problems for people who are looking for a completely desktop solution. Authentication is not supported in the screenshot program, which can be a problem for those testing gated websites and for those concerned about security.

Finally, Multi-Browser Viewer is only available for Windows environments. Since a significant amount of development is done on Mac environments today, this cuts out a fairly large subset of developers.

Pros: Virtualized standalone browsers, mobile simulation, HTML 4 validation, screenshot testing.

Cons: Limited options compared to the competition, somewhat clunky interface, Windows only.

Browsera

Automatically test for scripting issues and layout differences through a variety of Firefox, Safari and Internet Explorer versions.

browsera

Browsera lets developers quickly test their entire site for layout issues and JavaScript errors. The automated testing process initiated by Browsera will crawl the entire site very quickly — something that most of the other testing solutions won’t do. Browsera also supports authentication, so a website that has login gates can also be tested through the crawling process, in addition to dynamic, user-based pages. For developers that simply want to ensure that their website is working as it should as fast as they possibly can, Browsera is an excellent solution. The program will identify any inconsistencies and incompatibilities in layouts and JavaScript, highlighting them so that they can be adjusted and displaying errors, warnings and other informational messages. High resolution screenshots will also be taken in the process.

That being said, there are some problems with Browsera, too; only Firefox, Safari and Internet Explorer are supported. At $49 a month for a standard package, it isn’t one of the cheaper services — though there is a free very limited service that is more like a trial. But perhaps the most problematic part of Browsera is that it’s limited on a page by page basis. The free subscription allows 25 pages by one user, while the project-by-project subscription allows 300 pages over the course of fourteen days. The standard subscription allows 500 pages a month. As any web developer knows, 500 pages is not a lot. Subscriptions through Browsera only go up to 2,000 pages a month over three users, which again could be significantly limiting to any large development company.

Browsera is the perfect solution for users who want to quickly be able to automate their testing process and catch any significant layout or JavaScript errors, but the fact that it only supports a few major browsers and will only allow for a few hundred pages a month may hold its usefulness back for some companies. The project-by-project one time purchase ($39 for 300 pages) may be useful for those who need a testing service sporadically, however, so developers should keep it in mind. Most developers find that Browsera is more accurate than many other services, and the functionality — including the web crawling — operates very fast. Regardless of developer skill level, Browsera is also a very accessible and easy to use program.

Pros: Clean, accessible interface, authentication support, automated web crawling.

Cons: Only three major browsers supported (Firefox, Safari and Internet Explorer), fairly expensive compared to other screenshot services, very limited testing.

TestingBot

Automatic and manual virtual machine powered cross-browser testing service with a total of 138 browser and operating system combinations.

testingbot

TestingBot supports Internet Explorer, Firefox, Google Chrome, Safari and Android testing environments. Local support is offered through secured tunnel. In addition to standard cross-browser compatibility tests, Testingbot also runs detailed analytics. The reporting tools will note when tests have failed and other issues that could crop up throughout the process, which can be very valuable to a developer. As a cloud service, TestingBot is almost always lightning fast. In fact, TestingBot claims that their tests are much faster then virtually every other cross-browser testing solution available.

Out of almost all testing services, TestingBot offers the largest amount of automated testing time for its standard accounts. A free trial comes with 100 minutes of automated testing and a small team package comes with 1,000 minutes. Four simultaneous tests can be run through their basic accounts and video and screenshots are captured throughout the automated testing process. Taken together, this makes TestingBot one of the most useful programs available for automated testing. Developers who need to do a large amount of automated testing will find this program quite useful.

Tests can also be scheduled in advance or scheduled to be run on a regular basis, and push alerts can be sent to your phone or through text messages — TestingBot is one of the few programs to offer this. If you want to test your website’s responsiveness at a certain time of day, the TestingBot solution will let you do that and will notify you immediately regarding whether the test failed or succeeded. In this way, TestingBot can be used as a continuous website monitor.

Manual testing is also as easy and offers the ability for live testing, which can be shared with colleagues to facilitate collaboration; others do not need to have their own TestingBot account to review the testing, you can simply send them an invitation. That being said, TestingBot may not have as diverse a selection of development environments as some other solutions.

Why should you choose TestingBot? 

“The reasons why we stand out are our location in Europe, fresh virtual machines for every test and extensive selenium support.”

Jochen Delabie, TestingBot

Pros: Secured virtual machines, automated and manual testing, advanced reports, 138 combinations, push alerts regarding scheduled tests.

Cons: Not as many combinations as some others.

Spoon

A lightweight sandbox environment including a variety of operating systems, browsers and other emulators.  

spoon

Spoon.net is a bit of a unique inclusion on this list. Rather than being a strict cross-browser testing service, Spoon is an emulator and virtualizer for a variety of programs. Through Spoon, you can launch a solution and then immediately begin using it in a clean, fast sandboxed environment, doing everything on that software system that you could do if it was installed natively. Spoon supports all major browser. So while it may technically “support” mobile through Firefox Mobile, it doesn’t have extensive mobile emulation the way other testing services do. Spoon is less a testing service and more a tool to facilitate testing.

For developers who quickly need to do something in Chrome or another browser, Spoon is invaluable; you can simply load up the site and run a program, often for free if you don’t need any advanced features. The browsers themselves can be loaded with development features enabled.

Since Spoon does allow for a sandbox environment, developers can potentially complete rather complicated and specific testing functions, integrating and using a variety of plugins and customized features to see whether their site will fail to perform. That being said, Spoon.net is fairly obscure for those who are not advanced developers. The amount of choices can be overwhelming, especially when attempting to configure a browser or load a specific version of the browser. It is mostly for moderately advanced users who know exactly what they need.

Advanced developers who are wondering about a very specific issue, such as whether a certain feature works in Chrome, will find that Spoon may be the fastest way to find their answer. Hobbyists and beginners might instead consider just installing Chrome. Developers may want to keep Spoon in mind, though, because it isn’t just for browsers. Spoon.net allows the testing of a multitude of other solutions that relate directly to web development and web programming, all of which could be useful.

Pros: Robust sandbox environment, more than just operating systems and browsers, automated building and emulation.

Cons: Requires a high level of knowledge to use; may be intimidating for new developers. Limited mobile support.

Equafy

This open source friendly solution offers automatic scanning, manual testing and record and replay of your site on a variety of options.

Equafy

Equafy has a fairly robust free service — it isn’t a trial, as there is no time limit involved. The free service offers two hours of automated testing, unlimited results and the ability to both run automated scans and record and replay tests. In fact, it’s the only free service to offer all of these features. Their basic plan is $49 a month and includes 10 hours of automated testing and the software solution is free for open source projects. Enterprise plans are available for development companies that require more time.

Users can create testing scenarios that are then automated, to put their websites through the paces and ensure that the website is operating as it should. Automated tests are captured through video and screenshot recording. With authentication, the entirety of the website can be tested on every available platform. Once testing is run, users can check for any differences between screenshots. Automatic visual bug detection rides on top of the platform, ensuring that nothing is missed — and this can be a great boon for a developer who has simply been staring at their site for far too long.

With support limited to Windows, Linux and Mac OS, Equafy unfortunately does not have robust mobile support — and that could be problematic for developers seeking an all-in-one solution. Still, it is one of the most affordable ways to begin testing, making it ideal for the hobbyist developer who simply needs to test. The user experience is solid and clean, and with a few added features Equafy could easily become a top contender. The automatic visual validation service and automatic issue checking is a huge bonus. Open source developers may also want to become familiar with the software as it grows, assuming that open source support continues.

Why should you choose Equafy? 

“Equafy allows users to easily automate the testing of user activity and templates, repeat it on multiple browsers, devices and resolutions. All that without requiring development skills”

Slavy Slavov, Equafy

Pros: Free for open source projects, automated scanning, authentication and local testing.

Cons: Only supports Windows, Linux and Mac OS.

The Final Verdict

So which of the listed cross browser testing solutions is the best of the best? Well, it really depends on you. If you want something light, fast and clean, maybe Spoon is the tool that you’ve been looking for. If you want something that’s a good balance of affordability and features, you probably can’t go wrong with Sauce Labs. Local and secure? Maybe BrowseEmAll is the way to go. Either way, any of the above listed products will have you well on your way towards rapid cross browser compatibility. Occasionally, you may need to use two or more tools to create a complete testing environment — the more the merrier.

Many developers will use at least a few testing environments occasionally. Though most of these products run virtual machines or extremely advanced emulators, there is still some room for error. Some environments may introduce issues that don’t truly exist or may exaggerate issues that cannot be reproduced. By using multiple testing environments, a developer can get a better picture of what is actually going on in a user’s system. It’s not always possible to obtain complete cross browser consistency — it’s simply a goal.

The majority of cross browser testing services — such as automation or screenshot batching — can be found through one or more services for free, so those who have not yet committed to a product might want to try a few out to determine which solution will work best for them. At their core, most of these solutions work in very similar ways and are priced extremely competitively. It is the user experience that will likely be the determining factor in whether a specific solution feels right for them.

Feature Chart for Basic Accounts

Accurate as of April 2, 2015. “Configurations” refers to advertised browser, operating system and plugin options. All of the services below have free trials, so don’t hesitate to try them out before committing to a purchase!

 BrowserlingBrowseEmAllCrossBrowserTestingBrowserStackBrowserShotsSauce LabsMulti-Browser ViewerBrowseraTestingBotSpoonEquafy
Basic Price$19/month$12/month or $99/license$29.95/month$39/monthFree$12/month$139.95/license$49/month$40/month$19/monthFree
MobileNo$19/month or $159/licenseYesYesNoYesYesNoYesNoNo
Local TestingYesYesYesYesNoYesYesNoYesYesYes
AutomatedYesNoYesYesNoYesYesYesYesNoYes
Video / Screenshot CaptureYesYesYesYesYesYesYesYesYesNoYes
Deskop or Web ServiceWebDesktopWebWebWebWebDesktopWebWebDesktopWeb
Manual TestingYesYesYesYesNoYesYesNoYesYesNo

All services have been tested with an undisclosed paid account.

No Comments

Leave a Reply

Your email address will not be published.