January 3, 2022

Picking an API automation tool


Selecting an API automation tool has become increasingly difficult over the years. The plethora of tools available makes it very difficult to make a choice that fits the requirements of any project. This analysis represents our research and experience in testing and using some of these tools in our projects at Indexnine. We have identified criterion and fit the strengths and weaknesses of each tool to arrive at a guide to best select a tool for a problem at hand.


While every tool is not created equal, each tool has its strengths and weaknesses. We typically apply the following criteria to determine fit for a project.

Client requirements

Client requirements such as budget(s) and choice of language for maintainability and available talent plays a big role in determining what is best for the project. Out of these, particularly pay attention to the availability of talent while determining which tool to use.


Project structure and technologies used are an important aspect of every project as these are the pillars of the project. While selecting tools we have to consider the technologies used in the larger project. As far as possible try to match the language used for development and testing. This leads to better maintainability and much better collaboration between developers and QA team.

Open source vs Commercial automation tools

Budgets and required investment in commercial automation tools are an important factor that has to be considered. While startup customers typically like to go open-source, enterprise customers may have specific preferences in the tools used. We extensively work with open source tools that offer the best-fit to teams.


As the project grows, features and code base grows. As projects go live we routinely get asked some/all of the following questions:

  1. Can you share reports after automation runs?  
  2. Can the automation runs be integrated within a CI/CD process?
  3. Can we set up a quality gate to avoid releases if tests don’t pass?
  4. Can this be implemented in <Typescript>/<Python>/<Java>/<Pick a language>?
  5. Can you automate using BDD so tests are easily understood? 


The following table lists out the tools we have used and experimented with in various projects at Indexnine. Hopefully this table will help you determine which tool may be the best suited for your product team. We are comparing the following tools: POSTMAN, Katalon, Robot, RESTAssured and Karate. While this is not a comprehensive list of tools, it represents a cross-section that is most useful in the types of projects we execute at Indexnine.

Basic Capabilities
Installation of tool and getting started with automation project
Does the tool support SOAP API automation?
Does the tool support REST API automation
Test assertion templates:
Does the tool support test assertions?
Can automated test scripts be reused easily? 
Data driven:
A data source, like a spreadsheet or other table, is used for the input values and typically the testing environment is not hard-coded
BDD support:
Test cases are written in a simple plain english language that everyone can read and understand.
Default Test report:
Report helps understand execution details of automated tests, it helps visualize details of each iteration and test results
(plugins needed)
e.g: HTMLExtra
YesYesNo (plugins needed)
TestNG, JUnit runner and Allure Report
Environment Variables Setup:
Prepared scripts are easily executable on different environments such as local, DEV, QA, Production
Scripting languages:
Languages supported by tool for test assertion
Java scriptJavascript,
Complexity of test preparation:
Degree of complexity for test cases preparation and scripting in tool
Learning Curve:
How easy/difficult it is to learn this tool.
Tags and annotations:
User-defined tags for grouping of test cases/ apis for specific execution
Advanced Capabilities
UI Automation:
Do these tools have an option to perform UI automation
API performance testing:
Are these tools able to perform API performance automation
Yes (Limited)YesYesNoYes
Are these tools able to perform CLI action for execution to support CICD
Parallel execution:
Are these tools able to parallel execution of the automation suites
Yes (Paid)YesYesYesYes
Debugging and logs:
Do these tools provide support for debugging and if logs are easy to understand
What is the complexity of maintenance for automation suites and scripts
Community Support:
Do these tools are having community support
Yes, wide groupYes, wide groupYes, wide groupYes, wide groupYes (Limited)
Are these tools open source and pricing details if any
Open source, API monitoring – paid (Pricing)Basic version is free:(Pricing)Open sourceOpen sourceOpen source


We hope the above table helps you get started on your automation testing journey. Investing in a good automation test suite is the best way to ensure efficient addressing of regressions in products. 

The review ratings above are based on our experience and represent our opinion on these tools. 

As always, do consult the latest documentation on these tools to determine best fit for your project. Since the tools are constantly evolving, new features and capabilities may be available when you are ready to implement them.

Do reach out to Indexnine at info@indexnine.com or through the website form if you need any help setting up your QA automation teams. Happy product building!