Introducing Grasshopper – An Open Supply Python Library for Load Testing
We’re excited to announce {that a} new open-source venture has joined the Alteryx open-source ecosystem. Grasshopper is a library for automated load testing, written in Python.
Introduction
At Alteryx, one in every of our aims is to develop modern instruments and frameworks that improve software program testing talents. As a part of this mission, we’ve designed a user-friendly, open-source framework that allows people to optimize their efficiency testing efforts. This framework is particularly meant to assist measure software program efficiency by load testing in opposition to an API by producing metrics corresponding to response instances, failure charges, and customized developments and reporting these metrics to a timeseries database. Our intention is to assist everybody progress on their efficiency testing journey by way of this framework.
Our crew beloved how straightforward it was to get began with Locust, however, it lacked an out-of-the-box answer to measure the timings that span a number of HTTP requests. This was a vital consider assessing our software program’s efficiency. Moreover, we have been unable to combination metrics for a generalized conditional assertion or report these metrics to a timeseries database. These options have been important for us to conduct environment friendly and complete efficiency testing of our software program. In consequence, we prolonged locust to create Grasshopper, which incorporates all of those crucial capabilities.
So, we’re excited to announce a brand new addition to our open-source tasks: Grasshopper, a load-testing Python library. Grasshopper makes use of Locust below the hood, and offers just a few key options on high, specifically:
- Checks
- Customized Traits
- Timing Thresholds
- Timeseries DB reporting and tagging (specifically InfluxDB)
- Integration with PyTest
Background
Just lately, our crew was offered with a uncommon alternative that SDETs usually dream of – an opportunity to pause and consider the tooling we have been utilizing for our efficiency testing. After rigorously surveying the out there choices, we selected Locust as the inspiration for our answer. This determination was made with the advantage of hindsight, permitting us to include classes discovered from earlier experiences into our tooling technique.
The story of Grasshopper begins out with making an attempt out Locust, discovering the wonderful advantages it launched, after which constructing on high of these options.
Why Locust?
As a efficiency crew with SDETs, we have been taken with Locust for quite a lot of causes, together with:
- it is Python – Lots of our SDETs had experience in Python and needed entry to the simplicity of Python and the large ecosystem of python packages out there. Decorators, pytest, and breakpoints to call just a few.
- it is modular – Locust was constructed with customizability in thoughts. If we had some take a look at requirement that it didn’t initially fulfill, we could always add a custom listener or decorator to increase the performance.
- it is reasonable – Locust supplied some nice options round carefully replicating consumer conduct. For instance, one might outline an inventory of duties for every consumer to carry out, after which put weights on every job to find out how usually every will probably be carried out.
- we already had a shared API take a look at library in Python – Our high quality group had already developed a library for most of the API actions utilized in purposeful exams. If we have been capable of re-use these capabilities in our efficiency exams, then that may be enormous, as we might preserve each frameworks without delay with none duplicated code.
Quick Advantages
Our crew made a major breakthrough after we built-in Locust with the shared purposeful API take a look at library. Although there have been some particulars to iron out, we have been finally profitable in making it work. This achievement had a monumental impression on our crew’s effectivity and the general high quality of our work. By leveraging the shared library, we have been capable of eradicate the necessity for a devoted set of API actions for efficiency testing, releasing up our sources to focus on extra crucial duties. Moreover, our skill to contribute to the shared library garnered enthusiasm and satisfaction from our colleagues within the high quality group.
However, we would have liked extra… time for some innovation!
Regardless of the love for Locust, we recognized a lot of objects that extra mature instruments and frameworks present.
- Tag-based suites for development evaluation and different conditions the place we consider a proposed change to the product/configuration
- Customized developments. For instance, say you’ve got some timed motion that spans a number of HTTP calls. Locust doesn’t have the performance to maintain monitor of such a timing, so we added this as a decorator that may be wrapped across the technique that you simply care about.
- Checks. Checks validate boolean circumstances within the take a look at. For instance, a examine might validate {that a} response physique has a sure form. This examine is given a reputation, and is aggregated over time.
- Customized tagging for all metrics (checks, HTTP requests, Customized Traits)
- Information to our timeseries db (influxdb) & dashboards (Grafana). This contains HTTP response instances, customized developments, checks, and customized tags.
- Thresholds. For instance, you care that the ninetieth percentile of HTTP requests for a sure request aren’t above x ms.
- Reporting outcomes to different areas (console studies, reportportal, slack, and so forth.)
- And lastly, what each good testing framework wants – some reusable base lessons that care for the vast majority of the boilerplate that exams usually comprise
So, we got down to construct a package deal that mixes different open supply instruments and a few further code so as to add in lots of these things. General, our purpose was that you simply set up the package deal, make a replica of the instance take a look at and begin filling out the code that’s particular to the factor you need to take a look at. This ended up being a non-trivial quantity of labor and we’d like to avoid wasting you from this identical labor, so asserting Grasshopper – the complete(er) efficiency testing answer written all in python, constructed on high of Locust and Pytest, sporting many comfort capabilities.
Life After Grasshopper
Since grasshopper is very easy to get began with, we determined to evangelize this device on to our personal builders inside Alteryx, and have seen numerous success within the course of. Now, there are developer groups that independently use this device, and have the pliability to increase it to their very own particular use circumstances. This has been an enormous win for shifting efficiency testing left in our software program improvement lifecycle.
As an added bonus, we have been capable of reuse the api mannequin library constructed for purposeful exams, saving an amazing quantity of labor that may finally duplicate code. This additionally drastically hastens our skill to breed circumstances {that a} buyer has escalated.
Utilization
Set up
This package deal might be put in through pip: pip set up locust-grasshopper
Additional Studying
Please seek advice from the project README for essentially the most up-to-date documentation on how one can create load exams with Grasshopper. There may be additionally an in-depth instance that can be utilized to get began in your load testing journey!
Future Outlook
Sooner or later, we plan on including the next performance to grasshopper:
- slack reporting
- reportportal reporting
- prometheusDB reporting
Contributions
Particular because of the crew that constructed Grasshopper: Suzanne Ezell, Jacob Fiola, Logan Michalicek, Simon Stratton, Mykola Solopii, Vijaya Ayyappaneni, and Ashwin Chandrasekar. As well as, we wish to thank Laurie Linz. None of this could have been attainable with out her assist and management.
We might additionally like lengthen our thanks the maintainers of locust and locust-influxdb-listener. Grasshopper couldn’t exist with out the usage of these packages.
You probably have concepts for enhancing or enhancing Grasshopper, open supply contributions are welcome. To get began, try the Grasshopper “contributing” part within the venture README.
If Grasshopper doesn’t fairly suit your wants, listed below are another load testing instruments we evaluated, all open supply choices (definitively not exhaustive).
- k6 – javascript based mostly; the first cause that we didn’t select k6 was that the code is compiled to golang earlier than execution, so debugging was difficult. This additionally prevented us from utilizing most npm packages in our efficiency exams.
- Gatling – scala based mostly, contains recording function, good GUI for watching reside exams; we dominated this out due to a scarcity of scala expertise.
- JMeter – java based mostly, pretty mature device; However a significant draw back is that it makes use of a completely domain-specific language. Ultimately, we determined to go along with a device that would share our API library which was in python