Showing posts with label Performance Testing. Show all posts
Showing posts with label Performance Testing. Show all posts

Friday, 14 March 2014

Opensource Performance Test Tools

Description:
The main goal of the project is to create a distributed generic system collecting and storing various runtime metrics collections used for continuous system performance, health, quality and availability monitoring purposes. Allmon agents are designed to harvest a range of metrics values coming from many areas of monitored infrastructure (application instrumentation, JMX, HTTP health checks, SNMP). Collected data are base for quantitative and qualitative performance and availability analysis. Allmon collaborates with other analytical tools for OLAP analysis and Data Mining processing.
Requirement:
Platform independent

Description:
Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.
Requirement:
Solaris, Linux, Windows (98, NT, 2000). JDK1.4 (or higher).

Description:
benerator is a framework for creating realistic and valid high-volume test data, used for (unit/integration/load) testing and showcase setup. Metadata constraints are imported from systems and/or configuration files. Data can be imported from and exported to files and systems, anonymized or generated from scratch. Domain packages provide reusable generators for creating domain-specific data as names and addresses internationalizable in language and region. It is strongly customizable with plugins and configuration options.
Requirement:
Platform Independent

Description:
CLIF is a modular and flexible distributed load testing platform. It may address any target system that is reachable from a Java program (HTTP, DNS, TCP/IP...) CLIF provides 3 user interfaces (Swing or Eclipse GUI, command line) to deploy, control and monitor a set of distributed load injectors and resource consumption probes (CPU, memory...) An Eclipse wizard helps programming support for new protocols. Load scenarios are defined through XML-editing, using a GUI, or using a capture tool. The scenario execution engine allows the execution of up to millions of virtual users per load injector.
Requirement:
Java 1.5 or greater, with enhanced support for Linux, Windows XP, MacOSX/PPC

Description:
ContiPerf is a lightweight testing utility that enables the user to easily leverage JUnit 4 test cases as performance tests e.g. for continuous performance testing. It is inspired by JUnit 4's easy test configuration with annotations and by JUnitPerf's idea of wrapping Unit tests for performance testing, but more powerful and easier to use.
Requirement:
Windows, Mac OSX, Linux, Solaris and all other platforms that support Java 5

Description:
A C-written web application testing and load generating tool. The goal of the project is to provide a powerful open-source alternative to Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and TLS/SSL protocol stacks, simulating tens of thousand and hundred users/clients each with own IP-address. The tool supports user authentication, login and a range of statistics.
Requirement:
linux

Description:
D-ITG (Distributed Internet Traffic Generator) is a platform capable to produce traffic at packet level accurately replicating appropriate stochastic processes for both IDT (Inter Departure Time) and PS (Packet Size) random variables.
Requirement:
Linux, Windows

Description:
The Database Opensource Test Suite (DOTS) is a set of test cases designed for the purpose of stress-testing database server systems in order to measure database server performance and reliability.
Requirement:
Linux, POSIX

Description:
DBMonster is an application to generate random data for testing SQL database driven applications under heavy load.
Requirement:
OS Independent

Description:
An open-source web site stress test tool. Simulates multiple user types and counts. Includes proxy server for recording playback scripts, and log evaluator for generating result statistics. Note: this tool is no longer under active development although it is still available on Sourceforge. BEWARE: This tool has not been updated since 2002. It remains listed here in case anybody wishes to take it over.
Requirement:
OS independent

Description:
Dieseltest is a Windows application that simulates hundreds or thousands of users hitting a website. BEWARE: This tool has not been updated since 2001. It remains listed here in case anybody wishes to take it over.
Requirement:
Windows

Description:
Faban is a facility for developing and running benchmarks, developed by Sun. It has two major components, the Faban harness and the Faban driver framework. The Faban harness is a harness to automate running of server benchmarks as well as a container to host benchmarks allowing new benchmarks to be deployed in a rapid manner. Faban provides a web interface to launch & queue runs, and extensive functionality to view, compare and graph run outputs.
Requirement:
OS independent; JVM 1.5 or later.

Description:
FunkLoad is a functional and load web tester, written in Python, whose main use cases are functional and regression testing of web projects, performance testing by loading the web application and monitoring your servers, load testing to expose bugs that do not surface in cursory testing, and stress testing to overwhelm the web application resources and test the application recoverability, and writing web agents by scripting any web repetitive task, like checking if a site is alive.
Requirement:
OS independent - except for the monitoring which is Linux specific.

Description:
fwptt is an open source Web application testing program for load testing web applications. It can record normal and AJAX requests. It has been tested on ASP.Net applications, but it should work with JSP, PHP or other.
Requirement:
windows

Description:
The Grinder is a Java load-testing framework making it easy to orchestrate the activities of a test script in many processes across many machines, using a graphical console application.
Requirement:
OS Independent

Description:
GrinderStone is an Eclipse plug-in for Grinder load testing scripts development including debugging, modularity and pretty logging
Requirement:
All

Description:
Hammerhead 2 is a stress testing tool designed to test out your web server and web site. It can initiate multiple connections from IP aliases and simulated numerous (256+) users at any given time. The rate at which Hammerhead 2 attempts to pound your site is fully configurable, there are numerous other options for trying to create problems with a web site (so you can fix them).
Requirement:
Hammerhead has been used with Linux, Solaris and FreeBSD.

Description:
Hammerora is a load generation tool for the Oracle Database and Web Applications. Hammerora includes pre-built schema creation and load tests based on the industry standard TPC-C and TPC-H benchmarks to deploy against the Oracle database with multiple users. Hammerora also converts and replays Oracle trace files and enables Web-tier testing to build bespoke load tests for your entire Oracle application environment.
Requirement:
Platform Independent (Binaries for Linux and Windows)

Description:
Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro and macro level benchmarks. The three distinguishing characteristics of httperf are its robustness, which includes the ability to generate and sustain server overload, support for the HTTP/1.1 and SSL protocols, and its extensibility.
Requirement:
linux (Debian package available), HP-UX, perhaps other Unix

Description:
http_load runs multiple HTTP fetches in parallel, to test the throughput of a Web server. However, unlike most such test clients, it runs in a single process, to avoid bogging the client machine down. It can also be configured to do HTTPS fetches.
Requirement:
tbc

Description:
Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.
Requirement:
Platform Independent

Description:
Monitoring tool with great visualization and customization capabilities. It's quick to install and suitable for use in performance labs.
Requirement:
Windows, Unix

Description:
j-hawk is a Java based open source framework which can be incorporated in your application for performance testing. The idea is you have to define *module and its tasks (means method) inside your application and register the same with j-Hawk. j-Hawk executes the modules and generates a graphical performance report which can be analyzed to find performance bottleneck of your application.
Requirement:
windows,ubuntu

Description:
JChav is a way to see the change in performance of your web application over time, by running a benchmark test for each build you produce. JChav reads all the JMeter logs from each of your runs (one per build), and produces a set of charts for each test in each run.
Requirement:
JMeter

Description:
Stress-Testing Tool for web-applications. It comes with the crawling/exploratory feature. You can give JCrawler a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. The load parameters (hits/sec) are configurable.
Requirement:
OS Independent

Description:
loadUI is a tool for Load Testing numerous protocols, such as Web Services, REST, AMF, JMS, JDBC as well as Web Sites. Tests can be distributed to any number of runners and be modified in real time. LoadUI is tightly integrated with soapUI. LoadUI uses a highly graphic interface making Load Testing Fun and Fast.
Requirement:
Any

Description:
Lobo is a tool for performance testing and monitoring that allows you to monitor the evolution of performance along the time-line of the project. It was specially designed to be used in agile-iterative and evolutionary approaches.
Requirement:
Java

Description:
MessAdmin is a light-weight and non-intrusive notification system and HttpSession administration for J2EE Web Applications, giving detailed statistics and informations on the application. It installs as a plug-in to any Java EE WebApp, and requires zero-code modification.
Requirement:
OS Independant

Description:
Mstone started as a mail performance measurement system but now can test svn, etc. It can simultaneously test SMTP, POP, IMAP, and some HTML based systems. It measures transaction latency in multiple stages, and graphs the combined results from multiple clients.
Requirement:
multiple (perl based)

Description:
Multi-Mechanize is an open source framework for web performance and load testing. It allows you to run simultaneous python scripts to generate load (synthetic transactions) against a web site or web service.
Requirement:
Any

Description:
nGrinder is the platform of stress tests which enables you to execute script creation, test execution, monitoring, and result report generator simultaneously. The opensource nGrinder offers easy ways to conduct stress tests by eliminating inconveniences and providing integrated environments.
Requirement:
Windows / Linux / Mac

Description:
The NTime tool is very similar to NUnit tool to perform repeatable tasks that help managers, architects, developers and testers to test an application against its performance.
Requirement:
Windows 98 or above, .Net framework 1.1 or 2.0

Description:
A distributed software testing architecture based on CORBA. Using OpenSTA (Open System Testing Architecture) a user can generate realistic heavy loads simulating the activity of hundreds to thousands of virtual users. OpenSTA graphs both virtual user response times and resource utilization information from all Web Servers, Application Servers, Database Servers and Operating Platforms under test, so that precise performance measurements can be gathered during load tests and analysis on these measurements can be performed.
Requirement:
Windows 2000, NT4 and XP

Description:
OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing near real-time performance measurements of the application under test.
Requirement:
Linux, Windows

Description:
Ostinato is an open-source, cross-platform packet/traffic generator and analyzer with a friendly GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark.
Requirement:
Cross-Platform

Description:
An open source framework for unit test and performance benchmark, which was initiated by Andrew Zhang, under GPL license. p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time consumption, and generates the result in the form of plain text, image or pdf file.
Requirement:
OS Independent

Description:
Pandora FMS is a monitoring Open Source software. It watches your systems and applications, and allows you to know the status of any element of those systems. Pandora FMS could detect a network interface down, a defacement in your website, a memory leak in one of your server application, or the movement of any value of the NASDAQ new technology market. If you want, Pandora FMS could send out SMS message when your systems fails... or when Google's value drop below US$ 500.
Requirement:
32-bit MS Windows (NT/2000/XP), All POSIX (Linux/BSD/UNIX-like OSes), Solaris, HP-UX, IBM AIX

Description:
SMTP benchmarking tool. It is threaded, uses very little disk I/O (e-mail body content randomly generate text). It has an SMTP source, SMTP sink and POP server load tester (to pull sent mail)
Requirement:
Linux/UNIX; requires C compiler

Description:
Pylot is a free open source tool for testing performance and scalability of web services. It runs HTTP load tests, which are useful for capacity planning, benchmarking, analysis, and system tuning. Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI.
Requirement:
Python 2.5+. required.Tested on Windows XP, Vista, Cygwin, Ubuntu, MacOS

Description:
This application calls the URL you select as many times as you choose and tells you how long it took the server to respond. It writes some additional runtime details to the PHP log file so you can optionally do more granular analysis afterwards. Although the server processes most of the statistics, all URL requests come from the browser. You can run as many browsers and workstations simultaneously as you want.
Requirement:
PHP/JavaScript

Description:
Seagull is a multi-protocol traffic generator test tool. Primary aimed at IMS protocols, Seagull is a powerful traffic generator for functional, load, endurance, stress and performance tests for almost any kind of protocol. Currently supports Diameter, XCAP over HTTP, TCAP (GSM Camel, MAP, Win) protocols.
Requirement:
Linux/Unix/Win32-Cygwin

Description:
SIEGE is an http regression testing and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the internet. It lets the user hit a webserver with a configurable number of concurrent simulated users. Those users place the webserver "under siege." SCOUT surveys a webserver and prepares the urls.txt file for a siege. In order to perform regression testing, siege loads URLs from a file and runs through them sequentially or randomly. Scout makes the process of populating that file easier. You should send out the scout, before you lay siege.
Requirement:
GNU/Linux, AIX, BSD, HP-UX and Solaris.

Description:
SIPp is a performance testing tool for the SIP protocol. Its main features are basic SIPStone scenarios, TCP/UDP transport, customizable (xml based) scenarios, dynamic adjustement of call-rate and a comprehensive set of real-time statistics. It can also generate media (RTP) traffic for audio and video calls.
Requirement:
Linux/Unix/Win32-Cygwin

Description:
SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing and performance analysis of network-based applications.
Requirement:
Any system with Java 1.4 or higher

Description:
Network benchmark application which can put your network under load and conduct automatic benchmark and recording activities.
Requirement:
OS Independent

Description:
General-purpose stress test tool.
Requirement:
Windows NT/2000, Linux

Description:
TestMaker from PushToTest.com delivers a rich environment for building and running intelligent test agents that test Web-enabled applications for scalability, functionality, and performance. It comes with a friendly graphical user environment, an object-oriented scripting language (Jython) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write scripts for you, a library of fully-functional sample test agents, and shell scripts to run test agents from the command line and from unit test utilities.
Requirement:
Windows, Linux, Solaris, and Macintosh

Description:
The purpose with TPTEST is to allow users to measure the speed of their Internet connection in a simple way. TPTEST measures the throughput speed to and from various reference servers on the Internet. The use of TPTEST may help increase the consumer/end user knowledge of how Internet services work.
Requirement:
MacOS/Carbon and Win32

Description:
Tsung is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP and Jabber servers (SSL is supported). It simulates complex user's behaviour using an XML description file, reports many measurements in real time (including response times, CPU and memory usage from servers, customized transactions, etc.). HTML reports (with graphics) can be generated during the load. For HTTP, it supports 1.0 and 1.1, has a proxy mode to record sessions, supports GET and POST methods, Cookies, and Basic WWW-authentication. It has already been used to simulate thousands of virtual users.
Requirement:
Tested on Linux, but should work on MacOSX and Windows.

Description:
Valgrind is an award-winning suite of tools for debugging and profiling Linux programs. With the tools that come with Valgrind, you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling, to speed up and reduce memory use of your programs.
Requirement:
Linux

Description:
LoadSim is a web application load simulator. It allows you to create simulations and have those simulations run against your webserver.
Requirement:
JDK 1.3 or above

Description:
Benchmarking tool for caching proxies, origin server accelerators, L4/7 switches, content filters, and other Web intermediaries.
Requirement:
C++ compiler

Description:
WebLOAD Open Source is a fully functional, commercial-grade performance testing product based on WebLOAD, Radview's flagship product that is already deployed at 1,600 sites. Available for free download and use, WebLOAD is a commercial-grade open source project with more than 250 engineering years of product development. Companies that require commercial support, additional productivity features and compatibility with third-party protocols have the option of purchasing WebLOAD Professional directly from RadView.
Requirement:
Windows NT/2000/XP

Tuesday, 11 February 2014

Parameter Types in LoadRunner

The types of parameters available in loadrunner VuGen are:
1) File or Table Parameter Types
i) Data Files
ii) Data Tables
2) XML Parameter Types
3) Internal Data Parameter Types
i) Date/Time
ii) Group Name,
iii)Iteration Number
iv) Load Generator Name,
v) Random Number
vi) Unique Number
vii) Vuser ID
viii) Date / Time
4) User-Defined Function Parameters
Parameter Types:
  1. File 
  2. Table 
  3.  XML 
  4.  Random Number 
  5.  Iteration Number 
  6.  Unique Number 
  7.  Load generator Name 
  8.  Date/Time 
  9. Vuser ID 
  10. User-Defined parameter.
The above are the types of parameter files we can use in Load Runner. Let's see in-depth about types.

File and Table:

These are the most widely used parameter types in the performance testing domain. Vusers take value from the data file and table.
When a file type or table type parameter is used, a .dat file is created. By default all the data files are named as <Parameter Name>.dat and are stored in the script’s directory.
When VuGen opens the data file, it displays the first 100 rows. To view all of the data, click ‘Edit with Notepad’ and view the data in ‘parameter properties’.

 XML type parameters:

XML Parameter Types are used for multiple valued data contained in an XML structure. It replaces a complex structure in which an array can be made a part of another array.
The complex structure as required in a web-service call can be replaced with a single XML parameter.
E.g., An XML parameter by the name Contact can replace a name, email id, mobile number, and address.
XML parameterization helps to have cleaner input of the data, and enables efficient parameterization of Vuser scripts. XML parameters are widely used with Web Service scripts and with SOA services.

Random Number:

Random Number replaces the parameter with a random number. While creating random type parameter, a minimum and maximum value of the range (the range, out of which the random number is planned to be selected) is required to be set.
A Random type parameter can be used -
To sample a system’s behavior within a possible range of values. E.g. To randomly view the details of 50 incidents being created in the system, one can specify the range for 50 serial incident ids (e.g., Min – 40001, Max - 40050) and emulate the situation. Randomly incident details will be viewed by Vusers.
Percentage distribution of tasks’ execution by Vusers. Random number can be set to have a range of 100. Using if loop one can achieve appropriate percentage distribution of different tasks.
Iteration Number:

Iteration number parameter type replaces the parameter name with Vuser’s current iteration number.
An iteration type parameter can be used -
To print the Vuser’s iteration number in external file as sometimes required while debugging scripts for data-issues.
To allocate a unique data input per Vuser.
Unique Number:
Unique number parameter replaces the parameter with a unique value. This value is taken from the range set by entering the start value and the size of the range.
Unique number type of parameter is used -
To enter a unique value in the data field while script execution. (E.g. setting a unique username for a Signup transaction of a website).
To check system’s behavior for all possible values of the parameter.

E.g., Executing a query for all employees, whose ID numbers range from 100 through 199, create 100 Vusers and set the start number to 100 and block size to 100.
  

LoadGenerator Name:

A LoadGenerator is the machine on which Vusers perform the steps as in the scripts. During test execution, Load Generator Name type parameter replaces the parameter with the name of the Vuser script’s load generator.
A LoadGenerator type of parameter is used -
While debugging scripts, to understand if failures exist only at a particular loadgenerator.
To segregate results and logs between different loadgenerators.

Date/Time parameter type:

A Date/time parameter replaces the parameter with a date and/or time. One can specify one’s own format of date/time representation.
Date/time parameter type is used in -
Business scenarios in which vuser wants to get specific details for specific duration of time. In the form data, the ‘from’ field-type will have value of last month’s date-time and the ‘to’ field will have date/time of current date-time.
Scenarios in which vuser has to enter a date-time value.
Scenarios in which vusers have to enter a unique string value for a field.

Vuser Id parameter type:

A Vuser ID replaces the parameter with the ID number assigned to the Vuser by the Controller during a scenario run. When you run a script from VuGen, the Vuser ID is always 1.
Vuser id parameter type is used - 
To print the vuser id in an external file for script-debugging purpose.
To segregate transaction volume based on Vuser ids.

 User defined parameter type:

A User defined parameter type is a user-defined function in which call to the function is made and the value returned by the function replaces the parameter name.
This type of parameter is used in following cases -
Random function can be called to capture one out of the many co-related values stored in an array (created by web_reg_save_param).
To obtain a value in a specific format that is not supported by other parameter types.

We can observe while recording a script , a window with some buttons will be displayed.Which contains vuser_init, action,vuser_end. These are the options which are useful for the user at the time of recording script. 

Thursday, 6 February 2014

Performance Testing: Main Types, Top Tools (Infographics)

A software testing process usually includes many stages, methods and can have different goals. Among them performance testing is one the major components. In general, software performance testing is meant to identify how the system performs under some workload and measure its quality attributes such as speed, scalability, reliability and others. Performance testing is necessary to carry out before the system goes to market, otherwise you might unexpectedly face some cases when the system runs slowly.
There are plenty of useful tools for measuring the performance of web sites, web applications or any other objects. The list of the most popular performance testing tools is illustrated on the infographics below.



Performance testing includes several types of tests that aim to analyze different aspects.
  1. Smoke testing
    This is a number of base tests that show if the system actually works or not. Minimal smoke tests can indicate whether all of the components function as expected, in a correct way.
  2. Load Testing
    Load tests target to find out the normal performance metrics under the expected load. They are carried out, for instance, by setting some expected number of users making some transactions. Load testing refers to a non-functional type and reveals the response time for all the main procedures. When making changes in the code, load test can help to measure whether this improved the system performance or hurt it.
  3. Stress Testing
    Stress tests aim to figure out the behavior of the system under the extreme load and identify the high end of the system’s capacity. Constantly increasing the load, you will see the limits of good performance under a heavy load. When you reach the threshold you can see what components of the system start lagging and fail first. In order to increase the system’s robustness beyond normal conditions you can make these components more efficient.
  4. Soak Testing
    Soak tests (they are also called endurance tests) focus on long-time period. They determine whether the system can work under a continuous load. During soak tests it is important to pay attention to such an issue as the performance degradation which means that the response time after a long period would be worse than in the beginning.
  5. Spike testing
    Spike tests are carried out to analyze the system behavior in conditions of sudden load increase. The goal is to figure out if such short-time huge changes can hurt the system or will be successfully handled.
  6. Component Testing
    Component tests aim to check the performance of any separate component of the system such as a search function, a picture upload, online chat etc.
  7. Configuration testing
    This is an essential test for the dynamic system. Changes made to the system configuration always need to be tested for understanding the effect they cause to the performance and behavior.
Accomplishing all types of tests does not automatically mean than your project is perfectly tested. A performance tester should always remember that testing is rather a continuous and repeating process, than a one-time event. Therefore, you should include testing into every stage of development.

Tools For Performance Modeling

So What is Application Performance Modeling - Application Performance Modelling is the science of using modeling techniques to forecast application performance. Performance Modelling could be performed with several different objectives in mind, this could include, validation of the solution approach or solution architecture, validation of infrastructure requirements for a given solution approach and possibly impact analysis for change in business workload volumes. Application Performance Modeling can be performed using a combination of statistical, analytical and simulation modeling techniques. Unfortunately there aren’t any standard commercial or Open Source tools out there for performance modeling.
You can learn more about Performance Modelling at - Performance Modelling Fundamentals
Provided below is a list of frequently used tools for purposes of Performance Modelling. At this point this page includes just links to the various tool vendors or websites. Over a period of time we intend to introduce a summary of different tools, tool reviews including a short description of their strengths and weaknesses to help readers better grasp positioning of the relevant tools and frameworks within the enterprise.

VendorArea Of FocusLink
AnylogicDiscrete Event SimulationLink
JMTAnalytical ModellingLink
IBM SPSS (Statistical Modeling)Statistical ModellingLink
Minitab (Statistical Modeling)Statistical ModellingLink
PDQ (Queuing Networks)Analytical ModellingLink
R ProjectStatistical Modeling & Data VisualizationLink
StatsoftStatistical Modeling & Data VisualizationLink
SimpyDiscrete Event Simulation ModelingLink
Simul8Discrete Event Simulation ModelingLink
TIBCO SpotFire S+Statistical ModelingLink