Stress Testing¶
One of Clearwater’s biggest strengths is scalability and in order to demonstrate this, we have easy-to-use settings for running large amounts of SIP stress against a deployment. This document describes:
- Clearwater’s SIP stress nodes, what they do, and (briefly) how they work
- how to kick off your own stress test.
SIP Stress Nodes¶
A Clearwater SIP stress node is similar to any other Project Clearwater
node, except that instead of having a Debian package like bono
or
sprout
installed, it has our clearwater-sip-stress-coreonly
or
clearwater-sip-stress
Debian package installed.
What they do¶
Clearwater SIP stress nodes provide a set of SIPp scripts to run against your Sprout cluster. There are two kinds of stress available:
- Our recommended approach is to use scripts which emulate a P-CSCF and
send traffic to Sprout, stress testing the IMS Core directly. The
nodes log their success/failure to
/var/log/clearwater-sip-stress
and also print a human-readable summary of the stress (with information about percentage of failed calls, average latency, and so on). - We also have some older scripts which emulate UEs and send traffic to
Bono, stress testing the P-CSCF and the IMS Core together. The nodes
log their success/failure to
/var/log/clearwater-sipp
, but unlike the other scripts, do not print a human-readable summary of the stress.
Deploying a stress node¶
These instructions assume you’ve already installed a Clearwater deployment, either manually or through Chef.
Follow this process to bulk provision the number of subscribers you want. As a general guideline, we’d expect a small deployment (with one Sprout, Vellum and Dime, each with one full CPU core) to handle at least 30,000 subscribers.
You should also ensure that the reg_max_expires
setting is set to
1800 rather than the default of 300 - see our configuration
guide for
instructions. This matches the assumptions in our load profile of 2
re-registers per hour.
Chef¶
If you’re using Chef to automate your Clearwater install, create your
stress test node by typing
knife box create -V -E ENVIRONMENT sipp --index 1
.
Manual install¶
Otherwise, follow these steps to deploy a stress node manually:
- create a new virtual machine and bootstrap it by configuring access to the Project Clearwater Debian repository.
- set the following property in
/etc/clearwater/local_config
:local_ip
- the local IP address of this node
- for our recommended scripts which send stress to Sprout, run
sudo apt-get install clearwater-sip-stress-coreonly
to install the Debian packages. - for our older scripts which send stress to Bono, run
sudo apt-get install clearwater-sip-stress
to install the Debian packages.
The stress node needs to be able to open connections to Sprout on TCP ports 5052 and 5054, and Sprout needs to be able to open connections to the stress node on TCP port 5082.
Running stress (IMS core only)¶
To kick off a stress run, simply run:
/usr/share/clearwater/bin/run_stress <home_domain> <number of subscribers> <duration in minutes>
.
The script will then:
- set up the stress test by sending an initial register for all the subscribers
- report that that initial registration has completed and that the stress test is starting
- send traffic, using a fixed load profile of 1.3 calls/hour for each subscriber (split equally between incoming and outgoing calls) and 2 re-registrations per hour
- after the given duration, wait for all calls to finish and then exit
- print summary output about how the test went
The output will be in this format:
Starting initial registration, will take 6 seconds
Initial registration succeeded
Starting test
Test complete
Elapsed time: 00:15:07
Start: 2016-11-16 15:03:29.062755
End: 2016-11-16 15:18:36.414653
Total calls: 1625
Successful calls: 1612 (99.2%)
Failed calls: 13 (0.8%)
Retransmissions: 0
Average time from INVITE to 180 Ringing: 74.0 ms
# of calls with 0-2ms from INVITE to 180 Ringing: 0 (0.0%)
# of calls with 2-20ms from INVITE to 180 Ringing: 0 (0.0%)
# of calls with 20-200ms from INVITE to 180 Ringing: 1572 (96.7384615385%)
# of calls with 200-2000ms from INVITE to 180 Ringing: 45 (2.76923076923%)
# of calls with 2000+ms from INVITE to 180 Ringing: 0 (0.0%)
Total re-REGISTERs: 1800
Successful re-REGISTERs: 1792 (99.5555555556%)
Failed re-REGISTERS: 8 (0.444444444444%)
REGISTER retransmissions: 0
Average time from REGISTER to 200 OK: 15.0 ms
Log files in /var/log/clearwater-sip-stress
The summary statistics at the end just come from SIPp, not the Clearwater deployment. If you want to see our more detailed statistics, you’ll need to be running a separate monitoring tool such as Cacti (and we provide the start and end time of the stress run, to let you match up with these external graphs).
Extra run_stress options¶
The run-stress script has some command-line options:
--initial-reg-rate
- this controls how many REGISTERs/second the script sends in during the initial registration phase (defaulting to 80). On systems that can cope with the load, raising this value will let the test run start faster.--sipp-output
- By default, the script hides the SIPp output and just presents the end-of-run stats. With this option, it will show the SIPp output screen, which may be useful for users familiar with SIPp.--icscf-target TARGET
- Domain/IP and port to target registration stress at. Default issprout.{domain}:5052
.--scscf-target TARGET
- Domain/IP and port to target call stress at. Default issprout.{domain}:5054
.
Running stress (IMS core and P-CSCF)¶
In this mode, each SIP stress node picks a single bono to generate traffic against. This bono is chosen by matching the bono node’s index against the SIP stress node’s index.
This test includes two important scripts.
/usr/share/clearwater/infrastructure/scripts/sip-stress
, which generates a/usr/share/clearwater/sip-stress/users.csv.1
file containing the list of all subscribers we should be targeting - these are calculated from properties in/etc/clearwater/shared_config
./etc/init.d/clearwater-sip-stress
, which runs/usr/share/clearwater/bin/sip-stress
, which in turn runs SIPp specifying/usr/share/clearwater/sip-stress/sip-stress.xml
as its test script. This test script simulates a pair of subscribers registering every 5 minutes and then making a call every 30 minutes.
The stress test logs to
/var/log/clearwater-sip-stress/sipp.<index>.out
.
There is some extra configuration needed in this mode, so you should:
- set the following properties in
shared_config
:- (required)
home_domain
- the home domain of the deployment under test - (optional)
bono_servers
- a list of bono servers in this deployment - (optional)
stress_target
- the target host (defaults to thenode_idx
-th entry inbono_servers
or, if there are nobono_servers
, defaults tohome_domain
) - (optional)
base
- the base directory number (defaults to 2010000000) - (optional)
count
- the number of subscribers to run on this node (must be even, defaults to 30000)
- (required)
- optionally, set the following property in
/etc/clearwater/local_config
:- node_idx - the node index (defaults to 1)
To apply this config and start stress, run
sudo /usr/share/clearwater/infrastructure/scripts/sip-stress
and
sudo service clearwater-sip-stress restart
.