pS-Performance Toolkit

The perfSONAR Performance Toolkit (pS-Performance Toolkit) is a customized distribution of the CentOS operating system. The software consists of an ISO disk image that can be used in one of two ways: burn the image to a disk (recommended); or mount the ISO file system and run as a virtual machine (for example, under VMware, QEMU or xen). After performing either step, the result is a fully-configured system capable of running and using many advanced network tools and methodologies, serving either as a testing point or a means to solve performance issues.

Heartbleed

On April 7th 2014, the perfSONAR project received notice from the CentOS of a CVE related to the OpenSSL package: https://rhn.redhat.com/errata/RHSA-2014-0376.html

This alert was triggered by the recent exploit entitled "heartbleed", which targets TLS activity. perfSONAR is recommending that all users run yum update to download the latest packages from the CentOS repositories. Additionally, we recommend re-generation of the host's certificate information after the packages are updated, as it was generated with the vulnerable implementation of OpenSSL. The following steps will accomplish this goal:

  • # build a new key
    /usr/bin/openssl genrsa > /etc/pki/tls/private/localhost.key
    
  • # create a new self-signed cert
    cd /etc/pki/tls/certs
    make testcert
    
  • # restart apache
    sudo /etc/init.d/httpd restart
    

If you have installed a customized certificate, refer to instructions specific to that process.

New versions (3.3.2-2) of the LiveCD and LiveUSB product have been generated, and are available here.

NTP Amplification

Some of you may be aware of a recently highlighted risk of NTP amplification (http://www.symantec.com/connect/blogs/hackers-spend-christmas-break-launching-large-scale-ntp-reflection-attacks), and in some cases others may have already had a host that was used in an attack. A sample notice received may look something like this:

You are running a public NTP server that participated a very large-scale attack against a customer of ours today, generating UDP responses to spoofed requests with bogus timestamps that claimed to be from the attack target. Your server was particularly active in the attack, sending a significant percentage of the attack traffic we saw.

Please consider reconfiguring your NTP server in one of these ways:

  • To only serve your customers and not respond to outside IP addresses. If your NTP server runs as a standalone installation, setting the service to ignore all queries would work well for this. With ntpd, that can be done with "restrict default ignore" in /etc/ntp.conf; other servers should have a similar configuration option. A firewall rule to block UDP to the public IP address on port 123 would also work for this.
  • To rate-limit responses to individual source IP addresses
  • To limit queries to TCP-only
  • To ignore particularly unlikely queries, such as those representing dates far in the future or past
  • To limit the size of allowed responses

Starting with the 3.3.2 release of the pS Performance Toolkit, configuration changes have been implemented to correct the risk of NTP amplification. We highly recommend updating hosts to this version to mitigate the risk of being a part of future protocol abuses. For those that have not updated to the latest version, it is possible to make manual modifications to the local /'etc/ntp.conf' file to accomplish the same goal:

# by default act only as a basic NTP client
restrict -4 default nomodify nopeer noquery notrap
restrict -6 default nomodify nopeer noquery notrap
# allow NTP messages from the loopback address, useful for debugging
restrict 127.0.0.1
restrict ::1

Additional restrict lines can be added to allow trusted subnets, e.g.:

restrict a.b.c.d mask 255.255.0.0 nomodify notrap nopeer

Any changes to NTP should be followed by a restart of the service. More information on protection for hosts (and routing devices) can be found here:

http://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html

pS-Performance Toolkit 3.3.2 released

The final version of pS-Performance Toolkit 3.3.2 is now available for download. This release mostly contains bug fixes and minor enhancements. Release notes are available detailing changes.


pS-Performance Toolkit Recommendations for Use with Firewalls

The pS Performance Toolkit development team, working closely with the NTAC Performance Working Group, has produced a comprehensive guide to configuring site and host firewalls for use with the measurement tools. This information is provided because:

  • Well known ports will make it easier for sites that are using firewalls to specify the requirements of measurement on the host itself, and all network devices in the middle
  • Sites that are not using firewalls will be able to test to firewalled sites (as well as firewalled sites testing to them) and be guaranteed of success. Previous attempts in this space could have involved sites choosing different 'ranges' for their tools, and the tools did not effectively negotiate on a strategy that would lead to working measurement.

More information is available here.

purpose

When serving as a measurement point, a use case adopted by Large Hadron Collider (LHC) affiliated Tier 2 sites in the United States, the disk becomes a part of a global registry of perfSONAR tools. Affiliated sites and other interested parties may perform any number of tests (on demand, or in some cases regularly scheduled) to the installation including:

When used in the role of a network debugging infrastructure, the various tools can be used to methodically find, diagnose, and aid in the correction of network performance issues. Additional supporting components include:

  • Apache2 - Web server with supporting modules and extensions
  • MySQL - Relational database management system
  • NTP - Network Time Protocol

installation and disk contents

Please visit pS-Performance Toolkit QuickStart wiki for a complete set of instructions regarding installation and configuration. Click here for a complete list and explanation of the Disk Contents.

background

The original idea for the pS-Performance Toolkit came from the Network Performance Workshop series sponsored by Internet2. The old format of this workshop consisted of a day of tool installation (e.g. installing the Web100-enhanced kernel and other performance tools) followed by some time to test and use them. This particular format did not allow enough time to get to the primary reason many participants choose to sign up: solving network performance problems.

To lower the bar on this complex topic, this product offers a complete network performance suite pre-installed (and in many cases pre-configured) that is ready to be dropped directly into a network operations center (NOC) of any size. With the integration of perfSONAR tools, the results of both passive and active measurements can be shared in a federated environment with your peers enabling the early detection of performance issues.

“These tools give us better visibility into the network, allowing us to troubleshoot performance issues quickly.” -- Network Performance Workshop participant

mailing lists

performance-node-announce and performance-node-users, hosted at internet2.edu, are publicly archived and open to subscriptions.