NOTE: This web site is for the old version of perfSONAR. Please see www.perfsonar.net for information on the current release.

perfSONAR-PS is an open source development effort to create a colletion of easy-to-use and easy-to-install perfSONAR network performance monitoring services and tools.

perfSONAR 3.4 Released

The perfSONAR Toolkit, version 3.4, is now available. This release includes a number or changes:

  • A new measurement archive that stores all types of active measurement data. It boasts a RESTful interface and significant performance improvements over previous versions
  • A new full install distribution, similar to the NetInstall, but all installed packages are included on the downloaded ISO. This use case facilitates installing when networking is minimal or not available.
  • New graphs that can display results from throughput (bwctl+iperf3/iperf) and latency (owamp and ping) tools on the same graph
  • First steps toward unifying the projects formerly known as perfSONAR-MDM and perfSONAR-PS into one development effort with the inclusion of the OPPD Measurement Point
  • A number of security enhancements including the enabling of automatic updates by default for all new and existing installations
  • Updated documentation at http://docs.perfsonar.net

A detailed treatment of all changes can be found here: http://www.perfsonar.net/release-notes/version-3-4/. To install this software, please review the following options:

  1. Brand new installations may visit http://software.internet2.edu/pS-Performance_Toolkit/ to get the latest ISO images. Please see http://docs.perfsonar.net/install_getting.html for more information on choosing the right distribution and pointers to detailed installation instructions
  2. For NetInstall users updating from version 3.3.X of the Toolkit you should run "yum update". Note that all your data will automatically be migrated to the new measurement archive in a background process. See http://docs.perfsonar.net/manage_update.html#special-upgrade-notes for more details
  3. For NetInstall users updating from version 3.4rc2 or 3.4rc3, run the following:
      yum update
      yum clean all
      yum update
    
  4. For 3.3.X LiveCD/LiveUSB users, note that we are not providing a 3.4 LiveCD/LiveUSB. It is recommended all users transition to one of our other installation types prior to the previously shared EOL date of April 10, 2015. For instructions on how to do so see http://docs.perfsonar.net/manage_update.html#migrating-from-a-3-3-2-livecd-liveusb
  5. Upgrading from versions of perfSONAR older than 3.3 is NOT supported in this release.

IMPORTANT NOTE ABOUT AUTOMATIC UPDATES:

Version 3.4 of the perfSONAR Toolkit has enabled automatic updates via the yum system. This behavior is turned on by default, and can be disabled by the operator if they choose to do so. There are positive and negative connotations to automatic updates:

  • Positive Connotations:
    • When you install a brand new toolkit or update from a previous version of the Toolkit, automatic updates will be enabled by default.
    • Automatic updates will assist in getting the latest packages in a timely fashion, without requiring someone to physically access the machine.
    • All the packages on the system will be updated on a nightly-basis.
  • Negative Connotations:
    • Maintaining/securing the machine is something that can be ignored, every site is still ultimately responsible for ensuring things are updated. Orphaned machines can become a serious liability.
    • Automatic updates may download software that impacts configuration and operation of a running machine (from perfSONAR packages - or those upstream).
    • As configured, updates will be applied once per day. If a critical package becomes available after this cycle, it will not be installed until the following day. Stay connected to the perfSONAR project via the user and announce mailing lists to be aware of critical updates.
    • Some updates that require a reboot will still need human intervention.

The perfSONAR team does not get early access to third-party packages prior to their release, nor are we perfect when it comes to our own packages. Please see http://docs.perfsonar.net/manage_update.html#automatic-updates for more details on making this decision and how to enable/disable automatic updates.

perfSONAR Toolkit LiveCD and Live USB EOL Announcement

The perfSONAR project will be EOLing the LiveCD and LiveUSB products. This product will not be build for the 3.4 release, and the 3.3 version will be supported until April 10th, 2015. During this period of time the perfSONAR Project will be generating new products when a serious vulnerability is found, or when a critical kernel vulnerability is announced. We will continue the availability of new builds via the user and announce lists when they are available.

Those sites that are using the LiveCD/LiveUSB can upgrade using this documentation: http://docs.perfsonar.net/manage_update.html#migrating-from-a-3-3-2-livecd-liveusb

Version 3.3 of perfSONAR EOL Announcement

perfSONAR version 3.3 will be EOLed on April 10th, 2015. All users must upgrade to version 3.4 by this date, or risk using unsupported software. For individuals who would like to have access to the historic RPMs, the project has created a "vault" containing last builds of project software from prior versions:

http://software.internet2.edu/vault/

The Internet2 repository software (versions 0.4-2 and above for version 3.3 users) contains directives to enable this feature. The repository package can be found in these locations, or will be updated for existing installation via YUM:

http://software.internet2.edu/rpms/el6/i386/main/RPMS/Internet2-repo-0.4-2.noarch.rpm
http://software.internet2.edu/rpms/el6/x86_64/main/RPMS/Internet2-repo-0.4-2.noarch.rpm

Once enabled, the vault will allow existing toolkit builds to access the old RPMs. However, it will not allow someone to install a fresh instance from a netinstall ISO. To enable this repo:

a) Edit the "/etc/yum.repos.d/Internet2-Vault.repo" file and change "enabled = 0" to "enabled = 1" for both repositories listed in this file

b) Run 'yum update' to update the cache of packages

Shellshock Vulnerability

The perfSONAR Toolkit, and all systems that feature a vulnerable version of the bash application, are susceptible to the risks of the recently discovered shellshock vulnerability.  More information from the upstream operating system vendor on this vulnerability can be found here.

Packages to address the bash vulnerability are available from the upstream CentOS repositories as of September 26th, 2014.  Additional mitigation from the perfSONAR project to prevent software from exposing the bash vulnerability has also been released.  This mitigation involved searching for libraries and applications that invoked bash variables - our search found this to be limited to CGI scripts served through the Apache web server.  Tools like OWAMP/BWCTL and others were not exposed to the bash risk. 

We recommend that perfSONAR operators take the following steps immediately:

  • If you are running the netinstall (version 3.3 or 3.4), please run 'yum update' to pull in the latest packages, and reboot the system
  • If you are running a LiveCD, please download the latest release (version 3.3.2-10)
  • The following additional steps, to limit external access to the GUI, may also be used for any of the use cases:
    • Open the configuration file: /etc/httpd/conf.d/apache-toolkit_web_gui.conf
    • Search for instances of these permissions:

      Order allow,deny
      Allow from all
    • Modify the permissions to look like this (replacing the obvious fake address with a real one, multiple 'Allow' lines are permitted):

       Order deny,allow
       Allow from AAA.BBB.CCC.DDD/16
       Deny from All
    • restart httpd (sudo /etc/init.d/httpd restart) or reboot

For those that have not done so, please consider configuring yum to automatically update the system.  Instructions on how to do this can be found here.

Please subscribe to the perfSONAR  user or announcement mailing lists for additional information as it becomes available.  Patches from the upstream vendors may become available as more is understood about the impact of this vulnerability. 

Heartbleed

On April 7th 2014, the perfSONAR project received notice from the CentOS of a CVE related to the OpenSSL package: https://rhn.redhat.com/errata/RHSA-2014-0376.html

This alert was triggered by the recent exploit entitled "heartbleed", which targets TLS activity. perfSONAR is recommending that all users run yum update to download the latest packages from the CentOS repositories. Additionally, we recommend re-generation of the host's certificate information after the packages are updated, as it was generated with the vulnerable implementation of OpenSSL. The following steps will accomplish this goal:

  • # build a new key
    /usr/bin/openssl genrsa > /etc/pki/tls/private/localhost.key
    
  • # create a new self-signed cert
    cd /etc/pki/tls/certs
    make testcert
    
  • # restart apache
    sudo /etc/init.d/httpd restart
    

If you have installed a customized certificate, refer to instructions specific to that process.

New versions (3.3.2-2) of the LiveCD and LiveUSB product have been generated, and are available here.

NTP Amplification

Some of you may be aware of a recently highlighted risk of NTP amplification (http://www.symantec.com/connect/blogs/hackers-spend-christmas-break-launching-large-scale-ntp-reflection-attacks), and in some cases others may have already had a host that was used in an attack. A sample notice received may look something like this:

You are running a public NTP server that participated a very large-scale attack against a customer of ours today, generating UDP responses to spoofed requests with bogus timestamps that claimed to be from the attack target. Your server was particularly active in the attack, sending a significant percentage of the attack traffic we saw.

Please consider reconfiguring your NTP server in one of these ways:

  • To only serve your customers and not respond to outside IP addresses. If your NTP server runs as a standalone installation, setting the service to ignore all queries would work well for this. With ntpd, that can be done with "restrict default ignore" in /etc/ntp.conf; other servers should have a similar configuration option. A firewall rule to block UDP to the public IP address on port 123 would also work for this.
  • To rate-limit responses to individual source IP addresses
  • To limit queries to TCP-only
  • To ignore particularly unlikely queries, such as those representing dates far in the future or past
  • To limit the size of allowed responses

Starting with the 3.3.2 release of the pS Performance Toolkit, configuration changes have been implemented to correct the risk of NTP amplification. We highly recommend updating hosts to this version to mitigate the risk of being a part of future protocol abuses. For those that have not updated to the latest version, it is possible to make manual modifications to the local /'etc/ntp.conf' file to accomplish the same goal:

# by default act only as a basic NTP client
restrict -4 default nomodify nopeer noquery notrap
restrict -6 default nomodify nopeer noquery notrap
# allow NTP messages from the loopback address, useful for debugging
restrict 127.0.0.1
restrict ::1

Additional restrict lines can be added to allow trusted subnets, e.g.:

restrict a.b.c.d mask 255.255.0.0 nomodify notrap nopeer

Any changes to NTP should be followed by a restart of the service. More information on protection for hosts (and routing devices) can be found here:

http://www.team-cymru.org/ReadingRoom/Templates/secure-ntp-template.html

pS-Performance Toolkit Recommendations for Use with Firewalls

The pS Performance Toolkit development team, working closely with the NTAC Performance Working Group, has produced a comprehensive guide to configuring site and host firewalls for use with the measurement tools. This information is provided because:

  • Well known ports will make it easier for sites that are using firewalls to specify the requirements of measurement on the host itself, and all network devices in the middle
  • Sites that are not using firewalls will be able to test to firewalled sites (as well as firewalled sites testing to them) and be guaranteed of success. Previous attempts in this space could have involved sites choosing differen$ 'ranges' for their tools, and the tools did not effectively negotiate on a strategy that would lead to working measurement.

More information is available here.