pS-Performance Toolkit

FAQ (frequently asked questions)

  1. Where can I ask questions or report bugs?
  2. How do I use the NPAD command line client (diag-client)?
  3. How do I use BWCTL?
  4. How do I use the NDT command line client (web100clt)?
  5. How do I use OWAMP?
  6. Can I Use a Firewall?
  7. How many ports will BWCTL need to operate effectively behind a firewall?
  8. Where can the BWCTL port values be adjusted?
  9. Can the OWAMP test port ranges be added via the GUI?
  10. What should I enter for the Communities of interest configuration question?
  11. I do not think I am a member of a community, should I put anything?
  12. What are the hardware requirements for running the pSPT?
  13. What is the recommended configuration for the hard disk of a machine running the pSPT?
  14. Does my machine have to meet the System Requirements?
  15. The colors on my Console Configuration do not match what I see on the web. Some are green already.
  16. The "Services On This Node" screen shows many services in the non-running state when first started, what is wrong?
  17. The "Services On This Node" screen doesn't have any IP addresses or hostnames for the services. Some of the services are Not Running. What is wrong?
  18. I do not see my service in the Global Set Of Services, where is it?
  19. When looking at the data display for perfAdmin BWCTL/perfAdmin OWAMP I do not see results, but I filled out information in the Scheduled Testing area. Where is my data?
  20. What is the purpose of BWCTL Limits/OWAMP Limits?
  21. How can I set limits to prevent others from overusing BWCTL/OWAMP?
  22. How many NTP servers do I need, can I select them all?
  23. Service X is listed as Not Runnning on the web interface. How can I restart?
  24. NTPD has exited/is not running on my machine, why did this happen and how can I fix it?
  25. BWCTL seems to exit immediately after starting/restarting. Why won't it stay in a running state?
  26. Can I boot from a USB key instead of a CD?
  27. How can I upgrade an 3.2RC X NetInstall?
  28. How can I backup historical OWAMP/BWCTL data?
  29. A CVE announcement was made for the current pSPT Kernel, what do I do?
  30. After creating a user on the LiveCD and trying to log in, I see Could not chdir to home directory /home/USER: No such file or directory, what is wrong?
  31. I am trying to log in with the knoppix user, and it is not working. What is wrong?
  32. During the NetInstall, I see errors about a corrupt file being downloaded. What should I do?
  33. How do I interpret the headings of each column of the "Throughput Tests" page?
  34. How do I interpret the headings of each column of the "One-Way Latency Tests" page?
  35. If I enable SSH via 'chkconfig' it does not remain across reboots, what is wrong?
  36. How do I change the MTU for a device?
  37. How do I change the SSL certificate used by the web server?
  38. I forgot to enable IPv6 in CentOS when I installed the toolkit. How do I enable it?
  39. Why is the static IPv6 address I assigned during the net-install process not configured when my host starts-up?
  40. How do I fix my certificate when my browser displays "Error code: sec_error_reused_issuer_and_serial" when I try to access the toolkit web page?
  41. I'm seeing "tg3 0000:02:01.0: tg3_stop_block timed out, ofs=4800 enable_bit=2" errors on bootup. What should I do?
  42. My pSPT has an RFC 1918 (Private Internet) address, and I am not able to configure regulare tests. Is there a way around this?
  43. Version 3.2.2 of the perfSONAR-PS Toolkit LiveCD version won't boot. What should I do?
  44. Can I run owamp and bwctl on the same host?
  45. The drivers that are included for my "foo" network card old, what should I do?
  46. I setup a server from the latest ISO, but when I do a NDT test to the server, nothing happens. Is there a problem with the NDT tool?
  47. [BWCTL and OWAMP] Graphs fail to open on my browser. I am using IE 8/9. Is this a problem with my toolkit or my browser?
  48. What can I do in the case of a catastrophic hardware failure?
  49. Where can I find more resources regarding timekeeping for VMWare Virtual Machines?
  50. What TCP congestion control algorithm is used by the perfSONAR Toolkit?
  51. My host is only connected on R&E routed networks, and I have no ability to access commercial web sites that host packages for the pS Performance Toolkit. Yum is failing as a result, what steps can I take to correct this?
  52. My (OWAMP|BWCTL) measurements have stopped, and I notice mysql errors in the logs. What should I do?
  53. I have a limited amount of disk space in /var/log, and I have noticed that perfsonarbuoy_ma is writing a lot of INFO entries in the /var/log/perfsonar/perfsonarbuoy_ma.log which is filling up this disk space rapidly. Is there a way to suppress all INFO level logs?


Q:Where can I ask questions or report bugs?
A:

For questions, send email to performance-node-users at internet2 dot edu. You may also join the mailing list by visiting https://lists.internet2.edu/sympa/info/performance-node-users. For bugs, report at http://code.google.com/p/perfsonar-ps/issues/list. State in the bug description that the issue involves the pS-Performance Toolkit.




Q:How do I use the NPAD command line client (diag-client)?
A:

The diag-client is a command line version of the NPAD diagnostic client. Instead of needing a web browser, this client runs the tests from a terminal window. The basic options are to provide a server name/address and the connection IP. The NPAD server has two ports open:

  • 8000 for HTTP traffic
  • 8001-8020 for measurement traffic

Please connect to 8100:

[knoppix@Knoppix ~]$ diag-client HOSTNAME 8100
Using: rtt 10 ms and rate 20
Connected.
Control connection established.
port = 8003
Starting test.
Parameters based on 107 ms initial RTT
peakwin=27617 minpackets=3 maxpackets=83 stepsize=8
Target run length is 608 packets (or a loss rate of 0.16447368%)
Test 1a (11 seconds): Coarse Scan
Test 1b (11 seconds): ...
...

Connecting to the HTTP port will result in the following error:

[knoppix@Knoppix ~]$ diag-client HOSTNAME 8200
Using: rtt 10 ms and rate 20
Connected.
Protocol error: bad handshake.
Please make sure you have the latest client,  and you have the correct port number



Q:How do I use BWCTL?
A:

BWCTL (Bandwidth Test Controller) is a client/server program developed to simplify Iperf, thrulay, and nuttcp testing between hosts. At boot time, the NPTookit starts a BWCTL server process and leaves it listening on TCP port 4823. This server may then be accessed by remote BWCTL clients. Additionally, the disk contains BWCTL client applications that can be used to test to remote instances. The BWCTL server allows TCP tests with a maximum duration of 60 seconds. To run a test to a remote BWCTL server:

  • Logon to the pS-Performance Toolkit server using the root user, or other valid userid
  • Identify the remote server
  • Run bwctl -s remote-bwctl-server command to stream data for 10 seconds from the remote BWCTL server to the local instance.
  • Run bwctl -c remote-bwctl-server command to stream data for 10 seconds from the locally instance to the remote BWCTL server.
  • Results are displayed on our console or terminal window.

A more detailed document describing a debugging exercise using BWCTL can be seen here: DOCX or PDF




Q:How do I use the NDT command line client (web100clt)?
A:

The web100clt is a command line version of the NDT diagnostic client. Instead of needing a web browser, this client runs the tests from a terminal window. The basic options are to provide a server name/address and the connection IP. The NPAD server has two ports open:

  • 7123 for HTTP traffic
  • 3001, 3002, 3003 for measurement traffic

Please connect to 3001, 3002, or 3003:

[knoppix@Knoppix ~]$ web100clt -n HOSTNAME -p 3001
Testing network path for configuration and performance problems  --  Using IPv4 address
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
running 10s outbound test (client to server) . . . . .  164.00 kb/s
running 10s inbound test (server to client) . . . . . . 13.40 Mb/s
The slowest link in the end-to-end path is a a 622 Mbps OC-12 subnet
Information [C2S]: Packet queuing detected: 16.95% (local buffers)
Information [S2C]: Packet queuing detected: 67.10% (local buffers)
Server '128.193.128.237' is not behind a firewall. [Connection to the ephemeral port was successful]
Client is not behind a firewall. [Connection to the ephemeral port was successful]
Packet size is preserved End-to-End
Server IP addresses are preserved End-to-End
Client IP addresses are preserved End-to-End

Connecting to the HTTP port (or other ports) will result in the following error:

[knoppix@Knoppix ~]$ web100clt -n HOSTNAME -p 7123
Testing network path for configuration and performance problems  --  Using IPv4 address
Information: The server 'HOSTNAME' does not support this command line client



Q:How do I use OWAMP?
A:

OWAMP (One-Way Ping) is a client server program that was developed to provide delay and jitter measurements between two target computers. At boot time, the pS-Performance Toolkit starts an OWAMP server process and leaves it listening on TCP port 861. This server may then be used by remote clients. Additionally, the disk contains OWAMP client applications that can be used to test to remote instances (including a Java client and a console based application). By default, the OWAMP server sends a low-level data stream in each direction and measures the one-way delay and jitter between the two hosts. Separate measurements, one for each direction, are reported to the user at the end of the test. To run a test to a remote OWAMP server:

  • Logon to the pS-Performance Toolkit server using the knoppix or other valid userid.
  • Identify the remote server.
  • Run the owping remote-owamp-server command to make a pair of 10 second delay measurements (one in each direction) between remote OWAMP server and the local instance.
  • Results are displayed on the console or terminal window.



Q:Can I Use a Firewall?
A:

Firewalls are used to impart security into a network by controlling incoming and outgoing traffic based on header information. E.g. Firewall rules can limit traffic to specific subnets, IPs, ports, and protocols. Firewalls inspect packets, and thus impart a delay in the delivery of traffic end to end. This delay can introduce jitter into a flow (e.g. disruption of the variation of arrival times of packets), or potentially drop packets completely if memory is exhausted due to the amount of processing for a given set of flows. For these reasons it is recommended that the pSPT device be allowed to test outside of the firewall environment.

If the pSPT device must be behind a firewall, a set of ports must be opened for incoming and outgoing traffic. Even with a set of ports identified for traffic, note that outbound filtering (a common technique used in modern firewalls) is not advised unless the peers you are testing with are using the exact same set of port ranges. If this is not the case, failures may result when the non-local end of a test chooses a port that is not in the local end's expected range. Note that the following port recomendations are only enforced on the local end, and the tools make no attempt to negotiate via protocol with the remote end of a test:

Service Protocol Local Ports Direction
SNMP MA TCP 8065 Incoming
9990
Traceroute MA TCP (control), TCP/UDP/ICMP (measurement) 8086 Incoming/Outgoing
8087
PingER TCP (control), ICMP (measurement) 8075 incoming (control), outgoing (measurement)
perfSONAR-BUOY TCP 8085 Incoming
8569 Incoming/Outgoing
8570
Lookup Service TCP 8090 Incoming/Outgoing
8095
9995
BWCTL TCP 4823 Incoming/Outgoing
TCP/UDP Edit /etc/bwctld.conf, set peer_port to a range of values, open the tcp port for these values. It is recommended that 6001-6200 be used for this variable.
TCP/UDP Edit /etc/bwctld.conf, set iperf_port, thrulay_port, and nuttcp_port to a specific range, and open the tcp/udp ports for those ranges. It is recommended that 5001-5600 be split between these three variables (200 each). In the "Scheduled Tests" screen, click the "Configure BWCTL Test Ports Range" and use the same range of "peer_port" values as in the first step.
OWAMP TCP 861 Incoming/Outgoing
UDP Edit /etc/owampd.conf, set testports to a range of values, open the tcp port for these values. It is recommended that 8760-8960 be used for this variable. In the "Scheduled Tests" screen, click the "Configure OWAMP Test Ports Range" and use the same range of ports as in the previous step.
NDT TCP 3001-3003 Incoming/Outgoing
7123 Incoming
NPAD TCP 8000 Incoming
8001-8020 Incoming/Outgoing
Apache TCP 80 Incoming
443
SSH (Optional) TCP 22 Incoming
NTP UDP 123 Outgoing
DNS UDP 53 Outgoing
Ping ICMP - Incoming/Outgoing
Echo TCP 7 Incoming
Ping ICMP 33434-33534 Incoming

Even with this list of available ports, there is not guarantee that certain measurements (e.g. BWCTL and OWAMP) will succeed 100% of the time, namely because the choice of ports (or the decision to limit them to a certain range) is a site by site decision. Most tools pick a specific port range, and that is the only range that will work. These test tools wanted to provide the flexibility of allowing local sites to pick the local port ranges they use. To get this flexibility for what is accepted for local port selection, the trade-off is that you can not limit the destination port you send to. However, you always have control of the local-port selection. So, inbound traffic can be limited based on the destination port, and outbound traffic can be limited based on the source port.

In general, if a site is blocking outgoing traffic based on destination port, problems will result in using the BWCTL and OWAMP tools specifically since there is no way to limit the port choices of the remote sites. Likewise, if a site blocks traffic in either direction based on source port a similar situation will result.

Outbound filtering is therefore not recommended, and a policy to allow all outgoing traffic is a better choice for the measurement infrastructure. As for inbound filtering, sites may open up those destination ports that you are actually used as service ports, but all source ports have to be allowed as they are chosen by the other side.

Additional information can be found here: http://fasterdata.es.net/performance-testing/perfsonar/ps-howto/perfsonar-firewall-requirements/




Q:How many ports will BWCTL need to operate effectively behind a firewall?
A:

The pSPT development team recommends not limiting measurement tools to certain ports; this action may cause unexpected or unpredictable behavior. For instance tools such as BWCTL have two factors to consider if the ports are limited to a small subset:

  • Ports available for the regular testing infrastructure on the machine
  • Ports available for others to test to the machine

Both situations are controlled by setting values in the same configuration file on the local machine, and it can be hard to predict how many to allow. Some simple calculations can be used to determine a baseline number of ports for the first situation and are based on the parameters of the BWCTL test. Consider the following BWCTL parameters:

  • 10 Second long BWCTL tests
  • Maximum availability of 6 slots per minute

This would imply needing a total of 6 ports open. To allow for time range rounding errors we should increase this to 7 to be safe. Ideally this would work well, but there are complications due to the nature of the Linux kernel. A general behavior of the kernel is to not release a port from a previous use for up to a minute after it may be closed. This environmental consideration therefore has an impact on the above calculation. Instead of only allowing 7 ports, we should double this number to 14 to be completely safe.




Q:Where can the BWCTL port values be adjusted?
A:

The pSPT development team recommends not limiting measurement tools to certain ports; this action may cause unexpected or unpredictable behavior. If selecting a port range is still required, BWCTL has several settings defined in the bwctld.conf file that dictate which ports it may use for testing. The configuration options are:

  • iperf_port - Port range (e.g. 5001-5020) to run the iperf receiver.
  • nuttcp_port - Port range (e.g. 5021-5040) to run the nuttcp receiver.
  • thrulay_port - Port range (e.g. 5041-5060) to run the thurlay receiver.
  • peer_port - Port range (e.g. 5061-5080) to run the server processes of the above tests.

Note that the above ranges are examples, and that calculating the appropriate number of ports based on the FAQ item above is recommended.




Q:Can the OWAMP test port ranges be added via the GUI?
A:

The pSPT development team recommends not limiting measurement tools to certain ports; this action may cause unexpected or unpredictable behavior. If selecting a port range is still required, please see this section for instructions on how to alter the testing port range.




Q:What should I enter for the Communities of interest configuration question?
A:

This question can be confusing to answer for new users. The goal is to associate some loosely coupled labels to the data that the pS-Performance Toolkit disk will be making available to the larger world. Think of this step similar to assigning labels to photos or music. Some examples of valid answers are:

  • Internet2 - The data made available somehow connects the Internet2 backbone
  • LHC (CMS, ATLAS, etc.) - The disk is part of the LHC deployment structure
  • eVLBI - The disk is a part of the larger telescope community
  • MAX - A connector of member of the MAX gigapop
  • DOE-SC-LAB - US Department of Energy Office of Science Labs

Use as many community names as necessary to properly categorize the data from the installation.




Q:I do not think I am a member of a community, should I put anything?
A:

Communities are not required, but they allow other individuals and organizations to find and use your services. It is a good practice to join as many as you may think are applicable.




Q:What are the hardware requirements for running the pSPT?
A:

See this page. Note that the pSPT development team has not created hard minimum or maximum requirements - the pSPT will function on almost any form of hardware. Performance considerations do favor meeting or exceeding the minimum guidelines.




Q:What is the recommended configuration for the hard disk of a machine running the pSPT?
A:

There is only one requirement for partitioning the hard disk of a pSPT system: an ext3 partition must be available for storage. The user may configure as many partitions as they need for the pSPT disk, but for simplicity it is recommended that the entire disk be used in a single partition to allow for maximum storage resources. See also this section for hints on storage configuration.




Q:Does my machine have to meet the System Requirements?
A:

There is nothing on the pSPT that will prevent systems that do not meet the requirements from starting. Erroneous or inaccurate behavior is possible if the hardware cannot support the measurement tools.




Q:The colors on my Console Configuration do not match what I see on the web. Some are green already.
A:

If you are upgrading from a previous version of the pSPT, the colors may be green already because a particular aspect was configured previously. These do not need to be configured again.




Q:The Services On This Node screen shows many services in the non-running state when first started, what is wrong?
A:

Many of the services will be in this state because they are missing some key configuration items (e.g. from the Administrative Info). After following the configuration steps check this screen again, most should be functional.




Q:The Services On This Node screen doesn't have any IP addresses or hostnames for the services. Some of the services are Not Running. What is wrong?
A:

If the pSPT cannot grab a DHCP lease or it is not statically configured, there will be no access to the internet. Many of the services rely on knowing this information, and will therefore refuse to start unil this is corrected.

Example of a running service with a hostname:

Example of a running service, but without a hostname or ip address:

To check to be sure your installation has this information, check the /opt/perfsonar_ps/toolkit/etc/external_addresses file:

[root@localhost ~]# cat /opt/perfsonar_ps/toolkit/etc/external_addresses 
external_address=lab236.internet2.edu
default_accesspoint=lab236.internet2.edu
default_ipv4_address=lab236.internet2.edu
default_ipv6_address=2001:468:1420:3:21b:21ff:fe4e:9203



Q:I do not see my service in the Global Set Of Services, where is it?
A:

Much like DNS, the information that will populate the Global Lookup Service will take time to propagate. Please allow some time (e.g. a few hours) before your service will be fully visible.




Q:When looking at the data display for perfAdmin BWCTL /perfAdmin OWAMP I do not see results, but I filled out information in the Scheduled Testing area. Where is my data?
A:

There are several possible reasons for not seeing data:

  • Data from regular tests will take time to collect. In the case of a 4 hour testing interval, data may arrive anytime in a 4 hour window.
  • Database corruption may prevent the measurement tools from storing the results. Consult the MySQL daemon (e.g. ensure it is running and able to be accessed via the mysql -u root command
  • The measurement tools could fail to complete due to:
    • Firewalls blocking access to key ports at the source, destination, or middle connectivity of the test hosts. Attempt to run a manual test with the target tool to further explore this potential problem.
    • NTP has not fully synchronized on the source or destination host. BWCTL and OWAMP rely on stable clocks. If the clock is not stable due to system restart, lack of the NTP daemon running, or extreme clock skew (e.g. caused by using NTP servers that are not close enough to the target system), the measurement tools will fail to complete.
    • Version mismatch between source and destination systems. This occurrence is rare, and running a manual test with the target tool is the best way to further debug this problem.
    • Changes were made to the /etc/hosts file to re-direct traffic from the system's DNS name to the loopback (127.0.0.1) address. The following shows a modified version of /etc/hosts that would cause measurements to fail: 
      127.0.0.1	DNS_HOSTNAME localhost.localdomain localhost
::1		localhost6.localdomain6 localhost6
      It is suggested that /etc/hosts be similar to the following: 
      127.0.0.1	localhost.localdomain localhost
HOST_IP		DNS_HOSTNAME
::1		localhost6.localdomain6 localhost6
      The errors that may result from this modification, and will show up in the syslog output, are as follows for BWCTL: 
      bwctl: Unable to connect to HOSTNAME:4823
      And as follows for OWAMP: 
      powstream[PID]: OWPControlOpen([HOSTNAME]:861): Couldn't open 'control' connection to server: Invalid argument



Q:What is the purpose of BWCTL Limits/OWAMP Limits?
A:

These allow you to limit the influence that outside users have on your system performance. For example, to prevent your machine/network from being saturated with BWCTL tests, limit the duration and maximum bandwidth available. These screens allow a fine grained way to protect resources.




Q:How can I set limits to prevent others from overusing BWCTL/OWAMP?
A:

BWCTL and OWAMP have configurable dialog that allows the administrator to limit the resources consumed. To set the limits for BWCTL, consult this section. To set the limits for OWAMP, consult this section.




Q:How many NTP servers do I need, can I select them all?
A:

It is recommended that 4 to 5 close and active servers be used. The Select Closest Servers button will help with this decision. Note that some servers may not be available due to routing (e.g. non-R&E networks vs R&E networks - a common problem for Internet2 and ESnet servers).




Q:Service X is listed as Not Runnning on the web interface. How can I restart?
A:
  • Bandwidth Test Controller (BWCTL):
    • sudo /etc/init.d/bwctld restart
  • Lookup Service:
    • sudo /etc/init.d/lookup_service restart
  • Network Diagnostic Tester (NDT):
    • sudo /etc/init.d/ndt restart
  • Network Path and Application Diagnosis (NPAD):
    • sudo /etc/init.d/npad restart
  • One-Way Ping Service (OWAMP):
    • sudo /etc/init.d/owampd restart
  • perfSONAR-BUOY Regular Testing (Throughput):
    • sudo /etc/init.d/perfsonarbuoy_bw_collector restart
    • sudo /etc/init.d/perfsonarbuoy_bw_master restart
  • perfSONAR-BUOY Measurement Archive:
    • sudo /etc/init.d/perfsonarbuoy_ma restart
  • perfSONAR-BUOY Regular Testing (One-Way Latency):
    • sudo /etc/init.d/perfsonarbuoy_owp_collector restart
    • sudo /etc/init.d/perfsonarbuoy_owp_master restart
  • PingER Measurement Archive and Regular Tester:
    • sudo /etc/init.d/PingER restart
  • SNMP Measurement Archive:
    • sudo /etc/init.d/snmp_ma restart



Q:NTPD has exited/is not running on my machine, why did this happen and how can I fix it?
A:

NTPD may exit if the hardware clock on the host is too far off from the true time to make a difference. To skip the clock ahead to the correct time, try these commands:

[knoppix@Knoppix init.d]$ sudo /etc/init.d/ntp stop
Stopping NTP server: ntpd.
[knoppix@Knoppix init.d]$ sudo ntpdate owamp.newy.net.internet2.edu owamp.wash.net.internet2.edu
Looking for host owamp.newy.net.internet2.edu and service ntp
host found : eth-0.nms-rlat.newy32aoa.net.internet2.edu
Looking for host owamp.wash.net.internet2.edu and service ntp
host found : eth-1.nms-rlat.wash.net.internet2.edu
27 Jan 13:42:51 ntpdate[14891]: adjust time server 2001:468:9:12::16:34 offset -0.001660 sec
[knoppix@Knoppix init.d]$ sudo /etc/init.d/ntp restart
Stopping NTP server: ntpd.
Starting NTP server: ntpd.

If NTPD continues to exit on a periodic basis, there may be a hardware failure. Consult the machine's BIOS to see if there may be problems with the hardware clock or internal battery.




Q:BWCTL seems to exit immediately after starting/restarting. Why won't it stay in a running state?
A:

BWCTL relys on NTP (Network Time Protocol) to have an accurate representation of time for measurements. BWCTL (and ntpd) will simply exit if the system clock is too far from the recognized time. To check the status of the ntpd daemon:

[knoppix@Knoppix ~]$ ps axw | grep ntpd
 5146 ?        Ss     0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -u 115:121 -g

If ntpd is not running, you will get nothing back:

[knoppix@Knoppix ~]$ ps axw | grep ntpd
 5140 pts/0    R+     0:00 grep ntpd

To bring your system clock back up to date, try the following steps:

sudo /etc/init.d/ntp stop 
sudo ntpdate owamp.newy.net.internet2.edu
sudo /etc/init.d/ntp start

To check your system clock on the pSPT, try the following command (after restarting ntpd):

[knoppix@Knoppix ~]$ ntpq -p -c rv
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*chronos.es.net  .PPS.            1 u   30   64    1   25.016    7.335   0.170
+navobs1.oar.net .USNO.           1 u   29   64    1    7.299    5.607   0.065
+tick.usno.navy. .USNO.           1 u   28   64    1   40.144    7.690   2.046
-2001:468:1:12:: 130.207.244.240  2 u   27   64    1   27.212    5.913   0.052
-2001:468:2:12:: 64.57.16.34      2 u   26   64    1   30.335    5.501   0.047
assID=0 status=0644 leap_none, sync_ntp, 4 events, event_peer/strat_chg,
version="ntpd 4.2.2p4@1.1585-o Sun Nov 22 16:42:02 UTC 2009 (1)",
processor="i686", system="Linux/2.6.27.37-web100", leap=00, stratum=2,
precision=-20, rootdelay=25.016, rootdispersion=946.806, peer=21397,
refid=198.124.252.90,
reftime=cf7c5bc9.8f99a127  Fri, Apr 23 2010 13:47:53.560, poll=6,
clock=cf7c5bf1.8e0ffeba  Fri, Apr 23 2010 13:48:33.554, state=4,
offset=7.335, frequency=-47.691, jitter=1.113, noise=2.593,
stability=0.020, tai=0

If you find that your clock is stopping on a regular basis, the internal battery of your server may be failing. Consult your server user's manual or on-line references for more information.




Q:Can I boot from a USB key instead of a CD?
A:

Yes, the CentOS project, recommends the method found here.

Additionally, there are tools such as UNetbootin available to migrate an ISO to a USB key.




Q:How can I upgrade an 3.2RC X NetInstall?
A:
  • Download http://software.internet2.edu/rpms/i386/RPMS.main/Internet2-repo-0.2-5.noarch.rpm
  • Install the RPM, e.g. sudo rpm -ivh Internet2-repo-0.2-5.noarch.rpm
  • Edit /etc/yum.repos.d/Internet2-web100_kernel.repo and set enabled = 1 
    # Name: Internet2 RPM Repository
# URL: http://software.internet2.edu
[Internet2-web100_kernel]
name = Internet2 Web100 Kernel RPM Repository - software.internet2.edu - main
mirrorlist = http://software.internet2.edu/web100_kernel/rpms/mirrors-Internet2-web100_kernel
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Internet2
gpgcheck = 1
  • Run yum update
  • Reboot



Q:How can I backup historical OWAMP/BWCTL data?
A:

These steps will get rid of all data older than 3 months (adjust the 'maxmonths' if you want to allow more to stay in the db), and store them as SQL dumps on the host. Here are the steps to clean the database on the throughput machine:

sudo mkdir -p /var/log/BACKUP/bwctl
sudo /opt/perfsonar_ps/perfsonarbuoy_ma/bin/clean_pSB_db.pl \
--mysqldump-opts="--skip-lock-tables" --dbtype=bwctl --maxmonths=3 \
--owmesh-dir=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/ \
--dumpdir=/var/log/BACKUP/bwctl

Run the following command on latency machine to archive data older than 30 days (adjust the 'maxdays' to keep more):

sudo mkdir -p /var/log/BACKUP/owamp
sudo /opt/perfsonar_ps/perfsonarbuoy_ma/bin/clean_pSB_db.pl \
--mysqldump-opts="--skip-lock-tables" --dbtype=owamp --maxdays=30 \
--owmesh-dir=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/ \
--dumpdir=/var/log/BACKUP/owamp



Q:A CVE announcement was made for the current pSPT Kernel, what do I do?
A:

The perfSONAR-PS development effort subscribes to all major outlets that will announce kernel CVEs. In the event that a CVE is announce that directly effects operation of the pS Performance Toolit, the following steps will take place:

  • Announcements regarding the CVE will be posted to the performance-node-users mailing list, more information on the mailing list is available here: https://lists.internet2.edu/sympa/info/performance-node-users
  • A timeline will be relayed regarding availability of new kernels.
  • The CentOS project will make the patched kernel available first, and it will be available through the yum repositories on the toolkit before the perfSONAR-PS project is able to apply the web100 patches.
    • It is strongly suggested that pS Performance Toolkit users utilizing the NetInstall option upgrade immediately. Note this will break support for the NDT and NPAD tools, but use of measurement tools is secondary to security. Run the following command: 
      sudo yum update
    • pS Performance Toolkit users that are utilizing the LiveCD option can either continue using their instance, or shut it down depending on the severity of the CVE.
  • The perfSONAR-PS project will release a web100 patched version of the most recent kernel, and make these available through the yum repo. Announcements will be made again to the performance-node-users list.



Q:After creating a user on the LiveCD and trying to log in, I see Could not chdir to home directory /home/USER: No such file or directory, what is wrong?
A:

The Storage Section of the pS Performance Toolkit Quickstart Guide notes that if you make changes to the backing store during configuration - you should reboot immediatly. Adding users via the User Management commands before rebooting will result in home directory errors.

To correct this problem, delete and re-add the user account.




Q:I am trying to log in with the knoppix user, and it is not working. What is wrong?
A:

Version 3.2 of the pS Performance Toolkit no longer adds a knoppix user by default. If you have upgraded from version 3.1.x then you will still have this user on your system. Please use the root user as the default way to log into the system.




Q:During the NetInstall, I see errors about a corrupt file being downloaded. What should I do?
A:

During the NetInstall, you may see some errors about a corrupt file being downloaded along with buttons like Reboot and Retry. This happens if it fails to download an RPM from a mirror, which can happen for numerous reasons. Usually, that error can be solved by hitting Retry. You may have to hit that multiple times depending on which mirrors the install is trying to download the RPM from.




Q:How can I upgrade from the 3.1.x LiveCD to the 3.2.x NetInstall?
A:

There is not an automatic way to migrate existing data from a legacy LiveCD deployment to the hard disk environment. The following steps can be used as guidelines.

  1. Boot into 3.1.x, and backup the previous configuration/data.
    • MySQL: Dump all of the mysql databases (bwctl, owamp, cacti, pingerMA, and mysql), and store these on a different machine. 
      mysqldump -u root owamp > owamp.sql
mysqldump -u root bwctl > bwctl.sql
mysqldump -u root cacti > cacti.sql
mysqldump -u root pingerMA > pingerMA.sql
mysqldump -u root mysql > mysql.sql
    • Cacti (If using Cacti): Copy the contents of /var/lib/cacti to a different machine.
    • Host Configuration: Copy the file /mnt/store/configs.tbz to a different machine.
  2. Shutdown the machine, insert the NetInstall image (N.B. a different machine can be used also). Setup the NetInstall, do not configure via the steps in the quick start (e.g. do not run /opt/perfsonar_ps/toolkit/scripts/nptoolkit-configure.py or configure things via the web interface).
  3. Copy the contents of /var/lib/cacti back to this machine.
  4. The following step may bot be necessary, but will create the necessary databaes on the new machine: 
    mysqladmin -u root create owamp
mysqladmin -u root create bwctl
mysqladmin -u root create cacti
mysqladmin -u root create pingerMA
mysqladmin -u root create mysql
  5. Restore the database dumps into mysql (the complete dumps will preserve the tables etc. as they were on the old host - there were no database schema changes between versions) 
    mysql -u root owamp < owamp.sql
mysql -u root bwctl < bwctl.sql
mysql -u root cacti < cacti.sql
mysql -u root pingerMA < pingerMA.sql
mysql -u root mysql < mysql.sql
  6. Copy the config tarball to somewhere on the machine.
  7. Download the following script: http://anonsvn.internet2.edu/svn/perfSONAR-PS/trunk/perfSONAR_PS-Toolkit/scripts/upgrade_3.1.x.sh
  8. Edit the script:
    • Set STORE_LOCATION to be the directory of the tarball
    • Set the OLD_CONFIG_TARBALL variable to the location of the tarball, use the STORE_LOCATION variable (e.g. $STORE_LOCATION/configs.tbz)
  9. Run the script.
  10. Download the following script: http://anonsvn.internet2.edu/svn/perfSONAR-PS/trunk/perfSONAR_PS-Toolkit/scripts/upgrade_fix_permissions.sh
  11. Run the script.
  12. Networking configuration for the new machine vs the old machine may have been altered during the configuration move, particularly with respect to the numbering of the network interfaces. The networking configuration tool: 
    sudo /usr/sbin/system-config-network-tui
    Can be used to verify that the settings for DHCP or Static IP address are configured on the correct interface.
  13. Open and edit /opt/perfsonar_ps/toolkit/etc/discover_external_address.conf, be sure the external_interface= entry reflects the current externally facing interface.
  14. Reboot



Q:How do I interpret the headings of each column of the "Throughput Tests" page?
A:

  • First Host - The hostname of the first host in the test.
  • First Address - The IP address of the first host in the test.
  • Second Host - The hostname of the second host in the test.
  • Second Address - The IP address of the second host in the test.
  • Protocol - Indication that a TCP or UDP is being performed.
  • Duration - The length of the bandwidth test
  • Window Size - If a window size was selected for TCP tests, it will be displayed here. If a window was not selected (e.g. TCP auto-tuning) nothing will appear.
  • Bandwidth Limit - If a UDP test was selected, it is possible to limit the total bandwidth that can be consumed for a test. If the test is TCP or no limit was entered, this will be blank.
  • Bi-Directional - Indication that both directions of the test are available. OWAMP tests each direction, and when one direction has failed it may indicate:
    • Firewalls are preventing the test from completing
    • NTP is not synchronized on one of the hosts
    • Either host does not have available test slots
    • The results of the test could not be retrieved
  • Line Graph - A drop down that features different time ranges. When clicked a graph featuring connected points will appear.
  • Scatter Graph - A drop down that features different time ranges. When clicked a graph that only displays data points will appear.




Q:How do I interpret the headings of each column of the "One-Way Latency Tests" page?
A:

  • First Host - The hostname of the first host in the test.
  • First Address - The IP address of the first host in the test.
  • Second Host - The hostname of the second host in the test.
  • Second Address - The IP address of the second host in the test.
  • Bi-Directional - Indication that both directions of the test are available. OWAMP tests each direction, and when one direction has failed it may indicate:
    • Firewalls are preventing the test from completing
    • NTP is not synchronized on one of the hosts
    • Either host does not have available test slots
    • The results of the test could not be retrieved
  • Graph - A drop down that features different time ranges. When clicked a graph will appear.




Q:If I enable SSH via 'chkconfig' it does not remain across reboots, what is wrong?
A:

The initialization of certain services such as SSH or perfSONAR daemons, normally controlled by tools such as 'chkconfig', is managed by the pS Performance Toolkit web interface. The proper way to enable and disable services is to visit https://[host]/toolkit/admin/enabled_services to manage service startup.


Q:How do I change the MTU for a device?
A:

You can view the MTU of your network devices by executing the /sbin/ifconfig command.

To temporarily change the MTU for a device, you use the ifconfig command and specify the device and the new MTU. For example: 

ifconfig eth0 mtu 9000 up

To make these changes permanent you need to modify the specific devices configuration file. These files are in /etc/sysconfig/network-scripts/ and have names like ifcfg-eth0 for the device eth0 and ifcfg-eth1 for eth1.

  1. For example you could add the line MTU="9000" for IPv4 or IPV6_MTU="9000" for IPv6 to /etc/sysconfig/network-scripts/ifcfg-eth0.
  2. After making the changes you need to restart the network services by running the command 'service network restart' as root.




Q:How do I change the SSL certificate used by the web server?
A:

The toolkit by default generates a self-signed SSL certificate that it configures for use with the Apache web server. Some users may desire to replace this certificate with a certificate signed by a certificate authority (CA).

You may also need to replace the certificate due to a problem sometimes encountered with browsers not accepting the self-signed certificate. You may see an error like the following: 

    HOST uses an invalid security certificate.
    The certificate is not trusted because it is self-signed.
    The certificate is only valid for localhost.localdomain
    (Error code: sec_error_untrusted_issuer)

You can find instructions for installing a new certificate in Apache here.




Q:I forgot to enable IPv6 in CentOS when I installed the toolkit. How do I enable it?
A:

It is recommended that you always enable IPv6 during the CentOS installation portion of the toolkit setup. If you did not enable it, then you can do so with the following steps:

  1. Login to the toolkit as a user capable of running sudo
  2. Run sudo and enter your sudo password
  3. Open the file /etc/modprobe.conf in a text editor and remove the following lines: 
    alias net-pf-10 off
alias ipv6 off
options ipv6 disable=1
  4. Restart the host
  5. You can now assign an IPv6 address.




Q:Why is the static IPv6 address I assigned during the net-install process not configured when my host starts-up?
A:

When you perform the net-install of the toolkit, you will be prompted twice to enter networking information by CentOS. The first time is to define the networking to be used for downloading required packages. The second prompt is later in the installation and defines what will be configured on the host post-installation. It is a known CentOS behavior that IPv6 information entered at the first prompt is not automatically filled-in at the second prompt. This can be confusing because the IPv4 information does get automatically filled-in. If you do not manually enter the IPv6 information a second time, then your host will not have the address configured post-installation. You will have to manually assign the address if this happens.




Q:How do I fix my certificate when my browser displays "Error code: sec_error_reused_issuer_and_serial" when I try to access the toolkit web page?
A:

This error can occur with pre-3.2.1.1 installations of the toolkit due to the way the https certificate is generated. If you are running an older installation you may fix this issue by downloading and running the script here on your toolkit host.




Q:I'm seeing "tg3 0000:02:01.0: tg3_stop_block timed out, ofs=4800 enable_bit=2" errors on bootup. What should I do?
A:

The RedHat kernel version 2.6.18-308.1.1.el5 includes version 3.119 of the tg3driver. This driver seems to have the problem, but version 2.6.18-274.18.1.el5 of the kernel, which includes version 3.116 of the tg3 driver, does not have the issue. Unfortunately, since this is a problem in the upstream kernel, there's not much we can do much about it.

There is a workaround for the NetInstall version, though, by downgrading to the previous kernel.

Grab the previous version of the kernel: 

wget http://software.internet2.edu/web100_kernel/rpms/i386/RPMS.main/kernel-2.6.18-274.18.1.el5.web100.i686.rpm

You'll need to then force the kernel to install: 
rpm -ivh --force kernel-2.6.18-274.18.1.el5.web100.i686.rpm

After that, you can reboot the machine into the older version.




Q:My pSPT has an RFC 1918 (Private Internet) address, and I am not able to configure regulare tests. Is there a way around this?
A:

Edit 

/opt/perfsonar_ps/toolkit/etc/discover_external_address.conf
and set allow_internal_addresses to 1. Then reboot the machine.




Q:Version 3.2.2 of the perfSONAR-PS Toolkit LiveCD version won't boot. What should I do?
A:

When installing version 3.2.2 of the LiveCD, a number of users have reported getting error messages such as the following: 

WARNING: Cannot find root file system!

Create symlink /dev/root and then exit this shell to continue the boot sequence.
This is caused by a change in the latest version of the kernel which seems to cause incompatibility with some hardware configurations. At this time it is recommended that users encountering this problem downgrade back to version 3.2.1 of the LiveCD or consider a NetInstall.




Q:Can I run owamp and bwctl on the same host?
A:

bwctl testing will affect owamp results, so it is best to run them on separate hosts. If this is not possible, owamp is still a very useful troubleshooting tool, and it is better to run them both on the same host than to not run owamp at all.

If you do run owamp and bwctl on the same host, owamp may see increased latency and jitter during the bwctl tests, and on a 1G host you may even see loss. If you are using nagios to generate alerts based on these owamp results, you'll need to take this into account.




Q:The drivers that are included for my "foo" network card old, what should I do?
A:

The pSPT includes drivers from the upstream CentOS distribution and a very limited number of specially built drivers for cards that are known to be widely used (e.g. myicom). If you notice that there is a newer driver available than what the pSPT provides you can do the following:

  • Download the source or RPM from the manufactuers web site, build on your pSPT
  • If you are running a netinstall version of the pSPT, run modprobe after installation to load the new version
  • If you are running a LiveCD, you will need to modify the local init.d script to force modprobe on each reboot to load the proper version



Q:I setup a server from the latest ISO, but when I do a NDT test to the server, nothing happens. Is there a problem with the NDT tool?
A:

First check to be sure that the NDT server is running: 


/etc/init.d/ndt status
web100srv (pid 2430) is running...
fakewww (pid 12737) is running...

Also check to be sure you are running a web100 enabled kernel: 


uname -a
Linux HOSTNAME 2.6.18-274.7.1.el5.web100 #1 SMP Wed Nov 2 08:35:05 PDT 2011 i686 i686 i386 GNU/Linux

If you are not running a web100 related kernel, use 

yum update
to be sure you are running the latest version, and restart NDT if necessary.




Q:[BWCTL and OWAMP] Graphs fail to open on my browser. I am using IE 8/9. Is this a problem with my toolkit or my browser?
A:

If you are using the most recent version of the pSPT, the graphing package has been converted to dygraphs, which is HTML5 and JavaScript based. Not all browsers (namely recent versions of IE) fully support HTML5, and as such there will be compatibility issues outside of our control and support structure.


A write up on known problems and possible solutions here:

http://dygraphs.com/#ie

The pSPT project suggests using other browsers as a short term fix (e.g. Chrome, Firefox), as these support the required technology just fine. If you require the use of IE (8, 9, previous versions) the above page has suggestions on organizations using the pSPT can modify HTML to fool the browser (Web Content is located in '/opt/perfsonar_ps/toolkit/web' and '/opt/perfsonar_ps/serviceTest' directories). Our projet has not tested these items to date; caveat emptor.




Q:What can I do in the case of a catastrophic hardware failure?
A:

In the case of a catastrophic hardware failure you may be able to recover some or all of your data files if you have a recent backup. This section contains information on backing up your system.

If you don’t have a recent backup and you can read your disk you still may be able to recover your data. This script copies your data files and toolkit configuration to a tar archive. You can unpack this archive in the root directory of a fresh toolkit installation to restore your data and configuration. You may also want to copy your /home and /root directories to the new installation.

Note that there is no grantee of a full recovery. Depending on the condition of your system you may or may not have success.




Q:Where can I find more resources regarding timekeeping for VMWare Virtual Machines?
A:

VMWare has two resources worth reading:




Q:What TCP congestion control algorithm is used by the perfSONAR Toolkit?
A:

The perfSONAR toolkit uses the CentOS 6 default TCP congestion control algorithm, which is htcp. However, if you enable NDT or NPAD, these tools require the Reno congestion control algorithm, so the default congestion control algorithm for the host is changed to reno.




Q:My host is only connected on R&E routed networks, and I have no ability to access commercial web sites that host packages for the pS Performance Toolkit. Yum is failing as a result, what steps can I take to correct this?
A:

Version 3.2.x of the pS Performance Toolkit (CentOS 5 based) requires access to 4 major repositories:

  • CentOS
  • CentOS updates
  • EPEL
  • Internet2

Of these 4, only the last one is consistantly available via both R&E and commercial routing. The first 3 are available via both methods, but the defaults on CentOS will attempt to fetch a mirror list that might use a commercially routed server only; resulting in timeouts and other unexpected behavior.

To edit the existing repository choices, you need to swap out the mirrorlist with a baseurl in the relevant .repo files:

  • For CentOS-Base (/etc/yum.repos.d/CentOS-Base.repo):
    • set the 'updates' baseurl to one in http://software.internet2.edu/rpms/mirrors-Toolkit-CentOS-Updates
    • For 'base', you can take that same mirror for updates, and replace "/updates/i386" in the URL with "/os/i386" (e.g. http://linux.mirrors.es.net/centos/5/updates/i386/ -> http://linux.mirrors.es.net/centos/5/os/i386/ ). You'll probably need to disable the 'extras' with a "enabled=0" in it.
  • For EPEL (/etc/yum.repos.d/epel.repo), set the 'epel' baseurl to one in http://software.internet2.edu/rpms/mirrors-Toolkit-EPEL

Examples of what these files will look like after modification appear below:

  • For CentOS (/etc/yum.repos.d/CentOS-Base.repo):
  • [base]
name=CentOS-5 - Base
baseurl=http://linux.mirrors.es.net/centos/5/os/i386
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

#released updates 
[updates]
name=CentOS-5 - Updates
baseurl=http://linux.mirrors.es.net/centos/5/updates/i386
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

and

  • For EPEL (/etc/yum.repos.d/epel.repo):
  • [epel]
name=Extra Packages for Enterprise Linux 5 - $basearch
baseurl=http://linux.mirrors.es.net/fedora-epel/5/i386
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL



Q:My (OWAMP|BWCTL) measurements have stopped, and I notice mysql errors in the logs. What should I do?
A:

If you see the following types of messages in the logs: 

2013/06/06 17:00:35 (4864) ERROR> SQL.pm:218 perfSONAR_PS::DB::SQL::query - Query error "DBD::mysql::st execute failed: Table 
'./owamp/20130605_DATA' is marked as crashed and should be repaired at 
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/../lib/perfSONAR_PS/DB/SQL.pm line 213.
" on statement "(select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130530_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130531_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130601_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130603_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130604_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130605_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130606_DATA) order by send_id, recv_id, tspec_id;".
2013/06/06 17:00:35 (4864) FATAL> perfSONARBUOY.pm:1303 perfSONAR_PS::Services::MA::perfSONARBUOY::createStorage - Query error, 
aborting.
2013/06/06 17:00:35 (4864) FATAL> perfSONARBUOY.pm:304 perfSONAR_PS::Services::MA::perfSONARBUOY::init - Couldn't load the store 
file - service cannot start
2013/06/06 17:00:35 (4864) ERROR> daemon.pl:380 main:: - Failed to initialize module perfSONAR_PS::Services::MA::perfSONARBUOY on 
8085:/perfSONAR_PS/services/pSB

It is likely that your mysql instance has crashed and will need to be repaired. Follow these steps. Note that this assumes repairing both the latency and bandwidth functionalities at the same time - if you are running only one of the services, you can omit the comamnds for the one you are not using.

  1. Stop essential services:
    sudo /etc/init.d/crond stop
sudo /etc/init.d/owampd stop;sudo /etc/init.d/perfsonarbuoy_owp_master stop;sudo /etc/init.d/perfsonarbuoy_owp_collector stop
sudo /etc/init.d/bwctld stop;sudo /etc/init.d/perfsonarbuoy_bw_master stop;sudo /etc/init.d/perfsonarbuoy_bw_collector stop
sudo /etc/init.d/perfsonarbuoy_ma stop
  2. Kill anything left over (its always good to check to see if things wedged):
    ps auxw | grep powstream
sudo kill -9 $PID
ps auxw | grep owampd
sudo kill -9 $PID
ps auxw | grep powmaster
sudo kill -9 $PID
ps auxw | grep powcollector
sudo kill -9 $PID
ps auxw | grep bwctl
sudo kill -9 $PID
ps auxw | grep bwctld
sudo kill -9 $PID
ps auxw | grep bwmaster
sudo kill -9 $PID
ps auxw | grep bwcollector
sudo kill -9 $PID
  3. OPTIONAL, note that if you do this, you nuke results that may not be in the DB yet:
    sudo rm -frd /var/lib/perfsonar/perfsonarbuoy_ma/owamp/*
sudo rm -frd /var/lib/perfsonar/perfsonarbuoy_ma/bwctl/*
sudo rm -frd /var/lib/owamp/catalog/*
sudo rm -frd /var/lib/owamp/hierarchy/root/regular/*
  4. Stop the database:
    sudo /etc/init.d/mysqld stop
  5. Run a rebuild:
    sudo su -
sudo myisamchk -r /var/lib/mysql/owamp/*MYI
sudo myisamchk -r /var/lib/mysql/bwctl/*MYI
  6. restart the database:
    sudo /etc/init.d/mysqld start
  7. Backup old data (usually a good idea), in this case it saves 3 months:
    sudo mkdir -p /var/log/BACKUP/bwctl
sudo /opt/perfsonar_ps/perfsonarbuoy_ma/bin/clean_pSB_db.pl --mysqldump-opts="--skip-lock-tables" --dbtype=bwctl --maxmonths=3 
--owmesh-dir=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/ --dumpdir=/var/log/BACKUP/bwctl

sudo mkdir -p /var/log/BACKUP/owamp
sudo /opt/perfsonar_ps/perfsonarbuoy_ma/bin/clean_pSB_db.pl --mysqldump-opts="--skip-lock-tables" --dbtype=owamp --maxdays=90 
--owmesh-dir=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/ --dumpdir=/var/log/BACKUP/owamp
  8. restart services:
    sudo /etc/init.d/crond start
sudo /etc/init.d/owampd start;sudo /etc/init.d/perfsonarbuoy_owp_master start;sudo /etc/init.d/perfsonarbuoy_owp_collector start
sudo /etc/init.d/bwctld start;sudo /etc/init.d/perfsonarbuoy_bw_master start;sudo /etc/init.d/perfsonarbuoy_bw_collector start
sudo /etc/init.d/perfsonarbuoy_ma start
  9. check data sometime soon after (~5-10 minutes for OWAMP, several hours for BWCTL) on the web pages.
  10. As an additional step, you can add the following to the /etc/my.cnf file, in the [mysqld] section:
        set-variable = table_cache=12000

    Run /etc/init.d/mysql restart after this change.



Q:I have a limited amount of disk space in /var/log, and I have noticed that perfsonarbuoy_ma is writing a lot of INFO entries in the /var/log/perfsonar/perfsonarbuoy_ma.log which is filling up this disk space rapidly. Is there a way to suppress all INFO level logs?
A:

The perfSONAR-BUOY MA (and other perfSONAR services) use log4perl controls to set the logging level. You can make a change to the specific configuration file:

 /opt/perfsonar_ps/perfsonarbuoy_ma/etc/daemon_logger.conf

And change the 'INFO' in 'log4perl.logger.perfSONAR_PS=INFO, A1' to be any of the following:

  • OFF
  • FATAL
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • TRACE
  • ALL

WARN is normally sufficient.