On March 18, The Australian Academic and Research Network (AARNet) Operations Center reported a problem (which lasted from 2:30 am to 4:42 am) with their router located at the Pacific Wave, Los Angeles. The outage affected the path from the Mauna Kea Summit to Internet2.
On May 2, service between Maunalani and CFHT at Waimea was interrupted between 4:45 am and 10:13 am due to Time Warner fiber damage.
On May 20, a Time Warner power outage caused a brief network interruption to and from the AARNet hub at Maunalani around 3:00 pm.
On both of the last two incidents, traffic from observatories other than CFHT was automatically rerouted through Oahu and they did not experience an outage.
Starting at 11 pm on May 24, UH-ITS performed major maintenance which lasted until 2:00 am on May 25. In spite of the multiple outages on the interisland links, microwave, and interisland fiber, there was no complete loss of connectivity at anytime; connectivity was sustained through the various alternative routes.
In Hilo: SMA and Subaru were switched to the GE network in March. JAC and CSO are still pending.
1. Performance varies dramatically from under 10 Mbps to 250 Mbps with no apparent explanation.
2. Between the Big Island and the HIA in Victoria Canada, no better than 250 Mbps has been seen through the Southern path. Over the Northern path, Pui Hin has measured over 500 Mbps.
Toward the end of May, Alan Whinery got in touch with Matt Mathis, who wrote the performance analysing tool, NPAD. A telecon was held on May 30th to discuss these performance issues. Since NPAD gets more sensitive as the path segments get shorter, Matt Mathis suggested finding servers and clients along the path with the goal of testing progressively shorter and shorter segments in order to locate the source of excess loss.
Later that day, Darrell Newcomb from Cenic uncovered a "smoking gun": there seems to be a big packet loss between MaunaLani and LA. Attempts at further testing were hindered by the lack of known NPAD servers. Pui Hin's observation is that better documentation of the network, including equipment along the path, names of personnel responsible for the equipment, and availability of network performance measurement servers will be necessary to get a better handle on the problems.
Note: the location of this meeting has been moved to CSO in Hilo to encourage members from Hilo to attend in order to be able to make a decision on the fiber project.