On January 18, a transformer replacement at CFHT Waimea took down the SX south path as well as our path to the INET fibers starting at 8:30am. Since it was Martin Luthur King's day, Chris Zane from UH-ITS and Pui Hin were not able to get a hold of anyone from CFHT. Kanoa finally made contact in the early afternoon and apologize for having forgotten to make prior announcement. At about the same time, Jon Chock, who has been helping to track down the problem also heard from his friend about the transformer replacement.
Kanoa is working on buying additional battery pack for the UPS that supports the MKOCN switch as the UPS does not have the runtime capacity that Kanoa thought it has.
On January 20, the Mauna Lani Juniper m7i router was down from about 8:00am to 8:15am. Observtories traffic was rerouted through Manoa. It was found later that the problem was due to two loose power cables on the router. During the course of investigation, Chris found some software issues on the CFHT 3750 which warrant an OS upgrade to version 12.2.50 to be scheduled at a later time.
On January 22, a Juniper m7i router was successfully inserted into the observatories network after a year of antipation and failed attempts. Thanks to the careful planning, the process took less than an hour and well within the allotted time. However, IPv6 was not enabled during the process for fear of introducing uncertainties while trying to determine whether the migration was successful.
On January 27 Pui Hin learned that the OS on the Hilo m7i needs to be upgraded due to a vulnerability. The upgrade will shutdown the entire Observatories network for 10 to 15 minutes, and 30 minutes or more should back tracking be needed. Pui Hin has been in discussion with UH-ITS on the issue of redundancy in Hilo, to be elaborated later, and decided with UH-ITS to perform the upgrade at the same time or after the backup is in place. Meanwhile, UH-ITS will continue to mitigate our exposure by a combination of anti-spoof and TCP filtering. UH-ITS was unwilling to incorporate the OS upgrade with the installation of the Juniper on January 22 for the same reason why IPv6 routing was not enabled.
On February 3, UPS upgrade at the Summit Communication Room was completed within 10 minutes without a hitch. Also on the same day, at the same time, the OS on the Juniper at Mauna Lani was upgraded to fix the security issues discussed previously. That resulted in the rerouting of observatories traffic through Manoa.
On March 3, the Cisco 3750 at CFHT was down from 2:25pm to 3:10pm due to wide-spread power failure in Waimea and insufficient capacity of the UPS at CFHT. This again cut off our path to the SX south as well as the interisland INET fibers at Mauna Lani. Observatories traffic was rerouted through the 155Mbps microwave. It was at a busy time of the day and with no surprise, the microwave was loaded to its maximum capacity introducing high latency between the observatories and their Mainland counterparts. As usual, Bob Kibrick of the Lick observatory was the first to report the problem.
On March 9, due to damage of some fiber cables at the 16-mile marker of the Saddle Road, some Hawaiian Telcom services including phonelines, our backup T1 framerelay, and the T1 connection between VLBA and the Summit Communication Room were down from 2:00pm to 3:50pm. Fortunately, our GE services were sustained on a bypass. According to Hawaiian Telcom, an excavator driven by a contractor who was on the wrong road clipped the fiber cables.
IPv6 Support - As mentioned earlier, IPv6 was not enabled at the time when the Juniper router was installed in order not to introduce additional issues which could make it difficult to issolate problems due to the migration. In addition, we are also waiting for the reconfiguration of the MKOCN Hilo hub to provide redundancy and the upgrade of the Juniper OS to fix a vulnerability.
Network Performance Issues -
Pui Hin reported that things have been working very well within Hawaii since the Juniper installation except for the segment between Hilo and Manoa which on average has gone up from around 100 Mbps to 500Mbps. However, she is still seeing as low as 80 Mbps on occasion.
Pui Hin hasn't been able to check the traffic to Canada as the previously accessible iperf server is no longer available. Kanoa said that the server IP has changed and he would send out new information after the meeting.
On the effort to establish more peformance measurment server between Hawaii and our counterparts on the mainland, Pui Hin reported some success. Since the last meeting, she has identified or established a number of new servers, two in the LA area with the help from Bob Kibrick; one at Cambridge through Mac Cooper's effort; and through Ruisheng Peng's effort, a new server is being worked on at Caltech. Pui Hin peresented a list of all the currently known iperf servers.
Mac Cooper was interested in having jumbo frames turned on in our switches and routers to achieve better performance. During the discussion, almost everyone is doubtful that it would help over long distance. In addition, it would be impossible to get everyone to do the same. Having a mix of jumbo and regular frames would surely produce bad performance. Mac Cooper subsequently confirmed that the vender he talked to recommended it, but only on segments with latency no greater than 3-4ms.
Kanoa had some concerns that by adding an additional piece of equipment, we are inviting additional hardware issues. Also the additional switch will create an extra hop for CFHT's summit connections.
Pui Hin's response was that Cisco equipment has been very dependable in our experience, more so than the Junipers at UH campus and, we have a standby Cisco switch in Hilo. The additional switch is layer-2 only and would not have much effect on latency. In fact, with the additonal ports on the layer-2 switch, we can use more ports for the interconnects for additional effective bandwidth.
Last updated September 13, 2010 by by Miranda Hawarden-Ogata