On September 20 peering changes at the Pacific North West GigaPOP (PNWGP) in Seattle caused an outage of the commodity Internet that lasted from 8:15am until 8:25am.
On October 12, while Pui Hin and Miranda were working in the Summit Communication Room (SCR) with CFH, Keck, and JAC on their GE connections, one of the two UPSs in the room died. While trying to re-balance the power usage between the other UPS and a new one on loan from Keck, the ATM router was inadvertently rebooted twice between 4:00pm and 5:00pm causing loss of connection to HP and all Summit and Waimea facilitites.
On October 29 UH-ITS started to advertise observatories subnets to the SX networks so that observatory traffic would not only be going through the SX southern path on the outward bound, but will also return through the SX southern path. A number of routing issues arose and it took over a week to resolve them.
On October 29 observatories experienced problems getting to Microsoft related and NASA and ST sites.
On October 30 advertising was stopped to allow observatories to reach Microsoft related, NASA, and ST sites.
On October 31 routing issues were believed to have been resolved; advertisment of observatory subnets to the SX resumed. According to UH-ITS, the problems with routes to Microsoft sites, as well as NASA and ST sites were caused by the fact that traffic not accepted by the SX at the PNWGP was not routed correctly to the commodity network. UH-ITS resolved the problem by advertising specific /24 subnets rather than in bigger blocks. Note that NASA and ST would not peer with SX as it is a foreign network. As such, traffic to these sites has to be currently routed through the commodity networks.
From November 1 through 8 observatories continued to have problems with some Microsoft, NASA and ST sites.
On November 8 UH-ITS working with PNWGP figured out that some of the specific /24 routes advertised were not accepted by Arcnet's routing policy. Problems were fixed when Arcnet's routing policy was modified to accept the /24 routes.
On November 13 SMA experienced routing problem to Taiwan. UH-ITS implemented a temporary static fix that needs to be revisited at a later time.
For about 10 days, starting at the end of November, CFHT was seeing poor network performance between Waimea and the Summit. The problem eventually went away by itself while Pui Hin and Kanoa were running various tests to determine the cause. Pui Hin noticed software upgrades on the Arcnet routers during the period and the problem seems to have gone away after the last upgrade.
As expected, we are also seeing TCP performance issues over long distance.
We are still waiting to hear from Hawaiian Telcom regarding when SMA's private GE will become available. As a reminder, Hawaiian Telcom has agreed to offer a GE agreement to SMA at the same rate as with the other observatories, subject to PUC approval.
In Waimea, CFHT is connected to the UH-ITS GE router and has been using the shared GEs for connection between their Summit and base facilities.
In Hilo, Gemini and IfA are on the GE network. Yet to be scheduled are the rest of the Hilo facilities including CSO, JAC, SMA, and Subaru.
Pui Hin invited everyone to participate in the BOF and suggested a get-together at Manoa before or after the BOF.