Issues related to peformance of the SX south path:
On April 20, UH-ITS reconfigured the connections at IfA Hilo (the IfA Hilo facility) to have the 3750 going directly to a 6500 and the old 7206 to a new 3750 at UH Hilo. This change would provide us with two completely redundant paths to Oahu. Also, it allows UH-ITS to set up a VLAN between Mauna Lani and Kella Hall through the 3750 at IfA Hilo to bring BGP back to Manoa from Mauna Lani, and to provide IPv6 addresses on the 6500. Recall that in February, UH-ITS' effort to support IPv6 on the 3750 through an upgrade of the IOS on the switch was unsuccessful.
Immediately after the reconfiguration, Pui Hin noticed a drop in transfer rates between Hilo and Manoa but not between the Summit and Manoa, or Waimea and Manoa. A few days later, Kanoa reported asymmetric and poor transfer rates between Waimea and the Summit.
On May 15, while UH-ITS was still investigating the situation, Bob Kibrick from the Lick Observatory reported serious packet lost, high and extremely variable latency between their remote observing facility on the University of California Santa Cruz (UCSC) and the Keck Observatory.
On May 16 Chris Zane of UH-ITS reported that the 3750 switch at IfA Hilo was running at rather high CPU with minimal traffic which would explain packet loss and high latency when traffic is up. He considered rerouting all observatories traffic over the SX north path to bypass the switch.
On May 21 Alan Whinery discovered some problems on the SX south path to the UCSC. In light of this, he decided to reroute only the traffic between UCSC and Keck through the north path. After the reroute, performance was restored to normal. Apparently the ACKS due to the bad path to UCSC was enough to bring the 3750 to its knees.
A decision was made to purchase a Juniper to replace the 3750 routing duties.
Other unrelated incidents:
On May 6, at 2:30 pm, Oceanic Time Warner moved the connection between IfA Hilo and CFHT to a temporary link in preparation for some equipment relocation to take place on May 6 and 7. There was a 3 minutes interruption during the process.
On May 7, at ~2:00 pm, some configuration issues caused an outage of approximately 10 minutes after the Oceanic circuit was restored.
It is worth noting that Oceanic originally proposed a 2-4 hour interruption for the equipment relocation. They offered the workaround after stern opposition from Kanoa.
In the latter part of May, Pui Hin received sevaral reports that the wireless at HP has not been working reliably. It turned out that the Cisco APs for the VIS connection are still active even though the VIS is now connected through fiber cables and the APs are no longer connected back to the LAN. Pui Hin disabled the wireless interface on the APs while waiting for their physical removal. No new problems have been reported.
There seems to be continuous interest on the part on the US Naval Observatory to fund fiber connection to the VLBA on Mauna Kea. More inquiries were made but no actual plans.
It was decided early on that IfA will issue one PO to the selected vendor for the entire job. Each participating observatory will then submit a PO to the IfA in the amount of the observatory's portion of the work as an agreement to pay. It took a while for all the paperwork to get done, but a PO was generated and submitted to Hawaiian Telcom on April 28. Terms of delivery is "9 week ARO".
As of Monday this week, the fiber is still being manufactured. However, Maxwell Communications, Hawaiian Telcom's subcontractor, is hoping to start installation of the Maxcell innerducts and fiber housings on Monday, June 15 with hope to start fiber installation at the end of the month.
Pui Hin reminded everyone of when she first proposed the project in 2003 in conjunction with the construction of SMA Hilo facility. The current connection predated the IfA Hilo building. It was constructed around 1998 as the result of community efforts. It consisted of a fiber panel donated by JACH installed in the communication handhole outside of the Subaru building. MKO observatories tenants of the park pulled their own fibers to the fiber panel. In February of 2000, the IfA joined the community by pulling fiber from their new building into the handhole and, with equipment acquired through the Gemini network grants, provided Internet access for the park. It is a big relief to have the project finally on its way as many of the old fiber strands are no longer operational and the fiber panel is just about rusted through thanks to the humid Hilo climate.
Last updated September 13, 2010 by by Miranda Hawarden-Ogata