Minutes of Working Group Meeting

Joint Astronomy Center, Hilo

September 12, 2013




  1. Current Network Status
  2. On June 21, Kanoa reported problems with intermittent packet loss from the Summit to Waimea and to the Internet. The loss was as high as 20% at its height, at which point it affected their phone as well as data connections. Pui Hin found out through UH-ITS that maintenance work on the UH INET fibers was performed by Oceanic Time Warner between June 19 and June 21, and the problem was caused by OSPF flaps. Chris Zane from UH-ITS apologized for not having rerouted traffic between Mauna Lani and HURP to the 1 GE. That would have avoided the problem. These OSPF flaps were too short to reflect on the MKOCN MRTGs. However VLBA did see them on their router.

    Between June 22 and June 24, Pui Hin further observed OSPF flaps on the MKOCN MRTGs which turned out to be caused by a bad interface in a UH-ITS switch on Oahu. The faulty interface was replaced on June 24. This problem affected only the connection between the IfA facility at Manoa, not the observatories or their base facilities.

    On July 1, Kanoa alerted us to a serious traffic accident in Waimea where a utility pole was damaged and HawTel fibers were strewn out all over the road. However none of our connections were affected.

    On July 3, while working on preparing for the upgrade of the MKOCN backbone to 10Gbps in Hilo, Pui Hin noticed that the backup 1 GE from Mauna Lani to HURP was down. Oceanic was notified and they were able to clean and reseat the jumper at Mauna Lani and that restored the service. Oceanic couldn't tell us how that happened but the guess was that the connector was accidentally dislodged during other work around the area. Fortunately, the backup was never needed while the jumper was disconnected.

    On July 3, ~6:00pm, Nagios warnings indicated that the temperature in the Summit Communication Room has risen. We suspected it to be the air handler that was fixed in August. MKSS Douglas Astrande went up on July 4th to take a look. Indeed he found that the fan motor was dead again. Douglas propped open the door and set up fans to keep the temperature down. Since we weren't able to find a motor in Hilo, we weren't able to replace the motor until Monday, July 8. The fans Douglas put in there as a temporary measure did fine holding the temperature down.

    On July 22, the MKOCN backbone in Hilo was upgraded to 10Gbps by installing a Cisco c4500x-16 at HURP. A lot of preparation was done ahead of time so that at the scheduled time, we only had to move the Summit and the observatory connections from the old Cisco 3750G-12s to the new switch.

    The upgrade, which was scheduled to be completed between 12:15pm to 1:00pm, went mostly on schedule except for the connections to JAC and CTAHR. Miranda and I failed to notice that JAC was having intermittent outages until ~4:00pm. The problem was fixed by configuring their interface with "speed nonegotiate". The problem with CTAHR arose from some misunderstanding among UH-ITS network staff. New fibers were installed on June 21 for CTAHR to connect directly to the UH backbone. However configurations had not been modified and on July 22, CTAHR was still using their old connection through the MKOCN. The engineer who worked with us on the MKOCN upgrade did not realize this. Configuration for the new CTAHR fiber route was made by UH-ITS that evening.

  3. Brief Update of the Network Upgrades in Hilo and on the Summit
  4. On August 9th, Oceanic Time Warner successfully patched the new UH Fiber all the way from Hilo to HP, the length of the run was 167,000 feet. It was a great relief that, thanks to the hard work and tenacity of the cabling team, Oceanic was able to pull the fiber through the conduit that was crushed during the construction of the parking lot next to the HP common building.

    While we were having our meeting, UH-ITS was at HP installing equipment and performing end to end tests. Once that is done, Pui Hin and Miranda would be able to bring the two 10Gbps links to the Summit using existing MKOCN fiber.

    Other items yet to be completed:

    . The backup 3750 switches, at the summit and at HURP, are not in place. That could easily be done within the next month or so. The backup switches are designed to provide the observatories with the option to have a second connection to their facilites, mostly for equipment redundancy.

    . The fiber from HP to Mauna Lani along the Saddle Road won't be ready until sometime next year.

    Other Related Issues:

    Although the observatories are currently connected using static routes, Pui Hin will be happy to work with anyone who prefers to use OSPF. An example of where OSPF routing is necessary would be in Waimea, for the sharing of private alternate routes between the Keck and the CFHT facilities. In order to make it work, all IPs along the routes must be in the public UH address space and OSPF costs will need to be carefully configured on the various interfaces.

  5. Weather Display Station at HP
  6. During our meeting in March, the working group agreed that the development and maintenance effort for the new display station will be minimized by using displays and services already developed and served by the observatories on their own systems. In order to move forward, Pui Hin asked everyone to confirm what they currently have available that can be contributed for use by the new weather display station.

  7. Next meeting
  8. December 12th, Thursday, 10:00am at the Keck Waimea Facility.

MKOCN Homepage Back to Minutes

Last updated March 20, 2013 by Miranda Hawarden-Ogata