Minutes of Working Group Meeting


Keck Base Facility, Waimea

December 12, 2013

Present:

Absent:

Minutes

  1. Current Network Status
  2. On October 3, Simon Chan reported some problems accessing the Internet which are associated with DNS or routing issues. Pui Hin checked with UH-ITS but they couldn't provide any information. Pui Hin reminded everyone to send along some traceroutes with their problem reports.

    On October 5, Kanoa notified us that there was an extended power outage and only one of the two UH-ITS 4900s was supported on a UPS. The one UPS was enough to see us through as no incident was reported by anyone except that UH-ITS did call Pui Hin for the status of the switch that was down due to the power failure.

    On Friday, November 22, around 5:30pm, Kanoa reported slow connection between Waimea and the Summit that had been happening intermittently over a period of time. Iperf tests reviewed speed from under 5 to 900Mbps. Concluding that it was due to OSPF flapping, he went ahead to adjust the OSPF cost on the interface he thought was causing the problem.

    On Saturday, November 23, around 10:30pm, Kanoa was seeing packet loss of ~90% and high latency of 50 to 400ms between certain pairs of computers between their Summit and Waimea networks. Traceroutes showed that the routes with high packet loss and high latency all went through 128.171.249.1 which is the Manoa interface of the T1 between Manoa and HP. They managed to observe by changing the IPs of critical computers to IP pairs that were not affected.

    On Sunday, November 24, after some investigating, Pui Hin found that the problem was due to the existence of a default route generated at Manoa using the T1 as the gateway. Since the the T1 is known only in OSPF area 0 and the MKO network is in area 255, there wasn't a way to suppress the default without access to the UH-ITS router at Manoa. Instead, Pui Hin adjusted some OSPF costs at HP to make sure that the route through the T1 would not be preferred by the observatories' routers in Hilo and Waimea. Observing on the night of the 24th went without incident and it wasn't until the next morning that Pui Hin learned that certain HP subnets were affected by the change.

    On November 25, Pui Hin worked with UH-ITS to try to find a better solution by adjusting the cost of the default route on Oahu to no avail. In the end, since we were installing the two 10Gbps circuits at HP on that very same day, we decided we could safely turn off the T1 which was used only as a backup at least temporarily.

    On the night of November 30 - December 1, Kanoa observed some dropouts on their Summit-Waimea link. These dropouts could not have been caused by the T1 as that had remained shutdown since November 25th. UH-ITS could not find anything that would have caused the problem.

  3. Brief Update of the Network Upgrades in Hilo and on the Summit
  4. On September 12, while performing some end-to-end tests on the newly installed fiber between Hilo and the Summit, UH-ITS found a 10dB loss of signal on a fiber pair that cannot be explained. Time Warner was charged to figure out the cause of the loss. Finally on September 24, they found a kink in the fiber panel in the switchgear room in Hilo. Subsequently, on October 10, UH-ITS installed the 10Gb equipment at the IfA Hilo network room, and on November 30, they were back at HP to complete the installation and successfully turned on the two 10Gbps circuits.

    On November 21, in Hilo, Miranda and Pui Hin upgraded the MKOCN network to 10Gb by installing the Cisco 4500x and migrated the Hilo observatories' connections from the Cisco CAT3750G to the 4500x. With all the equipment preinstalled, they managed to keep the downtime to less than 30 minutes.

    On November 24, a backup switch was also installed in Hilo to provide support of a second (backup) connection for the observatories.

    Routing issues:

    In the old days, all the MKOCN sites were in the OSPF "common" area 0 which is the base OSPF area that covers everywhere. UH-ITS is now deploying NSSA extensively, in order to keep the number of routes fewer for smaller routers to handle. Also they are using it as a way to partition the network into separate administrative areas linked up by area 0 so that mistakes made in one area will not affect another area. However, this has created some problems with optimal paths not being preferred at times. As an example, right after our two 10GE links were up, Pui Hin noticed that the 10GE path was preferred from the Summit down, but not in the reverse direction. Also between Mauna Lani and Hilo, the TW SX extension 1GE that passes through CFHT was preferred instead of the 10GE path through the North Hawaii Education and Research Center. Pui Hin worked with UH-ITS to make adjustments to the OSPF costs over a number of links and, as of December 11, all of our routes are preferr ed over 10Gbps paths.

    Pui Hin went through the PowerPoint presentation she gave at the MKO Users' Committee Meeting as a summary of all the upgrades that have been completed since 2010. It also included some information on what the rest of the upgrade plans are for the near future.

    There was a discussion on what to do with the T1 circuit. It was decided that we will continue to pay for it until the alternate fiber route from the Summit to Mauna Lani is completed. However, we will keep the circuit down unless there is a need for it in an emergency.

  5. Weather Display Station at HP
  6. As the ongoing network upgrade effort is coming towards an end, Pui Hin suggested we put our time into getting the weather station replaced. Miranda was charged with getting in touch and working with everyone to get a firm idea of what we can readily use. If it is determined that much more development effort is needed to put together the non-interactive station, we will pursue the alternate option of an interactive system as suggested by Tim Minick, which requires little or no development work. Meanwhile, Pui Hin will check with Stewart Hunter on our current fund balance and whether there will be money for the project within the current fiscal year.

  7. Next meeting
  8. March 13th, Thursday, 10:00am at the Subaru Hilo Facility.


MKOCN Homepage Back to Minutes

Last updated March 20, 2013 by Miranda Hawarden-Ogata