Minutes of Working Group Meeting

Gemini Base Facility, Hilo

June 9, 2011




  1. Current Network Status
  2. During the last three months, networks out of Hilo have been plagued by Oceanic fiber breaks and hardware problems at the Hilo SOB (State Office Building). Pui Hin has discussed with UH-ITS the need for backup for the SOB as well as the Oceanic fiber and the idea of using one of our HT (Hawaiian Telcom) GE circuits. According to Chris Zane, using HT fiber for backup is one of the options being considered. Also, the fiber to be installed between Hilo and HP will continue to Mauna Lani along the Saddle to create an INET ring. CFHT, IfA Hilo, and HP will all be on that INET ring.

    Here's an account of what happened:

    On March 13 -- starting around 12:15pm -- a fiber break between Honokaa and Hilo brought down both the INET and the SX path out of Hilo. Traffic was rerouted via the OC3 microwave. Apparently, we were still relying on the temporary fiber that was installed after the March 8 fiber break and the temporary fiber cable was accidentally "trimmed" by state tree trimmers.

    The SX through CFHT was restored around 2:45pm and the INET path was back up around 3:10pm.

    On March 17 Oceanic respliced the fiber that was broken on March 8 between 12:01 and 4:00. Resplicing was done strand by strand and so each site took a short hit as it was taken off the temporarily fiber for the resplicing.

    On April 7 the LuxN wave equipment at the Hilo SOB failed which brought down the INET path for Hilo but the SX was ok. Traffic between Hilo and Oahu, and the commodity traffic to the Internet were rerouted over the microwave.

    On April 9 -- starting at about 4:00am -- SONNET equipment at Puu Kilea on Maui also failed which resulted in the failure of the microwave in addition to the loss of INET since April 7. The result was a complete interruption of traffic between Hilo and Oahu including the commodity traffic to the Internet. The microwave was restored at 5:20am.

    On April 11 the LuxN equipment at the Hilo SOB was temporarily restored.

    On April 25 -- starting around 9:30am -- the LuxN equipment at the Hilo SOB once again started to malfunction causing Oahu and Internet bound traffic to flap between the GE and the microwave path. The SX path was not affected. Around 2:00pm, the management card was pulled from the LuxN box which kept things stable until April 28.

    On April 28 -- around 5:30am -- the LuxN box became unstable once again and this time UH-ITS rerouted all Oahu and Internet bound traffic to the microwave. The LuxN hardware at the SOB was finally replaced at around 9:15am.

    On May 18 there was another Oceanic fiber break, this time near Hilo. Once again connection to CFHT and both the SX and INET paths from Hilo were cut. All traffic out of Hilo was rerouted through the microwave. Connection to the SX via CFHT was restored at 2:55pm. The INET was restored at 6:00pm.

    On May 31 -- at about 6:00pm -- the LuxN box at the SOB acted up again. The INET was down until 8:00pm. The SX was not affected.

    On June 1 the same thing happened and the INET was down between 4:41am and 6:24am.

    On June 3 a firmware update on the LuxN box at the SOB seems to have restored stability to the network connections out of Hilo.

  3. FY 11/12 Budget
  4. The Cisco maintenance budget has been increased by $3000 in anticipation of additional equipment to support 10Gb upgrade at the MKOCN Hilo Network HUB. The electricity budget has also been increased by $2000. There is a small decrease in salary, due to an over-estimate last year. The budget this year will be $93,294 vs $89,474 last year. However the rate per observatory will be lower than last year -- $7,275 vs $8,002 after taking into consideration the fund balance.

  5. Brief Updates of Various Projects
  6. VLBA - Fiber Connectivity to MKOCN

    Despite earlier concerns, the project was approved by the Office of Mauna Kea Management in late May. Although some of the conduits between the VLBA and the Summit Communication Room (where the new VLBA fiber meets up with the MKOCN fiber) do cut into the Natural Area Reserve, there are no handholes in those areas so there will be no need for HT to access the area. Also HT found an empty subduct that traverses the Puu which can be pulled out and replaced with a new subduct system without having to walk along the Puu.

    Fiber and subduct installation started on 6/1 and is expected to be ready for testing on 7/6 with in-service date of 7/8.

    Improvement of MK Summit Conduit Infrastructure

    For the VLBA project, HT will be installing a 4-way multi-path subduct product called Futurepath. Pui Hin was hoping initially to take advantage of the pull and have HT install a 2-way system for future use by the MKOCN. However she later backed out of it due to time constraints by the contract between UH-ITS and the Naval Observatory, the cost quoted by HT ($12,500 for approximately 1800' of conduit), and the fact that these subducts do not come with pull strings. Instead, she will continue to persue an as-built drawing of the current conduit system from HT and work with Bob McLaren on a plan to provision for subduct space in the future.

  7. Follow up from the Last Meeting
  8. Documents pertaining to the Wireless Policy on Mauna Kea

    In response to requests from several observatories, Pui Hin distributed documents pertaining to the Mauna Kea wireless policy. The documents include:

    . A memo on the restriction of mobile and cellular telephone use within the Mauna Kea Science Reserve, dated March 8, 1994.

    . A memo covering the Policy on radio transmitters in the Mauna Kea Science Reserve, dated April 2, 2001.

    . A request from the radio telescopes to refrain from the use of wireless and microwave technology on the summit of Mauna Kea.

    Road-condition Mailing List

    Pui Hin looked into the report that the road-condition mailing list was heavily spammed but saw no evidence of it except for a very brief period in February. Pui Hin speculated that perhaps one of the recipients had his/her computer hacked at that time. Jonathan Chock who reported the problem also confirmed that the problem seems to have dissipated.

    The issues with IRTF -- that their users were not receiving -- is due to configuration issues: that there were too many levels of aliases and include files. The problem was resolved once the configurations were simplified.

    Network Rack Security at HP

    Pui Hin has made a request to Stewart Hunter to lock the door of the network room. (This has happened since the last meeting - thanks to Stewart Hunter)

    Weather System at HP

    For the replacement/upgrade of the weather display system at HP, Pui Hin suggested using a Google TV of some form that would need no maintenance: a TV that comes with Google TV or a plain TV with a Logitech Revue box. Of the two, the Logitech Revue box is preferable because the keyboard and mouse combo that comes with it is more standard than the Google TV currently available. However, Simon Chan -- who has experience with the Logitech Revue -- pointed out concerns with the Logitech Revue related to certain Java streaming and Flash Video issues. In light of this, Pui Hin will look further into whether weather displays are affected by these problems before deciding whether we need to go back to a regular PC and a big monitor. As for the size of the screen, it was agreed that a TV with a screen size between 32" and 37" will probably work best. Some of the weather pages look almost too stretched on screens 40" and bigger.

  9. Next meeting
  10. September 8, Thursday, 10:00am at the JAC Base Facility, Hilo.

MKOCN Homepage Back to Minutes

Last updated September 13, 2010 by by Miranda Hawarden-Ogata