At about 5:00pm on December 13th, the Verizon fiber cable between Humu'ula and the Verizon Hilo central office was damaged. As a result, much of the communication to HP and the summit was lost. (The connection between CFHT and the Keck head offices and the summit was not affected.) A Verizon crew dispatched to repair the fiber cable at about 8:30pm turned around after reaching the site because of high winds and unsafe conditions. All services were restored around 2:00pm the next day.
During the outage, service for Subaru was restored immediately because the OC12 service provided to Subaru by Verizon provides for rerouting of traffic from Humu'ula through Kamuela and then through the Hamakua coast. According to Verizon, an attempt was made to manually patch the damaged cable to alternate routes. As a result of that effort, JAC's service was restored sometime in the evening of the 13th. However Verizon technicians then decided that patching all of the services would take about the same amount of time as simply repairing the fiber cable. The patching efforts were discontinued in favor of repairing the cable.
Verizon offers us the option to build route diversity at an additional monthly recurring charge. At the current level, Verizon will try to provide diversity on a temporary and non-guaranteed basis. Pui Hin has asked that the common DS3 be rerouted before other DS3s.
About six months ago, CFHT experienced loss of connection between their summit router and the MKOCN summit router. Our investigation showed numerous input and CRC errors on both routers. Cleaning fiber connectors on both ends reduced errors, but loss of connection continued to occur. The problem could be fixed temporarily with a reboot of the summit router or diverting the CFHT traffic to HP, although input and CRC errors then started to appear on the HP router. In recent weeks, the loss of connection has spread to JAC, UH88, Gemini, as well as Subaru. Since the observatories summit routers and the MKOCN summit router are directly connected through ATM, Pui Hin does not think that this is a routing issue. Instead, she suspects the problem is caused by either dirty fibers or a defective interface card. As a first step towards solving the problem, Pui Hin suggested scheduling a cleanup of all the summit fibers. A preliminary schedule was drafted to clean all the fibers between the summit communication room and each of the MK observatories.
Charles Yun from Internet2 visited the Big Island on an invitation by Kanoa who organized several meetings between Charles and the observatories. Charles suggested that we start gathering data on types and volumes of data produced by each observatory, and also to run tests on our existing network that will build a case for improved networking to the Big Island and the summit. Kanoa started running some tests in early March. No effect or degradation has been observed.
Pui Hin also reported on a new funding opportunity through NSF. The program's title is "International Research Network Connections" (IRNC).
Here's an excerpt from the program's synopsis:
"NSF expects to make a small number of awards to provide network connections linking U.S. research network with peer networks in other parts of the world. Links funded by this program are intended to support science and engineering research and education applications. Funded projects will assists the U.S. research and education community by enabling state-of-the-art international network services similar to and interconnected with those currently offered or planned by domestic research networks.
"Preference will be given to solutions which provide the best economy of scale and demonstrate the ability to link the largest communities of interest with the broadest services."
UH will most probably partner with other organizations in this proposal to achieve end-to-end connections. The goal is to bring the 10Gbps to a UH facility and to the Mauna Kea summit at a minimum of 1Gbps for the short term. For the longer term, to build the infrastructure that will provide observatories and other scientific organizations with opportunities to acquire cost-effective high speed connections.
Note: Since the last meeting, Verizon, citing applicable mileage charges, has given Pui Hin much higher numbers for Kamuela. Pui Hin has scheduled more meetings with Verizon, and will report on this at the next meeting.
For the dormitories, we are running into an unexpected problem. While we did not have any trouble getting the wireless signals to each of the buildings, the insulation in the walls is blocking distribution of the signals to the rooms.
Pui Hin asked the group for suggestions on how we might have the dormitory rooms connected without having to pull cables into every room. Kanoa suggested using Ethernet over AC powerlines.
Following are the number of fibers requested by each observatory:
| Gemini | -- | 6MM/6SM |
| JAC | -- | 12MM |
| SMA | -- | 12MM |
| Subaru | -- | 8MM+4SM or 4MM+4SM depending on the cost |
CSO will continue to patch into Gemini
It is not clear when the rewiring projects can be realized since each observatory's fiscal year starts in a different month:
| Gemini | -- | January |
| IRTF | -- | February |
| JAC/Subaru | -- | April |
| SMA | -- | October |
| UH88 | -- | July |
However, to move forward, Pui Hin will get a budgetary quote from Verizon.
Following the meeting, a presentation was given by Sun and AMD representatives on the AMD Opteron Processor, the cooperation between Sun and AMD, and the Sun Grid Engines.