Data Center Solves Communication Problem during Branch Circuit Monitoring System Implementation

Data Centers and Power Distribution

With over 1,900 customers in 39 data centers in 10 countries, Equinix is the global leader in network-neutral data centers. Their customer list is a virtual who's who of network providers, large content companies, and enterprises, including MSN, Verizon, AT&T and Comcast Google, PayPal, IBM, Electronic Arts, and YouTube. These customers rely on Equinix's Internet Business eXchange (IBX) data centers to keep their mission-critical applications running securely and reliably.

Inside an IBX center, Equinix provides customers colocation space, reliable power and connectivity to over 300 networks, including all the major global Internet Service Providers (ISP) and international carriers. Customers are putting an enormous trust level in Equinix's ability to support their systems, many of which are the customers' primary revenue source.

Equinix's IBX centers are under state-of-the-art security and they benefit from a shared power grid, with major energy savings through shared services in these compressed locations. Utilizing Kepware's KEPServerEX to tie into a variety of distribution panels gives Equinix the ability to monitor the power distribution within the data center. They can keep an eye on power loads in a cabinet, assess for sufficient available reserve power, and plan ahead for expansion as the site grows, as well as detect and manage any anomalies remotely.

Software/Hardware Configuration

Equinix uses over 300 Veris boards installed on a single Windows server in a typical IBX center. The diagram below depicts the general architecture, how they are utilizing up to 64 Veris boards, monitoring power use, per COM port on a single server. Digi serial port expanders expand one PCI slot into 8 COM ports and allow Equinix to configure up to 16 COM ports on a single server. Kepware's KEPServerEX provides the data access to Veris devices and exposes the resulting data via OPC communication standards to Iconics software, which provides the graphical interface.

Equinix has a monitoring system in place whereby personnel are able to manually track and analyze power data at each location. In order to obtain more detailed data on customer loads, they were looking to get to a Branch Circuit Monitoring level. This system would allow them to see customer loads at the cabinet and cage level more easily than standard power monitoring systems. It would also allow much better insight overall into how customers are loading their space and could be viewed at a remote level immediately rather than needing to send out engineering staff more often.

Debugging the Performance Delays

The pilot Branch Circuit Monitoring program began in early 2006 and Equinix's engineering team wanted to perform a full rollout to all U.S. IBX centers. Initially, the program had been running very smoothly. They had an extremely easy process in learning and implementing Kepware's software. With data reliability of extreme importance, they felt confident to go to a full rollout in mid-2007, as they were receiving valuable and consistent readings on the power distribution within the data center.

However, difficulties began during the full rollout when the server would run from several minutes to several hours, then lock up for no apparent reason. Pamela Brigham, IBX Systems Engineering Manager would need to re-boot the entire server to allow get it to run smoothly again. Equinix went through a thorough analysis of each segment of the software used to monitor their data systems. After several weeks of debugging with support engineers at Kepware and Iconics through a series of emails, phone calls, and WebEx sessions, Equinix was still unable to get the system to a fully functioning status. With so many variables in the configuration they were using, they needed to spend some time reviewing each piece.

Identification of Problematic Driver

Kepware provided easy access to a support group with diagnostics assistance and various white papers on the trouble they were having, during the several weeks of debugging. Since Kepware was unable to successfully duplicate the difficulty at their own facility, they determined that an in-person visit to Equinix's headquarters was in order.

After looking at each configuration of the multiple drivers forming the system and debugging each individually, it occurred that there might be a problem with the serial port cards. After talking with the engineers at Digi, it was discovered that the serial port driver was actually an older version with a known issue that had been shipped from old inventory. Digi had previously identified and solved the problem in the driver and had thought they had sent out the latest, corrected version rather than the problematic driver. "We were extremely grateful to have discovered the problem after so much analysis. Kepware really went above and beyond in working with us to identify and solve the communications problem," said Ms. Brigham.

Time and Travel Savings

With the serial port driver updated, the full rollout of the Branch Circuit Monitoring System is now in place and performing just as planned. "We now estimate that we are saving about 5 minutes per circuit, and with customers using up to a couple hundred circuits, this is a huge gain for us," stated Ms. Brigham. Equinix's engineering team is now utilizing Branch Circuit Monitoring in all US locations with plans to expand to some of the European locations next year. With an engineering staff now unburdened of much of their travel and able to focus on new development, the installation has been a huge victory for the company.

Return to Kepware Solution Chronicles Page »»