Cabling in Top-of-the-Rack Architectures
Top-of-the-rack architectures offer a clear access-layer migration path to an optimized high-bandwidth network and cabling facilities architecture that features low capital and operating expenses and supports a “rack-and-roll” computer deployment model that increases business agility.
This article is excerpted from a white paper published by Cisco.
Forward-looking IT departments are preparing their data centers for the future by integrating support for 10 Gigabit Ethernet and a unified network fabric into their switching and cabling strategies. Since the typical data center lifecycle is 10 to 15 years, cabling architectures have a tremendous effect on the data center’s ability to adapt to network architecture changes, evolving bandwidth needs, and technology moves, additions, and changes. Cabling architectures, if not chosen correctly, could force an early replacement of the cabling infrastructure to meet connectivity requirements as the network and computer technologies evolve.
Today’s data centers deploy a variety of cabling models and architectures. With the migration from Gigabit Ethernet to 10 Gigabit Ethernet, cabling and network switching architectures are being reevaluated to help ensure a cost-effective and smooth data center transition. The choice of cabling architecture will affect throughput, expandability, sustainability, optimum density, energy management, total cost of ownership (TCO), and return on investment (ROI). Anticipating growth and technological changes can be difficult, but the data center should be able to respond to growth and changes in equipment, standards, and demands while remaining manageable and reliable.
Data center managers and facilities administrators will choose cabling architectures based on various factors. The top-of-the-rack (ToR) model offers a clear access-layer migration path to an optimized high-bandwidth network and cabling facilities architecture that features low capital and operating expenses and supports a “rack-and-roll” computer deployment model that increases business agility. The data center’s access layer, or equipment distribution area (EDA), presents the biggest challenge to managers as they choose a cabling architecture to support data center computer connectivity needs. The ToR network architecture and cabling model proposes the use of fiber as the backbone cabling to the rack, with copper and fiber media for server connectivity at the rack level.
Why Use Top-of-Rack Architecture
Rapidly changing business requirements impose a corresponding need for flexibility and mobility in data centers. Because of the significant cost of building a new data center, designing an infrastructure that provides the flexibility to meet business objectives while increasing ROI is an IT imperative. By building the data center infrastructure—power and cooling, cabling, etc.—in a modular fashion, data center flexibility can be increased, which in turn improves business agility.
Many organizations are now deploying modular data centers. IT departments are increasingly deploying racks of servers at a time. Racks of servers, blade systems, and integrated rack-and-blade systems are often purchased in preconfigured racks with power, network, and storage cabling preinstalled so that racks can be commissioned within hours, not days, from the time they arrive on the loading dock. While server form factors are evolving, and some racks can host up to 96 independent computer resources, the rack form factor remains constant, making it the standard unit of deployment in many data centers.
ToR solutions complement rack-at-a-time deployment by simplifying and shortening cable runs and facilitating the replication of rack configurations. This rack-and-roll deployment model offers a solution by placing switching resources in each rack so that server connectivity can be aggregated and interconnected with the rest of the data center through a small number of cables connected to end-of-row (EoR) access- or aggregation-layer switches.
The TIA/EIA-942 specification provides a simple reference for data center cabling that supports different cabling schemes, EoR or ToR, to meet differing needs from a physical and operational perspective. The ToR model defines an architecture in which servers are connected to switches that are located within the same or adjacent racks, and in which these switches are connected to aggregation switches typically using horizontal fiber-optic cabling.
ToR switching allows oversubscription to be handled at the rack level, with a small number of fiber cables providing uniform connectivity to each rack. The advantage of this solution is that horizontal fiber can support different I/O connectivity options, including Gigabit Ethernet and 10 Gigabit Ethernet as well as Fiber Channel. The use of fiber from each rack also helps protect infrastructure investments as evolving standards, including 40 and 100 Gigabit Ethernet, are more likely to be implemented using fiber before any other transmission mechanism. By limiting the use of copper to within racks, the ToR model isolates the cabling that changes most often to the parts of the data center that change most frequently – the racks themselves. The use of fiber runs from racks provides a flexible data center cabling infrastructure that supports the transition from Gigabit Ethernet to 10 Gigabit Ethernet now, while enabling transition to 40 and 100 Gigabit Ethernet in the future.
Fiber and Copper Cabling Characteristics
A number of optics and media options are available for data center designs. Choices depend on the physical facilities requirements and media characteristics (physical medium dependent [PMD] factors) of the data center.
Fiber
When considering a data center buildout, evaluation of the near- and mid-term future is required to maximize the cable plant investment. Because 10, 40, and 100 Gigabit Ethernet will be standardized in the next three to five years, organizations should assess fiber-optic cabling.
Standard single-mode fiber (SMF) (ITU GR.652, TIA/EIA-493CAAA) is designed to support high-capacity, low-cost transmission components developed for the 1310-nanometer (nm) space. SMF features low dispersion and is optimized for use in the 1310-nm wavelength region. SMF is also used effectively with wavelength-division multiplexing (WDM) systems operating in the 15,550-nm wavelength region. SMF can be used for cross-aisle and inter–data center applications. SMF will support 40 and 100 Gigabit Ethernet with serial implementations.
The most cost-effective fiber cabling is multimode fiber (MMF). It is found in nearly all data centers today and can support 10 Gigabit Ethernet. However, as 40 and 100 Gigabit Ethernet become standardized, either MMF with parallel optics or SMF will be required. SMF, although simpler to manage and more amenable to higher bit-rate transmission than MMF, is more expensive to terminate and also more expensive in terms of the optical transceivers that are required at the switch and host devices.
Copper
Copper cabling can be used for 10 Gigabit Ethernet server connectivity.
Category 6A (Cat6a) copper cabling was developed in conjunction with the 10GBASE-T standard to achieve reliable 10-Gb/s operation over 100-meter copper twisted-pair channels. Cat6a shielded and unshielded products are designed to extend usable bandwidth up to 500 MHz and to drastically reduce alien crosstalk interference. In the context of ToR architectures, Cat6a can be used within the rack.
1X-based Twinax copper is an alternative copper solution that uses SFP+ direct-attach Twinax cabling. Although this solution has a limited cable distance of up to 10 meters, it provides a robust, power-efficient, and cost-effective solution for 10 Gigabit Ethernet transmission.
The SFP+ direct-attach solution is a fully integrated SFP+ cable that is available in multiple lengths up to 10 meters. As the cabling distance is limited, each server is directly connected to a ToR switch with no intermediate patch panels to manage. This approach dramatically simplifies cabling and termination as the cabling is contained within a single rack, and it works well with the concept of a modular data center. The SFP+ direct-attach solution draws 1.5W of power per port, has a latency of 0.1 microsecond, and is available today.
Top-of-Rack Cabling Recommendations
The ToR network architecture uses available cabling media options with flexibility at the rack level, using various server patch cable types, while taking advantage of fiber uplinks from the rack for horizontal cabling. Although CX-1 Twinax and fiber support high-speed 40- and 100-Gb/s transmission, fiber is the recommended horizontal cabling medium as it provides an optimal solution for high-speed 40 and 100Gb/s transmission over relatively long distances (up to 300 meters). Note that 40 and 100Gb/s transmission requires multiple fiber strands (OM3, OM4, and SMF fiber) plus optical interfaces that depend on the distance from the EDA to the HDA and aggregation.
Limiting the use of copper to within racks, the ToR model isolates the cabling that changes most frequently to the parts of the data center that change most frequently – the racks themselves. By using fiber runs from the racks, this architecture delivers a flexible cable infrastructure that supports the transition from Gigabit Ethernet to 10 Gigabit Ethernet today and allows adoption of 40 and 100 Gigabit Ethernet technologies in the future.
The ToR architecture enables data center managers to implement a single cabling model that can support Gigabit Ethernet and 10 Gigabit Ethernet and unified network fabric today, while supporting future 40 and 100 Gigabit Ethernet standards as they come to market. Using a single overhead cable tray for fiber-optic cable management, data center managers have the flexibility to deploy preconfigured racks with different connectivity requirements in any rack position. For example, a rack of servers running multiple Gigabit Ethernet connections can be placed next to a rack of servers with 10 Gigabit Ethernet and FCoE connections to each server.
Data centers are undergoing a fundamental shift. Application developments and server virtualization are causing IT to seek a more flexible, efficient, and available infrastructure that can dynamically adapt to the needs of the business. To achieve this vision, data center architectures are changing to become more modular, so that data center managers can easily add racks of servers by adding racks that are pre-cabled to a ToR switch and then connected to aggregation-layer switches using fiber-optic cables.
To read this white paper in its entirety, visit Cisco online.
