HP Cluster Platform Interconnects v2010 Myrinet System Interconnect Guide
HP Cluster Platform Interconnects v2010 Manual
View all HP Cluster Platform Interconnects v2010 manuals
Add to My Manuals
Save this manual to your list of manuals |
HP Cluster Platform Interconnects v2010 manual content summary:
- HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 1
HP Cluster Platform Myrinet System Interconnect Guide Part Number: A-CPMYI-1A May 2005 Revision/Update Information: Version 1.0 Product Version: HP Clusters This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform solutions. Hewlett-Packard Company Palo Alto, - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 2
contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 3
Contents About This Manual 1 Myrinet interconnect overview 1.1 Interconnect characteristics 1.2 Backplane and chassis types 1.3 Wiring rules 2 PCI cards 2.1 General characteristics 2.2 Single-port XP PCI card 2.3 Dual-port 2XP PCI 2.4 Single- - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 4
1-5 128-Port Myrinet System Interconnect 1-4 1-6 Fully Populated 128-Port Myrinet System Interconnect 1-5 2-1 PCI Identification Label 2-1 2-2 Single-port XP PCI Rev. D card 2-2 2-3 Myricom 2XP PCI Rev. E card 2-4 2-4 Myricom 2XP PCI Rev. F card 2-6 3-1 Myrinet system - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 5
• Is familiar with HP cluster architecture and concepts Organization This manual is organized as follows: Chapter 1 Chapter 2 Chapter 3 Chapter explanation of the LEDs and some diagnostic procedures. Provides instructions for replacing some components in the system interconnect. Describes how - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 6
in the caution could result in damage to equipment or loss of information. Important Text set off in this manner presents clarifying information or specfic instructions. Note Text set off in this manner presents commentary, sidelights, or interesting points of information. vi About This - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 7
1 Myrinet interconnect overview The high-speed Myrinet system interconnect is used in some HP Cluster Platform solutions. The chassis for the interconnect is 5U or 9U, depending on the cluster configuration. A PCI adapter card connects each application node in the cluster to the interconnect. The - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 8
1.2 Backplane and chassis types The Myrinet interconnect used in an HP Cluster Platform can be a 5U (64-port) or 9U (128-port) chassis. The backplane (Figure 1-2) provides connectivity for the 8-port switch cards. Figure 1-2: Myrinet System Interconnect Backplane For configurations greater than 128 - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 9
interconnect, consolidates the uplinks from two node-level switches. Each switch supports up to 128 links, 64 links from each node-level switch; therefore, two top-level switches are necessary to support configurations up to 256 nodes. The node-level switch is populated - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 10
Figure 1-5: 128-Port Myrinet System Interconnect xc-013B-IG Table 1-1 explains the chassis types, and the types and location of line cards within the interconnects. Table 1-1: Chassis types Chassis Type Component Bounded chassis M3-E128 17 slot chassis Monitoring Line Card 8 Port Switch Line - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 11
card 5 Card ejection/insertion handles 6 Cooling fan module Detailed wiring tables are presented in the HP Cluster Platform Myrinet System Interconnect Configuration and Cabling Tables Guide . Myrinet interconnect overview 1-5 - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 12
- HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 13
the port is powered, and a link is connected. Yellow indicates a fault. If the switch port is unlit, the switch is not powered, or there is a problem with the line card. The LEDs for the ports only light when there is a cable attached and the GM message passing system from Myrinet is - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 14
2.2 Single-port XP PCI card The single-port XP PCI card (Figure 2-2) is the Myricom Rev. D card. It supports one X-port interface and one PCI-X interface. Figure 2-2: Single-port XP PCI Rev. D card 2-2 PCI cards - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 15
, nonoperating -40C to 70C Relative humidity 90% @ 65C 2.3 Dual-port 2XP PCI The dual-port 2XP PCI NIC (Figure 2-3) is the Myricom Rev. E card. It supports two X-port interfaces and one PCI-X interface. PCI cards 2-3 - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 16
Figure 2-3: Myricom 2XP PCI Rev. E card 2-4 PCI cards - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 17
-40C to 70C Relative humidity 90% @ 65C 2.4 Single-port 2XP PCI The single-port 2XP PCI NIC (Figure 2-4) is the Myricom Rev. F card. It supports one X-port interface and one PCI-X interface. These NICS must use the Myricom GM-2 or Myrinet Express (MX) software. Other than the difference in the - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 18
Figure 2-4: Myricom 2XP PCI Rev. F card HPTC-0105 2-6 PCI cards - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 19
3 Switch, spine and monitoring line cards The Myrinet system interconnect uses three types of cards. This chapter presents the following information: • A description of the switch card (Section 3.1). • A description of the spine card (Section 3.2). • A description of the monitoring line card ( - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 20
Item 3 4 Description Port identification number Type LC fiber port Figure 3-3 identifies the external features of the Myrinet system interconnect 8-port GigE switch card. Each GigE port has two green LEDs in the RJ45 connector. The right LED indicates link activity, and the left LED indicates - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 21
Figure 3-4: Block Diagram of the gigabit ethernet switch card Myrinet Backplane Interface 7 6 5 4 3 2 1 0 Serial link Sense & Clocks Control Scan path XBar16 dual + 12V Power & hot-swap circuits Lanai-XM-based Myrinet-GbE conversion for each of these 8 ports 15 14 13 12 11 10 9 8 - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 22
Figure 3-5: Spine Card Block Diagram Backplane Interface 76 54 32 10 µC Serial link dual +12V Power & hot-swap circuits Sense & Clocks Control Physical-Level Conversion Circuits 7 6 5 4 3 2 1 0 Front-Panel Ports 3.3 Monitoring line card All system interconnects must have a - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 23
Ethernet ports also require a DHCP server on the network to give the card's microcontroller an IP address and subnet mask. The standard firmware supports two software interfaces over the Ethernet connections: HTTP Web server and SNMP. Figure 3-7: External Features of the Monitoring Line Card Item - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 24
- HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 25
the switch port is unlit, the switch is not powered, or there is a problem with the line card. The LEDs for the ports only light when there is a or on a port on a switch line card), use the following procedures to manually check the link: 1. If no green LEDs are illuminated, verify that the - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 26
POST (power-on self-test) errors in the HP ROM-Based Setup Utility User Guide. If the card is not visible to the system or is generating system hardware card on the interconnect to run diagnostic tools and check ports for problems. The easiest way to informally monitor events and counters on the - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 27
system to read the modified /etc/dhcpd.conf file, enter the following command: # service dhcpd restart The monitoring line card is hot swappable. After the restart command, you must of the line card is a supported version: # links -dump 172.28.0.132/all | grep version LED indications and - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 28
- HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 29
5 Replacing components Several components can be replaced in the Myrinet system interconnect: This chapter presents the following instructions relating to the replaceable components: • Installing a monitoring line card (Section 5.1). • Removing and installing an 8-port line card (Section 5.2). • - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 30
5.1.1 Rebooting the monitoring line card The easiest way to reboot a Myrinet interconnect is with a Web browser. In the following steps, uc denotes microcontroller: 1. Get the main page (http://switchname). 2. Click on the link "Slot 56 (big uc)." 3. Select "reboot" from the menu "commandForBigUc." - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 31
To insert a line card, reverse the previous steps. Be sure to start with the handles in their out position, and guide the line card slowly and gently into the card guides. The signal pins and ground blades on the high-density connectors on the line card align themselves by means of alignment - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 32
Note Do not operate a Myrinet interconnect for extended periods with missing line cards. Use blank panels to fill in any empty line card slots. The front of the enclosure needs to be closed both for the efficiency of the fan cooling and to avoid electromagnetic interference (EMI). 5.3 Replacing - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 33
Figure 5-6: Interconnect Fan Tray 5.4 Replacing an interconnect chassis The communications ports on the interconnect face toward the rear of the rack. The location of the interconnect in the rack depends on the specific configuration of your cluster. Up to four 14U interconnects might be installed - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 34
5. Disconnect and remove each fiber-optic cable until all ports are unplugged. Secure the cables out of the interconnect's removal path Note Observe the maximum bend radius for fiber-optic cables. Bending the cables too much can cause the glass core to break. A good rule of thumb is to never bend - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 35
Figure 5-8: Sliding Interconnect Chassis from the Rack Rear of Rack 8. Move the interconnect to a suitable work surface. Retain all screws for reuse. Use only the 6-32 x .375 in Phillips pan-head screws to attach the retainer brackets to the chassis. Use of incorrect screws might damage components. - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 36
5. At the rear of the rack, disconnect the cables after first ensuring that the cables are properly labeled for their appropriate ports. This is important if you are planning to replace a defective interconnect. Remove the cables from the cable management straps, and route them carefully out of the - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 37
the old chassis and immediately insert it into the corresponding slot in the new chassis. See Section 5.1 for installing a line monitoring card. See Section 5.2 for instructions for removing and replacing line cards. Replacing components 5-9 - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 38
Route the cables into the eight hook-and-loop cable management straps, located on both sides of the interconnect. Each cable management strap provides support for 24 fiber-optic cables. 7. At the front of the rack, reconnect the interconnect's power supply cable. 8. At the front of the rack, replace - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 39
6 Powering up procedure To power up a Myrinet system interconnect, plug the power cord into the standard C14 power jack labeled 100-127/200-240 V-. The power cord is rated 10A for the 5U interconnect, and 15A for the 9U chassis. There is no power switch on the enclosure, which prevents an - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 40
Characteristic Operating Nonoperating 5U Chassis 15% to 80% @ 104° F 90% @ 149° F 9U Chassis 6-2 Powering up procedure - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 41
Index Numbers and Special Characters 2XP PCI Rev. E, 2-3 Rev. F, 2-5 A application node, 1-1 B backplane power connector, 5-3 C chassis interconnect, 1-1, 5-1 configuration federated, 1-2 control node, 1-1 password, 4-3 crossbar switch, 3-3 D DHCP, 3-5, 4-2 monitoring line card, 5-1 downlink, 1-3 E - HP Cluster Platform Interconnects v2010 | Myrinet System Interconnect Guide - Page 42
spine card, 3-3 monitoring line card, 3-4, 4-2 installing, 5-1 network access, 4-2 rebooting, 5-2 reseating, 4-3 MPI, 1-1 N network access, 4-2 node-level switch, 1-2 nonfederated configuration, 1-3 P password default, 4-3 PCI card, 1-1, 2-1 identification label, 2-1 LEDs, 2-1 Rev. D, 2-2 Rev. E,
HP Cluster Platform
Myrinet System Interconnect Guide
Part Number: A-CPMYI-1A
May 2005
Revision/Update Information:
Version 1.0
Product Version:
HP Clusters
This manual describes the Myricom Myrinet system interconnects used in HP Cluster Platform
solutions.
Hewlett-Packard Company
Palo Alto, California