Mellanox Technologies ConnectX-5 Ex Manuale utente

Mellanox Technologies

www.mellanox.com

Mellanox ConnectX®-5 and ConnectX®-5 VPI Adapter

Cards User Manual

P/N:

MCX555A-ECAT, MCX556A-ECAT, MCX556A-EDAT

Rev 1.5

Downloaded from Arrow.com.

Doc #: MLNX-15-51362 5Mellanox Technologies

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,

CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,

GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, LinkX®, Kotura®, Kotura logo, Mellanox CloudRack®,

Mellanox CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox

Open Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox

ScalableHPC®, Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox

Virtual Modular Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, NPS®, Open Ethernet logo,

PhyX®, PlatformX®, PSIPHY® , SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The

Generation of Open Ethernet logo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and

Voltaire logo are registered trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks

NOTE:

THIS HARDWARE , SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT (S) ) AND ITS RELATED

DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-ISﺴ WITH ALL FAULTS OF ANY

KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT

USE THE PRODUCTS IN DESIGNATED SOLUTIONS . THE CUSTOMER 'S MANUFACTURING TEST

ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY

QUALIFY THE PRODUCT (S) AND/OR THE SYSTEM USING IT . THEREFORE , MELLANOX TECHNOLOGIES

CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY . ANY EXPRESS OR IMPLIED WARRANTIES , INCLUDING, BUT NOT LIMITED TO , THE

IMPLIED WARRANTIES OF MERCHANTABILITY , FITNESS FOR A PARTICULAR PURPOSE AND

NONINFRINGEMENT ARE DISCLAIMED . IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR

ANY THIRD PARTIES FOR ANY DIRECT , INDIRECT, SPECIAL , EXEMPLARY , OR CONSEQUENTIAL

DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO , PAYMENT FOR PROCUREMENT OF

SUBSTITUTE GOODS OR SERVICES ; LOSS OF USE , DATA , OR PROFITS ; OR BUSINESS INTERRUPTION )

HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY , WHETHER IN CONTRACT , STRICT LIABILITY ,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE ) ARISING IN ANY WAY FROM THE USE OF THE

PRODUCT (S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH

DAMAGE .

Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.5 1Mellanox Technologies

Table of Contents

Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

About This Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1 Product Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2 Features and Benefits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3 Operating Systems/Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.4 Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Chapter 2 Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.1 InfiniBand Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2 Ethernet QSFP28 Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 PCI Express Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4 LED Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 3 Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.2 Operating Systems/Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1.3 Software Stacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.3 Pre-Installation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.4 Bracket Installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.4.1 Removing the Existing Bracket. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.4.2 Installing the New Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.5 Card Installation Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.6 Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.6.1 Cable Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.7 Adapter Card Un-installation Instructions . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.7.1 Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.7.2 Card Un-installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.8 Identify the Card in Your System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.8.1 On Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.8.2 On Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Chapter 4 Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.1.2 Downloading Mellanox OFED. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.5 2Mellanox Technologies

4.1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Mellanox OFED 20

4.1.3.1 Installation Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.1.3.2 Installation Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4.1.3.3 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.1.3.4 Installation Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.1.3.5 openibd Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.1.3.6 Driver Load Upon System Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1.3.7 mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1.4 Uninstalling Mellanox OFED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4.1.5 Installing MLNX_OFED Using YUM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1.5.1 Setting up MLNX_OFED YUM Repository . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.1.5.2 Installing MLNX_OFED Using the YUM Tool . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.5.3 Uninstalling Mellanox OFED Using the YUM Tool. . . . . . . . . . . . . . . . . . . 29

4.1.5.4 Installing MLNX_OFED Using apt-get Tool. . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1.5.5 Setting up MLNX_OFED apt-get Repository . . . . . . . . . . . . . . . . . . . . . . . 30

4.1.5.6 Installing MLNX_OFED Using the apt-get Tool . . . . . . . . . . . . . . . . . . . . . 30

4.1.5.7 Uninstalling Mellanox OFED Using the apt-get Tool. . . . . . . . . . . . . . . . . 31

4.1.6 Updating Firmware After Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.6.1 Updating the Device Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1.6.2 Updating the Device Manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.6.3 Updating the Device Firmware Automatically upon System Boot . . . . . 32

4.1.7 UEFI Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4.1.7.1 Enrolling Mellanox's x.509 Public Key On your Systems . . . . . . . . . . . . . 33

4.1.7.2 Removing Signature from kernel Modules . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1.8 Performance Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

4.2 Windows Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.2.1 Hardware and Software Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.2.2 Downloading Mellanox WinOF-2 Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

4.2.3 Installing Mellanox WinOF-2 Driver. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.2.3.1 Attended Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.2.3.2 Unattended Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.2.4 Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.2.5 Extracting Files Without Running Installation. . . . . . . . . . . . . . . . . . . . . . . 43

4.2.6 Uninstalling Mellanox WinOF-2 Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.6.1 Attended Uninstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.6.2 Unattended Uninstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.7 Firmware Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.8 Deploying the Driver on a Nano Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.8.1 Offline Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.8.2 Online Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Chapter 5 Updating Adapter Card Firmware. . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.1 Firmware Update Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.5 3Mellanox Technologies

Chapter 6 Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6.1 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

6.2 Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

6.3 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Chapter 7 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

7.1 MCX555A-ECAT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

7.2 MCX556A-ECAT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

7.3 MCX556A-EDAT Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

7.4 Airflow Specifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.5 Adapter Card LED Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

7.6 Board Mechanical Drawing and Dimensions . . . . . . . . . . . . . . . . . . . . . . . 56

7.7 Bracket Mechanical Drawing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Appendix A Finding the GUID/MAC and Serial Number on the Adapter Card 58

Appendix B Safety Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

B.1 Safety Warnings in English . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

B.2 Avertissements de sécurité d’installation (Warnings in French) . . . . . 62

B.3 Sicherheitshinweise (Warnings in German) . . . . . . . . . . . . . . . . . . . . . 64

B.4 Advertencias de seguridad para la instalación (Warnings in Spanish) 66

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.5 1Mellanox Technologies

List of Tables

Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Table 2: Documents List. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Table 3: Single and Dual-port VPI Adapter Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Table 4: Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Table 5: Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Table 6: Installation Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Table 7: mlnxofedinstall Return Codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Table 8: Hardware and Software Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Table 9: General Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Table 10: Linux Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Table 11: Windows Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Table 12: MCX555A-ECAT Specification Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Table 13: MCX556A-ECAT Specification Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Table 14: MCX556A-EDAT Specification Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Table 15: Airflow Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Table 16: Physical and Logical Link Indications (Ethernet Mode). . . . . . . . . . . . . . . . . . . . . . 55

Table 17: Physical and Logical Link Indications (InfiniBand Mode) . . . . . . . . . . . . . . . . . . . . 55

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.5 1Mellanox Technologies

List of Figures

Figure 1: EMI fingers on QSFP28 Cage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Figure 2: PCI Device (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Figure 3: Installation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Figure 4: Mechanical Drawing of the Dual-port Adapter Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Figure 5: Dual-port Tall Bracket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Figure 6: MCX555A-ECAT Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Figure 7: MCX556A-ECAT Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Figure 8: MCX556A-EDAT Board Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.

Rev 1.5 2Mellanox Technologies

Revision History

This document was printed on November 1, 2018.

Table 1 - Revision History Table

Date Rev Comments/Changes

July 2018 1.5 • Updated Adapter Card LED Operations on page 55

January 2018 1.4 • Updated Adapter Card LED Operations on page 55

• Added a note to Hardware on page 10

November 2017 1.3 • Updated Adapter Card LED Operations on page 55

October 2017 1.2 • Updated the following specification tables:

• MCX555A-ECAT Specification Table on page 52

• MCX556A-ECAT Specification Table on page 53

• MCX556A-EDAT Specification Table on page 57

• Updated Linux on page 19

April 2017 1.1 • Updated Product Overview on page 5

• Updated Card Installation Instructions on page 12

March 2017 1.0 First release

Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.Downloaded from Arrow.com.