Problem Description
Trigger condition:
1. SSN1EAS2 transmission board use FPGA 110.
2. Cross-connect boards in slot 10 are active cross-connect boards.
3. The PCB of a cross-connect board is SSN1SXCS1, SST1PSXCSA1, SST1PSXCSA, SST2PSXCS, or SSN2SXCS.
4. When the proceeding three conditions are all met, the problem occurs occasionally (about 10% probability) and is determined by SSN1EAS2 boards and cross-connect boards.
Note:
During an upgrade on the live network, the problem is easily triggered when the cross-connect board in slot 10 replaces that in slot 9 as an active one.
When an SSN1EAS2 board works with a cross-connect board whose PCB is SSN1SXCS1, SST1PSXCSA1, SST1PSXCSA, SST2PSXCS, or SSN2SXCS, there is a high probability that the problem may occur. When an SSN1EAS2 board works with other types of cross-connect boards, this problem never occurs on live networks and in test environment. For details about risky types of cross-connect boards, see the Risky Types of Cross-Connect Boards That May Encounter Header Jitters on FPGA 110 of SSN1EAS2 Boards When They Work Together.
Symptoms:
1. Some packets of Ethernet services on an SSN1EAS2 board are lost, or Ethernet services are interrupted.
2. The board may repeatedly or occasionally reports service alarms related to huawei SDH or GFP services, such as B3_SD, HP_UNEQ, HP_RDI, T_LO**, ALM_GFP_DLFD, and FCS_ERR.
3. The active and standby cross-connect boards may report BUS_ERR alarms simultaneously, and alarm parameters indicate that the SSN1EAS2 board caused the alarm.
Cold reset cannot resolve the problem nor trigger the problem.
Identification methods:
1. If all of the following conditions are met, these fault symptoms are most probably caused by the SSN1EAS2 board:
− An SSN1EAS2 board uses FPGA 110.
− The cross-connect board in slot 10 is the active one.
− The cross-connect board is the type listed in the attachment Risky Types of Cross-Connect Boards That May Encounter Header Jitters on FPGA 110 of SSN1EAS2 Boards When They Work Together.
2. After the cross-connect board in slot 10 replaces that in slot 9, if fault symptoms on the SSN1EAS2 board are cleared, the problem is caused by the SSN1EAS2 board.
[Root Cause]
The design of the FPGA on an SSN1EAS2 board for cross-clock-domain has bugs. When the cross-connect board in slot 10 function as the active one, the headers output to the MAPPER and VSC9128 chips have jitters at the period of 77 Mbit/s. As a result, the VSC9128 chips fail to correctly receive service signals from the cross-connect boards. In addition, the services signals transmitted to the cross-boards may have jitters, resulting in bidirectional packet loss or service interruption.
[Impact and Risk]
Some packets of Ethernet services are lost, or Ethernet services are interrupted in the upstream and downstream direction.
Measures and Solutions
Recovery measures:
Replace the cross-connect board in slot 10 with that in slot 9 as the active one.
Workarounds:
When an SSN1EAS2 board uses FPGA 110 or earlier, avoid using the cross-connect board in slot 10 as the active cross-connect.
Preventive Solutions:
1. Upgrade the FPGA (BOM: 05020AAE) used on an SSN1EAS2 board to version 120 or later, because the FPGAs resolve the design bugs for cross-clock-domain.
− For V100R008 and V100R009 versions, upgrade the device to V100R010C03SPC203 or later.
− For V100R010 versions, upgrade the device to V100R010C03SPC203 or later.
− For V200R011 versions, upgrade the device to V200R011C02SPC106 or later.
− For V200R012 versions, upgrade the device to V200R012C00SPC101/V200R012C01 or later.
During an upgrade, the board version needs to match the device version specified in the version mapping.
On the NE whose version is V100R010C03SPC202, the SSN1EAS2 board can use the software of V100R010C03SPC203 in a weak mapping mode. The software includes BIOS, board software, FPGA, and EPLD.
2. SSN3EAS2 boards can be used. Both SSN1EAS2 and SSN3EAS2 boards are 10GE Ethernet service processing boards. When using an SSN3EAS2 board, ensure that the device version supports the SSN3EAS2 board.
