For this node (a CH121 V3) we replaced the motherboard 2 months prior to this issue. Now after 2 months customer face the following issue:
Other nodes in same chassis are working fine,
"I cant get this node to boot over PXE, when booting it cannot recognize any medium connected for NIC. It is the same node we replaced motherboard on, 2 months
ago. i dont know if this fault has been ongoing since then, or if this has occured recently.
I noticed one thing that differ when comparing this faulty node with another, "PXE slot" of this one is with 0400 and the other node is
0800 see screenshots attached below, why does it differ?
I have tried to reset ibmc locally and reset MMboard for whole enclosure but no success.
The first step was to ask if the customer can ping the PXE Server IP.
All the nodes in the chassis are ping-able and they can communicate with "Bright" PXE server => This is not a network problem
Second step was to as customer to provide the iBMC logs so we can see if perhaps there is a hardware issue. As customer didn't have a valid IP address for the node, he could not access the node's GUI and collect the logs. As such we recommended to collect the logs for the entire chassis.
The customer provided the logs for the entire E9000 chassis and we proceeded at analyzing the 2nd node.
After analyzing the logs from the "affected" node and comparing it with another node we found that MEZZ Card MZ310 was in slot 2 while the MEZZ Card MZ611 was in slot 1.
By comparison the healthy node had the MEZZ Card MZ310 in slot 1 and MEZZ Card MZ611 in slot 2.
The root cause of this issue is the fact that when 2 months prior the motherboard was replaced the MEZZ cards were swapped, MZ310 in slot 2 instead of being installed in slot 1 and MZ611 in slot 1 instead of being installed in slot 2.
The picture bellow represents the "affected" node:
The picture bellow represents a healthy node:
The Solution for this issue is to simply swap MEZZ Cards. In this case put MZ310 in slot 1 and MZ611 in slot 2.
After a motherboard has been replaced, make sure that the MEZZ Cards are in their proper slots as to avoid future issues such as this.