Sunday 23 June 2013

Avaya ERS4500 PoE failure / error and recovery.

The other day I was alerted to a POE problem on an Avaya / Nortel ERS4548GT-PWR when trawling through a syslog server. What drew my attention was the "PoE Main Power Usage On 512 Watts" log entry, which I had not seen before.

So I logged onto the switch and ran a few commands in an attempt to explore the problem.


ERS4500#show sys-info
Operation Mode:        Switch
MAC Address:           MA-CA-DD-RE-SS-00
PoE Module FW:         Unknown
Reset Count:           14
Last Reset Type:       Power Cycle
Power Status:          Primary Power
Autotopology:          Enabled
Pluggable Port 45:     None
Pluggable Port 46:     None
Pluggable Port 47:     None
Pluggable Port 48:     LX
Base Unit Selection:   Non-base unit using rear-panel switch
sysDescr:              Ethernet Routing Switch 4548GT-PWR
                       HW:12       FW:5.3.0.3   SW:v5.6.1.053
                       Mfg Date:20120121    HW Dev:  
Serial #:              SERIALNUMBER
Operational Software:  FW:5.3.0.3   SW:v5.6.1.053
Installed software:    FW:5.3.0.3   SW:v5.6.1.053
Operational license:   Base software
Installed license:     Base software
sysObjectID:           1.3.6.1.4.1.45.3.71.2
sysUpTime:             43 days, 01:22:00
sysNtpTime:            2013-05-28 04:07:52 GMT+00:00
sysServices:           6
sysContact:         

sysName:             

sysLocation:         

Stack sysAssetId:   
Unit sysAssetId:      

ERS4500#show poe-main-status
% Internal error

ERS4500#show poe-power-measurement
Port  Volt(V)  Current(mA)   Power(Watt)
----  -------  -----------   ---------------
% Internal error

ERS4500#show poe-port-status
      Admin      Current                               Limit         
Port  Status     Status              Classification   (Watts)  Priority
----  -------    -----------------   --------------   -------  --------
1     Enable     Detecting                 0          16       Low    
2     Enable     Detecting                 0          16       Low    
3     Enable     Detecting                 0          16       Low    
4     Enable     Detecting                 0          16       Low    
5     Enable     Detecting                 0          16       Low    
6     Enable     Detecting                 0          16       Low    
7     Enable     Detecting                 0          16       Low    
8     Enable     Detecting                 0          16       Low    
9     Enable     Detecting                 0          16       Low    
10    Enable     Detecting                 0          16       Low    
11    Enable     Detecting                 0          16       Low    
12    Enable     Detecting                 0          16       Low    
13    Enable     Detecting                 0          16       Low    
14    Enable     Detecting                 0          16       Low    
15    Enable     Detecting                 0          16       Low    
16    Enable     Detecting                 0          16       Low    
17    Enable     Detecting                 0          16       Low    
18    Enable     Detecting                 0          16       Low    
19    Enable     Detecting                 0          16       Low    
20    Enable     Detecting                 0          16       Low    
21    Enable     Detecting                 0          16       Low    
22    Enable     Detecting                 0          16       Low    
23    Enable     Detecting                 0          16       Low    
24    Enable     Detecting                 0          16       Low    
25    Enable     Detecting                 0          16       Low    
26    Enable     Detecting                 0          16       Low    
27    Enable     Detecting                 0          16       Low    
28    Enable     Detecting                 0          16       Low    
29    Enable     Detecting                 0          16       Low    
30    Enable     Detecting                 0          16       Low    
31    Enable     Detecting                 0          16       Low    
32    Enable     Detecting                 0          16       Low    
33    Enable     Detecting                 0          16       Low    
34    Enable     Detecting                 0          16       Low    
35    Enable     Detecting                 0          16       Low    
36    Enable     Detecting                 0          16       Low    
37    Enable     Detecting                 0          16       Low    
38    Enable     Detecting                 0          16       Low    
39    Enable     Detecting                 0          16       Low    
40    Enable     Detecting                 0          16       Low    
41    Enable     Detecting                 0          16       Low    
42    Enable     Detecting                 0          16       Low    
43    Enable     Detecting                 0          16       Low    
44    Enable     Detecting                 0          16       Low    
45    Enable     Detecting                 0          16       Low    
46    Enable     Detecting                 0          16       Low    
47    Enable     Detecting                 0          16       Low    
48    Enable     Detecting                 0          16       Low  

ERS4500#show log       
Type Time                          Idx  Src Message
---- ----------------------------- ---- --- -------
I    2013-05-22 19:51:50 GMT+00:00 35801     PoE Module Reset
I    2013-05-22 19:53:01 GMT+00:00 35802     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 19:53:01 GMT+00:00 35803     PoE Main Power Usage On 512 Watts
I    2013-05-22 19:55:25 GMT+00:00 35804     Trap:  pethPsePortOnOffNotification
I    2013-05-22 19:55:25 GMT+00:00 35805     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 19:56:35 GMT+00:00 35806     PoE Module Reset
I    2013-05-22 19:57:46 GMT+00:00 35807     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 19:57:46 GMT+00:00 35808     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:00:10 GMT+00:00 35809     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:00:10 GMT+00:00 35810     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:01:21 GMT+00:00 35811     PoE Module Reset
I    2013-05-22 20:02:32 GMT+00:00 35812     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 20:02:32 GMT+00:00 35813     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:04:55 GMT+00:00 35814     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:04:55 GMT+00:00 35815     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:06:06 GMT+00:00 35816     PoE Module Reset
I    2013-05-22 20:07:17 GMT+00:00 35817     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 20:07:17 GMT+00:00 35818     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:09:40 GMT+00:00 35819     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:09:40 GMT+00:00 35820     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:10:51 GMT+00:00 35821     PoE Module Reset
I    2013-05-22 20:12:02 GMT+00:00 35822     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 20:12:02 GMT+00:00 35823     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:14:25 GMT+00:00 35824     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:14:25 GMT+00:00 35825     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:15:36 GMT+00:00 35826     PoE Module Reset
I    2013-05-22 20:16:47 GMT+00:00 35827     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 20:16:47 GMT+00:00 35828     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:19:11 GMT+00:00 35829     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:19:11 GMT+00:00 35830     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:20:21 GMT+00:00 35831     PoE Module Reset
I    2013-05-22 20:21:32 GMT+00:00 35832     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 20:21:32 GMT+00:00 35833     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:23:56 GMT+00:00 35834     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:23:56 GMT+00:00 35835     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:25:06 GMT+00:00 35836     PoE Module Reset
I    2013-05-22 20:26:17 GMT+00:00 35837     Trap:  pethMainPowerUsageOnNotification
I    2013-05-22 20:26:17 GMT+00:00 35838     PoE Main Power Usage On 512 Watts
I    2013-05-22 20:28:41 GMT+00:00 35839     Trap:  pethPsePortOnOffNotification
I    2013-05-22 20:28:41 GMT+00:00 35840     PoE Port Detection Status:  Port 1 Status: Delivering Power
I    2013-05-22 20:29:52 GMT+00:00 35841     PoE Module Reset
I    2013-05-22 20:31:02 GMT+00:00 35842     Trap:  pethMainPowerUsageOnNotification
----More (q=Quit, space/return=Continue)----




So you can see from the above output that this is an ERS4548GT-PWR Rev:12 running software version 5.6.1.053. Along with this you can see that the switch cannot recognise the PoE module Firmware so it is displaying as "Unknown", the output of two of the PoE commands responding with "Internal Error" and finally the log is showing "PoE Module Reset" and associated logs almost every 5 minutes.

Avaya/Nortel obviously have seen this problem in the field before because over the years they have been releasing PoE firmware for their various switches in the event that the PoE firmware becomes corrupt. I found the following document on support.avaya.com, but had seen in earlier from the Nortel support website


ERS4500_PoE_Firmware_Readme_v1.pdf , 5.6.x


Looking for the PoE firmware on support.avaya.com, I was unable to find the correct PoE firmware for my model under the version 5.6.1 section, thinking it may be under the main release I checked the 5.6 section and there was still no PoE firmware available  for my switch model. After finally checking in the 5.6.2 section I was able to find the following two files.


4500_PoE_400b15.img , 5.6.x

PoE Firmware for ERS 4500-PWR (hardware revision 12 or later: 4526T-PWR HW:12, 4550T-PWR HW:12, 4524GT-PWR HW:12, 4526GTX-PWR HW:12, 4548GT-PWR HW:12)

4500_PoE_400.img , 5.6.x

PoE Firmware for ERS 4500 (up to hardware revision 11: 4526T-PWR HW:11, 4550T-PWR HW:11, 4524GT-PWR HW:11, 4526GTX-PWR HW:11, 4548GT-PWR HW:11 )
MD5 Checksum:5ed16e97d5bef9019cb9c00243603539


As I had a hardware revision #12 ERS4548GT-PWR I downloaded "4500_PoE_400b15.img" and copied it to my TFTP server.  I then attempted to download the file via TFTP to the switch


ERS4500#download address 10.0.0.1 poe_module_image 4500_PoE_400b15.img
% Invalid image


I had no idea why the switch was complaining that it was an "Invalid image", but I thought I would try the other file so I downloaded "4500_PoE_400.img", copied it to my TFTP server and attempted to download this new file via TFTP to the switch


ERS4500#download address 10.0.0.1 poe_module_image 4500_PoE_400.img
% Invalid image


I was confused, I thought there may be a bug or the files I downloaded were corrupt, but further investigation was required so I logged a job with Avaya. Within an hour I was advised to perform the following:
  1. Upgrade the software version on the switch to version 5.6.2 or later.
  2. Download the PoE firmware 4500_PoE_400b15.img to the switch.
  3. Leave the switch on the current version or downgrade the switch back to 5.6.1.

I took heed to their device, upgraded the switch to 5.6.2 and voilĂ  I was able to download the PoE firmware to the device, noting that the PoE firmware also requires a reboot.

Once the device was restarted there were no longer any PoE log errors, the switch could read the PoE module firmware and the PoE module commands were returning normal results.


Success!

No comments:

Post a Comment