There were two u1980 PBX exchanges deployed at two different sites. Each exchange was integrated with Nortel, Siemens and soft switch (SOFTEX3000). Everything was working fine but suddenly customer observed that there are huge number of call drops with every call that was placed between different exchanges. After some analyses we observed that every call drop was exactly after one minute. We tried the intra scenario in which two IP phones under one exchange tries to make a call and then i observed that there was no call drop. After some intense testing and analyses I realized that actually the call was not dropping but the caller and callee stops hearing the voice of each other and due to this other user things that the call is dropped and hanged the phone. After that I realized that if the voice is dropped after that if user holds the call and then resume it again after that users starts hearing the voice of each other and then after one minute again users stops hearing the voice. This issue was observed in almost every call and customer was really annoyed about this because there whole communication was down.
In the alarm information in e-sight there was alarm of frame slip error (CRC 4E1). But this alarm comes and after some seconds automatically restored to ok position. On the Local maintenance Terminal LMT there was no alarm being displayed on call logs and calls look like normal. But when we took the traces on IP phones between which voice was dropping after one minute we realized that the RTP stream was flowing from both phones but was not reaching at either site. From this we realize that may be in customer network some device is causing this problem. But because we are not allowed to get into the customer network we could not find the device which was causing this problem. Below is the alarm information which we get through LMT.
Device IP Alarm Level Device Type Alarm ID Alarm Name Alarm Content Recover Flag
22.214.171.124 Minor eSpace U1980 306000264 E1/T1 CRC error Slot=2 Trunkport=1 Yes
126.96.36.199 Minor eSpace U1980 306000266 E1/T1 frame location lost Slot=2 Trunkport=1 Yes
188.8.131.52 Minor eSpace U1980 306000262 E1/T1 frame slip Slot=3 Trunkport=0 Yes
172.16.70.103 Minor eSpace U1980 306000266 E1/T1 frame location lost Slot=2 Trunkport=1 Yes
172.16.70.103 Minor eSpace U1980 306000262 E1/T1 frame slip Slot=2 Trunkport=1 Yes
After this to be sure we thought that may trunks between the exchanges are causing this issue. So we analyzed the trunks as well between exchanges but found nothing abnormal.
1. At first that because the issue is just in inter scenario may be the trunks are having some problem which I verified again but the issue was still there.
2. Then when checked the wire shark packets we realized that because the IP phone are sending the RTP stream but not able to reach the other end may be caused by some firewall or may be due to some other network device. But then again customer replied that other exchanges are working fine under the same network then why only your site is causing this problem. But we showed customer the traces of wireshark after which they checked their network but still the issue didn’t resovled.
3. Then our R&D replied us to try to change the ip phones voice codecs that may be because of that voice is droping after one minute.
After all this issue was still there.
After testing different scenarios i was unable to resolve the issue. I have tried to change the trunk configuraiton, changed the voice codecs but still no use. Then there was one parameter in the configuration of trunk called heartbeat interval whose value was by default set to 120s. I tried to change this parameter to 255s but still same voice drop issue. But when i changed the heartbeat interval value to 30s the voice drop issue was resolved. I then tried to make many calls and no call showed me voice drop, the call behaviour went normal.