There were two u1980 PBX exchanges deployed at two different sites. Each exchange was integrated with N Company, S Company and soft switch (SOFTEX3000). Everything was working fine but suddenly customer observed that there are huge number of call drops with every call that was placed between different exchanges. After some analyses we observed that every call drop was exactly after one minute. We tried the intra scenario in which two IP phones under one exchange tries to make a call and then i observed that there was no call drop. After some intense testing and analyses I realized that actually the call was not dropping but the caller and callee stops hearing the voice of each other and due to this other user things that the call is dropped and hanged the phone. After that I realized that if the voice is dropped after that if user holds the call and then resume it again after that users starts hearing the voice of each other and then after one minute again users stops hearing the voice. This issue was observed in almost every call and customer was really annoyed about this because there whole communication was down.
In the alarm information in e-sight there was alarm of frame slip error (CRC 4E1). But this alarm comes and after some seconds automatically restored to ok position. On the Local maintenance Terminal LMT there was no alarm being displayed on call logs and calls look like normal. But when we checked on IP phones between which voice was dropping after one minute we realized that the RTP stream was flowing from both phones but was not reaching at either site. From this we realize that may be in customer network some device is causing this problem. But because we are not allowed to get into the customer network we could not find the device which was causing this problem. Below is the alarm information which we get through LMT.
Device IP Alarm Level Device Type Alarm ID Alarm Name Alarm Content Recover Flag
172.*.*.3 Minor eSpace U1980 3*****264 E1/T1 CRC error Slot=2 Trunkport=1 Yes
172.*.*.3 Minor eSpace U1980 3*****266 E1/T1 frame location lost Slot=2 Trunkport=1 Yes
172.*.*.3 Minor eSpace U1980 3*****262 E1/T1 frame slip Slot=3 Trunkport=0 Yes
172.*.*.103 Minor eSpace U1980 3*****266 E1/T1 frame location lost Slot=2 Trunkport=1 Yes
172.*.*.103 Minor eSpace U1980 3*****262 E1/T1 frame slip Slot=2 Trunkport=1 Yes
After this to be sure we thought that may trunks between the exchanges are causing this issue. So we analyzed the trunks as well between exchanges but found nothing abnormal.
1. At first that because the issue is just in inter scenario may be the trunks are having some problem which I verified again but the issue was still there.
2. Then when checked the signal we realized that because the IP phone are sending the RTP stream but not able to reach the other end may be caused by some firewall or may be due to some other network device. But then again customer replied that other exchanges are working fine under the same network then why only your site is causing this problem. But we showed customer the related informatica after which they checked their network but still the issue didn’t resovled.
3. Then we tried to change the ip phones voice codecs that may be because of that voice is droping after one minute.
After all this issue was still there.
After testing different scenarios i was unable to resolve the issue. I have tried to change the trunk configuraiton, changed the voice codecs but still no use. Then there was one parameter in the configuration of trunk called heartbeat interval whose value was by default set to 120s. I tried to change this parameter to 255s but still same voice drop issue. But when i changed the heartbeat interval value to 30s the voice drop issue was resolved. I then tried to make many calls and no call showed me voice drop, the call behaviour went normal.
Whenever this kind of issue occur first try to check the parameters involved in it. Like R&D was saying that this issue is caused by the customer network and customer was saying that this issue is something else so we must always ensure that each and every parameter is ok. Some how this Heartbeat interval parameter has some effect on the voice drop issue whereas according to R&D this parameter has nothing to do with RTP stream which takes the voice packets at each other ends. So by changing this heartbeat interval to 30s i was able to resolve this issue. May be some network device in the network was causing this problem that the network device did not consider the system alive after 1 minute and stops the RTP stream due to some reason.
My suggestion is to first recheck each and every parameter becasue from this any parameter can have the effect on anything possible. So we should be aware of this possility as well.