T1 !
It is 2021. T1? What does that mean?
Ha, well.. T1's were something that were popular when I was just getting into the industry in the late 1990's and early 2000's. But they were beginning to loose favor by 2010 as SIP / VoIP options began to become more economical and fiber backbones became more widespread.
But one thing can not be denied. In terms of robustness, a T1 line from point to point is typically VERY solid.
Where I work, we use T1's for several point to point critical infrastructure systems. We were actually still running a Cisco SONET fiber ring 15454 up until April 2021! The T1's would route over that system out to their various locations. Prior to this we used copper hardline and CO repeaters.
This system was Aging. I fully believe the Cisco SONET 15454 was made useless by the Y2K bug fix, as January 1st, 2020 the SONET user interface ceased to be functional, and it corrupted the kernel on our monitoring workstation. For over a year the SONET ran without the ability for us to maintain it or read logs, or do anything. Luckily it was solid enough to do it's job without complaining much. For a system that was 10 years beyond it's useful life, that is a huge show of hardness.
This year it was my goal to work all of the major systems off of the aging SONET system. And my self imposed deadline was April 2021. Now keep in mind, this is a project I started and worked on on my own. My workplace was so caught up in their own red tape and lack of understanding that - while this was on their drawing board, nothing was being done about it.
I disconnected the SONET in the first week of April! My new system, using an ERPS ring with AdTran 8044M's was complete, and all of my T1's were transferred over and functional enough to be useful. I completed my goal, and just in time. As a few hours before the last T1's were to be moved off of the old SONET, that SONET failed, and those T1's went down! Luckily the 8044M was in place as was the cabling to quickly transfer the T1's over to the new system and get it back online.
But then began a strange issue. See I have never really dealt with setting up a CO network of T1's. Usually when I dealt with T1's, I dealt with client side T1's, and the CO side of things was already set up. So yeah I was still learning when I set up this super important critical infrastructure system. I had read that there was only supposed to be one clock source, but I had 2 8044's at my CO location. Each of which handling 8 T1's. The T1's like I said, are point to point, so they all originate from the CO location, and head out to various locations.
I had wrongly set the 2nd CO 8044, to packet timing client mode, off of the first 8044.
So the 8 T1's originating from the 2nd 8044 at the CO all had packet timing issues. But I didn't quite catch that right away. For one, they all seemed to work... Kind of. Every 12 minutes or so, I would get an interruption from one of the Fire Department audio lines, which would unsquelch their radio system.
With several other projects on the go, it took me a while to come back to this issue to revisit it. But I did have a theory it was clock or timing related.
Tonight I was checking the units and saw that the T1 channel banks's all used to have clock inputs but the wire on the wire wrap was cut. So I was thinking okay if the CO T1's had timing at one time, then why don't they now? Then I realized... The SONET was the timing, and then I realized that the packet timing on the AdTran 8044 was also the timing. I don't know why that didn't quite click in until just now. But it hit me like a pallet of falling bricks. Hell yeah.
So I logged in and checked, and sure enough the 8 locations off of the 2nd 8044 were all showing packet alarms. So I fixed that up, by making the 2nd 8044 at the CO it's own packet server instead of a client, and then set up the other locations to point to that unit that they are pointed to anyhow with the T1, to look for packet timing there. And yeah... that fixed it :)
Woo. Fun stuff.
I hope my T1's live for many many more years. lol