Het Formule 1 seizoen staat met 23 raceweekenden weer voor de deur. Van de GP van Bahrein tot aan de GP van Abu Dhabi zendt Ziggo het hele seizoen uit in 4K-kwaliteit. Uiteraard exclusief op kanaal Zi...
(firstly apologies for English, if we where talking this would be easy in Dutch but written is harder)
I have been suffering intermittent packet loss on my link for some time now mostly noticeable during zoom calls as that seems to be most affected by packet loss but the kids complain about netflix and gaming drops as well.
I have just move home (Dec 2020) so at first I thought it was my setup and cabling and I have been through several changes including changing all my own cabling and even replacing my firewall (ubiquiti dream machine pro) which is connected direct to the modem which is in bridge mode. I have already had a call to the helpdesk who sent me a new cable and splitter of which I don't use the splitter anyway but I did replace the cable (even though it's the same model and length of cable as the 2 month old one I had plugged in before, you never know).
I looked at the logs from the modem (Connectbox Giga) (I will attach captures) and errors don't appear to be a problem, the only log messages that I can see appear to be around issues I can explain either from me power off the box or there being a known issue. But the packet loss is a lot more frequent than this and effects all of my destinations, which can only lead me to think that it comes from something very close to my location, either my own network (which is why I replaced the devices and the cabling (all new equipment and cables)) or the very first hops on the provider network (as it impacts every location I monitor).
So I wanted to provide you guys with the data and see if somebody can suggest something other than there being a problem in the Ziggo network that I can look to solve
I will attach some of the capture here and then some more afterwards, I have my smokeping results which I will add
Attached are my smokeping results for the last 30+ days, and there was some recovery but it's never been reliably great (which probably means I should have brought it up some time ago)
I monitor my first and 2nd direct hops (you can probably find out roughly where I live from the IPs I guess) and then the ziggo website, to see if there was something related to size I do the 1st and 2nd hops with larger packets (to force fragmentation and see if MTU causes problems) and while the results are not great it doesn't seem to be that size is always a problem)
I also monitor some websites just to see if there was a difference related to specific destinations or some crazy thing with peering. As you can see from the results the times of bad packet loss are roughly the same for all of the graphs. This really makes me think this is coming down to something very local to me, either my own house or the very first hop to the provider
The last thing I checked was my own FW, and I initially did see some issues so I upgraded the HW. I monitor both the internal and external side of the FW (not even sure if it really helps but I do see difference in RTT so I figure I am seeing some processing delay between internal and external (even if it's just the time taken to process the firewall rules) what I do not see there is any packet loss at all (I guess that is to be hoped or I have bigger problems)
Could you post one or more pictures of your AOP (main wall outlet located in utility closet or living room), splitter(s), signal amplifier (if applicable) and cables?
Additionally, could you post your modem’s log data (Networklog tab)?
The log is reporting noise-related issues in both the downstream (sync timing errors) and upstream (T4 errors). Could you swap the current modem cable (which is outdated) to the one included with the modem (red/green connectors)?
Since the modem log contains both downstream and upstream errors (sync/T4) while the modem is connected directly to the AOP (and AOP and modem cable are both up-to-date), a moderator (Ziggo employee) should diagnose the signal from their perspective. Perhaps other modems in your neighborhood are exhibiting similar issues.
A moderator will visit this topic within 2-3 days. Please verify your address details and update them if necessary.
Hi @bcrocker74, I'd like to confirm that the issue is on our side of the modem. But since your modem is in bridge mode I would be pinging your router (which is currently blocked by the firewall).
Could you switch your modem back to router mode and allow pingrequests on it (I'm not sure if this is an option in the Connectbox Giga)? I can then start a pingtest from our server to your modem to see if that shows the same amount of loss.
If so, I'd advise a mechanic to visit your address to investigate.
If not, then I'm afraid the loss is caused in your internal network behind the modem.
Thanks for looking at this Mark, just to make sure, I have the stats from the same period inside my network as outside, never a single entry for packet loss or even close to delay on both sides of my FW so it would be a surprise if you guys come back with that as the solution.
Hi Ben, have you had any issues today? The pingtest from our server to your modem seems fine so far, no signal issues to be seen either.
Your issues could be caused by a lack of upstream capacity, at times the upstream traffic peaks causing traffic to be delayed. Streaming services like Zoom, Netflix and online games are likely impacted by this. Did you experience any issues between 13:10 and 13:20 today for example?
Worst seemed to be between 10 -12:30 today, there have been bad bits during the whole day however. In general it seems to be a little less bad since we changed to router mode from bridge mode which would seem interesting as we effectively added a hop that is processing the traffic when we did that. Any chance there's a known issue on the connectbox in bridge mode? The other interesting part is that large packets seem to be very much more impacted than small ones (attached are large graphs)
As for upstream capacity, the FW is rated at 3.5 G (Unifi Dream Machine Pro) and has never apparently seen a dropped packet in its counters, I have smoke ping graphs for both the internal and external side of the FW at the same times and there has never been an instance of loss
You're pinging random hops in our network which don't prioritize ICMP requests, the amount of packetloss on those hops doesn't translate to packetloss when you're trying to have a MS Teams meeting of playing a video-game (examples).
There is a minimal amount of packetloss in the connection from your modem to the internet, between 10:00 and 12:30 on tuesday:
The upstream capacity however (I should have specified this) is worrying. I'm talking about the upstream capacity of our hub (CMTS) where traffic of all the customers in your area is bundled and sent off to the next hub in the network.
You can see below the hub is working at full capacity for ~1,5 hours straight which means there is more traffic than the hub can handle at that time, causing delays which influences livestreams like Zoom meetings most.
I've escalated this to our Network Capacity management team, they will check my findings and let me know what the next step will be. If it does come down to a necessary capacity increase, that could take several months depending on the circumstances.
I will update you on the situation as soon as I have news.
Thankyou for looking into this and the explanation.
I chose the two hops on the network as they are consistently the next two hops in any path I go to, it was the best I could do to look at the PE devices. The trend they show is also shown by the other external locations for the same times (I chose 11 or 12 other locations mixed in europe and US to try and make sure that any local provider network problems didn't effect all the destinations). When I get serious loss on the local destinations it is also shown at the far away ones so it suggested it was something common to both paths which can only be the close by local connections (my own CPE and the connection to the PE). As the CPE is the only bit I can impact I already replaced the firewall and the network cabling and then the cable to the modem so there wasn't much else to look at.
I have turned the modem back in to bridge mode and opened firewall rules to allow ICMP, if you are not able to ping the device please let me know the range I should include in the rules and I will open it for you as well.
Again, really appreciate your help.
Hi Ben, I've sent you this info in private as well but just to confirm:
The capacity of our hub doesn't seem to be that critical, I don't see a lot of extra packetloss or increased ping during peak hours. I'll schedule a mechanic to visit your address asap, since I also don't see any packetloss to one of your neighbours.
You'll receive confirmation of the appointment by e-mail, if the date and timeslot doesn't suit you, you can click on the link in the e-mail to reschedule.