Ongoing Packet Loss

(firstly apologies for English, if we where talking this would be easy in Dutch but written is harder)

 

I have been suffering intermittent packet loss on my link for some time now mostly noticeable during zoom calls as that seems to be most affected by packet loss but the kids complain about netflix and gaming drops as well. 

 

I have just move home (Dec 2020)  so at first I thought it was my setup and cabling and I have been through several changes including changing all my own cabling and even replacing my firewall (ubiquiti dream machine pro) which is connected direct to the modem which is in bridge mode.  I have already had a call to the helpdesk who sent me a new cable and splitter of which I don't use the splitter anyway but I did replace the cable (even though it's the same model and length of cable as the 2 month old one I had plugged in before, you never know).

 

I looked at the logs from the modem (Connectbox Giga) (I will attach captures) and errors don't appear to be a problem, the only log messages that I can see appear to be around issues I can explain either from me power off the box or there being a known issue. But the packet loss is a lot more frequent than this and effects all of my destinations, which can only lead me to think that it comes from something very close to my location, either my own network (which is why I replaced the devices and the cabling (all new equipment and cables)) or the very first hops on the provider network (as it impacts every location I monitor).

 

So I wanted to provide you guys with the data and see if somebody can suggest something other than there being a problem in the Ziggo network that I can look to solve

 

I will attach some of the capture here and then some more afterwards, I have my smokeping results which I will add

 

bcrocker74
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes
23 Reacties 23

Attached are my smokeping results for the last 30+ days, and there was some recovery but it's never been reliably great (which probably means I should have brought it up some time ago)

 

I monitor my first and 2nd direct hops (you can probably find out roughly where I live from the IPs I guess) and then the ziggo website, to see if there was something related to size I do the 1st and 2nd hops with larger packets (to force fragmentation and see if MTU causes problems) and while the results are not great it doesn't seem to be that size is always a problem)   

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

I also monitor some websites just to see if there was a difference related to specific destinations or some crazy thing with peering. As you can see from the results the times of bad packet loss are roughly the same for all of the graphs. This really makes me think this is coming down to something very local to me, either my own house or the very first hop to the provider

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

The last thing I checked was my own FW, and I initially did see some issues so I upgraded the HW. I monitor both the internal and external side of the FW (not even sure if it really helps but I do see difference in RTT so I figure I am seeing some processing delay between internal and external (even if it's just the time taken to process the firewall rules) what I do not see there is any packet loss at all (I guess that is to be hoped or I have bigger problems)

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Could you post one or more pictures of your AOP (main wall outlet located in utility closet or living room), splitter(s), signal amplifier (if applicable) and cables?

Additionally, could you post your modem’s log data (Networklog tab)?

tobiastheebe
Expert
Expert
  • 1154Reacties
  • 70Oplossingen
  • 356Likes

Doh, I meant to include the log last night, here it is, and the picture of the wall box, thankyou for looking Tobias

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

The log is reporting noise-related issues in both the downstream (sync timing errors) and upstream (T4 errors). Could you swap the current modem cable (which is outdated) to the one included with the modem (red/green connectors)?

tobiastheebe
Expert
Expert
  • 1154Reacties
  • 70Oplossingen
  • 356Likes

Done, I tried before and had no difference/improvement so I went for the shortest one 😄 Will post again when I have findings

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Could you re-post the downstream (including DOCSIS 3.1) and upstream data after a couple of hours?

tobiastheebe
Expert
Expert
  • 1154Reacties
  • 70Oplossingen
  • 356Likes

Capture of the logs, there's nothing in the network log since 10/02  so I didn't include

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

This is my ping results to the 1st hop in the provider network from thismorning. I am on the 2nd of the 3 cables I have now and none of them have seemed to make a difference.

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Since the modem log contains both downstream and upstream errors (sync/T4) while the modem is connected directly to the AOP (and AOP and modem cable are both up-to-date), a moderator (Ziggo employee) should diagnose the signal from their perspective. Perhaps other modems in your neighborhood are exhibiting similar issues.

 

A moderator will visit this topic within 2-3 days. Please verify your address details and update them if necessary.

tobiastheebe
Expert
Expert
  • 1154Reacties
  • 70Oplossingen
  • 356Likes

Thankyou Tobias, much appreciated

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Hi @bcrocker74, I'd like to confirm that the issue is on our side of the modem. But since your modem is in bridge mode I would be pinging your router (which is currently blocked by the firewall).

 

Could you switch your modem back to router mode and allow pingrequests on it (I'm not sure if this is an option in the Connectbox Giga)? I can then start a pingtest from our server to your modem to see if that shows the same amount of loss.

 

If so, I'd advise a mechanic to visit your address to investigate.

If not, then I'm afraid the loss is caused in your internal network behind the modem.

Mark Z
Community Moderator
Community Moderator
  • 5144Reacties
  • 585Oplossingen
  • 1858Likes

Hi Mark,

 

Change has been made, I am in router mode again. Regards

 

Ben

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Thanks for looking at this Mark, just to make sure, I have the stats from the same period inside my network as outside, never a single entry for packet loss or even close to delay on both sides of my FW so it would be a surprise if you guys come back with that as the solution.

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Hi Ben, have you had any issues today? The pingtest from our server to your modem seems fine so far, no signal issues to be seen either.

Knipsel.JPG

 

Your issues could be caused by a lack of upstream capacity, at times the upstream traffic peaks causing traffic to be delayed. Streaming services like Zoom, Netflix and online games are likely impacted by this. Did you experience any issues between 13:10 and 13:20 today for example?

Mark Z
Community Moderator
Community Moderator
  • 5144Reacties
  • 585Oplossingen
  • 1858Likes

Hi Mark,

 

Worst seemed to be between 10 -12:30 today, there have been bad bits during the whole day however. In general it seems to be a little less bad since we changed to router mode from bridge mode which would seem interesting as we effectively added a hop that is processing the traffic when we did that. Any chance there's a known issue on the connectbox in bridge mode? The other interesting part is that large packets seem to be very much more impacted than small ones (attached are large graphs)

 

As for upstream capacity, the FW is rated at 3.5 G (Unifi Dream Machine Pro) and has never apparently seen a dropped packet in its counters, I have smoke ping graphs for both the internal and external side of the FW at the same times and there has never been an instance of loss

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

You're pinging random hops in our network which don't prioritize ICMP requests, the amount of packetloss on those hops doesn't translate to packetloss when you're trying to have a MS Teams meeting of playing a video-game (examples).

There is a minimal amount of packetloss in the connection from your modem to the internet, between 10:00 and 12:30 on tuesday:
Knipsel.JPG

 

The upstream capacity however (I should have specified this) is worrying. I'm talking about the upstream capacity of our hub (CMTS) where traffic of all the customers in your area is bundled and sent off to the next hub in the network.

You can see below the hub is working at full capacity for ~1,5 hours straight which means there is more traffic than the hub can handle at that time, causing delays which influences livestreams like Zoom meetings most.

image.png

 

I've escalated this to our Network Capacity management team, they will check my findings and let me know what the next step will be. If it does come down to a necessary capacity increase, that could take several months depending on the circumstances.

 

I will update you on the situation as soon as I have news.

Mark Z
Community Moderator
Community Moderator
  • 5144Reacties
  • 585Oplossingen
  • 1858Likes

Hi Mark,

 

Thankyou for looking into this and the explanation.

 

I chose the two hops on the network as they are consistently the next two hops in any path I go to, it was the best I could do to look at the PE devices. The trend they show is also shown by the other external locations for the same times (I chose 11 or 12 other locations mixed in europe and US to try and make sure that any local provider network problems didn't effect all the destinations). When I get serious loss on the local destinations it is also shown at the far away ones so it suggested it was something common to both paths which can only be the close by local connections (my own CPE and the connection to the PE). As the CPE is the only bit I can impact I already replaced the firewall and the network cabling and then the cable to the modem so there wasn't much else to look at.

 

I have turned the modem back in to bridge mode and opened firewall rules to allow ICMP, if you are not able to ping the device please let me know the range I should include in the rules and I will open it for you as well.

 

Again, really appreciate your help.

bcrocker74
topicstarter
Meedenker
  • 13Reacties
  • 0Oplossingen
  • 0Likes

Hi Ben, I've sent you this info in private as well but just to confirm:
 

The capacity of our hub doesn't seem to be that critical, I don't see a lot of extra packetloss or increased ping during peak hours. I'll schedule a mechanic to visit your address asap, since I also don't see any packetloss to one of your neighbours.

You'll receive confirmation of the appointment by e-mail, if the date and timeslot doesn't suit you, you can click on the link in the e-mail to reschedule.

Mark Z
Community Moderator
Community Moderator
  • 5144Reacties
  • 585Oplossingen
  • 1858Likes

You're welcome! Do let me know if you have any further questions.

tobiastheebe
Expert
Expert
  • 1154Reacties
  • 70Oplossingen
  • 356Likes

Ik heb de moderators een seintje gegeven.

KBX458
Super Expert
Super Expert
  • 5125Reacties
  • 213Oplossingen
  • 1767Likes

You definitely need a Moderator to look into this matter. I will inform my colleague who can translate the data into an advice or action. This will be next Monday. 

Kind regards

Sander Z
Community Developer
Community Developer
  • 2473Reacties
  • 254Oplossingen
  • 1382Likes

Uitgelicht topic