Transient Connection Issue

Transient Connection Issue

[Ashish Madeti]

Hi All

Background: I am using Wireguard VPN to secure intra-server
communications among my 5-6 ubuntu servers sitting in different data
centers.

Today, we had a downtime of around 15 minutes because the server
running nginx was not able to connect to the web-application server
using the wireguard interface [0]. I ascertained that it was not a
connection issue between nginx server and web-application server by
trying to connect to web-application server via its public IP, which
worked [1]. I even tried restarting wireguard service [2] on both
nginx and web-application server but to no avail.
So, before investigating further, I decided to first route all the
traffic to a failover server (which was also a part of the VPN). It
took me around 5-10 minutes to pull the latest configuration and
application changes onto the failover server and then route all
traffic to it. Once our site was up, I again tried connecting to the
original web-application server from nginx server, using curl, but
this time it worked fine.

Can anybody help me understand the problem or anything I should try if
it happens again?

Please let me know if you need any more information.

[0] Tried via curl. curl 10.0.0.10:8080. Received the error
'Connection timed out'
[1] curl w.x.y.z:8080 returned the html content as expected.
[2] sudo service wg-quick@wg0 restart

Regards
-- 
Ashish Madeti

Re: Transient Connection Issue

[Tomcsanyi, Domonkos]

Hi Ashish,

With the amount of information given it is very hard to comment anything meaningful.
Have you gone through standard network connectivity issue investigation steps?
E.g.: does ping work? Do you have correct routes setup? What does wg show tell during downtime? What does tcpdump shows on the wire?

Cheers,
Domi


> 10.11.2020 dátummal, 0:21 időpontban Ashish Madeti <ashish@provakil.com> írta:
> 
> Hi All
> 
> Background: I am using Wireguard VPN to secure intra-server
> communications among my 5-6 ubuntu servers sitting in different data
> centers.
> 
> Today, we had a downtime of around 15 minutes because the server
> running nginx was not able to connect to the web-application server
> using the wireguard interface [0]. I ascertained that it was not a
> connection issue between nginx server and web-application server by
> trying to connect to web-application server via its public IP, which
> worked [1]. I even tried restarting wireguard service [2] on both
> nginx and web-application server but to no avail.
> So, before investigating further, I decided to first route all the
> traffic to a failover server (which was also a part of the VPN). It
> took me around 5-10 minutes to pull the latest configuration and
> application changes onto the failover server and then route all
> traffic to it. Once our site was up, I again tried connecting to the
> original web-application server from nginx server, using curl, but
> this time it worked fine.
> 
> Can anybody help me understand the problem or anything I should try if
> it happens again?
> 
> Please let me know if you need any more information.
> 
> [0] Tried via curl. curl 10.0.0.10:8080. Received the error
> 'Connection timed out'
> [1] curl w.x.y.z:8080 returned the html content as expected.
> [2] sudo service wg-quick@wg0 restart
> 
> Regards
> -- 
> Ashish Madeti

Re: Transient Connection Issue

[Mo Balaa]

Check your MTU across your paths. In my experience, transient connection issues are due to MTU oversize. 


> On Nov 10, 2020, at 2:20 AM, Tomcsanyi, Domonkos <domi@tomcsanyi.net> wrote:
> 
> Hi Ashish,
> 
> With the amount of information given it is very hard to comment anything meaningful.
> Have you gone through standard network connectivity issue investigation steps?
> E.g.: does ping work? Do you have correct routes setup? What does wg show tell during downtime? What does tcpdump shows on the wire?
> 
> Cheers,
> Domi
> 
> 
>> 10.11.2020 dátummal, 0:21 időpontban Ashish Madeti <ashish@provakil.com> írta:
>> 
>> Hi All
>> 
>> Background: I am using Wireguard VPN to secure intra-server
>> communications among my 5-6 ubuntu servers sitting in different data
>> centers.
>> 
>> Today, we had a downtime of around 15 minutes because the server
>> running nginx was not able to connect to the web-application server
>> using the wireguard interface [0]. I ascertained that it was not a
>> connection issue between nginx server and web-application server by
>> trying to connect to web-application server via its public IP, which
>> worked [1]. I even tried restarting wireguard service [2] on both
>> nginx and web-application server but to no avail.
>> So, before investigating further, I decided to first route all the
>> traffic to a failover server (which was also a part of the VPN). It
>> took me around 5-10 minutes to pull the latest configuration and
>> application changes onto the failover server and then route all
>> traffic to it. Once our site was up, I again tried connecting to the
>> original web-application server from nginx server, using curl, but
>> this time it worked fine.
>> 
>> Can anybody help me understand the problem or anything I should try if
>> it happens again?
>> 
>> Please let me know if you need any more information.
>> 
>> [0] Tried via curl. curl 10.0.0.10:8080. Received the error
>> 'Connection timed out'
>> [1] curl w.x.y.z:8080 returned the html content as expected.
>> [2] sudo service wg-quick@wg0 restart
>> 
>> Regards
>> -- 
>> Ashish Madeti