Dhcpd randomly returns the IP address or the backup IP

Discussion:

Wayne Gemmell | Connect

2021-03-19 06:22:27 UTC

Hi

When my dhcp server gets a DHCPRELEASE followed by a DHCPDISCOVER then it
randomly allocates the IP address or the backup IP address that are
assigned to the host in the lease file to the host. The assignments always
got to the correct host and the mac address doesn't change.

When this backup IP address is allocated then dynamic DNS isn't updated and
my container is unreachable.

I've put all the relevant details in the link below but am more than happy
enough to furnish anything that is missing. Can anyone help please?

https://gitlab.isc.org/isc-projects/dhcp/-/issues/172

<https://connect-mobile.co.za/mobile_marketing_solutions/>

Wayne Gemmell | Connect

2021-03-31 12:07:50 UTC

Permalink

I made a mistake and that file has all the mysql and dhcp in it. Oops. Oh
well.

I have no doubt it wil lhappen again because this happens on all the hosts
I test on.

<https://connect-mobile.co.za/mobile_marketing_solutions/>

On 31 Mar 2021, at 13.42, Wayne Gemmell | Connect <
Hi
I've posted a pcap file of the process. There is nothing hitting at
exactly the same moment but I'm digging into how to trigger this
differently.
https://drive.google.com/file/d/1mYIIlhq7JPdszT1zeasREIYcP4Fnj1lj/view?usp=sharing
I will take a look, but as Glenn mentioned this will likely be down to the
process the code uses to select a suitable and available address.
Stop the servers, edit the leases file to remove any mention of the
undesired address and restart the servers. They will initialise using the
information in the leases file and then the host has only one address that
it has ever used.
This should work until for some reason it gets a different address once -
then this will start again.
<https://connect-mobile.co.za/mobile_marketing_solutions/>

Hi
These two addresses are both in the same pool, so nothing illegal is
happening. It is still odd that it sometimes gets one address and then the
other, this should not happen.
Consider the leases file as a historical document, each time something
happens, the leases file gets added to. Once in a while it gets regenerated
with just the current status of all leases.
So for this sequence of events you should be able to reconstruct what
happened by looking into this file.
One thing that comes to mind is timing. The release and the discover
happens in the same second, so what if the server could not write to the
leases file before being asked about a new lease - this is not extending an
existing lease as the discover indicates that the device does not have an
address at this moment.
Would it be possible to add a pause of a few seconds between the release
and the discover? at least for an experiment. If that works, some timing is
at the bottom of this.
Possibly a wireshark trace could reveal the exact timing of these two
messages.
The leases file snippet did also not have both addresses as active, which
could uncover other interesting things.
--
Best regards
Sten Carlsen
A pessimist is a person that can find a problem for every solution.
On 29 Mar 2021, at 12.05, Wayne Gemmell | Connect <
Hi Sten
One of the host I've been testing with in the kannel pool alternates
between the following
inet 10.3.2.36/16
inet 10.3.2.60/16
Leases look as follows. I don't really understand why each would be both
free and active.
https://pastebin.com/UP6M1Lic
DHCP Lease list on responsible peer. The other one has similar data but
no host names. https://pastebin.com/WX3NwkM2
Here's snippet of the lease file https://pastebin.com/fKFiZpQ0
<https://connect-mobile.co.za/mobile_marketing_solutions/>

Thanks
Sten
On 26 Mar 2021, at 08.43, Wayne Gemmell | Connect <
Thanks Sten
I've moved my class out of the subnet and the issue still persists.
Well, that is one can of worms out of the picture.
What are the two addresses?
<https://connect-mobile.co.za/mobile_marketing_solutions/>

You define the class kannel inside the subnet, so if there are any
other subnets, it will be defined there as well. Class definitions are
global always, defining them inside a subnet may have some "interesting"
side effects as some options are taken from that subnet even if the host
gets an address in a different subnet - routers is one of them.
So one thought is, if the backup address is not in this subnet, the
host will be given incorrect information.
--
Best regards
Sten Carlsen
--------------------------------------------------
Aoccdrnig to rseerach at Cmabrigde Uinervtisy,
it deosn't mttaer in waht oredr the ltteers in a
wrod are, the olny iprmoatnt tihng is taht the
frist and lsat lteter be at the rghit pclae.
The rset can be a ttoal mses and you can slitl
raed it wotihut porbelm. Tihs is bcuseae the
hmuan mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. Amzanig, huh?
--------------------------------------------------
On 25 Mar 2021, at 13.01, Wayne Gemmell | Connect <
I tried it. No change. The issue still persists.
<https://connect-mobile.co.za/mobile_marketing_solutions/>
On Tue, 23 Mar 2021 at 16:19, Wayne Gemmell | Connect <

I'll give it a try, thanks.
<https://connect-mobile.co.za/mobile_marketing_solutions/>
On Tue, 23 Mar 2021 at 01:32, Bill Shirley <

update-optimization off;
Bill
Hi
When my dhcp server gets a DHCPRELEASE followed by a DHCPDISCOVER
then it randomly allocates the IP address or the backup IP address that are
assigned to the host in the lease file to the host. The assignments always
got to the correct host and the mac address doesn't change.
When this backup IP address is allocated then dynamic DNS isn't
updated and my container is unreachable.
I've put all the relevant details in the link below but am more than
happy enough to furnish anything that is missing. Can anyone help please?
https://gitlab.isc.org/isc-projects/dhcp/-/issues/172
<https://connect-mobile.co.za/mobile_marketing_solutions/>
_______________________________________________
ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________

ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users

Wayne Gemmell | Connect

2021-03-31 11:45:52 UTC

Permalink

Thanks Glenn, I agree completely. if the DNS updated this wouldn't be an
issue. I'm trying to work out what is going on there but it is a bit of a
busy DNS server. I will try and isolate the traffic though.

<https://connect-mobile.co.za/mobile_marketing_solutions/>

The on-disk leases file is just a representation of what is in memory -
that is the true status, so as soon as it is released it would be marked
so in the in memory copy. I don't think timing would affect that. What
Sten says though about the history in the leases file is correct - each
new operation is appended to the file, so you can see previous ones, at
least for the life of the file. My understanding is that the file is
re-written every hour to preven tit growing too large. The previous file
remains dhcpd.leases~ so between the two you can get up to 2 hours of
history.
When a lease is released, it is marked as available. Given this
particular client has had both IPs in the past, then the matching
algorithm to find a new lease could come up with both possibilities.
Without looking at the code I wouldn't know precisely how it decides
upon a particular lease from the list of candidates. The client should
be prepared to switch IPs upon release of the lease and subsequent
discover. That is not the same as renewing an existing lease. I guess
the real problem is that DNS is not being updated.
Packet capture between dhcpd and the DNS server will at least show if
updates are being sent and you can then narrow it down to a problem with
either server.
regards,
Glenn

On 29 Mar 2021, at 12.05, Wayne Gemmell | Connect
Hi Sten
One of the host I've been testing with in the kannel pool alternates
between the following
inet 10.3.2.36/16 [1]
inet 10.3.2.60/16 [2]
Leases look as follows. I don't really understand why each would be
both free and active.
https://pastebin.com/UP6M1Lic
DHCP Lease list on responsible peer. The other one has similar data
but no host names. https://pastebin.com/WX3NwkM2
Here's snippet of the lease file https://pastebin.com/fKFiZpQ0
[3]
Thanks
Sten
On 26 Mar 2021, at 08.43, Wayne Gemmell | Connect
Thanks Sten
I've moved my class out of the subnet and the issue still persists.
Well, that is one can of worms out of the picture.
What are the two addresses?
[3]
You define the class kannel inside the subnet, so if there are any
other subnets, it will be defined there as well. Class definitions
are global always, defining them inside a subnet may have some
"interesting" side effects as some options are taken from that
subnet even if the host gets an address in a different subnet -
routers is one of them.
So one thought is, if the backup address is not in this subnet, the
host will be given incorrect information.
--
Best regards
Sten Carlsen
--------------------------------------------------
Aoccdrnig to rseerach at Cmabrigde Uinervtisy,
it deosn't mttaer in waht oredr the ltteers in a
wrod are, the olny iprmoatnt tihng is taht the
frist and lsat lteter be at the rghit pclae.
The rset can be a ttoal mses and you can slitl
raed it wotihut porbelm. Tihs is bcuseae the
hmuan mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. Amzanig, huh?
--------------------------------------------------
On 25 Mar 2021, at 13.01, Wayne Gemmell | Connect
I tried it. No change. The issue still persists.
[3]
On Tue, 23 Mar 2021 at 16:19, Wayne Gemmell | Connect
I'll give it a try, thanks.
[3]
On Tue, 23 Mar 2021 at 01:32, Bill Shirley
update-optimization off;
Bill
Hi
When my dhcp server gets a DHCPRELEASE followed by a DHCPDISCOVER
then it randomly allocates the IP address or the backup IP address
that are assigned to the host in the lease file to the host. The
assignments always got to the correct host and the mac address
doesn't change.
When this backup IP address is allocated then dynamic DNS isn't
updated and my container is unreachable.
I've put all the relevant details in the link below but am more than
happy enough to furnish anything that is missing. Can anyone help
please?
https://gitlab.isc.org/isc-projects/dhcp/-/issues/172 [4]
[3]
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users

_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users
------
[1] http://10.3.2.36/16
[2] http://10.3.2.60/16
[3] https://connect-mobile.co.za/mobile_marketing_solutions/
[4] https://gitlab.isc.org/isc-projects/dhcp/-/issues/172
_______________________________________________
ISC funds the development of this software with paid support
subscriptions. Contact us at https://www.isc.org/contact/ for more
information.
dhcp-users mailing list
https://lists.isc.org/mailman/listinfo/dhcp-users