<div dir="ltr">In full on snark mode Ill comment that this must be why Rick was going off on the whole systemd thing.<div><br></div><div>Seriously, I found your post interesting and educational.</div><div><br></div><div>I have only begun to encounter systemd, so any posts on it are helpful.</div><div>Once you get the hand of it, I must admit that with several systems combined into systemd with a single way to manage them, </div><div>it has the potential for simplification.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Apr 18, 2020 at 5:49 PM Michael Paoli <<a href="mailto:Michael.Paoli@cal.berkeley.edu">Michael.Paoli@cal.berkeley.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Okay, got it solved.<br>
<br>
Turned out to be relatively simple ... but I missed it on earlier passes.<br>
The fix:<br>
# rm /etc/systemd/system/bind9.service.d/bind9.conf<br>
# systemctl daemon-reload<br>
<br>
Bit 'o background,<br>
the earlier:<br>
/etc/systemd/system/bind9.service.d/bind9.conf<br>
was put in place - and necessary - to do certain overrides<br>
for systemd launching bind9 ... notably for the chroot.<br>
However, now the unit file for bind9<br>
/lib/systemd/system/bind9.service<br>
had changed how it was<br>
calling / expecting to launch bind9. Notably:<br>
[Service]<br>
Type=forking<br>
So, that was conflicting with how<br>
/etc/systemd/system/bind9.service.d/bind9.conf<br>
was firing up bind9's named.<br>
Also, the newer<br>
/lib/systemd/system/bind9.service<br>
picks up the configuration bits in<br>
/etc/default/bind9<br>
(looks like the older did too ... but probably<br>
the much older did not)<br>
which also has the customizations for chroot,<br>
so the (custom)<br>
/etc/systemd/system/bind9.service.d/bind9.conf<br>
was no longer needed at all.<br>
Still not sure why it earlier didn't work<br>
when I removed the -f option in<br>
/etc/systemd/system/bind9.service.d/bind9.conf<br>
and I think I even tried removing that file entirely,<br>
but I might've missed the<br>
# systemctl daemon-reload<br>
step earlier on some of those attempts.<br>
In any case, cleanest "fix" for it:<br>
# rm /etc/systemd/system/bind9.service.d/bind9.conf<br>
# systemctl daemon-reload<br>
(well ... notwithstanding totally gutting systemd ;-))<br>
<a href="https://wiki.debian.org/Bind9" rel="noreferrer" target="_blank">https://wiki.debian.org/Bind9</a><br>
is also pretty good, but it could do with some more updating (which<br>
I may likely get around to if someone doesn't beat me to it).<br>
<br>
> From: "Michael Paoli" <<a href="mailto:Michael.Paoli@cal.berkeley.edu" target="_blank">Michael.Paoli@cal.berkeley.edu</a>><br>
> Subject: systemd 8-O ;-) ... bind9 chroot Debian 9 (stretch) --> <br>
> Debian 10 (buster)<br>
> Date: Sat, 18 Apr 2020 04:03:26 -0700<br>
<br>
> So, ... hitting a systemd issue I'd like to figure out and get resolved.<br>
> Yeah, I know, systemd, ugh ... but despite my also not much liking it,<br>
> if reasonably feasible, want to see if I can get this issue resolved.<br>
> So, bit 'o background:<br>
><br>
> So, ... working on (near) clone (balugclone) of system (balug).<br>
> Near? As in starting about identical, then mostly changing "just<br>
> enough" (<br>
> clone:<br>
> different Ethernet MAC address<br>
> (before even first booting) down interface link:<br>
> (<br>
> link=down; mac=52:54:00:67:20:40<br>
> virsh domif-setlink balugclone "$mac" "$link" --config<br>
> virsh domif-setlink balugclone "$mac" "$link"<br>
> virsh domif-getlink balugclone "$mac" --config<br>
> virsh domif-getlink balugclone "$mac"<br>
> )<br>
> change network from bridged to default (RFC-1918 + NAT/SNAT)<br>
> stop and disable potential conflicting services:<br>
> systemctl stop & systemctl disable:<br>
> mailman.service<br>
> exim4.service<br>
> apache2.service<br>
> spamassassin.service<br>
> rsync.service<br>
> mariadb.service<br>
> bind9.service<br>
> ...<br>
> )<br>
> to avoid conflicts with the running production balug<br>
> Virtual Machine (VM) and its data, etc.<br>
> And, what for? Do a pre-production Debian 9 (stretch) --> 10 (buster)<br>
> upgrade, to be able to plan for and have (theoretically) smooth actual<br>
> production upgrade. Alas, last time around, wasn't quite thorough<br>
> enough:<br>
> <a href="https://lists.balug.org/pipermail/balug-admin/2020-February/001018.html" rel="noreferrer" target="_blank">https://lists.balug.org/pipermail/balug-admin/2020-February/001018.html</a><br>
><br>
> Anyway, this time, fair bit more progress (yea!) (notably working<br>
> through sanity checks of at least basic functionality of important services).<br>
><br>
> But alas, still bumping into one gottcha I've not yet found a fix for.<br>
> And, yup, systemd specific.<br>
><br>
> So, nameserver - running BIND9 under chroot.<br>
> If I fire it up manually, in manner that sysvinit would were it present:<br>
> # PATH=/sbin:/bin:/usr/sbin:/usr/bin start-stop-daemon --start --oknodo \<br>
> --quiet --exec /usr/sbin/named --pidfile /run/named/named.pid -- \<br>
> -u bind -t /var/lib/named<br>
> Then all appears fine, it runs fine, functions, keeps working, etc.<br>
> (note to safely test it on clone, also:<br>
> clone:<br>
> /etc/network/interfaces disable interfaces except lo and change eth0<br>
> to inet dhcp<br>
> (eth0 & relevant configs later becomes ens3 through the upgrade)<br>
> shutdown<br>
> up interface link:<br>
> (link=up; mac=52:54:00:67:20:40<br>
> virsh domif-setlink balugclone "$mac" "$link" --config<br>
> virsh domif-setlink balugclone "$mac" "$link"<br>
> virsh domif-getlink balugclone "$mac" --config<br>
> virsh domif-getlink balugclone "$mac"<br>
> )<br>
> boot<br>
> and before enabling and attempting to (re)start bind9:<br>
> bind9 all notify off (no)<br>
> comment out notify-source and notify-source-v6<br>
> )<br>
><br>
> But alas, when started under systemd with:<br>
> # systemctl start bind9.service<br>
> Things go kind'a funky ... and fail in fairly short order.<br>
> First of all, as far as I can tell, from both systemd config,<br>
> and also looking at process arguments and such, looks like bind9<br>
> fires up properly under chroot in either case.<br>
> From: /etc/systemd/system/bind9.service.d/bind9.conf<br>
> we have:<br>
> ExecStart=/usr/sbin/named -f -u bind -t /var/lib/named<br>
><br>
> Also, without that -f option there (and after:<br>
> # systemctl daemon-reload<br>
> )<br>
> it then effectively doesn't (as far as systemd/systemctl is concerned)<br>
> work at all, failing quite immediately with:<br>
> systemd[1]: bind9.service: Control process exited, code=exited, <br>
> status=1/FAILURE<br>
> ... even though bind9/named is and continues to run fine in that case ...<br>
> but the systemd/systemctl status is all wrong, as it thinks it failed,<br>
> so, need the -f option. Anyway, back to with -f (foreground) option:<br>
><br>
> And ... smoking gun ... strace(1).<br>
> It looks like in both cases (manual sysvinit-like start, or<br>
> systemd:<br>
> # systemctl start bind9.service<br>
> named itself starts and<br>
> runs fine ... it's actually a systemd (configuration?) problem!<br>
> And, how did I find that? When the named process fails, it fails<br>
> because it's getting SIGTERM!!!:<br>
> 4539 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---<br>
> This seems to consistently happen about 90 seconds after systemd/systemctl<br>
> "starts" (attempts to start) it.<br>
> And ...:<br>
> 4689 kill(4690, SIGTERM) = 0<br>
> (the only reason the two PIDs between that and the earlier above don't<br>
> match, is they were captured in separate runs).<br>
> It's systemd/systemctl that's sending the signal that's causing<br>
> bind9 (named) to shutdown - that's also 100% consistent with what the<br>
> logs shows, e.g. (shortening the timestamps to MM:SS):<br>
> 51:42 balug-sf-lug-v2 named[5518]: resolver priming query complete<br>
> 53:12 balug-sf-lug-v2 named[5518]: shutting down<br>
> 53:12 balug-sf-lug-v2 named[5518]: stopping command channel on 127.0.0.1#953<br>
> 53:12 balug-sf-lug-v2 named[5518]: stopping command channel on ::1#953<br>
> 53:12 balug-sf-lug-v2 named[5518]: no longer listening on ::#53<br>
> 53:12 balug-sf-lug-v2 named[5518]: no longer listening on 127.0.0.1#53<br>
> 53:12 balug-sf-lug-v2 named[5518]: no longer listening on 192.168.122.245#53<br>
> 53:12 balug-sf-lug-v2 named[5518]: exiting<br>
> So ... at this point I'm trying to figure out why systemd/systemctl<br>
> is SIGTERMing named - when it ought not. I'm guestimating maybe<br>
> it tries to do some "health check", and does it improperly, and after<br>
> 90 seconds "gives up" and SIGTERMs the PID.<br>
> I also notice:<br>
> # systemctl start bind9.service<br>
> ... if done from terminal, that remains in the foreground the entire time<br>
> So seems systemd/systemctl is "waiting" for some check to pass before<br>
> "releasing", and instead times out waiting, gives up, and zaps the PID.<br>
><br>
> So, curious if any folks might know or have more clue(s) as to what<br>
> to look at and/or where to get down to the bottom of this<br>
> systemd/systemctl issue with bind9/named (also not seeing this issue<br>
> with any of the other services).<br>
><br>
><br>
> Other interesting bit ... (maybe just distraction / red herring):<br>
> /bin/systemd-tty-ask-password-agent<br>
> systemd/systemctl, done with interactive start from terminal,<br>
> fires up (forks (clone) and execs /bin/systemd-tty-ask-password-agent<br>
> with argument of --wait). If I redirect stdin from /dev/null,<br>
> e.g.:<br>
> # </dev/null systemctl start bind9.service<br>
> I don't end up with the /bin/systemd-tty-ask-password-agent process<br>
> hanging out for the duration ... but even in that case, named still<br>
> gets SIGTERMed by systemd/systemctl right around 90 seconds after it's<br>
> been fired up.<br>
> Also, on details, systemd/systemctl sends SIGCONT immediately<br>
> before the SIGTERM ... but it's the SIGTERM that has everything going<br>
> sideways and TERMinates the running bind9/named.<br>
><br>
> Also, if folks are curious, here are some of the key bits<br>
> that allow bind9/named to function under chroot:<br>
> $ grep named.\*bind /etc/fstab<br>
> /dev/null /var/lib/named/dev/null none bind 0 0<br>
> /dev/random /var/lib/named/dev/random none bind 0 0<br>
> /run/named /var/lib/named/run/named none bind 0 0<br>
> /usr/share/dns /var/lib/named/usr/share/dns none bind 0 0<br>
> $<br>
> That, and some symlink(s), etc., and it works under chroot ...<br>
> and stuff that needs and ought interact with it, from outside of<br>
> chroot, all works and plays nice together (almost the same as<br>
> Debian 9 (stretch) ... just one more directory from /usr for<br>
> Debian 10 (buster)). And with that infrastructure, it probably also<br>
> runs just fine outside of chroot too, without any changes ... but I<br>
> really don't want to be running it outside of the chroot.<br>
> Ah, what the heck, it's non-production, let's try ...<br>
> /etc/systemd/system/bind9.service.d/bind9.conf<br>
> ExecStart=/usr/sbin/named -f -u bind<br>
> # systemctl daemon-reload<br>
> # systemctl start bind9.service<br>
> ... and still fails same way (again shortening the timestamps to MM:SS):<br>
> 11:19 balug-sf-lug-v2 named[5991]: resolver priming query complete<br>
> 12:49 balug-sf-lug-v2 named[5991]: shutting down<br>
> 12:49 balug-sf-lug-v2 named[5991]: stopping command channel on 127.0.0.1#953<br>
> 12:49 balug-sf-lug-v2 named[5991]: stopping command channel on ::1#953<br>
> 12:49 balug-sf-lug-v2 named[5991]: no longer listening on ::#53<br>
> 12:49 balug-sf-lug-v2 named[5991]: no longer listening on 127.0.0.1#53<br>
> 12:49 balug-sf-lug-v2 named[5991]: no longer listening on 192.168.122.245#53<br>
> 12:49 balug-sf-lug-v2 named[5991]: exiting<br>
> And if I do it sysvinit-like start, without chroot:<br>
> # PATH=/sbin:/bin:/usr/sbin:/usr/bin start-stop-daemon --start <br>
> --oknodo --quiet --exec /usr/sbin/named --pidfile <br>
> /run/named/named.pid -- -u bind<br>
> ... it continues to stay up and running no problem, long past 90 seconds,<br>
> so appears it's not only not a chroot issue, but not even at all specific<br>
> to chroot.<br>
> FYI:<br>
> $ ls -l /etc/bind<br>
> lrwxrwxrwx 1 root root 25 Mar 15 2014 /etc/bind -> ../var/lib/named/etc/bind<br>
> $<br>
> Anyway, mostly that, and the bind mounts, and appropriate<br>
> permissions/ownerships, and it plays well in and/or out of chroot (alas,<br>
> probably the first time I fired it up outside of chroot in many years).<br>
<br>
<br>
_______________________________________________<br>
conspire mailing list<br>
<a href="mailto:conspire@linuxmafia.com" target="_blank">conspire@linuxmafia.com</a><br>
<a href="http://linuxmafia.com/mailman/listinfo/conspire" rel="noreferrer" target="_blank">http://linuxmafia.com/mailman/listinfo/conspire</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><br>R "Texx" Woodworth<br>Sysadmin, E-Postmaster, IT Molewhacker<br>"Face down, 9 edge 1st, roadkill on the information superdata highway..."<br></div>