The dhcpd.conf on the server corresponds to this commit:

commit 39fd9ae782506bdb112b4a27378bbad3fd6ad218 (HEAD)
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date:   Thu Feb 22 13:16:54 2018 -0800

    puppet: Drop incorrect request to use CNAME within prod
    
    BUG=chromium:814855
    TEST=None
    
    Change-Id: I2cf08a93074e9e3bc6b8844c9337717cd9fd223f
    Reviewed-on: https://chrome-internal-review.googlesource.com/575734
    Reviewed-by: Prathmesh Prabhu <pprabhu@google.com>
    Tested-by: Prathmesh Prabhu <pprabhu@google.com>

So, presumably some change to puppet shortly after that is the source
of the problem.

Ah, dns servers don't have the private key.  They are not in our prod server list so I forgot them when deploying the private key initially.

They didn't need the private key, but jkop is pushing forward with encrypting secrets now.

Richard, can you take a shot as Deputy?  The instructions are in chromeos-admin/ansible/README.md and there's a deploy_key.yml Ansible playbook in the same directory.

No doubt the README can be improved.

Along a similar vein, we should track ddns on our Puppet dashboards, but they are currently getting filtered out since they are not in our prod list.

> Richard, can you take a shot as Deputy?
> The instructions are in chromeos-admin/ansible/README.md
> and there's a deploy_key.yml Ansible playbook in the same directory.

If the missing key isn't causing this failure, then that should be a
separate bug.

I did more digging into a possible blamelist.

This CL is the last before the update that installed the current
versions of the config file:
    9aba9969 Replacing Row4-Rack1-Server

This CL was committed 2/23 16:39, so it post-dates any likely updates
that might have broken puppet:
    cc13e7cf This Is A Test

So, this is a starting blamelist:
cc13e7cf This Is A Test
39fd9ae7 puppet: Drop incorrect request to use CNAME within prod
42b6753f puppet: Install python-mysqldb package everywhere
1985cdd7 puppet: Remove unused upstart job
fdc3e508 puppet: Add new SSH keys for chromeos-test user
18d972f2 puppet: Explicitly list ssh files to install
9796afc4 puppet: Move ssh setup to subclass
e60b4013 puppet: disable 'respect_static_attributes'.
a89985b2 Revert "puppet: Use chromeos-test group for role accounts"
c7cb8654 [puppet] Include all validation targets
aa4ec17d [puppet] Add a second shard AFE server.
b4e5b2f6 Fix gmail credentials in the lab.
c7f57133 Fix gmail credentials in the lab.
dbeced32 [puppet] Add syntax validation
bc896803 Swapping servos for 6-4-18-12,14,16 and 6-4-19-2.
343d00f4 sml: Update purser ListPolicies call's field_mask.
b3bb1ea2 puppet: Use chromeos-test group for role accounts
63d0b4e4 ci_results_archiver: Switch to cros-bighd-0002.
6eb0fdb4 puppet: Start mysql on all new DB servers.


From that list, this change jumps out at me:
fdc3e508 puppet: Add new SSH keys for chromeos-test user

Looking at the suspect CL, I think it's possible that the
problem does stem from the missing private key.  So, I'll
go give that a whirl.

I installed the key, and forced a manual puppet run.  Now I see
this:

    chromeos-test@chromeos-ddns3:/tmp$ ls -l /etc/dhcp/dhcpd.conf
    -rw-r--r-- 1 root root 911469 Mar 23 16:46 /etc/dhcp/dhcpd.conf

So, don't know why, but I think that was it.