Thanks for the report(s)!
Interestingly, this particular one only happens on phone for me (Nexus 4, Android L), not on tablet (Nexus 7, Android M).
Looking into the issue.

It also happens when the dialog is closed with the "Cancel" button.

And it also happens after a successful export, passing the sharing intent to Gmail and then discarding the draft there (upon which the user is returned to Chrome's settings).

The reason why this reproduces unreliably is that the issue needs to happen under memory pressure. My older phone has that more often than the newer tablet, but setting "Don't keep activities" in developer options makes the reproducing reliable. SavePasswordsPreferences#onResume needs fixing, I'm working on that.

I'm slightly more confused. Perhaps the memory pressure was just a red herring. The debugger tells me that #onResume is not involved in re-showing the dialog.

OTOH, even with explicit dialog.dismiss() calls, the dialog appears again. Not sure why is that so, but will continue investigating.

Mystery solved, I forgot to call super.dismiss(dialog) in my custom DialogFragment.

That's easy to fix, but the patch might take me a while because of tests. Apparently, the instrumented version of Chrome differs in some key way of handling dismissal from normal Chrome, and thus just creating an Espresso test case repeating the steps from here did not reproduce the issue. I'll try to figure that out and update this bug with a CL URL once I have some.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c3c2e6c338ed0a054d2de7e5927fca5f90f0b908

commit c3c2e6c338ed0a054d2de7e5927fca5f90f0b908
Author: Vaclav Brozek <vabr@chromium.org>
Date: Tue Feb 13 12:59:57 2018

Call super.onDismiss in ExportWarningDialogFragment#onDismiss

By accident, ExportWarningDialogFragment#onDismiss did not call its
parent's onDismiss. This seemed to work fine on some occasions and
with some combinations of device and Android, but occasionally it led
to the dialog being re-opened.

This CL fixes that omission and with it also cures the reported issue
from the associated bug.

However, there seems to be no reliable way to test this. The author wrote an
Espresso test replicating the reproduction steps (tap export, cancel, view a
password entry, go back), but this test passed even before the fix. It might be
that the instrumentation interferes with the behaviour of the dialog parent's
onDismiss(). Therefore this CL lacks a regression test.

Bug:  811312 
Change-Id: Iea9be516e2ef06d44f3aa21e3a921e31ca22df97
Reviewed-on: https://chromium-review.googlesource.com/915366
Reviewed-by: Ted Choc <tedchoc@chromium.org>
Commit-Queue: Vaclav Brozek <vabr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536344}
[modify] https://crrev.com/c3c2e6c338ed0a054d2de7e5927fca5f90f0b908/chrome/android/java/src/org/chromium/chrome/browser/preferences/password/ExportWarningDialogFragment.java

Verified fix in 66.0.3348.0 with steps from #0. Export password dialog in not re-prompted.