As far as I know, nothing has changed with the tryserver itself. It's possible some intermediate plumbing broke.

The next question is: who sends these e-mails? Are these BuildBot-generated e-mails? Are they generated/sent by "cbuildbot" tool? +akeshet@, can you shed some light on what component is failing here?

afaik these were sent by buildbot, not cbuildbot. I don't know of any code in cbuildbot that was sending them. And they were sent all prettily html-rendered, buildbot style, with stages and links and colors, which also suggests to me that it has to have been buildbot (cbuildbot doesn't render pages or have access to rendered pages like this).

Whoever is responsible for working on this:  This is a serious problem that really needs to be fixed.  I submitted 3 tryjobs over the weekend, but again received no email about their status.  The stupid website drops the jobs off the page after about 25 hours, so now when I go to try to see if the jobs succeeded or failed, I can't find them at all, so I have to re-submit the jobs and remember to sit watching the web page the entire day to see what happens to them.  This is a waste of my time and your resources.

PLEASE FIX THIS SOON!!!

Aviv - is this cbuildbot?

This is not cbuildbot.

I pulled up an old email, it has from address "cros.tryserver@chromium.org". That string does not occur in chromite. It does occur in the buildbot site config for our tryserver.

https://chromium.googlesource.com/chromium/tools/build.git/+/master/masters/master.chromiumos.tryserver/master_site_config.py

Could we get somebody from chrome infra to look into this?

Erik - can you find someone who can look into this today?

Yeah, I'll ask around and see what we can do about this.

+trooper, there isn't a clear owner for this.

Marking untriaged since I think most troopers are filtering owned bugs these days.

re comment 13: that's correct.

code is here:
https://chromium.googlesource.com/chromium/tools/build/+/b9d4324e45281a5e1fae1fba173e3d93ca5465e2/masters/master.chromiumos.tryserver/master_site_config.py#15

and here:
https://chromium.googlesource.com/chromium/tools/build/+/b9d4324e45281a5e1fae1fba173e3d93ca5465e2/masters/master.chromiumos.tryserver/master.cfg#206

2016-06-29 12:16:52-0700 [-] sending mail (9343 bytes) to [u'stevenjb@chromium.org']
2016-06-29 12:16:52-0700 [-] Starting factory <twisted.mail.smtp.ESMTPSenderFactory instance at 0x2b2f123e3ef0>
2016-06-29 12:16:52-0700 [-] releaseLocks(<AutoRebootBuildSlave 'build272-m2', current builders: chrome-pfq,compile-only-pre-cq,tot-asan-informational,chrome-perf,cheets-release,depthcharge-full-firmware,incremental,pre-flight-branch,branch-util,no-vmtest-pre-cq,llvm,gcc,release-afdo-use,firmware,telemetry,factory,android-pfq,chromium-pfq,payloads,test-ap,chromium-pfq-informational,paladin,full,moblab-release,kernel-3_18,chrome-pfq-cheets-informational,wificell-pre-cq,pre-cq-launcher,sync-test-cbuildbot,asan,pre-cq,chrome-pfq-informational,chromium-pfq-informational-gn,toolchain,chromiumos-sdk,etc,release-afdo-generate,release,depthcharge-firmware>): []
2016-06-29 12:16:52-0700 [Uninitialized] SMTP Client retrying server. Retry: 5
2016-06-29 12:16:52-0700 [Uninitialized] SMTP Client retrying server. Retry: 4
2016-06-29 12:16:52-0700 [Uninitialized] SMTP Client retrying server. Retry: 3
2016-06-29 12:16:52-0700 [Uninitialized] SMTP Client retrying server. Retry: 2
2016-06-29 12:16:52-0700 [Uninitialized] SMTP Client retrying server. Retry: 1
2016-06-29 12:16:52-0700 [Uninitialized] Unhandled error in Deferred:
2016-06-29 12:16:52-0700 [Uninitialized] Unhandled Error
        Traceback (most recent call last):
        Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

Labs, can you see if master2a.golo's SMTP gateway is working properly?

Thanks for verifying the configs stip. For some reason postfix wasn't running, it is now and i see mail moving.

cmtice@, can you verify people are getting mail?

I haven't received any old/archived/delayed emails.  I will submit a new trybot job and see if I get any emails, but it will be several hours before I know.

You wont receive old email, as it couldnt be sent, and buildbot just drops it on the floor in that situation.

I've launched a new trybot job; I'll let you know about the email.

I received the status email; it looks like this is fixed. Thanks!