It sounds like there is only one issue here: you're not getting the new ShortName from the definition file.  (I wonder if Chrome is actually fetching anew the file or if it's hitting a cache somewhere in the way.)

The other complaint here is not a bug.  After you remove the search engine entry from settings ("Remove Search Engine entry from settings, repeat 5 and 6.") then do step 5 ("Go to the website again and do what usually gets the OpenSearch loaded").  It's natural that the search engine appears again.  This is working as intended.

BTW, last year I investigated when opensearch descriptions get updated and concluded that everything was working correctly (see bug 470604).  So if there's a bug here, it's likely just a problem with shortname rather than the whole file and the URL of search suggestions.

I can tell that it doesn't always fetch the file, when running locally and seeing the web server output. With a local machine, the only cache should be Chrome, but since I can't see the requests in Chrome (because of crbug/501185), I can't tell how far the request goes. 

Making sure the opensearch.xml file doesn't get cached and _then_ removing the entry from Search Engines makes a new request happen and get used. However, the old ShortName still gets used in the chip after hitting 'tab'.

I'm not sure about fetching the file.  You can try a program like wireshark to monitor your computer's network traffic.  Aside from the out-of-date shortname issue, could this simply be a problem with it being unclear when we're supposed to fetch a new file?  i.e., could the root issue be not triggering the fetch logic at all in step 5)

The shortname thing sounds a bit like bug 569770, which haven't been able to find a consistent repro for.  (I think it has something to do with authentication...)  If you have a consistent repro step that we can do on a new profile, let me and pkasting@ know and let us poke at your server with a debug chrome.

It is indeed unclear when you're supposed to fetch a file once a previous version has been fetched. It looks like the cached file sticks around when the search engine entry gets removed.

It would appear that removing the entry in settings and re-loading the XML file with cache disabled is enough to update most of the definition. However, I observed lingering shortnames from previous versions (at one point two different ones). 

Between the lack of insight into what actually happens, the caching problems, the divergent specifications, and the results not matching anything the specifications imply, I've decided to drop this approach for now, sadly. Fixing  bug 501185  (and/or making it possible to see details of the current search engine entries) would help.

Issue 664260 has been merged into this issue.

The merged-in bug, and its linked to some google-internal bugs, have several more examples of this.

pkasting: we need to figure out a policy on what we should be doing to update auto-generated custom search engines.  There are two types of such engines, those created with opensearch descriptions and those created due to a properly-formed role=search form.

My instinct: for the former, always download an opensearch specification if it exists and update the engine appropriately (if the spec is invalid, then don't do anything); for the latter, always try to replace the current auto-generated engine (if we decide based on the structure of the search form that we wouldn't create a new auto-generated engine with this form, then do not revise the current engine).

What are your thoughts?

Those seem fine to me.  Do you know what we currently do that isn't those things?

It's clear from this report and the merged-in bugs that at least opensearch is broken.  It's possible the non-opensearch / inferred search engines are broken as well.  When someone picks this up, they should check.

For reference,
tThe code for open search (xml-configured) search engines lives in
https://cs.chromium.org/chromium/src/components/search_engines/template_url_fetcher.h
This class does both the fetching and adding of these search engines.

The code for search engines that are inferred based on the structure of a <form> that someone submits lives partially in Blink, here:
https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/exported/WebSearchableFormData.cpp?q=WebSearchableFormData.cpp&dr&l=216
This gets sent back to Chrome and used somewhere in here.
https://cs.chromium.org/chromium/src/content/renderer/render_frame_impl.cc
I haven't traced how this form data gets transmitted all the way back to being added into the set of search engines.