New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 837770 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Apr 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Hack FileSystemGetter to work around recent problems.

Reported by jrbarnette@chromium.org, Apr 27 2018

Issue description

Because of problems explained in bug 771827, the
code for FileSystemGetter in server/cros/dynamic_suite/control_file_getter.py
is sometimes asked to search all of /usr/local/autotest, which
slows down the world.  Worse, the search code isn't safe in the presence
of symlinks, as described in bug 837388.

Because of recent outages caused by the search code, we need
to do better.  Unfortunately, fixing bug 771827 the right way
could be awkward (the bug has the details).

As it happens, there's an existing hack in the code that blacklists
selected directories:
        # Do not explore site-packages. (crbug.com/771823)
        # Do not explore venv. (b/67416549)
        # (Do not pass Go. Do not collect $200.)
        blacklist = {'site-packages', 'venv'}

The blacklist is conceptually wrong. It's also a maintenance hazard,
since there's nothing to guarantee that the blacklist stays up-to-date
in the face of content changes in /usr/local/autotest.  However, the
blacklist is useful for an immediate fix to prevent the worst of the
recent problems.  So, we should make two changes:
 1) Update the blacklist to contain at least the following:
        site-packages
        venv
        results
        containers
        logs
 2) Update this test:
                    elif os.path.isdir(fullpath):
                        directories.append(fullpath)
    Change it to test `os.path.isdir(fullpath) and not os.path.islink(fullpath)`

 

Comment 1 by ihf@chromium.org, Apr 27 2018

Cc: ihf@chromium.org
Cc: dgarr...@chromium.org gu...@chromium.org
Owner: jrbarnette@chromium.org
Status: Started (was: Available)
https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/1033693/

Project Member

Comment 3 by bugdroid1@chromium.org, Apr 28 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/be2a91150e6657d0a7571beef71df805f533cffa

commit be2a91150e6657d0a7571beef71df805f533cffa
Author: Richard Barnette <jrbarnette@chromium.org>
Date: Sat Apr 28 04:27:42 2018

[autotest] Prevent certain problems in finding control files.

In control_file_getter.FileSystemGetter, there's code that searches
a given list of direcotory hierarchies for test control files.
Some callers request searches of all of /usr/local/autotest; that's
kind of bad.

Additionally, the code is vulnerable to looping forever if a symbolic
link in the hierarchy produces a loop.

This change extends an existing hack that blacklists known big
directories.  It also skips searching through any symbolic link.

BUG= chromium:837770 
TEST=None

Change-Id: Iea8a4e03aa8ae870f7a79df7d632009be7bda70d
Reviewed-on: https://chromium-review.googlesource.com/1033693
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Congbin Guo <guocb@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/be2a91150e6657d0a7571beef71df805f533cffa/server/cros/dynamic_suite/control_file_getter.py

Status: Fixed (was: Started)

Sign in to add a comment