New issue
Advanced search Search tips

Issue 916644 link

Starred by 3 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

isolateserver lost a file

Project Member Reported by benjamin...@google.com, Dec 19

Issue description

It seems that isolateserver lost an isolated file.

Looking at this task: https://chromium-swarm.appspot.com/task?id=41e03caf373f4e10
the isolated output is: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=77097f70db453491ed08b75eeb51acf4c3834fe9
which contains the file test/90effd6a465f50f7c5f44dfbcc49aefe.png with hash c77f66d182fb71fbaab27b9e5f50cf97eb01e06d. However, clicking on that link returns 404: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=c77f66d182fb71fbaab27b9e5f50cf97eb01e06d&as=90effd6a465f50f7c5f44dfbcc49aefe.png

I noticed this issue because there is a dependent task: https://chromium-swarm.appspot.com/task?id=41e047292932c610
that has isolated input: https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=cdecac44fe6e5cd9c401379c9b85d3628b1553c3
which contains 77097f70db453491ed08b75eeb51acf4c3834fe9, the output of the above task. The task failed during setup with:
21003 2018-12-19 17:14:01.064 E: internal failure: Attempted to fetch from https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve; no data exist: default-gzip / c77f66d182fb71fbaab27b9e5f50cf97eb01e06d.
Traceback (most recent call last):
  File "/b/s/swarming_bot.1.zip/client/run_isolated.py", line 734, in map_and_run
    use_symlinks=data.use_symlinks)
  File "/b/s/swarming_bot.1.zip/client/run_isolated.py", line 531, in fetch_and_map
    use_symlinks=use_symlinks)
  File "/b/s/swarming_bot.1.zip/client/isolateserver.py", line 1294, in fetch_isolated
    digest = fetch_queue.wait()
  File "/b/s/swarming_bot.1.zip/client/isolateserver.py", line 960, in wait
    digest = self._channel.next()
  File "/b/s/swarming_bot.1.zip/utils/threading_utils.py", line 377, in _task_executer
    result = func(*args, **kwargs)
  File "/b/s/swarming_bot.1.zip/client/isolateserver.py", line 865, in fetch
    sink(verifier.run())
  File "/b/s/swarming_bot.1.zip/client/local_caching.py", line 586, in write
    size = file_write(path, content)
  File "/b/s/swarming_bot.1.zip/client/local_caching.py", line 41, in file_write
    for d in content_generator:
  File "/b/s/swarming_bot.1.zip/client/isolateserver.py", line 1032, in run
    for chunk in self.stream:
  File "/b/s/swarming_bot.1.zip/client/isolateserver.py", line 262, in zip_decompress
    for chunk in content_generator:
  File "/b/s/swarming_bot.1.zip/client/isolate_storage.py", line 349, in fetch
    source_url, self.server_ref.namespace, digest))
IOError: Attempted to fetch from https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve; no data exist: default-gzip / c77f66d182fb71fbaab27b9e5f50cf97eb01e06d.
Attempted to fetch from https://isolateserver.appspot.com/_ah/api/isolateservice/v1/retrieve; no data exist: default-gzip / c77f66d182fb71fbaab27b9e5f50cf97eb01e06d.
 
Labels: -Restrict-View-Google -Pri-1 Pri-2
Status: Available (was: Untriaged)
It's rare, but I saw it happen. I'm adding MISSING_INPUTS state as part of 916553 which will make monitoring for this much easier.

Removing RVG as there's nothing internal in the report.

Sign in to add a comment