Yes, definitely due to SHT_RELR sections that are now in every PIE binary.

I reproduced the problem in my chroot.

While gdb does print the "don't know how to handle section `.relr.dyn'" message, it works fine after that. I am able to single step, set breakpoints, print variables etc. from my test program containing SHT_RELR sections.

The inability to debug seems to be related to something else. Compiling with '--no-experimental-use-relr' will make the message about '.relr.dyn' go away, but does it make any difference in gdb's ability to debug?

Interesting. Without the LDFLAGS I was not able to debug. It was showing `Remote register badly formatted: xxxx`.

(grunt-gdb) b main
Breakpoint 1 at 0x3940: file ../mosys-9999/mosys.c, line 312.
(grunt-gdb) c
The program is not being run.
(grunt-gdb) run
Starting program: /build/grunt/tmp/sbin/mosys 
<locks up here>

When debugging I don't normally need to call `run`. 

GDB output from a working run:
(grunt-gdb) b main
Breakpoint 1 at 0x55555556f30c: file ../mosys-9999/mosys.c, line 313.
(grunt-gdb) c
Continuing.  
<breaks on main>

When you reproduced the problem, were you remote debugging?

I did not try remote debugging. Only tried loading and running binary containing SHT_RELR sections locally under gdb.
Cab you check if your remote device has updated glibc as well?
Running a binary containing SHT_RELR sections on a system with ld.so that does not process SHT_RELR sections will not work.
Are you able to run the binary remotely without gdb?

Yeah the remote device is able to run the binary without gdb. The device had an image build last week. Not sure what exact glibc version is required.

An image built last week should be fine. Support for SHT_RELR was added to our glibc in mid March (https://chromium-review.googlesource.com/965112).

Can you try loading the same binary under gdb in your chroot?
Since the binary is for x86_64, you should be able to run it under gdb, set breakpoint at main, single step, etc.

I reproduced the problem using a chell device (also x86_64, /usr/sbin/mosys is built with SHT_RELR sections enabled).

However, I could not get the gdb-chell script to work. My reproduction was using manually setup gdbserver:

# ssh to the chell machine (ip-address: 100.100.93.212) and forward local port 12345 (enter password "test0000" if using a test image).
$ ssh -L 12345:localhost:12345 chronos@100.100.93.212

# run gdbserver on the device, listening on port 12345.
$ sudo gdbserver :12345 /usr/sbin/mosys -vvvf memory spd print all

# In your chroot, run x86_64-cros-linux-gnu-gdb on mosys binary, and connect to the gdbserver running on the device.
$ x86_64-cros-linux-gnu-gdb -q /build/chell/usr/sbin/mosys 
BFD: /build/chell/usr/sbin/mosys: don't know how to handle section `.relr.dyn' [0x      13]
Reading symbols from /build/chell/usr/sbin/mosys...Reading symbols from /usr/lib64/debug//build/chell/usr/sbin/mosys.debug...done.
done.
(gdb) target remote :12345
Remote debugging using :12345
warning: Can not parse XML target description; XML support was disabled at compile time
Remote register badly formatted: T0506:0000000000000000;07:10e6ffffff7f0000;10:608dddf7ff7f0000;thread:p41e4.41e4;core:3;
here: 00000000;07:10e6ffffff7f0000;10:608dddf7ff7f0000;thread:p41e4.41e4;core:3;
(gdb) 

At this point, the gdb running in the chroot terminates connection with the gdbserver. The window with gdbserver shows:
Remote debugging from host 127.0.0.1
Remote side has terminated connection.  GDBserver will reopen the connection.
Listening on port 12345

and gdb never reconnects.

The problem seems to be a mismatch of architecture setting between gdb and gdbserver.

(gdb) show architecture
The target architecture is set automatically (currently i386)
(gdb) set architecture i386:x86-64
The target architecture is assumed to be i386:x86-64
(gdb) target remote :12345
Remote debugging using :12345
Reading symbols from /usr/x86_64-cros-linux-gnu/lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib64/debug/usr/x86_64-cros-linux-gnu/lib64/ld-2.23.so.debug...done.
done.
0x00007ffff7dd8d60 in _start () from /usr/x86_64-cros-linux-gnu/lib64/ld-linux-x86-64.so.2
(gdb) 

After I set the architecture explicitly, it is able to connect to the remote gdbserver. I can now run the program:

(gdb) continue
Continuing.
[Inferior 1 (process 16868) exited normally]
(gdb) 

The output from mosys appears in the window where gdbserver was running.

Note that this is with the default build of mosys with .relr.dyn section.
There was a message about "don't know how to handle section `.relr.dyn'", but it's harmless and can be ignored.

In my chroot:

$ for i in i686-pc-linux-gnu x86_64-cros-linux-gnu armv7a-cros-linux-gnueabi aarch64-cros-linux-gnu; do echo -n "$i-gdb: "; $i-gdb -q --eval-command="show architecture" --eval-command="quit"; done
i686-pc-linux-gnu-gdb: The target architecture is set automatically (currently i386)
x86_64-cros-linux-gnu-gdb: The target architecture is set automatically (currently i386)
armv7a-cros-linux-gnueabi-gdb: The target architecture is set automatically (currently arm)
aarch64-cros-linux-gnu-gdb: The target architecture is set automatically (currently aarch64)

The second one (x86_64-cros-linux-gnu-gdb) should print "i386:x86-64".

An interesting line from gdb output in the bad case:
  warning: Can not parse XML target description; XML support was disabled at compile time

Looks like gdbserver sent target description to gdb, but gdb could not parse it to set target architecture.

It seems to me that if XML support is enabled at gdb compile time, it might be able to understand the description sent by gdbserver and set architecture automatically.

Thanks for digging into it. The gdb-xxx scripts are a bit flaky.

So does this mean that gdb is having problems detecting the arch for SHT_RELR binaries?

Rahul, you should be able to build gdb with USE="xml" to test this.

Trying now :)

I tried USE=xml sudo -E emerge -v cross-x86_64-cros-linux-gnu/gdb

This resulted in the same error. I'm going to try building gdb for the target with xml support.

Looks like it is two issues that compound together to end up in the bad architecture setting:

1. gdb was unable to determine the architecture from the binary.
2. gdb was unable to parse the description sent by gdbserver.

(1) is indeed due to the presence of a SHT_RELR section.
For two binaries, one with SHT_RELR and one without, I see this behavior:

$ x86_64-cros-linux-gnu-gdb -q --eval-command="show architecture" --eval-command="quit" ./a.out.relr
BFD: /mnt/host/workspace/glibc/hello/a.out.relr: don't know how to handle section `.relr.dyn' [0x      13]
Reading symbols from ./a.out.relr...done.
The target architecture is set automatically (currently i386)

$ x86_64-cros-linux-gnu-gdb -q --eval-command="show architecture" --eval-command="quit" ./a.out.norelr
Reading symbols from ./a.out.norelr...done.
The target architecture is set automatically (currently i386:x86-64)

In the second example, the binary does not contain an SHT_RELR section.
The message from BFD: is gone, *and* gdb has the architecture setup correctly.
The message is not completely harmless. It also implies that gdb was unable to detect the architecture automatically.

The fallback of parsing the XML description sent by gdbserver would be nice to have, but the detection from binary needs to be fixed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/4dd8e137369070d2b8c6e9ac0f80a709e7da083f

commit 4dd8e137369070d2b8c6e9ac0f80a709e7da083f
Author: Rahul Chaudhry <rahulchaudhry@chromium.org>
Date: Thu May 03 22:51:40 2018

sys-devel/gdb: move SHT_RELR section type to the generic range.

Proposal for adding SHT_RELR sections in generic-abi is at
https://groups.google.com/forum/#!topic/generic-abi/bX460iggiKg

Experimental support for SHT_RELR sections is available in the
aosp gdb repository in master branch, with the section type and
dynamic array tags defined in the OS specific range.

This change updates to the latest commint in aosp/gdb, and moves
those definitions to the generic range so we can load binaries
containing SHT_RELR sections created by the experimental support
in sys-devel/binutils.

BUG= chromium:836845 
TEST='sudo emerge cross-x86_64-cros-linux-gnu/gdb' and loaded x86_64 binaries containing SHT_RELR sections.

Change-Id: I9632a485c749f23648063edfd9a814024b94453e
Reviewed-on: https://chromium-review.googlesource.com/1040680
Commit-Ready: Rahul Chaudhry <rahulchaudhry@chromium.org>
Tested-by: Rahul Chaudhry <rahulchaudhry@chromium.org>
Reviewed-by: Manoj Gupta <manojgupta@chromium.org>

[modify] https://crrev.com/4dd8e137369070d2b8c6e9ac0f80a709e7da083f/sys-devel/gdb/Manifest
[add] https://crrev.com/4dd8e137369070d2b8c6e9ac0f80a709e7da083f/sys-devel/gdb/files/gdb-8.0.1-sht_relr.patch
[delete] https://crrev.com/2ca21f62e8c677efbb5a6c1d0e744da0e7eaa4eb/sys-devel/gdb/gdb-8.0.1.20171030-r5.ebuild
[rename] https://crrev.com/4dd8e137369070d2b8c6e9ac0f80a709e7da083f/sys-devel/gdb/gdb-8.0.1.20180501.ebuild

The new sdk should have the fixed gdb.
Please update and give this another try.

It's still not working:

dut ~ # gdb --version
GNU gdb (Chromium OS 8.0.1.20180501 vanilla) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-cros-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://crbug.com/new>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

dut ~ # gdbserver --version
GNU gdbserver (Chromium OS 8.0.1.20180501 vanilla) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
gdbserver is free software, covered by the GNU General Public License.
This gdbserver was configured as "x86_64-cros-linux-gnu"



(chroot) $ gdb-grunt --debug --cgdb --remote 100.107.108.187 \
  -g "--eval-command=set complaints 10000" \
  -g "--eval-command=directory /build/grunt/tmp/portage/sys-apps/mosys-9999/work/" \
  /tmp/sbin/mosys -vvvf memory spd print all

Reading symbols from x86_64-cros-linux-gnu-gdb...Reading symbols from /usr/lib64/debug/usr/bin/x86_64-cros-linux-gnu-gdb.debug...done.
done.
BFD: /build/grunt/tmp/sbin/mosys: don't know how to handle section `.relr.dyn' [0x      13]
Load new symbol table from "/build/grunt/tmp/sbin/mosys"? (y or n) Load new symbol table from "/build/grunt/tmp/sbin/mosys"? (y or n) Load n
ew symbol table from "/build/grunt/tmp/sbin/mosys"? (y or n) y
Reading symbols from /build/grunt/tmp/sbin/mosys...done.
Remote debugging using localhost:55919
warning: Can not parse XML target description; XML support was disabled at compile time
Remote register badly formatted: T0506:0000000000000000;07:b0ecffffff7f0000;10:608dddf7ff7f0000;thread:pd10.d10;core:1;
here: 00000000;07:b0ecffffff7f0000;10:608dddf7ff7f0000;thread:pd10.d10;core:1;
Source directories searched: /build/grunt/tmp/portage/sys-apps/mosys-9999/work:$cdir:$cwd
(grunt-gdb) show architecture
The target architecture is set automatically (currently i386)

(chroot) $ x86_64-cros-linux-gnu-gdb --version
GNU gdb (Chromium OS 8.0.1.20180501 vanilla) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=x86_64-cros-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://crbug.com/new>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

This is really strange. Is there a way to find out which gdb is being spawned by gdb-grunt?

The gdb versions in your other gdb commands look good and they should have the fix. I tried this in my chroot (containing a chell build) and there is no error about .relr.dyn section from BFD:

$ x86_64-cros-linux-gnu-gdb /build/chell/usr/sbin/mosys 
GNU gdb (Chromium OS 8.0.1.20180501 vanilla) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=x86_64-cros-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://crbug.com/new>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /build/chell/usr/sbin/mosys...Reading symbols from /usr/lib64/debug//build/chell/usr/sbin/mosys.debug...done.
done.

Can you run x86_64-cros-linux-gnu-gdb directly on mosys and check if you still see the error from BFD:

$ x86_64-cros-linux-gnu-gdb /build/grunt/tmp/sbin/mosys

Here is the info from gdb-grunt

10:16:48: INFO: RunCommand: cgdb x86_64-cros-linux-gnu-gdb --quiet '--eval-command=set sysroot /build/grunt' '--eval-command=set solib-absolute-prefix /build/grunt' '--eval-command=set solib-search-path /build/grunt' '--eval-command=set debug-file-directory /build/grunt/usr/lib/debug' '--eval-command=set prompt (grunt-gdb) ' '--eval-command=file /build/grunt/tmp/sbin/mosys' '--eval-command=target remote localhost:55919' '--eval-command=set complaints 10000' '--eval-command=directory /build/grunt/tmp/portage/sys-apps/mosys-9999/work/'

Dropping the --cgdb makes it work. Looks like gdb-grunt is not passing the -d flag to cgdb to pick the correct debugger.

10:40:57: INFO: RunCommand: x86_64-cros-linux-gnu-gdb --quiet '--eval-command=set sysroot /build/grunt' '--eval-command=set solib-absolute-prefix /build/grunt' '--eval-command=set solib-search-path /build/grunt' '--eval-command=set debug-file-directory /build/grunt/usr/lib/debug' '--eval-command=set prompt (grunt-gdb) ' '--eval-command=file /build/grunt/tmp/sbin/mosys' '--eval-command=target remote localhost:41407' '--eval-command=set complaints 10000' '--eval-command=directory /build/grunt/tmp/portage/sys-apps/mosys-9999/work/'
Reading symbols from /build/grunt/tmp/sbin/mosys...done.
Remote debugging using localhost:41407
warning: Can not parse XML target description; XML support was disabled at compile time
Reading symbols from /build/grunt/lib64/ld-linux-x86-64.so.2...Reading symbols from /build/grunt/usr/lib/debug/lib64/ld-2.23.so.debug...done.
done.
0x00007ffff7dd8d60 in _start () from /build/grunt/lib64/ld-linux-x86-64.so.2
Source directories searched: /build/grunt/tmp/portage/sys-apps/mosys-9999/work:$cdir:$cwd
(grunt-gdb) b main
Breakpoint 1 at 0x55555555799c: file ../mosys-9999/mosys.c, line 313.
(grunt-gdb) c
Continuing.
During symbol reading...unsupported stack op: 'DW_OP_GNU_implicit_pointer'...

Breakpoint 1, main (
During symbol reading, incomplete CFI data; unspecified registers (e.g., rax) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., rdx) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., rcx) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., rsi) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., rdi) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r8) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r9) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r10) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r11) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r12) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r13) at 0x555555557997.
During symbol reading, incomplete CFI data; unspecified registers (e.g., r15) at 0x555555557997.
argc=6, argv=0x7fffffffecb8) at ../mosys-9999/mosys.c:313
313             mosys_globals_init();

Looks like https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1033920 should fix the cgdb error.

Thanks!