New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 809790 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner: ----
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

chromeos2-devserver9 database entry set up incompletely

Reported by jrbarnette@chromium.org, Feb 7 2018

Issue description

From the server database, you find this:
    $ atest server list chromeos2-devserver9.cros.corp.google.com
    Hostname     : chromeos2-devserver9.cros.corp.google.com
    Status       : primary
    Roles        : devserver
    Attributes   : {}
    Date Created : 2018-02-05 13:51:57
    Date Modified: 2018-02-05 13:51:57
    Note         : None

The devserver attributes are empty.  That shouldn't be:  Without
properly set up attributes, the devserver can't be entered into
shadow_config.ini, and without that, it can't be used.

We need to figure out what went wrong adding the server, and
fix the tool (preferable) or process (if it can't be isolated
inside a tool) so that it doesn't happen again.

 
... Assuming that the server is otherwise correctly configured,
we probably should now fix it so that it can into production.

Cc: davidri...@chromium.org
Technically that is a "beta" devserver to test new the equipment.  Currently being used by davidriley for some devserver work

It is not suppose to be used for production use yet.
Cc: englab-sys-cros@google.com
I've fixed the server attributes:
    $ HOST=chromeos2-devserver9
    $ atest server modify -a port -e 8082 $HOST.cros.corp.google.com
    Attribute `port` of server chromeos2-devserver9.cros.corp.google.com is set to 8082.
    Server chromeos2-devserver9.cros.corp.google.com is modified successfully.
    Hostname     : chromeos2-devserver9.cros.corp.google.com
    Status       : primary
    Roles        : devserver
    Attributes   : {u'port': u'8082'}
    Date Created : 2018-02-05 13:51:57
    Date Modified: 2018-02-05 13:51:57
    Note         : None

    $ atest server modify -a restricted_subnet -e 100.115.224.0/19 $HOST.cros.corp.google.com
    Attribute `restricted_subnet` of server chromeos2-devserver9.cros.corp.google.com is set to 100.115.224.0/19.
    Server chromeos2-devserver9.cros.corp.google.com is modified successfully.
    Hostname     : chromeos2-devserver9.cros.corp.google.com
    Status       : primary
    Roles        : devserver
    Attributes   : {u'restricted_subnet': u'100.115.224.0/19', u'port': u'8082'}
    Date Created : 2018-02-05 13:51:57
    Date Modified: 2018-02-05 13:51:57
    Note         : None

If it's not in prod, it should probably not be in the prod server database.

We also use that for filtering metrics and other things.
> Technically that is a "beta" devserver to test new the equipment.  Currently being used by davidriley for some devserver work
>
> It is not suppose to be used for production use yet.

OK.  The database entry as it was created problems, so I'd rather
not return to the status quo ante.

davidriley@ - can you describe the current use and why/whether/how
the devserver needs to be in the database at all?

Cc: xixuan@chromium.org
I'm no longer needing exclusive use for testing and I asked xixuan@ (current deputy) for it to be added to the production set of devservers.
> [ ... ] I asked [ ... ] for it to be added to the production set of devservers.

It should have become eligible for use in production the moment
the attributes were fixed in comment#4.

That said, I'd still like to clean up our process here:  This server
sat around unused for some time, and that was undesirable.  Even if
we figured out how to avoid that for the next time, there are two
points on which I'd like to be convinced:
  * That the current manual procedure has up-to-date documentation.
  * That current procedure is sufficiently automated.

Re #5 & #8, Where is the documentation for 'manually add one worked devserver to production'?
Labels: OS-Chrome
On a different note I would have preferred the devserver be return back to continue some optimizations but oh well.

Can we also test changing this
Attributes   : {u'restricted_subnet': u'100.115.224.0/19', u'port': u'8082'}

to
Attributes   : {u'restricted_subnet': u'100.115.128.0/17', u'port': u'8082'}


Re c#10: What optimizations are we talking about?  It's reasonably performant as-is, so I'm not sure if it's a good use of time doing further optimization.

I'd much rather have the time spent by bringing up the other similarly specced machines and turning off the 2x1Gbps and 2x2Gb machines in favour of these new ones.
Server side
Comparison of read cache settings on the RAID card

Network side
Comparison between cut-through vs store-and-forward buffering performance at max load on the network switch

It's one of those once I get it right I can duplicate across the labs.  I'm sure it's fast enough now but if I can squeeze a but more out of it it'd be nice :)

Also need to document the new setup for the rest of the englab-sys-cros team.

Although given jrbarnette's other work (network bandwidth management) can be made moot with an expedited manual rollout so plans are underway already.
Cc: pprabhu@chromium.org
Is this server finally 100% setup correctly?

If yes I would like to change the restricted subnet to 
100.115.128.0/17
Status: Fixed (was: Untriaged)
pprabhu@pprabhu:infra_internal$ atest server modify chromeos2-devserver9.cros.corp.google.com -a restricted_subnet -e 100.115.128.0/17
Attribute `restricted_subnet` of server chromeos2-devserver9.cros.corp.google.com is changed from 100.115.224.0/19 to 100.115.128.0/17.
Server chromeos2-devserver9.cros.corp.google.com is modified successfully.
Hostname     : chromeos2-devserver9.cros.corp.google.com
Status       : primary
Roles        : devserver
Attributes   : {u'restricted_subnet': u'100.115.128.0/17', u'port': u'8082'}
Date Created : 2018-02-05 13:51:57
Date Modified: 2018-02-05 13:51:57
Note         : None


That devserver now has a legit entry.
server_management_library does not have a task to add a devserver to prod. This means that we're currently setting up the server-db entries manually.

The instructions are here: https://sites.google.com/a/google.com/chromeos/for-team-members/infrastructure/add-devserver-to-prod
and are linked from the admin tasks reference page (after a few clicks)
Filed issue 820197 for automating that.

Sign in to add a comment