Skip to content

rename slave to worker #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Nov 9, 2017
Merged

rename slave to worker #291

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
50e653d
rename slave to builder
levlaz Nov 1, 2017
5939cfa
fix GitHub link formatting
levlaz Nov 1, 2017
3d474ee
update wording about GitHub
levlaz Nov 1, 2017
bf4ef83
fix punctuation
levlaz Nov 1, 2017
045dc14
Clean up based on feedback from zware
levlaz Nov 2, 2017
b851f16
add newline, clear up MacOS buildbot worker doc
levlaz Nov 2, 2017
a04ef78
add newline to conf.py
levlaz Nov 2, 2017
1f19863
re-add refernce to footnote
levlaz Nov 2, 2017
58aadf7
clean up mac instructions
levlaz Nov 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions buildbots.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Continuous Integration

To assert that there are no regressions in the :doc:`development and maintenance
branches <devcycle>`, Python has a set of dedicated machines (called *buildbots*
or *build slaves*) used for continuous integration. They span a number of
or *build workers*) used for continuous integration. They span a number of
hardware/operating system combinations. Furthermore, each machine hosts
several *builders*, one per active branch: when a new change is pushed
to this branch on the public Mercurial repository, all corresponding builders
Expand Down Expand Up @@ -209,7 +209,7 @@ Custom builders
When working on a platform-specific issue, you may want to test your changes on
the buildbot fleet rather than just on Travis and AppVeyor. To do so, you can
make use of the `custom builders
<http://buildbot.python.org/all/waterfall?category=custom.stable&category=custom.unstable>`_.
<http://buildbot.python.org/all/#/waterfall?category=custom.stable&category=custom.unstable>`_.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole link doesn't really work anymore. We could try http://buildbot.python.org/all/#/builders?tags=custom.unstable&tags=custom.stable instead.

These builders track the ``buildbot-custom`` short-lived branch of the
``python/cpython`` repository, which is only accessible to core developers.

Expand All @@ -233,4 +233,4 @@ recommend you change (temporarily, of course) the contents of the
the ``Tools/buildbot/test.bat`` script.

.. seealso::
:ref:`buildslave`
:ref:`buildworker`
134 changes: 67 additions & 67 deletions buildslave.rst → buildworker.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@

.. _buildslave:
.. _buildworker:

Running a buildslave
====================
Running a buildworker
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"buildworker" doesn't really fit the terminology now used by Buildbot and doesn't flow as easily as "buildslave", either. Just "worker" fits Buildbot's new terminology, with "buildbot worker" for disambiguation where necessary. This applies pretty much everywhere in this PR, and it's a judgement call as to whether each instance needs disambiguation.

=====================

.. highlight:: bash

Python's :ref:`buildbots` system was discussed earlier. We sometimes refer to
the collection of *build slaves* as our "buildbot fleet". The machines that
the collection of *build workers* as our "buildbot fleet". The machines that
comprise the fleet are voluntarily contributed resources. Many are run by
individual volunteers out of their own pockets and time, while others are
supported by corporations. Even the corporate sponsored buildbots, however,
tend to exist because some individual championed them, made them a reality, and
is committed to maintaining them.

Anyone can contribute a buildbot to the fleet. This chapter describes how
to go about setting up a buildslave, getting it added, and some hints about
to go about setting up a buildworker, getting it added, and some hints about
buildbot maintenance.

Anyone running a buildbot that is part of the fleet should subscribe to the
Expand All @@ -24,18 +24,18 @@ mailing list. This mailing list is also the place to contact if you want to
contribute a buildbot but have questions.

As for what kind of buildbot to run...take a look at our `current fleet
<http://buildbot.python.org/all/buildslaves>`_. Pretty much anything that isn't
<http://buildbot.python.org/all/#/builders>`_. Pretty much anything that isn't
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be workers rather than builders.

on that list would be interesting: different Linux/UNIX distributions,
different versions of the various OSes, other OSes if you or someone are
prepared to make the test suite actually pass on that new OS. Even if you only
want to run an OS that's already on our list there may be utility in setting it
up: we also need to build and test python under various alternate build
configurations. Post to the mailing list and talk about what you'd like to
up. We also need to build and test python under various alternate build
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the colon isn't quite right, but I'd rather just replace it with a semicolon than a full stop. And the space doesn't need to be removed before Post.

</nitpick>

configurations. Post to the mailing list and talk about what you'd like to
contribute.


Preparing for buildslave setup
------------------------------
Preparing for buildworker setup
-------------------------------

Since the goal is to build Python from source, the system will need to have
everything required to do normal python development: a compiler, a linker, and
Expand All @@ -45,33 +45,33 @@ everything required to do normal python development: a compiler, a linker, and
compiled python.

In order to set up the buildbot software, you will need to obtain an identifier
and password for your buildslave so it can join the fleet. Email
[email protected] to discuss adding your buildslave and to obtain the
needed slavename and password. You can do some of the steps that follow
and password for your buildworker so it can join the fleet. Email
[email protected] to discuss adding your buildworker and to obtain the
needed workername and password. You can do some of the steps that follow
before having the credentials, but it is easiest to have them before
the "buildslave" step below.
the "buildworker" step below.


Setting up the buildslave
-------------------------
Setting up the buildworker
--------------------------

Conventional always-on machines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You need a recent version of the `buildbot <http://buildbot.net/>`_ software,
and you will probably want a separate 'buildbot' user to run the buildbot
software. You may also want to set the buildbot up using a virtual
environment, depending on you manage your system. We won't cover how to that
environment, depending on how you manage your system. We won't cover how to that
here; it doesn't differ from setting up a virtual environment for any other
software, but you'll need to modify the sequence of steps below as appropriate
if you choose that path.

For Linux:

* If your package manager provides the buildbot slave software, that is
* If your package manager provides the buildbot worker software, that is
probably the best way to install it; it may create the buildbot user for
you, in which case you can skip that step. Otherwise, do ``pip install
buildbot-slave``.
buildbot-worker``.
* Create a ``buildbot`` user (using, eg: ``useradd``) if necessary.
* Log in as the buildbot user.

Expand All @@ -82,40 +82,40 @@ For Mac:
* Log in as the buildbot user.
* Install either the Python 2.7 bundle from python.org [#]_, or pip.
* Open a terminal window.
* Execute ``pip install buildbot-slave``.
* Execute ``pip install buildbot-worker``.

For Windows:

* Create a buildbot user as a "standard" user.
* Install the latest version of Python 2.7 from python.org.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This no longer needs to specify 2.7; we can use any version we want now :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This series of instructions was a tad bit confusing so I specified that the last step is only necessary if using pip.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little lost on your comment here. My comment was just that we can remove "2.7" from this line :)

* Open a Command Prompt.
* Execute ``python -m pip install pypiwin32 buildbot-slave`` (note that
* Execute ``python -m pip install pypiwin32 buildbot-worker`` (note that
``python.exe`` is not added to ``PATH`` by default, making the
``python`` command accessible is left as an exercise for the user).

In a terminal window for the buildbot user, issue the following commands (you
can put the ``buildarea`` wherever you want to)::

mkdir buildarea
buildslave create-slave buildarea buildbot.python.org:9020 slavename slavepasswd
buildworker create-worker buildarea buildbot.python.org:9020 workername workerpasswd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/buildworker/buildbot-worker/

This applies everywhere the buildslave command was used.


(Note that on Windows, the ``buildslave`` command will be in the
(Note that on Windows, the ``buildworker`` command will be in the
:file:`Scripts` directory of your Python installation.)

Once this initial slave setup completes, you should edit the files
Once this initial worker setup completes, you should edit the files
``buildarea/info/admin`` and ``buildarea/info/host`` to provide your contact
info and information on the host configuration, respectively. This information
will be presented in the buildbot web pages that display information about the
builders running on your buildslave.
builders running on your buildworker.

You will also want to make sure that the buildslave is started when the
You will also want to make sure that the buildworker is started when the
machine reboots:

For Linux:

* Add the following line to ``/etc/crontab``::

@reboot buildslave restart /path/to/buildarea
@reboot buildworker restart /path/to/buildarea

Note that we use ``restart`` rather than ``start`` in case a crash has
left a ``twistd.pid`` file behind.
Expand All @@ -126,13 +126,13 @@ For OSX:

mkdir bin

* Place the following script, named ``run_slave.sh``, into that directory::
* Place the following script, named ``run_worker.sh``, into that directory::

#!/bin/bash
export PATH=/usr/local/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:$PATH
export LC_CTYPE=en_US.utf-8
cd /Users/buildbot/buildarea
twistd --nodaemon --python=buildbot.tac --logfile=buildbot.log --prefix=slave
twistd --nodaemon --python=buildbot.tac --logfile=buildbot.log --prefix=worker

If you use pip with Apple's system python, add '/System' to the front of
the path to the Python bin directory.
Expand All @@ -147,14 +147,14 @@ For OSX:
<plist version="1.0">
<dict>
<key>Label</key>
<string>net.buildbot.slave</string>
<string>net.buildbot.worker</string>
<key>UserName</key>
<string>buildbot</string>
<key>WorkingDirectory</key>
<string>/Users/buildbot/buildarea</string>
<key>ProgramArguments</key>
<array>
<string>/Users/buildbot/bin/run_slave.sh</string>
<string>/Users/buildbot/bin/run_worker.sh</string>
</array>
<key>StandardOutPath</key>
<string>twistd.log</string>
Expand All @@ -167,27 +167,27 @@ For OSX:
</dict>
</plist>

The recommended name for the file is ``net.buildbot.slave``.
The recommended name for the file is ``net.buildbot.worker``.

For Windows:

* Add a Scheduled Task to run ``buildslave start buildarea`` as the
* Add a Scheduled Task to run ``buildworker start buildarea`` as the
buildbot user "when the computer starts up". It is best to provide
absolute paths to the ``buildslave`` command and the :file:`buildarea`
absolute paths to the ``buildworker`` command and the :file:`buildarea`
directory. It is also recommended to set the task to run in the
directory that contains the :file:`buildarea` directory.

* Alternatively (note: don't do both!), set up the buildslave
* Alternatively (note: don't do both!), set up the buildworker
service as described in the `buildbot documentation
<http://trac.buildbot.net/wiki/RunningBuildbotOnWindows#Service>`_.

To start the buildslave running for your initial testing, you can do::
To start the buildworker running for your initial testing, you can do::

buildslave start buildarea
buildworker start buildarea

Then you can either wait for someone to make a commit, or you can pick a
builder associated with your buildslave from the `list of builders
<http://buildbot.python.org/all/buildslaves>`_ and force a build.
builder associated with your buildworker from the `list of builders
<http://buildbot.python.org/all/#/builders>`_ and force a build.

In any case you should initially monitor builds on your builders to make sure
the tests are passing and to resolve any platform issues that may be revealed
Expand All @@ -196,19 +196,19 @@ only of failures on your builders, so doing periodic spot checks is also a good
idea.


Latent slaves
^^^^^^^^^^^^^
Latent workers
^^^^^^^^^^^^^^

We also support running `latent buildslaves
<http://docs.buildbot.net/current/manual/cfg-buildslaves.html#latent-buildslaves>`_
on the AWS EC2 service. To set up such a slave:
We also support running `latent buildworkers
<http://docs.buildbot.net/current/manual/cfg-workers.html#latent-workers>`_
on the AWS EC2 service. To set up such a worker:

* Start an instance of your chosen base AMI and set it up as a
conventional slave.
* After the instance is fully set up as a conventional slave (including
slave name and password, and admin and host information), create an AMI
conventional worker.
* After the instance is fully set up as a conventional worker (including
worker name and password, and admin and host information), create an AMI
from the instance and stop the instance.
* Contact the buildmaster administrator who gave you your slave
* Contact the buildmaster administrator who gave you your worker
name and password and give them the following information:

* Instance size (such as ``m4.large``)
Expand All @@ -221,63 +221,63 @@ on the AWS EC2 service. To set up such a slave:
The buildmaster cannot guarantee that it will always shut down your
instance(s), so it is recommended to periodically check and make sure
there are no "zombie" instances running on your account, created by the
buildbot master. Also, if you notice that your slave seems to have been
buildbot master. Also, if you notice that your worker seems to have been
down for an unexpectedly long time, please ping the `python-buildbots
<https://mail.python.org/mailman/listinfo/python-buildbots>`_ list to
request that the master be restarted.

Latent slaves should also be updated periodically to include operating system
Latent worker should also be updated periodically to include operating system
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/worker/workers/

or other software updates, but when do do such maintenance is largely up to you
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/do do/to do/ (please, I know it's not your typo :))

as the slave owner. There are a couple different options for doing such
as the worker owner. There are a couple different options for doing such
updates:

* Start an instance from your existing AMI, do updates on that instance,
and save a new AMI from the updated instance. Note that (especially for
Windows slaves) you should do at least one restart of the instance after
Windows workers) you should do at least one restart of the instance after
doing updates to be sure that any post-reboot update work is done before
creating the new AMI.
* Create an entirely new setup from a newer base AMI using your existing
slave name and password.
worker name and password.

Whichever way you choose to update your AMI, you'll need to provide the
buildmaster administrators with the new AMI ID.


Buildslave operation
--------------------
Buildworker operation
---------------------

Most of the time, running a buildslave is a "set and forget" operation,
Most of the time, running a buildworker is a "set and forget" operation,
depending on the level of involvement you want to have in resolving bugs
revealed by your builders. There are, however, times when it is helpful or
even necessary for you to get involved. As noted above, you should be
subscribed to ``[email protected]`` so that you will be made
aware of any fleet-wide issues.

Necessary tasks include, obviously, keeping the buildbot running. Currently
the system for notifying buildbot owners when their slaves go offline is not
the system for notifying buildbot owners when their workers go offline is not
working; this is something we hope to resolve. So currently it is helpful if
you periodically check the status of your buildslave. We will also contact you
you periodically check the status of your buildworker. We will also contact you
via your contact address in ``buildarea/info/admin`` when we notice there is a
problem that has not been resolved for some period of time and you have
not responded to a posting on the python-buildbots list about it.

We currently do not have a minimum version requirement for the buildslave
We currently do not have a minimum version requirement for the buildworker
software. However, this is something we will probably establish as we tune the
fleet, so another task will be to occasionally upgrade the buildslave software.
fleet, so another task will be to occasionally upgrade the buildworker software.
Coordination for this will be done via ``[email protected]``.

The most interesting extra involvement is when your buildslave reveals a unique
The most interesting extra involvement is when your buildworker reveals a unique
or almost-unique problem: a test that is failing on your system but not on
other systems. In this case you should be prepared to offer debugging help to
the people working on the bug: running tests by hand on the buildslave machine
the people working on the bug: running tests by hand on the buildworker machine
or, if possible, providing ssh access to a committer to run experiments to try
to resolve the issue.


Required Ports
--------------

The buildslave operates as a *client* to the *buildmaster*. This means that
The buildworker operates as a *client* to the *buildmaster*. This means that
all network connections are *outbound*. This is true also for the network
tests in the test suite. Most consumer firewalls will allow any outbound
traffic, so normally you do not need to worry about what ports the buildbot
Expand Down Expand Up @@ -325,10 +325,10 @@ suite.
Security Considerations
-----------------------

We only allow builds to be triggered against commits to the CPython repository,
or committer-initiated branches hosted on hg.python.org. This means that the
code your buildbot will run will have been vetted by a committer. However,
mistakes and bugs happen, as could a compromise, so keep this in mind when
We only allow builds to be triggered against commits to the
`CPython repository on GitHub <https://github.com/python/cpython>`_.
This means that the code your buildbot will run will have been vetted by a committer.
However, mistakes and bugs happen, as could a compromise, so keep this in mind when
siting your buildbot on your network and establishing the security around it.
Treat the buildbot like you would any resource that is public facing and might
get hacked (use a VM and/or jail/chroot/solaris zone, put it in a DMZ, etc).
Expand Down
8 changes: 8 additions & 0 deletions conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,3 +229,11 @@
('index', 'pythondevelopersguide', u"Python Developer's Guide Documentation",
[u'Brett Cannon'], 1)
]

# ignore linkcheck anchors for /#/$ANCHOR since it is used for
# dynamic pages such as http://buildbot.python.org/all/#/console
# http://www.sphinx-doc.org/en/stable/config.html?highlight=linkcheck#confval-linkcheck_anchors_ignore
linkcheck_anchors_ignore = [
# match any anchor that starts with a '/' since this is an invalid HTML anchor
'\/.*',
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing newline at the end of the file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, even my linter yelled at me and I ignored it. Thank you :)

2 changes: 1 addition & 1 deletion index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ Full Table of Contents
compiler
coverity
clang
buildslave
buildworker
motivations
gitbootcamp

Expand Down