717106 - Release automation for ESR

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

Now that the ESR announcement has officially gone out, I'm interested in
nailing down the specifics the releng perspective.

Here's the transcript of the conversation kev and I had yesterday about ESR:

http://pastebin.mozilla.org/1443760

I also just had a chat with catlee and rail about this, so now I'm
trying to reconcile the two discussions. Here are the key points as I
understand them:

* we need to use a distinct update channel for the ESR, both for the
mechanical aspect of actually updating but also for uptake monitoring

* for the initial ESR release, this *could* be done as a partner repack
with a distribution ID, but we would then need to create an update
specific to that distribution ID for the *next* ESR release so the users
did not get updated to the mainline release (e.g. 10.0 ESR -> 11.0
non-ESR). That ESR-specific update would either need to change the
update channel, or we would need to continue to prevent fallback updates
by offering ESR-specific updates until we did. AFAICT (I've done casual
testing), any browser set to a release-* channel will try to update to
the main release channel.

* we need to maintain a separate repo for the ESR for backporting of
fixes, security patches, etc.

My main concern is making sure that releng is not signing up for a whole
string of manual, one-off processes. We're bound to screw that up at
some point. We need to get automation is place for this ASAP.

According to the kev, enterprises will test the 10.0 ESR and will want
to be updated to 10.0.1 ESR when 11.0 is released. In this scenario,
does it make more sense mechanically to do the following:

* tag the repo for FIREFOX_10_RELEASE
* clone that changeset as the new FIREFOX_ESR branch
* change the update channel on the new FIREFOX_ESR branch to something
that has no chance of fallback-updating to a mainstream, rapid-release
build, e.g. channel="esr"
* run standard automation to produce builds against this new branch
(rail has concerns about which other channels we'd need for testing
here, e.g. esr-releasetest, ...)

Still not sure who is responsible for backporting fixes to the ESR under
any scenario.

kev: does the 10.0 ESR need to go out concurrently with 10.0
rapid-release, or do we have some leeway on the first one? If so, how much?

Comment 1

[Kev Needham [:kev]]

Backporting fixes falls to Eng, and JP will be leading that.

ESR will ideally go out with 10.0, but we have a little wiggle room (e.g. a day or two) if we need it. Sooner we can determine whether that's needed the better, just so we can give a heads-up.

I know John was concerned with using a completely separate channel and having to maintain code, but we can discuss that further, too.

Comment 2

[[e-mail] (:bhearsum)]

Kev, Catlee, Rail, and I all met about this last week. Here's what I remember from it:
* We'll be using a parallel set of configs for each ESR release - name to be determined. The ESR, from the initial version, will have its own release config, patcher config, update verify config, etc.
* We will use a separate, non-fallback channel for ESR releases
* For initial ESR releases, we will build from the same version as the accompanying general release channel release, but the builds will not be bit-for-bit identical. For the 2nd and later ESR releases we will build from a separate repository with backported fixes

Essentially, we'll be doing these exactly like we do the mozilla-1.9.2 releases right now. We did some looking at our config bumping scripts and as best as I can tell the only one that will need updating is update-verify-bump.pl, which hardcodes "betatest" as the channel: http://hg.mozilla.org/build/tools/file/default/release/update-verify-bump.pl#l203.

The patcher config scripts are OK, because they simply inherit the channel set in the initial config. So, just like with the beta and release configs, we'll have to create the first one by hand, and the bumper will work from then on. ReleaseUpdatesFactory's setChannelData method will probably need an update, too: http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l4747

Comment 3

[Rail Aliiev [:rail]]

We'll be using:

* Version with esr suffix, but the same appVersion; base tag with ESR:

releaseConfig['version']             = '10.0esr'
releaseConfig['appVersion']          = '10.0'
releaseConfig['milestone']           = releaseConfig['appVersion']
releaseConfig['baseTag']             = 'FIREFOX_ESR_10_0'

* Separate relbranch:
releaseConfig['relbranchPrefix']     = 'GECKO_ESR' # or soemthing similar

* mozilla-release l10n repos:
releaseConfig['l10nRepoPath']        = 'releases/l10n/mozilla-release'

* separate build notes URL (releaseConfig['releaseNotesUrl'])

* No beta channel:
releaseConfig['useBetaChannel']      = False

Q&D patch: https://gist.github.com/1622520

Comment 4

[[e-mail] (:bhearsum)]

Which reminds me, builds will be going in 10.0esr-candidates on FTP, and Kev said it was fine to put them in public, alongside the other candidates.

And one more thing, just to be clear: we're not using any of the partner repack infrastructure to do these, despite considering doing so at one point.

Comment 5

[[e-mail] (:bhearsum)]

Despite my efforts to avoid it, this somehow found its way to me!

We talked a little bit more about this today and a couple things came up:
* Where will snippets be pointing? They can't point at the full mirror network.

Comment 6

[[e-mail] (:bhearsum)]

Oops, hit enter too soon.

We talked a little bit more about this today and another thing came up:
* Where will snippets be pointing? They can't point at the full mirror network. They may be able to live on the internal mirrors in releases/, Kev said that we should check with Alex Keybl and Laura Mesa before finalizing that. We might be able to hide them from directory listings if it helps. I don't think we should point them at the candidates directories, because those will get deleted from nightly/ at some point.

If putting them in releases/ isn't OK, we'll have to find a new directory to plop them in. We could probably create /pub/mozilla.org/firefox/esr or equivalent, but I'm really hoping to avoid that, as it severely complicates the set of changes we have to make to the release automation. Given the short time frame, that's going to increase the risk of getting it done right, and on time.

Comment 7

[[e-mail] (:bhearsum)]

I e-mailed Laura, Alex, and Asa and everyone is in agreement that we can put these builds in /pub/mozilla.org/firefox/releases, so let's do that!

Comment 8

[Laura Mesa [:lmesa] [:lvmesa]]

Thanks Ben.  Would it also be possible to have one of the deliverables for this bug is also the bouncer link we talked about over email?

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #8)
> Thanks Ben.  Would it also be possible to have one of the deliverables for
> this bug is also the bouncer link we talked about over email?

Yup, absolutely. Do you want me to paste it here, or e-mail it somewhere?

Comment 10

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to Ben Hearsum [:bhearsum] from comment #9)
> (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #8)
> > Thanks Ben.  Would it also be possible to have one of the deliverables for
> > this bug is also the bouncer link we talked about over email?
> 
> Yup, absolutely. Do you want me to paste it here, or e-mail it somewhere?

Email is fine.  Also, one more question.  I'm  want these builds to be branded differently to be visibly identifiable as the ESR. The logo will stay the same, but we do have a wordmark that would be nice to use in the About window. It would also be nice that the official label be "Firefox ESR" on people's desktops.  Do you want me to attach the wordmark here?

Comment 11

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #10)
> (In reply to Ben Hearsum [:bhearsum] from comment #9)
> > (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #8)
> > > Thanks Ben.  Would it also be possible to have one of the deliverables for
> > > this bug is also the bouncer link we talked about over email?
> > 
> > Yup, absolutely. Do you want me to paste it here, or e-mail it somewhere?
> 
> Email is fine.  Also, one more question.  I'm  want these builds to be
> branded differently to be visibly identifiable as the ESR. The logo will
> stay the same, but we do have a wordmark that would be nice to use in the
> About window. It would also be nice that the official label be "Firefox ESR"
> on people's desktops.  Do you want me to attach the wordmark here?

Here is the wordmark: http://cl.ly/3P460u1y0c470n3Y3T3E

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #11)
> (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #10)
> > (In reply to Ben Hearsum [:bhearsum] from comment #9)
> > > (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #8)
> > > > Thanks Ben.  Would it also be possible to have one of the deliverables for
> > > > this bug is also the bouncer link we talked about over email?
> > > 
> > > Yup, absolutely. Do you want me to paste it here, or e-mail it somewhere?
> > 
> > Email is fine.  Also, one more question.  I'm  want these builds to be
> > branded differently to be visibly identifiable as the ESR. The logo will
> > stay the same, but we do have a wordmark that would be nice to use in the
> > About window. It would also be nice that the official label be "Firefox ESR"
> > on people's desktops.  Do you want me to attach the wordmark here?
> 
> Here is the wordmark: http://cl.ly/3P460u1y0c470n3Y3T3E

Laura, do you have a plain PNG version of the Wordmark? Comparable to this existing one: http://hg.mozilla.org/releases/mozilla-beta/raw-file/default/browser/branding/official/content/about-wordmark.png

Comment 13

[[e-mail] (:bhearsum)]

Spun the branding changes out to bug 720837, where Gavin will hopefully help us sort them out.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: wip buildbot-configs patch

Comment 15

[[e-mail] (:bhearsum)]

Attachment: wip buildbotcustom patch

Comment 16

[[e-mail] (:bhearsum)]

Attachment: wip tools patch

Comment 17

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #10)
> (In reply to Ben Hearsum [:bhearsum] from comment #9)
> > (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #8)
> > > Thanks Ben.  Would it also be possible to have one of the deliverables for
> > > this bug is also the bouncer link we talked about over email?
> > 
> > Yup, absolutely. Do you want me to paste it here, or e-mail it somewhere?
> 
> Email is fine.  Also, one more question.  I'm  want these builds to be
> branded differently to be visibly identifiable as the ESR. The logo will
> stay the same, but we do have a wordmark that would be nice to use in the
> About window. It would also be nice that the official label be "Firefox ESR"
> on people's desktops.  Do you want me to attach the wordmark here?

Ben, can you send me the bouncer link when you have a second.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #17)
> (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #10)
> > (In reply to Ben Hearsum [:bhearsum] from comment #9)
> > > (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #8)
> > > > Thanks Ben.  Would it also be possible to have one of the deliverables for
> > > > this bug is also the bouncer link we talked about over email?
> > > 
> > > Yup, absolutely. Do you want me to paste it here, or e-mail it somewhere?
> > 
> > Email is fine.  Also, one more question.  I'm  want these builds to be
> > branded differently to be visibly identifiable as the ESR. The logo will
> > stay the same, but we do have a wordmark that would be nice to use in the
> > About window. It would also be nice that the official label be "Firefox ESR"
> > on people's desktops.  Do you want me to attach the wordmark here?
> 
> Ben, can you send me the bouncer link when you have a second.

I'll get that to you by EOD. I'm not 100% sure what it's going to be yet :).

Comment 19

[[e-mail] (:bhearsum)]

Bouncer links will be (for en-US):
http://download.mozilla.org/?product=firefox-10.0esr&os=win&lang=en-US
http://download.mozilla.org/?product=firefox-10.0esr&os=osx&lang=en-US
http://download.mozilla.org/?product=firefox-10.0esr&os=linux&lang=en-US
http://download.mozilla.org/?product=firefox-10.0esr&os=linux64&lang=en-US

Comment 20

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to Ben Hearsum [:bhearsum] from comment #19)
> Bouncer links will be (for en-US):
> http://download.mozilla.org/?product=firefox-10.0esr&os=win&lang=en-US
> http://download.mozilla.org/?product=firefox-10.0esr&os=osx&lang=en-US
> http://download.mozilla.org/?product=firefox-10.0esr&os=linux&lang=en-US
> http://download.mozilla.org/?product=firefox-10.0esr&os=linux64&lang=en-US

Ben, is there one link I can attach that would then allow people to choose their download by platform? The way the FAQ is set up, four links is going to be hard to layout.  If not, I can try and find a solution.

Comment 21

[Kohei]

Can someone confirm that the upcoming Firefox ESR 10 builds would be available in all current locales? We know that the discoverability of ESR on the website will be limited to select locales (only English and Japanese).

Comment 22

[[e-mail] (:bhearsum)]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #20)
> (In reply to Ben Hearsum [:bhearsum] from comment #19)
> > Bouncer links will be (for en-US):
> > http://download.mozilla.org/?product=firefox-10.0esr&os=win&lang=en-US
> > http://download.mozilla.org/?product=firefox-10.0esr&os=osx&lang=en-US
> > http://download.mozilla.org/?product=firefox-10.0esr&os=linux&lang=en-US
> > http://download.mozilla.org/?product=firefox-10.0esr&os=linux64&lang=en-US
> 
> Ben, is there one link I can attach that would then allow people to choose
> their download by platform? The way the FAQ is set up, four links is going
> to be hard to layout.  If not, I can try and find a solution.

You'll have to ask Kev or Alex for that - I'm not involved with the website side of things at all, sorry :(.

(In reply to Kohei Yoshino from comment #21)
> Can someone confirm that the upcoming Firefox ESR 10 builds would be
> available in all current locales? We know that the discoverability of ESR on
> the website will be limited to select locales (only English and Japanese).

To the best of my knowledge this is the plan. Kev, can you confirm?

Comment 23

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to Ben Hearsum [:bhearsum] from comment #22)
> (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #20)
> > (In reply to Ben Hearsum [:bhearsum] from comment #19)
> > > Bouncer links will be (for en-US):
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=win&lang=en-US
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=osx&lang=en-US
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=linux&lang=en-US
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=linux64&lang=en-US
> > 
> > Ben, is there one link I can attach that would then allow people to choose
> > their download by platform? The way the FAQ is set up, four links is going
> > to be hard to layout.  If not, I can try and find a solution.
> 
> You'll have to ask Kev or Alex for that - I'm not involved with the website
> side of things at all, sorry :(.

Ok, will ask webdev for a solution. 
> 
> (In reply to Kohei Yoshino from comment #21)
> > Can someone confirm that the upcoming Firefox ESR 10 builds would be
> > available in all current locales? We know that the discoverability of ESR on
> > the website will be limited to select locales (only English and Japanese).

The plan that Patrick drafted for the ESR release states that the only builds that would be available would be in en-US. This is because the EWG group is only in english and that list is the only way users will be able to get information about bug issues, schedule changes etc.  Therefore, if we have localized solutions for the EWG, that's great, but as far as I know, Mozilla Japan is the only other locale that has a similar solution to the EWG. In that sense, I think we should only then be doing en-US and Japanese builds. People can nominate other locales in the EWG list and then it would be Stormy or Kev's responsibility to relay that to rel-eng. I have addressed this question in the FAQ fwiw.  Kev, please correct me if I'm wrong. 


> 
> To the best of my knowledge this is the plan. Kev, can you confirm?

Comment 24

[[e-mail] (:bhearsum)]

Attachment: tools patch, minus update changes

Since we're not going to be running updates for this initial release I decided to drop the update changes for now to reduce the risk and complexity.

This patch is simple, it just adds the mozilla-esr10 release branch to one of the masters, and fixes a regex in the signing scripts to support '10.0esr' versions.

Comment 25

[[e-mail] (:bhearsum)]

Attachment: esr buildbot-configs

Here's the buildbot-configs changes we need for the initial ESR release. We don't need anything in buildbotcustom now, since we're not changing the updater just yet. Here's a summary:
* Create mozilla-esr10 branch on build & test masters, set channel to 'esr'. Otherwise, it's configured like mozilla-release
* Create mozilla-esr10 release config & l10n-changesets.
* Fix staging configs to point at proper mozconfigs (linux vs linux32)
* Create mozconfigs for l10n, and 32-bit linux (used only be the source factory)

Comment 26

[[e-mail] (:bhearsum)]

Attachment: updated tools patch to make mar signing work

Sorry, I also had to update the mar regex.

Comment 27

[[e-mail] (:bhearsum)]

Attachment: add graph server branch for esr

Also adding the lion slaves that are missing re: https://bugzilla.mozilla.org/show_bug.cgi?id=713746#c41

Comment 28

[[e-mail] (:bhearsum)]

Attachment: add builders, too

Almost forgot to add builders for codesighs/leaktest

Comment 29

[[e-mail] (:bhearsum)]

Comment on attachment 592197 [details] [diff] [review]
add builders, too

bug 721831 for the graph server DB update.

Comment 30

[[e-mail] (:bhearsum)]

Still to do here: test out buildbotcustom and updates related tools changes with a 10.0.1esr build in staging

Comment 31

[[e-mail] (:bhearsum)]

I added the following to rsyncd-mozilla-releases.exclude to make sure none of the esr builds go to external mirrors:
- firefox/releases/*esr*
- thunderbird/releases/*esr*

Comment 32

[Laura Mesa [:lmesa] [:lvmesa]]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #20)
> (In reply to Ben Hearsum [:bhearsum] from comment #19)
> > Bouncer links will be (for en-US):
> > http://download.mozilla.org/?product=firefox-10.0esr&os=win&lang=en-US
> > http://download.mozilla.org/?product=firefox-10.0esr&os=osx&lang=en-US
> > http://download.mozilla.org/?product=firefox-10.0esr&os=linux&lang=en-US
> > http://download.mozilla.org/?product=firefox-10.0esr&os=linux64&lang=en-US
> 
> Ben, is there one link I can attach that would then allow people to choose
> their download by platform? The way the FAQ is set up, four links is going
> to be hard to layout.  If not, I can try and find a solution.

Ben, it it possible to get non-locale specific bouncer links? Getting complaints on the EBG list that people can't find the builds outside on en-US.

Comment 33

[[e-mail] (:bhearsum)]

(In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #32)
> 
> (In reply to Laura Mesa [:lmesa] [:lvmesa] from comment #20)
> > (In reply to Ben Hearsum [:bhearsum] from comment #19)
> > > Bouncer links will be (for en-US):
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=win&lang=en-US
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=osx&lang=en-US
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=linux&lang=en-US
> > > http://download.mozilla.org/?product=firefox-10.0esr&os=linux64&lang=en-US
> > 
> > Ben, is there one link I can attach that would then allow people to choose
> > their download by platform? The way the FAQ is set up, four links is going
> > to be hard to layout.  If not, I can try and find a solution.
> 
> Ben, it it possible to get non-locale specific bouncer links? Getting
> complaints on the EBG list that people can't find the builds outside on
> en-US.

This is something outside of the purview of RelEng. There's no such thing as a non-locale specific bouncer link, so you need to wrap the Bouncer links in a page, like http://www.mozilla.org/en-US/firefox/all.html. That's something you'll probably have to work with Kev or Alex on.

Comment 34

[[e-mail] (:bhearsum)]

Attachment: minor fixes to update verify, patcher config bumpers

The patcher config part of this isn't ideal, but having an esrtest-url in all of the configs isn't going to hurt anything, so I didn't want to invest effort trying to avoid that.

The update verify bumper change is to prevent it from adding erroneous lines to an empty config when it tries to split the previous test (which doesn't exist).

This patch also creates the esr10 update verify configs.

Comment 35

[[e-mail] (:bhearsum)]

Attachment: support esr in ReleaseUpdatesFactory

This patch adds a required releaseChannel argument to ReleaseUpdatesFactory. It can be specified in the release config as 'releaseChannel', and falls back to looking at 'update_channel' in the branch config if that's missing. This gets passed along to the update verify bumper, and used in determining the channel data.

I added some tests to the setChannelData function to make sure I wasn't breaking things for other branches, and made the following changes to it:
* Only add the 'beta' channel to self.channels if useBetaChannelForRelease is True, because that's the only time we use it.
* Get rid of useBetaChannel flag because it's no longer necessary now that we have an explicit channel name being passed.
* Use a hacky 'if esr in version' to set the testChannel properly for ESR. I couldn't figure out a generic way to do this.

Comment 36

[[e-mail] (:bhearsum)]

Attachment: buildbot configs updates

* Drop useBetaChannel, add releaseChannel for 1.9.2 (because update_channel is set to 'nightly' in the branch config).
* Bump staging esr release config to 10.0.1, and stage-ffxbld account (instead of mine)

Comment 37

[[e-mail] (:bhearsum)]

Attachment: initial patcher config

Comment 38

[[e-mail] (:bhearsum)]

Callek, the buildbotcustom patch currently up for review probably affects you, too -- you'll need to remove any usage of 'useBetaChannel' that you have, and replace it with 'releaseChannel' and possibly 'useBetaChannelForRelease'. comment #35 has some details.

Comment 39

[Rail Aliiev [:rail]]

Comment on attachment 594247 [details] [diff] [review]
buildbot configs updates

Review of attachment 594247 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +34,5 @@
>      'aus2_ssh_key': 'cltbld_dsa',
>      'hg_username': 'ffxbld',
>      'hg_ssh_key': '~cltbld/.ssh/ffxbld_dsa',
>      'graph_selector': '/server/collect.cgi',
> +    'compare_locales_repo_path': 'users/bhearsum_mozilla.com/compare-locales',

All look good except this hunk. Be careful! :)

Comment 40

[Rail Aliiev [:rail]]

Comment on attachment 594246 [details] [diff] [review]
support esr in ReleaseUpdatesFactory

lgtm

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 594251 [details] [diff] [review]
initial patcher config

Looks fine, lets drop beta-url if that'll not cause bustage in the bumper.

Comment 42

[[e-mail] (:bhearsum)]

Comment on attachment 594247 [details] [diff] [review]
buildbot configs updates

Landed, but will fail checkconfig until the buildbotcustom patch gets landed.

Comment 43

[[e-mail] (:bhearsum)]

Comment on attachment 594251 [details] [diff] [review]
initial patcher config

Apparently I landed a version of this back on the 30th....wth? In any case, I've overwritten it with this version, minus the beta-url lines.

Comment 44

[Rail Aliiev [:rail]]

Comment on attachment 594246 [details] [diff] [review]
support esr in ReleaseUpdatesFactory

http://hg.mozilla.org/build/buildbotcustom/rev/70f0b8a5ac12
I had to land this to fix preproduction failures.

Comment 45

[Rail Aliiev [:rail]]

+ bustage: http://hg.mozilla.org/build/buildbotcustom/rev/120b412257d1

Comment 46

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 594246 [details] [diff] [review]
support esr in ReleaseUpdatesFactory

Yay, no more useBetaChannel hacks.

Comment 47

[Aki Sasaki (not active)]

Comment on attachment 594247 [details] [diff] [review]
buildbot configs updates

Bustage for 11.0b2:
http://hg.mozilla.org/build/buildbot-configs/rev/e6332f35f4e1

We're going to have to do the same for 10.0.1esr or we're going to fail.

Comment 48

[[e-mail] (:bhearsum)]

This all worked fine, save me forgetting to put 'releasetest' as a test channel, which is being tracked in bug 725662.