I became a Mozillian more than twelve years ago. I’m not sure whether the term “Mozillian” was even being used back then, I definitely didn’t hear it. Also, I didn’t actually realize what happened — to me it was simply a fascinating piece of software, one that allowed me to do a lot more than merely consume it passively. I implemented changes to scratch my own itch, yet these changes had an enormous impact at times. I got more and more involved in the project, and I could see it grow and evolve over time.
Not all of the changes were positive in my eyes, so this blog post hit a nerve with me: is Mozilla still an open source project? How is Mozilla different from Google or Microsoft who also produce open source software? See Android for example: while being technically open source, the project around it is completely dominated by Google. Want to contribute? Apply at Google!
In my opinion, making the source code accessible is a necessary requirement for an open source project, but clearly not the only one. It should be possible for people to get involved and make a difference. I could identify the following contributing factors:
1. Source code availability and documentation
This sounds trivial but it isn’t, particularly with a project the size of Mozilla. Finding the Firefox source code is fairly easy, but it already gets more complicated if you need the build configuration of the Firefox release builds. And when you start looking into web services that Firefox relies on things get really tricky, with the source code hidden under cryptic codenames and distributed more or less randomly between hg.mozilla.org and various GitHub organizations. Even if you find the source code, chances are that setting up your own test environment will be hard to impossible.
There is also another aspect here: seeing a piece of code often isn’t enough, you need to see the reasoning behind it. That’s why the information contained in the revision history is so important. Unfortunately, I’m getting the impression that this aspect is increasingly getting neglected. I frequently see changes linking to bugs without any description whatsoever, or where the description is so cryptic that it is only understandable to some inner circle of developers. It also seems that Mozilla no longer has a consistent policy of opening up security-relevant bugs after they have been fixed. So the context of these changes stays hidden for a very long time.
2. Contribution processes
Contribution processes have always been a sore spot with Mozilla. Finding corresponding documentation has always been too hard and the process in general too complicated. Contributors had to deal with depressingly long review times and lack of mentors. This is a tough topic, and finding the right balance between “just do it yourself” and “help somebody else do it” is hard. However, I’ve had the impression that Mozilla got too comfortable hiring people to do the job and neglected making contribution easier.
3. Decision transparency
This might be the most disappointing aspect of all of them: that you can contribute your effort but have no way of making your voice heard or even understanding the direction in which the project is heading. I think the Australis project was one of the positive examples here: the goals and each step of the project were documented, feedback requested during various stages of the project, and the result was released when it was really ready. Sure, you cannot make everybody happy and lots of people ended up opposing Australis for whatever reasons. It’s a fact: when you ask your existing user base then typically most people will either be indifferent or oppose a change, simply due to self-selection bias. But I did have the impression that feedback was taken seriously on this one.
Sadly, that’s not always the case. I seem to recognize a tendency within Mozilla to delay public discussion until late stages of a project in order to avoid negative press. This is very unfortunate because people outside Mozilla cannot contribute until all the major decisions are already done (occasionally this even seems to apply to key figures within Mozilla). Contributors feel excluded and complain about Mozilla’s lack of transparency.
There are more issues here of course. There is the sheer amount of information, and the fact that it’s very non-obvious which information sources to follow for each particular topic — is X being discussed on Planet Mozilla or Google Groups? Bugzilla? Discourse? Air Mozilla? Something else?
There are also unclear hierarchies and responsibilities within Mozilla. Do you still remember those press articles citing a person with an impressive title who wasn’t actually qualified to speak for the project in question? Yes, journalists are very confused by this, and so are contributors. Already figuring out who is in charge of which project is a non-trivial task. I guess that this wiki page is the way to go but it isn’t easy to find and it doesn’t seem terribly up-to-date either.
And then there are people making strange statements on the Mozilla blog which seem to be very hard to justify, yet these are announced as new Mozilla policies without any obvious way to comment. How come that they are speaking for the entire Mozilla project? They definitely didn’t ask for community input, and I am unable to recognize relevant contributions to the project either (that’s assuming that Mozilla is still a meritocracy).
Conclusion
Don’t get me wrong, Mozilla is still a fascinating and unique project. Building up a large open source project and creating the right balance between contributors and paid employees is a very complex problem, and there is a lot to be learned from Mozilla here. However, I have the impression that the balance is increasingly shifting towards paid employees, and that the people at Mozilla understanding why this is a problem are getting fewer. I would love to have somebody prove me wrong.
Comments
How is contributing hard atm? I mean, I would really like to know so that we can improve that. I try to prioritize reviews from non-Moco employees over the ones coming from employees, and I think many others have the same principal.
(though, I admit, right now there is a review from a non-MoCo employee which has waited for too long, it is just that it is rather difficult one and there has been plenty of other review requests).
And about decision transparency, that seems to vary greatly depending on the subject. Core Gecko decisions, at least the ones I know about,
very rarely happen behind the closed doors, and I for example don’t participate to any ‘hidden’ meetings (except workweeks, but I don’t think anything really new related to core Gecko work has ever decided there) about what we should do next. It is all in bugzilla, IRC and newsgroups/mailing-lists. You’re very right that “the sheer amount of information” is hard to manage if you’re not following all the channels, often
including also external channels like what is happening in WhatWG or W3C. But that has nothing to do whether you’re paid or volunteer contributor, it is equally hard for them both.
In the core Gecko level at least meritocracy is very much still there. As a good example you must get a DOM peer review for any webidl changes, otherwise commit hook will prevent pushing the patch. (webidl is the way to express the APIs Gecko exposes to the web.)
Contributing isn’t hard – for me that is. But for somebody who has never done it the process outlined on https://developer.mozilla.org/en-US/docs/Introduction is rather overwhelming. And that one doesn’t even mention unit tests, which is something new contributors frequently seem to fail at. Back in 2004 I outlined a bunch of code issues in a bug and then some Brendan Eich guy jumped in and fixed them – that was great, because I wouldn’t even have known how to test the changes in a build. And – sure, I know that I simply got lucky there and I don’t really have any solutions to offer. It’s really complicated as I said already. I merely have the impression that the discussion around making contributing simpler more or less died away.
As to getting relevant information: Mozilla employees aren’t really in the same position there, simply because they always have somebody who can tell them what information sources to follow. Also, they have eight hours a day to spend on their project – staying up-to-date is part of the job description. Contributors who just need to figure out something don’t have anybody to tell them what is relevant and what isn’t, and they typically don’t have the time to read everything. That clearly is something that can be solved, by documenting the most relevant information sources for each project and making that documentation easy to find. But that also implies that some channels have to be used consistently for all important communication, even if it means duplicating discussions.
Mozilla certainly has the same policy we always had for security bugs: we open them up once we’ve dropped support for the versions affected by the bug. That’s if we ignore complications around bugs that also affect other software projects.
What that means in practice is that if a security fix is not backported to an ESR, then it stays hidden until the next ESR release. I expect that’s the long pole on opening up security bugs nowadays…
For the rest, I think the issue with reviews is very strongly module-dependent. And I agree that the decisionmaking has been becoming more opaque. :(
I cannot confirm that. I just went through https://www.mozilla.org/en-US/security/known-vulnerabilities/firefox/ again and there is still this issue: most bugs all the way down to Firefox 31 are still closed, and even bug 1005584 which was fixed in Firefox 30. This makes very little sense security-wise given that the current ESR is Firefox 38, and that finding the change by bug number is trivial.
Note that this policy was never applied to projects other than Gecko/Firefox in the first place. For AMO changes for example I always had to ask, otherwise the bugs would stack locked forever.
As someone who’s been involved with Mozilla for a long time and is now an employee, it made me sad to read this post. I don’t recognize the Mozilla you describe.
For source code – can you explain what you mean by “cryptic codenames” and “various github organizations” ? I’m only aware of one Mozilla github organization, and you don’t need any code from github to build mozilla-central into Firefox – just hg.m.o.
As for commits – at least in hg.m.o, I’m pretty sure the only commits without anything besides a bug number and reviewer are security-related commits. What other commits are you seeing that only list bug numbers?
We are actively working to make contributing to Mozilla easier. We’ve improved triage in bugzilla (ie more bugs filed are now getting attention more like what you described Brendan did to yours), we are improving our reviewing tools, people are working on making it possible to submit pull requests from github through that infrastructure (I think scheduled for Q4, but don’t quote me on that), we’ve improved our build system and are continuing to do so (so that for frontend changes, you will no longer need to compile all the native code), we’ve improved our mentoring infrastructure immensely (mentor field on bugs, http://www.joshmatthews.net/bugsahoy/, etc.), we’re getting people involved through outreachy – a lot of us are actively involved with making this better.
Unlike the other sections, you didn’t really provide specific examples regarding contributing as to what was difficult, beyond “long review times”. I think the difficult thing used to be finding a reviewer (which bugzilla now helps with)… I deal with most if not all review requests with 1 day, and I know a number of the other Firefox peers have similar goals for how they treat reviews. (Ironically, I’m at a conference tomorrow until Wednesday, and so I’m not accepting review requests at the moment – because I know that it would be difficult or impossible for me to do them in a timely manner. Others on the Firefox team will be able to do them. I believe that now that we make this explicit in bugzilla (ie it is actually impossible to ask me to review your patch during these days), that is actually better than accepting the requests and then not replying to them for a week.). I don’t believe there’s currently a way to easily search for outstanding delayed review requests in a module. I’ll ping some of the folks on the contribution/community teams to see what’s happening to improve that (and to suggest a few things). In the meantime, if you know of languishing review requests in the Firefox product, please consider this an explicit request to needinfo me on those bugs (which I’ve not blocked while I’m away). I’ll either deal with them myself or try to forward them to an appropriate person.
For discussion and decisionmaking: everything I’ve ever been involved with has been discussed on mailing lists, which are almost all mirrored to newsgroups & google groups. We moved Firefox discussions to firefox-dev (https://wiki.mozilla.org/Firefox/firefox-dev) a few years back, which is mirrored to Google Groups but not the newsgroups, because it is a moderated list (and we can’t moderate NNTP). Planet is not normally used for structured discussion about projects, though people sometimes post updates about things they find exciting. I don’t know of any decisions (or discussions) relating to Firefox development that happen on Discourse.
The Mozilla project is pretty big though – it seems to me like generally, people figure out “which bit” they want to get involved with, and then they go do that, and in the process they realize which parts of the firehose are in use by that part of the project. That makes sense to me: newsgroups and IRC are not an easy medium for discussions about web literacy and digital rights with people whose core competency isn’t engineering. Likewise, if you told all the engineers they’d have to move away from those technologies to Discourse or airmo, I expect there’d be uproar. :-)
I’d also say a lot of what I do day-to-day doesn’t need structured decision-making. Of course we should fix security bugs; of course we should make Firefox work properly with on-screen keyboards on Windows tablets; etc. etc.
Do you have concrete examples of what decisions you felt were made behind closed doors and should have involved the community earlier?
I’m also a bit mystified about the “strange statements on the Mozilla blog”. I looked through the last few months of posts on blog.mozilla.org. I didn’t see anything surprising (we work on webmaker, Firefox Hello, open video, policy stuff about net neutrality, board/org shuffles that warrant publishing about … which bit of that is surprising? Did I miss it?). Can you be more explicit about what surprised you or needed an explanation?
As I said, finding Firefox source code is easy. Maybe I’m going in too deep by I’m currently aware of at least five Mozilla organizations on GitHub: mozilla, mozilla-jetpack, mozilla-services, mozilla-metrics and mozillaweb. And I know by now that Mozilla’s crash processing server is called “socorro” and that AMO is called “olympia” – but getting there took some time.
As to lack of context in bugs, the most recent example was https://bugzilla.mozilla.org/show_bug.cgi?id=1197206 vs. https://bugzilla.mozilla.org/show_bug.cgi?id=1210498 (I filed the latter because I had no idea what the former was about). I cannot help with specific examples from Firefox right now, but I’ve definitely hit a number of bugs simply referencing some internal (or unlinked) spec without bothering to provide sufficient information.
You can probably guess what decision got me surprised the most lately and was presented as a fact: WebExtensions. So far I’m not aware of any attempts to consult the add-on developers about the shortcomings of the current add-on system (or Chrome’s) before making a decision. Not just that, it seems that people with relevant experience within Mozilla were left out as well.
If you want to know specific examples of Mozilla Blog posts, there is for example https://blog.mozilla.org/netpolicy/2014/08/21/trust-should-be-the-currency/ and its sequel https://blog.mozilla.org/blog/2015/09/24/mozillas-vision-for-a-healthy-sustainable-web/. Maybe that’s just because I have a bit of experience in this area, but the attempts to declare privacy as the central problem and deny all other aspects here seem extremely constructed.
Wladimir, I hear you. I was a Mozilla employee for a few years from 2010 to 2014. During that time, I initially tried to behave as if I were “just another contributor” in an open-source project, and I initially tried to spend as much time as possible mentoring new contributors, and I tried to document status and projects on wiki pages. What killed that in my experience was:
1) with the shift to “mobile” and Firefox OS and generally the top-down reorg that we saw between 2011 and 2013, it became had to argue for doing anything that didn’t fit in our quarterly goals (which, for engineers like us, consisted mostly of steps required to make mobile / Firefox OS work).
2) with mozilla becoming a larger organization, more things became political, including in engineering (especially under the rule of the evil two-headed CTO and its lieutenants). So a lot of “decisions” didn’t have a very rational justification that could have been explained on a public wiki page.
Hmm, looks like I can’t reply to your reply.
I’m not sure what’s going on with bug 1005584 but I will find out.
Yes, Textpattern doesn’t really support replies – I added a hack in order to reply myself.
It’s really not just bug 1005584, that’s merely an extreme example. There are loads of bugs that should have been opened months ago but weren’t for some reason.
So the repository names… I don’t know – all I use in mozilla’s github repos is gecko-dev (for CVS history) and readability, both of which are pretty clear in their purpose. I don’t know why webdev and IT use codenames for their repos, but I agree that it isn’t likely to be conducive to contribution…
Regarding the posts you flagged, Denelle is “Chief Legal and Business Officer at Mozilla Corporation” as it says on the blogpost. I think the first post is more related to trust than privacy, but either way – they don’t seem that different from other legal/policy posts we’ve had in the past when we had different legal counsel. I’m sure you could email Denelle or legal if you think they’re missing something important – but yes, it probably does have something to do with your perspective. I didn’t bother reading the posts at the time they were posted, as they clearly spoke about our views, intentions and mission (which I guess I assume I am familiar with), rather than about concrete steps we are/aren’t taking (which obviously does change, and so it tends to be worth reading about)… :-)
The WebExtensions thing… I’m sorry if you felt that came as a surprise. No, I didn’t think it was a surprise, and I couldn’t guess that it was what you were going to bring up. This was discussed pretty extensively at Whistler, and I had heard mention of it before then. I’m not sure to what degree there was secrecy about this rather than just people starting to work on it because they thought it was a good idea, and not finding it necessary to talk about it until it was more than just a concept. I would agree that there wasn’t a wide pre-initial-implementation discussion to sort of ask permission/guidance from people outside MoCo to do it. I also think that the community outside of MoCo is now pretty involved in both that and the other work going on with add-ons. I’m not sure to what degree it makes sense to talk about pre-decision there… What would you have wanted people at MoCo to do instead?
The thing is, this doesn’t seem to me as a thing we’re doing for its own sake – it’s a consequence of other realities. Take XUL. Its bad interactions with HTML layout and its poor support for commonly desired things (<label class=“text-link”/> is horrible and needs to go away) both mean that it’s often clunky and hard to work with, and it doesn’t help people contribute, either (which was one of your other points…). The tight link between the implementation of Firefox and add-ons caused by the XUL style of writing add-ons is constantly causing issues. The final nail in the coffin is security and tabs-in-a-separate-process. We could have decided to do a third go at the “add-on” concept that wasn’t webextensions, which would have been just as hard to switch to for existing add-ons, but on top of that we would have been left as the only browser beside Edge, Chrome, Opera, … that didn’t support the existing de facto standard. I guess I don’t really see what the alternative was, and from that perspective it feels much less like a decision.
As I said, I was not surprised by XUL being deprecated – it was obvious that this would come eventually. The weird part was the fact that the Add-on SDK was de facto deprecated at the same time (just after it finally matured with JPM and Firefox 38), and that the people involved didn’t seem to be aware of the shortcomings of Chrome’s extension APIs that they decided to copy. When I asked people kept arguing that the Add-on SDK underachieved – but it never had the kind of resources that were committed to WebExtensions now. I cannot shake off the feeling that WebExtensions are a seriously misinformed decision, people deciding to start from scratch just because that’s more interesting than improving what you already have. But it’s clearly too late to discuss that now, now we can only talk about WebExtensions implementation details.
Gijs, a good example of absence of any public discussion, and rationale except “Chrome did that, and we should too” can be found in this bug – https://bugzilla.mozilla.org/show_bug.cgi?id=1187005
Actually, the discussion around NPAPI removal has been going on for a while, and plenty of rationale has been given. Sure, issues during the transitioning period aren’t really avoidable. But adding support for legacy features to new platforms (and 64 bit Firefox is a new platform) doesn’t make much sense. As you might have noticed, Mozilla is addressing this problem from a different angle: by making the problematic websites switch away from Silverlight. They have to do that anyway, in order to support Chrome and Edge, so it’s only a matter of time.
If Firefox doesn’t provide unique, why users would use it instead of Edge or Chrome? Sites will be working fine in them, Edge is built-in in majority of desktop OSes, Chrome is providing integration with Google services, what Firefox will provide? Better “privacy”? Doesn’t looks like users really care about it, despite all fuzz from Mozilla regarding that, Firefox market share doesn’t grow (or even shrink).
And personally I don’t care about Silverlight much, more about other plugins (gaming, corporate, etc), throwing people using them away, or forcing to pay for updating to “new technologies” doesn’t looks like good plan
I don’t think that security hazards are the kind of unique functionality somebody should provide. That’s exactly the reason why the browser vendors are cooperating on this one – so that they can get rid of NPAPI and all the issues related to it once and for all.
FWIW, re “improving what you already have” for Jetpack… Jetpack’s security model was never seriously enforced, and bolting it back on after the fact would have been difficult/impossible, because everything and their dog uses third-party modules which just require(“chrome”), which boils down to a XUL add-on in how tightly it’s linked to the platform (and how much breaks/broke due to e10s and other changes).
There might have been ways to deal with that (ie get rid of require(“chrome”), replace with more modules, add actual security enforcement), but it would have meant an equal amount of deprecation of existing add-ons and in-product code, I think. That and the technical debt inside jetpack with all the actors and all kinds of other code which, to the best of my knowledge, nobody was particularly fond of, made “make jetpack better” less appealing than the webextensions option (to me, anyway) — at least with webextensions we get compatibility with existing other add-ons to make up for all the stuff that will start breaking. If we didn’t do that, we would have had to start from scratch, and even most if not all jetpack add-ons would have needed changes.
Sure, most Jetpack-based add-ons are breaking out of the sandbox – not because they like that, but simply because there is no other option. Mozilla finally decided to invest some resources into giving add-on developers the APIs they need, so why not add them to Jetpack? Once the APIs are in place, the low-level APIs could be deprecated – meaning warnings in Browser Console, pushback on add-on reviews etc. You know the drill, this has been done a number of times already, also for Jetpack changes. The difference to WebExtensions is: no trial-and-error developing everything from scratch (Google devs have some experience with their API concepts but Mozilla has none), a clear migration path for existing extensions instead of boiling the ocean, existing knowledge of the extension developers can be reused. Sure, nobody can claim that this could be done in six months – but that timeline is completely unrealistic for WebExtensions as well.
Yes, Jetpack has technical debt – but so do Chrome APIs. People seem to have been overestimating the former and massively underestimating the latter (mostly because they weren’t familiar with Chrome extensions). And compatibility with existing add-ons is a bogus argument in my opinion – it’s one that gets you into the bug-for-bug compatibility hell and limits your ability to innovate.
Regarding bug 1005584 specifically (while recognizing you mentioned it as one of a class, not something to fixate on). Maybe Boris followed up directly, I dunno. But in case he didn’t.
I don’t have any special information on that bug, beyond what I can see in it — so no offline discussions, IRC private-message discussions, things of that nature. (But still access to that particular security bug, admittedly more than all but a few people have.) But if I read the bug correctly, particularly comment 46 (feel free to check me eventually!), there’s a specific reason that bug remains closed, that’s unrelated to your concerns, that — given I lack full context on that bug — I won’t attempt to explain, even hand-wavily.
I agree security bugs can just be frustrating, tho. Even security group members have been hit here, recently, as security-bug access has become increasingly compartmentalized by team. (So for example I can usually access JS bugs, but by default I can’t always access even those DOM bugs that are relevant to me.) This is of course totally #securitygroupproblems. :-) But the point remains that security often mucks things up and hinders the flow of useful information, and nobody likes it.
Yes, Boris already explained why some security bugs stay open way too long. There are some exceptions (and also some bugs that were simply forgotten), but the main issue is Gecko-based products that are updated much less frequently than Firefox. This is a very suboptimal situation, and I hope that Mozilla will eventually address it at the source (meaning more timely updates for those products).