Teaching People to Fish

[denise]

The transcript and video of damned_colonial's OSCON keynote about women in open source are now available! Well worth a read/listen; she provides some really thought-provoking material.

After she presented the lecture, someone sent her an email saying (in summary) that encouraging beginners is all well and good for small projects, but for larger or more complex projects, it's impractical: beginners will make mistakes, the project maintainers will reject their patches, the newcomer will get frustrated and leave, and nothing will get accomplished. There was also a suggestion that in order for open source software to achieve wide adoption, it has to be high quality, with an undercurrent (if not outright statement) that a). newcomers' contributions won't be quality contributions and b). it's all well and good for small projects to encourage newcomers, but "real", serious open source projects should be staffed entirely by experienced developers. His point seemed to imply that the world is divided into programmers and nonprogrammers, and if you aren't born A Programmer, you have no hope of ever achieving that status. I'm not certain that was what he meant to imply, but it's what was there between the lines.

Skud copied me (and Mark) in on her response to him, in the hopes that one or both of us would have something to say, and indeed I did, because that attitude is exactly what drives people away from contributing to projects. Skud suggested I take my response and turn it into a blog post nearly verbatim, but after some consideration, I decided I'd rather rewrite it to be a little less off-the-cuff and a little more well-structured.

It's slightly facile to say that everyone contributing to Open Source was a beginner once, because some people are better at self-education than others, some people had the benefit of a university computer science degree, some people had the benefit of hands-on coaching from a parent or mentor in their teenage years, etc. But 'experience' isn't a binary, all-or-nothing quality.

If a project keeps turning away people who don't have enough experience for them, the end result will not be that project having a large pool of only experienced developers, while interested people who don't quite match the appropriate experience level go off, educate themselves, and return when they're qualified enough to match the project's requirements. The end result will be that project having a small pool of developers who are 'experienced' enough to meet the project's requirements, while interested people who don't match that level of experience go away and never come back. And because nobody stays with a project forever -- burnout, life commitments, and the lure of other shiny projects ensure that there will be perpetual turnover -- the project is likely to see its contributor pool slowly dwindling over time.

To that end, I've put together a list of steps that any project can take to lure in new contributors and ensure that beginners have a successful learning experience with them, without compromising that project's quality and professionalism:

1. Lower the barrier to entry.

Make a list of the steps a newcomer to your project would need to go through in order to start hacking on your code. Downloading the source, obviously, but what else? Do you require libraries that the average user isn't likely to have installed? Do you assume the programmer will have, or require the programmer to have, certain tools or utilities on their system? Is your code architecture weird or nonstandard, so people will have trouble finding things in the code? Is your install process well-documented?

Barriers to installation aren't the only barriers to analyze, either. Take a look at the process by which people can contribute patches to you. Is it well-documented? How long would it take for a newcomer to figure out where to send a patch? Do you make it clear what form you require patches to be in? Are your coding style guidelines clearly available somewhere? Is your code review process transparent?

(The classic measure of this is the "typo test" -- if someone notices a typo in your project and wants to submit a patch to correct it, how long would it take them to obtain your code, make the fix, generate a patch, and submit the patch?)

Then, once you've done this analysis, figure out how you can streamline the process. For example, on Dreamwidth, we knew that the process of installing the code was tedious, confusing, and poorly documented. The technical requirements for a development environment are also fairly strict. We're doing work to improve the documentation and the install process, but even if it were much simpler, the technical requirements alone would be a barrier.

Our solution was to add another VPS server solely dedicated to providing development sandboxes for anyone who wants to contribute to the project. We call them Dreamhacks, and anyone who's interested in coding with us gets their own install (with all the libraries, plus the code already checked out for them, plus a few useful scripts), their own database to run against, and a little bit of perlbal magic to make their assigned port answer to requests for username.hack.dreamwidth.net. A little scripting on our part, and setting up a new instance takes less than five minutes for us, while it might take a newcomer several days of fighting with it.

2. Keep a public task list -- no matter how small the tasks. (Especially the small tasks.)

Every project collects them like they're candy: the little flaws or bugs that have been there since the dawn of time. You know, the ones that will take an experienced developer five minutes to fix. Typos. Missing fields. Missing options. Legacy code that doesn't actually do anything, but is still there because nobody ever bothered to take it out. Cases where you switched over to a new way of doing things halfway through the project, so half your code uses the old way and half of it uses the new way.

In most projects, those little flaws and bugs are on everyone's "when I get a chance" list: "it'll only take me five minutes; I'll do it when I get a chance". Except there are always bigger and shinier things to do, or bugs that are more crucial to fix, or new features that will knock everyone's socks off. (There's also, sometimes, a pervasive atmosphere of "those little things are beneath me".)

If you keep a public "when I get a chance" list, categorized by amount of effort required to complete the task, a new developer won't have to go looking. Some people come to a project with a particular itch that they want to scratch, and they won't have any problems finding something they want to work on. Other people come in to the project knowing that they want to help out, but not really having any particular direction they want to go in. Being able to pick a task off the list means that those people won't have to go searching for something they want to do. The idea is to present them with a wide variety of options, in the hopes that one will catch their eye and appeal to them. This increases the chance that a casual, drive-by viewer will get lured in and decide to stick around.

On Dreamwidth, we handle this by logging everything -- no matter how big or small, no matter if it's a bug or a planned feature -- into our Bugzilla. Through judicious use of keywords, we separate them into "minor effort", "medium effort", and "major effort". People who are looking for a quick hit can choose something off the "effort-minor" list, while people looking for something to dig their teeth into can choose from the "effort-major" list.

Categorizing every task by the amount of effort and experience it will take to implement it gives newcomers a solid idea of what might be suitable for them to tackle. And the benefit of having everything logged and recorded means that you're not going to forget it when it pops off your mental stack. I've seen multiple projects use their bug trackers either solely for major bugs, or as a release tracker -- only entering items that are planned for the next release and saving the others for "later", in a list in someone's email or on a wiki or whatever. By keeping the list public and entering everything on it, no matter how minor, you increase the chances that a random passerby will see an item, think "that's been annoying me for ages, too!" and provide you a fix.

Think of your bug list/issue tracking list not as a list of flaws that you have to correct (which will lead to you getting annoyed or angry every time another one gets added) but as a list of opportunities to improve. That way, every item that gets added to the list is not only proof that people are using your product and want it to be more awesome, but another item added to the list of things that can be used to lure in new people.

3. Have clear coding guidelines.

Many projects have a series of best-practice guidelines -- sometimes written, sometimes unwritten -- for coding style or architecture style. If the best-practice guidelines are written, they're often written in a shallow or confusing fashion, or phrased/presented in such a way that it's assumed everyone reading the guidelines will be able to extrapolate the guidelines that are only written between the lines. This leads to a newcomer submitting a patch that doesn't meet the project's guidelines, then having the patch rejected because it's not good enough.

It's also common for code reviewers to treat "it would be nice, but not necessary" guidelines as Holy Writ and reject patches that meet all of the required guidelines, but violate one or more of the undocumented, optional guidelines or the community standards that have evolved over the years. This is frustrating as hell, and often leads to people having a patch rejected, and rather than rewriting it just throwing up their hands and saying "screw you guys, I'm going home".

Fixing this problem involves that thing that so many developers hate: writing documentation. First, look at your coding guidelines. (You do have written code guidelines, right? If you don't, go write some. Make them detailed. Then go back and make them more detailed.) Is there anything in there that could be misinterpreted? Are you using language that assumes your reader already knows everything about your particular programming language? Are there any unwritten rules that will cause a reviewer to reject a patch that aren't documented in your guidelines?

Then, ask someone who's new to programming, or new to programming in your project's language, to read over the coding guidelines, noting down sections that are unclear or that assume a familiarity that not everyone will have. You don't have to turn your coding guidelines into a tutorial on programming in $language (and in fact you shouldn't, as that will make more experienced people take one look at them, assume they know everything that's in there, and stop reading), but if there's heavy jargon, or the guidelines assume a computer-science education, it would be a good idea to footnote a "further reading" list for any particularly jargony bits.

You should also include clear, concise examples of each point you include. Instead of saying:

Use postfix conditionals whenever possible. Postfix conditionals should not use parentheses.

Say this instead:

Use postfix conditionals whenever possible. Instead of:

if ( $u->is_person ) {
    return "User is a personal account";
}

Do this:

return "User is a personal account" if $u->is_person;

Don't use parentheses in postfix conditionals. Instead of:

return "User is a personal account" if ( $u->is_person );

Do this:

return "User is a personal account" if $u->is_person;

Ideally, your coding guidelines document should be the only document anyone new to your project should have to read in order to write a patch that will pass your code review on the first try (assuming that there are no technical errors with the patch, of course).

4. Lower your pedantry level.

This is perhaps the hardest thing for a lot of developers to accept, because to them, it sounds like "lower your standards". There are often a number of reasons for reviewers to reject a patch, though:

1. This patch is wrong/doesn't work/doesn't compile.
   
2. This patch introduces a security flaw.
   
3. This patch fixes part of the issue, but not all of it.
   
4. This patch doesn't meet the functional style guidelines (weird logic, bad architecture, etc)
   
5. This patch is perfectly okay as is, but the reviewer sees a more elegant way to implement it.
   
6. This patch doesn't meet the cosmetic style guidelines (missing spaces, wrong formatting, etc).
   
7. This patch fixes the issue as described, but the reviewer would like to expand the scope of the issue (e.g. changing "fix a missing </table> in the HTML this produces" to "rewrite the page so it uses <div> instead of <table>")

In order to attract, train, and retain newer or less-experienced developers, it's important for project leaders to recognize that there's a difference between all of those reasons -- and that some of those reasons are valid reasons to reject a patch, and some of them are pedantry that will drive a newcomer away.

Obviously, if a patch doesn't work, doesn't compile, or introduces a security flaw, that's a perfectly justified reason to reject the patch. Whether or not you reject a patch for fixing part of the issue but not all of it depends on your particular project and on the nature of the issue. For projects that have a very rapid release cycle, or for issues that are particularly severe, it can be better to take a patch that fixes 80% of the issue, assuming that you'll get the next 20% in a later release: it can be better to fix the issue for a subset of your users now instead of waiting for someone else to come along and fix the issue for everyone later, particularly if the remaining 20% requires a specialized skillset that only a few people have.

Rejecting a patch because it doesn't conform to the functional style guidelines is also something that depends on your particular project. If your project combines a number of different functional styles, due to issues with legacy code or different maintainers over the years, it can be better to commit the patch and open a new bug to improve it later. (For instance, if you have a functional rule that anything that can be abstracted into a function that can be called anywhere should be, but the existing code doesn't always do that, it might be better to commit the patch now and open up a new bug to refactor it later.) If the particular functional rule is for performance reasons, or the entire codebase follows that rule, rejecting the patch for that reason is justified.

The other issues, however, are things you should think twice -- or three times -- before rejecting a patch for, especially if the patch comes from a new contributor. If the fix is perfectly valid, but the reviewer sees a more elegant way to implement it, it's often better to commit the patch as-is and then open a new bug to patch the patch later (especially, again, in projects with a rapid release cycle), under the theory that it's better to have a functional fix now than an elegant fix later. If the patch has cosmetic style flaws, it's better for the committer to add the missing spaces or remove the extraneous parentheses or convert tabs to spaces (or vice versa) at the time of commit.

And if the patch fully fixes the issue as described, but the reviewer rejects it because he or she wants to expand the scope of the issue, that is the fastest way to drive people away from contributing to your project. Your community developers will learn that nothing they submit will ever be good enough and stop even trying.

5. Never reject a patch without explaining.

If you reject a patch, explain why you're rejecting it. Tailor your explanation for the contributor's experience level. If you know that the person who submitted it has been programming for twenty years, it's okay to bounce a patch with just a quick note about what's wrong, but if someone's newer, getting back that one line is more likely to leave them sitting there staring at your response and wondering what the heck you mean by it.

If you accept the patch, but there's a better way to do it, explain it. If you make corrections to the style when you commit it, explain the corrections that you made, and why. The idea is to train people with specific, real-world examples, offering them feedback on things that they've done so that they'll learn from their mistakes and not make the same mistake again. (And if they are making the same mistakes frequently, ask yourself why. Are your explanations not clear enough? Are your coding guidelines not explicit enough? What can you improve to make sure that it doesn't keep happening?)

In either case, provide your feedback in as constructive a method as possible:

• Explain any problems with the patch in a clear, explicit fashion.
  
• Provide concrete suggestions about how to improve the code.
  
• Direct the newcomer to resources they can use to learn about the problems in more depth.
  
• Give examples of what would be considered good coding practice in that particular situation.

And, as a side note to this item: Review and commit patches quickly. Ideally within a day or two, especially for small fixes. If someone's a frequent contributor and their patches sit for a while, they'll likely be off doing something else while they're waiting, but if someone's new to the open source world and their first patch sits for a while, they'll be sitting there wondering what they did wrong.

Think back to how excited you felt the first time you saw your name in the changelog of a project you liked and respected. If you make someone's first experience with you be positive, quick, and seamless, they're far more likely to stick around.

6. Get buy-in from your existing contributors.

This methodology is, without a doubt, a burden on the existing contributors, especially on bootstrap:

• It requires them to spend less time coding and more time coaching.
  
• It requires them to take the time to log everything into an issue tracking database, with enough detail that a random passerby will be able to understand the issue.
  
• It requires a heavy commitment to code review and rapid commit cycles.

The benefit to this methodology, though, is that all of that effort you put into coaching and training will pay off. The people you mentor in this fashion will not only gain technical experience that can be applied to your project, they're likely to be tremendously loyal to your project, thanks to all of the effort you put into teaching them. (Humans are social animals: if you welcome people as part of the pack or the tribe, and demonstrate to them that they are valued, they will continue to participate with you.)

This is not to say that every single one of your developers needs to take on the task of mentoring newcomers. (In fact, it's often a good idea to keep certain people far, far away from those who are just starting out.) A good percentage of your team should commit to mentoring, though, and you should provide fora for newcomers to help each other, and experienced developers to help newcomers, in as many methods as possible.

Designate at least one, and possibly more, people on your team (who have good social skills) to be the "newcomer liaison". This should be a person who believes that there's no such thing as a stupid question and is willing to explain the same basic steps over and over again, in whatever fashion the newcomer learns best. (Some people learn just fine from a link or a book reference or a basic pointer about what to research; some people need painstaking hand-holding.) "JFGI" should never, ever be the answer to a question about how to do something. At most, the response should be "Here, let me link you to this page where someone else explained this, and if you still have questions after reading that, I can answer them for you."

Remember that not everyone you train in this fashion will stick around after doing one or two small things. Maybe you're not a good fit for them; maybe their life gets insanely complicated; maybe they've run out of things they feel like hacking on. Don't expect to have a 100% retention rate. If your retention rate stays at 50% or better, though, you're still ahead of the game.

*

On the surface, this whole methodology looks like a tremendous amount of effort for very little payoff. After all, if you have to keep holding newcomers' hands, when will you have time to do any hacking of your own? And the stuff that the newcomers are working on, well, you could do that in five minutes, and it takes someone who's new to programming five days.

But as you deploy this method, your newcomers gain experience, and start coaching other people. Because they were welcomed into your project with these methods in place, it will seem natural and normal to them to do the same for others, and thus your pool of available mentors will grow over time. By fostering a communal, convivial atmosphere, where everyone helps everyone else, your project will be a pleasant place for contributers to be, where people want to spend time helping to achieve your goals. And sure, you could have done those little tasks in five minutes, but when was the last time you really spent any amount of time doing those little, five-minute tasks, instead of viewing them as annoyances? By using your little tasks to train your newcomers, it relieves you of the burden of having to do those little tasks yourself.

In Skud's keynote at OSCON, she said something that really stuck with me: You can teach programming. You can't teach passion. And that, really, is the core of the Dreamwidth development methodology: we take people who are passionate about the project, people who want to contribute, and teach them the skills. Because they're so passionate about the project, they put incredible effort into learning. Because we teach them the skills they want to learn, they turn around and pay it forward by teaching others.

We average 30-50 commits a week. We've had contributions from some 40 unique contributors since the project began. Of those, I'd estimate that anywhere between 50%-65% have either never programmed in Perl before, never contributed to an Open Source project before, or never programmed before, period. At the beginning, mark was our only committer; he spent several months coaching and mentoring others, using these methods, and now we have five committers and about three or four more people doing regular code reviews.

I'm really happy with how well our way is working for us. I encourage any project that's looking for more people to try these methods, too, because experienced programmers are made, not born. By turning away people who have the passion, but don't have the experience, projects are narrowing their potential range of contributors down to a very small talent pool -- the group of people who already have the narrow skillset the project is looking to recruit. Those skillsets might be relatively common in the open-source world, but there are thousands of projects competing for that talent pool.

A project looking to recruit new participants could do far worse than to train their own.

Comments

[kerravonsen]

Excellent post.

*bookmarks*

[samvara]

Love this. You realise I'm now going to be doing a run-through of the OTW practices to see what else we can improve :)

*g* I don't think of the Archive of Our Own as a small project and some of our most passionate and valuable contributors are newbie coders. We are not experiencing this "beginners will make mistakes, the project maintainers will reject their patches, the newcomer will get frustrated and leave, and nothing will get accomplished" phenomenon of which you speak.

What we *are* experiencing is excellent communication, group mentoring, a 'run and find out' approach that Riki Tiki Tavi would envy, and strong individual investment in the project. We still get frustrated, usually because we're all volunteers and don't have enough free hours to devote, or because we want to be able to achieve something right now and don't want to wait.

I am frequently inspired and humbled by the humour, creativity and goodwill that the committee I work with radiates - and the scope of the work we undertake + achieve. It's totally working for us.

Re: *bookmarks*

[azurelunatic]

Interestingly, the OTW was the other example that Skud held up for "Beginners (and women) in Open Source: You're Doing It Right!".

[tesserae]

I know less than nothing about open source coding but I will tell you that this is entirely true for non-profit organizations. Keeping management tasks in the hands of the people "who know what's going on" is almost a guarantee of organizational death, or at least a long period in the nonprofit ICU after their reign ends. You need to bring in and train the next generation of leaders, plus you are going to get creative solutions from people who don't already know what can't be done. Which I suspect may apply in your field as well!

[kate]

ILU.

And I wish I could get across to the people in my office (actually in any working environment anywhere) that this is true. We've hired two new people to take over some of our burden, which would be great except that the other people in the office clutch their work to their bosoms and say 'but what if they don't do it right?'

On the other hand, I plan on mentoring my specialist, giving her whatever work she's interested in doing, and watching her grow (and probably move on because there's no real room for advancement in my office). Fostering people always creates better results. Why people don't get this, I will never understand.

[hypatia]

you might want to share this link around: http://webchick.net/embrace-the-chaos

it's focused on open source projects, but the same applies in closed source dev for sure.

[zdashamber]

It has always seemed like it would be great to know how to program, but as a kid the "learn Basic" articles in magazines didn't work, and there were never programming classes, and then in college the programming classes were for EE/CS majors who already knew stuff. HTML, I learned from one intro and then the web, but for actual programming languages I couldn't figure out what was up... There are a ton, and apparently many are complex and terrible, and some are irrelevant to modern times?

It sounds like from this that once I ground through a "learn Perl" book I might be able to figure out what was going on by helping with DW?

It is somewhat encouraging-sounding.

[zooey_glass]

I don't know how much prior knowledge Denise and her gang require, but you can join the OTW coding team without swotting up first! I started out knowing enough html to code an LJ entry, and I have contributed actual code to the Archive of Our Own now :D (Or you can do both! Not trying to steal people from DW, because we LOVE DW, but we always need more coders in the OTW too, and we like beginners!)

[thistleingrey]

Thank you for writing this up and posting it!

[zooey_glass]

I love this! I've been so excited by what you're doing with Dreamwidth, and as a member of the OTW coder trainers, I'm pleased to think we tick most of your boxes (although now I feel the need to scrutinise us for any failings). Our two projects make me so happy - every time I see a positive reference to OTW or DW I feel all warm and happy inside :D

[turlough]

I don't have the spoons to even think of trying to do any programming but if I ever did this is the kind of project I would love to work on.

I also want to take the opportunity to THANK YOU for Bugzilla and the way you use it. It's simply invaluable to me as an interested bystander who wants to keep informed on what's going on.

[reddragdiva]

The larger Wikimedia projects are having similar problems keeping (as in, not immediately repelling) n00bs. I've forwarded this post to foundation-l for discussion.

The barrier to entry isn't language, it's how n00b-repellent the culture can get. And it's not just English Wikipedia - it's quite a few of the projects. And every n00b-bump has a good historical reason, but that doesn't mean it isn't a problem.

[ninetydegrees]

Is it ok if I point out a few things regarding the way DW doesn't make it easy for newbies? Don't get me wrong, I think DW's doing a good job and I don't want to imply it doesn't.

[jeshyr]

I'm not Denise, but I'd love to hear about the problems and I'm pretty sure others would too.

We know that the IRC-centric-ness is a speedbump for many and that's being worked on but I'm sure there are others. I am not a developer here but flattening barriers is my job (I do accessibility stuff) - so please, tell on.

[ysobel]

encouraging beginners is all well and good for small projects, but for larger or more complex projects, it's impractical

...because DW is not at all complex, nooo.

(yeah, I know, it's hardly the biggest thing out there, but it's not Hello-world-in-BASIC either.)

I have said this before, and will again, because it's true: I *love* DW, and the openness of the dev community. And as someone who's gone from ... I think my first non-english-stripping patch was a typo fix (</b> to <a>) ... to overhauling an entire freaking page (and sure it's not as pretty or as elegant or as concise or as shiny as it could be, but it's functional and it's making the page better and more usable) ... um, I had a point, but lost it in there.

Still, the world-view this guy has -- that beginners will make mistakes, so the patches will be rejected, so they will get frustrated and leave -- is such a self-fulfilling cycle. (Preaching to the choir, I know, but.) Because yes, beginners make mistakes, but what happens next depends on how the established dev community reacts: "Ew, that's horrible, you shouldn't be doing coding, go back to the kitchen" will drive people away, "I suppose it does what it's supposed to but it doesn't conform to our arcane unwritten standards that we won't tell you" will drive people away, but "Hey, if you could fix X and Y and Z that would be great" / "Committing the patch, but keep in mind P and Q for the future" will get people to try again, until they are part of the established dev community. Which will then grow instead of shrinking.

Or, to sum up: Rah, ILU and Mark and DW ♥ ♥ ♥

[csjewell]

If you keep a public "when I get a chance" list, categorized by amount of effort required to complete the task, a new developer won't have to go looking. Some people come to a project with a particular itch that they want to scratch, and they won't have any problems finding something they want to work on. Other people come in to the project knowing that they want to help out, but not really having any particular direction they want to go in. Being able to pick a task off the list means that those people won't have to go searching for something they want to do. The idea is to present them with a wide variety of options, in the hopes that one will catch their eye and appeal to them. This increases the chance that a casual, drive-by viewer will get lured in and decide to stick around.

I have to admit that Adam Kennedy mentioned an itch that needed scratched on the Strawberry Perl homepage - ask him how successful he was at getting someone lured in (meaning me)!

Saw a video of the OSCON talk - thumbs up to whatever her username is here. She said things that needed said.

I'm not a contributor to DW in particular - although a pointer to the list of things that need tuits applied might help in that department for when I have some!

[minxy]

Your mentoring skills are awesome. I always learn when you write something like this, even though I'm not in programming.

Well said.

[kyrielle]

Love this. Lots. And as I often do when reading what's being done here, I wish I had the time to be more than a user of the site, to be a contributor. I don't; I don't even have the time for all the stuff in my daily life; and I won't commit to doing something and then not do it, that's not useful. But oh I wish I did, because what you are doing here is something I wish I was part of!

[pne]

This.

[juliet]

This is an excellent post :)

As one of the hadn't-done-Open-Source-before people (though not a Perl newbie), I found the DW atmosphere massively encouraging. Unfortunately I'm in the "life insanely complicated" bin for another little while, but I fully intend to come back again when everything calms down :)

Go DW!

[jld]

Depending on how large and complex the project is, the confused newbie patch submitters might even be people from that project itself, but whose experience is with a different part of it.

(I mention this because I've done that, and my patches turned out to be wrong, and in fact neither I nor the reviewer were entirely clear on what the right thing actually was, so the bugs are still open. Which is its own problem.)

That's a different case from people entirely new to the project or to programming as a whole, of course.

[sorrel]

The extent of my coding knowledge is limited to the most basic of html, but I still found this fascinating. As a methodology for business, it's something that applies to far more than open source and I wish I saw more often. Instead of competing to get people with an existing skill set and treating them like a finite resource, use a little extra time and patience to train people yourself and if done correctly, you not only expand your talent pool exponentially, you also create loyalty. The companies who tend to get first pick aren't necessarily the ones who pay the most extravagant salaries; they're often the ones who are just considered cool to work for. Someone who not only chooses passion over experience but is willing to take the time to shape that passion into a needful tool is, in my opinion, the very definition of "cool to work for."

[jeshyr]

I am with you ALL the way on this one. The other advantage of training people up from scratch, albeit a secondary one, is that you get people who do things your way. People who already know how to do something can often be resistant/defensive about changing the way they do it - the coding style, for example.

For a completely non-geeky example I find this with my personal carers (I'm disabled). I prefer to train people with no prior training because they're still enthusiastic about the job (there's a high burn-out rate) and because they're willing to learn things my way. Getting people with existing training means I first have to convince them to forget everything they thought they knew because my disability's so odd it doesn't fit the "rules" they know.

r

[yvi]

I love this post so much! I have read it twice now and had some interesting discussions with people about it.

I tend to see it very much like the attitude in my country when it comes to job-training - it's formalized for most jobs here, it takes two to three years, you get paid for it and you go to school about half of the time, work the other half. Sure, in the short-term you will cost your employer money because you can't work as efficiently as the already skilled workers and they need to train you. But after one year, you are starting to be worth your money. After two years, you can do the job well. In the end, you have someone who knows their way around and it pays off to have trained that person. And that person can train others.

The other issues, however, are things you should think twice -- or three times -- before rejecting a patch for, especially if the patch comes from a new contributor

Yes! With you, I knew I could reject the /support bug because you missed on .bml. With a new contributor, I never would have rejected it, I would have accepted it and said "hey, that works great. you just missed one instance [points], can you do that as well and submit a patch for that as well? thanks".

And no-one who doesn't want to train has to be forced to do training for this to work. There are always people around who want to help.

[lapistablet]

the end result will not be that project having a large pool of only experienced developers, while interested people who don't quite match the appropriate experience level go off, educate themselves, and return when they're qualified enough to match the project's requirements.

... I have to say, there are a few, very small, very magical and privileged contexts in which this does in fact hold true. I live in one of them; Cambridge UK, otherwise known as Silicon Fen. The guy you can't see who lives in the next bedroom over <---- there is a hardcore cryptography programmer; he got his current job on the strength of having written a crypto library for kicks. He is the kind of person who sees a project he wants to contribute to, goes away, self-educates, and then comes back and dives into the fray. He gets a lot of respect, and with good reason, because he's damn good at what he does.

He's also a member of a tiny, tiny minority of people I've met, mostly but by no means exclusively male, who actually work like this. Over the dozen years I've been in Cambridge, in and out of various parts of the high tech business world, I've learnt that the true geek is a very particular kind of person, and thinks in a very particular way, and I think the open source culture that puts such stumbling blocks in the beginner's way evolved out of the group interactions of people who, actually, really are that awesome. I quite clearly remember learning a whole new idiom and system of reference when I first started hanging around with programmers; I'm an author by trade and my first degree was in foreign languages, so you might say I'm something of a professional communicator, and it was fascinating to immerse myself in that and learn the world in a different way. In fact, I've devoted quite some time over the years to studying the language and culture that naturally springs up among the geeks who are my partners and my friends.

And at the end of twelve years doing that, what I've concluded is this: the open source community as it stands is pretty much composed entirely of maverick geniuses, who manage by hook or by crook to work together. Because they're maverick geniuses, working together for them happens in a very quirky way, and that's what's given rise to the open source culture as it stands. I think part of the "open source attitude problem" (insofar as it's a problem rather than just a deep failure to communicate) is actually a question of the classic geek assumption that "everyone else is like me". The problem is that with open source, the people making that assumption are almost all people who have the genius factor. They're the gifted few who have that instinctive flair for the task at hand; a natural love of and aptitude for hacking that makes it second nature to work both by and for yourself. Most of the rest of us aren't so lucky, even if we're bright and able enough that we're perfectly capable of competent hacking given training in the basics. I think getting that point across might do a lot to stop this debate polarising too hard...

[valentinite]

As a professional programmer (who does not volunteer for DW because I want to come home and NOT look at bug tracking and code, sorry), I have to say what planet are these people on if they think this isn't how software should be done, period. All of those things are things that I can tell you how well (or poorly) my company does, and which ones have caused the most pain for new developers.

This is one of the big things that fandom can teach the world. That people can do amazing things just for love, all they need is a friendly assist.

Good ideas, wrong target?

[sc3d.myopenid.com]

Your guidelines for mentoring are excellent. Unfortunately, there are a lot of projects that simply don't have the wherewithal to implement them. Without seeing the original email, I can only treat it as a straw man or Aunt Sally, but it does seem to miss the point, as does the rebuttal given above. The projects that can least easily offer mentoring and development to less experienced volunteers are mostly small. Some are large, but what they have in common is little spare bandwidth on the part of maintainers, and large user bases for whom the project's product is critical. They'd all love to have more participation, but getting from here to there is impossible without additional experienced developers to act as mentors. Further, these projects tend to be un-sexy or perceived as "hard", or both, and hence attract few volunteers. Finally, many maintainers are discouraged from interacting with potential volunteers by the users who seem to expect free service from whom they are often hard to distinguish. This is not a surprise, as volunteers usually start out as users, and many won't have considered volunteering to start with; they often need educating themselves that when a maintainer responds to a bug report or feature request with "agreed; can you write a patch for that?" they are not being either facetious or unreasonable.

A couple of examples of the sort of project I'm talking about: at the "large" end is the Linux kernel, which attracts bug reports like flies (mostly from the clueless, many from the rude; and it's worth remarking that being clueless is a form of rudeness, albeit mild: while ignorance is blameless, the expectation of being able to use a product for free without learning about it isn't), and whose core developers long ago learned that in order to avoid spending all their time replying to emails they had to be ruthless. In fact, ruthlessness is a lesson that all productive technologists learn (and indeed, one could generalise far beyond IT, but that would be too much scope creep for this comment). The trick as far as educating the next generation goes is learning how to be productively ruthless: that is, balancing encouragement with the demand that volunteers actually help themselves to learn. (Again, there are interesting wider parallels with how we educate people into both their social responsibilities, and the rewards that come with exercising them.)

But the Linux kernel probably won't die, despite its forbidding aspect; there are simply too many developers who aren't put off, even if a lot leave burnt out and most only ever write device drivers. It's the projects that still of great importance but much smaller maintainer pools, typically numbering from 1-6, that could stand most to gain from more volunteer work, but who find it hardest to do so. I'm thinking in particular of GNU projects like autotools (autoconf, automake, libtool...), grep (absolutely crucial to most Linux systems, but with only one active maintainer), file (just about creeping along with one "full-time" maintainer) and SoX (the sound toolkit, currently with about one and a half maintainers). These are all programs on which much other free and open source software relies, and yet it's a miracle that they continue to be developed at all. As a contributor to all of them, I groan inwardly every time I see a clueless post to a mailing list that results in yet another polite reply from a thinly-spread maintainer, typically a wrong-headed invalid bug report whose author is typically never heard from again (whether they go away satisfied or frustrated). Hardly any offer contributions, and those often consist of patches which are not updated to meet criticisms; almost none offer sustained help.

As a contributor whose contributions have ranged from spending months or even years working on a project to submitting a patch, having it rejected, and realising that it probably wasn't worth submitting in the first place (not to mention endless bug reports and dozens of feature requests), I submit that the greatest frustration is the maintainer who simply ignores one's contributions. On most occasions on which I have been treated with what seems like brusqueness, or even rudeness, I've ended up concluding that, if I'm not actually wrong, at least the maintainer has a point when they criticise my wrong-headed feature request, unclear or downright incorrect bug report, and I've come to the conclusion that while it would be lovely if all maintainers could learn to write brief and polite notes in such circumstances, brief and brusque is better than long and polite, because it sucks less time and motivation from the real task in hand, which is to develop and maintain useful software.

In short, we all have responsibilities, users and volunteers no less than authors and maintainers. In particular, in a world in which authors are not paid, volunteers have to recognise that their offerings impose costs as well as offering benefits. Authors and maintainers too need to find ways to encourage new-comers, not seeing their education as a burden to be borne elsewhere, and taking a long-term view that sees both the practical necessity and social duty to educate the next generation; however, some are in particularly responsible positions in the free software ecosystem, and their primary responsibility is likely to be to those who depend on them.

My own position is fortunate: I have quite a bit of time to devote to free software, but I'm not responsible for any of these "overloaded" projects. The main project I maintain, GNU Zile (a tiny Emacs clone) has a small and relatively well-educated user base, who contact me rarely and on the whole cluefully, and with whom I can therefore take pains. Nonetheless, despite the occasional expression of interest, no-one has offered me more than the occasional patch in about 8 years, and I'm the only "full-time" maintainer other than Sandro Sigala, Zile's original author, from whom I took it on; in 12 years, the project has had only one other major contributor (i.e. someone whose name made it into the copyright notice). In short, Zile's position is fairly weak, but it doesn't matter. (In the worst case, there will still be Emacs, after all!)

Thanks for writing this article; I have recently been considering technical fixes to the problems of out-but-still-crucial projects (essentially, how to boost programmer productivity), and this is a timely reminder not to forget about straightforwardly recruiting new members to teams that certainly can afford to grow!

Re: Good ideas, wrong target?

[denise]

Oh, I absolutely agree that there are projects that aren't suitable for beginner work. Something like the Linux kernel, or MySQL, or any of the other complicated, highly-technical infrastructure projects that exist out there, probably won't have an abundance of the suitable-for-beginners tasks that I'm talking about -- that particular piece of advice is more geared towards projects that exist as helper applications for other hobbies or things that people get passionate about, not projects that are technology infrastructure projects that require people to be passionate about the technology itself. A project such as -- oh, say, book cataloging software -- can recruit brand-new developers from the people who use the project, but who've never programmed before, and use the tiny cleanup tasks to lure them into "leveling up" and taking on harder, more complex issues.

The tiny tasks I'm talking about are things like dialog changes, altering preferences, changing the way the page renders (for browser-based apps) -- stuff that people can do quickly and easily as a beginner, stuff that has a really high 'wow' factor. If you get a patch accepted into a project you use regularly, one that scratches your particular itch (however modest), every time after that you use the product or the app, you see your change and think, "I did that!"

From there, a little effort on the part of the maintainer can nurture that accomplishment glow and make the beginner really think that their contribution is valued and appreciated, no matter how little it is (on DW, we get one-line -- hell, one-character -- patches with about as much frequency as we get the big sweeping patches). Once you get someone used to the feeling of accomplishment, once you provide them with an atmosphere of unalloyed positive safety to make mistakes and ask questions and try things out one by one until something works, they gain the confidence necessary to try the harder stuff, because they know that they won't be judged or shamed if they get something wrong.

It's the shift in conceptualization -- hell, the shift in cultural/project worldview -- necessary to turn flawed or incomplete patches from a detriment to an asset (because they indicate interest in the project and someone who's passionate enough about the end result to spend their time/energy/effort learning things in order to contribute). Using this method, the project maintainer absolutely has to ride herd on everyone involved, because it's fairly common in the OSS world (from what I've seen in my forays, at least) for active contributors to view mistakes with scorn. Project contributors are expected to do things right the first time, because otherwise it just wastes everyone's time.

And it's true that this method of doing things does place an incredible burden on the project maintainers, on the active contributors, and on the people doing code review and commit to mentor and encourage others, or at the very least to tame their worst expressions of annoyance, and that is a time-suck. It's also true that the rate of return on this method is nowhere near 100%, and there will always be instances where your thinly-spread maintainer is going to spend an incredible amount of time and effort on someone only for that someone to disappear without any tangible gain being returned to the project.

This method is more of a "bread cast upon the waters" method, where you have to look, not at the rate of abandonment, but at both the rate of return and the rate of exponential growth: if 50% of the people you mentor at bootstrap turn around and get lured into the project, and those people then turn around and mentor others with another 50% rate of success, and so on and so forth, the maintainer's initial outlay of time and effort will start building up concentric circles of mentorship, where people who have been given a hand up will then turn around and do the same for others, etc. After that initial bootstrap phase, it takes the burden of handling the basics off the maintainer, freeing him/her up for the more complicated technical bits and allowing the basic education to stay in the hands of those who have just recently gone past it themselves. (The other benefit of that is that if it's someone who's just learned teaching someone who's learning, the teacher knows and remembers all of the pitfalls, because they went through it themselves so very recently -- someone with 20 years of experience isn't going to remember all of the things that confused them in their first six months!)

I disagree that being clueless is inherently a form of rudeness -- I disagree vehemently, in fact; rudeness is something culturally defined and not at all concurrent from culture to culture. Cluelessness is a form of rudeness in the most-prevalent open source culture, true -- but I submit to you for consideration the proposition that that very socially-construed definition of rudeness is part of what makes so many OSS projects unwelcoming. There are certainly ways to gently deflect someone with more enthusiasm than experience to the existing newbie documentation for self-education.

It really doesn't take that much more time to type out something like "That's a great idea, and not something that I'd ever thought of before. I'm overbooked right now in terms of tasks that I personally need to get done for the next release, so I won't be able to get to that myself, but if you want to put together a patch, I'd eagerly accept it. (I don't know your experience, so if you need some suggestions on how to get started, check out $newbie_documentation_url for some pointers.)" than it does to type "agreed; can you write a patch for that?" (After a while, it just sort of rolls off your fingers without you having to think about it.)

The former acts to a). validate the person's suggestion, which to them is of course going to be the most imporant thing in the universe; b). provide the person information about why you aren't immediately leaping to implement it, because of course they feel you should see the light of their genius and drop everything to go do it; c). let them know that you aren't just blowing them off, that you do want to work with that person to improve the product; d). provide pointers to implementation information that makes no assumption about the experience level, or lack thereof, of the questioner. (Thus neatly avoiding both the "I've been doing this for twenty years, how dare you assume I don't know what I'm doing" reaction and the "I've never done this before, how dare you assume everyone is as experienced as you are?" reaction.) The latter operates from a number of different assumptions that might have been true in the infancy of the OSS culture (that everyone reading has a certain skill/ability/interest level) but aren't as true now.

Existing OSS culture, in my experience, is predicated around newcomers already understanding existing OSS culture, and reacts swiftly and violently to those who don't. It's also -- and I think this is a little harder to spot, especially for the people who suffer from it -- a culture where the focus is placed on the individual, not on the project: where most projects are centered around the maintainer, not the end result. These two things combine to create a culture where innocent mistakes made out of desire to improve the project are shot down with a vehemence totally out of proportion to the weight of the error, and leave people smarting (and way more likely to say "fuck you, I don't need this kind of crap in my spare time").

You say "the real task in hand [...] is to develop and maintain useful software", which is absolutely true, but which also carries an unexamined assumption: who defines 'useful'? Obviously, the answer to that is going to be different for each project, depending on what the projected use case is, but I've seen a lot of projects where the implicit answer to that is "the maintainer", not "the people who are using it". There's always going to be an element of that in any project, since the maintainer is God for all intents and purposes, with the final say about what goes in and what direction the project is going to be taking, but I think there are ways for a maintainer to keep that leadership role and shepherd the project without conforming to a very narrow personal vision and interpretation of what the software should be. I know that with DW, there are tons of people working on things that Mark and I never imagined that people would want, but which they get incredibly passionate about -- so we let them go for it, and take the end results gladly! That makes the end-result software more broadly usable and useful.

(This is, of course, only applicable to really broad applications. For smaller projects, a narrower focus is far more appropriate.)

I dunno, I'm just sort of noodling around these really big and amorphous concepts, trying to analyze why we've been so successful in recruiting such a broad base of contributors and how those lessons can be abstracted to fit the OSS community as a whole. I've been passionate about OSS use and OSS advocacy for over a decade, but I've never found a project where I felt that my contributions would be welcomed. I've been trying lately to put my finger on why, and why DW would feel different even if I weren't half-owner, so as to be able to articulate it for others. So much of our work with DW has been to actively oppose the unwelcoming cultural practices we've both seen over the years (such as the ones that killed LiveJournal's open source contributor culture), while retaining and strengthening the cultural practices that let LiveJournal's other volunteer cultures thrive and develop.

Thanks for the thoughts, and for letting me poke at some more of my ideas in response to yours. :)

[brainwane]

I'm the fairly new volunteer development coordinator for MediaWiki, and I am basically trying to follow this exact playbook. I know it is and will be rewarding but it's hard to bootstrap.