Processing feedback

4 Replies

Some of you might remember when Andrew Starr-Bochicchio send out the Developer Advisory Team’s analysis of the feedback new contributors gave us. It was a great read, told us a lot about how new contributors get involved, but it was a also quite a bit of work.

As we conducted most of our interviews via mail, we had lots of answers in our inbox. As a team, we put them into a Google Doc and after the release cycle had passed, we had amassed 6 months worth of feedback. Sometimes just a few lines which concentrated on just one topic, but sometimes heaps of text covering each and every point of their developer experience.

As we feel this was quite a bit of work, but also worth doing, we want to continue this effort. Still we are looking for ways to improve this. If you have ever done anything along these lines, we’d love to learn from your experience. How can we optimise this process?

A few requirements we have:

  • No surveys. When we reach out to new contributors we are interested in a conversation with them and not necessarily interested in getting them to rate and judge each and every bit they might have interacted with. By reaching out to them, we already ask them to spare us some of their time, a long survey would likely give us more structured feedback, but less responses.
  • No expensive text analysis software.

If you have any ideas how we can optimise the process, please let us know.

Also if you are interested in helping out the Developer Advisory Team, please get in touch.

 

  • K.P. Smith

    This may sound archaic and possibly less automated then your requirements, but what I’ve used successfully multiple times in the past to gather and sort information from a pool of 13,000 employees is as follows:
    - Data submission should occur through an online form, not a survey, an open web form
    - Have predefined, mandatory schematics (skims) the user must select to help define their submission: (complaints, system, bug – Improvement, source, availability …. etc)
        – The skim helps sort the data but also you could use them to define suggestions categories, complaints, wishes, etc also
    - Because the submissions were sorted upon receipt, the process of manually scanning is much easier to accomplish with less resources, and manual review is always more accurate than any algorithm/scripted/automated sorting process. 
    - Additionally, you could put an automated sorting system in place after the skimmed submission but before human review, which if the automated process is executed correctly, could result in even less human intervention.

    If you put the proper support in up front, manual review is less daunting than it sounds and helps ensure every grain of detail is accurately accounted for in the results.  When soliciting open feedback anything can and will be said and no amount of coding can account for that :)

    Cheers and keep up the good working with improving the community!!

  • Pingback: Daniel Holbach: Processing feedback | txwikinger-ubuntu | Scoop.it

  • http://twitter.com/jspaleta Jef Spaleta

    i want to be clear… you are looking to distill some trend or guidance information out of the long form written responses you have already generated? Or are you looking for a new process to generate more structured feedback?

    If I understand the problem statement with a bit more clarity I can probably suggest something.

    Also I take it the existing feedback is considered confidential and is not a public dataset which different mining methodologies can be thrown at in an experimental fashion?

    -jef

    • https://launchpad.net/~andrewsomething andrewsomething

      What I’m personally interested in is a more efficient (perhaps computer assisted) way of doing qualitative analysis of responses to open-ended questions. I think the open-ended nature of this undertaking is important as it allows respondents to give answers that prioritized their own experience.

      Part of our assumption is that we’ve been doing this too long to really understand the issues facing new contributors. So a quantitative survey would very likely miss important things. We also use the initial email to create an engagement and rapport with the contributor. Part of the DAT’s mission is to make development “more social.” We generally feel that surveys can come off as too impersonal.