Autodesk Labs versus Beta: Experts Elite Presentation

photo courtesy of Adrianna Schneider, @LEEDing-Lady

The Expert Elite program was spearheaded by many, including Technologist, Shaan Hurley, and launched at AU during our Launch & Lunch event with our CEO Carl Bass. The Expert Elite program is designed to recognize community members who make extraordinary contributions to our online and social communities. Members of the Expert Elite group are characterized by their regular and responsive participation in Autodesk’s discussion forums and social channels and advanced knowledge of the company and our products. Our goal with our Expert Elite program is to deeply engage this important group of individuals and empower them to better assist their peers and support the Autodesk community.

On May 8 and 9, Autodesk hosted our first annual North American Autodesk Expert Elite Summit in San Francisco. We had about 40 members joining our Summit, and it was an opportunity for us to share program, product, and strategic information with all of our members which are under NDA with Autodesk. The 2-day Summit was filled with Expert Elite Program topics and product breakout sessions where Product Managers collaborated with our members on features and functionality. I was involved into 2 capacities:

• Fellow Autodesk Gallery Ambassador, Jonathan Rowe, and I gave the Expert Elite members a tour of the Autodesk Gallery.
• I gave a presentation on Autodesk Labs and Autodesk Beta program.

I oversee Autodesk Labs, so I could cover that easily. Shaan oversees the Beta program. To prepare for the Beta portion of the presentation, Shaan gave me the skinny which he has documented in his Between the Lines blog article. It was great getting to talk to users whom we regard as evangelists because of their combination of technical knowledge and active participation in our development processes. I enjoyed myself. From a few tweets I saw, I think the experts did too.

Here is what I covered. You can download my PowerPoint. Although the notes section is not word for word what I said, it does capture the essence of what Shaan sent me, and I crafted myself.

Download Labs_Beta.pptx (9399.2K).

I think the two main questions that the presentation answered for the experts were:

1. I updated my Beta profile to say I wanted to help test product X, but I did not get included in the Beta for product X. Why?

   Betas are limited by size. Some such as AutoCAD 2014 are large — about 30,000 participants. Others are as small as 200. So when product teams determine whom to accept into a beta, they have to make hard choices, Since how our products get used varies from locale to locale, teams are looking for geographic diversity. With this in mind, only one-third of beta program participants are in the United States. So if you don't get included, it's nothing personal.

2. I went to all the trouble to report a bug during the Beta, but when I got the released product, it was not fixed. Why?

   Feedback during a beta can be a feature request or a defect report. Although a beta is typically months before a product ships, there is often not enough time to add new capabilities. In fact, that is why Autodesk Labs was formed - so we could get feedback early in the process where there is enough time to consider dramatic changes to approach or functionality. So most feature requests during a beta get added to a wish list. They can be considered down the road.

   For defects, the severity, frequency, and risk are evaluated.

   • The severity reflects how bad it is (e.g., typo, annoyance, feature does not work as promised, crash, loss of data).
   • The frequency reflects how likely it is to happen (e.g., everyone will encounter this every time, this only happens with a certain type of data or on a certain hardware configuration).
   • The risk associated with a defect reflects the complexity and amount of code that has to change to correct it.

   Anytime we fix a defect, we run the risk of breaking something else. So as a beta progresses, the bar gets raised higher and higher as to what gets fixed and what gets deferred. All of this is done by product core teams (i.e. project management, QA, SW DEV, marketing) who hold daily bug scrub meetings using a list of bugs in a defect tracking system. So if you report a defect, and it does not get fixed, it was probably just too risky to fix for the initial release. It's in the tracking system, so you may see it in a later service pack.

In the question and answer portion, the attendees made the following points:

• With regard to betas, why does Autodesk have a target release date? Why doesn't Autodesk just keep fixing defects until they are all fixed? How does customer participation help? See my answer.
• It would be easier to submit crash reports if users could copy/paste from the application into the crash report.
• Users would like to use video to supplement their crash reports. Many times it would be easier to explain what they were doing when they encountered a crash by simply retracing their steps while recording their screens with narration. Perhaps Project Chronicle is the answer?
• Users without access to local discussion forums would like Autodesk to persuade VARs and resellers as to the value that these forms of interaction offer.
• Autodesk University presenters lamented the fact that although the conference costs are paid by Autodesk, the hotel costs are not.
• Autodesk should consider an all-in-one system that analyzes the user's system, suggests possible solutions, and captures data that can help technical support personnel diagnose problems.

Explanations are alive in the lab.