Usability Testing: Fusion 360, Autodesk

Autodesk Consulting Project

Collaborated with Autodesk, Inc to process Autodesk Fusion 360 Usability Testing, which focuses on Competitive Analysis.

Role

 
 

This is a team of four HCI students conducted a usability evaluation on Autodesk Fusion 360. I was mainly responsible for testing design, testing, and statistical analysis

Executive Summary

Jan 2018 - May 2018

Date

Group

Work

Jia-Hua Li, Lingfei Li, Tanvi Kulkarni, Peter Yang

Participants

Client

Autodesk, Inc.

Each of the 10 participants of the study was asked to perform two tasks for SolidWorks and Autodesk Fusion 360.

 

The data collected included quantitative data, qualitative data, and performance data.

 

The results of the study endorse several recommendations to improve the usability of Autodesk Fusion 360.

Research Questions

 

Our research questions for the usability testing were:

  1. From Autodesk Fusion 360 and SolidWorks, which one has an easy interface to complete the given task?
     

  2. What are the difficulties faced by the participants while using Autodesk Fusion 360?

Methodology

 

Methodology | Participant Characteristics

10 participants, 6 graduates, and 4 undergraduates have experience using 3D modeling software for at least six months, on average 2.1 years.

 

40% of participants use 3D modeling software 3-5 times per week, almost every day.

 

20% of participants use it 1-2 times per week.

 

40% of participants use it 1-2 times per month.

 

Almost all participants use SolidWorks, CREO, Pro/ENGINEER, and AutoCAD.

Methodology | Usability Testing Lab

The study was taken in GOL-2289 and GOL-2293 in the Golisano at Rochester Institute of Technology.

Entire study happened from Tuesday,  April 10 through Wednesday, April 18.

Methodology | Process

Methodology | Task Scenarios

Solidworks

You are working on the project with your team member, Brian, who is very stressed

and can’t create the 3D model with the blueprint provided. Brian gives you the

handouts and wants you to create the 3D model with SolidWorks by looking at the

blueprint.

Fusion 360

You are working on the project with your team member, Brian, who is very stressed

and can’t create the 3D model with the blueprint provided. Brian gives you the

handouts and wants you to create the 3D model with Fusion 360 by looking at the

blueprint.

To mitigate any transfer of learning effect between SolidWorks and Autodesk Fusion 360, we counterbalanced the task order. This table shows the order we followed for each of the 10 participants.

Methodology | Test Design

Methodology | Measurements for Data Collection

We concentrated on the quantitative data, qualitative data and performance data for the usability of Autodesk Fusion 360.

The quantitative attribute:

  • Count of incorrect menu choices

 

The qualitative attributes:

  • Any positive comments that the users have while using the “think aloud” procedures

  • Any confusions or misunderstandings about Autodesk Fusion 360 while performing each task

  • Any feedback on how to improve any elements or functions of Autodesk Fusion 360

 

The user’s preference data:

  • Usefulness of terms and labeling

  • SUS questions as the measurement of the results

Methodology | Deviations from our Test Plan

After pilot testing, we found out that the original task was difficult to create. We simplified it down.
 

We did not specify durations for three tasks. We add time limitation to the success criteria of the main task (12 minutes) and warm up the task (5 minutes).

 

Results | Findings

Summaries and data analysis - Preference

System Usability Scale (SUS)

 

Mean of SUS score:

20.6 x 2.5 = 61.5 SUS score, (48.272, 74.728) 95% CI

 

Matching adjective

= OK/fair - Good   

 

Findings: Ok/Fair - Good, with a relatively weak confidence interval 

Summaries and data analysis - Preference

System Usability Scale (SUS)

 

Different Groups

Beginner vs Expert vs General

Findings: Usability for Fusion 360 are different in different groups, Expert group owns higher SUS score, the general group owns the lower score.

Summaries and data analysis - Preference

Expectation vs Experience

Findings: Fusion 360 and Solidworks both own high quality. The fusion 360 still need to focus on promotion and fixing some problems.

Summaries and data analysis - Preference

Post-task questions

Satisfaction for different Softwares after  finishing the same task

Hypothesis:

Is SolidWorks better than Fusion 360 in satisfaction?

Findings:
Satisfaction for Fusion 360 and Solidworks may be the same.

Summaries and data analysis - Preference

Open-ended questions

 

Like: Full functioning, Labels, Easy, User-friendly, Visual design, Angle Change

Dislike: Different from SW, Rotation way, Model Tree

Summaries and data analysis - Performance

Counts of finding functions in the wrong menu

Summaries and data analysis - Performance

Think Aloud

 

Positive Comments: 35.9%


Challenges / Suggestions: 51.3%


Misunderstandings: 12.8%

The participants were asked to think aloud while performing a task for Autodesk Fusion 360. They gave some positive comments.

The participants had a few challenges for the Autodesk Fusion 360 and how is Fusion 360 different from other software.

Ordered list of usability issues found

-Inspect and edit the sketch dimension were confusing

 

During the test, almost every participant didn’t find the ”sketch dimension” for the first time.  Some of them thought that the inspect/measure function was to define the sketch dimension and some participants stuck in this function.

-Features didn’t  associate with sketches in the model tree

 

Many participants tried to find the “edit feature” function in this area and couldn’t find it. This issue prevented users going further.

-Visibility of extruding cut function

 

In Fusion 360, the user can change the cut or join option in the “extrude” function. Since extrude and extrude cut are two different functions in Solidworks, some participants couldn’t find the extrude cut function.

 

We reported this as one of the issues in the heuristic evaluation:  Visibility of system status after functions selected.

-Visibility of bottom navigation bar

 

There are several ways to orbit the object in Fusion 360. One way is to move the mouse pointer to the top right area, hold the scroll wheel, and drag. Some of the participants didn’t find out another easier way. They rotated their view in this way, so it took them more time to finish the task.

Interview about Software Choice

Short interview: mainly used software

Creo or Pro/E: 3       SolidWorks: 5 Fusion 360: 1

Short interview: factors affect mainly used software choices

Course, Co-op, Prestige in Industry, Function, Easy to Use, Easy to learn, Be Familiar

Conclusion

 

Were the research questions answered?

  1. From Autodesk Fusion 360 and SolidWorks, which one has an easy interface to complete the given task?

  • Partially answered-  Easiness comes from familiarity and participants were more familiar with Solidworks

  • Around 43% of the participants never used Fusion 360 or prefer other software

  • Few participants liked how well the functions are organized under categories in Fusion 360

  • Satisfaction for both software has no difference, and most of them used Fusion 360 for the first time

 

   2. What are the difficulties faced by the participants while using Autodesk Fusion 360?

  • It takes time for participants to find specific functions for the first time

  • Different design of the model tree

  • Difficulty in finding the option for the constructed line

  • Unexpected placement of “Cut” option

  • Difficulty in finding how to change the dimension of the object

Recommendations

  • Separate student mode so that users can customize tools and place it where they expect the functions to be.

  • Change the interaction way of rotation - fully accessible to everyone.

  • Make a model tree more active and own more functions levels.

  • Consider most users’ experiences from other software, increase external consistency or provide specialized reminder according to users’ previous experience in software, especially when users use Fusion 360 in their first time.

​Promote way:

      Provide more chance to help potential users for Fusion 360, such as courses, rewards, and competitions

Future Areas of Study

Limitation & Gap:

  • Only several functions have been tested in this Usability Testing

  • Only test CAD, not test CAE and CAM (One of the advantages of Fusion 360 is that it combines the whole three design process in one software)

  • Because of the time period limit, learnability was not able to test

  • Expert, general, beginner, and user group own different SUS score

​Future areas of study:

  • Test more functions

  • Test functions used in CAE and CAM design process

  • Test learnability of Fusion 360 in a cycled period

  • Explore the reason that expert, general, and beginner user group own different SUS score

  • Provide different improvement and promotion for different user groups

Do you like this Design Work?

  • Grey Facebook Icon
  • Grey Instagram Icon
  • Grey LinkedIn Icon

©2020 BY LINGFEI LI | ALL RIGHTS RESERVED