This post deals with the results and conclusion of the user trial discussed earlier. The findings of the study can be divided into two main areas: the results of the drawing exercise and the success of developing the drawings into a 3D CAD model, and the results of the two CAD modelling exercises. It’s important to stress that in both cases the objective was not to judge or analyse the quality of the design, but rather to gain subjective feedback from participants about which activities they enjoyed or disliked, and which approach resulted in the product they were most happy with.

Drawing exercise and 3D CAD model development

The table below shows an analysis of the drawings returned by the participants:

All drawings were analysed and coded according to the following criteria:

I(i). Number of sheets – How many sheets of paper were used in the exercise?
I(ii). Number of drawings – The total number of sketches made during the exercise, including sketches of ideas which were rejected.

II(i). Understanding of Safe Model concept – Did the participant understand and follow the instructions regarding the images of the safe model?
II(ii). Inconsistency between drawings – Did sketches exhibit inconsistent or contradictory information?
II(iii). Understanding of orthographic projection.
II(iv). Use of annotation.

III(i). Evidence of design iteration – Did the participant develop and test the validity of a design through sketches?
III(ii). Number of different designs drawn.
III(iii). Final design identified – Did the participant make obvious which was the final design?

IV(i). Functionality – Did the participant design a functional element in addition to the basic functionality of the USB memory stick?

Attach (A) – A method of attaching the product
Grip (G) – A feature which allows the product to be held more easily
Retain (R) – A method of keeping the cap in place
Other (O) – Any other form of functionality

IV(ii). Functional detailing – Did the participant include functional details such as screws or split lines in the design?
IV(iii). Cosmetic detailing – Did the participant include cosmetic details such as fillets or chamfers in the design?
IV(iv). Colour and Texture – Did the participant include details whose colour or texture were specified?
IV(v). Manufacturing constraints – Did the participant consider details imposed by manufacturing such as draft angles or material wall thicknesses?

V(i). Existing Designs – Did the participants look at the envelope of existing designs before, during or after the exercise?

VI(i). Degree of interpretation – a measure of the degree to which the CAD operator had to interpret the participant’s drawings in order to build the CAD model. Measured on a scale of 0-5, where:

0 = no interpretation needed, the drawings were accurate and fully resolved;
2 = some interpretation needed, the drawings were accurate but some details were unresolved
5 = significant interpretation needed, the basic idea was communicated but details were unconsidered or unresolved

The three most important findings, highlighted in orange in the table above, relate to design iteration, functionality and the degree of interpretation required to translate participants’ sketches into 3D models. The sketches showed that only four participants drew more than one design option. Even fewer (three) engaged in any form of design iteration, i.e. a process in which a design idea was modified. The most common form in which drawings were returned was a single idea, drawn from multiple viewpoints. As such, the ability of the participants to engage in design exploration through sketching was extremely limited.

This finding is supported by existing research into the manner in which designers use the activity of drawing. Designers tend to use drawing in two ways: firstly as a means of ‘exploration and manipulation,’ and secondly as a means of communication. The first is a creative activity in which multiple sketches are used to develop a design from the first idea to the ‘best’ idea. Such sketches do not need to be accurate or even realistic provided they offer an insight into the problem or possible solution. A communicative sketch, by contrast, is a method of explaining a (partial or full) design solution.

Drawings returned by participants appear to show an inability to utilise sketching as a method of exploration. Instead most participants attempted to draw the ‘correct’ design immediately, i.e. they tried to communicate a final design without testing whether it was, in fact, the best solution. There is a ‘randomness’ within design exploration which can be attributed to a lack of inhibition among designers to the act of drawing. Designers are often taught that mistakes when drawing have value and can lead a design in new directions; in contrast a number of participants’ drawings showed evidence of the use of an eraser to remove ‘wrong’ sketches. This inhibition or discomfort with drawing was further borne out by responses from participants, nine out of ten of whom preferred the process in Part II where drawing was not involved.

Sketch returned by one trial participant showing a lack of design iteration

Although their sketches showed a lack of design exploration, the need for design iterations was implicitly recognised by all participants in their reactions to the CAD model representation of their design. Initially all participants believed the CAD model to be an accurate interpretation of their drawings. However, all participants subsequently accepted the invitation to modify the CAD model, and all believed that the design was improved by this process of modification. Participants perceived the CAD model as a ‘sketch’ or work-in-progress which required development, and recognised that design iteration was necessary to arrive at a better design.

The results clearly demonstrate the value which participants attached to their self-designed products: a clear reason for this was the ability to introduce additional functionality to the product. All ten participants added additional functional elements to the USB memory stick, for example, details which enhanced grip, or methods for ensuring the cap was not lost. One participant shaped the device such that it could act as a bottle opener, whilst another attempted to decrease the possibility of the product being knocked and broken when plugged into a computer. This value placed on functionality rather than just aesthetics, together with a preference for a design which more closely fits the consumer’s needs, also coincides with findings from mass customisation literature.

Design with a method of retaining the cap when removed

The degree of interpretation required to translate a participant’s sketch to a 3D model is also a factor which needs to be highlighted. The drawing below shows a not-atypical sketch returned by one participant.

A ‘final’ design sketch from a trial participant

The drawing shows no indication of the preferred design, nor does it show design ‘refinements’ such as fillets or curved faces. Thus as the trial went on, it increasingly became obvious that one of the tasks, when translating a sketch to a 3D model, was to ‘read between the lines’ and second guess what the participant might want, rather than attempt to faithfully reproduce what was drawn. When presented back to the participant, the model which was built from the sketch showed a hook feature, as well as small fillets and slightly curved side surfaces. However the model was built and constrained in such a way that it could be updated to ‘close’ the hook, thus making the feature a loop. This was, in fact, what the participant had intended, and after modifying the model the final design is much more rounded than the original sketch.

The 3D CAD model produced form the sketch above

The final design after reconsideration and manipulation by the participant

3D CAD Model Modification

Results in this area refer to the tasks within both Part I and Part II of the trial which involved modifying the CAD model. The results largely consist of a comparison between the design process of Part I and Part II, and which process yielded the most favoured design. Opinions of participants were recorded during and immediately after the trial, and are summarised in the table below.

Discussion of Results

The most significant finding from the trial comes from the paradox highlighted in the table above. As previously mentioned, nine out of ten participants felt uncomfortable with the drawing exercise, preferring to modify a CAD model representation of a design. However every participant believed the drawing exercise ultimately led to the best design. When questioned, the main reason given was, as might be expected, that the design more closely matched their needs and wishes than the pre-existing model. Three participants stated that they would like to imagine their design was unique, and did not feel certain a pre-existing model would not be modified in similar ways by others. One participant said he would be proud to show the product to friends and explain to them why he had designed it in the way that he had. Thus the trial shows that the best results, in the participants’ own opinions, came from the less enjoyable process.

Conclusions and Future Research

The trial clearly shows that participants placed significant value on the ability to design their own USB memory stick. Products which were self designed were valued more highly than those which were customised from pre-existing designs, despite the fact that most participants felt uncomfortable with the process of self design and preferred the process of customisation. Participants were generally unable to engage in design iteration through sketching, and used drawing as a method of recording and communicating a design rather than exploring it. However, when presented with a CAD model representation of their own design, participants recognised the value of developing that design through iteration in order to arrive at a better solution. Participants placed a high priority on the ability to incorporate additional functionality into the basic usage of the memory stick.

The trial raises a number of questions which would benefit from further research. Of most interest to me is the question of how to resolve the apparent paradox between the preferred design process and the preferred outcome. Participants unanimously favour self designed products over modified or customised pre-designed products, however a clear majority did not enjoy the drawing task required to initiate the self design process. In a setting outside of a user trial it might therefore be expected that consumers would not engage in self design at all, and thus never arrive at a point where they were able to assess the value of their self-designed product. Future research should therefore investigate ways of capturing consumers’ design intent without requiring the consumer to sketch or draw those ideas.

A further question stems from the issue of how to interpret a consumer’s design intent, particularly when that intent is not well explored or developed, or when it is physically impossible to meet all criteria. In the trial this was overcome by the use of a designer/CAD operator, who was able to use experience and intuition to ‘second guess’ what the participant wanted to achieve with their design. In a commercial setting this scenario would likely be impossible, as it would require the input of a professional designer for every consumer created design. This would suggest the need for an automated process which would replace the designer’s intuition, or more realistically, which applied certain rules to constrain and condition the consumer’s design. Such a system, whilst inevitably limiting creative freedom to some extent, would also give the consumer confidence that the self designed product would always be manufacturable. Such a system, therefore, could be considered an advanced form of customisation toolkit, one which enabled the consumer to move beyond configuration and engage in freeform design.

In general participants understood the need for design exploration and iteration, even though this was rarely displayed in their sketches. Within the CAD model manipulation tasks, some participants were helped by the use of Genoform, which was able to suggest new directions for exploration which the user was otherwise unable to see. It may be that an automated iteration would be of benefit to some users in improving their designs.

A final question arises from the value which participants placed on adding additional functionality to their design. The trial was able to recognise that a participant valued their design of, for instance, a grip detail. However it did not ask the participant to compare that grip detail with, for instance, a method of retaining the cap when removed. The results do not record whether the participant considered a cap retention method but rejected it as less important than a grip detail, or failed to consider a cap retention method at all. Thus the trial could not disclose whether participants saw value in added functionality in general, or only in the added functionality that they had designed. Such knowledge would be valuable, particularly if a future trial were based on a more complex product.