R4: User Research Case Studies

In both papers, authors provide some research techniques that related to the techniques that our group are using for gathering information from stakeholders.

In sustainable interaction design for mobile phones, authors used a qualitative survey and interview-based study of mobile phone ownership. It revealed the investigation that the reasons why people to replace and give up their mobile devices and also showed information that how people dealing with old phones. Authors noted that they used web survey to investigate participants and phone to interview participants from survey. Same strategy as our group, we chose survey and questionnaire method to investigate a broad and general overview of the needs and practices of current UC Irvine students who are seeking and hosting events. We will use both web-based survey and paper-based survey to identify fundamental issues within the current techniques.

We setup 6 questions that related to our project. Unlike phone interview in the paper, we will interview people in person; we more concern about the real awareness of the users. We believe the face to face interview will provide us a narrow domain that helps us to find out their perceived wants and needs.

In another paper, secondary research is also applied to assist analysis background. This method reveals historical development on guidebook in museum. Similarly, we will supplement our information from additional materials. These secondary research will be assist to compare with survey and help us to seek potential problems that how we understand users’ awareness in an appropriate way.

In the paper about revisiting the visit, they used questionnaire and interview to gather data in order to explain the pros of guidebook used in museum. In the other paper, they used similar research methods to investigate how do people deal with their used phones.

In our project, we have designed a questionnaire of 10 questions trying to figure out that does people interest in our idea, their current custom using similar products, the affordable price and their basic requirements of the product etc. From the copies we collect back now, we’ve found some weakness in our questionnaire which need to pay attention on next time when designing another one. The most obvious weakness of our questionnaire is that in some questions, we did not considerate to list all of the possible situations for people to choose from.

Then, we’ve designed one-to-one interview of eight people to understand what the prospective users need the product to be in detail. Such like the size of the product, their expectation of the product, their advice for the design team, the different roles who will use the product, their complaint about the current product, etc.

It seems that questionnaire provides an overall perspective of the project, reflecting what do people care most about the product and their common desire of the new one. It is a quick way to make sense what the basic or original requirements are in shortest time. While when it comes to the on-to-one interview, our team found out some reasons why people chose specific option during the questionnaire phrase. They also generously gave us some valuable design suggestions, as well as pointed out the unreasonable aspects in our questionnaire.

In the paper, ‘Opportunities for sustainable interaction design for mobile phones’ authors conducted a 34- question web survey
and a series of in-depth, semi-structured phone interviews. Interestingly, the phone interviewers were selected based on the web survey
responses. However for the web survey, participants were recruited via snowball sampling. The different methods of sampling for each
research method is interesting. The tactic of using the survey to select people to interview can prove to be useful
for finding very qualified people for interviews. For our project we are conducting online surveys, interviews, and observations. Structured
interviews will be conducted, as in the literature, however the people we select to interview won’t be based off of the survey. However,
Interviewees are not excluded from the survey. Our interview questions are also inspired by our survey questions, and are
more in depth and attempt to ‘probe’ the interviewee for more detailed information and data about experiences.
In the paper, ‘Revisiting the Visit: Understanding How Technology Can Shape the Museum Visit’, authors recruited interviewees
using signs located inside a visitor center. A clever tactic of using incentives were used to recruit people. Incentives like, “the use of technology”
or viewing something that was “off limits” to the general public helped bring people to sign up for the study. In contrast to our group project, we did not think of
incentives to lure people into taking our surveys/ interviews. We used advertising on the web (via social networking) to recruit people
and our own personal friends/coworkers/acquaintances to select people to interview.

In the paper “Breaking the Disposable Technology Paradigm: Opportunities for Sustainable Interaction Design for Mobile Phones”, the researchers observed the visitors activities to analyze user’s habits, and collected data by survey and interview. And In the paper about Sustainable Interaction Design for Mobile Phones, researchers used web-based survey to get to know perspective user’s preferences and used semi-structured phone interviews based on the survey response.

We also used survey to get to know more about users, but the different thing is that we used only paper-based survey. We came up with a questionnaire with 10 questions. Most of our survey questions are close-ended questions, so the only thing the respondent need to do is to choose an answer he or she prefers.
And our group also use one-to-one interview to know more about perspective user’s idea about our product. By asking them open-ended questions, we got more information about different user’s special preferences and more details of user’s requirement. And some interviewees provided us some creative ideas that could help us a lot.

In “Opportunities for Sustainable Interaction Design for Mobile Phones” researchers employed surveys and interviews in their data collection. Similarly, “Revisiting the Visit” described a method of interviewing asking participants to consider their museum experience with and without the use of a guidebook.

Our group intends to develop a tool to improve the process of lesson planning for teachers, particularly for the first time. Because of the very specific task with the limited participant pool we have outlined, we found surveys would not be ideal, and in situ observation would be nearly impossible. The literature helped to clarify real-world applications, and we simultaneously looked to the IDEO method cards to inform our approach. We supplemented semi-structured interviews with a flow analysis of the lesson planning process, and also performed role-playing exercises in which we attempted to create original lesson plans. The interview process informed our flow analysis, and roleplaying exposed us to difficulties that helped direct our design process.

The relation between the techniques used in the paper and our project can be observed by highlighting similarities coupled with differences; classified under stages of selection, execution, exploration and evaluation of the user research as follows:

Selection: The Huang & Truong study used snowball sampling for recruiting the participants through brief and simple online surveys. These surveys were used as basis for selecting candidates for the in-depth phone interviews as well. The Sotto Voce research used opportunity sampling by observing visitors at Filoli, and approaching them based on judgments of the recruiter. They also made the population at the historic house aware of the ongoing study with the help of an advertisement. We are using a combination of both the sampling approaches i.e. the friend of friend approach for the online survey, along with the opportunity sampling by targeting our surveys to specific online groups. Also, we are interviewing people who we think will be appropriate for covering the depth for our research, this selection is independent of whether or not a person currently uses a similar tool as ours. We are involving present users of the tool as well as novice users.

Execution: Unlike the Sotto Voce we are not having any written consent forms for the participants, nor are we giving any sort of training to the participants before interviewing them, participation for survey is voluntary and ad-hoc. Rather than doing the phone interviews with people selected based on survey responses like in Huang & Truong paper, we are conducting one-on-one interviews in parallel with the online survey; the answers are being logged in the form of notes and audio recordings.

Exploration & Evaluation: In both the studies the data collected is being referenced to the data that is handpicked in the secondary research. This is helpful in validating the findings in surveys and interviews, i.e. statistical results and quotes from the participants. The use of affinity diagrams also seems intriguing for clustering and idea brainstorming, but we have not currently employed them for the exploration and evaluation, rather our focus is on. We are adopting a similar approach for analyzing, validating and bench-marking our findings by doing a competitive product survey. We see our competitors as our baseline for requirements and a means by which we can cultivate a meaningful, engaging and usable design that is helpful for our stakeholders. We like the Huang & Truong team want to be creative while being relevant and grounded. Interestingly, few of the competitor products are the ones that our participants are using currently; we call them our “prime gurus”. Participant experiences with these products are proving to be very helpful in figuring out the features of foremost importance and the gaps that can be filled with our design.

In terms of older readings, we are actually implementing many of the techniques we have read about into our projects, though I would say that our method matches design ethnography and contextual inquiry more than ethnography. In our techniques (e.g., interviewing target users, role-playing…), we are trying to understand what educators do when they design new courses, how they go about designing new courses, why they use existing methods, and what they want and need when they are designing new courses. We would like to understand how the design we create can fit into their lives. Though we role-played as new educators, we are not really participating with them, as ethnographers would. We also already have the clear cut goal of using this information to create a new design or improve upon existing designs in a specific context.

In terms of this week’s readings, although we did not conduct surveys, I did incorporate some of the elements in the “Tailored Design” reading into the way I approached my interview. For instance, I emphasized my gratitude to my potential interviewees for helping me in advance, and thanked them after they agreed to participate, and after the interview. I also gave them information about what the interview was for and why I wished to interview them. I tried to make it convenient for them to participate by allowing them to communicate with me via phone, rather than in person, and being as flexible as possible with the time of the interview (my interviewees live on the East coast). The first interview I conducted (the other one should be conducted tomorrow) followed the template of an ethnographic interview as outlined in “Interviewing an Informant.” I greeted him, told him why I was interviewing him, and asked friendly questions about how he was to get him comfortable rather than launching directly into questions. Then I explained to him I was ignorant about designing courses so that he would feel like he was making a very valuable contribution. I was directive in our interaction and asked him descriptive questions (“Can you describe to me, step-by-step, the process through which you design a syllabus? How about a lesson plan?”) and structural questions (“What are the different resources that you use in creating a syllabus? How about a lesson plan?”). I would express interest in what he was saying and restate what he said to further emphasize my interest, and to make sure that I was understanding him correctly. He had a lot to say–it was a little difficult to get him to stop talking sometimes and to move onto the next question…but I feel like these methods worked overall. I wonder how the interview would have gone if I did not already know him though; it could be harder to establish rapport.

In the museum Sotto Voce paper, the authors conduct observation method to get some background information about visitors need. Then the authors combine prototype experience, questionnaires, surveys and interview method. Compared to our project, we don’t have prototype at this point. But we should definitely have user to experience our design and get feed back from them. Also anther difference is we don’t have incentive for users to participate in our study. Most research methods have been done by our social connections. Other than that, we almost used all the rest of research methods mentioned in this paper, such as observation and interview.

In Huang and Truong’s paper, they collect data by two components: a 34-question web survey (n=79), and a series of in-depth, semi-structured phone interviews with participants selected based on survey responses (n=10). Our class project is basically follows the same routine as I mentioned above. Similarly, our interview expectation would be n=4 considering time limitation. But the online survey has 60 participants response till today. Besides, we plan to do another 4 observation studies to complement the drawback of our small number of interviews. Our interview is also semi-structured, as discussed in the paper. Since we all group members will do interview separately, a semi-structured interview could maintain uniform format while leave space response for different situation.

I think our research methods choice is good because we considered both depth and breadth. We are using online web survey to collect data broadly and by conducting interview and observation we could get details of the user demands, which ensures depth of our data.

(continued)
in Huang and Truong’s paper, they conducted a survey and phone interviews. In Sotto Voce, the methodologies were more diverse from observations to prototyping to surveys and interviews. This highlights the triangulation method discussed in class since some methods have weaknesses that can be compensated by other methods. For instance, interviews may highlight pain points by users, but observations may reveal a deeper pain point that users did not realize.

Furthermore, at my group’s rate, we may build a prototype and evaluate + iterate its design and functionality

Both the Grinter and the Huang paper report usages of user research techniques that are quite similar to the techniques being applied to our project. More specifically, our group has decided to conduct surveys and interviews to further understand which user requirements are crucial in achieving our design goals.

Huang’s paper reported that surveys were conducted to facilitate the process of picking participants for their interviews. Such methods were implemented to assure that the participants involved with the research possessed a diverse range of characteristics. This is somewhat similar to the methods being used for our group’s project. However, our group has decided to conduct surveys alongside the interviews due to the time constraints given by the project; therefore, our surveys will not be used to recruit individuals for the interviews. This is better for our project anyway, since our survey sample is smaller than the sample used in Huang’s research. Having a smaller sample would limit the selectiveness on the types of people we want in our interviews. Additionally, the interview group would be less selective due to the fewer number of questions present in our survey compared to the survey in Huang’s research. Instead, people who we think would be appropriate for providing depth to our research will be interviewed, along with those who are friends of our friends.

In Grinter’s paper, interviews were conducted based on the experiences in using the guide book provided; in other words, some sort of product was presented to those being interviewed. However, our project does not provide those being interviewed/surveyed with a prototype in order to receive feedback. Our group is analyzing and validating our observations and results by doing a competitive product comparison. Such a comparison will allow our group to see what user requirements are necessary to design an engrossing and easily usable/efficient design that is beneficial for all of our stakeholders. Therefore, understanding user’s experiences with current products is extremely beneficial as it provides us with details on which features are essential for a specific design and what concerns need to be addressed in order to achieve our design goals.

Our project deals with the helping people learn a new language. In the paper, Breaking the Disposable Technology paradigm, they spoke only to those who owned mobile phones. I think that with regard to learning a new language, in order to come up with a unique way to do this, we should speak also to people who do not have a mobile device or those who do not reply on it much, and ask them if they were to learn a new language, how would they do it. The reason is that there could be a group of people who would say the screen of their mobile device too small for such a hobby.

For this data gathering, they undertook web surveys. These surveys were aimed at collecting a broad set of data. In our project, we will focus on smaller groups for data since, we need specifics from them, and would prefer more of the interview method of data gathering as a more reliable method than surveys. This is because we would like to integrate the non-verbal cues as well. They did use interviewing, but only after their survey and they used survey questions to guide them.

In the paper, Revisiting the Visit, I stumbled upon an important design constraint i.e audio. In learning a new language, pronunciation will definitely be a key factor while designing. So the audio system for a mobile device would be the headphones, just like its use for music. Also, they gathered data through observation of people in the museum. In our project this will not be possible since, we do not know in which leisure part of the day people choose to do this.

In the paper of mobile phone, they used both a qualitative and a qualitative method to complement each other by combining a web-survey and a phone interview. Although we also conducted the web-survey and an interview, the interview presented in the paper was conducted based on the survey responses, which is different from our methods. With limited time of our project, we could only conduct the different methods simultaneously. The limitation leads to the iteration and we have do another interview which includes the questions inspired by the survey responses and the previous interview answers.

In the project Sotto Voce, they conducted three design-evaluation iterations. This iteration reminds me of our methods. Although the techniques are different, the iteration is inevitable. In our project, we interviewed several people based on our first design questions. When we tried to collect the data about people’s sleepiness, compared to the techniques mentioned in the paper, it’s hard for us to ask about their feeling or experiences while sleeping. Besides, after we analyzed the data, we found that the complexity of actions and activities couldn’t be revealed within few questions. Hence, we also used video camera to record the participants’ activities and tried to explore the implicit thoughts and impressions of the users’ experiences.