Theme 5, Reflections

What's design research?

Previous to this weeks discussion on design research my view of the topic was limited to the design and  testing of prototypes. Now my scope has broadened to include the design of non-material ideas, such as frameworks and research methods. Instead of classifying research that includes prototype developing as design research, my view is now to classify research that relies on the development and testing of ideas as design research. This view was also supported by the lecture held by Haibo Li, where a lot of focus was put on the importance of good ideas, and how good ideas can be filtered from bad ones. Research that explore ideas, should therefore put effort into explaining and developing the concept phase, where the idea is born, just as much or even more than the testing and development phase. 

This weeks seminar also brought up that the main difference between many design research papers where the methods used to come up with the idea. And the development and testing(if there was one) was more straightforward and followed logically upon what the idea claimed to achieve. 

One thing that I want to argue is crucial to design research, and that was not brought up so much in the lecture or the seminar, is the importance of the conclusion and suggestions for further development in a design research paper. The artefact/idea that the research produces is not of much value to other researcher unless a critical discussion on how it can be improved is made. This is also what I believe differentiates design research from product development, where the goal of the research is to improve something not only by designing a new artefact that adresses what to be improved, but also to contribute  to the knowledge of how something could be improved even further by others. This can be put in contrast to product development by a company where the goal is to improve something to get a market advantage and further improvements are motivated by just that(I guess that this is somewhat the classical argument for why capitalism is an effective system).

  

Theme 5, Design Research

In preparation for this weeks theme I've read the papers "Turn your Mobile Into the Ball: Rendering Live Football Game Using Vibration" by Shfiq ur Réhman et al. and "Parametric Time-Frequency Representation of Spatial Sound in Virtual Worlds" by Mikko-Ville Laitinen et al. Both papers could be classified as design research, where the authors examines a proposed design.

The papers differ in that the paper written by Réhman et al. has a clear focus on a practical application to which the proposed design could be used, that is, providing real-time information about a soccer game. The paper written by Laitinen et al. is more vauge, and argues that getting spatial information of virtual worlds through a sound display has various applications, but the technology is not aimed at any specific example.

Evaluation of Media Technologies

Both papers ground their evaluation methods in standards. The paper by Réhman et al. uses ISO recomndations to develop a evaluation scheme that measures three aspects of usability: Effectivness, Efficiency and Satisfaction. The other paper evaluates the design with a formal listening test, conducted in a listening room following recommendations for listening test where small changes in sound are produced(http://www.itu.int/dms_pubrec/itu-r/rec/bs/R-REC-BS.1116-1-199710-I!!PDF-E.pdf). 

It's interesting to see that both papers choose to base their evaluations to some extent on proposed standards, and this probably gives the results some credibility since they are more likely to be comparable to similar research.

Both papers used some form of user evaluation as their method of evaluation. This can of course be tricky when testing novel designs, since users have no previous experience of the designs. This might be the reason for both papers to include some form of semi-informal user training in their evaluation to give the users the opportunity to familiarize themselves with the technology to be evaluated.

Communicating Design Research

To express in a paper what a design consists of can be challenging and to me this is espessially noticable in the paper about Spatial Sound. This since sound design is really hard to grasp without acctually experiencing it. The paper relies heavily on the knowledge of the reader, so that for example the reader through a matematical explanation of how sound intensity varies through spatial parameters can understand how the design works. For me, being not an expert in acoustics, it proved challenging to get a mental picture of the design as such. The other paper had an easier task of describing the design since the vibration output of a cellphone is something that I'm familiar with, and also something that has fewer parameters that vary. I understand how the design is suposed to work, and I can picture to my self how the different vibrations would feel.

Conclusions

The two papers where similar in their approach to design research, and followed the scheme of prototype developing followed by user testing. Compared to the article that was discussed in last weeks theme, they presented a research methodology that was more in the scope of to what I'm used to. The conclusions and discussions of the papers where, due to this approach, more focused on the success of the prototype, and values measured in the user tests. For this kind of research it seems to me that close focus has to be paid to the arguments for why the prototype should be developed, since a successful user test may be conducted even if the prototype isn't useful in any real-life scenario.

Theme 4, Reflections

Qualitative data is hard to gather and analyze

This  week I've come to realize that gathering and analyzing qualitative data requires a high level of skill. I contrast with quantitative methods, the qualitative methods seem to require more thought to be put into to every step of the method. One should strive to keep the sampling random, but on the other hand the requirements on participants, or data seem to be more detailed in most cases in qualitative research. The data then have to be both gathered and analyzed in a systematic way, and this without losing the important nuances in qualitative data that will help to explore the subject.

A big difference between quantitative and qualitative seems to be in the exploratory nature of the former. Qualitative methods seem to be more exploratory in nature, and I think it would really hard to argue for any hypotheses when conducting qualitative research.

The method used in the paper I read seemed quite obvious for the research question. It could be described as an analysis of essays written by participants on the subject of "my experience of growing up with interactive media". Even so I have not read any papers where that method have been used before. I really like how the authors argue for that the participants memory would be self selecting, and thus only milestones of experience with interactive communicaton would be noted in the essays that the participants wrote.

Design research and qualitative methods

I still find it hard to understand the method described in Ylva Ferneaeus paper. As I understand it a design can be motivated by theory, and tested through prototyping. Thus no further evaluation would be necessary. What I find hard to grasp is the lack of motivating why the design should be prototyped, this since I'm used to research of the type that evaluates a design because it could be used by someone for a specific task. To pursue such research further evaluation would probably be nessesary, but Fernaeus research seems to have the purpose test a design because it could be motivated trough grounded theory. To me it looks like a fine line between research, and just testing stuff to explore possibilities, and it's probably hard to formulate such research.

Theme 4, Qualitative Methods

Brief summary of the paper I read

I read the paper "Coming of age with the internet: A qualitative exploration of how the internet has become an integral part of young people's lives". It is an explorative study of how the view college students in the US have been shaped by growing up in the same time as the Internet was introduced in the western world(Mcmillan, Morrison, 2006). The paper also look at how the Internet now is integrated into their daily lives. It was published in 2006 and  examines personal narratives writen by student in the years 1998 to 2000 about how interactive media have affected their lives. The students where also asked to reflect on how they use interactive media today.

The Method

The method that is used in the paper is a qualitative analysis of written material, that could be called a document analysis. The authors argue that the method in this case provides better data to analyse than oral interviews because subjects have more time to reflect on their past and current experiences. As opposed to oral interviews where some of that information may be lost. I follow this argument and think that even though more non-verbal information might be communicated in a face to face setting, it might be hard to analyse this information in a systematical way.

How was the data analysed

The analysis of the data was built up around an itterative systematical coding and sorting of the texts. First they where sorted uppon passages relevant to the research question, then they where broken down into themes and topics, and finaly more selectively examined to create a picture what was found in the first two stepes.

My reflections on the method

The method is in my opinion well balanced and thought through, and I feel that they have found some themes that are relevant to the papers discussion. The paper showed me in a really clear way how qualitative analy sis can be performed in a very systematical way, even though the data was personal narratives that didn't follow a strict patern. I also belive that written data, just as the authors argue, might in many cases provide more information than what can be extracted from a face to face interview.

Design research

I think that the paper illustrates design research in a good way. It defines the goal of the designs as an programming interface that can be understood by "ordinary" people and thats doesn't require a screen to illustrate the actions of the programmer. Whith these clear goals the research prosses then becomes a question of trying to achieve these concepts on a technical level,  which makes it possible to evaluate the research. The study can't hovewer tell us if the approaches suggested are going to be successfull or not, just that they are doable.

The question i have about design research is how you find the balance between potential and novelty in a desing? The ultimate design is of course something in wich you can se a potential(for example commercial) and also is very new and unheard of. But you could think of designs that are not very innovative but have a great potential in improving something, or resulting in a succesfull product. On the other hand you could think of the designs that are very new and innovative, but that you can't realy see how it would be usefull in the "real" world. Where do you find the balance between these two?

References

Fernaeus, Y. & Jacobsson, M. 2009. "Comics, Robots, Fashion and Programming: outlining the concept of actDresses".  Proceedings of the 3rd International Conference on Tangible and Embedded Interaction.  

Mcmillan, S. & Morrison, M. 2006 "Coming of age with the internet: A qualitative exploration of how the internet has become an integral part of young people's lives" New Media & Society. 

Theme 3, Reflections

What have I learnt?

The main thing I take away from this week is how quantitative methods provide support for arguments in research. Before I found it hard to relate quantitative studies to more intriguing research questions, and it often stopped at the frequency of som occurrence. The lecture brought up the examples of divorce statistics, and before I thought of quantitative methods as just counting the number of divorces, now I see that such data could be just a step in drawing up conclusions on why divorces occur, and that the data you collect is necessary to back those conclusions.

My understanding of the method used in the paper we read as preparation for this week grew after the lecture. Before I was unsure of the purpose of the paper, and how the method was developed, but now I think that I see how the results of the quantitative analysis are used to test the model of Community Inquiry that the paper presents. I also see how mixed methods should be thought of as answering research questions that can't be answered by either quantitative or qualitative methods. Before I saw mixed research as partitioning the research objective into one quantitative part and one qualitative part, but then it's probably better to do two separate experiments or surveys and present them independent of each other.  

The lab exercise did not give me that much. I learned how to use the software, but since it's seemed quite intuitive that wasn't something that I feel was the main thing to take away. Now I know know of the existence of the software, but there are probably other tools aswel, also the examples we did in the lab exercise could probably be done using a more general calculation software such as excel. As usual when using software or learning a programing language i feel that the main difficulty is not how to do something(that can be discovered through google or trial and error) but rather what to do. This of course would vary with the experiment or research one is doing, but I still feel like I'm missing some core understanding in how to use statistics in a meaningful way apart from just comparing frequencies. I do however take with me the books Ester Appelgren recommended  and I hope that they will be a good read before doing my Masters Thesis. 

Theme 3, Quantitative Methods

Main points in the paper "Emotional presence, learning and the online environment"

In preparation for this theme we read the article "Emotional presence, learning and the online environment(Cleveland-Innes, Campbell, In press). Below follows a brief summary and critique of some of the main point pf the paper.

What is the purpose of the study?

The main purpose of the study is to evaluate through exploratory research if it's possible to expand the model of Community Inquiry applied to an online learning environment, with a fourth element of emotional presence added to the three elements: social presence, teaching presence and cognitive presence. 

The community of inquiry model(CoI model)

The community of inquiry(CoI) model is briefly explained in the paper as the way community emerges in suport of learning through the relationship between social presence, teaching presence and cognitive presence(Cleveland-Innes, Cambell, In press). The common illustration of the model is seen in figure 1.

Figure 1(http://communitiesofinquiry.com/model).

The purpose of the paper is thus to expand this model to include emotional presence as its own part in the model illustrated in figure 1.

What method is used?

The method used in the article could in my opinion be called a quantitative content analysis. However it's a bit unclear since the authors refer to the data analysed as qualitative. The data was collected from open-ended questions and conference discussion transcripts from online college courses. Then it was examined for indicators of the different parts of the CoI model, plus indicators for emotional presence.

The paper is in my opinion insufficient in the way it describes the method used. Especially in when it comes to explaining how the content analysis of the data was conducted. It's stated in the paper that

 "..coders discussed the range of emotions noted in the textual data, and discussed the use of terms to label expressions deemed to be 'emotional.'..."(Cleveland-Innes, Campbell, In press).

Whats lacking is an explanation of what was discussed, and how they came to the conclusion of what emotional labels to use.

Mixed research

The second paper read in preparation for this theme was a paper on key points for improving mixed research in the field of online learning by Patrick R. Lowenthal and Nancy L. Leech(2009).

What are the difficulties in mixed research?

One of the main point in the paper is: mixed research is such a new type of research that most researchers doesn't feel comfortable in conducting it. Most researchers come from a background of either qualitative or quantitative research wich influences the approach they have on mixed research. The main difficulty in mixed research is thus to construct research that is fittet to a mixed approach, instead of applying a mixed approach to a research question and design appropriate for a quantitative or qualitative method.

The paper is much to the point in describing what obstacles lay in the way of conduction good mixed research, and most cases it proposes a solution to how one should approach these obstacles. One component that I believe is missing however is a discussion of why to conduct mixed research, which also would to som extent answer the question of when to do it.

References

Lowenthal, P. R. and Leech, N. 2009. "Mixed research and online learning: Strategies for improvement" Online education and adult learning: New frontiers for teaching practices. Hersey, PA: IGI Global.

Cleveland-Innes, M. and Campbell, P. In press "Emotional presence, learning and the online learning environment" The International Review of Research i Open and Distance Learning.

Theme 2, Reflections

What have I learnt?

What is theory?

My understanding of what theory is have gone from basically none, to at least an understanding of what the main components of a theory is. I still find it hard to identify theories when reading research papers hoverer and this might be a shortcoming from my side, but I also think that many papers are quite unclear in describing what theory is used if there even is one. From now on I will try to identify theories in every paper I read, since I think that I'm missing those types of frameworks when thinking about research and trying to write my own papers. Hopefully the list on KTH-social will come in handy in the future outside of the context of this course. 

The paper "The Nature of Theory in Information Systems" was in my opinion a bit to focused on classifying theories. I think that theory is a little bit what you make it, and the what type of theory it is depends more on how you use it rather than the intention of the one formulating it.

What did I learn from the paper of my choice?

I think that I was somewhat lucky finding a paper that used theory in a very explicit and clear way. This was great since I haven't really read a paper like that before, at least not one with a connection to media-technology. It made me realize how powerful theory can be if it's used in a meaningful way. It also showed me how to relate theory to the methods and purpose of research, something that I was definitely lacking when writing my B.Sc thesis. 

What's still missing after this week for me is the answer to the question "how to formulate theory?". I have a better understanding of what theory is, and how it can be used in research. To formulate theory is still a mystery however, even when I know that a theory should be based on logical arguments that explains causality I still think that it would be hard to formulate such arguments. But that is of course something which you cannot learn in a week, but rather comes from experience and reading other peoples theories; and now I feel much better equipped to do that.  

   

Theme 2, Theory

What is theory?

Theory could be understood as logical reasoning, that argues for why something that can be observed might lead to another observed phenomenon. The theory also has to argue why these two are related. This can be seen in many of the more famous theories such as the logical reasoning behind why objects acted upon by a force stay in motion until acted upon by another force. Here the force is related to the movement of the object. 

It's important to note that a theory is not something that is only observed(data). So observing that an object acted upon by a force stays in motion is not a theory, the theory is constructed by reasoning why this is the case, and then making it more certain by observing occurrences when the theory is proven to work.

When writing a scientific paper its also not enough to use references to constitute the theory of that paper. The theories also have to be explained, and the author/authors have to argue why the theory is relevant to their research. 

Theory used in the paper I selected

The paper I selected to read was a paper in the field of Medical Information Technologies that was published in the journal: Journal of American Medical Information Association. I choose the paper because of it's deliberate use of grounded theory to construct a framework of how consumer health information technologies(CHIT) should be developed and understood from the patient perspective(Or et. al 2010).

The grounded theories that are used in the paper is the theory of planned behavior that was proposed by Icek Ajzen and the unified theory of Acceptance and use of technology proposed by Viswanath Venkatesh et. al  (1991, 2003). The theory that's laid up in the paper builds on these two theories and proposes that the effective use of a CHIT system can be understood in terms of behavioral intention built up from how useful the patients think the system will be and the healthcare knowledge of the patient. The theory is then tested on a group of home care patients using a web-based system for self-care of chronic cardiac disease. 

I would place this theory in the category of Explaining and Predicting(EP-theory) found in Shirley Gregors paper on the Nature of theory in Information Systems(Gregor 2006). This because it both explains the causal relationships that can be used to predict the effective use of a CHIT system, and those causalities are explained using the two more general theories. The theory is also tested and its concluded that the theory can predict the effective usage of a CHIT system. 

I think that the benefits of using the theory in the paper are related to the theory's limitation. The benefit I see is that the theory can be applied to any user-group since its not limited to any parameters in the user group in such as what age, gender, income etc. This is also a limitation of the theory since some of those parameters are likely to be excluded by the theory even though they might have an impact on the effective use. The theory also doesn't help in understanding "how" CHIT should be designed but only explains the causal relationships that predicted to what extent the system might be used. So the theory could be used to understand the user-group and how likely they are to use a system at all, but not how that system should be designed to meet the needs of those users. 

References

Gregor, S. 2006. "The Nature of Theory in Information Systems". MIS Quarterly Vol. 30, No. 3

Or, CKL. Karsh, BT. Severtson, DJ. Burke, LJ. Brown, RL. Brennan, PF. 2010. "Factors affecting home care patients' acceptance of a web-based interactive self-management technology". J Am Med Inform Assoc Vol 18, No 1

Ajzen, I. 1991  "The theory of planned behavior. Organizational Behavior and Human Decision Processes". 

Venkatesh, V. Morris, MG. Davis, GB. Davis FD. 2003. "User acceptance of information technology: toward a unified view". MIS Quarterly Vol. 27, No. 3