Research - HowTo

A Personal How-To

Like other faculty, I often find myself explaining the process of research and how to get started to graduate students embarking on research. Different people have different views - again like other researchers, I advise my students with the best I know how.

Here is a brief presentation of that same talk that I give my students, which may be useful to prospective or continuing students; if you do read it, remember that it is a personal view. It also necessarily relates a little to my own research area, because in very different fields, the process may be different also (as in fields where empirical studies is a large component of research).

It is probably easier to view the material below if you resize your browser window to the widest possible size.

See also my tips on research reading after the presentation, or download it here.

Slide 1 image: text reads:
Introduction to the Process of Research: A Personal View.
Rudra Dutta, Computer Science, North Carolina State University.

As we go down the long road from first grade to Ph.D., the two basic changes are: (a) we take more and more responsibility for our own learning, instead of assuming it is on teachers and parents, and (b) more and more, the balance of our learning shifts to how to learn, rather than learning specific material.
As many people have pointed out, a funny thing happens if you extend the definition of an expert to the logical extreme after the manner of the Dirac Delta function: the best expert is somebody who knows "everything" about "nothing"! Realistically, we would stop short of this ideal, while there is still something we actually are an expert on!

Slide 3 image: text reads:
Stages of Research Project.
Literature.
Problem Definition.
Modeling.
Contribution.
Validation.
Presentation.
Archival. The forward direction is obvious - you go through these stages successively. As the backward arrows indicate, it is often necessary to backtrack from the modeling to the problem definition or even the literture review stage. There is also a possible backtrack from the validation stage - if you find your wonderful new algorithm is 10 times worse than the well-known dumb one, you have to re-think your contribution. A particularly unpleasant backtrack is the one leading back from the archival stage, which can lead all the way back to square one, much like the biggest snake in "Snakes and Ladders". This happens when you submit your perfect paper to a conference or journal, and a reviewer points out some elementary mistake which invalidates the whole thing; or maybe they just say "this exact thing has already been published - see paper such-and-such." Unpleasant as it is, this happens on occasion; this is why we do not consider any research project finished until the results have been archived in some peer-reivewed forum.

Slide 5 image: text reads:
Anatomy of a Research Paper.
Title, Abstract.
Introduction, Context, Background.
Prior work - literature survey.
Problem identification, definition, (formulation).
Problem modeling, formulation.
Approaches, solution, contribution, insight.
Validation, verification, experimentation, results.
Conclusion, speculation.
References, bibliography. There is a line just before the "Problem modeling" part because essentially the part of the paper before that is tutorial in nature. If you read several papers on the same topic, the content of each upto this point are going to be very similar to each other. This helps in understanding a research area, and saves on the reading effort.

See also the "Reading for Research" section just after this presentation.

Slide 7 image: text reads:
Problem Definition.
Describe problem precisely.
Specify context of problem.
Broad context - less articulation.
Specific context - more complete description.
Specify scope of problem.
Define aspects of the problem that are not being addressed.
Specify motivation of problem.
State research question as briefly yet as specifically as possible. There are two basic considerations that must be honored in finding a research problem:

It must represent an actual problem somewhere - a "pain point" for some practitioner of some useful activity. Otherwise you will end up with a solution looking for a problem, even if your own research is very successful. In the academia, we are comparatively relaxed about this; our goal is not to be able to affect the bottomline in six months (as it often is in the industry). Nevertheless, we must not completely lose sight of this ultimate goal of all research - especially in a College of Engineers. This is what "motivation" basically refers to.
It must be something that has not yet been found, obviously. Remember the perceived "hole" in the field of knowledge. Re-inventing the wheel, no matter how nice a color your paint it, counts as development, not as invention or discovery.

A couple of techniques that can help are:

Think like an engineer, not a researcher. Ask yourself: "how can I do this now? If the answer is completely available by tying together several pieces of existing knowledge and technology, there might not be a research problem there. If, on the other hand, there is a missing link in the chain, that might be your research problem.
Try to describe the problem using text at various scales; try to express it in one paragraph, one page, one sentence, etc.,.

This is often the scary part. Many students back away from research because of this very thing, that the "spark" is not possible produce on order. My favorite example here is that of lawn mowing. Suppose you have to mow your lawn, sometime in the next month. You cannot mow the lawn if it is raining, or if it rained in the last 24 hours. You obviously cannot produce two successive rain-free days on order. Will you despair?
The point is that things which are not possible to produce on order may nevertheless not be rare, especially if one puts oneself in a receptive position. In research, you can do this by reading in that area, and trying out simple ideas which you can produce without any "sparks". Remember the dictum that "luck" is just opportunity meeting preparednesss. Without a lot of preparation, you will always have bad luck. But hard work is guaranteed to change your luck. Abraham Lincoln is credited as having said: "If I had eight hours to cut a tree, I would spend six of them sharpening the saw." Whenever you think you have nothing to try, sharpen the saw, don't underestimate the role of perspiration, and the lightbulb will come on, sooner or later.

Very few of us are lucky enough to make intellectual advances that are so pure and abstract that it stands by itself in the mathematical space. Most of us engineers do things that require validation - the ultimate proof of the pudding is in the eating.
Knowledge is not useful if it is not shared. Perhaps it is not even knowledge. Also, sharing provides a necessary step in validating - no matter how much validation you yourself carry out, you could be consciously or unconsciously fooling yourself. In science, we must honor the concept of experimental reproducibility - ideally, another researcher reading your paper should be able to reproduce your experiments just from the information in your paper.

People don't have as much time as they like. You yourself have very little time for others. Some of the others have even less time than that for you. If you will get only 20 minutes to present the research work that took you a year, it is worth a week or two to get ready to utilize those 20 minutes as best as possible.
Remember that in a technical presentation, the audience is expected to challenge you, and you are expected to address the issue raised clinically and correctly. Do not put anything on the slides or say anything that you cannot defend. You will be challenged and tested on your understanding of the topic you are presenting. In fact, if you get absolutely no questions during the presentation or at the end, this means you have completely failed, because nobody listened to what you said.
Note: Even the most experienced speaker flounders when trying to speak accompanying slides without preparation and practice. Plan what you will say for each slide you have; practice if possible. Have speaker notes and/or the source papers handy if you think they will help.

Similar considerations as above. Language, grammar, spelling, all matter. It is not other people's privilege to read about your research - it is your privilege to have them read it. Make it as easy for them as possible, and as difficult to mis-understand as possible.

Slide 13 image: text reads:
The Role of Skepticism.
Skepticism vs. gullibility.
Keep an open mind, but not so open that your brains fall out.
Open mind - do not reject without verification.
Skepticism - do not fail to verify.
Baloney Detection Toolkit.
A scientist's skepticism aid.
Adapted from The Demon-Haunted World: Science as a candle in the dark, Carl Sagan, Random House, NY, 1995. The last few slides, starting with this one, is a general overview on the scientific method. For a more detailed exposition, I can do no better than point to the book by Sagan that I have cited. I recommend it to anybody that has anything to do with science.
I hope this helped you. If it did, feel free to point links to it or use the material otherwise; but please do credit the source.
Thank you.

Slide 14 image: text reads:
Baloney Detection Kit - Do...
Seek independent confirmation of so-called facts.
Encourage debate on the evidence.
Authority carries no weight in argument.
Consider multiple working hypotheses.
Avoid NIH syndrome.
Quantify.
Insist on complete chain of evidence.
Apply Occam's razor.
Set predefined failure standards for hypothesis.

Slide 15 image: text reads:
Baloney Detection Kit - Avoid...
Ad hominem (to the man).
Argument from authority.
Argument from adverse consequences.
Appeal to ignorance.
Special pleading.
Begging the question.
Observational selection.
Statistics of small numbers.

Slide 16 image: text reads:
Baloney Detection Kit - Avoid...
Inconsistency.
Non sequitr (it does not follow).
Post hoc, ergo propter hoc (after, therefore because).
Confusion of correlation and causation (together, therefore because).
Meaningless question.
Excluded middle (false dichotomy).
Short term - long term.
Slippery slope.
Straw man.
Suppressed evidence.
Weasel words.

Reading Literature for Research

This is a personal view on reading research papers with the purpose of getting started with some research area. It seems to work for some of my students - feel free to use it to get started in your own reading, but know that you may have to come up with your own techniques in addition to these.

0. Choosing "Seed Paper". Very often, we have a rough idea of the area in which we want to (or are being asked to) do research in, and very often, we have at least one research paper in mind that epitomizes this research area. Not necessarily the first paper in this area, not necessarily the latest or the best, just a good one. We shall call this your "seed paper". Sometimes the seed paper will be assigned to you by an advisor or a superior.

1. Focusing Literature Survey. For this paper, do the following:

a. Get familiar with some Bibliographical indexing tool - I wholeheartedly suggest BibTeX, which accompanies LaTeX. Start the bibliography of your research project by including the full citation for your "seed paper."
b. Understand well the basic design problem that the paper addresses. Briefly describe the problem (in half a page or less).
c. State the research problem in one sentence. To do this, you may have to generalize/broaden the problem somewhat, so that you are describing a narrow problem area, rather than a specific problem.
d. Find the most prestigious journal in your research area, for my area this is probably IEEE/ACM ToN. Make a list of all the papers published in that journal over the past one year.
For each paper, answer the question "Does this paper address the same research question (part c) as the seed paper?" For most papers on your list, you will only need to read the title, or a few sentences in the abstract or at most the entire abstract, before ruling them out. For papers which seem as if they might be related to the seed paper, you should delve deeper to decide. You will have to read beyond the abstract for these - you should definitely read the introduction section, and the context or background section is there is one. You may have to read the problem definition or problem formulation section, but try to avoid it. Under no circumstances should you read beyond the problem formulation section. At this time, you must have a definite "yes" or "no" answer for each of the papers on your list. For each paper you answer "no" to, enter them in a separate "reject" bibliography, and keep a one sentence description of your reason for so answering. For each "yes" paper (if any), enter them in your research bibliography.
Note: As a guideline to the effort you should be expending at this stage, I suggest you should not spend more than 2 to 5 minutes on considering any individual paper before deciding your answer.
e. If you answered "no" to every paper, consider revisiting your answer to part (c) to see if you were too specific with the research question; however, if you have good reasons for each "no" answer then you are done. If you answered "yes" to 5 or more papers on your shortlist, you have definitely made the research question too general - go back to part (c), make the question more specific, and repeat.

2. Expanding Literature Survey. Not everything gets published in one top journal in the field. Realistically one should be aware of papers published in several journals and conference proceedings. In order to do this, we cannot follow the model used above (starting from the set of all papers and narrowing down), because with several good conferences and journals over several years, we are talking of several thousand papers in the original list. In this case, we have to expand our list of papers rather selectively.

a. Identify two or three keywords or keyphrases that describe the research question you are homing in on. The seed paper and the other papers on your current list may actually provide keywords, which might help. Conduct a search on the INSPEC database (available through the D H Hill Library's website) with combinations of these keywords and keyphrases. Search over approximately the last 5 years of the database.
b. Repeat your keyword searches independently on the IEEE Xplore database and the ACM Digital Library database, and report if you turn up any papers that your search on INSPEC did not turn up.
c. Repeat part (d) of Step 1 with the papers that you turned up. At this time, you should have between 5 and 20 papers that you have classified "yes". We shall call this your "core list".
d. Read the bibliographies of each paper on your core list, and make a list of papers that also address the core research topic, and appear in the bibliographies of more than one of the core list papers. Add these papers to your core list.
e. By this time, from the list of authors of the papers on your core list, you have a good idea of who the most active or influential researchers in your research area are. Visit their websites - they will usually have a list of their own publications. If you find any paper that addresses the core research topic and that you have not found so far, add them to your core list.
f. So far we have searched backwards. Now we have to try forward searching, which is more difficult. Fortunately, it is much less difficult now than it was even 10 or 15 years ago. There are two fairly good public citation search engines - Citeseer and Google Scholar. Identify one or two papers on your core list that have the maximum citations within the core list. Conduct a search on Citeseer and Google Scholar to find other papers that refer to one or more of these papers, and that you have not already encountered before. For each such new paper, repeat part (d) of Step 1. If you answer "yes" to any of these papers, add them to the core list.
g. Enter all the papers on your core list into your bibliography.
Note: Retain all the intermediate lists you have made, and save all the keyword searches you performed.

3. Reading and Remembering. Now you have to read all the papers in your core list. There is no shortcut and no way around; fortunately it gets much easier after the first couple of papers in any research area. I have only a couple of suggestions on reading.

a. Research papers and presentations, textbooks, almost all archived material is presented sequentially, but this is not how the human mind works in understanding, learning, or creating. When reading, you have to jump back and forth a little bit. But if you keep referring back, you may never make much progress. One technique that helps is to underline or highlight the key things, such as definitions, notations, formulae, as you come to them, because these are things you are likely to want to refer back to from later in the paper. Another is divide-and-conquer; which in this context means if you do not understand some part of it on repeated reading, then leave it behind for the moment. Just make sure you understand what the consequence of that part is for the rest of the paper, then take the author's word for it, and move on. Re-visit it later; it might be much easier to understand in the light of the understanding of later parts of the paper, or a different paper.

b. Every time you read a paper, annotate its bibliography entry to build up an annotated bibliography. An annotated bibliography is a regular bibliography (list of cited references), with a descriptive short note (one or two paragraphs, not more than one page) written by you documenting your understanding of this paper. It is hard to believe after you just spent hours reading and understanding every detail in some paper, but you will forget all about that paper in a matter of months or even weeks. An annotated bibliography saves you from having to undertake the entire effort again - usually reading the annotation is enough, or at most a few minutes worth of the actual paper in addition. More details are available at various sources, for example, D. H. Hill Library has a webpage on annotated bibliographies.

c. Re-run all the literature searches that you made to come up with your core list of papers. The intent is to become aware of research work on your topic that may be have been published since you started work on it. A secondary goal is to tune your literature survey to your research topic, if that has taken a turn.

Additional Resources

A paper in ACM CCR, July 2007, by Keshav provides a three-pass paper reading method. The references cited in this paper are also useful. You need access to the ACM Digital Library to use the link above (automatically available from NCSU); the citation of the print copy is: "How to read a paper", by S. Keshav, in ACM SIGCOMM Computer Communication Review, Volume 37 , Issue 3 (July 2007), Pages: 83 - 84, April 2007, ISSN:0146-4833.


	As we go down the long road from first grade to Ph.D., the two basic changes are: (a) we take more and more responsibility for our own learning, instead of assuming it is on teachers and parents, and (b) more and more, the balance of our learning shifts to how to learn, rather than learning specific material. As many people have pointed out, a funny thing happens if you extend the definition of an expert to the logical extreme after the manner of the Dirac Delta function: the best expert is somebody who knows "everything" about "nothing"! Realistically, we would stop short of this ideal, while there is still something we actually are an expert on!
	The forward direction is obvious - you go through these stages successively. As the backward arrows indicate, it is often necessary to backtrack from the modeling to the problem definition or even the literture review stage. There is also a possible backtrack from the validation stage - if you find your wonderful new algorithm is 10 times worse than the well-known dumb one, you have to re-think your contribution. A particularly unpleasant backtrack is the one leading back from the archival stage, which can lead all the way back to square one, much like the biggest snake in "Snakes and Ladders". This happens when you submit your perfect paper to a conference or journal, and a reviewer points out some elementary mistake which invalidates the whole thing; or maybe they just say "this exact thing has already been published - see paper such-and-such." Unpleasant as it is, this happens on occasion; this is why we do not consider any research project finished until the results have been archived in some peer-reivewed forum.

	There is a line just before the "Problem modeling" part because essentially the part of the paper before that is tutorial in nature. If you read several papers on the same topic, the content of each upto this point are going to be very similar to each other. This helps in understanding a research area, and saves on the reading effort.
	See also the "Reading for Research" section just after this presentation.
	There are two basic considerations that must be honored in finding a research problem: It must represent an actual problem somewhere - a "pain point" for some practitioner of some useful activity. Otherwise you will end up with a solution looking for a problem, even if your own research is very successful. In the academia, we are comparatively relaxed about this; our goal is not to be able to affect the bottomline in six months (as it often is in the industry). Nevertheless, we must not completely lose sight of this ultimate goal of all research - especially in a College of Engineers. This is what "motivation" basically refers to. It must be something that has not yet been found, obviously. Remember the perceived "hole" in the field of knowledge. Re-inventing the wheel, no matter how nice a color your paint it, counts as development, not as invention or discovery. A couple of techniques that can help are: Think like an engineer, not a researcher. Ask yourself: "how can I do this now? If the answer is completely available by tying together several pieces of existing knowledge and technology, there might not be a research problem there. If, on the other hand, there is a missing link in the chain, that might be your research problem. Try to describe the problem using text at various scales; try to express it in one paragraph, one page, one sentence, etc.,.

	This is often the scary part. Many students back away from research because of this very thing, that the "spark" is not possible produce on order. My favorite example here is that of lawn mowing. Suppose you have to mow your lawn, sometime in the next month. You cannot mow the lawn if it is raining, or if it rained in the last 24 hours. You obviously cannot produce two successive rain-free days on order. Will you despair? The point is that things which are not possible to produce on order may nevertheless not be rare, especially if one puts oneself in a receptive position. In research, you can do this by reading in that area, and trying out simple ideas which you can produce without any "sparks". Remember the dictum that "luck" is just opportunity meeting preparednesss. Without a lot of preparation, you will always have bad luck. But hard work is guaranteed to change your luck. Abraham Lincoln is credited as having said: "If I had eight hours to cut a tree, I would spend six of them sharpening the saw." Whenever you think you have nothing to try, sharpen the saw, don't underestimate the role of perspiration, and the lightbulb will come on, sooner or later.
	Very few of us are lucky enough to make intellectual advances that are so pure and abstract that it stands by itself in the mathematical space. Most of us engineers do things that require validation - the ultimate proof of the pudding is in the eating. Knowledge is not useful if it is not shared. Perhaps it is not even knowledge. Also, sharing provides a necessary step in validating - no matter how much validation you yourself carry out, you could be consciously or unconsciously fooling yourself. In science, we must honor the concept of experimental reproducibility - ideally, another researcher reading your paper should be able to reproduce your experiments just from the information in your paper.
	People don't have as much time as they like. You yourself have very little time for others. Some of the others have even less time than that for you. If you will get only 20 minutes to present the research work that took you a year, it is worth a week or two to get ready to utilize those 20 minutes as best as possible. Remember that in a technical presentation, the audience is expected to challenge you, and you are expected to address the issue raised clinically and correctly. Do not put anything on the slides or say anything that you cannot defend. You will be challenged and tested on your understanding of the topic you are presenting. In fact, if you get absolutely no questions during the presentation or at the end, this means you have completely failed, because nobody listened to what you said. Note: Even the most experienced speaker flounders when trying to speak accompanying slides without preparation and practice. Plan what you will say for each slide you have; practice if possible. Have speaker notes and/or the source papers handy if you think they will help.
	Similar considerations as above. Language, grammar, spelling, all matter. It is not other people's privilege to read about your research - it is your privilege to have them read it. Make it as easy for them as possible, and as difficult to mis-understand as possible.
	The last few slides, starting with this one, is a general overview on the scientific method. For a more detailed exposition, I can do no better than point to the book by Sagan that I have cited. I recommend it to anybody that has anything to do with science. I hope this helped you. If it did, feel free to point links to it or use the material otherwise; but please do credit the source. Thank you.