Suggestions on Research-based Thesis

Background: I was previously invited by my university’s Computer Science department to write up some suggestions for undergraduate students who have few research experience on their final year (thesis) project. I believe there are values in sharing this piece publicly, given many inquiries I received in the past.

Epistemic effort: Spent ~10 hours writing and improving this post based on people’s feedback.

Introduction

I struggled a lot when I first started out doing research, and I hoped there was more written guidance on how to progress more effectively and spot unknown unknowns. In summer 2021, I submitted my thesis project to a top conference in its field and got accepted - my third first-author top-conference paper in my undergrad years. You might share my initial frustrations if you are taking up a research-based thesis while having zero research experience beforehand. In this post, I will share some practices and opinions that I gradually got to appreciate throughout my (albeit limited) time trying to be a better researcher.

Research-based projects can be very different from the types of engineering projects we get trained on in coding courses: We don’t have perfect answers to whether a certain system works, or if there exist recognizable patterns behind data. It implies that we will need to handle uncertainties more carefully throughout our idea iteration cycles, which I will elucidate below. While I expect this essay to be more helpful for students doing a research-based thesis, some general principles can be useful for engineering-intensive projects too.

Caveat: I am more trained in conducting machine learning research, so although I’ve tried my best to keep my advice general for most Computer Science (CS) subjects, some of them might still be less applicable to other areas, especially in the section “Qualities I look for when I review papers” as it might not fit for other CS areas’ review process. If you have any questions, your supervisor’s opinions should certainly be weighed more than mine :).

Some good practices of doing research

Be clear in your purpose while reading papers

Depending on your project’s progress, you might be reading papers for different purposes: At the beginning of the project, you might be reading papers for literature review. At later stages, you might want to check out others’ opinions on a particular topic. Therefore, the type and location of the information that will be useful for you vary depending on your purpose. I almost never read a paper from its beginning to the end in one pass: What’s the meaning of burying myself into experiment details if I am just curious about its unique insights? Therefore, it can save you a lot of time if you know exactly what information you are looking for, and go directly for it.

There are many good suggestions on reading papers effectively. One resource that helped me a lot is How to Read a Paper by S. Keshav.

Choose a good topic

Deciding on specific topics (or directions within the current topic) to pick is an arguably tricky question, as this relates to the “research taste” people talk a lot about. Independently picking concrete research projects requires one to read widely and think about what has been missing in the literature, and people’s approaches vary a lot. I usually ask myself three questions when I read papers and look for topics to work on: (1) What insights from this paper can be useful to lay the foundation for a solution to problem X? (2) Why is this approach (which I just came up with) novel and tractable? (3) Why aren’t people already working on it?

There are many more to flesh out on improving one’s research taste, as it is a skill that can take years to improve. Two resources I find useful are (1) A Survival Guide to a PhD by Andrej Kaparthy (especially on “Developing Taste” and a few paragraphs after it) and (2) Research Taste Exercises by Chris Olah. Although their intended readers are more experienced researchers, attempting some methods within these two posts even when you have relatively limited knowledge can still be helpful.

Know your project’s intended contributions

Setting the right intended contributions can influence many crucial decisions in the future: What are the gaps we are trying to fill through our work? What experiments should we conduct to best support our claims? What type of analysis can be useful for the community? Etc. These intended contributions can change as your project progresses - some hypotheses might be rejected. Even so, I find writing a short introduction to the project early on very helpful: It prevents me from losing track of the actual project’s scope and informs me on what to prioritize.

In my opinion, for a research project to be meaningful, it needs to find a certain balance between novelty and tractability: Projects that make only incremental improvements are less interesting, and those that aim too high can’t be feasibly achieved within a foreseeable timeline. Talking to your supervisors and fully understanding what you try to achieve with your project can be very helpful.

Prioritize tasks that are riskiest and most informative

As I discussed above, we have high uncertainty in whether certain systems work or not. While it seems right to prioritize the hardest tasks, it is the riskiest tasks that decide the actual progress of your project. If someone nailed to engineer the hard ones (e.g., building a complex user interface) within the system but eventually failed on a risky and crucial component (e.g., a machine learning agent), it is likely that the time left would not allow him/her to produce another high-quality work.

Another type of task that I find helpful to prioritize is the most informative task. I define “informative” as the level to which it informs your future decisions, or your time saved by knowing certain things won’t work. For example, knowing that method A fails can quickly rule out your attempt to try method A in all future plannings and save your time.

It’s even better if both considerations above can take the time cost of verification into account: If tasks A and B are equally risky (or potentially informative), but task A takes 10 hours less than task B to complete, then prioritize task A. In general, trying to solve the simplest version of your problem first is a good practice because it informs you whether your method can solve the problem in its simplest instance. In practice, among all possible next steps, my personal proxy for risk and informativeness is my confusion: Which task am I most confused/uncertain about? You might have other proxies depending on your projects, and talking to people with prior experience can be very helpful too.

Possibly helpful resource: Research as a Stochastic Decision Process by Jacob Steinhardt

Know your purpose for this cycle and cap the time

I treat the progress of research as an iterative process:

• Coming up with hypotheses.
• Conducting experiments to validate the hypotheses.
• Observe experiment results and accept/reject them.
• Extract insights and decide what is most valuable to study in the next cycle.

Depending on your type of research, the way you come up with and validate hypotheses can be different. This is very similar to the debugging process, where we form hypotheses on lines that produce the bug, then test to verify and fix the bug. I usually try to cap each cycle within 2 weeks, or 1 month for harder ones (assuming I have other courses to take care of while working on my thesis). This means you get some conclusions that will help you work towards your goal every 2 weeks or 1 month.

Organize your findings and thoughts every now and then

Since doing research is like searching for valuable conclusions from a huge hypothesis space, you accumulate insights (and/or confusions) along the way you do your work. At certain points, you might observe a mixture of thoughts messing around in your mind, which is likely to be a combination of your insights and questions. Writing down a one-page document recording your thoughts every now and then can help organize your ideas and share them with others. In these documents, I usually informally write down the project’s current problems, my insights/observations on it, my key uncertainties, and actionable items. I sometimes use it to decide my to-do list for the week, or if I find it valuable, the material to be discussed in the group meeting.

Effectively communicate with your group mates and supervisors

The meeting time is precious, so leave it for things that actually require interactions. If anything can be written down and shared beforehand without taking too much extra time, I usually prefer to share that before the meeting. For example, if I have new theoretical/experimental results, I usually share them with the group immediately without waiting until the next meeting. I almost always write up my updates a few hours before our next meeting to make time for my group mates to read them. If we were meeting with professors, I would make slides and prepare some write-ups at least 12 hours beforehand and send them through emails. In this way, both your group mates and professors have a chance to get updated before your meeting starts, so you can save time discussing more important topics that can’t be easily done asynchronously. (But also be kind if they don’t get time to read them ahead - They might be busy too!)

Therefore, it is a good practice to record your experiment results whenever possible, and have everyone’s progress in one place. Another benefit is traceability. It happens a lot when you find some experiments conducted in the past or some discussions that happened in previous meetings helpful for future decisions. It will be much easier to trace back if you keep a good record of them. During the year I was working on my thesis, we shared a common Google Doc recording our experiment results. We eventually accumulated 255 pages of results that we had to split the doc according to the month we conducted those experiments. Our fortnightly meeting minutes also grew to 81 pages. It’s not the page number we were optimizing for - It is more about how much information can be communicated outside of meetings, and how much potentially valuable information you can keep with fairly low cost throughout the process.

There are many great suggestions on how to keep a lab notebook for logging experiments. A good template should typically include a table of content (convenient for navigation), the date, experimenter, goals of the experiment, plots or tables of results, explanations, observations/analysis of results, future actions, and the path where these results are stored. See this article for details. For meeting minutes I suggest including a section for everyone’s updates for the past week.

Side note: The lab notebook and individual updates can be strong evidence for professors to decide your contributions to the project, and reporting free-riders.

Perfecting your project

Maybe you have higher ambitions than simply finishing your thesis. You might want it to have high quality and be actually valuable for the community. In this section, I share some of my thoughts on the type of projects that I think are of high quality and can thus be considered as top conference/journal papers, along with some general tips to actually do that. If you are not aiming for submission, it can still be helpful as a reference to improve your thesis.

Qualities I look for when I review papers

I have reviewed papers and had my papers reviewed at different conferences. In this section, I share some common qualities that I usually look for when I am a reviewer, and those often mentioned in the comments given by other reviewers for my papers.

Minimal requirements. Does the paper follow the right page limit and format? Is the paper written clearly with few grammatical errors and typos? Have the authors explained their graphs and tables? Etc.

Significance. Deciding on the significance of a paper requires an adequate familiarity with the field and knowing some of its recent updates. Also, it can be challenging to produce papers with significant contributions to a field if it is your first research paper. Consult your supervisor on what he/she thinks about the significance (or potential significance) of your project, as it relates to your success rate submitting your project to different conferences/journals.

Whether the method or argument makes sense. One type of insight experienced researchers accumulate over time is their estimated chance that certain methods could work or certain arguments are correct. If you write something unlikely to be true, it can leave bad impressions on your work even if this statement is irrelevant to your main contribution. In most parts of your paper, write opinions and arguments that are either backed by your experiment results or originated from credible sources if you are just starting. If you really want to include your speculations, clearly state your confidence level while checking with your supervisor if your claim sounds reasonable.

Whether the experiments are solid and support their claims. Solid experiments are usually compared against reasonable baselines, validated on different benchmarks, and sometimes come with extensive ablation studies. If you are proposing new methods, your experiments should show that your methods work well on important benchmarks and even better, generalize across different benchmarks. If you are trying to find patterns within data, use different tools from various angles to show that they all echo with your main observations.

Value insightful observations over SOTA results. You don’t need to beat the performance of state-of-the-art algorithms on a benchmark to get your paper accepted. They should just be decent and support your claim. If this is your first research project, in most cases, your time can be more valuable spent on finding insights and observations that can inform future research rather than diligently adding tricks to push up the performance. Depending on your area of research, insightful observations can help deepen people’s understanding of the problem, point out possible sources of failure, or core reasons why some methods worked. They can come from various experiment results or your theoretical analysis. You can consult your supervisor on what types of experiments/analyses are likely to provide such insights.

Tell a convincing story and improve your writing. Many academic papers follow a very similar logic flow. If you read a few in your field, you might notice they have very similar section arrangements or even the way they structure each section. Paper writing is a very important part of unspoken norms within research communities. It is not only about your English proficiency, but also the type of language you use and the logic you deploy underneath your writing. If it is your first time writing a paper, I suggest finding a few accepted papers that have similar goals to you (e.g., proposing a new algorithm or analyzing existing phenomenon), and imitating how they structure and write their papers. For non-native English speakers, it is sometimes tempting to use fancy phrases to exercise your vocabulary, but academic communities value clarity more than that. Good writers can convey their thoughts using fewer and simpler words.

Submitting your thesis to conferences and journals

Have the right mindset and be prepared to work for a prolonged period. Conferences and journals might have very stringent requirements that even experienced researchers get their papers rejected, not to mention it is your first research and your first submission. Don’t hold on to acceptance too much but instead treat it as a learning experience. Also, be prepared for it to be rejected (It’s very common!) and the project timeline may extend beyond your thesis submission deadline.

Start writing early and leave enough time for feedback. As I discussed above, learning the writing norm can take quite a while if you are relatively inexperienced. Also, aggregating all your results and making reader-friendly graphs and LaTeX tables usually take more time than expected. I personally think my writing is decent among non-native English speakers (Getting full marks on TOEFL writing), and I spared 1.5 weeks just for getting feedback and adjusting my writing for the submission where I got my first paper accepted. I suggest for people who write papers and aim for a conference/journal submission for the first time, try to write at least half of your paper with most of your experiment results 2 weeks ahead, and leave 3 days for improving your writing and proofreading for the final version. Maybe I am on the more cautious side of the spectrum, but don’t overlook the importance of writing your paper when you just started - It is the only “user interface” to communicate your ideas with others, just like those perfect slides made by consultants and entrepreneurs.

Good luck with your project, and I hope you’ll enjoy this process :)!

Acknowledgement

I am especially grateful to receive support and feedback from Yawen Duan, Yulong Lin, Brian Tse, Ziya Huang, and Zhonghao He in perfecting this post.