Author: Sanjay Goel, http://in.linkedin.com/in/sgoel
This checklist can be progressively used by the research students, especially in engineering disciplines, to carry out a reflective assessment of their research work. Different elements of this checklist can help them to find the potential gaps in their work and opportunities of further improvement. Professional R&D engineers and resaerchers in disciplines other than engineering will also find this checklist useful wrt their work.
A. Reflecting upon your research/project problem
1. What problem have you tried to solve?
2. What is your motivation for taking this problem?
3. Why is this problem significant and relevant to the context?
3a. Have you validated the significance and relevance of the problem through external feedback from diverse sources – domain experts, potential users, peer community?
4. Have you checked the feasibility issues?
4a. Do you have any prior experience with this or similar problem or underlying domain?
4b. In case of multidisciplinary and interdisciplinary problems that require engineering R&D wrt some other application domain, are the application domain related aspects of the problem sufficiently documented and understood?
4c. Have you considered the issues of current state of the art?
4d. Have you considered the availability of required resources, domain expertise, and permissions?
5. Have you considered the alternate decomposition strategies for your problem?
5a. What functionality will you consider to allocate to hardware, software, and human element? Justify your partitioning?
6. Is there any novelty in your problem identification and/or formulation?
7. Have you surveyed and critiqued existing approaches for solving this problem?
B. Reflecting upon your Literature Review
Please see the first article in this series – Research Method for Engineering Research Students – Part I: A Checklist for Literature Review
C. Reflecting upon your solution, results, and future scope
1. Did you identify the causes of the problem?
2. Have you attempted to reformulate the problem in terms of known approaches?
3. How did you do constrain the problem?
3a. What is the need to constrain the problem in this manner?
4. What is your solution?
4a. What alternate strategies did you consider? What is your justification selecting your strategy?
4b. Have you considered the alternate decomposition strategies for your solution?
5. What are the main results?
6. How reliable are your results? Have you validated your results and claims?
6a.i Have you validated the results and claims using mathematical/logical reasoning?
6a.ii. Have you checked the theoretical consistency?
6a.iii. Have you considered contradictory theories, if any?
6b.i. Did you collect enough size and variety of data to avoid premature and misleading conclusion from small or ideal case test data?
6b.ii. Are you confident about the accuracy and reliability of your data collection method?
6b.iii. Did you collect data from multiple sources?
6b.iv. Have you considered the data that does not support your findings and conclusions?
7. Is your solution novel in terms of:
7a. Applying a theory/approach known within a specific discipline in a new context within the specific discipline of engineering
7b. Applying a theory/approach known (outside the specific discipline of engineering) within a specific discipline in a new context
7c. Theory enhancement/integration/building about some aspects of engineering: theory, application, or education of engineering activities – planning, design, realization, evaluation, and client interface activities, and also some ubiquitous and over-arching activities? For example, wrt software engineering, for a detailed catalogue of software development related activities, interested readers may refer Software Development Activities: A Catalogue of Technical and Technically Oriented Activities. Other disciplines of engineering also have similar activities.
7d. Providing a new framework or paradigm for thinking about some aspects of engineering (as indicated in ‘c’ above).
8. Compare your solution approach with other existing approaches-
8a. In what way is your solution approach is better than other existing approaches? Consider multiple dimensions –
8a.i. Time/ resource/energy requirement
8a.ii. Wastages and/or pollution
8a.iii. Performance efficiency
8a.iv. Consistency, completeness, generality, scalability, open-endedness
8a.v. Reliability, elegance, usability, maintainability
8a.vi. Safety, health, dignity
8b. What are main shortcomings/limitations/weaknesses of your approach? Consider multiple dimensions –
8b.i. Time/ resource/energy requirement
8b.ii. Wastages and/or pollution
8b.iii. Performance efficiency
8b.iv. Consistency, completeness, generality, scalability, open-endedness
8b.v. Reliability, elegance, usability, maintainability
8b.vi. Safety, health, dignity
9. List the problem statements for future research/projects as extension or improvisation of your research.
9a. What kind of new inter-project collaboration possibilities can you now propose with any of the other ongoing research/projects within your department?
D. Reflecting upon your research method
1. Have you made sure that you have not indulged in intellectual plagiarism by claiming others’ ideas as your ideas?
2. Have you taken due care to avoid common errors in logical and analytical reasoning – misplaced focus, storage limitation, information availability, hypothesis persistence, selective support, limited reviewing, inadequate data, multiple variables, misplaced causality, dealing with complexity, decision and probability? Refer a summary of Metzger’s collation of common logical and analytical errors at Software Debugging: Errors in Logical and Analytical Reasoning – Metzger’s Collation.
3. Did you develop and incorporate a risk management strategy during your research? For example, wrt software engineering, an overview of risk categorisation schemes is given at Some Software Risk Categorization Schemes. Other disciplines of engineering also have similar categories.
3a. Did you identify, assess, prioritize, minimise, monitor, and control the risks of various categories?
3b. Did you face any un-identified risks? If so, what and why?
3c. Was the impact or likelihood of any category of risk severely misidentified? If so, what and why?
4. What were the main steps in your research methods?
4a. Did you build some hardware/software as part of your research?
4a.i If yes, did you test your hardware/software properly as per the laid out procedures?
4b. What was the task division among group members, if any?
4c. Did you cooperate/collaborate with any other project group? How?
4d. What did you do to get external feedback from (potential) users and others?
4e. How did your revise your solution in order to respond to the feedback?
4f. What were the main challenges?
4f.i. How did you address these challenges?
4g. What mistakes did you make with respect to your research method?
4g.i. If you were to start again, how would you modify the method?
E. Reflecting upon your report writing
1. In the thesis/report, have you clearly explained the problem domain and research questions, significance of the problem, motivation, critical review of existing literature, theoretical and empirical background, constraints, solution approach, research method, design of experiment, data collection method, data analysis method, results, comparative evaluation with existing approaches, inferences, theoretical contribution, conclusions, and summary.
2. Did you take care not to copy the material from elsewhere without paraphrasing/ quoting and appropriate citation?
3. Is the material structured, presented, and interlinked properly?
3a. Have you structured the text into cohesive and consistent chunks at multiple levels of text hierarchy – chapters/annexures/appendices, sections, subsections, paragraphs?
3a.i. Were tedious details organized in annexures and/or appendices?
3a.ii. Have you labeled long pieces of text, graphics, tables, and equations?
3b. Are the key ideas summarized in a chapter, and also at the end of sections and chapters?
3c. Have you given a glossary of key terms?
3d. Does the thesis/report title and abstract convey the essence of problem and solution approach?
3e. Is the presentation of the material sequenced in the logical order?
3e.i. Were the inferences drawn logical and supportive?
3e.ii. Were floating objects like table and figures collocated with the relevant text?
3f. Have you given backward and forward internal inter-linkages between scattered but closely related material?
3g. Are the table of contents, list of figures, list of tables consistent with the main body of the material?
4. Have you exploited the opportunities of non-textual representations?
4a. Have you exploited the opportunities of graphical representation in the form of figures, tables, charts, concept maps, etc?
4b. Have you considered representation of key ideas as mathematical models?
4c. Are the textual, graphical, and mathematical representations consistent in terms of presentation, terminology, typography and layout?
4d. Have appropriate typographic convention been used for inline and displayed mathematical material?
5. Have you taken care of spelling, grammar, and punctuation?
6. Are the references as per the format, and cited?
F. Reflecting upon your learning
1. What new things did you learn through your engagement in this research/project?
2. What technical and other professional competencies have you been able to strengthen by your engagement in this research/project? How?
This checklist is an outcome of my long engagements in teaching, software development, research, project management, research review, supervision of hundreds of student projects, and teaching Research Methodology to UG and PG students. Many valuable insights have also developed as a result of enriching experiences of leading the overall planning, monitoring, and evaluation of thousands of major/minor student projects under the supervision of 40+ colleagues at the department of CSE & IT at JIIT since 2004.
I am highly indebted to Dr. Mukul K. Sinha, my guru for the last 23 years, to have helped me experience most of the elements in this checklist. I thank Harkesh Singh Dagar, for numerous discussions on various issues related to engineering in the last seven years. I am also thankful to all my colleagues of CSE & IT department at JIIT, especially the BTech final year project coordination team of Manish K. Thakur, Sangeeta Mittal, and Chetna Gupta, for their efforts and support in collective administration the monitoring of hundreds of final year student projects every semester through a multi-stage structured evaluation.
I invite the learned community of researchers and professional engineers to suggest further improvements to this checklist. Useful suggestions and comments received from several experts after the initial posting on 15th April, are being incorporated to further update this checklist. The suggestions given by Dr. Vasudeva Verma (IIITH) wrt intellectual plagiarism and Prof. Thomas Marlow (Seton Hall University, USA) regarding risk management are valuable additions to the original list. I also strongly recommend the readers to read the thoughtful comments of Prof. Rao Vemuri, UC Davis.
You may also like to see: