Skip to main content

Researcher's Toolkit: C2 - Research management

This researcher's toolkit has been developed to offer you practical advice and suggestions to help you design, carry out and write up a research project.

Research management

In academia, emphasis is no longer solely on the findings that are generated from research, but also the research data itself.

Applications for public research funding must include a data management plan. If the application is successful the contract is likely to include stipulations on how you store, archive and share the research data.

Aside from meeting funder requirements in a competitive environment, there are many benefits to managing your research data effectively: such as maximising the impact of your research, increasing citation rates, increased transparency and accountability, economic benefits and the increased potential for collaborations.

Planning your data

Before you start your project, consider what data will be generated and how it will be managed - not just during your project but beyond the end of the project. A formal data management plan is highly recommended - and, for some funders, a requirement at application stage.

Describing your data

Research data is only useful if it can be understood. Make sure you document your data and generate metadata. This will help you and your collaborators understand your own data later in the project - or years down the line - and will help others to understand and potentially reuse your data.

Storing your data

Data will need to be stored safely from the moment it is created, during the lifetime of your project and beyond. Check that you have sufficient project storage and are applying safe practice for data gathered in the field. You may need to select and store data for the longer term. Contact the Research Data team for advice. If you are submitting a project bid, make sure sufficient project and long term storage is budgeted for. During the project storing data minimise the risk of data loss, ensure version control and make it easier for researchers to share data throughout the project. 

Citing and being cited

If you are using data generated by others, ensure you formally cite the data in your publications. Apply the same level of scholarly practice regarding citation of data as you would with any other type of research output. If you deposit your own research data with a repository service, other researchers will be able to cite your data unambiguously.

Sharing your data

Who else may be interested in your data? Does your funder expect you to make your data available? There are various ways to share your data; a robust way to share is deposit with a repository service. There are subject specific data repositories and OARS, the University of Suffolk repository.

[Based on advice by Leeds University]

"Are you addressing research data management?"
A blog post providing a good starting point on managing research data:

Directions in Research Data Management for UK Universities
Published by Jisc, this looks at a strategy for the next 5 years 

DMPOnline tool
A data management planning tool

Manage your research information quick guide

MIAO – My Individual Assessment of Open Access
A self-assessment tool for researchers

Open access and the research lifecycle: a guide for researchers

Loading ...

Bibliometrics and research impact

[This section is based on guidance by Leeds University]

​Bibliometrics is the measuring influence or impact in the journal literature. It is the quantitative analysis of research literature, based upon citations, and can be used to evaluate the impact on the academic community of a research paper, an individual researcher, a research group or institution, or a journal. Although bibliometrics cannot be used to measure the quality of research, it can be useful for understanding the significance or impact of research.

Bibliometric data are increasingly being used to assess the impact of research e.g. it is included by the UK Higher Education's Research Excellence Framework (REF)

Key metrics include impact factors, h-index and citation counts. 

Bibliometrics - an introduction (online tutorial)

Why use bibliometrics?

Bibliometrics can be used to demonstrate the impact of your own research or that of your research group. It is also useful when applying for tenure, promotion or grants and on your CV a sit allows you to demonstrate the value of your research to your institution
by showing return on investment to funding bodies, industry and the general public.

It can be used to identify areas of research strength and weakness to inform future research priorities for an institution.

Bibiometrics can also be used to identify top performing journals in a subject area, in order to

  • decide where to publish
  • learn more about a subject area
  • identify emerging areas of research.

It can also help identify top researchers in a subject area, in order to

  • locate potential collaborators or competitors
  • learn more about a subject area
  • inform a recruitment process.

Quality

  • A large number of citations does not automatically mean that a work is of high quality. A work may be heavily cited because other authors are refuting its research.
  • Bibliometrics does not measure quality. It is important to put the data in context using a combination of metrics and other qualitative information where appropriate, such as funding received, number of patents, awards granted and qualitative measures such as peer review when evaluating quality of work.

Discipline Variation

  • Citations patterns differ greatly between disciplines so direct comparisons cannot be made.
  • Bibliometrics predominantly focuses on journal article citations, but some disciplines such as the arts, humanities and social sciences publish research in different types of publication.
  • Different fields of research publish at different rates. For example, in biomedicine, there is generally a much stronger culture of publishing in journals and citing the work of peers than in engineering which makes more use of conference papers.

Database Variation

  • The bibliometric databases do not cover all research areas and do not index all publications e.g.conference proceedings or reports are often poorly covered.
  • Results will vary depending on the database you use, so do not rely on just one.

Bias and Discrepancies

  • Citation bias. People may inappropriately cite their own work, their colleagues, or work from the journals in which they publish. A number of bibliometric tools allow you to exclude self-citations.
  • Experienced researchers have an advantage over early career researchers as they will have produced more outputs over a period of time and so will have more citations.
  • There is a bias towards English language material.
  • Time is needed before a meaningful citation analysis can be made, so new journals tend to fare badly.
  • Bibliographic tools cannot always reliably differentiate between researchers who share the same surname and initials, meaning that citation counts may be inflated. Researchers can use unique researcher IDs to reduce the risk of this. Find out more about using author identification systems.

[Guidance by Leeds University]

The journal impact factor (JIF) is the importance of a particular journal and is is based on the average number of citations received per paper published in that journal in the preceding 2 years. JIF averages the number of citations received by a journal in a given year and is dependent on how often articles from that journal have been cited in other journals. This might be a measure of quality or prestige in a particular field.

For the Journal Citations Reports tool (JCR) the Impact Factor is the average number of times articles from the journal published in the past two years have been cited in the JCR year. So a high impact factor is reliant on an article being cited many times in the journals indexed in JCR. If your impact factor is 2.5 then articles from the last two years have been cited two and a half times by other academic journals.

TIP!
Only factoring the last two years enables the factor to be more relevant as it disregards articles which are commonly cited e.g. as background or classic texts.

Issues to consider

Journal impact factors can help you determine the importance of the article and of the journal itself. However this is only part of the picture:

  • Not all journals are included in the Journal Citation Reports tool.
  • An article may be cited as part of a counter argument or to illustrate poor practice
  • A journal title may be too new to have had time to build up a citation reputation.
  • There may be a spike due to an exceptionally well cited article.
  • Different subject areas will vary, so it may be misleading to compare impact factors between very different academic subjects.
  • For some areas it is relatively easy to name top titles e.g. Nature, New Scientist, BMJ, Lancet in medical topics, but this becomes less easy with e.g. arts topics.
  • There can be valid reasons for publishing in a trade or practitioner journal.
  • You may wish to publish more quickly than is possible with a high impact factor title.
  • Results may include different authors with similar surnames, so do check clearly and remember to search with all first name initials.

Journal Citation Reports is a comprehensive and unique resource that allows you to evaluate and compare journals using citation data drawn from over 11,000 scholarly and technical journals from more than 3,300 publishers in over 80 countries. It is widely used to identify the top journals in a field based on citation behaviour. The journals are ranked in JCR using the well-known and accepted journal measure JIF (Journal Impact Factor). JIF is often used by individual researchers to identify the best titles in which to publish and gain recognition and potential impact.  Citation varies considerably between disciplines and therefore disciplines cannot be compared.

. Journal Citation Reports can show you the:

  • Journal Impact Factor (JIF)
  • Most frequently cited journals in a field
  • Highest impact journals in a field
  • Largest journals in a field

Citation and article counts are important indicators of how frequently current researchers are using individual journals. By tabulating and aggregating citation and article counts, JCR offers a unique perspective for journal evaluation and comparison.

Learn more about Journal ranking and analysis (online tutorial by Leeds University)

Learn more about Evaluating the importance of journals in your subject area (PDF)

Tracking your research impact (online tutorial): this tutorial explains how to use Web of Science and Scopus to track and assess research performance at individual, departmental and institutional level.

The Highly Cited Index (h-index) was developed by Professor Hirsch (University of California).

  • It is an indicator of impact that measures the productivity and impact of a researcher's outputs.
  • It is based on the number of publications as well as the number of citations they have received. 
  • It can help to quantify the output of an individual researcher but is only meaningful when compared within the same academic discipline. H-indices will therefore vary between the different academic disciplines.
  • It is useful because it discounts the disproportionate weight of highly cited articles or articles that have not yet been cited.

The Web of Science help pages describe the H-Index as “The h-index is based on a list of publications ranked in descending order by the Times Cited. The value of h is equal to the number of papers (N) in the list that have N or more citations.”

An author has an H-index of n if they have published n papers, each of which has been cited at least n times.

Example: If 4 papers have been cited at least 4 times you have an H-Index of 4; if 15 papers have been cited at least 15 times you have an H-index of 15.

Web of Science

The main database is the Web of Science via the Web of Knowledge. Web of Science is a citation database of more than 12,000 journals and over 160,000 conference proceedings. Coverage includes science, social science and arts and humanities dating back to 1900. Web of Science covers:

  • Scholarly output
  • Citation counts
  • H-index

Web of Science provides access to citations and abstracts from scholarly journals. 

Searchable databases include: 

  • Science Citation Index Expanded -1970-present
  • Social Sciences Citation Index -1970-present
  • Arts & Humanities Citation Index - 1975-present
  • Conference Proceedings Citation Index- Science - 1990-present
  • Conference Proceedings Citation Index- Social Science and Humanities - 1990-present

When you log on to the Web of Knowledge, be sure to select the Web of Science in order to select the Cited Reference Search option.Only references in articles published in journals indexed by Web of Knowledge are included in the JCR.

Other sources of citation information include:

Google Scholar offers some citation searching, click on the Cited by link under search results.

  • Jstor offers some citation searching in the Citation Tracker.

What are Altmetrics?

  • Altmetrics are based on the number of times an article is shared, downloaded or mentioned on social media, blogs or newspapers. In today's digital world, researchers share materials online, and the wider public can engage with research outputs via a variety of different media such as Twitter, blogs, news reports etc. Often this means impact can be seen more quickly, with research having a broader reach.
  • Attention (shares, retweets) = engagement (analysis, discussion, reviews) = impact (shapes policy, influences, working practices).
  • Altmetrics aims to capture this alternative impact from digital communications. it is an emerging method of analysing the impact of an article in social media.

More about altmetrics

Loading ...

Research types - qualitative, quantitative and mixed methods

Quantitative data is obtained when you measure or count things. For example:

  • The weights of a group of patients (measured in kilograms)
  • The times spent on a task by a group of children (measured in minutes)
  • Bone mineral density of a group of elderly women (measured in g/cm2)
  • Marital status of a group of students (counted by category: married, single, divorced etc)
  • Frequency of incidents of bullying (counted per person over a given time)

 Your dissertation proposal needs to make clear who (or what) you will be measuring or counting. You might be recommended to list your “inclusion criteria” and “exclusion criteria”. For example, a study might be based on children in years 5 and 6 at a local primary school, excluding those who have opted out of religious education.

In practice you might not be able to collect data from everyone who meets your criteria, so you may need to select a sample of people from the larger group. There are a number of different ways of choosing a sample, eg random sampling, quota sampling, etc. The research methods books on your course reading list should explain this in detail and you can find more information about sampling from HealthKnowledge

You should explain clearly how you will implement your sampling method.

You should also give an indication of the size of your sample – how many people (or items) will you include, and why did you come to this decision? Your dissertation supervisor could guide you on what is reasonable here, and this advice could vary from fairly informal (“Try to interview around 20 people if you can”) to very strict (“You need to carry out a formal power calculation with 80% power to establish the appropriate sample size to correctly reject the null hypothesis with 95% confidence”).  Just make sure you take appropriate advice, because a quantitative analysis might not be valid if your sample is too small.

 Having got clear who your sample is, you then need to explain how you will collect the data from it. Will it be direct measurement under controlled conditions (eg height, blood pressure, test score etc) or will it be observation counts (eg did the person walk under the ladder) or will it be via a questionnaire? Questionnaires are very popular, but need care to be done properly, and there are numerous books and websites covering questionnaire design. The student edition of the British Medical Journal published these helpful guidelines about questionnaire design

You might be expected to include a draft questionnaire as part of your dissertation proposal, so do spend time trying to get it right. Try piloting it on unsuspecting volunteers to make sure they can understand it and know how to fill it in properly.

Finally, having collected your quantitative data appropriately, you will have a set of numbers (or frequency counts) that need analysing, and you need to explain how you intend to do this. Your supervisor might be happy with a descriptive approach (working out summary averages or percentages and producing a graph), but you are more likely to need to carry out some inferential statistical analysis to assess the significance of your results. The StatsTutor website is an excellent place to start, and also includes information on using the SPSS software for data analysis. The Khan Academy also has a collection of videos on common statistical methods. 

Qualitative data is obtained when you are interested in experiences, ideas or beliefs. It is not a set of measurements or numbers, but rather a collection of descriptive narratives.

 For example:

  • Transcripts of interviews
  • Recordings of speeches
  • Minutes of meetings
  • Collections of field notes
  • Literary texts
  • And even photographs, pictures or music recordings

 Your dissertation proposal should make clear what sources of qualitative data would be appropriate to provide evidence to support your research aim, and how you propose to acquire those sources. This could be done by primary research (eg carrying out the interviews yourself) or by secondary research (eg searching existing archives).

You should also explain and justify how many sources you intend to use, and confirm that this will give a representative overview of the field of interest.

The analysis of qualitative data is not as simple as calculating averages or other statistics. In fact, because qualitative data is usually text rather than numbers, you will in general not need to do any calculations at all! The standard approach is to identify themes that become apparent as you sift through the sources, and to get a feeling for which themes are most common and if any themes tend to go together, which would indicate associations within the data.

 For example, if you were researching the experiences of factory workers who have suddenly been made redundant, you might expect themes of “financial insecurity”, “missing comradeship” or “loss of self-esteem” to be common themes running through their narratives. (Note: you also need to be alert for themes emerging that you had not anticipated, for example “increased frequency of sexual activity”.)  As you study your sources, you note each incidence of a theme through some kind of “coding” (equivalent to highlighting it in a particular colour), and then summarise the prevalence of the codes.

 There are numerous textbooks outlining qualitative research methods. This paper summarises the key approaches Analysing and presenting qualitative data

Some research designs use a “mix” of quantitative and qualitative data analysis. Typically, a questionnaire might comprise of some quantitative responses (eg How old are you? How many hours a week do you watch TV? Tick your favourite programme from this list.) and some qualitative responses (e.g. Describe in your own words how you think television has changed over the past decade.)

You should make it clear in your research proposal if you intend to adopt this approach. You will need to include a methodology statement that covers both quantitative and qualitative methods.

Loading ...

SPSS

This presentation is an introduction to SPSS, a computer programme for analysing quantitative data.

Also relevant:

Attribution

All images included in this guide are available through Creative Commons licensing CC-BY-2.0