impact

Open Access Champion 2013 – Open Journal Project

dannykingsley Leave a comment

To celebrate Open Access Week 2013, the Australian Open Access Support Group is recognising two ‘Open Access Champions’ – an individual and an organisation.

The Open Access Champion 2013 – Organisation Category has been awarded to the not-for-profit international development organisation Engineers Without Borders Institute’s Open Journal Project.

Julian O’Shea, the Director of the EWB Institute, who is heading up the project, spoke to Danny Kingsley about what has happened in the three months since the AOASG featured a story about the project in July.

Award

In bestowing the Open Access Champion 2013 award the AOASG is recognising the excellent work that Engineers Without Borders (EWB) has been doing to open access to research. The Open Journal Project publishes the Journal of Humanitarian Engineering (JHE)  which is not only open access, but provides easy to understand interpretations of the technical papers, translated into the local language(s) and addresses other issues of accessibility.

This project exemplifies a true commitment to open access in its pure form. The Project has considered all aspects of accessibility, well beyond the first step of simply providing access to the original research. In addition, the stated intention of the Project to act as a stimulus for others to follow the example set further increases the already impressive impact of the Project.

Increasing academic engagement

Since the project launched there has been a great deal of interest, explained Julian. “What is really pleasing is the level of academic interest,” he said.

The EWB have been talking to practitioners in the area of WASH (Water Sanitation and Hygiene Wash) research. Despite the focus on development, much of this research is still published in closed journals.

“This is probably one of the most important groups for development practice”, he explained. “But even people who would like to be out in the open publish in closed journals”.

The group has been offering them the opportunity to publish in the JHE. And to increase interest, the EWB have started discussions with organisers of the international WASH conference being held next year in Australia. The JHE is intending to publish a WASH-themed issue at the same time.

“The conference brings together leading academics and practitioners in WASH, so we will be using this as a platform to showcase our relatively young journal,” said Julian. “We want to build a network of the WASH and to formalise this as another opportunity to publish”.

The project is also looking to engage researchers in many of the countries where the humanitarian work is being conducted.  “We are looking at opportunities for them to publish”, said Julian.

The Project has considered the issue of different academic standards in different countries. “If you apply a fixed standard to your journal you are ruling out many parts of the world that do not have that experience,” he said. “English is the primary language of our journal”.

Many people who work in the area of humanitarian engineering may not have higher education degrees and may not have done formal academic writing. “If English is their second language we are happy to provide people in international research who can support and collaborate on the work”, said Julian. This is an open invitation, he explained, noting the journal is still a peer-reviewed journal committed to academic and technical rigour. “Their ideas have no less merit to their work but they do face academic hurdles”.

Sharing the message in the community

The target audience for the journal’s articles are practitioners in the field in developing countries. “What we have found is the idea that something is on the web so therefore it can be accessed is a bit of a stretch”, explained Julian. The EWB have a program which is a design challenge for Australian students – to come up with research outcomes that can be more readily understood.

“They make plain language guides rather than just the 10 page article,” he said. “So part of the process is we have readable understandable summaries of the research.”

The EWB are also planning to start spreading the word in person. “As of next year we will be disseminating these outcomes in country,” he said. “Because we have a network of people in country, with our local partners we will be holding local workshops targeting the groups we know about and share in person about what some of these outcomes have been”.

The group hopes to run some workshops in Nepal, one on water and one on energy. This will be using a human connection. “We can’t underestimate that,” said Julian. “Sharing in person makes it a lot more real. We will be working on the networks within their communities that spread the word”.

This AOASG award is not the only recognition the project has had. It was shortlisted for the World Youth Summit awards which recognises ICT and technology solutions addressing poverty alleviation. “We were the only program nominated from Australia”, explained Julian.

Future plans

The Project continues to innovate, with a summer project planned. “A student will develop a technology solution that converts an academic paper’s pdf or preprint into a low bandwidth version”, explained Julian.

The EWB are hoping to be able to automate the conversion, to allow the process to be scaled up across whole journals. “We expect to have a prototype by early next year that will enable editors and publishers to have a version that is low bandwidth friendly,” he said. “So these can be accessed in the developing world where downloading a pdf can be a technical challenge – this will give practitioners more scope to download. The outcome of the project will be open source so it can be shared.

Julian and the EWB team are brimming with ideas, but time is an issue as the Open Journal Project is just one of the projects currently running. Julian would like to develop a resource pack for editors and publishers to help them with these access issues. “It could help them with the change from one type of licensing to other”, he said.  “That would make it more of a movement.”

Assessing research impact: AOASG submission

dannykingsley 2 Comments

Measuring the impact of research has been on and off the government’s agenda for many years now. Originally part of the Research Quality Framework, impact was removed from the 5880919002_b92743b247_mExcellence in Research for Australia during its trial in 2009.

Due to its increasing relevance, measuring impact was trialled again in 2012 by the Australian Technology Network and the final report from this study: “Excellence in innovation: Research impacting our nation’s future – assessing the benefits” was released in November 2012.

Plans to assess impact

The Department of Innovation is currently exploring options for the design and development of an impact assessment program. It intends to pilot this in 2014.

As part of this process, the Department released a Discussion Paper in July 2013– “Assessing the wider benefits arising from university-based research”

The Paper seeks the “views of interested parties regarding a future assessment of the benefits arising from university-based research”.

Before research administrators throw their hands up at yet another assessment program, the Discussion Paper does recognise the overwhelming compliance burden on universities and the need to simplify this burden. . The Preamble states that plans include “scaling back and streamlining a number of current data collection and analysis exercises”.

Overall, the Government believes that a research benefit assessment will:

  1. demonstrate the public benefits attributable to university-based research;
  2. identify the successful pathways to benefit;
  3. support the development of a culture and practices within universities that encourage and value research collaboration and engagement; and
  4. further develop the evidence base upon which to facilitate future engagement between the research sector and research users, as well as future policy and strategy.

Submission from AOASG

AOASG prepared a submission in response to the discussion paper proposing that open access should be a measurable for assessing impact and that some reward should be associated with making work freely and openly available.

All submissions will be made available to the public on the Dept of Innovation website. In anticipation, the AOASG submission is copied below.

Response to principles from the paper

NOTE: AOASG chose to only respond to Principles 1, 3 and 5.

Principle 1 – Provide useful information to universities

Principle 1 is to be applauded. It is sensible and practical to marry the types of data required with the types of data the Universities are already producing. This will minimise the burden on Universities in aggregating data and producing reports.

Open access repositories in Australian universities are developed in a finite set of software with common underlying code – OAI-PMH. This allows for aggregation and harvesting across multiple platforms. Such repositories usually maintain statistics about individual works, such as the number of downloads and places where these downloads have originated.

Prior to developing or recommending any specific data for reporting on impact, we suggest that a survey be conducted of university libraries to gather information on the type of data collection methods already in place within open access repositories. This also has the benefit of supporting Principle 2 – Minimise administrative burden.

Principle 3 – Encourage research engagement and collaboration, and research that benefits the nation

Principle 3 notes that this assessment should encourage and assist universities to “better recognise and reward (for example in recruitment and promotion exercises) the contribution of academics to engagement and collaborative activities”. A fundamental component of this assessment is an academic’s involvement in open access and their approach to making research freely available.

Many Australian researchers share their work with the broader community by placing a copy of it in their institutional repository, or in a subject-based repository such as PubMed Central, SSRN, arXiv, or RePEc. The ARC & NHMRC open access policies are likely to encourage more researchers to follow this trend. However, currently there is no aggregated data, and little individual data, on the extent to which Australian researchers are making their own work available. In addition, some researchers also widen the accessibility of research outputs by working as editors, publishers and reviewers for open access journals published out of Australian universities. A definitive list is currently being developed of these journals however this list does not indicate the level and extent of open access activity in the country.  The efforts of academics and researchers to share research openly is currently not measured nor rewarded through any promotion or funding incentives.

Principle 5 – Collect and assess at the institution level, with some granularity by discipline

Principle 5 – is a good suggestion given that some types of research will naturally have a wider impact than others. Impact will also vary over time with some research outputs producing impact after a considerable time and others making immediate significant impact. It is more challenging to articulate the benefit to wider society of research in, say, pure inorganic chemistry than, for example, forestry. When considering the need to granulate the information available, the benefit of using data from open access repositories as suggested above, is the metadata for each record contains information about the author, subjects and clearly the institution.

Response to methodological considerations

What considerations should guide the inclusion of metrics within the assessment?

It has become clear that the established measurement systems such as the Journal Impact Factor (JIF) can be affected by those who seek to manipulate the outcomes. A recent clear example this is occurring is the JIF decision to suppress a larger than usual number of titles this year due to “anomalous citation patterns resulting in a significant distortion of the Journal Impact Factor”. Any reliance on metrics as a measure of quality and/or use of research needs to consider attempts to manipulate new measures as a potential outcome. One way of minimising data manipulation is to use a mix of qualitative and quantitative measures.

What information do universities currently collect that might form the basis for research engagement metrics?

As noted above in section 2 all Australian universities and CSIRO have an open access institutional repository. Such repositories usually collect information on the research that is available, how often it has been downloaded and where the interest has originated.

What metrics are currently available (or could be developed) that would help to reveal other pathways to research benefit?

The act of making a work open access creates a pathway to research benefit. Open access increases the potential impact of the work because it ensures the work can be accessed, applied or built upon by other researchers, industry, practitioners and the public. On this basis, we propose that the act of making research publicly available is a fundamental metric of assessing research benefit. This would also support and endorse the open access policies of the ARC & NHMRC. The metric could be twofold – at the individual researcher level (in terms of promotion) and/or at the institutional level.

While in some cases publisher copyright conditions will prevent work being made available, having an appropriate version of the work deposited in a repository with a ‘Request copy’ button facilitating access could be considered ‘making it available’ for this purpose.

Repositories capture article download information at the level of the individual article. This data could also be used as a metric of a pathway to research benefit. There is a proven link between making work open access and citations. The general collection of download statistics would be only one of several measures that can be aggregated to demonstrate interest in an use of research (see the next point).

In addition to ERA, NSRC, GDS, AusPat and HERDC data, are there other existing data collections that may be of relevance?

Recently there has been a move to develop a series of metrics that assess the value of individual articles rather than placing value on the journal or publication in which the article appeared. These article level metrics offer real time feedback about the way a research article is being used.

One example is the metrics page provided for a published article that lists the number of HTML views, PDF downloads, XML downloads as well as the number of citations and where the article has been shared on social media, such as through Twitter. There are however, many other examples of article level metrics already in existence. Examples include: Altmetric, ImpactStory, Plum Analytics & PLOS ALM. Discussion of research on social media sites indicates a level of impact beyond the confines of the scholarly publication system, with the added benefit of being instantly and easily quantifiable. The timeliness and convenience of these metrics addresses the need for “current information on the prospect of benefits from research” as identified in the Discussion Paper.

What are the challenges of using these data collections to assess research engagement?

It will be necessary to determine which sets of article level metrics are the most appropriate for specific purpose. There may be a need for some aggregation to correlate several sets of metrics about the same item.

Response to ‘other comments’ section

We have two suggestions for additions to Appendix A – “Examples of possible metrics”.

An additional Research engagement mechanism could be “Provision of research outputs in a publicly and freely available outlet”. The Measure could be “The percentage of research that is freely and publicly available within 6 months of publication”, and the Source would be “Institutional repositories, subject based repositories or open access publications”.

Currently one of the research engagement mechanisms listed is “Research engagement via online publications”. The measure suggested is “Unique article views per author” and the source is “Websites such as The Conversation”. We are in full support of this suggestion. The Conversation is an opportunity for researchers to discuss their work in accessible language and the author dashboard for The Conversation provides comprehensive metrics about readership.

However we suggest there are other metrics within the classification of ‘online publications’. Open access repositories can provide metrics on unique article views per author. We therefore suggest an additional source being “Institutional repositories, and other article level metrics”.

Accessibility is more than making the paper OA

dannykingsley 4 Comments

Journals

Proponents of open access generally agree that there are many benefits to open access, but discussions about the processes involved in achieving open access often stop at making the published research available. But what happens when the issues of accessibility are considered?

A remarkable project is underway in Australia, spearheaded by the Australian chapter of a not-for-profit international development organisation, Engineers Without Borders (EWB). The Open Journal Project aims to explore and promote techniques to make academic information genuinely open and accessible – with a focus on groups that are often excluded from access to this type of information.

EWB is a volunteer organisation, sending volunteers overseas to a local non government organisation to work on the ground on a project. The Open Journal Project considers the needs of individuals and practitioners in other countries.

“The Project doesn’t finish the day you press publish – that’s when it starts,” explained Julian O’Shea, who is the Director of the EWB Institute, the education, research and training section of EWB, and is heading up the Project.

“We are thinking about what we can do to make the work more accessible.”

The EWB Institute is based in Melbourne, and is publishing a peer-reviewed journal as a pilot and case study in their work. The Journal of Humanitarian Engineering (JHE) is piloting innovations in open access, including multi-language access, developing country access, low-bandwidth websites, and disability-accessible content.

“We want to pilot innovations and share our experiences with doing this,” explained Julian. “We want to work out what is world’s best practice, do it and live it and show it is not too hard.”

The problem

The EWB has noticed that practitioners overseas are under-served by the current publishing process. As an example, Julian stated that the leading university in Cambodia does not have access to the largest database in the field of engineering.

The idea for the focus of the journal began because the group saw there were very few that focused on experience, drawing on outcomes from developments and disseminating that information.

“The aim of the journal is not to be published or cited, but to provide outcomes in communities,” explained Julian. “This is different to other research organisations as a metric of success. It gives us a different angle or lens.”

The group wanted to encourage this as a field of research in academia. They were not sure what level of interest there would be in the journal because from a purely technical point of view they are not publishing innovative technologies. Rather, the focus is on new ways of applying this technology.

“We have been surprised and pleased that the journal has been really positively responded to,” said Julian.

Open access

The journal is published open access, with no cost to the author or to the reader. It uses an open source program called Open Journal Systems to run the administration of the peer review and publication. All papers in the journal are available under a CC-BY 3.0 license.

“We have had no negative feedback at all from people wanting to publish in the journal,” said Julian. “People doing this kind of research don’t have any issue with making their work freely available.”

Accessibility – language issues

Academic papers can be difficult to read even for people within a field. They can become impenetrable to researchers in parallel fields. This problem is further exacerbated in an international environment, working with practitioners on the ground who may not have any tertiary education.

There are several issues with language. The first is the problem of making the technical reports understandable to the lay person. Often the papers in this area are very technical, including many equations, and can reach 300 pages.

To solve this problem authors in the JHE are required to submit a two page plain summary about the paper with the formal paper. This means a project manager on the ground can make a decision about applying the technology or approach and then pass the full paper on to the technical manager.

But many of these projects are in countries where English is not the primary language. The Project addresses this by making the reports available in the language of the country it is targeted towards. The Project translates the plain language guides into both the local language of most importance and into other general languages.

The Project called on goodwill to obtain the translations. They sent articles out to the world, asking for volunteers to translate the papers. This had a good response from universities, companies or simply people to help out on the website.

The Project now has an approved translator list. The first time an article is translated it is sent to a native speaker to approve it and once this is done the translator can go onto the list. The quality of translations has been very high, said Julian, with only one that had to be sent back.

To date the plain language summaries have been translated into Indonesian, Italian, Portuguese, Russian, Hindi, Chinese, Spanish, Danish, Khmer and French. The number of languages is growing.

Accessibility – distribution

Another consideration is bandwidth. In many countries the internet connection is through a mobile telephone which prevents the download of large documents. The Project decided to produce the journal on a low bandwidth, and this opened up new issues.

“Generally the journal system distributes through pdfs,” explained Julian. “The problem is it is all or nothing – if the download cuts out at 90% you get nothing.” So the Project looked at releasing html versions of the papers. This has reduced the size of the website to 4.3KB and the journal articles are about 18KB. “We can put about 80 journals onto a floppy disk,” he said.

The Project also has plans to further improve the distribution to remote areas. “We haven’t done it yet but we will have a system that says ‘here’s a postcard, send it to us and we will send the paper to you by donkey’”, said Julian.

Accessibility – inclusion

With a philosophy of sharing research, it was important to the project to provide versions of the papers in an accessible format for people with disabilities.

The choice of publishing html versions of papers assists people with vision impairment, as they translate better using text-to-talk programs than pdfs.  In addition, the project is being proactive about embedding helpful metadata within the document such as describing images.

The Project has used the guidelines for Vision Australia to release a large print edition of the papers. “The first one took about couple of minutes – after that it was very simple,” said Julian. “That is what we are trying to show in this project, to meet a need for some people can be solved in literally two minutes.” The team has also produced Braille editions of the plain language guides.

Future plans

The project hopes to share their experience and inspire others. “We are doing this through the case study approach,” said Julian.  “This is my goal – to be able to communicate better. I am an author – what can I do? I am a publisher – what can I do?”

The Open Journal Project is hoping to formally launch later this year. Meanwhile, Volume 2 issue 1 is about to be released.

Twitter handle – @OpenJournal

Dr Danny Kingsley
Executive Officer
Australian Open Access Support Group