| dc.description.abstract | At the onset of the ALICE RAP project, the following objectives, description of work, and main tasks were agreed for Work Package 21: 1. To evaluate the overall functioning of the collaborative research project, using a state-of-art systems model of partnership functioning. 2. To document the interactions and linkages between project inputs, throughputs and outputs as the project unfolds over five years. 3. To facilitate structured discussions involving all partners, to be held at the end of every project meeting, to identify ways to optimise partnership functioning. 4. To produce a report about what is learned about optimising partnership functioning, to benefit future European research collaborations. The frameworks for the evaluation are the Quality Criteria of Transdisciplinary Research (QCTR) and the Bergen Model of Collaborative Functioning (BMCF), an established systems model with five elements: (1) Inputs including projects aims, partner financial, intellectual and technical resources; (2) Production processes (work directly related to meeting objectives) including positive and negative interaction amongst inputs, and positive and negative aspects of leadership, communication, and roles/structure; (3) Maintenance processes (work directly related to keeping the partnership functioning effectively); (4) Outputs (e.g., delivery on objectives) including additive, synergistic and antagonistic (negative) outputs, and (5) Feedback processes whereby outputs generated underway affect continuing input and throughput processes. These are the main Work package 21 tasks as agreed at the onset of the ALICE RAP Project; subsequent modifications are noted [in brackets]. Task 1: During the first three months of the project, at the half way point and again near the end of the project, all willing WP leaders and key administrative personnel will participate in 15-25 minute recorded, structured telephone interviews. The interview guide will include brief questions related to all five evaluation elements, with greater detail given to the element(s) having the most relevance at the various phases of the project. In others words, element 1 will be emphasised at the first round of interviews and elements 4 and 5 will be emphasised during the last round of interviews. [The planned first round of telephone interviews was completed, while the 2nd and 3rd rounds were substituted by self-report questionnaires completed either at partner meetings or online. This change was made based on preferences expressed by the Work Package Leaders, and agreed to by ALICE RAP leadership and the European Commission’s Project Officer.] Task 2: Following each round of interviews, selected recorded material will be transcribed and analysed and report will be generated and circulated to all partners prior to the next project meeting. The reports will be structured to fully protect the confidentiality of interviewees. The three reports will be synthesised into a final report following the third round of interviews. The reports will be used by the WP leaders to illuminate ways to improve the functioning of the project. [This task was completed with data generated by questionnaires, in addition to telephone interview data.] Task 3: At the conclusion of each project meeting, WP 21 will facilitate a structured discussion (recorded, guided by a structured interview guide) that will briefly address each of the five elements listed above. However, at each meeting, one of the elements will be intensively in focus, corresponding to the progression of the project. In other words, element 1 will be the focus of the discussion at the first meeting, elements 2 and 3 will be the main focus of the second meeting, elements 4 and five will be the main focus of the third meeting, and so on. [All the planned project meeting discussions took place, but not all were recorded due to practical limitations. When recording was not feasible, self-report questionnaires were used to gather needed data. The discussions were augmented by online surveys.] Task 4: After each project meeting, WP 21 will transcribe recordings and produce an analysis and report to be circulated to all partners. The reports will be used by the WP leaders to illuminate ways to improve the functioning of the project. [Following each project meeting, data analysis and reporting were done as planned, but the data came not only from recordings; qualitative and quantitative data were also used, collected by self-report questionnaire (both paper-and-pencil and online).] Task 5: As an ongoing task, WP 21 will produce a running analysis using all data as they become available, using the BMCF as the evaluation framework. This will result in a final evaluation report, publication of results in the international peer review literature and presentations at scholarly conferences. To meet the goals and accomplish the evaluation tasks, a range of qualitative and quantitative methods was used in a participatory/exploratory sequential mixed methods study design. The data sources were evaluations following ALICE RAP Partner meetings, data from several computer-mediated communication sessions at ALICE RAP partner meetings, self-report questionnaire data gathered at partner meetings and via the internet, individual telephone interviews with Work Package and Area leaders, and members of the Media and Communications Group and the Global Science Group, and project documents. This is a ‘dynamic’ design including a series of sub-studies, conducted over the entire project period, with each new sub-study’s aim and design based on the accumulating knowledge. The entire Area 7 team participated in decisions about the aims and designs of the sub-studies, and all Work Package Leaders had the opportunity to provide input. The ALICE RAP official list of project participants is the population from which evaluation participants were drawn. The list was modified over the course of the study as some participants left the project and other joined. The current list was used at each stage of the evaluation. In some instances the entire list was used and the sample was the population. In other instances the subset of participants attending a partner meeting and present at the time of evaluation composed the study sample. At yet other times a purposive sample was drawn, as was the case for the collection of telephone interview data. The details of participation are given in the descriptions of the Sub-studies, later in the Methods section. Data collection Post-meeting evaluations At the conclusion of each of the annual partners’ meetings, a meeting evaluation questionnaire was distributed to all participants in attendance. These data were analysed and fed back to ALICE RAP leadership to assist them in planning the next meeting. As the meeting evaluations had mostly to do with the practical aspects of conducting meetings, the results of these evaluations are not included in this report. Computer-mediated Communication (CMC) participatory evaluation When actors from disparate backgrounds and viewpoints gather to formulate and implement solutions, as in ALICE RAP, achieving effective communication is a special challenge. Computer-mediated communication (CMC) is often helpful in this regard, and it was used at some partner meetings to foster general discussions. A CMC innovation addressed by this evaluation process was to use scientific methods to analyse, interpret and feedback CMC data and results to ALICE RAP management. Interviews Following the ALICE RAP meeting in Barcelona in May, 2011, telephone interviews were undertaken with Work Package leaders and the Chairs of the Media and Communications Group and the Global Science Group. The interview guide is in Appendix A. Self-report questionnaires Self-report questionnaires were used in several Sub-studies, distributed either during partner meetings, or via the internet. Documents All official ALICE RAP documents were available for analysis, including the original application and the evaluation that led to the funding of the project. These were used, but were data sources of relatively minor importance. Evaluation Sub-studies aims and main findings Mission and Values Sub-study This sub-study, undertaken early in the project’s development, aimed to initiate a wide- ranging partners’ discussion on how the ALICE RAP partnership should function, ways to make it work smoothly and efficiently, the project’s mission, and the degree to which there is a diversity of core values among the ALICE RAP scientists. The issue of partnership functioning was a focus in all the Sub-studies, while the issues of mission and values were the particular foci of this Sub-study. The main result with regard to management practices, rather expectedly, was that the participants wish for a leadership style that balances firmness and clear direction on the one hand, with openness and a consultative style on the other hand. Perhaps less expected was the degree to which participants focussed on challenges related to diversity in the ALICE RAP research team. A warning was sounded about possible diversity-related dominance problems of two types: dominance by one or several disciplines over the others, and dominance of native English language speakers over the others. A finding of particular significance that emerged from the CMC has to do with values. It might have been the case that as scientists, the participants would have focussed on the eternal debate about value-laden versus value-free science. However, the widely held position was that science is value-laden, that values should therefore be illuminated, and that ALICE RAP should make an effort to develop shared values. Considerable attention was also paid to societal values regarding the nature of addiction, and the need to reframe the problem of addiction so as to reduce stigmatisation and marginalisation of people with addictions. However, it is open to question which values actually predominate in ALICE RAP, since the CMC is not the right methodology to study values, attitudes and predispositions. Further study of values, attitudes and predispositions held by ALICE RAP scientists could be useful in understanding the unfolding of the project’s science. Leadership Sub-study A telephone interview survey was used to gain insight about ALICE RAP’s leadership strategy, and its coordination and communication methods. The results of this Sub-study were discussed by the ALICE RAP leadership team, and it was agreed that ALICE RAP should develop Vision and Mission statements. There ensued a round of consultations, leading to the adoption of these statements, which are featured on the home page of the ALICE RAP web site: Vision Statement -- Promote well-being through a synthesis of knowledge to redesign European policy and practice to better address the challenges posed by substance use and addictive behaviours. Mission Statement -- Advance synergy among sciences that address substance use and addictive behaviours, through a five-year programme of European trans-disciplinary research. Science and Policy Dialogue Sub-study The aim of this Sub-study was to address a key goal stated in the ALICE RAP Description of Work: to include “clear and targeted dissemination and valorisation strategies, addressing and involving not only scientific audiences, but also policy-makers, other key actors and stakeholders…” The starting point for the CMC session on policy-science dialogue was the assumption that ALICE RAP has to engage stakeholders, as that is an essential aspect of TDR. The issue, then, is how to involve stakeholders. Scenarios were provided to illustrate possible levels of involvement, ranging from simply engaging stakeholders in dialogue, to acting together and taking responsibility together. The findings suggested a differentiated strategy for stakeholder involvement, tailored to particular aspects of addiction, to country or community levels of interest, and to the different stages of the project as it progresses. This could be developed as an element in ALICE RAP’s Communication Strategy, in collaboration with Areas and Work Packages. The essence of a Stakeholder Communication Strategy with this tone would be its emphasis on ‘communication with’ stakeholders, rather than ‘communication to’ stakeholders. Research Orientation Sub-study This sub-study was undertaken to provide ALICE RAP with insight about its scientists’ orientations to research collaboration after two years of the project, and again at the conclusion of the project. Those in ALICE RAP with the greatest expressed proclivity for inter/transdisciplinary research are older men in the biomedical sciences and younger women in the social sciences. Yet, as Tables 6-8 show, respondents in this study generally expressed strong proclivity for inter/transdisciplinary research, regardless of scientific background, sex and age. This quite likely is due more to a selection effect than to an adaptation effect. The ALICE RAP leadership intended to recruit scientists with the needed scientific expertise and with proclivity for transdisciplinarity. While the experience in ALICE RAP between 2013 and 2015 may have affected scientists’ attitudes towards transdisciplinary working both favourably and unfavourably, we conclude that at near the mid-point of ALICE RAP (2013), the partners’ orientations to collaboration was already consistent with the transdisciplinary Mission of the Project. The 2013 and the 2015 rounds of data collection yielded data that should be compared with substantial caution, since the sample frames (partners in attendance at the 2013 Barcelona and the 2015 Lisbon meetings) were different, even if drawn from approximately the same population (participants listed on ALICE RAP lists). Notwithstanding this caution, we conclude that amongst partners attending the two meetings and participating in the Research Orientation Scale data collection, the mean Research Orientation Scale score was similar, at about 28 in 2013 and in 2015. Experience in ALICE RAP seems to have neither decreased nor increased partners’ proclivity for transdisciplinary research collaboration. Collaboration Sub-study The aim of this Sub-study was to map collaborative contacts within and between ALICE RAP Areas and Work Packages. ALICE RAP is conceived as a transdisciplinary research (TDR) project. Great effort has been made to organise the project in ways that encourage the partners to cross administrative boundaries (Areas, Work Packages). Substantial parts of partner meetings have been devoted to discussions and consultations having the aim to stimulate the TDR atmosphere. The data suggest that a great deal of inter-Area and inter- Work Package collaboration happened in the period before data collection. Yet, the pattern was of substantial heterogeneity, with Area 3 seeming to be a beehive of inter-Area collaboration, and Areas 2 and 5 beehives of intra-Area collaboration. In their interpretations of the instructions for completing the survey, there is reason to believe that respondents pondered personal instances of collaboration. The Futures Sub-study The Futures Sub-study’s aim was to use the Bergen Model of Collaborative Functioning, to document partners’ perceptions of ALICE RAP collaborative processes and the degree to which they experienced synergy in the project. Data were collected using the online ALICE RAP Futures Questionnaire in 2014, with repeated data collection using the same questionnaire in 2015. In the measurement of ALICE RAP synergy, the data show consistently that a large majority of respondents perceived the project to have synergistic effects. There were interesting sex differences. Males were more than twice as likely as females to have experienced synergy frequently, but males were far more likely to be critical of the scientific quality of ALICE RAP. There was more moderate evidence for synergy in the other synergy construct – continuity -- as expressed by respondents’ intentions to participate in ALICE RAP- related activities after the project is completed. In Futures I, the continuity scale was close to normally distributed. In Futures II, the majority of respondents indicted intentions to participate by endorsing and supporting activities, or playing a modest role. For the more active types of future participation, women were more likely than were men to indicate the possibility of extensive futures participation. This was the case for connecting to networks with like interests, writing research applications, and writing papers. Men, on the other hand, were more likely to intend active participation in more general activities like endorsement and advocacy. Maintaining momentum Sub-study This Sub-study focused on the outputs element of the BMCF (Figure 1), aiming to ascertain the degree of synergy regenerated by ALICE RAP (indicated by partners’ intentions to keep aspects of the project underway after the EU funding period). Two main themes arose, research and networks. Five specific ideas were identified related to possible future social science research, seemingly with an emphasis on descriptive aspects (not intervention research). The five topics shown are listed in arbitrary order. The second research stream that received attention was related to intervention for policy enhancement, for which three highly interrelated topics were identified – again, these are listed in Figure 1 in arbitrary order. Together, eight social science research topics received attention, while no bio-medical topics were suggested. The other emergent theme was networks, loosely divided into ideas related to less and more formal networks. Five network ideas received attention, with various purposes: the gathering of vested interests; advocacy; building transdisciplinary science capacity across addiction arenas and micro to macro levels of systems; research. The findings suggest at least some degree of interest in continued ALICE RAP work. Interestingly, no participant suggested just letting ALICE RAP fade away. | en_US |
