Evaluation of the results and consultancy services

11/09/2020

Introduction

The anonymous survey aimed at facilitating a self-assessment of the proposal as well as the evaluation of the support and tools by the involved team including lessons learned.
The proposal evaluation used the same questions and scoring scale as the call for proposals.
The resulting dataset contains 20 variables including metadata from 9 participants.
Median answering time: 4.1 minutes (Min.: 1.9’; Max: 6.1’).

Collecting opinions from participants concerning the proposal following the official evaluation grid.
Collect feedback data from participants concerning tools, consultancy support, lessons learned and suggestions for improvement.

Qualitative assessment options	Numerical interval
Very good / Very high	4.21 to 5.00
Good / High	3.41 to 4.20
Regular / Average	2.61 to 3.40
Poor / Low	1.81 to 2.60
Very Poor / Very Low	1.00 to 1.80

Mean score by the participants: 44.67 points (49% higher than the min. 30 points).
Minimal score for acceptance: 30 out of 50 points.

Results of a reproducible Bayesian bootstrap re-sampling

We used a re-sampling statistical algorithm to bootstrap a 95% confidence interval for the mean score by participants. This re-samples data sets with replacement 4,000 times registering the draws to generate representative data from smaller data sets.
The histogram in the next slide contains the Bayesian highest density interval (HDI), which indicates a score prediction of 44.7 points for the proposal following the survey results.
This prediction lies within a confidence interval between 43.1 and 46 points at 95% confidence level (minimum score threshold for approval: 30 points).

Visualisation of text-based questions uses simple word clouds (see next slides) filtering automatically for “stop words” (e.g., articles and prepositions).
The size of words in word clouds represents the numbers of time (frequency) that they occur in the dataset.
Each specific frequency is also associated with a colour. Words with same frequency in the dataset have same colours.
Computer-based random sampling has been used to select examples of answers from the dataset in a reproducible way.

O excesso de instrumentos virtuais, por vezes alongam demais os processos, por vezes confundem.
Prazo muito reduzido para elaboração do produto
Extensão das reuniões
Infelizmente, o tempo era curto para podermos aprofundar a construção do projeto.
Pouco tempo disponível para refletir sobre a proposta, principalmente seu público.
Tudo correu muito bem
Tempo muito curto para a elaboração.
O tempo, que infelizmente foi pouco para a elaboração.
As funcionalidades ou interface com Word-MS é diferente e torna o processo mais lento. É uma questao de prática e adaptacao.

As reuniões e debates.
Dinâmica colaborativa
A capacidade técnica dos envolvidos
A dinâmica proposta.
Novos instrumentos tecnológicos.
Trabalho compartilhado, ferramentas online
Disposição da equipe Movimentar, capacidade de síntese e as ferramentas apresentadas.
O aprendizado da construção e as ferramentas utilizadas para essa construção remota.
Gostei de todo o processo, de modo geral

Há resistèncias ao uso de ferramentas digitais
Preciso maior discussão interna para poder levar à consultoria uma proposta mais coesa quando o tempo for curto.
Não tenho comentário
Como construir projetos para editais da UE, trabalhar em grupo remotamente em um desafio grande de ficar horas em frente ao computador.

Instruir a equipe, no início da reunião, sobre funcionalidade do Go to meeting, como o zoom, em especial, que foi um ponto citado por mais de uma participante.
Ter mais tempo para explicar o uso das ferramentas.Apontar de forma mais direta quando algo não estiver coerente na proposta, sem receio de não ser aceito pela organização.

Inputs to the results chain, outputs, and beneficiaries were shared by the local team right in the beginning of the assignment. The team used very well the online collaborative templates of the tables including the results chain, outputs by activity, and beneficiaries. Most importantly, the team followed the recommended sequence of steps in the process. This was very important for the design process and is a best practice.
There was not enough time for the local partner to fill out the provided template for a budget summary. This did not impact negatively the proposal design since it is not a must at the concept note stage. We recommend to start the design process earlier so as to allow for a summary budget, which works as a final ‘reality check’ of output and beneficiary figures.
Assuming that the concept note is accepted, it is recommendable to establish stronger processes for monitoring of third-party support. Currently, the co-applicant requires from grantees traditional project reports but has no system for grantees to track data on direct beneficiaries. Particularly as many are small initiatives, they require a more systematic approach to reliably reporting results. A custom system can allow sub-grant beneficiaries to report on the activity implementation and not only upon project completion. This will better ensure accountability due to reduced dependency on grantees’ estimations while avoiding delays and imprecision in reports to the donor.

We see ourselves as facilitators of participatory processes for the design of the funding application by, and following inputs and ideas from, the local teams. We try to include local knowledge as much as possible for improved relevance, ownership, and better adaptation to local needs and context. That is why we avoid a traditional, top-down expert approach and prefer to build on the inputs suggested by client and partner staff.
Scale up and develop capacities on the use of online collaborative document editing as well as management information systems such as Teamwork Projects, Trello, Asana or Basecamp. They can help to increase productivity and to reduce face-to-face meetings and risks, particularly in the context of the COVID-19 pandemic.

movimentar GmbH | contact@movimentar.eu