Section 1
Introduction to and background for the evaluation

1.1 Evaluation of the Danish Research and Technological Institutes

In "Guidelines for Authorised Technological Service in Denmark" (Danish Agency for Development of Trade and Industry, 1997), the Council for Technological Service has ruled that the authorised technological service institutes (in Danish: GTS institutes) shall undergo an evaluation every three years. The evaluation is a management tool that is used by the board and management of the institutes and by the Council for Technological Service.

The purpose of the evaluations is to determine whether the GTS institutes’ development activities, professional competence and services have the necessary content and quality for the institutes’ present and coming activities to meet the needs of industry and society in both the short term and the long term.

The evaluations are confidential because they are evaluations of independent institutions. Confidentiality also helps ensure the highest degree of openness on the part of the institutes in connection with the evaluations. However, as the institutes have a special status as authorised technological service institutes that can obtain public cofunding in the form of government contract funds, the public has a right to know whether the institutes are managing the funds allocated to them in accordance with their result contract.

Therefore, in 1998, the Council for Technological Service ruled that a summary of each evaluation report should be prepared and published.

In this report, the Danish Agency for Development of Trade and Industry publishes the summaries of the evaluation reports on the four GTS institutes evaluated in 1998. They are: the Danish Hydraulic Institute (DHI), FORCE, Danish Standard Association (DS) and the Danish Institute of Fire Technology (DIFT).

The evaluation – background and method
The evaluation includes an investigation of matters that are of relevance to all the GTS institutes, including whether the strategy plan for 1995-98 has been followed. It can also include an evaluation of¹

• the institute’s technological and scientific level, including the quality of the professional work performed with cofunding from the Council for Technological Service
• fulfilment of the criteria for authorisation
• the relevance of the technological area to industrial policy
• the technological basis of the planned activities with government cofunding and their relevance to industrial policy

Besides the general factors, a number of factors specific to the individual institutes were examined. The institute-specific factors evaluated by the expert panels are described below under each institute.

The evaluation reports are used primarily as the platform for a dialogue between the Council for Technological Service and the individual institute concerning the institute’s strategy plan, which is the basis for the institute’s authorisation as a GTS institute, and its result contract, which is the basis for allocation of government contract funds. However, the evaluation report can also be used as an internal management instrument at the institute and provides the Council for Technological Service with a good indication of the institute’s current situation.

It must be stressed that the Council does not regard the evaluation reports as the ultimate answer to how the institutes should conduct themselves. Each evaluation is carried out by an external expert panel. It is the panel’s impartial but qualified evaluation of the institute’s strongholds and weaknesses and its suggestions concerning possible directions for the institute’s development.

The method used for the evaluations is a so-called "peer review". The evaluation report thus presents the recommendations of an international panel of external experts from relevant sectors of industry and the research world who, in the best collegial spirit, give their evaluation of the institute’s activities and make recommendations for its development in the coming three-year period.

The panels’ evaluations and recommendations must therefore be seen in the light of the panels’ technical/scientific breadth and competence and the personal criteria of the panel members. It should naturally also be remembered that the evaluations themselves are limited in both time and scope for purely practical reasons.

Despite the limitations in the panels’ technical/scientific breadth that are inherent in the composition of every peer review panel, both the institutes and the Council for Technological Service are generally very satisfied with the expert panels’ work.

As a supplement to the evaluation reports the Danish Agency for Development of Trade and Industry carries out horizontal studies of companies’ knowledge of the GTS network².

The preparation of terms of reference for the evaluations, introduction of the panels to the GTS system and the organisational set-up for the evaluation have been coordinated by a team at the Danish Agency for Development of Trade and Industry composed of an evaluation coordinator and the heads of section in charge of the everyday liaison between the Agency and the institutes in question. The panels and the institutes have been able to contact the team at any time, and the institutes’ liaison officers have attended the panel’s meetings as observes on several occasions.

The team had five members:

• Mr. Jens Peter Vittrup (evaluation coordinator)
• Mr. Jens Dinesen (the Agency’s liaison officer for the Danish Hydraulic Institute)
• Mrs. Jette Merete Nøhr (the Agency’s liaison officer for FORCE)
• Mr. Jan Kristiansen (the Agency’s liaison officer for Danish Standard Association)
• Mr. Kjeld Andersen (the Agency’s liaison officer for the Danish Institute of Fire Technology)

Basically, the summaries of the evaluation reports are presented unedited. However, editing has been necessary in some places for reasons of context.

1.2 Authorised technological service in Denmark – in brief³

The Authorised Technological Service Institutes are independent companies that sell technological consulting services to companies and public authorities. Denmark has 14 so-called GTS institutes.

The institutes have the task of building a bridge between research and industry. They must thus promote the development and use of technological, managerial and market knowledge and promote innovation in companies. The GTS institutes also have a special obligation to serve small and medium-sized enterprises. The institutes differ greatly. Some are very R&D-intensive (e.g. the Danish Hydraulic Institute and FORCE), while others focus more on providing companies with service checks, approvals and standards (Danish Institute of Fire Technology and Danish Standard Association).

The GTS institutes sell their services on normal commercial terms in Denmark and internationally. They are at the same time part of Denmark’s research, technology and innovation system, which comprises institutions carrying out research and contributing to technological development. The institutes work closely together with the Council for Technological Service and the Danish Agency for Development of Trade and Industry on technology-based industrial development with a view to improving Denmark’s international competitiveness.

A GTS institute is an independent, non-profit institution that has been accepted by the Danish Minister for Business and Industry as an Authorised Technological Service Institute. The institutes are thus not publicly owned companies but non-profit technological institutes working within technological fields that are of interest from an industrial policy point of view and having a high professional level and a satisfactory financial basis and management. The technological service institutes are independent institutions, each with its own mission and business focus and its own management with sole responsibility for the institute’s financial survival and professional development. The institutes sell their services on normal commercial terms in Denmark and internationally and are thus market-oriented. At the same time, the institutes are non-profit institutions and do not pay dividends – any profit remains in the institute and is used for consolidation, development of services and R&D. Being non-profit institutions, the institutes are impartial suppliers, offering their knowledge to all companies wishing to buy technological service.

It is because of the institutes’ objectives, their professional and business profile and their non-profit status that the State involves them in Denmark’s technological infrastructure through authorisation of the institutes and cofunding of some of their activities.

Authorisation enables the institutes to apply for government contract funds. Contract funds help to finance some of the competence-building activities that enable an institute to supply the latest technological knowledge to companies. The money for these competence-building projects is allocated by the Council for Technological Service, which is also the body that recommends institutes for authorisation by the Minister for Business and Industry. In 1997, contract funds amounted to approx. DKK 254.5 million⁴.

The Council provides a GTS institute with cofunding for agreed and clearly defined activities in accordance with a result contract entered into between the Council and the institute. Contract funds are thus not operating grants that may be used to reduce the cost of the institutes’ commercial services.

The result contract is entered into between the Council for Technological Service and the institute for a 3-year period. The contract is preceded by an extensive process that includes the external technical evaluation, preparation of a strategy plan and a continuous dialogue between the institute, the Danish Agency for Development of Trade and Industry and the Council for Technological Service.

The interaction between the GTS institutes, the Danish Agency for Development of Trade and Industry and the Council for Technological Service is provided for in the Danish Act on Technological Service⁵, which has been translated into Guidelines for Authorised Technological Service in Denmark⁶.

Table 1 Turnover, contract funds and number of employees at the 14 GTS-institutes, 1997

1) Standard terms of reference are attached in appendix 1
2) See, for example "The GTS Institutes & Small Enterprises" (Danish Agency for Development of Trade and Industry, May 1998 - in Danish)
3) See also the Danish Agency for Development of Trade and Industry’s homepage www.efs.dk and the Institute Council’s home page www.atg.dk
4) Contract funds used to be called basic grants.
5) Act on Technological Service, Act No. 221 of 27 March 1996.
6) Guidelines for authorised Technological Service in Denmark, the Danish Agency for Development of Trade and Industry, May 1997

2.1 Introduction to DHI⁷

DHI develops and uses advanced methods and technologies within the fields of coastal hydraulics, hydraulic structures, environmental investigations, and offshore technology and water resources management. DHI was established in 1964 and has carried out projects in 120 countries. Besides research, development and consultancy, DHI sells technology – in the form of modelling software – to companies and public institutions and equipment to laboratories all over the world. The institute has three divisions: a marine technology division, a marine environment division and a water resources division.

DHI covers the following activity areas:

• Harbours and coastal structures
• Coastal hydraulics and littoral processes
• Oceanography and marine environment
• Offshore structures and pipelines
• Rivers and inland hydraulic engineering
• Water resources
• Urban drainage

Within each of these seven areas of activity, three basic methods or disciplines are used:

• Field measurements and real time monitoring of physical and chemical parameters
• Physical models, usually at a reduced scale
• Information technology and numerical models (computer simulation)

DHI’s mission is to advance Danish technological development and competence in the fields of hydraulic and water resource engineering and related fields, with a view to strengthening the potential of Danish society and the corporate sector to carry out projects in Denmark and abroad at a financial and environmental optimum. DHI serves commercial clients of all sizes - small enterprises as well as large Danish and international clients.

DHI is also one of the GTS institutes affiliated to the Danish Academy of Technical Sciences (ATV). The Academy appoints the boards of directors of the institutes, normally from the academic world or potential clients of the institutes and approves the by-laws governing the work of the institutes.

Affiliation to ATV implies that DHI is committed to carrying out its work in accordance with the Academy’s object, which is to promote technological and scientific development to the benefit of Danish society and industry.

DHI's vision is to be the world leader in selected technologies pertaining to coastal zone management, water resources management, flood management, and the urban infrastructure.

Table 2 Key figures for DHI, 1997

Composition of the expert panel
The evaluation was carried out by an international panel of experts, consisting of three persons appointed in February 1998 by the Danish Agency for Development of Trade and Industry. The members of the panel were:

• Mr. Joe Ploeg (Chairman), Senior Consultant in Engineering Consultant, Canada, former Director General of Institute for Mechanical Engineering and Canadian Institute of Industrial Technology, Canada.
• Professor A. Jacob Odgaard, Iowa Institute of Hydraulic Research, USA
• Mr. Peter Lundhus, Technical Director Øresundskonsortiet, Denmark

The expert panel was assisted by the Danish consultant Mr. Ivan Ring Nielsen, Technoconsult.

Evaluation Approach
The evaluation process was guided by the Terms of Reference, developed by the Council for Technological Service. The State - and therefore Danish society in general - directly subsidises the institutes of the GTS system and is at the same time a quality-conscious purchaser of the services provided by the institutes. The State therefore has not only an interest in the performance of these institutes but also an obligation to evaluate their performance. Such an evaluation should include the quality and relevance of the research programme, together with the organisational structure of the institutes, their international networks and their marketing activities. The comments to the Danish Act on Technological Service define four main requirements which society makes concerning the GTS system:

• It must ensure effective transfer of knowledge
• It must possess high and relevant competence
• The services must be easily accessible
• It must ensure effective management and use of resources

One of the focuses of the evaluation has been the quality and relevance of DHI’s research and development activities, which must be considered in an international perspective. The spearhead functions have received particular attention in the evaluation:

• Water resources engineering
  (including both highly specialised modelling and technologies/services for broader planning and management activities)

• Waves and their interactions with ports and coastal and offshore structures
  (including new modelling approaches for wave-current-structure interactions)

• Sediment transport and morphology
  (including cohesive and non-cohesive transport in rivers and coastal areas)

• Hydroinformatics
  (including new emerging techniques, GIS tools and integration of data acquisition, -processing and modelling into software products)

• Numerical methods
  (including combined use of numerical and physical modelling)

2.2 Executive Summary

An evaluation of the Danish Hydraulic Institute (DHI) has been carried out by an

international panel of experts on behalf of the Council for Technological Service.

The purpose of this evaluation has been to verify that DHI's R&D activities, professional competence and services have the content and quality needed to make DHI's present and future activities optimal in relation to both the short-term and the long-term needs of trade and industry and society. Furthermore, the expert panel has considered whether DHI has responded to the recommendations following from the previous evaluation carried out in 1995.

The panel is of the opinion that DHI is a generally well-reputed institute with highly skilled employees. The panel has met many DHI staff members at different levels of experience; they all seem highly qualified and motivated, and fully capable of carrying out the tasks assigned to them. The panel supports DHI's strategy of employing staff from foreign countries with the necessary specialised knowledge, thereby effectively supplementing the existing staff.

The panel finds that DHI fills a real need in society and industry for the kind of services it provides. The R&D programme of DHI is found to be of the highest international standard and DHI is using the most suitable methods and state-of-the-art knowledge in both its research activities and its technological services. The panel considers DHI to be among the world leaders in its fields of expertise.

DHI's consultancy services are consistent with industry's need for competent advice. The institute will have to ensure an appropriate balance between small and large contracts. With its mandated responsibility of serving small Danish companies and local authorities, there is a danger of DHI spending too much of its resources on small, usually less profitable, projects. The panel recommends that DHI evaluate alternative methods of serving the needs of these clients.

The panel has found that most of the recommendations from the previous evaluation have been or are in the process of being implemented. In particular, the panel is pleased to see that a 3D shallow water wave basin is under construction in order to sustain an adequate ability to model coastal zone processes. The institute’s cooperation with the Water Quality Institute (VKI) has progressed well, with the development of ecological models for contaminated land (MIKE SHE). The panel has also found that DHI's internal organisation has been restructured, reflecting a move from centring on hydraulic disciplines to greater market orientation.

Guided by the Terms of Reference, the panel has conducted interviews in order to be able to assess whether the needs of society and industry are indeed being served by the different divisions of DHI and at the same time to identify any issues that might lead to problems in the future. Three important issues have been identified, as described below:

Management issues
Performance indicators have become an important tool for managing companies with relatively large R&D budgets. The use of performance indicators started in the manufacturing and production industries but has been successfully introduced in a much wider range of companies in the last ten years or so. DHI should also be able to demonstrate that its R&D programme, which receives significant grants from the State, generates a positive return on investment. DHI has made some effort to define performance indicators, but only in a very general sense. As indicators it has used achievements of its staff in terms of scientific output (papers, reports, and other publications) and DHI's participation in EU-sponsored R&D projects. While these traditional indicators provide some evidence of the quality of DHI's R&D programme, they do not allow estimates of any financial benefits.

The panel recommends that the management defines a time frame for developing a comprehensive system of monitoring the output of DHI's R&D programme in terms of specific goals and targets, obviously in line with its overall strategic plan. This will eventually lead to being able to calculate performance indicators. It would be advisable to start with one or two well-defined R&D projects, for instance in connection with the new shallow water wave basin or the development of 3-D numerical models.

Box 1 Example of Performance Indicator

Operational issues
The panel sensed that DHI management tends to rely on perception of its present situation rather than on having in place firm strategies and plans for developing the next generation of tools, technologically and financially. There seems to be confidence at management level that the status of competitors’ software tools is still lagging behind DHI. The panel is concerned whether such a comparative advantage for DHI really exists. A major effort, requiring big R&D investments, is necessary to keep DHI at the leading edge. Some new methods, such as Navier Stoke equation solvers, may come from the research at the International Research Centre for Computational Hydrodynamics. It is vital to DHI's future to be able to continue an R&D programme at its current level of funding.

In the panel's view, DHI should concentrate on the development of the immature software packages with the greatest potential such as MIKE SHE and MIKE 3. The panel recommends that DHI concentrate on deciding which software packages to focus on in the future. This analysis could beneficially take its origin in a life-cycle analysis of each of the technology areas covered.

The panel fully supports the MIKE Zero initiative aimed at establishing a common user interface for all DHI software products.

The panel strongly recommends that DHI continue the strategy of offering integrated products based on its three well-established areas of competence, field measurements and onsite monitoring, physical modelling and numerical modelling. This requires all staff members to be encouraged and trained to promote the combined services from all divisions.

The panel recommends that DHI focus its activities on a limited number of business areas in which the vision of being number one in the world could become a reality.

The panel questions whether DHI is applying an appropriate market strategy. It could be that a more focused marketing effort would benefit its software sales as well as its consultancy service. DHI's consultancy projects seem to be driven by opportunities rather than marketing strategies. A key element in a marketing strategy for DHI is the ability to understand the clients' situation. The panel finds that several departments need professional marketing effort.

DHI’s level of competence
DHI's modelling capabilities, both physically and numerically, are undoubtedly world-class. The involvement in European R&D projects also indicates that DHI's work is of a very high quality. DHI has indeed an excellent worldwide reputation in its field.

There is no doubt that "excellence" is an integral part of DHI's culture. The concept of "relevance", on the other hand, has not been quite as widely accepted by the DHI staff as a second criterion for evaluating performance. On the whole, however, the panel finds that DHI fills a real need in society and industry for the kind of services it provides in the areas of activity listed above.

Financial issues:
DHI is facing major investments in R&D. That means that no reduction in the level of R&D funding can be made. In the last five years, the basic research work of ICCH has provided DHI with most of the essential knowledge for software development. The termination of ICCH this year will make it essential to increase DHI’s core grant to the level of the combined R&D programmes of DHI and ICCH. The panel is convinced that a large multiplier effect from this kind of R&D investment is certain. It recommends that, as a condition for an increase in its basic R&D grant, DHI commits itself to developing a performance indicator system within a specific time frame, including the definition of realistic and achievable goals and targets.

7. For a more detailed presentation see DHI’s homepage: www.dhi.dk

Section 3
FORCE

3.1 Introduction to the FORCE Institute⁸

The services and products of the FORCE Institute are available to all parts of Danish and international society. The main target groups lie within the following industrial sectors:

• Chemical/petrochemical
• Oil/gas
• Construction/building/civil engineering
• Energy/power generation
• Material processing
• Mechanical engineering/machinery
• Shipbuilding
• Other manufacturing industries

In 1997, more than 7,000 Danish companies bought services from the FORCE Institute on full commercial terms. Clients range from the largest Danish corporations to very small technology-based companies with only a few employees. Over 50% of the Danish client base is small and medium-sized companies.

On the international scene, the FORCE Institute is constantly increasing its customer base and turnover. Over the years, clients from more than 60 countries have bought services or hardware from the FORCE Institute.

The services and products offered by the FORCE Institute are organised in ten technical divisions. All ten divisions have special know-how in relation to R&D and service activities in their technical fields. The ten divisions are:

• NDT and Information Technology
• Mechatronic and Sensor Technology
• Inspection
• Material and Maintenance
• Welding and Production Technology
• Energy and Metrology
• Material and Product Testing
• Education and Training
• Quality Management and Metrology
• Polymer Technology (Aalborg)

In addition to the above divisions, the FORCE Institute operates two branches offering a wide variety of services and products from all divisions and has a wholly owned subsidiary in Sweden (STK) providing mainly inspection services. The evaluation programme does not cover the Swedish subsidiary.

Table 3 Key figures for FORCE, 1997

Composition of the expert panel
This evaluation has been carried out by an expert panel that has had the overall responsibility for the professional standard and validity of the evaluation. The evaluation was conducted on a peer review basis. The panel had two Danish members and one international member. When the expert panel was appointed, importance was attached to a broad professional standard, expert knowledge, independence from the FORCE Institute and experience from the evaluation in 1995. The panel members were:

• Mr. Torben Riber, M.Sc.Mech.Eng., Managing Director of I/S Vestkraft, Denmark (Chairman)
• Mr. Christian Schwartzbach, M.Sc. Mech.Eng., Managing Director of NIRO Holding A/S, Denmark
• Dr. Xaver Edelmann, Director of Swiss Federal Laboratories for Materials Testing and Research, EMPA, St. Gallen, Switzerland.

The expert panel was assisted by the Danish consultant Mr. Thomas Konradsen, Implement A/S.

Evaluation approach
In the general terms of reference for evaluation of the institutes, the two main headings are summarised as follows:

• How is the institute managing its activities in relation to the recommendations of the preceding report, the strategy plan and the result contract? (In the case of FORCE, the strategy plan and the result contract are one and the same.)

• Is the institute sufficiently effective?

In relation to main heading 1, it is stated that this "refers to a preliminary audit of the implementation of the recommendations and objectives described in the preceding report and strategy plan, together with the milestones included in the result contract. In other words, this is an expanded mid-way evaluation that reveals whether the institutes’ research and communication activities are up to expectations."

A number of matters following from the terms of reference have been evaluated under this heading, e.g.:

• the institute’s general development of knowledge and resources
• the institute’s areas of competence within subsidised knowledge development
• the institute’s knowledge-transfer activities, areas of spearhead competence, areas of sector-specific competence, and networking activities.

In relation to main heading 2, it is mentioned that it has to be "investigated more specifically whether the grant is sufficient for the steps taken by the institute to comply with its strategy plan and whether the institute lives up to the objectives and milestones agreed upon with the Danish Agency for Development of Trade and Industry in the result contract. It should also be investigated what the clients think of the institute’s activities. In this connection, clients can be both public and private clients".

Subheadings therefore include:

• Is the institute fulfilling its result contract?
• What results have the institute’s core activities produced (e.g. turnover, number of clients, patents, etc.)?

Background and purpose of the evaluation
In 1995, the Industry and Trade Development Council decided to implement an expert evaluation of six authorised technological service institutes in the GTS-network of institutes. The FORCE institute was one of those six institutes.

This 1998 evaluation of the FORCE Institute is a midway evaluation that focuses on the "excellent parts" of the institute’s activities, including the results achieved since the last evaluation and the expectations for the future. This means that the more routine service business units are not specifically included in the evaluation.

The evaluation was carried out during the period from March to mid-May 1998 by an expert panel with overall responsibility for the professional standard and validity of the evaluation.

Main conclusions
The expert panel has reached the following main conclusions:

• FORCE is still maintaining its role as a GTS institute in a very satisfactory way, in relation to both society and industry.
• FORCE is well managed. The institute has a clear strategy that is understood, accepted and followed by all divisions.
• FORCE’s divisions have all proven that they are able to ensure an effective transfer of knowledge to industry – to both large international companies and small and medium-sized companies.
• FORCE has also maintained a high level of technological competence in all important areas. The institute is showing continued leadership in selected areas.

However, there is still a need for further improvement in the organisation, especially with respect to its sales and marketing orientation and its organisational efficiency, including the synergy between the different divisions and companies, which could strengthen the overall performance of FORCE.

Main recommendations
The expert panel has the following main recommendations:

Expansion in foreign markets

• To maintain its role in Danish society and industry and to be able to deliver services of a high quality to companies in-and outside Denmark, it is important that FORCE continue its expansion outside the Danish market. This will undoubtedly require conditions, in which it is possible for FORCE to operate, acquire and manage companies outside Denmark on normal commercial terms.

• In this connection, the expert panel recommends "strategic alliances" as a tool for increasing the organisation’s access to international competencies and knowledge and as a way to keep specialisation without excluding itself from broader opportunities.

• The expert panel recommends an evaluation of the implications of the guidelines under which FORCE works with a view to identifying obstacles to expansion outside the Danish market.

Sales and marketing

• The expert panel recommends that FORCE further intensify the development of the sales force. The first step has been taken by training a group of employees in basic sales techniques and customer service. The next step must be to develop and implement a structure and a sales management system that will ensure that sales and marketing resources are being used in the most effective way.

• The expert panel recommends that FORCE implement a structure in which the responsibility for sales rests with people with special sales competence, solid sales experience and talent within this specific field.

• The expert panel recognises the appointment of a corporate sale and marketing manager but that is not enough to transform the "institution culture" into a modern sale and market oriented organisation. Top management’s role and involvement in sales and key client contact should also be strengthened. There is a clear need for more dedicated sales people in each division.

• The expert panel recommends that FORCE implement a structure that will ensure better synergy between divisions and more effective selling of all of FORCE’s competencies - not only in the local market but also across different national markets - a structure that will ensure, for example, that the Swedish organisation becomes better at selling not only its own services but the whole range of FORCE’s services. This should apply to all organisational units.

Government grants

• The expert panel notes that a comprehensive internal application form and new application and control procedures have been introduced. However, the expert panel still has difficulty in discerning the strategic and political structure behind the allocation.

• The expert panel therefore endorses the Industry and Trade Development Council’s policy of making the result contract the basis for provision of government funding.

• The expert panel will not recommend any specific amount of government grants but simply wishes to stress that without government grants FORCE will have to redirect its priorities towards more business oriented activities on international markets.

• Assuming that the above-mentioned policy and transparent reporting are implemented, the expert panel recommends that FORCE be allowed to work under less supervision.

Publications, conference presentations and patents

• The Expert Panel recommends that FORCE consider a more unified policy regarding publications and conference presentations – one that gives enough room for a minimum of publications and conference presentations in all divisions.

• The expert panel recommends that FORCE formulate a corporate patent policy.

• The expert panel recommends that FORCE establish a Human Resource function and that each division work out an HR development plan describing how the division, in relation to its business strategy and goals, intends to establish, develop and maintain the necessary staff and competencies. The HR development plan should include a successor plan.

• The expert panel recommends that FORCE continue FORCE 2000 and the integration process. Overall, the integration process seems to be proceeding well but there is still a need for further integration and a common approach. Integrated interdisciplinary solutions are still not common enough. The divisions are still in some ways acting as "small fiefdoms"- which, in the long term, may be a barrier to development and efficient utilisation of existing resources.

Future evaluations

• The expert panel recommends that future evaluations be based partially on a formal form and that, in future, the institute be measured using measurable performance criteria/standards. These performance criteria/standards should be agreed upon in advance in a performance contract/result contract with the institute. This will make the evaluation more objectives and make the purpose and result of the evaluation process more clear to all parties. Examples of performance criteria/standards could be number of patents, number of publications, number of Industrial Research Fellowship Programme (Ph.D.) ⁹ employed, number of interdisciplinary projects involving more than one division, market shares and traditional financial figures, shares of money spent on small and medium-sized enterprises (SMEs) or spearhead functions etc. This is in line with the Council for Technological Service’s strategy for the GTS institutes.

• As a supplement to this more formal evaluation form, a benchmark between FORCE and other similar institutes in Denmark and Europe could be considered.

Other areas

• The expert panel recommends that Industrial Research Fellowship Programme (Ph.D.) activities be continued and expanded throughout the organisation.⁹

8. For a more detailed presentation see FORCE´s homepage: www.force.dk

9. The Industrial Research Fellowship Programme is a business promotion programme cofinanced by Agency of Development of Business and Industry and the Company or Institute in question

Section 4
Danish Standard Association

4.1 Introduction to Danish Standard Association

Danish Standard Association works for standardisation and certification, to the benefit of industry, business life and society in general. Danish Standard Association is the officially recognised central body for standardisation in Denmark and the leading Danish certification body.

Danish Standard Association is governed by a board with up ten members elected from among a broadly composed committee of representatives appointed by Danish industry, authorities, organisations, etc. Danish Standard Association’s management is in charge of the day-to-day running of the institute.

Standardisation
Danish Standard Association administrates the work of standardisation in Denmark and is responsible for liaison with European and international standardisation organisations. Around 3,000 voluntary experts from Danish companies, authorities and institutes provide technical expertise in the standardisation work.

Danish Standard Association is a service institute that helps Danish trade and industry and the public sector to gain a say in the common standards that create the basis for mutual recognition of products and help to remove technical barriers to trade in the internal market and the rest of the world. Danish Standard Association is therefore a member of:

• ISO:International Organisation for Standardization
• IEC:International Electrotechnical Commission
• CEN:Comité Européen de Normalisation
• CENELEC: Comité Européen de Normalisation Electrotechnique
• INSTA:Det Internordiske Standardiseringssamarbejde

In its standardisation work, Danish Standard Association prioritises nine technical areas that are of importance to Danish trade and industry and to society in general. The nine prioritised areas are: health & safety and the working environment; metallic structures; building technology; quality; information technology; electrotechnology; logistics and transport; health-care; and external environment.

Certification
Certification documents that a product or a management system, within, say quality management or environmental management, fulfils requirements laid down in standards. Danish Standard Association is Denmark’s largest certification body, with branches in Eastern and Western Denmark. Danish Standard Association certificates are internationally recognised and have market acceptance.

Danish Standard Association is accredited by both DANAK under the Danish Agency for Development of Trade and Industry and the Dutch RvA to certify companies’ quality systems in accordance with the DS/EN ISO 9000-series of standards. Danish Standard Association is also accredited by DANAK to certify environmental management systems in accordance with Danish Standard Association/EN ISO 14001 and to carry out verification of the EMAS regulation. Danish Standard Association’s certification department houses DGM, Dansk Godkendelse af Medicinsk Udstyr. DGM is notified by the Danish Ministry of Health for approval of medical equipment that must be CE-labelled in accordance with EU directive 93/42/EEC.

Danish Standard Association also certifies both products and persons and audits environmental accounts.

Table 4 Key figures for Danish Standard Association, 1997

Source: "GTS-institutterne i Danmark, Den Grønne" (The GTS institutes in Denmark, The Green Book) Institute Council, May 1998.

Composition of the expert panel
The expert panel was composed of persons with special knowledge and experience of standardisation work, both in Denmark and at European and international level. The composition was as follows:

• Mr. Jørn Rex, Head of Division Novo Nordisk A/S (chairman)
• Mr. Geir Bekkevold, Head of Section Norwegian Ministry of Foreign Affairs, Oslo
• Mr. Kurt Sørensen, Managing Director A/S Brødrene Müller (withdrew from the panel before commencement of the work)

The expert panel was assisted by the Danish consultant Mr. Bent Schou, ProMentor Management A/S.

Purpose of the evaluation
The purpose of the evaluation has been to determine whether Danish Standard Association’s "development activities, technical competence and services have the content and quality needed for the institutes’ present and coming activities to meet the needs of industry and society in both the short term and the long term (see Terms of Reference for Evaluation of Danish Standard Association).

The terms of reference indicate two main questions on which the evaluation must focus:

• How is the institute managing its activities in relation to the recommendations made in the preceding evaluation report and the institute’s strategy plan?
• Is the institute organised sufficiently effectively?

Besides these two main questions, the terms of reference indicate a number of special matters that should be evaluated: Danish Standard Association’s involvement in international activities; the voluntary experts, the balance between public and private clients in the financing of the institute’s activities; Danish Standard Association’s communication and knowledge-transfer activities; and Danish Standard Association’s support for small and medium-sized enterprises.

The evaluation covers only Danish Standard Association’s standardisation activities. By agreement with the Danish Agency for Development of Trade and Industry and Danish Standard Association, the institute’s certification activities, which are 100 per cent user-financed, have been omitted from the evaluation.

4.2 Summary of the evaluation report

Danish Standard Association has undergone intensive and positive development in the last few years. Following the merger in 1992 and subsequent consolidation, Danish Standard Association has initiated and developed a number of important initiatives in the strategy plan period 1996-1998. The institute is currently facing major challenges because interest in standardisation remains high while there are signs of a weakening of the institute’s financial basis.

With a view to the evaluation of Danish Standard Association, the panel identified four main perspectives to which the institute’s tasks relate:

1. the industrial perspective: support for Danish enterprises – particularly small and medium-sized ones
2. the democratic perspective: information and communication on standardisation to decision-makers and society
3. the EU perspective: influencing the international work on standardisation
4. the welfare perspective: production and sale of standards.

The panel has interviewed voluntary experts (in 5 standardisation committees) and public authorities. Most of the feedback falls naturally into the four perspectives from which the panel has chosen to look at Danish Standard Association. Other views are collected under more focused headings discussed later.

1. The industrial perspective
Danish Standard Association should give higher priority to its work for Danish industrial companies – particularly the many small and medium-sized enterprises. The panel has evaluated and considered a range of activities in this area:

• In the light of the limited results from the action in relation to small and medium-sized enterprises, the panel recommends that Danish Standard Association carry out an analysis of the needs and wishes of enterprises – especially small ones – and of how these enterprises can be contacted and develop an interest in the work on standardisation.

• The panel sees, as a general tendency, that none takes responsibility in the daily committee work for communicating interesting problems/results. The panel recommends that Danish Standard Association initiate the formulation of a clear communication policy for standardisation committees and working groups so that responsibility for communication is clarified.

• The panel recommends that Danish Standard Association work in a more targeted way to involve industrial organisations and trade associations in the standardisation work, especially with respect to communication to small and medium-sized enterprises.

• It is very unfortunate when standardisation leads to blocking of small and medium-sized enterprises’ access to the market. The panel recommends that Danish Standard Association assess how the voluntary experts can be supported in such situations.

• In the panel’s view, contact with small and medium-sized enterprises and, in that connection, involvement of trade associations, is not solely Danish Standard Association’s responsibility. The panel recommends that public authorities, e.g. the Agency for the Development of Industry and Trade, also take action in this area.

2. The democratic perspective
Standardisation as a function and Danish Standard Association as an institution have a very low level of public visibility. Danish Standard Association has a number of activities in progress to strengthen the visibility of standardisation work.The panel finds these initiatives praiseworthy but insufficient. More basic and targeted action is needed in respect of public authorities and decision-makers.

The panel therefore recommends:

• that Danish Standard Association take steps for a more aggressive public information policy with a view to making the general public aware of the importance of standardisation to society

• that Danish Standard Association offer to participate in relevant ministerial committees
• that Danish Standard Association actively draw attention to the growing need for participation and cofunding by the relevant public authorities.

• that Danish Standard Association consider developing a kind of "Watchdog" – or monitoring function in relation to relevant EU proposals, mandates and other initiates.

3. The EU perspective
Danish Standard Association functions effectively in relation to the European and international standardisation process. Although Denmark is only a "small" country, it has a major influence by reason of the professionalism and effectiveness of its standardisation work.

• The panel recommends that Danish Standard Association do all it can to maintain its position among the best in Europe.

4. The welfare perspective
Danish Standard Association functions well and effectively with respect to its core activity: standardisation. It should, however, constantly seek further improvement – particularly with respect to the many voluntary experts. The very large quantities of paper are probably a serious barrier to participation by voluntary experts in the work on standardisation.

• The panel recommends that Danish Standard Association carry out a detailed analysis of document and paper flows with a view to limitation and increased goal orientation.

Development of production and results
In its report, the expert panel has defined a number of so-called evaluation parameters in relation to Danish Standard Association. The parameters are based on the above-mentioned perspectives and tasks.

• On that basis, the panel has concluded that the development of production and results at Danish Standard Association was positive in most areas in the period up to 1997. There is nothing to indicate any deviation from this positive trend in 1998.
• The panel has noted that the growth in published standards has not kept up with the growth in approved standards in 1997. The panel therefore recommends that Danish Standard Association continue initiatives to solve the bottleneck problems in the production and sale of standards.
• In particular, it should be noted that an international comparison of progress in the standardisation process shows that Danish Standard Association is among the best in Europe. The panel recommends that Danish Standard Association maintain this good position.
  The panel has noted that Danish Standard Association makes only limited use of systematic measures and associated measuring methods in relation to the prioritised tasks. The panel recommends that Danish Standard Association develop methods for more systematic establishment of criteria for measuring performance and, in that connection, develop the necessary measuring methods.

Training of voluntary experts
The panel has noted a positive development of Danish Standard Association’s training activities.

• The panel recommends that Danish Standard Association keep up and expand the work in this area.

Translation of standards
Lack of translation can naturally be regarded as a kind of "technical barrier to trade" and action should be taken to remove that barrier.

• The panel recommends that Danish Standard Association engage more actively in solving the translation problem. In particular, the panel recommends that Danish Standard Association analyse and assess the need for translation of a large proportion of published standards and consider how the necessary funding can be found for this work.

It should be possible for translation of standards to form part of Danish Standard Association’s service contracts with the responsible ministries.

Danish Standard Association’s international image
The panel has conducted interviews with important people in Danish Standard Association’s sister organisations in other countries. The picture of the institute that emerges from its international sister organisations accords well with the reports received by the panel from both voluntary experts and public authorities. Danish Standard Association is an efficient, service-oriented organisation in the standardisation field. Danish Standard Association is to a lesser extent an institution that pursues political goals and plays a visible role in the public debate.

• The panel recommends that Danish Standard Association consider closer liaison with other Nordic standardisation organisations.

This should also be seen in the light of the development of the Nordic countries as a region in the cooperation with the EU/EEA.

Following up on the 1996-98 strategy plan: the strategic challenges
The panel has reviewed the strategy plan and, together with the management of Danish Standard Association, has taken stock of the progress made. On this basis, the panel finds that many of the planned activities have been implemented or are being implemented. This means that a substantial part of the strategy plan will have been realised by the end of 1998. However, the panel has also found that a number of important initiatives have not yet been implemented owing to lack of grants.

It is clearly positive that Danish Standard Association has implemented a large part of its 1996-98 strategy plan, or is on the way to doing so, despite the lack of grants for special strategic activities. However, it must still be pointed out that the failure to implement important parts of the strategy plan is only partially due to a lack of grants from the Agency for Development of Trade and Industry. Danish Standard Association should itself assume more of the responsibility for prioritising the use of funds for both operating and development activities.

• The panel therefore recommends that, in future, Danish Standard Association’s management assume more direct responsibility for the strategy plan and the implementation of the planned initiatives. It seems inappropriate for operation to be financed out of current revenue, including government grants, while development is made dependent upon extra grants.

It is also difficult to judge from the strategy plan whether Danish Standard Association is achieving its own or its clients’ goals for the standardisation work. The strategy plan focuses greatly on strategic action areas and less on the objectives which the plan and the individual initiatives are intended to achieve.

• The panel recommends that Danish Standard Association develop a more systematic approach to setting the objectives that the concrete initiatives in the strategy plan are intended to achieve. This will lead to greater goal orientation and improve the possibility of regular stocktaking and follow-up.

Some of the initiatives contained in the strategy plan have already been dealt with above in connection with the feedback from voluntary experts and public authorities. In addition to these, there are a number of areas/challenges arising directly from the strategy plan:

The information technology challenge
Although Danish Standard Association has achieved much on the IT front, targeted action is called for in several areas if the needs of Danish Standard Association’s collaborators are to be met:

• The panel recommends substantial development of Danish Standard Association’s homepage and its applications.

For example, callers should at least have the possibility of accessing a short description of the scope of all standards, both those that have been completed and those that are still in the pipeline, the composition of committees, chairmanships, secretariats, etc. Danish Standard Association could easily meet the users’ needs in this area and should do so.

• The panel recommends that Danish Standard Association closely review its working processes in connection with the transition from paper-based procedures to IT-based procedures..

This is not the first time that inappropriate work procedures have been maintained in the computer age despite the fact that the new technology offers entirely new possibilities.

The challenge in relation to the international organisations
The panel has noted that Danish Standard Association is participating very actively in the Danish-initiated work on rationalising the international standardisation process.

• The panel recommends that Danish Standard Association continue to prioritise action to rationalise the European standardisation process, including – first and foremost – the organisation of CEN.

The panel has also noted Danish Standard Association’s unique possibility of mediating between the views of the European standardisation organisations.

• The panel recommends that Danish Standard Association be more aware in future of its possibilities of mediating between CEN/ISO views and CENELEC/IEC views, mainly with a view to avoiding too many and overlapping standards.

Resources, qualifications, organisation and conditions
The development of expenditure and revenue gives only a slight indication of the big challenges facing Danish Standard Association in the years ahead. There is much to indicate major changes, particularly in the funding of the institute’s activities.

In a situation with changes (possibly a reduction) in its funding, Danish Standard Association needs to prioritise the application of its resources and thus ensure that they are used optimally. It does not have a clear prioritisation mechanism at the present time.

• In this connection, the panel recommends that Danish Standard Association develop a reliable prioritisation mechanism for allocating resources to personnel, project and development activities.

In the view of the panel, Danish Standard Association must take targeted action on developing competence in order to equip itself for the coming challenges.

• The panel therefore recommends that the methods already used by Danish Standard Association today be thoroughly systematised and targeted. This will also enable Danish Standard Association to establish the necessary linkage between its strategy plan and its competence development plan.

The panel finds it extremely positive and important that Danish Standard Association has taken steps to restructure its organisation and management, since this will make it better equipped to cope with the coming challenges.

• However, the panel recommends that Danish Standard Association ensure that its managers maintain their practical knowledge of the standardisation work through some personal participation in this.

The panel does not consider that Danish Standard Association can, in practice, be compared with the other GTS institutes. It also believes that Danish Standard Association might benefit from standing outside the GTS system and thus achieve a more independent status and relationship with the Danish Ministry of Business and Industry:

• Danish Standard Association would become more visible to the central decision-makers with respect to assignments and grants.
• Danish Standard Association would be able to establish a stronger, independent profile.
• Danish Standard Association would be able to enter more directly into a dialogue with the Ministry of Business and Industry with a view to getting the necessary synergy created between industrial policy and the practical daily realities of industry.

If Danish Standard Association is to have a more independent status, it will need a suitable organisation and a good management that will ensure the right balance between strategy and effective operation.

The panel considers that, with the new organisation and the current management team, Danish Standard Association is well equipped for a more independent and visible position.

• The panel recommends that Danish Standard Association’s affiliation to the GTS network is subjected to an independent, interministerial analysis.

10. For a more detailed presentation see Dansk Standard’s homepage: www.ds.dk

Section 5
Danish Institute of Fire Technology

5.1 Introduction to DIFT¹¹

The Danish Institute of Fire Technology (DIFT) is the centre for fire safety in Denmark. The object of the institute is to promote active and passive fire protection and to combat and prevent damage to the environment from fires. DIFT covers the need of companies and authorities for consulting services, documentation and education/training within active and passive fire protection and prevention of damage as a consequence of fire. DIFT is accredited by DANAK for testing, inspection and certification. It has defined a number of primary tasks and functions for its activities:

• Research and Development: performance of both practical and theoretical R&D projects in the field of fire

• Fire inspection: regular fire inspection at companies and checking of automatic fire-protection systems; service assignments for insurance companies and insurance brokers

• Fire investigations: investigations of the causes and development of fires for the police and insurance companies

• Consultancy: fire-technical consulting services in connection with building projects and evaluation of fire protection solutions; preparation of contingency plans, risk analyses, etc.

• Testing: laboratory testing and classification of the fire-technical properties of materials and products

• Certification: Issuing of certificates and approvals and control activities on the basis of Danish and foreign authorisations, including certification of automatic fire protection systems.

• Information: Implementation of courses and theme days, sending out fire technology publications, library activities and general information activities on fire prevention.

 Table 5 Key figures for the Danish Institute of Fire Technology, 1997

Composition of the expert panel
The expert panel was composed of two persons, both of whom participated in the evaluation of DIFT in 1995. The panel thus had extensive technical insight into DIFT and DIFT’s sphere of work.

The panel consisted of:

• Professor Dr. S. E. Magnusson, Institute of Fire Technology, University of Lund (chairman)
• Mr. Michael Petersen, M.Sc., Glasuld A/S

The expert panel was assisted by the Danish consultant Mr. Jeppe J. Edens, Krüger Int. Consult.

Special factors to which importance was attached during the evaluation
In accordance with its terms of reference, the panel has evaluated whether DIFT and its staff have the necessary professional level to ensure that the institute’s activities remain relevant. The panel has also evaluated:

• the institute’s involvement in national and international codification and standardisation work to verify that this is of suitable scope and quality
• the institute’s proactive work with Danish companies at the product development stage with a view to ensuring that Danish products are of good fire-technical quality
• DIFT’s collaboration with similar institutes in nearby countries and, in that connection, the potential for an international division of work in this field.

Profile of the institute
The Danish Institute of Fire Technology (DIFT) provides authorities and companies with consulting services, documentation and education/training within fire safety and fire prevention.

DIFT has been evaluated with a view to determining whether its activities have the content and quality needed to meet the needs of society. A thorough evaluation was carried out in 1995 and another is scheduled for 2001. The present evaluation is a limited midway evaluation that is focused on the development since 1995 and that includes recommendations for the coming three years.

DIFT’s main activities are:

• Servicing through testing, consultancy, investigation, certification, inspection and control,

and preparation of codes and regulations to improve the level of knowledge with respect to fire technology,

• Participation in R&D projects to gain new knowledge and contribute to technological development within fire protection, and
• Compilation of fire technological data and dissemination of documentation and know-ledge supported by consultancy.

Additional funding
As a result of the 1995 evaluation, DIFT was allocated additional government funding. The panel in charge of the present evaluation has focused on the activities carried out by DIFT as a consequence of this additional funding, i.e. the activities that have been initiated by the DIFT in order to improve its situation, the results of these activities and the manner in which the activities have been carried out.

To a very large extent, DIFT has followed the recommendations made in the 1995 report. It has increased its resources as well as its competence within fire technology, primarily by initiating eight project activities for knowledge build-up that will provide the institute with basic R&D within fire technology and improve testing facilities.

The panel has studied the available project documentation, e.g. descriptions, reports and other material. A number of factors are relevant in the integrated evaluation of these project activities, which have been running for less than 2 years:

• At present, Denmark has no university level education in the field of fire safety engineering.
• Consolidated and successful research requires a research group with a certain critical mass.
• International cooperation may be possible, but only to the extent that previous research experience can be demonstrated.

On the basis of these remarks, the panel offers the following summarising comments:

DIFT has succeeded in starting up a number of new developments activities that are important and relevant to the future status of the institute.

Many of these activities cover the main activities of DIFT and can be regarded as important integrating tools for its work.

In quantitative terms, produced research results are moderate, for some activities even modest, although the short period of time that these activities have been under way should be kept in mind.

In a long term perspective, and, if continued, the newly initiated activities should put DIFT at the same engineering and scientific level as other leading Nordic laboratories.

Continued funding should be contingent on DIFT presenting an R&D/knowledge build-up strategy that provides a clear-cut objective and prioritisation of activities to achieve the optimum mix of projects.

Generally speaking, the funds allocated to DIFT in the period 1996-98 have enabled DIFT to:

• expand the level of technological know-how within DIFT,
• increase the R&D knowledge and background of DIFT’s staff,
• obtain new tools in the institute’s work for authorities and companies,
• become a natural partner for authorities and international institutions, and
• provide a platform for attracting R&D projects and relevant R&D employees.

Challenges for DIFT
Based on DIFT's present strategy, developments and requirements of the outside world, the main challenges to DIFT in the next three years will be:

• to acquire and maintain sufficient technological know-how in the organisation.
• to increase Denmark’s influence on international work on harmonisation of rules and regulations.
• to make DIFT more visible to the public.
• through R&D to secure the development of adequate tools for implementation of fire protection requirements based on functionality.
• to improve the visibility and tangibility of R&D activities and results.

Conclusions and recommendations
Since the 1995 evaluation, interest in fire protection in Danish society has been increasing. DIFT has profited from this situation by receiving additional funding for implementation of a number of the initiatives recommended in the preceding evaluation.

However, the R&D activities carried out in Denmark within fire technology are modest by international standards. Danish investments in the fire protection field are far lower than in the other Nordic countries. Even though DIFT has started some R&D for the additional funds provided, there is still a long way to go. DIFT has succeeded in initiating a university education in Denmark for fire engineers, which is an essential condition for R&D within fire technology.

The fact that the different departments have been relocated to Avedøre Holme has improved the efficiency of DIFT and made it more visible to its clients. At present, DIFT is generally accepted by the professional fire technology establishment as the centre for fire technology. However, the institute still has to improve its image in Danish society.

Even though its fire technology monopoly has officially ended, DIFT still has a virtual monopoly because more than 90% of Danish fire technology resources at institutes and universities are to be found at DIFT. The institute also seems to have kept most of its customer base.

The panel has outlined a number of recommendations and initiatives for improving the performance of DIFT:

• DIFT should develop an R&D strategy, possibly in cooperation with foreign research institutes.
• The strategy should have a clear-cut objective to provide DIFT with an R&D image internationally.
• DIFT should utilise its enormous amount of data in a database to be used for future clients.
• DIFT should try to develop a proactive strategy to improve its public image.
• DIFT should develop a strategy for applying for funds from the authorities.
• DIFT should develop a strategy for the role of DIFT in establishing fire requirements based on functionality.

Lastly, the authorities should be made aware of the necessity of DIFT receiving additional financial support for some years to come if the present positive trend is to be maintained.

11. For a more detailed presentation see DBI’s homepage: http://inet.uni-c.dk/-dift

1 Technological Service
The purpose of the Danish network of authorised technological service institutes (GTS institutes) is to strengthen technological service in Denmark, including promoting the development and use of technological and managerial knowledge, and to stimulate innovation in companies. The comments to the Danish Act on Technological Service set out four main requirements which society, including trade and industry, make concerning the GTS system:

• It must be able to ensure effective transfer of knowledge

• It must possess high and relevant competence

• The services must be easily accessible

• Effective management and use of resources must be ensured

It is also stated that the GTS institutes’ principal and dominant business concept is to act as enterprises offering consulting services to other enterprises on commercial terms. Authorised technological service institutes must at the same time contribute to the industrial promotion system by solving agreed tasks that are partially funded by the government, including the Danish Ministry of Business and Industry.

2 Purpose of the evaluations
The purpose of the evaluations is to verify that the GTS institutes’ development activities, professional competence and services have the content and quality needed to make the institutes’ present and future activities optimal in relation to both the short-term and the long-term needs of trade and industry and society.

The GTS institutes differ greatly with respect to both professional profile and choice of target group. However, despite the great heterogeneousness, their non-profit and business activities lie within four functions:

• a consultancy function

• a spearhead function

• a sectoral function, and

• a social function.

In addition, the GTS institutes have a special duty to ensure the dissemination of knowledge at all levels to small and medium-sized enterprises.

The basis for the evaluation is therefore that the State - and thus also the general public - by reason of its direct and indirect role as subsidising authority and quality-conscious client for the GTS system’s services, has a clear interest in, and obligation to evaluate, not only the quality of the research at the individual institutes, but also the institutes’ organisation, international networks and marketing to the extent that these matters are of importance to their function as GTS institutes. (See also: "Guidelines for Authorised Technological Service in Denmark", Danish Agency for Development of Trade and Industry, 1995)

As an innovation, the evaluation is being made part of a six-year cycle, alternating between a comprehensive evaluation of the entire institute’s activities and a more limited evaluation at three-year intervals. The six-year cycle commenced in 1995 with a comprehensive evaluation of all the institutes over a three-year period that ended in 1997.

The present terms of reference therefore apply to the limited round of evaluations, which will commence in 1998 and ends#. When the present six-year cycle ends in the year 2000, new terms of reference will be prepared for the next comprehensive round of evaluations. The new terms of reference will reflect the social and industrial needs applying at that time.

3 Description of the institute to be evaluated
XXX

5 Organisation and content of the evaluation
The evaluation is carried out by a panel comprising at least one expert, who may have participated in the preceding evaluation, and one expert with experience in technical and scientific evaluations, who is appointed as head of the panel. The Danish Agency for Development of Trade and Industry and a consultant provide the panel with secretariat assistance.

The main headings in the evaluation can be summarised as follows:

• "How is the institute managing its activities in relation to the recommendations in the preceding evaluation report, the strategy plan and the result contract?"

Management in relation to the strategy: The first main heading refers to a preliminary situational audit of the implementation of the recommendations and objectives described in the preceding evaluation report and the strategy plan, together with the milestones included in the result contract. In other words, this is an expanded midway evaluation that reveals whether the institutes’ research and communication activities are up to expectations. A number of matters should be investigated under this heading, e.g.:

• The institute’s general development of knowledge and resources

• The institute’s areas of competence within subsidised knowledge development

• The institute’s knowledge-transfer activities, areas of spearhead competence, areas of sector-specific competence, and networking activities

• Present and potential clients are contacted in a survey carried out to determine whether there is harmony between supply and demand.

Is the institute sufficiently effective? Under this heading, it is investigated more specifically whether the grant is sufficient for the steps taken by the institute to comply with the strategy plan and whether the institute lives up to the objectives and milestones agreed with the Danish Agency for Development of Trade and Industry in the result contract. It should also be investigated what the clients think of the institute’s activities. In this connection, clients can be both public and private clients. Subheadings therefore include:

• Is the institute fulfilling its result contract?

• What results have the institute’s core activities produced (e.g. turnover, number of clients, patents, etc.)?

The panel can use several different methods in its evaluation. Depending on the form of analysis in question, these can include a peer review, client interviews, questionnaire-based surveys, random interviews at the institute, or theme workshops. The actual methods of analysis used are agreed between the panel and the secretariat (see annex 1 on proposal concerning evaluation aims and tools).

As a part of the evaluation the panel will assess the spearhead function in accordance with annex 2. The assessment is to be used as a check list in the evaluation to give a rough indication of the institute’s standard in its principal areas of competence.

6 Special factors to which importance must be attached in the evaluation
XXX

7 Result of the evaluation
The evaluation report must include a summary and a main report. The summary, which must be suitable for publication, must contain an overview of the institute’s strongholds, the development of its performance indicators, and the panel’s critical comments and recommendations. The overview must be amplified in the remainder of the summary and explained in the main report.

The main report must be structured as follows:

• The institute’s development and current situation in relation to the strategy plan and the result contract

• The institute’s resources, areas of competence, knowledge development and quality

• Result of the user survey.

The complete report, i.e. the summary and the main report, should be between 25 and 50 pages long and must, as a general rule, be written in English.

8 Interested parties
The authorising and decision-making body is the Danish Council for Technological Service.

The users of the results of the evaluation will be primarily the institute in question and its board and management, the Council for Technological Service and the Danish Agency for Development of Trade and Industry.

It must be stressed, however, that the summary of the evaluation report will be publicly available and might be included, either separately or together with evaluation results from other GTS institutes, in the Danish Ministry of Business and Industry’s general fact-finding activities.

9 Time schedule
The time schedule for evaluation of the institutes is as follows:

February 1998: Approval by the Council for Technological Service of the specific terms of reference for the evaluation of the institutes and of the planning and organisation of the evaluations, including:

• Submission of the composition of the evaluation panels to the Council for Technological Service for approval after prior consultation with XXX and the Danish Academy of Technical Sciences (ACTC).

• Call for background material for the panels from the institutes

March 1998: Opening meeting

March to May 1998: The evaluation

15 May 1998: Delivery of the evaluation report and subsequent consideration of the report by the Council for Technological Service.

10 Budget
The Danish Agency for Development of Trade and Industry, together with a consultant, will provide the panel with secretariat assistance. The members of the panel will be paid a small fee. The cost of the services provided by the consultant and the panel will be paid by the agency.

Annex 1 Proposal for measures and tools to be used in the evaluations

Annex 2 Assessment of the institute’s spearhead function

The panel defines spearhead function in a dialogue with the institute.