A REVIEW OF PROGRAMS

FOR MINORITIES AND MINORITY SERVING INSTITUTIONS

 

 

 

 

 

 

 

 

 

By

 

Committee of Visitors (COV)

 

The National Science Foundation

 

February 1-2, 2001

 

 

 

 

 

 

 

EXECUTIVE SUMMARY

 

Overview

 

            This Committee of Visitors (COV) report summarizes the findings, recommendations and strategic outcomes for each of four programs serving minorities and minority serving institutions.  The COV conducted an extensive review and analyses of the Louis Stokes Alliance for Minority Participation (LSAMP), Historically Black College and University -  Undergraduate Program (HBCU-UP), Alliances for Graduate Education and the Professoriate (AGEP), and the Centers for Research Excellence in Science and Technology (CREST) programs on their implementation strategies and results relative to the Government Performance and Results Act (GPRA) guidelines, NSF-wide strategic plan, and the EHR directorate and HRD divisional goals and objectives.

             The primary strategic objectives for the four programs under review by this COV are the following:

 

·        To develop educational capacity at minority serving institutions;

·        To leverage existing capacity to substantially increase baccalaureate graduates from minority serving and majority institutions;

·        To leverage existing capacity and linkages to substantially increase doctoral graduates and new faculty production; and

·        To build research capacity, principally among senior researchers at minority serving institutions.

 

To maximize the effectiveness of the programs under review, four primary coordination strategies for achieving program linkages have been established, (1) linkages among programs for minorities and minority serving institutions, (2) linkages with other HRD programs, (3) linkages with other EHR programs, and (4) linkages with other NSF programs and other Federal programs with comparable goals and objectives.

            The underlying philosophy of these programs is that they will ultimately meet national needs with respect to the future workforce in the SMET fields.  This philosophy is strategically aligned with the following GPRA goals, as listed below:

 

People -  A diverse, internationally competitive and globally engaged workforce of  scientists,  engineers, and well-prepared citizens.

Ideas    -  Discovery across the frontier of science and engineering, connected to learning innovation and service of society.

Tools    - Broadly accessible, state-of-the-art and shared research and education tools.

 

 

 

 

Approach

 

In the conduct of the COV, the committee considered a number of informational sources, including a random sampling of award and declination files for FY 1998, FY 1999, and FY 2000; presentations and individual interviews from the program staff and senior administrators; and printed documents pertaining to NSF strategic outcomes and evaluation data.  The committee paid particular attention to the integrity and efficiency of program processes and management through an analysis of the following four factors:

 

·        Effectiveness of each program's use of merit review procedures

·        Program ratings on the use the new NSF merit review criteria

·        Reviewer selection

·        Resulting portfolio of awards

 

The Committee of Visitors (COV) was convened by the National Science Foundation on February 1-2, 2001 was chaired by Dr. Orlando L. Taylor, a member of the EHR Advisory Committee and comprised of the members listed below under the following sub-groups:

 

Undergraduate Sub-Group

 

Dr. Linda Mantel, University of Portland, (Sub-Group-Chair)

Dr. Alfredo G. de los Santos, Jr., Arizona State University

Dr. Charles Pickett, Mississippi Institutions of Higher Learning

Dr. Robert Harvey, Knoxville College

 

Graduate and Research Institutes Sub-Group

 

Dr. Fitzgerald Bramwell, University of Kentucky, (Sub-Group-Chair)

Dr. Jeanette Jones, Alabama A&M University

Dr. Warren Buck, University of Washington, Bothell

Dr. Joseph Ortiz, Kansas State University

 

             

 

           

 

 

 

 

 

 

 

SUMMARY OF MAJOR FINDINGS

AND RECOMMENDATIONS

 

 

Major Findings

 

The following represents a summary of some of the major findings of this COV report:

 

v     The overall design, including appropriateness of review mechanism (e.g., panels, ad hoc reviews, site visits, etc.) for all programs under review, are adequate based on the use of merit reviews.

 

v     The six months which the Foundation allows itself between proposal submission and notification of award or decline is longer than desirable for the LSAMP program. For the HBCU Program, the due dates and notification timetables appear adequate. 

 

v     All funded proposals fulfilled the priorities and criteria stated in the solicitations and guidelines for all programs under review. Within AGEP programs, the alteration of program criteria encouraged the formation of institutional alliances to achieve objectives, resulting in an increased emphasis on regional partnerships.

 

v     Generally, among CREST programs there was consistency with priorities stated in the program solicitations, announcements and guidelines.  We note the recent suggested addition of a matching funds requirement in CREST.

 

v     Among programs such as LSAMP and HBCU-UP, it is very difficult to separate the intellectual merit from the broader impact of the project. The current program announcement of the importance of linking the HBCU-UP and LSAMP programs to other NSF-funded graduate programs (such as AGEP and CREST) and to graduate institutions is commendable.

 

v     Within AGEP and CREST programs, program officers adequately monitored review panel activity with respect to intellectual merit. The impact of diversity and pipeline activity on basic research programs is adequately addressed.

 

v     Reviewers show deep commitment to-- and annual reports reveal--the establishment of a community of graduate programs that are intimately networked with one another and with undergraduate feeder institutions.

 

v     The identification of and support for emerging opportunities and the creation and support of existing opportunities are difficult to distinguish. Existing opportunities appear to provide a leverage for NSF support and are crucial to the continuing success and expansion of program efforts.

 

 

 

v     Potential support for new investigators is enhanced through the LSAMP program by the ability of researchers from small or developing institutions to partner with investigators from major research institutions.

 

v     Both undergraduate programs show specific support of minority students, programs, and institutions. Also, there is a good balance of multidisciplinary and innovative projects in funded proposals.

 

v     AGEP projects appear to uphold high professional standards; however, there is a large number of unfunded proposals that also meet these criteria. Out of seventy proposals submitted during FY 98-00, twenty-two were awarded. This award ratio of 31% approximates the Foundation-wide rate.

 

v     Among CREST projects, the overall quality of the science/engineering programs is very good, with  a good  portfolio of awards.

 

v     CREST institutions generally take advantage of emerging opportunities to expand grant award portfolios such as REU’s, instrumentation awards, and awards from other agencies such as the Department of Energy, Department of Defense, Federal laboratories and industrial alliances.

 

v     AGEP projects have broad support for new and established investigators, benefiting the creation of faculty mentoring relationships. This is true with respect to post-docs, senior researchers and new faculty.

 

v     AGEP proposals that are funded generally emphasize the integration of research and education in an academic context. CREST institutions often emphasize outreach, mentoring and research even more than research-intensive institutions.

 

v     AGEP projects have produced a steady increase in the number of minority scientists in the pipeline.

 

v     AGEP projects have a tendency to emphasize incremental advances in research and to adopt well-recognized pedagogical strategies. The majority of the CREST institutions have multidisciplinary projects and few high risk projects.

 

vHBCU and LSAMP programs do not specifically have the goal of improving mathematics, science and technology understanding and skills for U.S. students at the K-12 level; however, several of the LSAMP alliances, e.g., Chicago State and Howard, have teacher preparation programs as part of their project.

 

 

 

 

v     The vast majority of the faculty teaching in institutions funded by HBCU-UP is African-American, and a significant majority of the students in these institutions is also African-American, thus having the potential to strengthen the diversity of the scientific workforce.

 

v     Increased numbers of minority undergraduate students at LSAMP institutions enroll in and complete college level courses in all areas of SMET, thereby assisting greatly in meeting the goal of a better-informed citizenry. Across the 27 alliances, over 187,000 students were enrolled in SMET courses in 1999.

 

LSAMP programs also assist greatly in increasing the diversity of the scientific workforce. For example, in 1999, there were 20,567 BS degrees awarded in SMET fields through the LSAMP alliances.  [LSAMP MARS database]

 

v     The increased use and further development of the new LSAMP-Mars database will make an enormous contribution, not only to the success of LSAMP, but also to its linkages with NSF-funded graduate and research programs, by demonstrating the quality and productivity of LSAMP projects.

 

v     LSAMP funding does not provide international activity; however, virtually all grantees find creative ways to prepare their students for work and study in contexts beyond their local campuses.

 

v     CREST/AGEP programs have performed excellently in growing the numbers of students exposed to NSF funded programs. Coordinated efforts exist within AGEP to access and/or improve the quality of graduate support systems, expand graduate education curriculum options, gain access to a variety of academic partners and improve the preparation of graduates for broad career options.

 

v     The transition of the AGEP to emphasis on institutional alliances has provided a workable mechanism through which graduate students from diverse backgrounds can gain access to globally engaged science and engineering professionals with outstanding reputations.

 

v     As noted in a recent analysis [SED 9255369] of the LSAMP program, the alliance model of LSAMP links advances in discovery (led by research universities) with the advances in learning and societal benefits contributed by the other components of an alliance. [Westat Study of LSAMP Program]

 

v     The institutions that are part of an LSAMP alliance are proactive in linking research and education. Curriculum reform is a major component of undergraduate programs in LSAMP schools.

 

For example, North Carolina A&T State University in Greensboro, NC [HRD 9909058], as part of a target strategy of improving institutional capacity, will revise three “killer” courses simultaneously: calculus, chemistry, and physics. Teams of faculty from the three disciplines will work together to develop complementary course objectives, using active learning pedagogies, and the same web-based technology. Thus, students in calculus will be taught the mathematics concept that they need in order to solve the problems that are being assigned in chemistry or physics, and the homework they turn in will be graded by both the calculus faculty and the science faculty. Four classrooms will be remodeled to facilitate this new manner of teaching and will be equipped with the most current technology available. Faculty in all three departments will be provided with workshops, seminars, and technical assistance to implement this new way of teaching and learning. 

 

v     CREST researchers and programs require shared use instrumentation, cross campus collaboration, and inter-university collaboration to enhance the probability of the discovery or creation of new knowledge. The collaboration in physics of the CREST Center at Hampton University [HRD 9154080; HRD 9633750] with the University of Michigan and with Duke University is an excellent example of such collaborations.

 

v     Networking and connectivity that take full advantage of the Internet and make SMET information available to all citizens are exploited fully within the AGEP community. Among CREST programs, this activity is critical to success. 

 

v     AGEP projects have created effective lines of institutional communication and student transition between participating institutions.  This collective result derived from meeting the needs of individual institutions has contributed to an effective use of science and engineering resources. 

 

Major Recommendations

 

The following represents a summary of the major recommendations found within this COV report:

 

Evaluation Recommendations

 

v     A longer preparatory period after the release of the program solicitation and before the review process is advised.

 

v     In order to provide a fair and comprehensive evaluation of the CREST/AGEP  programs, the COV panel should attend meetings of the program directors.

 

v      The composition of the external review panels for AGEP programs could be enhanced by increased representation from disciplinary representatives within NSF.

 

v      A review mechanism that embeds professional contributions (e.g., review panel membership, site visit participation, etc.) from other NSF directorates should be implemented for CREST programs.

 

 

 

 

 

 

Program Recommendations

 

v     The capacity of CREST/AGEP centers should be maximized.

 

v     Activities to complete the transition of MGE programs to AGEP programs should be enhanced until all MGE programs are fully transitioned.

 

v     Linkages between LSAMP programs in Research I universities, HBCUs, Hispanic-serving institutions, Tribal colleges, and two-year, four-year, liberal arts and comprehensive institutions should be improved so that students can be better assured productive careers in SMET, both in academia and elsewhere.

 

v     A common set of professional standards should be established to maximize the potential of engagement of targeted communities and the scientific community as a whole. CREST proposal submission and evaluation should place greater emphasis on the research to be conducted and should be judged in a manner consistent with that used for other NSF-funded centers.

 

v     A database should be developed for LSAMP-Mars programs to access, update and track information on individual students regarding enrollment, degrees obtained, stipends and awards.

 

v     CREST centers should serve as nationally recognized exemplars in their areas of research.

 

v     CREST and AGEP/SMET programs should work toward greater access to the larger graduate scientific communities and professional organizations through Internet connectivity.

 

 

 

A.    INTEGRITY AND EFFICIENCY OF THE PROGRAM’S PROCESSES

        AND MANAGEMENT

 

        Based on the COV’s study of 269 proposal actions completed within the past three fiscal years, comments were provided on each of the following aspects of the program’s review processes and management.  Constructive comments indicating areas for improvement were encouraged.

 

1. Effectiveness of the Program’s Use of the Merit Review Process.

 

The COV considered the following in determining the effectiveness of the program’s use of merit review procedures in assessing integrity and efficiency of processes and management:

 

a.                  Overall design, including appropriateness of review mechanism (panels,

            ad  hoc reviews, site visits);

b.                  Effectiveness of program’s review process;           

c.                   Efficiency; time to decision;

d.         Completeness of documentation making recommendations;

e.         Consistency with priorities and criteria stated in the program’s solicitations,       announcements, and guidelines.

 

The COV made the following observations of graduate and undergraduate programs based on the above indicators:

 

a.                      Overall design, including appropriateness of review mechanism (panels, ad hoc reviews, site visits)

 

UNDERGRADUATE PROGRAMS

          

             LSAMP Program

 

Merit reviews for the LSAMP program are conducted by a combination of panel reviews and site visits (or reverse site visits) before and/or after funding. Panelists review the applications in advance, and, after extensive discussion at the panel meeting, prepare a panel summary. Annual reports provide information on each project and requests for continuation. They are reviewed by panel and reverse site visit. NSF program officers prepare reports and make recommendations based on the panel reviews and on staff analyses.

 

In addition, the LSAMP has had Program Effectiveness Reviews, carried out by NSF staff, which involve the preparation of substantial summaries by the funded institutions and participation in a combined reverse site visit. These reviews provide data on the production of minority graduates in SMET fields compared with baseline efforts and compared to state and national trends. Thus, they are measures of the effectiveness of the LSAMP project on individual campuses. In addition, program staff participate in the activities of the funded projects and carry out additional site visits as necessary. These multiple forms of review provide both detailed information on each participating institution, as well as a sense of the institutional situation and commitment to the goals of the program.

       

      HBCU Program       

 

This program has completed only two annual competitions so far. Merit review is carried out as discussed above. In addition, an annual report is submitted that addresses the program goals and objectives included in the proposal. NSF staff makes recommendations for continuation based upon these reports.

 

GRADUATE PROGRAMS AND RESEARCH CENTERS

           

      AGEP

 

A diverse set of reviewers has been systematically assembled to assess the prospects and realization of the recruiting and retention goals of AGEP. External panels have executed a balanced and rigorous review process. The composition of the review panels could be enhanced by increased representation from disciplinary representatives within NSF.

 

      CREST

 

The overall CREST design incorporates a number of self-checking accountability measures and possesses critical strengths which have been pivotal to the reviews of the proposed activities. The COV recommends that a review mechanism that embeds professional contributions (e.g., review panel membership, site visit participation) from other NSF directorates be implemented.

 

b.        Effectiveness of program’s review process

 

UNDERGRADUATE PROGRAMS

 

The LSAMP and the HBCU-UP Programs

 

Effectiveness of the review process for the two undergraduate programs is indicated by the ability of the panels and the NSF staff to differentiate clearly between competitive and non-competitive proposals, and to give clear indications when non-competitive re-submissions have reached competitive strength. Consistency in the ratings from reviewers are the norm (3 VG, or 2E, 1VG are common for highly competitive proposals) and individual reviews show strong commonality of noted strengths and weaknesses of each project. In the HBCU program, most proposals submitted in the first annual competition were not funded. Some of the institutions whose proposals were not funded in the first annual competition submitted proposals for the second annual competition. Those of high quality were funded. Others were not funded in either annual competition.

       

      GRADUATE PROGRAMS AND RESEARCH PROGRAMS

 

AGEP

 

Individuals with an understanding of and commitment to the goals of the AGEP program have conducted candid and informed reviews. Ample opportunity was provided for proposing institutions to obtain feedback from reviews. It is likely that the thoroughness of the review process was reinforced by site visits.

 

CREST

 

As designed, the review process enhances the research competitiveness of the submitting institution. This outcome is reflected in carefully stated reviews by a diverse expert panel review team.

 

c.        Efficiency and time to decision

 

UNDERGRADUATE PROGRAMS

 

LSAMP

 

Proposals are due each year on October 15. Annual reports from continuing institutions are due October 31. For successful proposals, the date of the Cooperative Agreement ranges from August to October of the following year. Thus, the Foundation’s target of providing notification of award within six months is longer than desirable. However, because of the complexity of the proposals, the multiple institutions that are involved, and the multiple steps involved in the proposal process at both the NSF and among the cooperating institutions, it is often difficult to complete the process more quickly.  As of FY 2001, the due date for LSAMP is January 31.

 

HBCU Program

 

The due date for proposals is October 15, and the notifications are made and cooperative agreements signed within six months.

 

 

 

 

 

 

 

 

GRADUATE PROGRAMS AND RESEARCH CENTERS

 

AGEP

 

A thorough review of selected proposals indicates that a timely review process has been implemented with deliberate, but optimal, responses to proposing institutions.

 

CREST

 

Based on a sampling of reviews of proposed activities, the NSF’s response time following proposal submission appears to be good.

 

d.                  Completeness of documentation making recommendations

 

UNDERGRADUATE PROGRAMS

 

LSAMP and HBCU-UP Programs

 

Documentation of recommendations is provided by detailed summaries and analyses by the NSF staff of panel findings and responses of institutions to questions arising from reviews.

 

GRADUATE PROGRAMS AND RESEARCH CENTERS

 

AGEP

 

The completeness of documentation for making recommendations is excellent.

 

CREST

 

The completeness of documentation making recommendations is also excellent.

 

e.                  Consistency with priorities and criteria stated in the program’s solicitations, announcements, and guidelines.

 

UNDERGRADUATE PROGRAMS

 

All funded proposals fulfill the priorities and criteria stated in the solicitations and guidelines for both LSAMP and HBCU-UP.

 

 

 

 

 

 

GRADUATE PROGRAMS AND RESEARCH CENTERS

 

AGEP

 

There is a close correspondence between promulgated information and the execution of the AGEP review procedures. The alteration of program criteria to encourage the formation of institutional alliances to achieve objectives has resulted in an increased emphasis on regional partnerships.

 

CREST

 

Generally, there is consistency with priorities stated in the program solicitations, announcements and guidelines. However, there does appear to be an inconsistency in communicating the need for matching or institutional contributions between program solicitations and the review process. Explicitly, this can be found in the apparent contradiction of “cost-sharing on page 14 of the draft CREST Program Announcement (April 15, 2001), a requirement that was not present in the previous Program Announcement and Guidelines NSF 98-19.

 

2. The Programs’ Use of the Merit Review Process

 

The COV considered the following in determining the effectiveness of each program’s use of the NSF Merit Review Criteria (intellectual merit and broader impacts):

 

a.                  Performance Goal: Implementation of Merit Review Criteria by Reviewers: NSF performance in implementation of the merit review criteria is successful when reviewers address the elements of both generic review criteria.

 

b.                  Performance Goal: Implementation of Merit Review Criteria by Program Officers: NSF performance in implementation of the merit review criteria is successful when program officers address the elements of both generic review criteria.

 

The COV considered the reviewers and program officers of the programs under review to be successful in implementing the merit review criteria.

 

 

 

 

 

 

 

 

 

 

The following observations were made relative to whether reviewers adequately addressed the intellectual merit criteria in their reviews:

 

UNDERGRADUATE PROGRAMS

 

LSAMP: 

 

Example 1.  “..This is a well thought out project with the potential of affecting education in the whole state of () by the creation of bridges between high schools and the universities in the state.” 

 

Example 2.  “The potential for benefits is much greater than most due to the high number and percentage of Blacks and Hispanics.”

 

Example 3. “I rated this proposal as excellent because its approach is based on important research, because it is innovative in its use of technology, because it

builds the ability to change into the curriculum, and because all of these qualities give it a high potential to accomplish its purposes.”

 

Example 4.   “This is an excellent opportunity to model successful approaches to increasing minority students.”

 

HBCU-UP

 

Example 1: “The proposal has clearly stated strategies that sufficiently address program goals. It reflects a complete understanding of the significance of faculty development, curriculum reform and enhancement, and student research experience to strengthen the SMET undergraduate education and research infrastructure. The intrinsic merit of the proposal is stre