A Systematic Review of Postgraduate Theses on Curriculum Evaluation

Curriculum evaluation is a process carried out to decide the effectiveness of training programs. In this process, the strengths and aspects of the training programs that are open to improvement are emphasized. Therefore, curriculum evaluation studies are an important part of curriculum development. In this respect, it is important to consider the research studies on curriculum evaluation studies together in order to develop curriculum. This study aims to review the postgraduate theses completed on the evaluation of curricula in Turkey between 2007 and 2019 according to several variables and to determine the research trend in this field. The method of the research was formed by systematic review. In this context, 586 postgraduate theses written on curriculum evaluation were reviewed. It was determined that the theses showed a balanced distribution according to years and were generally at master’s level. The postgraduate theses were mainly based on quantitative research methods


Introduction
To be able to realize educational needs, the characteristics of curricula should be understood well (Yüksel & Sağlam, 2012). Regarding this, determining the quality of curricula, curriculum evaluation is needed. With the evaluation of curricula, not only the curricula, but also the materials, technologies, time, and budget used in the curriculum and many sources employed through the curriculum can be evaluated. The quality of all elements of a curriculum is judged by revealing what kind of process is followed in which content in line with the objectives of the curriculum. Therefore, the evaluation of curricula is the process of systematically describing, acquiring, reporting, and applying the descriptive and judgmental information about the accurateness, practicability, reliability, significance, and equality of the objectives specified (Stufflebeam & Shinkfield, 2007). Curriculum evaluation is an important element and stage of a curriculum development activity, and constitutes the continuity of the curriculum development process in education. The results or feedback obtained in this process are used to improve or update the curriculum in a healthier way (Varış, 1988). Oliva (2009) stated that the primary purpose of the curriculum evaluation is to determine whether the curriculum aims and objectives are being carried out. Besides, they emphasize that it is also necessary to determine whether the curriculum is functioning while in operation; whether the materials and methods used are appropriate and sufficient; whether the graduates are successful in their next education level and their jobs; whether they can function in daily life and contribute to society; and whether curriculums are cost-effective, and worth the cost.
Curriculum development and evaluation studies in Turkey accelerated with proclamation of the Republic and have continued regularly in parallel with the developments in Turkey and in the world (Özdemir, 2009). Studies that started with the development of curricula mainly for primary education in 1924 appeared also at the secondary education as from 1930. In this context, primary and secondary education curricula were prepared and put into practice in 1924-26, 1931-36-38, 1948-49, 1968-69, 1974-77, 1990-92-98, 2001-05, and 2013-17-18. Taking the dates of curriculum development studies into consideration, as the political and economic situation of the country, the educators who visited Turkey (John Dewey-1924, Alfred Kühne-1925, Berly Parker-1934, Kate Vixon Wofford-1951, the regulations in the education system (Primary Education Law numbered 222, National Education Basic Law numbered 1739-1973, 8-year compulsory education-1998 and 12-year compulsory education-2012), the wars around the world (War of Independence, the First and Second World Wars, Syria Civil War), and finally the technological developments (Sputnik, Computer, Internet) have been effective on the studies over learning and teaching. The first curriculum evaluation studies in Turkey were conducted by the Ministry of National Education (MoNE) in 1944 to define the deficiencies, malfunctions, and problems of the two curricula practiced in five-year urban schools and three-year rural schools and accordingly to develop a new and unique curriculum. For this purpose, in 1944, a questionnaire was applied to teachers and meetings were also held with managers and inspectors by the MoNE Department of Education and Discipline to evaluate both curriculums (Arslan, 2000). Curriculum and evaluation studies took a systematic format in 1952 (Çelenk, Tertemiz, & Kalaycı, 2000). Curriculum development and evaluation studies which were practiced in the MoNE until 1980s also started to be dealt at the universities with the cooperation of the lecturers in these years (Küçükahmet, 1997). In this regard, postgraduate theses started to be written in relation to the development and evaluation of formal and informal education after 1980s (Hazır-Bıkmaz, Aksoy, Tatar, & Atak Altınyüzük, 2013).
In the context of formal and informal education, the availability of different practices in curriculum evaluation process enables the curriculum evaluation studies to be used not only in the field of educational sciences but also in different fields such as social sciences, science and health sciences, and in business and industry sectors (Fitzpatrick, Sanders, & Worthen, 2004, p. 19). Research in Turkey related to curriculum evaluation studies (Aslan & Sağlam, 2017;Dündar & Meriç, 2017;Gökmenoğlu, 2014;Kurt & Erdoğan, 2015;Özüdoğru, 2018;Yetkiner, Acar Erdol, & Ünlü, 2019) are intended to specific areas and have limited samples.
For example, Dündar and Meriç (2017) found that curriculum development and evaluation studies in the context of ESL/EFL are generally aimed at evaluation, based on teacher and student opinions, and questionnaire, interviewing, and document analysis were used in data collection process. Like Dündar and Meriç, Özüdoğru (2018) also handled curriculum evaluation studies in the field of foreign languages and determined that most of the studies focused on the evaluation of elementary school curriculums, language course curriculums and preparatory school curriculums. However, postgraduate degrees, secondary school curriculums and distance foreign language curriculums were evaluated in very few studies.
In her research study, Gökmenoğlu (2014) examined curriculum evaluation studies in only four dimensions -curriculum evaluation models and research approaches, the scope of evaluation studies, stakeholder participation in studies, and disciplines. In Kurt and Erdoğan's (2015) research, results regarding a narrow sample of only 38 studies were presented. While some studies only deal with theses in the field of curriculum and instruction, others deal with curriculum evaluation theses in a specific field (English education), doctoral dissertations between certain years or, only studies whose title is curriculum evaluation. This situation does not fully reflect the availability of curriculum evaluation in different areas. In this study, the theses made in all institutes and disciplines on curriculum evaluation were handled and it is different from other studies in this respect.
This study aims to analyze the postgraduate theses completed on the evaluation of curricula in Turkey between 2007 and 2019 according to several variables and to determine the research trend in this field. In this context, master's theses and doctoral dissertations on the evaluation of curricula were analyzed in order to determine how curriculum evaluation studies affect the postgraduate education programs in our country. The distribution of the postgraduate theses on the evaluation of curricula according to universities, institutions, departments / disciplines were examined and the method, sample group, data collection tool used in the curriculum evaluation studies and the studies related to the subject evaluated were determined. Thus, general situation regarding the studies in the field of curriculum evaluation was initially revealed.
Curriculum evaluation studies are considered as important in terms of guiding the educational sciences experts in the field of curriculum development. Furthermore, this study is considered to provide essential findings to researchers in terms of providing an overview of topics of interest and research methods in the field of curriculum development in Turkey. It will provide researchers with an idea of what different evaluation processes are carried out on similar curriculums. Especially, carrying out a comprehensive study for the distribution of curriculum evaluation theses according to discipline areas, institutions, and years will guide new studies and postgraduate theses. Within this scope, it is aimed to contribute to different shareholders such as teachers, students, administrators, and experts who work in the education field.

Method Design
This research study has the feature of systematic review. Postgraduate theses published in the field of curriculum evaluation were scanned with the systematic review method. Then their completion year and whether they were available for access were displayed. The theses encountered for review were synthesized under titles such as method, data collection tool, and sample (Karaçam, 2013).
Systematic review studies are deemed important in terms of presenting more scientific information together. In addition, studies conducted with systematic reviews are among the studies preferred in terms of being repeated cases by different researchers (Hemingway & Brereton, 2009).
A systematic review can be prepared in these stages (Hemingway & Brereton, 2009;Higgins & Green, 2011;Uman, 2011): Identify the review question, define inclusion and exclusion criteria, search for studies, select studies for inclusion based on pre-defined criteria, extract data from included studies and present results and assess the quality of evidence. Systematic analysis within the scope of the research was carried out in line with the following stages: • Identify the review question: Existing researches were examined to determine the gaps in the field and avoid duplication with similar studies. And then, a clear and well-defined research question was formulated.
• Define inclusion and exclusion criteria: Some criteria were clearly stated for the selection of master's theses and doctoral dissertations. In this context, date ranges, scope and accessibility were taken into consideration.
• Search for studies and select studies for inclusion based on pre-defined criteria: The search was carried out based on the keywords and the years of theses on the official web site of the National Theses and Dissertation Center and the theses accessed were saved by the researcher. Detailed information is given under the title of Selection Process of the Theses.
• Extract data from included studies: The postgraduate theses studied in the research were examined and compared with each other. In this way, categories were created to analyze the documents. Descriptive analysis approach was preferred for analyzing the data.
• Present results and assess the quality of evidence: Findings obtained in accordance with the purpose of the study are presented in tables and interpreted under the title of Results. The findings were supported by comparing them with the literature under the title of Discussion and Conclusion.

Selection Process of the Theses
Some criteria were taken into consideration in the selection of master's theses and doctoral dissertations. The postgraduate theses were included in the study if they were related to the evaluation of a program, curriculum, system, course or activities (These are concepts "activity evaluation, course evaluation, system evaluation" in thesis titles or keywords.); if their completion dates were between 2007 and 2019; and if they were open to access in the National Thesis and Dissertation Center. Within the scope of the National Thesis and Dissertation Center, 607 postgraduate theses meeting the criteria were determined. Twenty one of those theses were excluded from the scope of this research study since they were not open to access. In the final run, 586 postgraduate theses were accessed and constituted the study's sample.
The postgraduate theses selected were summarized through the tables of specifications that were prepared by the researchers. The tables of specifications were prepared as two-dimensional so as to include the keywords and the years in which the theses were completed. Through the twenty-eight keywords determined in the table of specifications, the postgraduate theses on the evaluation of curricula were classified between 2007 and 2019 and included in the study. The keywords used in order to access the postgraduate theses are in the following: Curriculum evaluation / Curriculum + Evaluation / the evaluation of Curriculum / of Curriculum + the Evaluation / the evaluation of Curricula / of Curricula + the Evaluation / the evaluation of 'Curriculum' / of 'Curriculum' + the Evaluation / the examination of Curriculum / of Curriculum + the Examination / the examination of Curricula / of Curricula + the Examination / the examination of 'Curriculum' / of 'Curriculum' + the Examination / the Evaluation of Syllabus / of Syllabus + the Evaluation / the evaluation of 'Syllabus' / of 'Syllabus' + the Evaluation / the Evaluation of Course / of Course + the Evaluation / the Evaluation of 'Course' / of 'Course' + the Evaluation / the evaluation of System / of System + the Evaluation / the Evaluation of 'System' / of 'System' + the Evaluation / the Evaluation of Activities / of Activities + the Evaluation. Activity evaluation, system evaluation, and course evaluation studies were carried out in some of the theses on curriculum evaluation. However, it was observed that activities or courses were evaluated in order to evaluate the curriculum in these studies, and curriculum evaluation model was used in some theses. For this reason, these theses, which are thought to be related to curriculum evaluation, have also been included in the research. Keywords have also been added for these theses. In addition, the theses found with this keyword but were not related to curriculum evaluation were not included in the research. As stated, postgraduate theses made for curriculum evaluation purposes were included in this research study.
The postgraduate theses included in the scope of research by the table of specifications were accessed directly through the official web site of the National Thesis and Dissertation Center. The postgraduate theses in the National Thesis and Dissertation Center were saved without any intervention. The two researchers conducting the research independently checked the accuracy and reliability of the theses. Also, in the evaluation of curricula, a specialist outside the research team was asked for opinion in terms of the storage and classification of the theses included in the research in accordance with the table of specifications in the process of data collection. At this stage, the keywords used in the research study were given to the specialist and asked to search in the theses center. The obtained theses were compared with the theses obtained by the researchers. It has been determined that the theses obtained by the experts and researchers are the same. Then, the theses were classified according to years and the analysis phase was started (Creswell, 2015). When the distribution of the theses in the research according to their levels is examined, it was seen that the sample majorly consisted of studies conducted at Master's thesis level (71.67%, 420 theses). Besides, the number of female postgraduate students (57.33%) who wrote theses on curriculum evaluation were higher than their male counterparts (42.66%).

Data Analysis
In this study descriptive analysis approach was preferred while analyzing the postgraduate theses. At first, a thematic framework was created by examining the literature. Opinions of three curriculum development experts were received regarding the suitability of the determined framework for the research purpose, and the categories (Publication year, university, institution, departments/disciplines, research methods, research models, data collection tools, samples, main subjects, type of curriculum evaluated and curriculum evaluation models of the study) were finalized. Postgraduate theses were examined in line with the thematic framework. The data were organized under certain categories, and frequencies were determined for each category. The total values in the tables presented in findings as a result of document analysis differ from the number of the theses included in the sample in some of the tables. For instance, in thesis studies which include both undergraduates and academicians in the sample group marking is made for both study groups. In this case, the total value differs in the table including the findings for the sample group. These tables are explained in the findings.

Trustworthiness and Transparency
In order to ensure the trustworthiness and transparency in qualitative studies, researchers are expected to select the method in accordance with the research objective and to access the richest data sources by making use of the appropriate data collection tools. In this process, it is also required for researchers to report all their actions in the research process objectively and in detail (Yıldırım & Şimşek, 2013). Concerning this, in the research, the method suitable for the research objective was preferred, access from official sources was realized in the selection of documents, and the data analysis process was reported in detail.

Limitations
This study is limited to the year of completion and accessibility of the postgraduate theses included in this research study. The findings of the research were obtained based on the postgraduate theses written on curriculum evaluation, which were completed between 2007 and 2019 and are open to access at the National Thesis Center. For this reason, different findings can be reached from the theses written in different years or those which were completed between 2007 and 2019 and are inaccessible.

Results
In this part, findings obtained in accordance with the purpose of the study are presented in tables and interpreted.

Distribution of Theses According to the Years
The distribution of postgraduate theses according to the years is presented in Table 1. As it is seen in Table 1, 586 theses in total were examined within the scope of this research. In the distribution of the theses examined, it was concluded that the theses on the evaluation of curricula were generally greater in number in 2019 (62 theses

Distribution of Theses According to the Institutions
The distribution of postgraduate theses according to the institutions is presented in Table 3. As it is seen in Table 3, theses in the field of curriculum evaluation were mostly carried out in the educational sciences institute (51.70%) and social sciences institute (41.29%). This case is related to the fact that the institutes of educational sciences have not been established in some of the universities in our country and educational sciences are conducted as affiliated to the institute of social sciences. Additionally, studies for the evaluation of curricula were carried out in the institutes of sciences and health sciences besides the educational sciences institute. The theses in the environmental sciences, fine arts, and forensic sciences institutes also indicates that the curriculum evaluation is benefited in different fields and disciplines.

Distribution of Theses According to the Departments/Disciplines
The distribution of postgraduate theses according to the departments/disciplines is presented in Table 4.  Based on Table 4, the theses on curriculum evaluation between 2007 and 2019 were generally completed in Curriculum and Instruction program (23.20%). This is related to the fact that curriculum evaluation is one of the main fields of study within the scope of Curriculum and Instruction discipline. In addition, there are many thesis studies in educational sciences, pre-school, and classroom education. These studies were conducted in a wide range of departments/disciplines such as elementary mathematics teaching (8 theses), psychological counselling and guidance (8 theses), physics education (7 theses), philosophy and science of religion (6 theses) under the title of "other".

Distribution of Theses According to the Research Methods
The distribution of postgraduate theses research according to the research methods is presented in Table5. As it is seen in Table 5, the research methods preferred were in three categories as quantitative, qualitative, and mixed method. Quantitative (56.65%) followed by qualitative (26.79%) and mixed method (16.55%) was preferred as research methods. However, when the distribution of research methods according to years was considered, it was determined that the distribution of quantitative and qualitative research methods by years was balanced, and mixed method was used far less between 2007 and 2011 and its use increased in general between 2012 and 2019.

Distribution of Theses According to the Research Model
The distribution of postgraduate theses according to the research model/design is presented in Table 6.  Table 6 shows that the research model or design mostly used in the theses was the survey (36.51%). However, in a considerable part of the theses prepared with survey design, it was not stated whether the research was in the single or correlational survey design, only that the research was realized in general survey model. Another model or design was experimental research (20.13%). In the theses conducted with the experimental design, situations that usually occurred before and after the implementation of any curriculum were generally investigated within the framework of curriculum evaluation. For example, the effect of a technique used in the learning-teaching process on students' achieving goals was investigated. In addition, case study and descriptive research were also preferred mostly in the studies on curriculum evaluation. Explanatory sequential mixed methods design (19 theses), simultaneous mixed methods design (19 theses), content analysis (18 theses), phenomenological (13 theses), embedded mixed methods design (11 thesis) were the other research models and/or designs mostly preferred in the studies.

Distribution of Theses According to the Data Collection Tools
The distribution of postgraduate theses according to the data collection tools is presented in Table  7.  Based on Table 7, data collection tools mostly used in the theses were questionnaires (43.34%) and interview forms (38.56%). This also shows similarity to the findings obtained regarding the research methods used in the theses. Quantitative research methods were more preferred in the theses and this is related to that the attitude scale and tests were often preferred in data collection tools, as well as the questionnaires. It was also found that the use of questionnaire showed a balanced distribution between 2007 and 2019 and that the use of interview form was gradually increasing. The least used data collection tools observation form and document analysis form. Depending on the increasing use of mixed and qualitative research methods, it is supposed that these data collection tools will also be preferred in the coming years. On the other hand, the total number of data collection tools in Table 9 was more than the total number of theses in the research. This shows that more than one data collection tool was used in the theses. Also, even if they were less in number, data collection tools such as inventory (13 theses), individual evaluation forms (11 theses), diary (11 theses), and development file (2 theses) were used in the theses.

Distribution of Theses According to their Samples
The distribution of postgraduate theses according to their samples is presented in Table 8.  Table 8 shows that documents (31.39%) mostly constituted the samples of the theses completed in the field of curriculum evaluation. The curricula in primary education, secondary education, and higher education, articles and research, institutional reports, and etc. were considered as documents. However, the total number of samples was different from the total number of thesis in the research. This indicates that different sample groups were used simultaneously in one thesis. Especially, based on the preference of documents, it can be interpreted that documents are used in several theses. On the other hand, the samples in the theses consisted mainly of students and teachers. This shows that the opinions of students and teachers were given importance in the process of the evaluation of curricula. In addition, there are also samples consisting of undergraduates and academicians in the theses. The fact that the regulations made in the curricula at higher education level are less when compared to the curricula determined by the MoNE explains the situation in sample distribution.

Distribution of Theses According to the Main Subject
The distribution of postgraduate theses according to the main subject is presented in Table 9. Based on Table 9, when the theses completed in the field of the evaluation of curricula were examined, the theses were written mostly on the curriculum usage (73.89%), and then, on the curriculum development process (15.87%) and outputs of curriculum (10.23%) at the least. Here, it can be said that the theses under the title of curriculum usage are an evaluation for one dimension. In addition, theses completed using the curriculum usage path and curriculum are included under this title. Theses under the title of curriculum development and assessment are studies that deal with the preparation and the processes of the curriculum on a specific subject. In these theses, it is seen that the curriculum development stages are applied and the curriculum is applied at the same time. When the outputs of the curriculum were examined, we discussed the effects of the curriculum with on course success, affective development, and psychomotor behavior. Concerning the distribution of the theses according to the research subjects, it was found that the study for the evaluation of curricula was carried out for a curriculum that was generally existing and implemented in the field of curriculum evaluation. Additionally, there were a significant number of theses within the scope of which a new curriculum was developed, implemented and assessed. The theses completed in order to evaluate the effect of changes such as method, technique, material, and model, and etc. within the curriculum were also presented under the title of the effect of curriculum.

Distribution of Theses According to the Type of Curriculum Evaluated
The distribution of postgraduate theses according to the type of the curriculum evaluated is presented in Table 10. In the theses completed in the field of the evaluation of curricula, mostly the curricula for the primary education stage 1 (23.20%) and stage 2 (25.08%) were evaluated. This shows similarity to the findings obtained in the samples of theses completed in curriculum evaluation. Teachers and students at the second stage of primary education and teachers and students at the first stage of primary education were usually included as samples. This was considered as related to that the curricula for the primary education stage 1 and stage 2 were mostly evaluated in the field of curriculum evaluation. Also, the curricula for the first and second stage of the primary education were evaluated more because the arrangements made in the primary education curricula were compared more to the ones made in the curricula for higher education.

Distribution of Theses According to the Curriculum Evaluation Models Used
The distribution of postgraduate theses according to the curriculum evaluation models used is presented in Table 11. When the use of curriculum evaluation models was considered, it was determined that an evaluation model was used only in the 105 (17.91%) of the theses included in the research. However, the Context-Input-Process-Product (CIPP) model developed by Stufflebeam (41.90%) was mostly preferred in the theses curriculum evaluation model was used. The reason why the CIPP curriculum evaluation model was often preferred comparing to the others may be that it is easy to understand and apply. The fact that it also has an application area in different disciplines especially in educational sciences can be seen as another reason for the model to be preferred. Moreover, the use of curriculum evaluation model in theses differed according to the years and there was no proportional increase or decrease. Discrepancy Evaluation Model developed by Provus (3 theses), Four-Level Curriculum Albeit small in number, Evaluation Model by Kirkpatrick (2 theses), Analytical Curriculum Evaluation Model by Demirel (2 theses), Cube Model of Evaluation by Hammond (2 theses) and Responsive Evaluation Model by Stake (2 theses) were the other evaluation models that have been preferred by the researchers.

Discussion and Conclusion
In this study, which aims to examine the postgraduate theses written on curriculum evaluation in Turkey between 2007 and 2019, 586 studies were analyzed. For the analysis, the theses were collected under three main titles as curriculum usage (433 theses), outputs of curriculum (60 theses) and curriculum development process (93 theses). Also, at this point, the trend is towards the evaluation of existing curricula. According to the research by Gömleksiz and Bozpolat (2013) (Dündar & Meriç, 2017;Hazır-Bıkmaz et al., 2013;Kurt & Erdoğan, 2015;Schreglmann, 2016). This situation is supported by the increase in the number of institutes at which postgraduate programs are conducted in the field of Curriculum and Instruction in recent years and also the increase in the number of theses approved every passing year (According to the data presented by the CoHE, approximately 53000 postgraduate theses were approved in the academic year 2017-18). Moreover, while this mentioned increase supports the high number of theses in the field of curriculum evaluation, especially the changes made in the curricula of primary, secondary and higher education in the last 15 years can be considered as the sign of the trend towards studies for the evaluation of curricula. A great majority of the postgraduate theses examined in this research study (71.67%) are at master's degree level. These studies are generally for the evaluation of existing curricula or determination of their effect. In the PhD dissertations examined, it is mostly focused on the development of a new curriculum and its evaluation. This is similar to the tendency to research which is to present an easier and short-term result in master's theses and to provide longer-term and authentic product in doctoral dissertations. Then, the evaluation of existing curricula is easier than the evaluation of a new curriculum after being developed, and the former can be realized in a short time. Another reason for this may be that there are more students at master's level and more master's theses have been written. According to the data by the CoHE (2019), nearly 46000 of the 53000 postgraduate theses approved in the academic year 2017-18 are master's theses and 7000 are at the PhD level. When the authors of the postgraduate theses are considered, it can be said that female researchers have completed more thesis studies than males, but the distribution is close to each other. This is similar to the rates of females and males who further their education at postgraduate level in Turkey. Based on the CoHE data for 2019, in Turkey, the number of people continuing postgraduate education is 490.373. 561% of these are men and 43.9% are women.
Considering the distribution of the theses completed in the field of curriculum evaluation according to universities, more theses were written on the evaluation of curricula in the universities such as Ankara, Atatürk, Gazi, Hacettepe, and Marmara. According to the CoHE 2019 data, when the number of postgraduate students of the universities in the last decade was considered, the universities such as Istanbul, Marmara, Ankara, Istanbul Technical, Hacettepe, Gazi, Dokuz Eylül, Yıldız Technical, Sakarya, Ege, and Atatürk ranked among the top ten. This shows that the universities with the high number of postgraduate students and the ones in which the theses for curriculum evaluation are mainly studied are similar. Especially, the number of students in Educational Sciences, Sciences, and Social Sciences Institutes of the universities (Ankara with 14136 students, Atatürk with 7913 students, Gazi with 9844 students, Hacettepe with 10702 students and Marmara with 19112 students) (CoHE, 2019) also explains the high number of theses on curriculum evaluation in these universities. In parallel with this, the theses on curriculum evaluation, included in the research, were written in the institutes of educational sciences (n = 303) and social sciences (n = 242). Also, in the studies carried out for the postgraduate theses completed in Curriculum and Instruction (CI), it was found that the universities such as Abant İzzet Baysal, Anadolu, Ankara, Atatürk, Fırat, Gazi, Hacettepe, and METU produced more theses in that field in the last fifteen years (Aslan & Sağlam, 2017;Kozikoğlu & Senemoğlu, 2016;Özüdoğru, 2018;Yetkiner et al., 2019).
When the change by years in the distribution of the theses according to institutes was examined, while more theses were written on the curriculum evaluation in the Institutes of Social Sciences between 2007 and 2010, it was determined that more theses were completed in the Institutes of Educational Sciences after 2010. According to the CoHE statistics (YÖK, 2019), the number of Educational Sciences Institute (n = 41) and Social Sciences Institute (n = 138) that were opened and are still operating in Turkey is 179. Based on these statistics, the number of Social Sciences Institute is about three times more than the number of Educational Sciences Institute. Of 41 Educational Sciences Institutes, 31 were opened after 2010. In addition, departments and/or disciplines such as Educational Sciences and Basic Education that were previously under the Institute of Social Sciences were assigned to the Institute of Educational Sciences. This explains the tendency in the distribution of theses from Social Sciences towards Educational Sciences after 2010. According to the study by Hazır-Bıkmaz et al. (2013), dealing with the PhD theses completed in the field of CI between 1974 and 2009, 285 of 358 PhD theses were completed in the social sciences and 58 of them in the educational sciences institutes. Moreover, the studies carried out for postgraduate thesis in CI, especially for the ones completed after the 2000s, revealed that there was a significant increase in the theses completed in the institutes of educational sciences (Kurt & Erdoğan, 2015;Özüdoğru, 2018). Regarding the distribution of the theses according to departments, it was revealed that more studies were conducted for curriculum evaluation in Educational Sciences in general, and those studies were mainly realized in the field of Curriculum and Instruction. The main reason for this may be that Curriculum Evaluation in Education is taught as a course under the Department/Discipline of Curriculum and Instruction. It was determined that, after Educational Sciences, thesis for the evaluation of curricula were completed in the field of Basic Education. Aslan and Sağlam (2017) reached a similar result in their research for theses on curriculum evaluation. According to the study, while most of the theses based on curriculum evaluation were completed at educational sciences division, a significant number of them were realized at basic education division.
Considering the research methods used in the theses completed in the field of the evaluation of curricula, quantitative methods were mainly preferred and qualitative and mixed methods were employed subsequently. In terms of the distribution according to years, it can be stated that quantitative research methods were much more preferred especially till 2012, and after 2012, qualitative and mixed methods were used more besides quantitative methods. In addition, the mixed research method which had been selected only seven times until 2012 was used ninety times from that year to 2019. This finding shows that the mixed method started to be selected for the theses on curriculum evaluation. When the postgraduate theses in the fields of CI and educational sciences were examined, it was found that quantitative methods were mainly preferred in general; there was a significant increase in the use of mixed methods especially after 2000, and qualitative methods were relatively less preferred (Eskici & Çayak, 2017;Gökmenoğlu, 2014;Gömleksiz & Bozpolat, 2013;Hazır-Bıkmaz et al., 2013;Yetkiner et al., 2019). Some factors such as quicker data collection through quantitative methods, the ability to determine the limits of variables more clearly, enabling generalizability thanks to the use of tools whose validity and reliability were already proved statistically, and the objectivity of the researcher may constitute the basis for the preference of quantitative research methods in the majority of the research. According to Patton (2014), in quantitative research, the perspectives and experiences of people can be placed into the predetermined answer categories each of which a certain number is assigned to. Thus, the method can measure the reaction of a large number of people on the research subject with a limited number of questions, can enable data to be compared and collected statistically, and provides a generalizable set of findings presented in a brief and concise way. The similarity of findings obtained through quantitative research and the need for reaching in-depth and different information about the same curricula could be the reason for the increase in the use of qualitative and mixed methods in recent years. According to Yıldırım and Şimşek (2013), human behavior can be investigated by a flexible and holistic approach through qualitative research and with this approach, the opinions and experiences of the individuals participating in the research can be reached. By using quantitative and qualitative methods together in mixed research methods, both the weaknesses of these methods can be eliminated and a multi-perspective can be provided to the researcher. In this way, more reliable and detailed information is obtained by gathering both statistics and words (Creswell & Plano Clark, 2015).
It can be said that survey and experimental designs were preferred and case study and descriptive research were also used in the theses on curriculum evaluation. In addition, mixed methods such as explanatory sequential, simultaneous, embedded, and exploratory sequential were applied. This shows that the preference of research methods in the theses is consistent with the design preference. Considering the research design preferences based on the years, survey designs were preferred mostly until 2012, but from then on, transition to experimental research and case study was realized. Also, after 2012, a significant increase in the use of designs within the scope of mixed methods occurred. Similar studies for the field of CI also revealed that survey (single and correlational) was generally preferred in the postgraduate theses and experimental design was often used as of the 2000s. Also, case study has become another design preferred in recent years (Aslan & Sağlam, 2017;Kozikoğlu & Senemoğlu, 2016;Schreglmann, 2016;Uysal, 2016). Similar trend is also seen in the studies for teacher training and educational sciences (Ayaz, Oral, & Söylemez, 2015;Eğmir, Erdem, & Koçyiğit, 2017;Karakoç, Özpolat, & Kara, 2018). The trend in research designs is in line with the tendency to prefer quantitative, qualitative, and mixed methods. Based on this, it can be said that the reasons for the trend seen in the preference of research designs are similar to the reasons for the tendency in research methods. It was found that questionnaires and interview forms were mostly used in data collection process in the theses included in the research. Attitude scale, skill and achievement tests, and document analysis form were also used as data collection tools in the theses. Regarding the distribution of data collection tools preference by years, questionnaires were mainly preferred especially until 2012. Later on, the trend shifted towards the use of attitude scale, skill and achievement tests, observation, and interview forms. Document analysis form was preferred at a similar rate in each period. When the trend in CI was considered, it was seen that some techniques, particularly questionnaire, such as interview, attitude scale, achievement test, observation, and document analysis were preferred. There has been a decrease in the use of questionnaires in recent years, and especially, interview, observation, and document analysis have also started to be preferred (Gökmenoğlu, 2014;Gömleksiz & Bozpolat, 2013;Kozikoğlu & Senemoğlu, 2016;Özüdoğru, 2018;Yetkiner et al., 2019). This indicates that questionnaires have an important role as data collection tool in the studies in CI. Additionally, it can be said that data collection tools varied after the 2000s and different tools such as interview, scale, observation, and document analysis were also preferred. The reasons why questionnaires are mainly used in the research studies may be that it takes shorter time and is easier to prepare and evaluate the questionnaire than the other tools and it is possible to apply the questionnaire to larger groups and in different ways (face-to-face, online, via e-mail, and etc.). According to Wolf (1988), questionnaires have specific advantages such as easy preparation, the possibility of applying quickly to much larger groups from different regions, and having lower cost when compared to the other data collection tools such as scales, interviews, and observations (Cited by Büyüköztürk, 2005). In recent years, data collection tools such as interview, observation and document analysis have been used. According to Yıldırım (1999), data collection techniques corresponding to the qualitative approach are more effective in investigating the research problems that require in-depth and comprehensive information.
Based on the sample chosen from the theses on curriculum evaluation, it was determined that both the instructors implementing the curricula and the students at the target group of the curricula were preferred in numbers close to each other. Also, curricula that evaluate in researches are chosen as sample within the scope of document review. In the distribution of the participants included in research sample according to their stages, it was seen that the curriculum evaluated was effective and, in this regard, the teachers and students at Stage 1 in primary education were preferred more. When the studies for postgraduate theses were examined, it was determined that teachers and students (at pre-school, primary, secondary and higher education levels) were preferred in the selection of sample and academicians were also included in the sample (Dündar & Meriç, 2017;Eskici & Çaylak, 2017;Gökmenoğlu, 2014;Küçükoğlu & Ozan, 2013;Özsoy, Bayrak Özmutlu, & Gündüz, 2017;Şahin, Calp, Bulut, & Kuşdemir, 2013). The main factor in preferring mostly teachers and students as sample in research may be related to that the population and sample are larger and more accessible than other groups and they are the primary sharers of the curricula.
Considering the types of curricula evaluated in the theses, it was determined that the curricula for the Primary Education Stage 1 and Stage 2 were particularly dealt, and the curricula for secondary and undergraduate education were also preferred primarily in evaluation. It can be said that a wide range of curriculum-based arrangements made within the MoNE (2005, 2006, 2011, 2013, 2015) and the CoHE (2007 had an effect on the fact that the theses written in the field of curriculum evaluation between 2007 and 2019 focused especially on those fields. It can be said that the curricula of primary, secondary, and undergraduate education are evaluated in similar studies (Akşan & Baki, 2017;Özüdoğru, 2018;Yetkiner et al., 2019). When the postgraduate theses were examined based on the disciplines, it was determined that the curricula of Science and Technology / Sciences, Social Studies, English, Science of Life, and Turkish at the Stage 1 and Mathematics, Science and Technology/Sciences, Social Studies, English, and Turkish at the Stage 2 in Primary Education, and Biology, Physics, Geography, Mathematics, English and Counseling in Secondary Education, and English Preparatory and Elementary School Teaching in Undergraduate Education were evaluated much more than the others. When the postgraduate studies conducted within the field of Curriculum and Instruction were analyzed, it was found that the same courses were addressed (Akşan & Baki, 2017;Aslan & Sağlam, 2017;Kurt & Erdoğan, 2015). In the choice of Science and Technology/Sciences, Social Sciences, English and Turkish Language curricula as the subject in more postgraduate theses, the fact that these courses appeal to wider age groups and can be addressed at many grade levels can be considered as an important factor. Only in one fifth of the curriculum evaluation postgraduate theses examined in the research (105 theses), the curriculum evaluation models were taken as basis. In this regard, the most preferred model is the CIPP (Context, Input, Process, Product) model developed by Stufflebeam. In addition to this, "Element-Oriented Curriculum Evaluation Model" developed by Erden and "Objective-Centered Evaluation Model" developed by Tyler were based on. The important point here is that the curriculum evaluation models have not generally been used in the postgraduate theses based on the curriculum evaluation. Similar results were also reached in different studies for postgraduate theses in the field of the curriculum evaluation. It was found that the CIPP model was the mostly used curriculum evaluation model in the studies and that Tyler's objective-centered evaluation model, Stake's congruence-contingency model and Eisner's educational criticism model was also preferred. However, it was also revealed that any curriculum evaluation model was not benefited in most of the theses (Akşan & Baki, 2017;Aslan & Sağlam, 2017;Gökmenoğlu, 2014;Özüdoğru, 2018). In this respect, it can be stated that curriculum evaluation models were not used in a great part of the curriculum evaluation studies carried out in Turkey in the last thirteen years. The fact that only 197 of the 586 postgraduate theses included in the research scope were written in the field of Educational Sciences and Curriculum and Instruction, the remaining theses were in the fields which do not include a curriculum evaluation course. Therefore, they were written in the fields which do not require specialization in that subject of study can be related to the lack of model use. This can be interpreted in the way that the mentioned theses without an evaluation model may be weaker than the ones in which a model is used.
In conclusion, a significant differentiation arises after 2012 in the research methods, designs, and data collection tools preferred in theses on curriculum evaluation from 2007 to 2019. In this context, there is a trend from quantitative methods to qualitative and mixed methods, from survey to experimental research and case study, from collecting data with questionnaires to collecting data with attitude scales, skill and achievement tests, observation, and interview forms. The curriculums developed by MoNE and CoHE and its' years are influence in the curriculums evaluated. In sample selection, the trend is related to the curriculum subject to the study. In new thesis studies on curriculum evaluation, especially pay attention to the trends in research methods, designs, and data collection tools. By using mixed methods and applying more than one data collection tool, evaluation studies can be multi-dimensional. Besides, curriculum evaluation models preferred very few in the theses on curriculum evaluation. For the more systematic research, using a significant curriculum evaluation model is essential.