Evaluation of the Professional Development Program in Digital Material Design According to the Kirkpatrick’s Model *

Today's developments have revealed the necessity of using digital materials in educational environments. The digital material design competencies of the on this subject are essential. In this context, it is necessary to develop and evaluate a professional development program. The purpose of the current study is an evaluation of the professional development program developed for the digital material design according to Kirkpatrick’s Model. In the current study, a digital materials design professional development program consisting of 10 modules and 21 subjects was developed. Kirkpatrick’s Program Evaluation Model was used in the evaluation of the program. In this context, teachers' reactions to the program, their learning levels, their ability to transfer the knowledge they have acquired to the classroom environment, and the target audience's opinions on the practices were determined. “The Digital Materials Design Competences Scale” and "The Scale of Self-Efficacy Perception of Technology Integration" was used to collect quantitative data. Interviews and the observation technique were used to collect qualitative data. According to the findings, it was determined that the teachers expressed positive opinions about the program and that they were satisfied with the program. It was concluded that after the professional development program, the teachers' digital material design competencies and their self-efficacy perceptions of the integration of technology developed. The follow-up evaluation study determined that teachers could transfer the knowledge and skills they gained in the professional development program to the actual classroom environment, and students expressed positive opinions about these practices.


Introduction
Advocating the idea that "Education is not preparation for life, but life itself," Dewey (2015) draws attention to the effects of social changes on education in his work "School and Society." According to him, education must undergo a complete change in order for social changes to have meaning for life. The frequent use of information and communication technologies in searching, organizing, storing, developing information, and solving problems causes every social system to expect educational institutions to train individuals who can use technology (Kurt, 2013). Our world and our society no longer need people who know the information but who produce information and know-how to access information (Çağlar & Reis, 2007). Therefore, in the current information age, individuals are expected to follow science and technology developments and apply them daily. With the transformation that society has undergone, the technological competencies expected from individuals have made the use of technology almost compulsory in educational environments, and the concept of technology integration has gained importance. Van-Melle, Cimellara, and Shulha (2003) defined technology integration as the process of using and disseminating information and communication technologies based on applications in accessing information and communicating to increase learning.
Teachers have a great responsibility in the successful accomplishment of technology integration. The development of teachers' knowledge and skills on this subject is essential in getting the return on investments. In this direction, integrating technology into learning-teaching environments in today's education system is seen as one of the vital teacher competencies (Brush & Saye, 2002). Teachers' preparation of digital materials required by techno-pedagogical education suitable for their lessons and students can be expressed as the first step of this process. Today's students are now described as digital students, and it is stated that these students adopt and use technology to explore, question, and improve their learning and contribute to the knowledge of others (Smaldino, Lowther, Mims, & Russell, 2015). In this connection, it can be said that the materials used for today's learners, in whose lives technology occupies an important place, should be digitized.
In addition to the technological developments experienced today, some critical situations have brought digitalization into education. The pandemic, which has recently affected the world, has significantly affected people's health, psychological, economic, social life, and education. The prominent effect of this in the field of education is that students stay away from the learning environments in their schools and participate in distance education from their homes where they go through the isolation process (Koçoğlu, Ulu Kalın, Tekdal, & Yiğen, 2020). Due to the Coronavirus (Covid-19) pandemic, many countries such as China, the USA, Italy, Spain, France, Korea, Turkey, and Germany were forced to interrupt their face-to-face educational practices. At the global level, open and distance education applications have come to the fore as a fundamental learning resource to find a solution in the face of the failure to implement formal education practices (Can, 2020). The involvement of educators in the distance education process with their students has increased the importance of using digital materials in these environments. In order for the distance education process to continue effectively, it has become necessary for teachers to develop digital materials that they can use in virtual classrooms.
It is stated that learning experiences equipped with digital teaching materials will enable learners to be fast, effective, and willing to have access to information (Yavuz-Konokman, 2019). Teaching materials transmitted through digital media are defined as digital materials, and digital textbooks, applications, and online supplementary resources are shown as examples of these materials (Edson & Thomas, 2016). When it comes to digital material, animations, simulations, presentations, digital texts, and videos come to mind (Taşlıbeyaz & Karaman, 2015). Karademir (2018) defined digital teaching material as any material prepared for teaching purposes through digital media such as computers or mobile devices.
It is stated that the interest in the technology of today's technology-literate students will make it easier for them to be interested in the information provided with digital teaching materials and that learning experiences designed with these materials will contribute to their positive attitude towards learning (Yavuz-Konokman, 2019). However, when the relevant literature is reviewed, it is seen that teachers have various problems in the use of digital materials and digital materials design (Akıncı, Kurtoğlu, & Seferoğlu, 2012;Dursun, Kuzu, Kurt, Güllüpınar, & Gültekin, 2013;Kaya & Koçak Usluel, 2011;Kayaduman, Sırakaya, & Seferoğlu, 2011;Keleş, Dündar Öksüz, & Bahçekapılı, 2013;Sarıtepeci, Durak, & Seferoğlu, 2016;Soydan, 2018;Yadigaroğlu, 2014;Yörük, 2016). In these studies, it has been revealed that; • Teachers do not have sufficient knowledge and skills in digital material preparation, • Teachers experience difficulties in obtaining digital materials, • Teachers' digital material needs are not met sufficiently, • Teachers use ready-made materials in their teaching processes, • Preparation of digital materials by teachers is adequate, • Teachers should be provided with expert support in digital materials design.
In related studies, it is recommended that a professional development program should be given by field experts so that teachers can prepare digital materials. The literature findings show that there is a general need for teachers in this regard. Usluel, Mumcu, and Demiraslan (2007) state that in-service training is an essential requirement on teachers' ability to use instructional technologies. Researches also support this view, and it is stated that teachers need professional development program on issues such as using technology in education, using the internet for educational purposes, and using teaching materials effectively (Gökmenoğlu, 2012;Sarıtepeci, Durak, & Seferoğlu, 2016). In this context, a professional development program was developed to improve teachers' competencies in digital material design, and its effectiveness was evaluated. The evaluation of this professional development program, which was developed as the focus of the research, was taken, and the program was evaluated according to Kirkpatrick's Model. The purpose of the current study is an evaluation of the professional development program developed for the digital material design according to Kirkpatrick's Model. The four-stage model proposed by Kirkpatrick and based on the results is the most frequently mentioned model in the relevant literature in terms of evaluating training programs and is used mainly in adult education and in-service training (Kirkpatrick, 1976). After implementing a model professional development program, involves first determining the positive or negative opinions of the participants about the course or seminar, then evaluating the acquired knowledge and skills, and then determining what changes have occurred in teachers' teaching methods. In this context, it is stated that Kirkpatrick's Model can be used effectively in the evaluation of adult and professional development programs (Sönmez & Alacapınar, 2015).
To this end, answers to the following questions were sought: The use of digital materials in today's educational environments is an element that will positively affect students' learning and contribute to technology integration in education. Teachers must have the necessary knowledge and skills in this regard. It is not enough for teachers to use only ready-made digital materials for them to realize techno-pedagogical education. For techno-pedagogical education to be implemented, the teacher is expected to design digital materials suitable for his/her students, course content, and gains. This issue was also mentioned in the "2023 Education Vision Document (Ministry of National Education [MoNE], 2020)", which includes the objectives to be achieved in education in order to raise individuals suitable for the age. The document contains targets for digital content and skill-supported transformation in our country. These goals are explained with the justification that teaching processes require a digital transformation per students' lives. Emphasis is placed on the importance of teachers developing and using digital teaching materials. In line with these developments, it is an actual situation that today's teachers have the skills to prepare and use their digital materials. In the research, a professional development program for digital material design was developed, and it was aimed to contribute to the digital material design competencies of teachers with the program. In this context, it is thought that the research will contribute to the field by developing and evaluating a program for the knowledge and skills needed in today's educational environments.

Research Method
The mixed-method was used in the current study. In the mixed method, quantitative and qualitative research methods are used simultaneously or sequentially (Teddlie & Tashakkori, 2010). Creswell and Plano-Clark (2015) state that mixed-method research will help answer questions that cannot be answered with qualitative or quantitative research methods alone. A multi-stage mixed model was used in the research.

Study Group
In the current study, there are two different study groups. The teachers participating in the professional development program developed with digital materials design constitute the first study group. The maximum variation sampling method was used in the study. Teachers in different branches from primary, middle, and high school levels participated in the professional development program. Demographic features of these teachers are given in Table 1. Teachers participating in the professional development program were coded as P1, P2, P3 ... P20. This coding was used to give quotations from interviews with teachers. The second study group in the current study comprises the teachers participating in the follow-up evaluation and their classes. Information about the participants of the follow-up evaluation is given in Table 2. As shown in Table 2, one of the teachers participating in the follow-up evaluation is working in a high school while the other is working in a middle school. A math lesson of the high school teacher A delivered to 9th-grade students was analyzed within the follow-up evaluation study. On the other hand, a middle school teacher B delivered to 8th-grade students was included in the follow-up evaluation study. Interviews were conducted with 13 students (7 students from teacher A and six students from teacher B).

Data Collection Tools
In the current study, "The Digital Materials Design Competences Scale (DMDCS)" developed by Göçen Kabaran (2020) and "The Scale of Self-Efficacy Perception of Technology Integration (SSEPTI)" developed by Wang (2004) and adapted to Turkish by Ünal (2003) were used to collect quantitative data.
In the current study, interviews and observation techniques were used to collect qualitative data. Interviews were done with the teachers participating in the professional development program and with the students of the teachers who participated in the follow-up evaluation. The semi-structured interview form prepared by the researchers was used in both interviews. The validity and reliability of the semi-structured interview forms were checked through expert opinion. There are five questions in the form prepared for teachers and four questions prepared for students. During the follow-up evaluation phase, the observation performed in the teachers' classrooms was carried out by one of the researchers. The unstructured observation form prepared by the researchers was used in conducting the observations.

Data Collection and Research Process
Firstly, the professional development program was developed. The literature states that the Taba-Tyler Model and System Approach Model are primarily used to develop in-service training courses (Çam, 2018). In the current study, these two models were used in the development of the professional development program. Planning, design/development, implementation, and evaluation stages were involved in the program development process. In the planning stage, need assessment studies were conducted; in the design/development stage, program design and pilot application were conducted; in the implementation state, the implementation of the developed program with the participation of teachers and data collection were performed, and in the evaluation stage, analysis of the data was made, and effectiveness of the program was evaluated. The program developed in the current study consists of 10 modules and 21 subjects. The program was implemented face-to-face with the participation of teachers on 24-28 June 2019. The program's content included various subjects that teachers can use in their teaching processes, such as online digital materials, mobile-supported digital materials, digital design, digital games, digital measurement, and evaluation tools. A follow-up evaluation study was conducted with two volunteer teachers who had participated in the professional development program. Kirkpatrick's model was used to evaluate the program in the study. The model has four primary stages: reaction, learning, behavior, and outcome/effect (Kirkpatrick, 1976). Thus, the effect of the professional development program was evaluated following the stages included in Kirkpatrick's Model.  Kirkpatrick & Kirkpatrick, 2006).
The quantitative data collection tools used in the study were applied face to face to the teachers at the beginning and the end of the program. At the end of the program, interviews were done with the students at the stage of follow-up evaluation with the teachers. All interviews were conducted as a focus group and face to face. Participatory and non-participatory observations were made during the professional development program and the follow-up evaluation phase.

Data Analysis
In analyzing the data obtained from the scales, the assumptions of normality and homogeneity were primarily checked. It was thought that the Shapiro-Wilk test was suitable for the normality test since the sample size was smaller than 50 (n = 20) in the current study (Büyüköztürk, 2018). Along with the normality test, the Skewness and Kurtosis values were also examined. Besides, the results of Levene's Test for Equality of Variances were used to test the homogeneity of the distribution. Then, it was decided to use the dependent samples t-test, one of the parametric tests, to analyze the data Reaction obtained from the scales. To determine the effect size in the current study, the Cohen-d formula, which calculates according to the difference of group means, was used (Pallant, 2016).
Qualitative data obtained in the current study were subjected to descriptive and content analysis. The data obtained in the descriptive analysis are summarized and interpreted according to previously determined themes. The purpose of this analysis is to present the findings to the reader in an organized and interpreted form (Yıldırım & Şimşek, 2016). Content analysis is "revealing the meanings remained hidden in texts or transcripts or the messages desired to be communicated there following a certain system in the form of concepts and categories and then conducting a quantitative and qualitative analysis of these concepts and categories" (Güler, Halıcıoğlu, & Taşğın, 2015, p. 333). For this reason, in the analysis of qualitative data in the current study, these two approaches were used together, and data were analyzed in depth.
In the research, validity and reliability studies were made for quantitative and qualitative dimensions. For the quantitative dimension, expert opinion was received, normality assumptions were met, and the reliability coefficient was calculated. In the qualitative, necessary procedures have been made for credibility, transferability, dependability, and confirmability. For teacher interviews, the percent of agreement value for the coding made by different researchers was calculated as .91, and the percent of the agreement for the coding made by the researcher at different times was calculated as .92. For student interviews, the percentage of agreement for the coding made by different researchers was calculated as .88, and the percent of the agreement for the coding made by the researcher at different times was calculated as .90 for student interviews.

Ethics Committee Approval Information
This study was granted ethically appropriate with the decision of Muğla Sıtkı Koçman University Human Research Ethics Committee, dated 04.03.2019 and issued 27.

Teachers' Reactions to the Program
The reactions of the teachers participating in the program were determined through the data collected from the interviews. The opinions of the teachers were gathered under four themes: "The Positive Aspects of the Program," "The Usability of the Information Learned in the Program," "The Negative Aspects of the Program," and "Suggestions for the Program." It has been determined that the theme of the positive aspects of the professional development program consists of four categories: digital materials, process, training, personal and professional contribution. The categories and codes related to the theme of the positive aspects of the program are given in Table 3. Helpful 3 Conducive to the effective delivery of lessons 2

Personal and Professional Contribution
Design skills 16 Self-confidence 7 Professional skills 4 The teachers stated that they did not know the digital materials included in the content of the professional development program before and that digital materials were new, various, pleasant, and valuable. They stated that the implementation process of the professional development program met their expectations, was enjoyable and beneficial, that they worked in collaboration with other participants and were able to raise their awareness. The teachers stated that the professional development program shared helpful resources with them, encouraged and motivated them, helped them cope with their difficulties, and was conducive to the effectiveness of their lesson delivery. The teachers stated that the professional development program they participated in contributed to designing and using digital materials, increasing their self-confidence and competencies to meet students' expectations. Some excerpts taken from the teachers' opinions about the positive aspects of the professional development program are as follows: It has been determined that the theme of usability of the information learned in the professional development program consists of four categories: applicability, intended use, features of digital materials, hardware, and network infrastructure. The categories and codes related to the theme of the usability of the information learned in the program are given in Table 4. The high majority of the teachers stated that the digital materials included professional development programs are applicable in teaching processes. They stated that they could use the digital materials learned within the professional development program's scope to attract the students' attention and attend to them. Teachers stated that they could apply the digital materials they learned in the program during the teaching process because they are effective, enjoyable, and accessible. Some teachers stated that they could not apply it because the smartboard in their school is broken and there is a problem with internet access. Some excerpts taken from the teachers' opinions about the usability of the information learned in the professional development program are as follows: P13: "I am an English teacher in primary school; since my students are young, I need to draw and keep their attention to the smartboard while teaching something; their attention span is also short. I will use most of them and even create variety." P15: "I can prepare crossword puzzles; I can use a 2D code. Students find philosophy a bit boring; therefore, I need to draw their attention by bringing some excitement to lessons." P3: "I think Kahoot is very feasible and not difficult to use. Preparing word cloud was very nice; I will use it for sure. The 2D code was wonderful. I will use it at the end of each unit. I want to use 2D code outside the class; when it is summer, children get easily bored, and I also get bored. Therefore, I will use it outside the class." P14: "In our school, the boards are broken, and there is no internet, so I will not be able to apply them." It has been determined that the theme of the negative aspects of the professional development program consists of two categories: process and participant characteristics. The categories and codes related to the theme of the negative aspects of the professional development program are given in Table 5. Teachers stated that the implementation time of the professional development program was the seminar period, but instead of attending the seminar at the school, it was necessary to attend the professional development program throughout the day. The duration of the program was found insufficient by some teachers. Some of the teachers participating in the professional development program had problems with foreign language as some of the applications in the program were in English. However, some teachers stated that technical competence was not considered in selecting the participants, and therefore there were problems in the process due to the level differences. Some excerpts taken from the teachers' opinions about the negative aspects of the professional development program are as follows: P4: "Coming here from the seminar was tiring for us because there were things we had to do at school, too. Nevertheless, I wish we had always been here…." P20: "More time could have been allocated to the training program." P8: "Some of the applications were in English, which was challenging for me." P5: "There were differences in terms of technological competence. It caused problems from time to time. Some teachers experienced problems in having access to the related sites." It has been determined that the theme of suggestions for the professional development program consists of two categories: process and participant characteristics. The categories and codes related to the theme of the suggestions for the professional development program are given in Table 6. Teachers stated that the implementation time of the professional development program should be the seminar period. However, it would be more beneficial to attend this program and not go to school during the seminar period, that the practice should be increased by decreasing the content a little, and that this program could save time if implemented through distance education. They also suggested that groups should be determined considering the level differences and branches of the participants. Some excerpts taken from the teachers' opinions about the suggestions to be made for the professional development program are as follows: P19: "This program was handy for us. However, in future training programs, teachers can participate in this program instead of going to school during the seminar period. In this way, it will be more useful." P11: "…In my opinion, content can be reduced. Through assignments, we can do more practice." P2: "The programs were all very nice. If there are studies to be done by universities through distance education, both participants will be more, and efficiency will be enhanced. It would be nice as I would not always have to come here." When a general evaluation is made, it can be said that the teachers expressed positive opinions about the content of the professional development program, the digital materials used, the instructor, and the process. During the interviews, teachers stated that they learned new and various digital materials in the professional development program, that the process met their expectations and benefited them, and that their personal and professional competencies improved. Almost all of the teachers stated that the information they attained could be used in teaching processes. Although a few teachers gave negative opinions about the process and participant characteristics, this does not indicate that they are not satisfied with the program. They stated that they offered these opinions as suggestions for making the professional development program more qualified. When these findings obtained from interviews with the teachers are evaluated, it is seen that the teachers' reactions towards the professional development program are generally positive, and they are satisfied with the program.

Changes in the Teachers' Competences of Digital Materials Design
To understand the changes in the participants' competencies of digital materials design, "The Digital Materials Design Competences Scale" was applied to the participants at the beginning, and the end of the training, and their pre-test and post-test scores were calculated. The relevant results are given in Table 7. When Table 7 is examined, it is seen that there is a statistically significant difference between the participants' pre-test and post-test mean scores of the digital materials design competencies scale sub-dimensions (p < .05). When the mean scores are examined, it is seen that the participants' post-test mean scores are higher than their pre-test mean scores.

Changes in the Teachers' Self-Efficacy Perceptions of Technology Integration
To examine the changes in the participants' self-efficacy perceptions of digital materials design, "The Scale of Self-Efficacy Perception of Technology Integration" was applied to the participants at the beginning. The end of the training and their pre-test and post-test scores were calculated. The relevant results are given in Table 8. It is seen from the Table 8 that there is a statistically significant difference between the participants' self-efficacy perception of technology integration pre-test and post-test mean scores (t = -3.815; p < .05). When the mean scores are compared, it is seen that the participants' post-test mean scores are higher than their pre-test mean scores.

Digital Material Applications that Teachers Perform in the Classroom Environment
In order to examine the extent to which the teachers could transfer the information and skills they had acquired in the professional development program into the classroom, observations were made by the researcher in three lessons. Teacher A designed three different digital materials, which he/she had not known before and learned within the scope of the professional development program (Photograph 2). Since the teacher preferred digital materials that required students to use computers, he/she conducted his/her lessons in the school's computer classroom. Teacher A used Kahoot, a digital questionnaire (Poll Everywhere), and a quiz show (LearningApps) in his/her lessons.

Photograph 2. Digital materials application of the teacher A
Teacher B implemented three different digital materials design applications that he/she had learned in the professional development program (Photograph 3). The teacher performed the digital material applications in his/her classroom. Since applications required students to use mobile phones, the teacher asked them to bring their phones to the lesson before the lessons she would deliver. Teacher B used Kahoot, 2D code, and puzzle applications in his/her lessons.

Photograph 3. Digital materials applications of the teacher B
According to the findings obtained from the observation data, it was determined that teachers could use digital materials effectively in the classroom. Besides, when the observation findings were examined, it was seen that the students' reactions were generally positive, and they actively participated in the process. Since the students did not have any problems with the digital materials, they did not receive technical support from the teacher during the process. In light of these findings, it can be argued that teachers A and B could transfer the information and skills they had acquired in the professional development program to the classroom environment.

Students' Opinions about the Digital Materials Applications
In the outcome/effect stage of Kirkpatrick's program evaluation model, the program's effect on the target audience is investigated. In this context, interviews were conducted with students in the two teachers who participated in the follow-up evaluation study. The answers given by the students to the questions were grouped under three themes: "The Teacher's Use of Digital Materials," "Effects of Digital Materials," and "Suggestions for the Use of Digital Materials." It has been determined that the theme of teacher's use of digital materials consists of two categories: intended use and materials used. The categories and codes related to the teacher's use of digital materials are given in Table 9. Students stated that the teachers (A and B) used digital materials for evaluation, reinforcement, and drawing attention. They stated that their teachers (A and B) used digital materials such as Kahoot, video, presentation, puzzle, game, 2D code, questionnaire, and competition. Excerpts from the students' opinions about the teacher's use of digital materials are given below: AS3: "Our teacher prepares questions about what we have learned at the end of the lesson. He/she asks us about these, and we answer them on the materials. Thus, he/she evaluates us." BS4: "…Our teacher uses such different applications as they motivate us to listen to the lesson more carefully…." AS3: "He/she uses materials such as Kahoot, smart boards, games, puzzles, competitions, videos." BS1: "He/she uses 2D code. We like this a lot. We use Kahoot a lot. We play games on the smartboard…." It has been determined that the theme of effects of digital materials consists of three categories: affective effects, supporting learning, and encouraging participation in the lesson. The categories and codes related to the theme of effects of digital materials are given in Table 10. Students stated that digital materials made them enjoy the lesson, aroused excitement in them, attracted their attention, increased their motivation, and made them more curious. They stated that digital materials enabled them to learn by having fun, facilitated learning, increased the retention of information, drew their interest to the lesson, and encouraged them to think creatively. Students stated that digital materials motivated them to participate in the lesson actively, increased their interaction with their teachers and friends, and made them more eager to participate in the lesson and take responsibility for the activities. Excerpts from the students' opinions about the effects of digital materials are given below: AS7: "Math is too hard. I am already bored; I always sit in the back row in the class. Nevertheless, when there are such activities, I enjoy it." AS6: "For example, the teacher asks questions to us with Kahoot. After every question, the name of the winner appears on the board. When I am the winner, I feel motivated." BS3: "When we do such activities, we love the lesson more, and we want to do more of such activities." BS4: "We have much fun while applying them. We learn more easily …." BS6: "I understand more easily with these materials…." AS1: "We all participate in the lesson. I think they motivate everyone in the class to participate in the lesson…." AS3: "When we play/compete as a group, interaction within the classroom increases." It has been determined that the theme of suggestions for the use of digital materials consists of two categories: the state of use and features of digital materials. The categories and codes related to the theme of suggestions for using digital materials are given in Table 11. Students suggested that digital materials should be used more, that they should be used in each class, and that all teachers should use them. They suggested using game-based, collaborative, interactive, and mobile digital materials in the teaching process. Excerpts from the students' opinions about the use of digital materials are given below: AS2: "I think these applications should be used more because the quality of the lessons gets high." AS6: "I want them to be used in all lessons, not just math." AS7: "I think we have more fun when we play. Such materials should be used." BS6: "I think the materials we play with are better. I want it to include games and competitions." AS3: "There may be things that encourage every student to participate in and help each other…."

Discussion, Conclusion, and Implications
According to the findings, it was concluded that the teachers participating in the professional development program expressed positive opinions about the content of the program, the digital materials used, the trainer, and the process. In the study conducted by Karademir (2018), it was found that the training received to develop digital teaching materials positively affected teachers. When teachers saw that they were successful in the education process, their prejudices and resistance to developing digital teaching materials decreased. In their study, Sánchez-García, Marcos, GuanLin, and Escribano (2013) determined that teachers showed a high level of satisfaction with the program due to their development own teaching skills. In this context, it is seen that the results obtained in the current study are similar to the results of different studies. Today, designing digital materials that can be used in teaching processes has become necessary for educators, and a training need has arisen on this subject. It can be said that positive reactions were given to the professional development program prepared based on this need, as the program included various digital materials, there was new information and applications in the program, and it allowed teachers to develop themselves and acquire competencies that would contribute to their professional development.
According to the findings obtained in the current study, it was determined that the teachers' digital materials design competencies and scores are taken from the sub-dimensions varied significantly after the professional development program. This result shows that the implemented professional development program contributed to the digital materials design competencies of the teachers. The professional development program contributed to developing the teachers' design competence, technical competence, techno-pedagogical competence, application and evaluation competence, and general digital materials design competencies. Similarly, in the study conducted by Gisbert-Cervera and Lázaro-Cantabrana (2015), it was concluded that a training program with ICT content for the professional development of teachers improved the digital competencies of teachers. Besides, in the literature, it has been determined that the results obtained from the studies carried out in order to foster the professional development of teachers in subjects such as the use of TPACK and ICT contributed positively to teachers' competencies (Guzey & Roehrin, 2009;Harris & Hofer, 2011;Kaleci, 2018;Yadigaroğlu, 2014).
According to the study's findings, it was determined that the teachers' perception of self-efficacy towards technology integration significantly varied after the professional development program, and their mean scores at the end of the program were found to be significantly higher. This result shows that the professional development program applied contributed to teachers' perceptions of selfefficacy towards technology integration. Perception of self-efficacy towards technology integration is defined as teachers' self-confidence in using technology effectively in learning environments (Nathan, 2009). The relevant literature states that teachers' characteristics such as perception, attitude, belief, and self-confidence are important in technology integration (Ertmer, 1999;İnan & Lowther, 2010;Kaya & Koçak Usluel, 2011). It is crucial to positively develop teachers' attitudes and thoughts on technology integration in education in this context. In the current study, the development of self-efficacy perceptions of technology integration is seen as a result that will contribute to teachers' ability to transfer the knowledge and skills they have gained in the program to the classroom environment and to integrate technology into teaching processes.
According to the findings obtained from the observations made in the classroom of two teachers who have participated in the professional development program, it was concluded that both teachers were able to transfer the competencies they had acquired in the program to the classroom environment. Both teachers prepared their applications regarding the digital materials they learned in the professional development program without supporting and using them in their classrooms. In the study conducted by Kaleci (2018), the follow-up evaluation of two teachers who have participated in the professional development program was made. It was determined that they could transfer the information they had acquired in the program to the actual classroom environment. Karademir (2018) stated that teachers' basic computer skills and self-efficacy in developing digital teaching materials affect their development. It can be said that the development of digital materials design competencies and self-efficacy perceptions of technology integration of the teachers have participated in the followup evaluation after the professional development program has an effect on digital materials design and use in the teaching process. In the professional development program, besides the teachers' acquiring the necessary knowledge and skills in digital materials design, it is thought that the instructor's demonstration of sample practices for the teaching processes of each digital material and the exchange of ideas with the teachers on how to apply digital materials in the classroom contributed to this result observed in the follow-up evaluation.
According to the students of the teachers who participated in the follow-up evaluation study, the teachers used various digital materials that they had learned in the professional development program in the classroom. The students stated that the teachers used these materials for evaluation, increasing retention, and drawing attention. These results show that the teachers used the knowledge and skills they had acquired in the professional development program in their classrooms. Besides, the fact that teachers used digital materials for different purposes shows that they used these materials at different lesson stages. Taşlıbeyaz, Dursun, and Karaman (2018) stated that digital materials are used in educational settings to make more effective and sustainable learning possible. In light of the findings, it is thought that both teachers continue to apply the digital materials they learned in the professional development program in their classes in order to make the teaching process more effective.
It was also concluded in the current study that digital materials had effects such as making the lesson more enjoyable, motivating students, attracting their attention, and arousing their excitement and curiosity. Another finding is that digital materials supported students' learning. A study reported that digital materials increased interest in the lesson, eliminated monotony, appealed to different types of intelligence, and increased self-confidence and motivation (Soydan, 2018). In the study conducted by Yağcı (2017), it was determined that computer-aided instruction, which included digital materials such as video, animation, slide, and computer-based tests, increased the academic achievement of students and the retention of the learned information. In another study, it was concluded that the use of digital teaching materials increased students' positive attitudes towards the course (Yang, Weng, Yang, & Wu, 2014). Karademir-Coşkun and Alper (2019) stated that digital learning materials are effective in focusing attention and concentration. It is thought that digital materials have positive effects on students in terms of addressing more than one sense of students and providing education based on individual differences. Besides, it can be said that students' interest in technology affects their thoughts about these materials.
It was concluded that the students want more digital materials to be used and these applications to be conducted in each lesson. Besides, it was concluded that students want to use digital materials that are game-based, cooperative, interactive, and include mobile applications. The positive effects of digital materials on students can be seen as to why students want these materials to be used more. It is expressed as an expected result that students mostly prefer game-based materials that make learning while having fun possible because students stated that when they learn with fun, the lessons are better for them and what they learn is more permanent. In the study conducted by Özer (2016), it was determined that game-based digital materials had positive effects on students. These materials increased students' interest in the lesson, contributed to cooperative learning, facilitated learning the lesson, and helped the lesson foster active learning. In the study conducted by Kula and Erdem (2005), students stated positive opinions about an instructional computer game and stated the elements in the game such as hint, bonus earning, collecting points, scoring, and facilitators as motivating. It also seems possible that the motivating elements in game-based digital materials are why students prefer these materials more.
The findings obtained within the scope of the research are limited to the data obtained by data collection tools from the teachers in the study group. The research is limited to the activities of the professional development program developed for digital material design. It has been determined that the professional development program developed in line with the results obtained within all these limitations is effective. Krueger, Hansen, and Smaldino (2000) stated that teachers have a crucial role in integrating teaching-learning processes with technology. Schoepp (2005) stated that there is a lack of training on using technology in teachers' teaching process, which is one of the most significant difficulties to be encountered in terms of technology integration in education. In this context, it is thought that the program developed with the research made will contribute to the solution of the problems encountered in the use of technology and digital material design mentioned in the literature. The research sets an example for professional development programs to develop teachers' digital material design competencies.
In line with the results obtained, it may be suggested to organize professional development programs by experts in digital material design and ensure the continuity of these programs. In the studies, programs with different contents can be prepared to improve teachers' digital material design competencies. Professional development programs can be developed in differ as blogging, website preparation, coding, artificial intelligence. To determine the needs in the studies to be carried out, with questionnaires and interviews, class observations can be carried out to examine the current situation of the teachers in depth. The developments in teachers' proficiency on this subject can be examined in detail by conducting more extended studies.