This class has been a struggle for me.  The information was harder for me to digest and understand than most other classes.  At the beginning and towards the middle I felt despair, but after talking with Frank I started to grasp the concepts better.  I now know a lot more about educational research than I did four short weeks ago.  I don’t know how I will use it on a regular basis, but it is good to know that I have that knowledge to utilize in my teaching career.

Ecological validity: (p. 355, McMillan) generalizability to other settings, times, treatments, and measures

Sentence: The study had ecological validity because the findings were applicable to other situations.

Hawthorne effect: (p. 356, McMillan) individuals’ realization that they are subjects in a study

Sentence: Unfortunately, sometimes in research subjects become aware that they are being observed, creating a Hawthorne effect.

Reflection:

The purpose and nature of Discussion:

Interpretation of results in light of problem/hypothesis, theory, methodology (selection of subjects, measurement of variables, treatments), statistical processes used, and literature

Conclusions:

Limitations related to: subjects, situation, time, treatment, measures

Recommendations/Implications – usually in conditional language

We worked in groups to determine the possible limitations of the studies and interpretation of subject selection.  My group was not present, so I worked with the S.E.S. group.

What do I think is the most important part of the Conclusion/Discussion?  I think it is all important, but the findings and recommendations are the most useful.  I like how the findings are summed up and application is given.  I think that the Limitations section is also very important.  It helps me know how accurate or not the study might be.  It also shows the thoughtfulness and awareness of the researcher(s).

General comments on research:

Use of qualifiers

Why is this important?

What role does the generally conservative approach or researchers play?

What aspects of the research process surprise you?

Single-subject design: (McMillan, p. 241) individual behavior recorded before and after an intervention, use one or just a few subjects to study the influence of a new procedure

Sentence: The researcher decided to use a single-subject design to observe the effect of a new diet on the beluga whale.

Withdrawal design: (McMillan, p. 242) treatment removed after implementation, so the subject is back to the baseline behavior

Sentence: The researcher were using a withdrawal design, since they planned to stop the treatment at the end of the study.

Multiple-baseline design: (McMillan, p. 243) more than one subject, behavior, or setting

Sentence: The design was a multiple-baseline design because the research was being done with many single-subjects simultaneously.

Reflection:

The Advantages of Single-Subject Research Methods

Useful for:

Behavior modification research

diagnosing teaching and learning problems

studying classroom management methods

development of students’ skill

studying a problem in detail

A form of action research

Experimental control

It is a quantitative design.

multiple, reliable observations of student behavior

can be observation, questionnaire, test

can not have pre-test effect

must be able to give it frequently

must be reliable

Detailed description of the treatment

replication of the treatment effects

I do single-subject studies all of the time in my classroom.  It might not be a “formal” study, but I am observing reactions to what I do and what is expected in my classroom.  Often I base what I do off of what worked in the past.

Reliable observations of behaviors

Operational definition of to-be-observed (target) behaviors

Careful training of observers

Frequent checks of observer reliability

Control of observer bias

SINGLE-SUBJECT RESEARCH DESIGNS

AB designs

Basic to all single-subject research is “A-B” logic: (1 Problem – low internal validity)

Condition A: environmental conditions are constant, target behavior is observed to occur at consistent, stable rate

Condition B: one of the environmental conditions changes, corresponding changes in the target behavior observed

A-B-A design

B-A-B design

A-B-C-B design

Multiple Baseline Designs

are used when: you cannot withdraw or reverse a treatment, or you cannot demonstrate a treatment effect with an A-B-A design

Summary: 2 Basic Designs

ABA and Multiple Base

Both based on the concept of multiple measures across time and the AB sequence of treatment

Require: reliable frequent measurement, clear specification of observation, and inter-rater reliability

I do this for my students through my actions, expectations, and the running of the classroom.  I try to be a good example for my students, an example of how to treat others, how to broach difficult situations, how to work hard, etc.

My classroom expectations are stated clearly at the beginning of the year, and, as a class, we go through the expectations, talking about why they are in existence and whether or not this is reasonable.  The discipline that accompanies the breaking of the expectations is fair and fits the “crime”.  After discipline I assure my students that the slate is clean.  We start fresh.  Misbehavior does not define my students, and I tell them this.

My classroom is very inclusive, and everyone in it have a role to perform.  I teach.  My students learn and help each other.  My students work alongside me to keep the room going (putting supplies away, cleaning, etc.).  We are a team and all citizens of the classroom!

Ethnography: (p. 276, McMillan) in-depth involvement in a culture to describe naturally occurring behavior, an ethnographic qualitative study, an in-depth description and interpretation of cultural patterns and meanings within a culture or social group

Sentence: The research involved an ethnographic approach, studying to see how the Cherokee culture was affected.

Emic: (p. 283, McMillan) participant wording, information provided by the participants in their own words

Sentence: The interview was full of emic data, giving a better indication of how the participant really felt and thought.

Etic: (p. 283, McMillan) researcher representations of emic data

Sentence: The data the researcher had learned through the interview with the participant was stated as etic data.

Grounded theory: (p. 293, McMillan) theory generated from qualitative data

Sentence: The researchers came up with a grounded theory study in order to generate a theory.

Triangulation: (p. 296, McMillan) compares the findings of different techniques

Sentence: Using triangulation, the researchers decided which technique would work best for their research.

Reflection:

The larger the sample, the smaller the error (the variance).  The closer to the population mean you will be with your estimate.

Statistical Inference

Sources of Error

Different or the same?

Logic of the decision (you assume there is no difference) – p value (overlap of groups, confusion)

Be careful of small samples (< 10 for comparisons and < 30 for correlations).

Inference is expressed as a statistical probability.

Characteristics of Qualitative Research

Natural setting – no lab settings

direct data collection

rich, narrative descriptions (sort of like novels)

process orientation

inductive data analysis (takes the collected data and tries to find findings from it) (vs. deductive – coming from particular slant, researching focused on it)

participant perspectives

emergent research design (don’t know what they’ll do when they start)

TYPES of qualitative research

ethnographic studies

phenomenologival studies (go to the population and watch and study) (participants selected because of their experiences) (Data analysis: constant comparative, narrative analysis)

grounded theory (to discover or generate a theory) (iteration of: data collection, data analysis) (Referral of analysis to participants until: experience can be predicted, participants are satisfied with description)

case studies (in-depth analysis of one or more issues through the review and description of a “bounded system”) (can be qualitative or quantitative) (quite flexible) (constant comparative, thick description, narrative studies)

DATA analysis – I

Constant comparative model

Live with the data!

Unitize (segment) it into smallest sections

Begin to reassemble data into related groups

Develop rules for inclusion in each group

Compare new and existing data to the rules

Continue categorization and/or revision of the categories

DATA Analysis – II

Thick description (describe the experiences in great detail!) (try to help the reader feel as if they were there in the room during the experience) (Realistic!) (can be read almost like a novel – develop character, describe setting in depth)

Semiotic, narrative analysis

CREDIBILITY of Qualitative Research

Triangulation (conclusions supported from at least 3 sources including various combinations of: interviews, observations, document reviews)

Reliability (Member checking – making sure you got the right information from the people you talked to, research audit – has to do with constant comparison analysis)

INTERNAL Validity of qualitative research

Clear match between researchers categories and experiences of reality being shared.

Member checking

Research audit

Threats to internal validity

—–

There are fewer rules and guidelines in qualitative research than in quantitative studies.  It is more complex.  I think it is a more subjective form of research.  But it gives a richer, more human view than the quantitative.  It is good to always read both forms of research with a critical eye.

I think I would enjoy doing qualitative research more though.  I would be really interested in doing an ethnographic study.  Other cultures are very intriguing to me.  Too bad I don’t have the time (or resources)!

Inferential statistics: (p. 252, McMillan) infers characteristics of a population, necessary to understand the precise nature of descriptions, relationships, and differences based on the data collected in a study

Sentence: Today we learned about the different types of inferential statistics, parametric and non parametric.

Analysis of covariance (ANCOVA): (p. 264, McMillan) adjusts for pretest differences between groups

Sentence: The analysis of covariance was made before the study occurred in order to help the results be accurate despite the differences in the groups being studied.

Reflection:

Statistical inference is taking the information from a study and putting the findings onto the expectations for the larger group.

Sources of error: measurement error,

Inference tests tests the areas of overlap between 2 tests.

Inference Logic (Is the observed difference between 2 samples due to error, or due to a real difference in the populations they represent?)

Start with the assumption that there is NO difference in the populations represented by the sample (Null Hypothesis)

Null Hypothesis: Since we know there is some error involved in sampling and measurement, we start with the assumption that there isn’t really a difference between the groups – we assume that any observed difference is simply due to error.

Types of Inferential Statistics

Parametric Statistics (normal distribution, equal variances in each group, interval-level measurements)

Non Parametric Statistics (less power to detect differences)

*Both test a null hypothesis and report a level of significance.

TYPES OF TESTS I

t tests

One-tailed, v. two-tailed test

Independent v. dependent samples

Used to compare 2 group means.  Yields a point on a distribution that can be assigned a probability.

TYPES OF TESTS II

f test

Used to measure differences between 2 or more means

Post Hoc analysis used to determine where difference lies if there are more than three means.

TYPES OF TESTS III

Chi Square

Non Parametric

Used to test frequency counts in different categories

The null hypothesis is that there is no difference between the observed number and the expected number of frequencies in a category

CRITERIA FOR EVALUATING INFERENTIAL STATISTICS

Look for descriptive statistics to help interpretation of inferential statistics

Understand the difference between statistical and practical significance

Inferential stats provide no indication of internal or external validity

Power of a test depends on size of sample

Make sure appropriate stat is used

Be careful of small samples

Level of significance interpreted correctly

My group is a non-equivalent groups pretest post test design.

———-

Law of Large Numbers – The larger the sample, the closer to the population mean you will be with your estimate.

———

Samples have different means than the population.  Good to remember.  Studies will not completely accurately represent the population as a whole.

I really enjoyed the hands-on participation today with the M&M count.  Chocolate makes everything better!  And I know that I learn better through experience and visual/tactical learning.

Why do we do this?  We need BIG samples.  Lots of sources of error with lots of researchers.  There were variations in the samples (M&M colors)

Internal validity: (p. 220, McMillan) control of extraneous variables

Sentence: There was internal validity because the outside variables were not able to affect the study.

Subject attrition: (p. 223, McMillan) threat from loss of subjects, subject systematically drop out of or are lost from the study and their absence affects the results

Sentence: The results of the study were not accurate because of the large number of subjects that left it, causing subject attrition.

Statistical regression: (p. 223, McMillan) threat from change of extreme scores to those closer to the mean, the tendency of subjects who score extremely high or low on a pretest to score closer to the mean of both groups on the post-test, regardless of the effects of the treatments

Sentence: The study suffered because of the statistical regression of the subjects.

Factorial designs: (p. 234, McMillan) containing 2 or more independent variables, extensions of the designs that study more than 1 independent variable (or factor) and the interaction between independent variables

Sentence: Factorial designs are more complicated than other designs because there is more than one independent variable in play.

Reflection:

What does validity mean in relationship to a test?

Internal validity – focus on lack of control of extraneous variables

Can we really attribute the change observed to the change in the independent variable or are unknown variables at work?

External validity – the generalizability of the results.  Can we apply the results of this study to our situation?

The threats to external validity are more individual and unique than the threats to internal validity.

“Buyer beware” Look for similarity in:

Subjects (sample bias)

situations – is my situation the same as the research study?

time

treatments – might be too expensive

measures

Reflections:

Internal validity is an attempt to control known extraneous variables.

External validity has to do with generalization

True Experiments:

directly manipulate the independent variable

control for extraneous variables (random assignment to and of groups)

Various designs protect against various threats to validity

—–

In class today we worked more with nonlinguistic representations.  My group drew examples of “maturation”, “instrumentation”, and “diffusion of treatment”.  Thanks for this form of learning, Frank!  It really helps condense and bring material to life for me.  Maturation is when the subjects in the sample change during the course of the study.  Instrumentation is how the study is conducted.  We used the example of one teacher reading the instructions out for the test for one group.  In another group a teacher had the students read through the instructions on their own.  Diffusion of treatment is the independent variable of one group having an inadvertent affect on another group.

We began talking about different forms on experimental design.  These seem really interesting compared to non-experimental designs.  And my talk with Frank helped me understand the content a little better.

Reflection:

We talked about more kinds of research (continued from yesterday).

NON-EXPERIMENTAL RESEARCH

Correlation research: describes the relationship between 2 or more variables in a sample (Do women really have more trouble grasping math?)

Causal-comparative research: carefully matched groups, intervention carefully monitored as it unfolds; gender, school attendance, socio-economic background; during the fact

Ex post facto research: after the fact; designed to explore causality when manipulation of the independent variable can’t be done; depends on locating groups that are the same except for the independent variable

SURVEYS: Tools used in research; subject selected (usually probability sampling) and then asked questions about predetermined topic

Results used to predict what the population thinks about the topic (like political surveys, market surveys – what people will buy, nielson ratings – what you’re watching); can be cross sectional (all data collected at same time) or longitudinal (data collected from the same sample multiple times)

Sample is CRITICAL!

Topics for which non-experimental are the only methods likely to be use: class size, nutrition, recess, curriculum, length of day, motivation, substance abuse, culture, religion, homelessness, pass/fail, year-round school, ethnicity, class make-up, absences, private vs. public

—–

EXPERIMENTAL RESEARCH:

What makes a “true” experiment? Direct control of independent variables and control of extraneous influences

What does “true” mean in the question above?

What assumptions are behind a quantitative experiment?

What assumptions are behind a qualitative research project?

Can you do a qualitative experiment?

Experimental Designs: only way to demonstrate cause from quantitative perspective

Intervention Research (direct control over when and how much of the intervention the participants receive); “Control” vs. “Contrast” group (Contrast is better for education, not control)

How to control for things you don’t know? random assignment to groups, random assignment of groups to conditions

Experimental Validity:

What does validity mean in relationship to a test?

What might validity mean to an experiment?

2 kinds of validity: internal – is the design set up to demonstrate control over independent variable and extraneous influences

Threats to internal validity (chart on p. 226, McMillan)

——

Today I had a hard time focusing, partly because of the long (2-hr.) drive I had to take to get to school.  Traffic was awful!  But we learned a little more about non-experimental research and some about experimental research.  While my group’s project is non-experimental, I think if I were a researcher I would be more interested in conducting experimental research.  It’s sort of like a mystery, and the outcome is not as set.  The possibilities seem more endless and interesting to find out.

Relationship design: (p. 186, McMillan) how information is obtained

Sentence: The relationship design for the research project was qualitative in nature.

Multiple regression analysis: (p. 197, McMillan) combines several predictor variables, provides a single index of the predictive power of all the predictor variables together

Sentence: In order to simplify the information the researcher used multiple regression analysis.

Coefficient of multiple intelligences: (p. 197, McMillan) the combined correlation of several predictor variables (independent variables)

Sentence: The coefficient of multiple intelligences was a combination of independent and dependent variables.

Logistic regression: (p. 197, McMillan) combines several variables to predict a dichotomous outcome, used to explore the relationship between the explanatory or predictive variables and the outcome

Sentence: In order to see how their hypothesis related to the outcome of the study the researchers used logistic regression.

Reflection:

Definition of reliability

Consistency – If a measure were given over and over to the same person, would they get the same score all of the time?

Estimation of reliability

Test-Retest – Test is given once and then again later.  A correlation is developed between the two administrations.

Internal reliability – split half, one half of the test is compared to the other half

Inter-rater reliability – Do two people see the same thing? Useful for observation.  Come up with a percentage of agreement (in decimal form).

Need to have very specific kinds of things to look for.  Or spend a LOT of time together (a lot of training).

Very subjective.

VALIDITY: bound up with purpose (This test is valid for….)

Does the test do what it is ‘spose to do?  Involves characteristics of the test and of the purpose for which it is being used.  Tools need to be used for what they’re meant for.

Estimation

Face validity – Does the test do what it’s supposed to do? (Expert review of the test)

Construct validity – Factor analysis to examine underlying structure.  Are the constructs that make up the test congruent with the concepts behind the test?

Predictive or concurrent validity – compares test against a known test or measure in either a predictive or side by side comparison.

Predictive test examples: SAT or ACT – were supposed to predict student success in college

Standardized test (standard conditions of administration  (no “blah, blah, blah”) and interpretation)

2 Kinds: Criterion referenced (compare in particular criteria) and norm referenced (compared to others/people)

Standardized test quality:

* clear directions for administration (location, tone of voice or facial expression of teacher, time of day, in relation to eating time)

* representative norms (Are the norms really representative?)

- Think “sample”

- How have they been described?

- Age of norms (when was the norm sample sampled?)

- Demographics of the group

Test Review Includes:

Ethnicity, age, gender

Demographic information about the test

Your judgement on: standard conditions of administration (are the instructions clear?), norm group, reliability(how is it measured, measured more than 1 way?), validity (for what purpose would the test be used?))

Overall judgment of usefulness of the test

Types of Educational Measurements

Observations (lab, structured, inference – high or low, observer effects – bias, contamination, halo effect (people wanting to please the observer))

Interviews (structured, unstructured, semi-structured, interviewer effects (interviewer characteristics, interviewer bias, …))

Questionnaires

Tests (criteria referenced, locally developed, large scale standardized tests, norm referenced (raw scores, standard scores, percentiles, grade equivilants))

Non-Experimental Research

Descriptive research (describe current situation or circumstance) (involves descriptive statistics)

Comparative research (compares 2 or more groups on a variable) (described in general terms (larger than, smaller than, similar, etc.)) (does not establish a causal relationship – they existed already) (non-distortion of impact/relationships, graph on p. 192)

Correlation research (describes the relationship between 2 or more variables in a sample) (can’t be used to demonstrate cause/effect) (includes more than 1 variable) (Correlation table show results – p. 195) (uses correlation coefficient to describe size of direction (-1.0 to 1.0)) (multiple regression studies combine several predictor variables) (can be used to make predictions by association)

—–

In class we heard a lot about different forms of research, some about the test reviews we will be doing, different types of educational measurements, and about validity vs. reliability.  It was a very full day!  I am not quite sure how all of the information ties together, but I appreciate the amount that was covered.

In my classroom I tend to use verbal-linguistic and visual-spatial the most.  But my students always develop their own skills through intrapersonal and interpersonal strategies.  I have used musical by having students focus on the music and create how the music feels to them.  What emotion does the music bring out in them?  Students use bodily-kinesthetic through sculpture (molding clay) and gesture drawing (conveying movement through drawing).  Some logical-mathematical is used for ratios of paint to chemicals or paint to water or firing clay, etc.  But in all of these areas I am excited to go beyond what I have done in the past and to brainstorm new ways to incorporate different multiple intelligence strategies.