Program evaluations

How can the reliability of an outcome evaluation scale be determined? How valid is this scale? Does the scale measure what it purports to measure? Why?

“The reliability of a measure refers to its consistency. For example, what is the probability of obtaining the same results upon repeated use of the same measuring instrument (i.e., test-retest reliability)? We want to be sure that the measure is somewhat consistent over time, and that results don’t vary dramatically from one time to the next.”(Welsh & Harris, 2012, p.182). In this case, we would need to see the ratio of the pre-test ans post-test results remain consistent over several iterations of the program.

Validity is a different concept then reliability; it is the “idea that the measure adequately or captures the concept of interest” (South University Online, 2014, main graphic). On the face validity, the delinquncy scale is valid; if the intervention works, then the rates of delinquency should go down. On the content validity, the scale is valid ONLY if the researchers had found that skipping school, using weed or booze, or running away from home were the major factors in delinquency. The criterion-related, or construct, validity is dependent on self-reporting. There are several issues with self-reporting. Hagan states that “self-report studies may be subject to lying”, amongst other issues (2012, p. 163).

Hagan, F. (2012). Essentials of research methods in criminal justice and criminology. (3^rd ed.) New Jersey. Peason Education, Inc.

South University Online. (2014). MCJ6004: Criminal justice planning & innovation: Selecting a research design (2 of 2). Retrieved December 2, 2014 from myeclassonline.com

Welsh, W. and Harris, P. (2012). Criminal justice policy and planning [VitalSouce bookshelf version]. Retrieved December 2, 2014 from http://digitalbookshelf.southuniversity.edu/books/9781437735000

Scenario 1

Is it fair to say that the program, as it was conceived, was a failure? Are there any reasons that ex-convicts who have been trained with important job skills such as plumbing, electricity, and computing have failed to land a job? Would you consider these to be complicating factors? How?

It is fair to say the program conception was a failure. To begin with, there was no empirical basis for the program; to say “it is believed “ is completely different then saying “it has been tested and re-tested”. This is the reason that planned changes are researched before implementation. Such research should have noted that “a survey in five major U.S. cities found that 65 percent of all employers said they would not knowingly hire an ex-offender (regardless of the offense)” (Petersilia, 2001, para .23). In addition, there are other issues not dependent on job skills;”Mental health, substance abuse, and health problems also pose significant barriers for ex-prisoners seeking employment” (Rakis, 2005, p.8). These are not confound issues; “Confounds are factors, other than the treatment or intervention, that are responsible for the observed outcome changes. In other words, these are factors, other than the program or policy, that caused the change in the outcome“ (South University Online, 2014, para. 1). In this case, there were no outcome change - the change agent simply failed to account for underlying factors that realistically impeded his design goal. Confounding issues should not be “accounted for” in studies so as to bias the result of the study.

Scenario 2

Will the death of the student present a threat to the evaluation of the program? Does the drop-out rate present a threat to the evaluation of the program?

The death of the student does present a confounding issue; some students may stop binge drinking in fear of their lives as opposed to being affected by the program itself. The drop-out rate will also affect the consitency of the reported results, unless the analyst can identify any pre-testing associated with the drop-outs and remove them from the final result comparisons.

Petersilia, J. (2001). When prisoners return to communities: Political, economic, and social consequences. Federal Probation, 65(1), 3. Retrieved November 17, 2014 from http://web.b.ebscohost.com.southuniversity.libproxy.edmc.edu/ehost/detail/detail?sid=da792b0f-f642-418b-9715-c17cb133e72e%40sessionmgr198&vid=1&hid=128&bdata=JnNpdGU9ZWhvc3QtbGl2ZSZzY29wZT1zaXRl#db=pbh&AN=5124169

Rakis, J. (2005). Improving the employment rates of ex-prisoners under parole. Federal Probation, 69(1), 7–12. Retrieved November 17, 2014 from http://search.ebscohost.com/login.aspx?direct=true&db=pbh&AN=19951432&site=ehost-live&scope=site

South University Online. (2014). MCJ6004: Criminal justice planning & innovation: Selecting a research design (1 of 2). Retrieved December 2, 2014 from myeclassonline.com

I had meant to post this early in the course; this is a recommendation for software I have been using for research and citations

Zotero is both a Firefox add-on and a desktop application (you can use it if you don't use Firefox as your browser) that saves citation information AND saves the source doc to your computer if you'd like to use the source again or access it offline.

It works well with the South digital library, with one-click additions to your database.  It also allows tagging to organize references.  It also does limit formatting for your cites and bibliographies.

For those of you that plan on doing more research, it is a great tool.

and it's free! https://www.zotero.org/