The Family Center on Technology and Disability

Thank you for helping us launch what we expect to be an interesting and informative discussion of evidence-based decision making in assistive technology. Please take a minute to read the introductory remarks of Dr. Reed and Dr. Hill ("speakers' positions"). Whether you're new to the topic or are a leader in the field, your questions, comments, and experiences will be extremely valuable to people across the country. The discussion will continue until December 17th. We hope to hear from you often.

Welcome to this important and timely topic. I am looking forward to hearing your stories, questions and comments.

As I prepared to help moderate this discussion, I couldn't help but think what progress we have made in the last few years in our understanding and use of assistive technology. That doesn't mean that every service provider (I use that term to include all paid staff who serve children, including teachers, therapists, administrators, and assistants) in every school district knows what they need to know, but we are making progress. In many cases family members are leading the way in helping the servic providers gain that knowledge. In the best situations families and service providers are working together to learn all they need to know to meet the needs of the children they serve.

We have so many excellent resources these days, including this one. During the course of our discussion, I will try to point out others that may be of value to you. I look forward to hearing from you and invite you to post a story or question to get this discussin going. I think it will prove to be both lively and stimulating.

Penny Reed, Co-Moderator

This is a very timely topic, and one I am very glad to see addressed. (Hi, Penny!) I have seen far to many decisions in AT based on gut instinct or prior experience with a product- without concern for an individual's opinion or preference. However, this become problematic when students have an opinion- and they do!
I have seen many administrators in particular be hesitant to purchase expensive devices, due to a history of AT abandonment. If we are able to show through careful data collection and analysis that a particular device is appropriate and useful to the student, the administrator will be much happier about spending special ed dollars.
One concern I think will be interesting to address is how long is appropriate for the data collection, particularly when we are comparing outcomes between several sophisticated devices.
Joan Breslin Larson
Minn. Department of Education

• Re:need for data posted by Penny Reed on Nov 12, 2003

  You make a great point, Joan. I think administrators should be cautious about purchasing assistive technology. It should never be too easy to get something or we get careless and request things for which we have not proven the need. At the same time, it should not be impossible to get approval to purchase a needed item.
  
  How long to collect data is a difficult question. The answer, I think, is that famous AT answer, "It depends." I think the question is not only, how long, but for what do we look. If I see significant change in a week, then a week was enough. However, if the child needs to learn new skills in order to use a device, several weeks may not be enough.
  
  One of the issues the team must address is, what skills the child currently and what skills must be learned in order to successfully use the assistive technology. The answers to those questions must help determine the length of the trial use.

My current practice when providing an Alpha Smart or a computer-based program such as CoWriter and WriteOutLoud is to put the tech in place for the student to use on a trial basis. The "data" I use to make the final determination is feedback from the teacher: Do you see a significant improvement in quality/quantity of written output? and from the student: Do you think this is helping? Do you like using it?

Would you consider this sufficient evidence? What else would you recommend?

• Re:AT for writing posted by Curtis Griesel on Nov 12, 2003

  "sufficient evidence" is ultimately up to the funder. You know your evidence is sufficient when you are successful in getting the device funded.
  
  But I think the evidence you suggest could be stronger. Subjective perceptions about improvement in quality, quantity, or whether or not the students "likes" something seem very susceptible to unpredictable influences and may not point to the device that provides the most productivity for the student.
  
  How about a more objective measure, such as what Dr. Hill is doing with AAC devices? I could imagine background software that captures a student's actual work and allows the practitioner to analyze both the quantity and quality of the work actually produced with the device over a set trial period.
  
  As far as I know, no such tracking tools exist for the Alpha Smart or for general desktop PCs using AT devices, but I think they could easily be developed.

  • Re:AT for writing posted by Joan on Nov 12, 2003

    I would look at the data in different way- What is the base rate of student performance without any intervention (e.g. Using paper and pencil, Tim is able to write 3 four word sentences in 25 minutes, with an average of 5 misspellings) and, measure his achievement after training and experience with an identified device (Using a portable word processor with word prediction, Tim is able to write six seven word sentences in 15 minutes with no misspellings.)
    I generally suggest the use of tracking forms, such as those found in How Do You Know It, How Can You Show It, (written by Penny Reed, our facilitator in partnership with Gayl Bowser and Jane Korsten) as a good and easy tool for data keeping.
    However- Margaret has pointed out something REALLY important. How does the student like something? Sometimes there can be all sorts of improvement in performance, but if the student does not like it, and does not want to use it beyond the experimental stage, it probably will not be useful technology for that student.
    Joan

    • Re:AT for writing posted by Margaret Poore on Nov 12, 2003

      okay, I like comparing words per minute and number of spelling errors. Now, considering the amount of time available to most of us for data collection, would you consider more pragmatic measures such as number of completed assignments turned in pre and post tech intervention to be valid, appropriate measures?
      
      I should know where to go to find the tracking forms you refer to, Joan, but how about a hint;-)?

      • Re:AT for writing posted by Joan on Nov 12, 2003

        Margaret- if a meaningful improvement for a student is the increase of assignments turned in (and as a Mom who just went to parent teacher conferences, I can really identify with a student's need for that one!)then that is a significant thing to measure. It all depends on what the student needs to do, and what you identify as being a realistic thing to measure.
        As for the forms I referenced, Reed, Bowser and Korsten wrote the book titled How Do You Know It, How Do You Show It. I think it can be purchased through the Wisc. Assistive Technology Initiative (www.wati.org)- look in their materials section. I think it costs about $25.00. I have distributed the book to MN educators, who really like having a resource to guide them in making effective decisions.
        Joan

        • Re:AT for writing posted by Penny Reed on Nov 12, 2003

          Thanks for the plug, Joan. None of us authors get any money for this manual, so I can comfortably say that we get wonderful feedback about its usefulness. We wrote it in response to the huge need for ideas about how to collect and use data. It has more than 30 examples of students and the type of data that their teams decided would be useful. The forms themselves may or may not be immediately useful. We authors believed that the readers would find the ideas valuable, but would construct their own forms for each unique need.
          
          I think the most important question to ask yourself is, "What would convince me that this tool is making a difference?" I use the word tool, because many of the tools are things that are not originally designed to be AT. Only the data tells us that it is or isn't. Something like the software program Inspiration is a good example.It is a great tool for many students.But if for a specific student with a learning disability, it increases, maintains, or improves the functional capability of writing essays, then it is AT for that student and needs to be included on the IEP.

      • Re:AT for writing posted by Penny Reed on Nov 12, 2003

        How many assignments turned in may or may not be a valid measure. If a behavioral issue is impacting the turning in of assignments, then the impact of the tool use will be invalidated. I would prefer to look at the things that Joan mentioned. Keeping a portfolio can be very helpful.
        
        If you are inquiring about How Do You Know It? How Do You Show It? It is available from www.wati.org or www.attainmentcompany.com.

  • Re:AT for writing posted by Penny Reed on Nov 12, 2003

    Those are good suggestions. In the meantime, actual observation of the student working both with and without the tool can yield some of this information. In addition there are several individual software programs that do collect data about a students mistakes and repairs.
    
    Don't discount the student's opinion. Depending upon the age and experience of the student, they can be powerful predictors of outcome. Think of your own use of software and other tools. How many times do we sit around and discuss the pros and cons of a particular tool where one of our friend loves something that we hate or vice versa. How we feel about a tool definitely affects whether or not we choose to use it.

• Re:AT for writing posted by Penny Reed on Nov 12, 2003

  Before this is over, I may become famous for saying, "It depends"! Those are certainly important factors. I like the fact that you are pairing two types of data: review of the finished product produced with the AT and the opinion of the user. Whether that is enough is determined by whether you feel confident that the evidence you have gathered is sufficient to answer the two critical questions:
  Is this tool AT (because it increases, maintains, or improves a functional capability)?
  If it is AT, is it required for the student to receive FAPE?
  
  One of the difficulties is that you included quantity as a criteria. MacArthur's research in 1999 with students with learning disabilities showed that the use of tools such as word predicition and talking word processing produced improved quality of written text, but did not change quantity or speed. If you use quantity as a criteria in some cases, your outcomes may not show a significant change.
  
  AT the same time Marcia Sherer's research on abandonment indicated that the user's feeling aobut the usefulness of the tool was critical. So that is an important piece.
  
  You might want to compare types of errors and whether this AT makes a difference. You might want to look at the number of errors made both with and without AT and how long it takes to correct them. Those are other factors that show that the selected tool is making a difference.
  
  
  
  

  • Re:AT for writing posted by Discussion Board Guest on Nov 14, 2003

    I agree with your statement "it depends"
    
    Is MacArtur 1999 research posted? I am very interested in reading more.

My son uses cowriter and intellitalk 2. It helps relieve the angst of writing and creates more product. What I have trouble with is his initiation. I still need to give him ongoing verbal prompt questions to keep him going and organize his work. Any suggestions on strategies or software? ( Inspirations' text is TOO SMALL to be helpful)

Thanks!

• Re:initiation posted by Penny Reed on Nov 13, 2003

  Organizing thoughts and getting them into text in a meaningful way is certainly the challenge in learning to write. It's good that you are working on it. How closely are you working with his teacher to insure that you both have similar expectations and are both using the same type of prompts. It would be good to coordinate your efforst and begin to look for things that work that you both can use consistently. Is the teacher is using any specific writing program such as 6 + 1 Traits? You can learn more about that at the Northwest Regional Educaiton Lab website:
  http://www.nwrel.org/assessment/about.asp?odelay=1&d=1
  
  As far as software, I don't know if Kidspiration would offer any larger font. Does anyone out there know?
  
  DraftBuilder from Don Johnston uses 18 point font Arial Bold (Windows) and Helvetica Bold(Mac) as its largest font. That may be an option.
  
  I know there are other writing programs, but don't know which ones offer an outlining feature that may offer larger font size. Does anyone have something to suggest?
  
  Word processing programs like Microsoft Word have outlining capabilities that may work, although they are more complicated than Inspiration.
  
  You could work with flashcards or other low tech tools to capture his ideas as he generates them and then organize them.
  
  But we are getting the tool before the task. If he is stuck at an instructional step, he will probably benefit from instructional strategies that could then be used with any of these tools to accomplish the task. Data about where he is stuck is needed.

"Research on the effectiveness of assistive technology is neither sophisticated nor adequate. The emphasis on the scientific-based evidence in NCLB [No Child Left Behind] challenges the professional community to build the knowledge base about the effectiveness of assistive technology."
- Dave L. Edyburn, Ph.D from Assistive Technology and Evidence-based Practice, http://www.connsensebulletin.com/edyatevidence.html

I recently ran across this article written by Dr. Dave Edyburn. This particular quote is based on what he sees as the current state of affairs in AT outcomes. I would be interested in hearing your thoughts on current and future trends in measuring the effectiveness of AT, especially in the context of NCLB. Are the mandates and ideas set forth in NCLB changing the way people measure the effectiveness of AT? Is NCLB raising the bar on measuring the effectiveness of AT or limiting the ways we measure the utility and effectiveness of assistive technology? How do these trends effect families selecting AT for their children?

Thank you! Julie

• Re:AT and No Child Left Behind posted by Penny Reed on Nov 13, 2003

  I do believe that the mandates and ideas set forth in NCLB is changing the way educators measure effectiveness of all our interventions including AT.
  
  That is a good question about how these trends may affect families selecting AT for their children. I hope it has a positive effect. It seems to me that we cannot haphazardly provide AT for anyone. That we always must look for evidence of the need for AT and as we move through a process of determining what AT might work, we always base decisions on evidence. In all cases the critical focus is on what tasks the child needs to accomplish, how he or she does it now and what aspects of performance we expect to see changed by the use of AT. This focus on function and on evidence will help all of us, both families and service providers, do a better job.

Hi,
For those of us who have used the trial and error methods of determining which AT works for so long, how do you suggest we start making the transition to the new Evidence Based Practice? What's the first thing we need to do? It seems very overwhelming - how can the transition be easier?

Thanks for your input!

• Re:Making the transition to EBP posted by Penny Reed on Nov 13, 2003

  Well, I guess it depends on how much "error" there was in your trial and error. I think we sometimes throw that term around too loosely. When we have used well planned trials with AT and have collected data about changes in function on which to base our decisions, I do not think that is "trial and error". Trial and error implies a randomness and lack of process that I hope we have already left behind.
  
  I guess I have always believed that its all about process. That we need a well thought out procedure for handling all AT questions. For example if a teacher requested a computer for a student with a disability and the AT Specialist just set one up in the classroom and never checked to see what it was being used for and whether it was working. That would be trial and error. But if a teacher requested a computer and the AT person said, "What is it you want him to be able to do with the computer?" Then we have the beginning of a process. We would identify a specific task, collect data on how that task is currently performed, research what AT has been effective for that task, plan a trial or sequence of trials to whether AT makes a difference, collecting data about changes in rate, accuracy, legibility, successful completion, etc. That is not trial and error. That is making evidence based decisions.
  
  Would you consider sharing an example of the "trial and error" methods you currently use? Perhaps you are not as far off as you think.
  
  

  • Re:Making the transition to EBP posted by Katya HIll on Nov 18, 2003

    Several questions were asked regarding how to make the transition to EBP that included “Where to start?” and “How to make is easier?” I feel that the first place to start is to evaluate or appraise what you’ve been doing. Most teams haven’t really been using a “trial and error” method, but have attempted to use a systematic approach to making decisions. However, applying a principled approach on top of an old paradigm doesn’t necessarily produce the expected results. Hence the frustration and feeling that EBP may be overwhelming.
    
    Usually, I start appraising the accepted processes and procedures of teams by evaluating the personal (data) evidence that has been collected, reviewing what external (research) evidence was used, identifying what strategies/interventions were selected, and seeing how change is being monitored. If the supporting evidence doesn’t add up, then I find that the expected outcomes typically haven’t been achieved. So, to get started with EBP, I have teams first identify (brainstorm) where they are today with AT/AAC processes, procedures, and policies. I have team members talk about:
    · What data they tend to collect? Are the data specific to measure progress for the intervention?
    · What external evidence is used to make AT/AAC decisions?
    · What’s the level of evidence? Is it backed up by research or simply something that was recommended by “an expert.”
    · How is change monitored once an intervention is implemented? Were the outcomes measurable? Were the data valid and reliable?
    
    As people may be thinking or talking about how their current process, procedures, and policies compare with EBP, I’m sure we’d all love for anyone to share the insights gained for your appraisals.
    
    Katya
    

There are many ways to collect data (or information or evidence) about a student’s assistive technology use. Four that we (Jane Korsten, Gayl Bowser and I) have identified are:

1. Interview the student
2. Review products created by the student (e.g. math worksheets, creative writing assignments, etc.)
3. Observe the student
4. Video tape the student


Each of these can be valuable at different times. Interviewing is particularly useful when you want to know a student’s preference or when you want to know why a student prefers one tool over another (“I don’t like the way it feels.” Reviewing products can tell us if the student produces written work with fewer errors or math problems that are better aligned. Reviewing finished products created both with and without assistive technology is probably the most common technique used to gather evidence about AT use. Unfortunately it doesn’t reveal how much time it took the student to complete the work, how many errors were made and then corrected . etc.

It is obvious that if we need to know how a student did something we must observe the student in action. That observation may be ongoing (occurring every time the student does the specific task) or episodic (twice a week, once a month, etc.) The number and frequency of observations will depend upon the task and what we need to know. Finally, video taping the student doing a specific task can be a useful way to gather data because it can be reviewed as many times as needed and discussed without the student being present.

Thinking about what it is we need to know can help us determine how and when to collect data.

What do you feel is the biggest challenge to using evidence based decision making? It would be good to hear from many of you.

• Re:Biggest challenge posted by Margaret Poore on Nov 14, 2003

  My response is in regards to AT for writing, not for AAC. The biggest challenge to using EBD for me is data collection. As a specialist who currently has active cases in 18 different buildings I find it difficult to get teachers to actually collect the data. I often don't have the time it takes to build a TEAM whose task is to determine what is needed. Our district has about a hundred buildings, so most referrals are from folks I haven't any "history" with. Often I am regarded as the expert who comes in and makes the determination of what is needed and then puts it in place. The concept of trial usage and data collection on observed changes is not a comfortable one for teachers and therapists who are all burdened with significant overloads. Which I why I need to use practical and easy data collection methods.
  
  I would love to have the time to videotape a student and then the time to review the tape. And lack of time is not the only limiting factor. Unless taping students is a regular event in their classroom, the process itself will make the kid nervous, silly, or resentful for being singled out. Unless your kid is a serious and mature student who won't be disturbed by having a camera (with or without the adult running it)right over his left shoulder (close enough to observe keystrokes and/or what appears on the screen)you are not going to get a sample of typical behavior, IMHO. I wonder if anyone is able to do this in a public school setting. This seems like something that could only occur in a university or research setting. If someone is out there doing it, I'd love to hear about it!
  
  

• Re:Biggest challenge posted by Virginia on Nov 16, 2003

  My biggest challenges to using evidence based decision making are time and teachers.

  • Re:Biggest challenge posted by Joan on Nov 16, 2003

    Virginia and Margaret talk about some big issues. Having others assume responsibility for data when we are supposed to be the "AT Wizards" is tough. When I was in practice, I was supposed to be the expert who did it all. This was a problem! I remember the first time I "got it right" for a student. I wanted to see what increase word prediction made for his performance in writing. I loaded the computer software, trained and asked for both data kept on a form, and for a hard copy of all documents he produced in 1 month using the software. The data forms showed he wrote 2 papers, with some progress over base rate. The hard copies, however, showed he produced major work with great improvement. The school staff had not been able to take the time for manage data as I had asked. Keeping the hard copy was effective data- but unless we had saved it, we would not have had evidence. Look for multiple ways to save and record success stories. Yup- often it is hard, and requires creative thinking. The outcome for this student proved to me (and his team)that it was worth it.
    

    • Re:Biggest challenge posted by Margaret Poore on Nov 16, 2003

      I love the idea of keeping a hard copy, Joan. This is an idea that is so simple and practical that it makes me slap my head and say "now why didn't I think of that"!!
      
      I have a sixth grade student that I will be training this week. I'm sure his resource teacher would be willing to keep a hard copy for me. Since he has not done any assignments so far this year that require anything more than a word or phrase answer, anything he writes that's more than a sentence will be an improvement!

      • Re:Biggest challenge posted by Joan on Nov 17, 2003

        Let us know how it works! It does mean you will have to do the counting- but it is a simple solution.
        

  • Re:Biggest challenge posted by Penny Reed on Nov 17, 2003

    Boy, you hit that one right on the head! Time is always our biggest problem in education these days as we all do more with less.
    
    By "teachers" I assume you mean that it is difficult to get someone else to collect data for you. I agree that is a challenge too.
    
    I don't know any easy answers. I think Joan really gave good examples. Sometimes the only way to get change is to find concrete evidence. The first few times are always the most difficult. Once the teachers with whom you work begin to get the realization that taking time up front to measure what is happening will save much more time in the long run, not to mention money saved, they will be more interested in gathering the needed evidence.
    
    Change is difficult. I see that in schools where the administrator understands and supports the use of evidence based practice, it is easier to implement.

• Re:Biggest challenge posted by Joan on Nov 17, 2003

  One of the biggest problems I faced was the systems change issue- when I was called into a situation, I was the outside consultant who was supposed to deliver the quick fix. When I would suggest we try things out and compare outcomes, I suspect some felt I was dodging my responsibility. I needed to present the situation smarter- "We need to insure we are finding which device/tool/strategy from among these 3 with similar functions really does increase Tim's performance." However- the administrators nearly always loved the result- because we found answers that worked rather than sat in closets.
  

In the Fall 2003 edition of "Research Connections in Special Education" there are three articles that are relevant to this discussion and are worth reading. They are:
"Using Data: Innovative Ways to Improve Results for Students with Disabilities" http://ericec.org/osep/recon13/rc13sec1.html.
"Data-based Decision Making-A Core Feature of Implementing Interventions"
http://ericec.org/osep/recon13/rc13sec2.html.
"Using Data from Participatory Action Research to Support Change and Innovation" http://ericec.org/osep/recon13/rc13sec3.html.

• Re:Additional Resources posted by Penny Reed on Nov 17, 2003

  These resources suggested by Jackie are excellent. In the second one, George Sugai, a leading researcher in the area of behavior intervention suggests following these principles:
  
  GUIDELINES FOR CREATING A DATA-BASED DECISION MAKING SYSTEM
  
  * Data should be readily available.
  * Procedures for collecting data must be easy to use and not require excessive staff time and resources. (According to Sugai, data collection systems should not consume more than one percent of someone's time each day.)
  * Purposes for collecting data must be relevant to ongoing activities.
  * Only a small number of questions should be addressed.
  
  Although addressing behavior interventions, this advice rings true for data about assistive technology also. The old KISS principle (I like to think of it as Keep it Simple Sweetie!) always applies.
  
  

I agree completely with the need to utilize hard copies (or finished products) of things the student has created. It is a great starting place. But a story I have heard Jane Korsten relate always reminds me not to stop there. Jane tells of a student with whom she worked who used a single switch and an onscreen keyboard with scanning in order to create written work. It was effective, but slow. He needed these tools because he had cerebral palsy. He was fully mainstreamed and getting above average grades. The occupational therapist who served him was always looking for other tools for him that might be faster. AT a conference she learned aobut using Morse code with a switch. She investigated and felt it might work for him. After a brief trial, she was even more convinced. He was very smart and learned Morse code in just a few weeks (much faster than the staff). He loved it and used it most of the time. However, the OT observed that he still wasn't very fast. With his permission they set up a data colleciton system to determine which was faster the onscreen keyboard and scanning or Morse code. After two weeks of data collection it was clear that even though he loved it, Morse code was slower. Based on that data they worked out a plan to use the onscreen keyboard for timed writing tasks and to let him choose for other tasks. Morse code became the source of a great hobby, HAM radio. Both were good tools, but for different purposes.

• Re:Data from more than one source posted by Penny Reed on Nov 17, 2003

  I should add here, that had they looked only at hard copy of products he produced with each method they would not have been able to see any difference unless they knew how much time each took.
  
  At the same time, if they interviewed him, his preference was Morse Code. He loved it.
  
  It was the direct observation to determine how much time each took that provided the additional and much needed piece of information.
  
  All three sources of information led to the final plan that really met his needs.

  • Re:Data from more than one source posted by Joan on Nov 17, 2003

    Penny- great story! It really shows how we have to explore all the avenues to get at the right answer. I know I have stopped short a few times. It also points out something important- know what you are looking for when you start to collect data. Are you looking for volume, fluency, speed, personal preference? All of these are valid. But- not always at the same time, and not always carry the same weight for a student at a given point in time.
    Thanks for the thoughts. (Darn- now I have to go think some more...)

In Minnesota, the Department of Education has written a manual to assist IEP teams in the process of considering and choosing appropriate assistive technology for a student with a disability. This easy to use manual is based on the best practices from national experts in AT. In addition to the five-step consideration process, there are several worksheets which would allow a parent or professional to collect data about the student's use of an AT device.

This manual is available for FREE from the MDE web site at http://education.state.mn.us/stellent/groups/public/documents/translatedcontent/pub_005769.pdf
Joan Breslin-Larson (who has posted here several times) is the main author of this manual. A few other states have similar consideration processes and worksheets and perhaps others can offer information about those options.

Starting on page 76 of the PDF file are worksheets which would be very useful for tracking data about a device's effectiveness. While I would encourage people to review ALL the worksheets since they build upon each other, the ones I think are most specific to data collection are: the Planning and Implementation Summary worksheet which helps the IEP team get organized when trying different devices. Questions such as "How will success be determined?" can be discussed at the IEP meeting so the whole team can offer ways to measure success and come to an agreement about what factors will indicate success with the device.

• Re:Data Tracking Worksheets Available from MN Dept. of Ed (PART 1) posted by Discussion Board Guest on Nov 19, 2003

  This link does not work and I found a way to search for the manual, but that link says the document can't be found. Not sure who to inform....

Seems my note was too long! Here's the rest:

The Extended Consideration Log worksheet allows IEP teams to track the effectiveness of a variety of devices in different environments which the child is in during the day. Questions such as what device was tried in what environment, how long the device was used, how many trials were observed, what was the criteria used to judge success and was the criteria met? These questions can help determine whether the IEP team will recommend a particular device for a student.

I'd be interested in knowing if there are other tracking worksheets available for download on the Internet for free. Expecting already strapped schools to pay additional costs for data tracking materials may deter them from trying this method.

• Re:PART 2 (More Data Tracking Worksheets) posted by Penny Reed on Nov 18, 2003

  Thanks, those are a great idea.
  
  There is something similar on the WATI (Wisconsin Assistive Technology Initiative) website, www.wati.org. Go to WATI Assessment forms and download them. They are all there as pdf files. Pages 34 and 35 are a Trial Use Guide which also has a form for collecting information about the trial and outcomes.
  
  I know many districts have developed forms. I second the request for more resources. That is a great idea.

Hello everyone!

Returning from the ASHA Convention in Chicago, I see our discussion is well underway. Many of the threads are similar to the ideas that came out of discussions during ASHA sessions that focused on evidence-based practice and outcomes measurement.

I’m very excited to be part of this topic discussion. In looking over the comments already posted, I see mentioning of one or more of the three components of EBP: external evidence, evidence at the personal level, and the knowledge and skills of practitioners. In working with teams attempting to get started with evidence-based practice, I find some concern about the time needed to apply EBP principles. I remind teams that more time is required in re-doing work when outcomes aren’t achieved because evidence hasn’t been used. I summarize the concerns as a hesitancy in jumping into searching and appraising the research evidence, and limited collection of quantitative or useable data. The following are the basic steps of EBP: 1) characterize and measure the performance of the individual, 2) ask the best question(s), 3) seek the best evidence, 4) appraise that evidence, 5) apply the evidence, and 6) monitor the change.

First, I’m hoping participants will share which, if any, EBP steps seem to stall teams from moving forward with an EBP approach. So the question is, which step is creating the biggest hurdle in applying EBP? Then, I’d like us to share what strategies we are using to improve our knowledge and skills in applying the various steps. I’d like to suggest that although EBP may have become an approved policy or process for professionals and an expected standard by individuals and families receiving AT/AAC services, our mind sets, skills, and access to tools and resources that support EBP and performance/outcomes measurement may NOT have kept up to pace.

We’re off to a great start with several insightful comments already contributed to add fuel to our discussion. I look forward to sharing more on this important topic which is contributing to a major paradigm shift in AT/AAC service delivery.

Katya

Katya Hill, Ph.D. CCC-SLP
Executive Director
AAC Institute

• Re:Introduction posted by Margaret Poore on Nov 18, 2003

  I work in a district that has about a hundred schools. I currently have cases open on 25 students in 18 different locations. Even when students are in the same building, they don't always have the same people on their IEP teams. So my biggest hurdle in terms of EBP is being able to get the message out to so many different people. I think it takes a good working relationship with a team to be able to convince them of the importance of EBP when in the past the process was so informal (so undocumented!). I think I have that with a few teams and will start with them. I have printed out some of the MN worksheets and will use them with the teams as we determine what trials and what criteria of success we need.

  • Re:Introduction posted by Katya HIll on Nov 19, 2003

    Margaret,
    
    Thanks for sharing your hurdle. Before deciding to get my doctorate I had provided AT services to 17 school districts with approx 60 new student referrals for AT each year. It took time and consensus to build skills and shift paradigms. I attribute teams seeing successful outcomes by others (and wanting to achieve similar outcomes) has the biggest factor in motivating change. Although several initiatives and strategies certainly helped!
    
    Before I share some ideas I'd like to see others provide suggestions about how to convince teams about the importance of EBP. I'm sure we can come up with a good list of ideas that can be posted and used to fit different environments.
    
    Looking forward to creating a list.
    
    Katya

Katya, I really like the basic steps of EBP that you posted:
1) characterize and measure the performance of the individual,
2) ask the best question(s),
3) seek the best evidence,
4) appraise that evidence,
5) apply the evidence, and
6) monitor the change.

Sometimes I think when we hear Evidence Based Practice we think that we are now expected to do formal research, (and there is a need for more research about AT as pointed out in Dave Edyburn's article that was mentioned earlier in the discussion), but all we are really being asked to do is just good practice. As a teacher, there is not one of those steps that I would ever want to leave out.

I think one of the problems in the AT field is that it has been viewed as some magical separate thing that some AT Specialist rides in and does. When in reality AT must be part of the services provided by everyone who serves a child with disabilities and we must apply the logical steps of EBP just as we would with any other service we provide. I can't imagine teaching reading without following those steps.

• Re:Steps of EBP posted by Katya HIll on Nov 21, 2003

  Penny, I agree with your comment that At frequently is viewed as some magical separate thing that some AT Specialist rides in and does. From my beginning in the field, the first thing I remember is someone riding in and showing me all the technology I was responsible for keeping track of and having to set up. At no time did anyone mention the scope and sequence of effective teaching strategies or the learning outcomes expected from the use of the AT. We were suppose to be impressed by the technology, and technology became first on everyone’s list.
  
  Of course, we realize that a thorough understanding of the student or individual is needed before even considering technology. That’s using evidence at the personal level. By starting with the student and collecting performance data about abilities and skills we are at a much better starting place to identify the types of AT strategies and/or technologies that will achieve the best outcomes. For AAC, to me that means language first, technology second. And, I can't imagine working on language without following the EBP steps.
  

I want to explore a little bit the different types of evidence that can be used in EBP. Steps 3 & 4 of EBP deal with seeking and appraising evidence. Dr. Hill in her opening remarks mentions two types of evidence, "external evidence" and "evidence at the personal level".

My understanding of EBP as it is practiced in medicine is that the goal is to accumulate a large enough body of "external" evidence so that once an individual has been appropriately "characterized and measured" (step 1), this external body of evidence can be referenced to select an appropriate treatment. This places less reliance on doing time-consuming and costly tracking of "personal evidence" before a solution can be chosen.

In other words, EBP only becomes really effective when there is a large body of external evidence that we can all draw from (external evidence) rather than having to treat each student as an individual subject for formal research (personal evidence) which can be both intimidating and time consuming.

The current problem with AT EBP is there is almost no body of external evidence to draw from, so we end up treating each AT assessment as an individual research project. Very costly and time consuming.

Dr. Hill alludes to this problem when she describes the "limited collection of quantitative or useable data".

What efforts are underway, or could be taken, to standardize data collection among AT assessments for various types of disabilities, so that the data could be accumulated into a central database, analyzed, and the results disseminated for use by other AT professionals when they do their assessments? This is the non-existant "knowledge base" that Dr. Edyburn refers to.

We won't really see benefits from EBP unless we have a good base of external data to draw from. Without external data, we are left with treating each AT candidate as an individual research project, which is far too expensive to be accepted by most families and schools as an ongoing practice.

• Re:External vs. Personal Evidence posted by Katya HIll on Nov 21, 2003

  In applying “external” evidence and evidence at the “personal level", we have to be realistic in the differences that exist among medicine, education, and related health care providers such occupational therapists, social workers, and speech-language pathologists. The fields of AT and AAC are unlikely to ever have the large clinical trials and randomized controlled studies as in medicine. However, that doesn’t excuse AT team members from searching and appraising the research evidence. The unfortunate habit in our related professions is that we frequently accept the word of authorities as evidence and don’t ask to see the data. For AT that means that I might implement an AT/AAC strategy or technology simply because of a causal recommendation.
  
  Today I was editing a presentation for ATIA on AAC dynamic display technology. Although this technology has a successful history in the market and is frequently recommended, very little empirical data are available on performance outcomes. However, I wouldn’t suggest that teams stop recommending AAC dynamic display systems. I do recommend that teams KNOW what evidence exists when considering a system, start asking to see performance and outcomes data (from presenters and manufacturers), and provide a disclosure to individuals or families about how evidence was used to support the team decision. This is an inexpensive, but powerful start to EBP.
  
  Evidence at the personal level should not be confused with single subject research data, and is neither intimidating or time consuming. It’s something that everyone is expected to do either as part of IDEA (mandated outcomes measurement), the ASHA Scope of Practice (use instrumentation to collect data), or being accountable to third-party payers. Today I helped one of my graduate clinicians analyze a language sample from a therapy session in less than 5 minutes. That included showing her how to operate the software analysis program. During one of my sessions at ASHA, participants not only collected a language sample from an individual who relies on an AAC system, they analyzed the samples and printed out a performance report. Again, this type of personal evidence can be collected and analyzed inexpensively, but is very powerful. Believe it or not, it's actually fun! We are only now just starting to “standardize” the tools and resources to measure performance and outcomes. With better tools, the establishment of larger performance databases is possible. I believe that is what Dave Edyburn envisions.
  
  I purpose that it is much more expensive not to do EBP, because we have a greater loss in time by doing it again, purchasing equipment that goes unused, and providing interventions that don’t achieve the desired outcomes. Teams, administrators, and families (not to mention the student) become frustrated in the investment of resources when evidence is not used to support decisions. I guess you need to accept that by using EBP some of your decisions are going to be different, but you'll be getting better results.
  

  • Re:External vs. Personal Evidence posted by Penny Reed on Nov 24, 2003

    You are so right. We cannot afford to waste our limited resources on equipment that is not the right choice or interventions that don't produce the desired outcomes. Every service provider should be able to identify and collect the personal evidence that is needed.
    
    When we first started the Wisconsin Assistive Technology Initiative (WATI) and interviewed dozens of service providers in schools to find out what they needed. We found that they did not do their own AT assessments because they "didn't know the questions to ask". That was 10 years ago. These days there are many tools available at no cost that can help us figure out the questions to ask. I think that the WATI assessment tool available from www.wati.org, the AT assessment tool available from the Georgia Project for Assistive technology, www.gpat.org, and the Student access map available from Boston Public Schools at www.boston.k12.ma.us/teach/technology/emmanuel.asp are all examples of that.
    
    These are not the only tools, but can be a place to start for the team that doesn't feel that they know what they should be asking.

  • Re:External vs. Personal Evidence posted by Jim Tobias on Nov 29, 2003

    I hope I'm not misunderstanding this topic, but I don't think Curtis was asking for "large clinical trials and randomized controlled studies as in medicine". I heard him making the call for studies on what works and *how well it works*, using the goals and objectives of the user's task environment. That is, if educators are concerned that a particular student doesn't write as much as his classmates without disabilities, then we're trying to help that student write more, even if it means only counting words. Can we in good conscience say, today, that we know that a particular type of intervention averages a 25 330ncrease in word output for students of that age with that disability? [Note that this is different from recommending a particular product. It's about justifying a class of products.] Studies like that should have been done decades ago. This clearly relates to the funding issues!
    
    My point here is that the metrics cannot be ours -- they must be those of the use environment, whether it's kindergarten, a clerical worksite, or a jet cockpit. Sometimes it seems that we are afraid to put our tools and clients up against the cold hard world of productivity measures. Let's not be shy! It's true that some clients will never "measure up" to all the tasks in the environment to which they've been assigned. That may mean that the environment is wrong. But look what such a rigorous approach will do for clients who can perform at high productivity with AT and other interventions: they will have all the "certification" they need for full participation. And we will know when our job is done (at least for the present).

    • Re:External vs. Personal Evidence posted by Joan on Dec 01, 2003

      I will be honest- this is very hard for me to get my head wrapped around this. Jim- I appreciate your clarification and think I am starting to get a handle on the bigger picture. Was it last year when the question was raised at the CEC conference asking how we knw AT makes an impact on the education of children with disabilities. I think this is one of the major the bottom lines. And- there is more than one bottom line...
      What is the impact on the performance of an individual using AT AND
      What is the impact on the performance of a group of students using AT.
      
      Particularly in education, when we are dealing with the impact of No Child Left Behind, the second question is essential. And, this is where we have little to no data. Although my personal preference is to examine the first question (I like the warm fuzzy child oriented data), it is obvious that it is past time to examine the second question in greater depth. Not only will this help us in doing our work well, but will also provide us with the data to validate funding.

I share the concerns that have been expressed about external evidence. One of the things that frustrates me is that they is not a good vehicle to share the outcomes of research that is done. There is a long delay, sometimes a couple of years or more, between the end of the research study and the dissemination in a journal. Other research is shared at a conference, but only the attendees hear it. Finally, even when published, it is often in a discipline specific journal so that other practioners never see it. The federal government continues to fund studies, but even with the technology we have available, there doesn't seem to be anything new or creative in dissemination that would make available the external evidence that exists.

Am I wrong? Or do others see this same problem?

• Re:External evidence posted by Jim Tobias on Nov 29, 2003

  You're absolutely right! And thanks for stating it so clearly!
  
  We have to do a better job of identifying the audiences we are trying to influence with our research. We should be able to create a packet for advocates, another for parents, another for school boards, another for employers, another for Medicaid authorities, etc. Because they all have different buttons and they all need to hear the results in different "languages". [BTW, this is a job that ATIA would probably be glad to help with.]
  
  As you all know, the AT Act is up for elimination or renewal. I have been invited to present ideas on the program to Senate staffers this Friday. Any "evidence-based" ideas you have would be most appreciated.

• Re:External evidence posted by Jeff on Dec 01, 2003

  Hi, this is my first post. Hope it's relevant. I have a couple of comments about external evidence.
  
  First, I think we need to pay particular attention to the quality of our evidence. And part of the problem comes from trying to apply research results to practical problems. Group-level experimental research has a hard time dealing with individual differences in clients, applying real intervention programs in real settings, the effectiveness of the intervention and its generalization to other tasks, contexts and impact on the quality of life of the person. I'm not knocking group-level research, but I think we have to be careful how we digest science.
  
  The American Psychological Association Task Force on the Promotion and Dissemination of Psychological Procedures published guidelines on evaluating treatment efficacy: http://www.cpa.ca/documents/empiric_p1.html
  
  Criteria for Judging Research: http://www.cpa.ca/documents/empiric_p8.html
  
  Clinical Practice Guidelines: http://www.cpa.ca/documents/empiric_p10.html
  
  The APA and CPA both accept single-case research as sufficient evidence for determining treatment efficacy. You can take a look at the web pages to get more details.
  
  
  Second, It seems to me that good external evidence comes out of the peer-review journal process, because it is the only place that research can be evaluated by expert peers. Conference proceedings, book chapters, reports, personal web pages - all of these are important sources of information, but in my mind, don't pass the test for good quality external evidence. ONe of the biggest problems with innovative AT research is that as individuals, we may invest extraordinary amounts of time and energy on a project that *we know works*. And that personal investment is the researcher's Achilles' heal, because it blinds us to those things that we may not want to see. The best we have at this point in time is peer review.
  
  Ok I've gone on way too long. Look forward to your comments.
  
  Jeff Higginbotham
  University at Buffalo

  • Re:External evidence posted by Katya HIll on Dec 03, 2003

    The comments on this thread all seem to be highlighting the challenges practitioners and consumers experience in tackling the component of external (field) evidence required of evidence-based practice. I feel I’m safe to write that most practitioners do not routinely read peer-reviewed journals to keep up on the latest research. Let’s assume many practitioners are reading AT/AAC peer-reviewed journals, are those same readers adequately appraising the strength of the evidence? In my own field of speech-language pathology, I know from attending this year’s ASHA Research Symposium that we are still reaching consensus on appraisal techniques/methods for EBP and learning from each other. And I think this is great! For those of us presenting on EBP, I believe it would be unrealistic to assume practitioners are going to invest in acquiring advanced statistical analysis skills with so much else to do in delivering services. However, here are a few practical starters:
    1) most practitioners can learn how to apply basic systematic approaches to looking at the research-based.
    2) Clinical training programs can start teaching appraisal skills from introductory level courses, and not limit research design and methodology to the “research course.” In many ways, as researchers we have become our own worse enemies as gatekeepers to the research processes.
    3) Conferences can start to ask for research and data based presentations, such as Closing the Gap and ATIA.
    4) Journals for practitioners can be geared toward pragmatic articles that report clinically useful data and outcomes such as the new ATIA publication.
    
    Penny mentioned the delay that occurs between posing a research question and publishing the findings in a peer-reviewed journal. In attending an NIH grantsmanship workshop, NIH presenters cited a minimum 10 month period needed to critique a grant application. Most grants are not awarded on the first submission. If you resubmit and are successful on the second attempt, approximately 24 months have transpired. For AT/AAC Research and Development (R&D) Departments, waiting for funding on a planned innovation would not be profitable. That’s not to say that research isn’t being done at the R&D level. It’s just that most manufacturer’s don’t perceive a value in publishing the data. That seems to be changing with more of us asking questions about the performance differences among various systems. More of us have to ask about the research that has gone into development decisions.
    
    One of the research tasks I’ve been involved in over the past five years, is the development of an integrated controller for wheelchairs and computer access. The team has several RESNA Proceeding publications as we worked through the various phases of development - which included taking over 9 months for IRB approval to use the joystick indicated in the study. My humble opinion is that if an R&D department had conducted the research, they would have arrived at the same conclusions and a product would be on the market today.
    
    So, I guess I’m thinking that we have to weight our values and question how we arrived at those values. Now I’ve gone on again.
    
    Katya
    
    
    
    

I'm working with a child with ASD and severe language impairment. At the moment he can read words and some kind of sentences, writes sentences by word selection in a PECS way, he communicates in signed spanish (we are from argentina)but his grammar is very poor; his sentences have words and some verbs and locatives, but don't follow a gramatical order. He is very good in computer, can play PC games, no motor difficulty, discriminates words into frases, can type words, but by typing forgets the whole mesage.
I'm looking a tool wich should help him in 1º: word selection and word prediction in Spanish son he can write mesages, prefferently with voice output. 2º textual structure, so he can organize his ideas and write upon them.
Up to the moment we have been working with low tech, but now he needs more training in language production as wrinting is the main source of language acquisition for him.
Which kind of tool should you recommend for his case considering that it must be Spanish based? I should apreciate any comments
Sincerely
Mariana Sigstad

• Re:decision making in writting posted by Penny Reed on Dec 01, 2003

  Mariana, I am not able to make any recommendations for you about products. But, you might find help if you would post your questions on the list serve that you can join through the Quality Indicators for Assistive Technology, http://www.qiat.org
  
  There are over 700 individuals with a great deal of knowledge and someone may know a child with similar needs who has been helped by a specific intervention. Good luck.

• Re:decision making in writting posted by Perrine Dailey on Dec 01, 2003

  I researched word prediction programs which can be used in Spanish and there are a few to choose from. As a non-Spanish speaker, I can't offer my opinion on their quality, but you can learn more from the web sites listed below.
  
  Aurora Systems: www.aurora-systems.com/pf/foreign.html
  Applied Human Factors: www.ahf-net.com/new_products.htm
  Text Complete: www.visorvillage.com/software/pc/TextComplete-Spanish-2003-7-31-palm-pc.html
  Click'N'Type: www.lakefolks.org/cnt/#LP-Spanish-Mexican
  Keystrokes 3 for Mac: http://www.macnn.com/news/20061
  
  Perrine

• Re:decision making in writting posted by Katya HIll on Dec 03, 2003

  Mariana,
  
  You have me thinking about providing suggestions for Spanish-based AAC systems. I'm aware of some products that support Spanish as well as some work being done related to AAC and Spanish. The next ISAAC conference is in Natal, Brazil in October 2004. I would think individuals are in the process of preparing presentions on the topic to present at this venue. I want to see if I can track down some more information for you. In the meantime, I'm wondering what experience others in the discussion might have related to AAC and Spanish. Any thoughts on evidence at the personal level? Is anyone aware of research related to this question? Considering the emphasis ASHA places on multicultural awareness and service delivery, it shouldn't take an exhaustive search to find evidence. However, we know that's probably not the case.
  
  Katya

I've enjoyed reading the interaction on this topic. A struggle that keeps coming up in discussions on evidenced based practice is the quality of the data and how to define quality data while keeping a balance between efficient research strategies and formal research methodologies. So it seems that there needs to first be a common language between practitioners, researchers and lawmakers. Just in the discussion to this point I've seen data-driven, evidence-based, and research-based used interchangeably.

(Thinking outloud here) NCLB calls for research-based. Practitioners have seemed to attempt to redefine research-based with evidence-based as a means of skirting scientific practices to develop a mangeable means of meeting the legal requirements. Data-driven seems to be an inital step before jumping into evidence or research.

So, what are your thoughts? Is there a difference in the terms? Are all the terms relating an equally valued measure?

Beth

• Re:Data, Evidence, Research posted by Penny Reed on Dec 01, 2003

  Beth, That is a great question. I don't know that I have the answer. I think that data is necessary for both research based and evidence based practice. Data is the specific information that we collect. I agree with you that we often use these words without thinking of their relationship.
  
  Let me take a stab at explaining what they mean to me.
  
  Research based means formal research with a control group, random assignment to group, etc. It is very difficult for us to do that with AT because we dont' have groups, we have individuals scattered about in different settings. However, it is not impossible and we need our university based researchers to do more research on AT.
  
  Evidence based means that we follow the steps outlined by Dr. Hill in asking specific questions and utilizing what has been learned from research as well as what we have learned about the specific student.
  
  Data driven would seem to be similiar to evidence based, but one could overlook the research or external evidence and still be "data driven" using only the child specfic data. So, I think that our goal as practitioners should be to be evidence based and within that we make decisions that are driven by both child data and external evidence.
  
  Does that help at all? Or did I just make it worse? Communication is such a challenge for all of us sometimes!

• Re:Data, Evidence, Research posted by Katya HIll on Dec 03, 2003

  Beth,
  
  I do believe we have different meanings or implications on the use of terms. Of course, so much of educational/health care semantics is our own perspective. I believe we need to first look at the original sources in defining the terms. So returning to how the terms are applied in evidence-based medicine helps us as we start to apply evidence-based practices in our fields. Remember, the term evidence-based education is now being used, also.
  
  From attending and presenting courses and seminars on this topic, I realize we do not place the same level of importance on the different components for evidence-based practice. To use a metaphor, the emphasis on research-based, data-based, and evidence-based takes on a different shade of a color and sometimes the hues can be at far ends of the spectrum.
  
  For example, to me research-based is most closely related to external (field) evidence. When researchers or those in academia tend to present, the importance of peer-reviewed research evidence is stressed, sometimes to the point of ignoring other data and evidence at the personal level. Those of us in academia can understand why is important. However, we’ve already discussed the difficulty in finding adequate research or external evidence to answer our questions about AT/AAC. That being the case, we don’t stop providing services, because our search and appraisal didn’t result in a conclusive answer.
  
  Data-based implies that I’m using some quantitative data to support my decisions. However, David Edyburn and Roger Smith of the ATOMS Project have identified the unavailability of large AT databases. In addition, this type of data is frequently collected based on what question(s) was/were asked. If I’m a funding agency my data may only reflect cost effectiveness of outcomes and not clinical or educational outcomes.
  
  To me evidence-based implies equal consideration is given to all three components of evidence-based practice: external evidence, evidence at the personal level, and the knowledge and skills of the clinician. When one of the components is weak, then I’m recommending the other two components have to be stronger. I believe practitioners should be concerned that organizations, agencies, and policies start to use the term research-based in it’s narrowest interpretation. We just don’t have the clinical studies or trials equivalent to medicine to be able to answer our questions if this happens.
  
  Your thoughts are very critical as we start applying these terms. Thanks for offering the comments.
  
  Katya
  

  • Re:Data, Evidence, Research posted by Blaise Liffick on Dec 06, 2003

    If I could add to this discussion from a different perspective, namely the subfield of human-computer interaction (HCI) within computer science (CS)...
    
    This area of CS is largely about usability. Toward that end, we spend a lot of time doing "usability studies," which are clearly distinct from formal scientific experiments. It sounds to me like evidence-based practice is like usability studies, where the emphasis is not on proving an hypothesis, but on collecting data to demonstrate the relative accomplishment of some goal. While such studies clearly do not carry the weight of formal experimentation, their purpose is also clearly different, typically showing the relative effectiveness of some hardware, software, technique, methodology, etc. Such studies are not meant to "prove" something in the formal sense, but instead to act as a guidance mechanism for future decisions. There may be some value to relating this "new" area of evidence-based practice to the (mostly) accepted practices of HCI, which has been doing this sort of thing for 20 years or so.
    
    My current work is to look at this connection between HCI methods and the assistive technology field...
    
    Blaise Liffick
    Department of Computer Science
    Millersville University

    • Re:Data, Evidence, Research posted by Katya HIll on Dec 09, 2003

      Blaise,
      
      I love your comments! Thanks for mentioning this area of research and sharing it with the audience. HCI "usability testing" is frequently used at the R&D level of AT/AAC product development. However, consumers rarely see the data. This methodology was used in the SBIR development and testing of LAM and with PeRT. Heidi Koester and Ed LoPresti used similar methods in COMPASS development. I find like-minded interest in HCI at the RESNA conference.
      
      It would be interesting to see where people would identify "usability studies" as Levels of evidence.
      
      Thanks for making us look at the question from a different field.
      
      Katya
      
      

I've noticed that over the past few years many more software programs have become available incorporating built-in data collection technology. One example of a company that has been collecting data and providing progress reports for many years is Laureate Learning Systems. After completing each lesson, the user sees how many attempts they made at various activities and their success rate. Don Johnston's Start to Finish book series offers a comprehension quiz after each chapter and right after completing the quiz, the user gets immediate reinforcement and can see how well they did. Even some commercial software programs offer progress reports. One example is Knowledge Adventure's Curious George Reading and Phonics, another is Disney's Winnie the Pooh Toddlers.

Jim mentioned earlier that the ATIA might be interested in getting involved with data collection or setting up a national database. How about having a discussion about a standardized method for collecting data from various software programs and pieces of technology? I'm envisioning a standard progress report based on NCLB and grade level competencies. I know it would be very difficult to get all (or even most) vendors on the bandwagon, but what if the progress reports were automatically emailed to a central location where the data could be anonymously tallied and the effort to collect and share the data was minimal?

I think teachers and parents would be more likely to collect data if software publishers included an easy to use progress report that could be printed and saved for later review. If the progress report is built in, there is really no excuse not to use it. It's just a matter of interpreting the data given.

I know Dr. Hill has some experience with AAC devices which track user usage and provide data, perhaps she will share her insights on automatic data collection.

Perrine Dailey

• Re:Using data collection tools built into software & technology posted by Katya HIll on Dec 03, 2003

  Dear Perrine,
  
  Since working on the NIH research task to develop the language activity monitor (LAM), the use of data logging has become routine for many clinicians and is being taught at several Universities. However, the early and first adopters of LAM were individuals who rely on AAC and parents. I feel their support pushed research and development efforts. (refer later to external evidence thread). In many areas of AT, consumer involvement drives advances which is exciting, but can sometimes seem threatening when we feel we have to catch-up.
  
  At Closing the Gap 2003, the AAC Institute presented a session on the Performance Report Tool (PeRT) software which was conceived as a product during the two NIH grants involving LAM. At that session, a language sample was collected from an individual using an AAC system. As the audience asked questions, a logfile was projected on the screen for everyone to see. At the end of the discussion, the logfile was saved on the computer. In addition, the logfile from the built-in feature in the AAC system was uploaded and saved. This allowed the audience to see the two methods available to collect LAM data. The saved logfile was opened in PeRT, analyzed, and a report with 17 summary measures was generated. Along with the Power Point presentation, this activity was done all in less than 1 hour. This hopefully demonstrated the possibility of routine reporting of quantitative data to support decisions and monitor change.
  
  I can discuss the multimodal nature of communication and the limitations of LAM data if asked. However, you should know that when I first proposed this feature, my goal as a clinician was wanting to gain a better appreciation of how an AAC system was being used and wanting to gather data on the “user-device” interface. These data cannot be captured with audio, video, or traditional methods of language sampling recording.
  
  Since many people on the discussion are probably at different levels of experience with LAM and other tracking tools. I’m open for specific questions on the topic, because (as those who know me) I could go on, and on, and on….
  
  Katya
  

  • Re:Using data collection tools built into software & technology posted by Discussion Board Guest on Dec 16, 2003

    as a parent who uses LAM to monitor my chil'd progress, I can assure people that it really is easy. It is as if someone copied down everything the child said, what time it was said, and how it was said. I can assess who well the child can physically access the device, what kind of utterances were said, and how they accessed the words. By looking at the LAM, I can not only see that the device is working for my child, but I can use it to select which new words to target to help improve the communication skills. Armed with this information, I can use teacheable moments in daily life to introduce those new targeted words. Once again, a peek at the LAM tells me if the word I taught has become part of the child's regular vocabulary. Anyone who is nervous about using LAM should try it just once. It can really increase learning!

I am a training coordinator /AT consultant at Michigan Assistive Technology Resource (MATR) Supported by Capacity Building Grant)
We need to develop tools to evaluate our services (training and training materials) and also show how its impacting students performance. We are also getting feedback from educators in the school districts we support that they are also being asked to prove (show data) the same.
There are so many factors that impact student performance its nearly impossible to determine whether a students success is based on the tool, the teachers training (teachers are also unique learners themselves) ,student ability, environmental factors, teaching style etc...safe home environment
Does any one have eval tool models for assessing effectiveness on teacher training or training materials?
and can we realistically expect to tie training quality to changes in student performance.
This is a different area of data collection than device effectiveness. Much more nebulous than counting or measuring speed, accuracy or frequency.

How do you measure educator’s skills around what they learned? Most post training tools ask more questions around the learners perceptions and do not go as far as finding out if the learner is actually implementing what they learned.
Another issue specific to our organization is that we are one step removed from direct student contact. We support training state wide to local school districts.

I just completed grading two classes worth of evidence-based practice (EBP) notebook projects. No small task! One of the first steps in applying the principles of EBP is formulating the best possible question for the patient/client/student. Good questions aren’t possible without a thorough understanding of the individual and problem. In my classes and workshops I go over the PICO model for formulating a question. PICO stands for Problem, Intervention, Comparison, Outcome. EBP questions can be built around these four components, however, you not required to seek evidence that compares intervention approaches.

I felt the students in my AAC course asked very relevant, critical clinical questions typical of the questions asked in workshops and the challenges faced by today’s AT teams. Here are two example questions:
> For children with autism, in comparing use of PECs and AAC voice output systems which intervention approach results in the most effective independent communication?
> For adults with CCN and significant neuromotor deterioration, is Morse Code an effective input strategy for maintaining functional, independent communication?
Other questions included 1) comparing language representation methods; 2) comparing language application programs, 3) comparing dynamic versus static display AAC systems.

I’m sure this audience wouldn’t be surprised at the conclusion students arrived at after searching and appraising the evidence. Several student teams had emailed field leaders on the question topic to seek advice about where to find more substantial evidence. I’d like to publicly thank Tom King for his correspondence on Morse Code, and Joanne Cafiero for her input on autism. The students had some nice role modeling.

Now I’d like to ask participants, what’s the burning EBP question you would like answered? Is it around a particular population? A specific intervention approach? A comparison of approaches/technologies? The results would be beneficial for all of us to see. (If we have enough responses, I'll do some tabulation.) We may be surprised at the diversity or lack of differences in what evidence we’re searching for to help support our decisions. Of course, the next question will be what’s the best piece of evidence you can find to answer your question.

Looking forward to your input.

Katya

• Re:Asking EBP questions – what’s your question? posted by Marge on Dec 07, 2003

  Katya and other school practitioners,
  
  I particularly appreciated this posting and wanted to contribute to this tread. As a public school SLP two years out of graduate school, I know the emphasis placed on research and univsersity faculty views. But I feel overwhelmed in keeping up with all the research needed to keep current on all the communication disorders on a school caseload. No communication area seems more critical than AT/AAC, since that’s such a small number in my school district. I feel that much of the knowledge and skills I had with AAC when I took my AAC course (over five years ago) doesn’t apply with what is being considered and used by our school teams.
  
  For me the most compelling question is about the AT/AAC assessment process. What does the research say about how long it takes to assess a student for an AAC system? How many trials on different systems are needed before making a decision? Many times I feel that the device trials are more “trial and error” approaches to say that we’ve conducting an assessment rather than identifying common variables to compare. I’m wondering if the annual nature of the IEP somehow influences the length of our assessment process.
  
  I want to make the best decision possible for the student, but see frustration by all team members when the process goes on and on and on and no final decision is made. We're using various documents from state AT programs suggested on earlier threads to support assessment, but these were adminstrative decisions. In the meantime as we move through the trial process, the student’s communication skills seem stagnant. I’m wondering if other school-based SLPs feel discouraged by the process and pressure too. What does the research-base offer to address these questions? What research are other SLPs in schools using to choose their AT/AAC assessment model?
  
  I'd love to hear from other school-based AT team members. Thanks for some comments and suggestions!
  
  Marge
  

  • Re:Asking EBP questions – what’s your question? posted by Katya HIll on Dec 09, 2003

    Hi Marge and others,
    
    I'm waiting to give a reply, but really am hoping some school-based AT team members will jump into the discussion first. Maybe this quote from the "Evidence-Based Education Manifesto" will get discussion participants thinking about how the AT assessment policies and procedures were identified for their school system.
    
    "It is part of a pre-scientific approach to social "science" that would look extremely incongruous in applied sciences such as engineering or medicine. No one would advocate building a particular structure, investing in an industrial process or using a medical treatment without good evidence to justify it. But it is part of the culture of social science (education) that opinion can often have a status equal to - or even greater than - that of evidence as a basis for action.
    We need to change that culture so that the question, "Where is the evidence?" becomes the first thing we think of when presented with a suggested change of practice or policy."
    -quote from EBE Manifesto
    
    I really appreciate your question and concerns, and would like to ask my own related questions. How long does the typical AT or AAC assessment take? What is a typical (average) number of trials? What quantitative data is used to compare trials? Do team members feel (outcomes not needed to have an opinion) that it's an effective process?
    
    Katya
    

  • Re:Asking EBP questions – what’s your question? posted by Katya HIll on Dec 15, 2003

    Assessment is the most frequently requested topic for seminar presentations and continuing education courses when I survey audiences. Several questions about the AT/AAC assessment process were asked in this posting. Since the posting included a request for school practitioners to respond, I’ll assume that the client is a school-aged child who might benefit from AAC. Now I’ll pose a question using the COPES concepts from Gibbs (2003). Of course, others can substitute another population or specific communication disorder. I’ll insert generic assessment models. What AT/AAC assessment models come to mind? What model(s) are you using? From the original questions, I believe the important measures are the time, expensive, and reliability that will result in the most effective communication. Therefore, the question might be, “if school-aged children with complex communication needs are assessed for an AAC system using Assessment Model A or Assessment Model B, which assessment model will be the briefest, most inexpensive, valid, and reliable to determine the AAC system that will result in independent communication.
    
    To date, no empirical data are available that test the validity and reliability of an AT/AAC assessment process, and I’m not aware of research that has compared the most commonly cited assessment models. The field consensus on the AT/AAC assessment process has been based on authoritative literature (evidence). Most practitioners agree that they follow the “feature-match” process. Many state AT programs have assessment forms designed around feature-matching. Yet, no research-base has been established over the years that documents the outcomes of applying “predictive profiling” or “feature-matching.” I believe that the term feature-matching gained popularity from the Pennsylvania Assistive Device Center, which is now PATTON. However, early use of these concepts comes from Yorkston & Karlan (1986), Costello & Shane, (1994), Glennen (1997).
    
    My favorite definition of the AAC assessment is “a process whereby data are collected and information is gathered to make intervention and/or management decisions.” (Lloyd, Fuller & Arvidson, 1997, p. 524). Now teams need to be conscientious and judicious in what data are being collected and compared, and what data different models consider as the most valid and reliable for decision-making. You may wish to compare the following models:
    >The Participation Model (Beukelman and Mirenda, 1998)
    >The Human Activity Assistive Technology (HAAT) Model (Cook and Hussey, 1995)
    >Matching Person and Technology (MPT) Model (Scherer, 1991)
    >Language-Based Model (Hill, 2000)
    
    A very wise place to start after reviewing assessment models and identifying the model being followed by your team is to start collecting short and long-term outcomes data as you complete the process. Keep a scorecard on the research being published, but keep your own scorecard on your outcomes too! In the meantime, I'll be happy to compare my outcomes with you. Your team may have something to share with the rest of us!
    
    Katya
    
    References:
    
    Beukelman, D. R., & Mirenda, P. (1998). Augmentative and alternative communication: Management of severe communication disorders in children and adults (2nd ed.). Baltimore: Paul H. Brookes Publishing Co.
    
    Cook, A. M., & Hussey, S. M. (1995) Assistive technologies: Principles and practice. St. Louis: Mosby.
    
    Costello, J., & Shane, H. (1994). Augmentative communication assessment and the feature matching process. Presented at the annual convention of the American Speech-Language-Hearing Association, New Orleans.
    
    Glennen, S. L., & DeCoste, D. C. (1998). Handbook of augmentative and alternative communication. San Diego, CA: Singular Publishing Group.
    
    Hill, K. (2001). Achieving success in AAC: Assessment and intervention. Edinboro, PA: AACI Press. Vol. 1, No. 1
    
    Lloyd, L. L., Fuller, D. R., & Arvidson, H. H. (1997). Augmentative and Alternative communication: A handbook of principles and practices. Boston: Allyn and Bacon.
    
    Scherer, M. J. (1991). Matching person and technology (MPT). Rochester, NY:Author.
    
    Yorkston, K., & Karlan, G. (1986). Assessment procedures. In S. Blackstone (Ed.), Augmentative communication: An introduction (pp. 163-196). Rockville, MD: American Speech-Language-Hearing Association.
    
    
    
    
    
    
    
    
    
    
    
    

• Re:Asking EBP questions – what’s your question? posted by Jessica Sunday on Dec 08, 2003

  Katya,
  In response to you asking for "burning EBP questions" that we would like answered... here is one that follows the PICO model that you also mentioned: In preschoolers with cerebral palsy and complex communication needs, do voice output devices support language development and build independent communication? I have been searching exhaustively for peer-reviewed evidence, and cannot find any that has helped me answer my question. However, we know that AAC systems are recommended frequently for this population with good results being presented.
  Thank you for the help!
  Jessica
  

  • Re:Asking EBP questions – what’s your question? posted by Ralf Schlosser on Dec 08, 2003

    Jessica,
    
    In searching for evidence it is best to first seek out a review, preferably a meta-analaysis. In absence of a meta-analysis may I suggest the following narrative review, which was part of a special AAC issue on speech output:
    
    Schlosser, R. W. (2003). Roles of speech output in augmentative and alternative communication: Narrative review. Augmentative and Alternative Communication, 19. 5-28.
    
    While the review is not specifically focused on your question, it does contain sections for various outcomes of using speech output including various communicative functions such as requesting and commenting. At the very least, this review will lead you to individual studies that bear relevance to your question. Hope this is helpful and best of luck.
    
    Ralf W. Schlosser, PhD
    Associate Professor
    Department of Speech-Language Pathology and Audiology
    151C Forsyth
    Boston, MA 02115
    Tel. 617-373-3785
    Fax. 617-373-2239
    R.Schlosser@neu.edu
    
    
    
    
    
    
    
    
    
    
    

  • Re:Asking EBP questions – what’s your question? posted by Katya HIll on Dec 09, 2003

    Jessica,
    
    I was going to refer you to some of the chapters in Ralf's book, but he beat me to it. Here's the difficulty with the question as you formulated it. In providing AAC services and supports to preschoolers with CCN, an extremely wide range of cognitive-linguistic skills as well as motor skill challenges exist. When you're thinking about this population or a specific child, are you considering abilities at the presymbolic stage? Or considering a beginning communicator? Since your outcomes concerned language development and independent communication, I'm assuming you're referring to preschoolers beyond the symbolic stage.
    
    Here's where clinicians in the trenches have difficulty with their questions being answered directly. For example, you will find a rather extensive number of peer-reviewed studies on graphic symbol selection along with studies identifying inital vocabularies. However, I believe an analysis of the research-base indicates limited implications for clinical practice. At least my graduate students arrived at that conclusion after a meta-analysis. Only a few (if any) of these studies provide outcomes related to how symbols are used or language is generated on commercially-available AAC systems. I believe this is the heart of your question or the decisions you face.
    
    Consequently, your knowledge and skills are critical in interpreting this evidence and continuing to seek and appraise evidence at other (maybe lower) levels. You may need to disclose the limited peer-reviewed or Level I evidence you have in making a recommendation. However, when you build testing (data collection) into the intervention process you can measure learning to monitor change in a timely manner.
    
    I usually discuss with teams and parents the evidence (or lack of) being used to guide decisions, and we've measured what isn't working. So the question becomes, are we willing to try "this" intervention based on the external and personal evidence?
    
    I have my favorite evidence for this question, but I'm saving it for later.
    
    Keep tuning in,
    
    Katya
    

• Re:Asking EBP questions – what’s your question? posted by Katya HIll on Dec 13, 2003

  Practitioners seldom ask specific questions that are central to good judgment and decision-making about practice (Gibbs, 2003). Evidence-based practitioners realize time cannot be spent on seeking and appraising evidence that doesn’t produce usable outcomes. Therefore, we need to formulate the best questions that result in the most valid answers for our clients within a reasonable time. Frequently, formulated questions (by both practitioners and researchers) do not meet rigorous criteria for a precise, clinically relevant answer. Of course, the knowledge and skills of the practitioner are required to appraise and interpret the evidence.
  
  Gibbs provides specific guidelines for posing questions based on “client-oriented, practical, evidence-search (COPES) concepts. He identifies five types of practice questions: effectiveness, prevention, assessment, description, and risk. The four features of a well-built question include:
  1) client type and problem,
  2) what you may do,
  3) alternate course of action,
  4) what you want to accomplish.
  
  Below are some example questions from Gibb’s book. See if you can identify the four features of each question.
  
  Effectiveness question: If disoriented aged persons who reside in a nursing home are given reality orientation therapy or validation therapy, which will result in better orientation to time, place, and person?
  
  Assessment question: If aged residents of a nursing home who may be depressed or may have Alzheimer’s disease or dementia are administered depression screening test or short mental status examination tests, which measure will be the briefest, most inexpensive, valid, and reliable screening test to discriminate between depression and dementia?
  
  Description question: If members of a hospital team who are concerned about team functioning take the preliminary checklist (clinical) team effectiveness test or take the interdisciplinary team weekly inventory, which measure will most reliably and validly reflect the team’s ability to accomplish tasks?
  
  Think of the question(s) most relevant to the clients you provide services to and then identify the question type along with formulating a specific question using the four features of a well-built question.
  
  If you don’t have a book about evidence-based practice in your library or are thinking about expanding your EBP references, this is the book I recommend as my best pick:
  
  Evidence-based Practice for the Helping Professions: A Practical Guide with Integrated Multimedia by Leonard E. Gibbs from Thomson Brooks/Cole, Pacific Grove, CA. 2003. ISBN 0-534-53923-8
  
  Good luck using the COPES concepts to pose some questions important to you.
  
  Katya
  
  
  
  
  
  

This has been a very interesting discussion thanks to the two presenters and the many people who have contributed. This is my first posting and I wanted to take a few minutes to share some of my thoughts on what has been said.

What is the process?

Earlier a process has been proposed that began with an individual’s performance. Most process steps published in text books and the peer-reviewed literature begin with the asking of a well-built question (e.g., Helewa & Walker, 2000; Law, 2002; Sackett et al., 1997; Schlosser, 2003a; Schlosser & Raghavendra, in press). This question usually stems from a clinical or educational uncertainty about how to proceed with and for a particular individual. I see know logical reason to start with performance. Schlosser and Raghavendra (2003) have argued that it would be beneficial to also disseminate what has been learned from the process. For example, a practitioner who finds limited help from research evidence should pass on her knowledge of this gap so that it can be tackled in the future. Otherwise researchers will not learn what future research should be done. This would result in the following process:

1. developing a well-built question
2. selecting evidence sources
3. executing the search strategy
4. examining and synthesizing the evidence
5. applying the evidence
6. evaluating the application of the evidence, and
7. disseminating the findings.

How much evidence do we need?

Curtis Griesel suggested earlier that EBP only becomes effective when there is a large body of external evidence. This is one of the many myths surrounding EBP (for other myths see Schlosser, 2003b). The fact is that there is never ever going to be enough evidence. Because of the complexity of human nature and its interaction with AT there will always be unforeseen scenarios that evoke clinical/educational uncertainty. Even in such fields as medicine, which has a huge evidence base, you will hear doctors saying that we need more evidence. So, what the EBP process is asking us to do is to identify the BEST and MOST CURRENT evidence at any given time. Sometimes, a case study is the best and most current we have. The alternative to considering this evidence is to do nothing until the evidence comes to a level that we consider THE BEST. This does not sound like a good alternative. Of course, having an extensive research base would help, but it is not necessary by any means to implement EBP.

What is the role of relevant stakeholders?

So far the role of stakeholders in EBP has been neglected in this discussion. Yet, in my mind stakeholders have primacy in EBP, after all it is their lives that are most affected by the decisions being made. That is why EBP for the AAC field has been defined as “…the integration of best and current research evidence with clinical/educational expertise and relevant stakeholder perspectives to facilitate decisions for assessment and intervention that are deemed effective and efficient for a given direct stakeholder” (Schlosser & Raghavendra, 2003, p. 256). This definition should be applicable to AT as well. The direct stakeholder is usually the individual using AAC or AT. In addition to the direct stakeholder, relevant other stakeholders, depending on what decision is at hand may include the direct stakeholder family members, friends, peers, or even employers and laypeople. We have proposed a diagram of the EBP decision-making process including these three cornerstones (research evidence, clinical/educational expertise, & stakeholder perspectives) and the only arrow into the decision-making box is through the stakeholder perspectives (Schlosser & Raghavendra, in press). It is conceivable that the stakeholders have perspectives that lead them to proceed differently or not at all. In the EBP we envision, research evidence should never translate directly into a decision and neither should clinical/educational expertise.

“Personal evidence”

In my opinion, the term “personal evidence” does not do justice to what is done in an AAC or AT assessment nor is it logical to separate it from clinical and educational expertise. For example, evidence is also collected about current and future environments, communication partners, rather than just the performance of the person per se. I would rather think of clinical and educational expertise as including everything that clinicians or educators are supposed to do based on the Knowledge and Skills Documents of their respective professions. Subsuming “personal evidence” under the larger umbrella of clinical and educational expertise is consistent with various EBP texts.

Research-based versus evidence-based

This has been an excellent discussion about terminology. The APA uses a similar term, namely “empirically-validated treatments” to refer to those treatments that meet certain quality standards. The term EBP provides us with a process that helps us incorporate treatments at the best level of evidence available into practice. Terms such as “research-based treatments” and empirically-validated treatments only tell us about the quality of the evidence, but don’t tell us how we get knowledge of these into our practice.

Appraising evidence from single-subject research

Dr. Higginbotham provided several resources for appraising evidence. I also wanted participants to be aware of several checklists that have been developed for that purpose and that might be useful (see Schlosser, 2003a).

Challenges we face

These are perhaps some of the challenges we face in moving EBP forward in AT and AAC (Schlosser & Raghavendra, in press):

1. Instruction in EBP knowledge and skills of practitioners
2. Instruction in EBP knowledge and skills in pre-professional programs
3. Building an adequate infrastructure for practitioners to engage in EBP
4. Making stakeholder participation in EBP work

Sorry for being so lengthy, but this way I was able to respond to many issues at once rather than reply to several individual responses. I very much enjoyed this discussion and wish that there were more opportunities like this to move EBP forward in AT.

Ralf W. Schlosser, PhD
Associate Professor
Department of Speech-Language Pathology and Audiology
Northeastern University
151C Forsyth
Boston, MA 02115
Tel. 617-373-3785
E-mail. R.Schlosser@neu.edu

References and Resources

Augmentative Communication News (2002). Issue on EBP. Issues 3-4.

Corwin, M. & Koul, R. K. (2003). Augmentative and Alternative Communication Intervention for Individuals with Chronic Severe Aphasia: An Evidence-Based Practice Process Illustration. Perspectives on Augmentative and Alternative Communication, 12(4).

Helewa, A., & Walker, J. M. (2000). Critical evaluation of research in physical rehabilitation: Towards evidence-based practices. Philadelphia: WB. Saunders Company.

Law, M. (2002). Evidence-based rehabilitation: A guide to practice. Thorofare, NJ: SLACK Incorporated.

Olsson, C. (2003). The EBP experiences of an AAC service provider: Diving in deep. Perspectives on Augmentative and Alternative Communication, 12(4).

Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (1997). Evidence-based medicine: How to practice and teach EBM. New York: Churchill Livingstone.

Schlosser, R.W. (2003a). The efficacy of augmentative and alternative communication: Toward evidence-based practice. New York: Academic Press.

Schlosser, R. W. (2003b). Evidence-based practice: Frequently asked questions, myths, and resources. Perspectives on Augmentative and Alternative Communication, 12(4), 4-7.

Schlosser, R. W. (2003c). Evidence-based practice: Meeting the challenge. Perspectives on Augmentative and Alternative Communication, 12(4), 3-4.

Schlosser, R. W., & Raghavendra, P. (2003). Towards evidence-based practice in augmentative and alternative communication. In R. W. Schlosser, The efficacy of augmentative and alternative communication: Towards evidence-based practice (pp. 259-297). New York: Academic Press.

Schlosser, R. W. & Raghavendra, P. (in press). Evidence-based practice in augmentative and alternative communication. Augmentative and Alternative Communication.

Sigafoos, J., & Drasgow, E. (2003). Empirically-validated Strategies, evidence-based practice, and basic principles in communication intervention for learners with developmental disabilities. Perspectives on Augmentative and Alternative Communication, 12(4).

• Re:EBP: What it is and what it isn't posted by Katya HIll on Dec 07, 2003

  Ralf,
  
  Thank you for your informative response. I’m sure participants will benefit from the information being archived as a long-term reference. I very much agree with the importance of appreciating that EBP is viewed and interpreted from the perspective of the stakeholder. I think various threads of the discussion could be used to highlight perspective and value differences among practitioners, administrators, academics, etc. The reason practitioners are placing value on applying the principles of EBP, is because we believe that EBP will result in better outcomes for the individual. The stakeholder with the biggest investment in the process is the individual receiving services.
  
  In teaching the principles of EBP, I start by reminding students and audiences that they are the stakeholders with the most to win or lose in regards to their own health care. So how do they expect health care providers to be making decisions when they have a health problem? I’m going to provide a recent example from my own life.
  
  This fall I had my own family experience with EBM in seeking treatment for my daughter who is having severe headaches. Our family physician prescribed common headache relief medication, but we advocated for more testing (reflecting my bias toward quantitative data) to determine the cause. From my experience with EBM, the problem has to be clearly defined before the physician can ask the best question to identify the best intervention and then monitor change. We accept as routine that the nurse or doctor “uses instrumentation to collect data in order to apply the principles of EBM (to paraphrase the ASHA Scope of Practice).
  
  To make a long story short, the MRI found a small tumor. I didn’t feel that the best EBM question had been asked and answered until we saw the neurosurgeon. This didn’t happen until the MRI was performed and we had collected specific data about the nature, time, duration, radiation of pain, etc. for the headaches. The neurosurgeon actually discussed the external evidence with my daughter and I at a visit. Evidence I might add that I had found by conducting my own Medline search and appraisal. However, we would have never even known to have narrowed the search to this external evidence if testing (collecting data to characterize the problem) had not been adequately performed. The neurosurgeon was not too impressed by the earlier intervention approach, by the way.
  
  I certainly would have changed neurosurgeons if at our visit something like the following would have been said, “I don’t bother running tests I have a lot of experience with these types of tumors.” Since that is what my primary providing physician said, we will be making a change!
  
  We might consider applying what could be learned from the above scenario to the delivery of AT services. As a coordinator of an AT program that provided services to 17 school districts, over a hundred teams were involved in AT decision making each year. As a consultant, I found that supporting teams to collect the most relevant performance data to ask and answer the best questions was the biggest asset I could provide. Once that was accomplished providing the external evidence was not a big challenge, and for the most part teams arrived at decisions that resulted in successful AT outcomes. Not all students with the same initial problem benefited from the same decisions. Clearly characterizing the student’s problem and performance led teams to consider different interventions, because different external evidence came into consideration. Involving students and families in the process increased appreciation for the decisions being made.
  
  Perhaps others will share their perspective on the contributions different stakeholders can make in the EBP process and/or what knowledge and skills are considered most important for different stakeholders. What are your viewpoints on how the EBP process can be implemented effectively yet pragmatically?
  
  Katya
  

For participants working in the educational sector, I’ve pulled slide material from one of my workshops on Evidence-Based Education (EBE). First, I provide the two most widely quoted definitions of evidence-based medicine and then reword those definitions to apply to education. Then the three ways education can become more “evidence-based” are listed along with a quote from the EBE Manifesto. I conclude with some recommended websites at Internet resources for EBE.

What is evidence-based medicine (EBM)?
“…conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”
- Sackett, et al., 1997, Centre for Evidence-Based Medicine website

What is Evidence-based Practice (EBP)?
“…an approach to decision making in which the clinician uses the best evidence available, in consultation with the client, to decide upon the option which suits that client best.”
- Muir Gray, 1997

What is evidence-based education (EBE)?
“…conscientious, explicit and judicious use of current best evidence in making decisions about the (special) education of a student.”
-adapted from Sackett

“…an approach to decision making in which the educational team uses the best evidence available, in consultation with the student/parent, to decide upon the option which suits that student best.”
- adapted from Gray

EBE Manifesto
There are three main ways in which education could become more "evidence-based". These concern:
1. development of evidence-based policies,
2. applying evidence-based practice
3. general promotion of a "culture of evidence".
- from E-BE UK website

Culture of Evidence
• It is part of a pre-scientific approach to social "science" that would look extremely incongruous in applied sciences such as engineering or medicine. No one would advocate building a particular structure, investing in an industrial process or using a medical treatment without good evidence to justify it. But it is part of the culture of social science (education) that opinion can often have a status equal to - or even greater than - that of evidence as a basis for action.
• We need to change that culture so that the question, "Where is the evidence?" becomes the first thing we think of when presented with a suggested change of practice or policy.
- quote from EBE Manifesto

For more information:
http://www.hsl.unc.edu/lm/ebm/index.htm
•Teaching/Learning Resources for EBP at www/rctsh/ebp/main.htm' title='http://www.mdx.ac.uk/www/rctsh/ebp/main.htm'>http://www.mdx.ac.uk/www/rctsh/ebp/main.htm
• Evidence Based Education UK (E-BE UK) http://www.cemcentre.org/ebeuk/default.asp
• Parents for Evidence Based Education at http://www.educationallycorrect.com
• From the US Department of Education: http://www.ed.gov/nclb/methods/whatworks/eb/edlite-slide001.html


My father received his Ph.D. in education from the University of Minnesota. Actually, one of the first in his program to use a computer (that took up a whole floor) for statistical analysis. He instilled in me at an early age the importance of using evidence (and citing the data) in making sound educational decisions.

Hope those in education find the information useful and enjoy some of the sites.

Katya

• Re:Evidence-Based Education posted by Discussion Board Guest on Dec 16, 2003

  Definitely, more information about evidence-based education (EBE) is becoming available. This may be a sign of the times to find resources on EBE at this website: http://www.ed.gov/about/offices/list/ies/news.html#guide.
  
  

We’ve had quite a bit of discussion on the types of external evidence needed for evidence-based practice. With several comments emphasizing the importance of peer-reviewed evidence. However, we’re also aware that extensive searching of the AT/AAC peer-reviewed literature may not even begin to answer the question.

One of the finer points of EBM is seeking Patient-Oriented Evidence that Matters (POEM) once you’ve asked your question. POEMs have to meet three criteria:

>The evidence addresses a question faced by clinicians.

>The evidence measures outcomes that clinicians and patients care about, such as symptoms and quality of life.

>The evidence has the potential to change the way we practice.

POEMs are valid, show improved important patient outcomes, and change what we do. Therefore, as practitioners we have an obligation to seek and appraise the evidence on these criteria. One of the major skills I want my students to acquire is the ability to distinguished between peer-reviewed evidence and POEMs. In evaluating evidence used to support therapy plans, unless students can discuss how the evidence addressed the question, measured meaningful outcomes, and had the potential to change the delivery of service, they had to keep looking. Many times they stop at the keywords and assume, here’s evidence.

Participants on the discussion might want to do a search on “patient oriented evidence that matters” to see what you find. Take a look at http://www.infopoems.com/description.html as the source of POEM information. Although medical, you'll see good examples of how POEM is applied.

Katya

• Re:Patient Oriented Evidence that Matters posted by Discussion Board Guest on Dec 12, 2003

  Dear Katya,
  
  The evidence I feel would be of most value to my clients, families, and myself is related to comparing the various AAC language application programs. I’m not sure how I’d phrase the question. I see several new language programs being offered by various manufacturers. However, I haven’t been able to find comparison information on the various programs, and even less on outcomes possible using a specific program. In fact, I don’t see a very practical or extensive research-base for most of the AT/AAC questions that meet the POEM criteria.
  
  So if researchers aren’t asking the right questions, how can clinicians hope to find the research evidence we need for EBP?
  
  Just some thoughts as I’m watching the snowfall.
  
  
  

This discussion has been extremely rich and we're grateful to Dr. Hill and Dr. Reed and to the people who have posted. We know from Webalizer data that hundreds of others have been following the discussion silently. The discussion will end on Wednesday, December 17th, so I urge those of you who may have been laying in the weeds to post your comments and questions soon. The discussion will of course be archived for future reference, so feel free to return and to refer colleagues to the site. We will also produce a summary of the issues addressed and the information shared, which will be available on both our website and on our next AT Resources CD-ROM. On January 12th we'll begin our next online forum - on Website and CD-ROM accessibility - led by two of the country's leading accessibility experts. We hope to see you there.

I was reading through this discussion thread today and wanted to add some thoughts and say I have enjoyed reading all the posts. Great information.

I am working with a group of people to facilitate a shift to a “research oriented” culture, where teachers and other professionals use data analysis as a standard operating procedure when making decisions that affect students. As has been noted in prior discussion, this can be difficult for professionals who are extremely busy already. I am currently using two strategies to help with this shift. One is reducing the fidelity of the research process, similar to the way a software simulation is not as complex as the real thing. An example I noted in the prior discussion was the collection of hard copy, or print output, as a data collection procedure. While it still left gaps, such as how long each copy took to produce, it was a way for a professional to start to get used to collecting data. Becoming aware of the gap and addressing that issue could be a great way to increase the fidelity of the research model over time. I chose this strategy because of the research that indicates novice learners will learn faster using a low fidelity model, with increasing fidelity being provided as the learning increases.

The second strategy I am using is embedding this entire process in professional development activities. In Florida, Action Research and Study Groups are recognized professional development activities that teachers can get inservice points for. It is completely acceptable to take a particular student case and formalize the research and decision making process, which is exactly what evidence-based practice is. The Study Group model is a great way to formalize meeting together and discussing findings. It makes it much easier for teachers to collect data and meet on/discuss data when they are getting inservice points for what they are doing, and their administrators see it as “state-of-the-art” professional development, than if they are doing it but it seems to have no relationship to their professional development plan or program. Tying it all together to create a learning community helps to push a change in culture.

David Davis

• Re:Changing the Culture posted by Katya HIll on Dec 16, 2003

  David,
  
  Thank you for sharing these strategies. Effective adult learning strategies include building on incremental steps toward a goal, doing, and teaching. In my classes I use the Multidisciplinary Interactive Team Thinking Test (MITTT) video clips from Len Gibbs (2003) to illustrate EBP principles. Students and practitioners learn by example, and the clips stimulate some good discussion on how to start with EBP. You might want to consider this resource in your professional development activities.
  
  I really like your approach to building skills through the study groups, because an IEP meeting is not the place to start. Although large workshops can introduce the concepts, small working groups are needed to acquire the skills. In talking with Len, he told me the final in one of his classes was based on posing an EBP question to small teams of students who then needed to conduct the search, appraise the evidence, and answer the question all within the class period. He was excited to report that all the students successfully accomplished the task! May we all reach this level of performance.
  
  We're just starting to change "our culture" to accept as second nature posing EBP questions and following through to seek the best answer. EBP is NOT a new term for something we've always been doing. That's important for administrators to understand. You may be interested in attending the AAC Institute Symposium on EBP this June in Orlando. Len is one of the presenters, and I'll be including information in another posting.
  
  Thanks for your contribution.
  
  Katya

  • Re:Changing the Culture posted by Discussion Board Guest on Dec 16, 2003

    Katya,
    
    I am very interested in getting more information on how Len set up his final activity, the types of information and data he supplied, etc. I would appreciate it if you could let me know how to contact him to request this information, or rely my request with my email to him. davisd@paec.org
    
    One of the things I have been doing this year is opening my workshops with a metacognition activity where the participants practice identifying specific student needs and issues (rather than broad issues like ".... can't write"). They then list interventions they have found to be useful and must describe what data they used to decide the intervention was useful, which leads into a discusson on data. They write their respones on a form that they keep during the training to refer back to as they get new ideas. It's a simple, "loose" activity but it does help establish a mindset or way of thinking.
    
    I agree that EBP is new and not business as usual, which is why I think it is perfect for setting up a professional development activity in such as way that participants get points for the actual work they are doing. I think it is great.
    
    Thanks for your information and any help you can give me in getting up with Len.
    
    David

    • Re:Changing the Culture posted by Katya HIll on Dec 17, 2003

      David,
      
      Len's book uses a tutorial approach providing practice examples to build skills in posing questions and then conducting a search. He presents step-by-step screen captures of websites showing the search process. I assume he required students to perform these steps in his final. Len has taught at the University of Wisconsin-Eau Claire since 1977. His background is in psychiatric social work. I'll send you his email address privately.
      
      I know Len will be using the multimedia materials he has developed during his presentation at the Symposium on EBP on June 19th as part of the RESNA conference. Hope you can make it. Specifics about the Symposium will be up at the AAC Institute website soon at www.aacinstitute.org. Also, I will be presenting at ATIA in January as well as at the AACI and RESNA booth in the exhibit hall. Please, stop by if you attend so we can chat and share professional development ideas.
      
      Katya

I wanted to take an opportunity before tomorrow to share my favorite AAC evidence. Perhaps after reading my list, others will be motivated to share some AT/AAC POEMs they find valuable. What evidence may have made you change the way you did things or look at problems differently?

One of the questions on my AAC final required the students to identify and provide the reason for selecting the evidence they found best met the POEM criteria in answering their EBP question. Brown (Brown’s Stages) was identified by the largest number of students, Romski & Sevcik’s work came in a close second, one of Beukelman et. al. studies on vocabulary frequency was cited depending on the cohort looked at by a student. (I was pleased that only one student mentioned my work.)

These nine references are listed in alphabetical order, not by level of evidence or my frequency of use. Many of the authors have several publications that could have been selected. However, all nine articles had some impact on my clinical thinking. I will follow the model of my students and not include any of my work, although I’m tempted to show some favoritism. I’m leaving the tenth slot blank in honor of Chomsky or the contribution linguistics make to AAC in general.

Enjoy some good reading for the winter!

Katya

Favorite AAC Evidence:

Balandin, S., & Iacono, T. (1999). Crews, wusses, and whoppas: Core and fringe vocabularies of Australian meal-break conversations in the workplace. Augmentative and Alternative Communication, 15, 95-109.

Beukelman, D., Yorkston, K., Poblete, M., & Naranjo, C. (1984). Frequency of word occurrence in communication samples produced by adult communication aid users. Journal of Speech and Hearing disorders, 49, 360-367.

Brown, R. (1973). A first language, the early stages. Cambridge, MA: Harvard University Press.

Gardner-Bonneau, D.J., & Schwartz, P.J. (1989). A comparison of Words Strategy and Traditional orthography. In Proceedings of the RESNA 12th Annual Conference. Arlington, VA: RESNA Press. 286-287.

Jagacinski, R.J., & Monk, D.L. (1985). Fitts' law in two dimensions with hand and head movements. Journal of Motor Behavior, 17, 77-95.

Koester, H.H., & Levine, S.P. (1994). Modeling the speed of text entry with a word prediction interface. IEEE Transactions on Rehabilitation Engineering, 2 (3), 177-187.

Miller, J. F., & Chapman, R. (1984). Disorders of communication: Investigating the development of language of mentally retarded children. American Journal of Mental Deficiency, 88, 536-545.

Romski, M. A., & Sevcik, R. A. (1989). An analysis of visual-graphic symbol meanings for two nonspeaking adults with severe mental retardation. Augmentative and Alternative Communication, 5, 109-114.

Schneider, W., & Fisk, A. D. (1982). Degree of consistent training: Improvements in search performance and automatic process development. Perception & Psychophysics, 31(2). 160-168.

Dear Discussion Participants,

This has been a great pleasure co-leading this discussion with Penny. The volume of information has been a lot to digest, and we’ve all helped to develop a rich archive of comments on evidence-based practice related to AT/AAC service delivery. I have been overwhelmed with private emails from discussion participants, too. I would like to take this opportunity to mention that I will be replying back to everyone personally, although it may take some time. I appreciate your consideration with the holidays fast approaching.

Evidence-based practice (EBP) is really not a new term for the same old ways of doing things. EBP is a paradigm shift for practitioners, other stakeholders, and most important, individuals who rely on AT/AAC. Applying the principles of EBP requires all of us to build new skills which we never master. EBP is a life-long process of posing the best questions in order to seek and evaluate the best evidence to make the best decisions with our clients. Data (from several sources) is critical to effective decision-making. Results of a search conducted today should not be the results of the future.

The final thoughts I would like you to consider are based on a metaphor used by Stephen Covey in Principle-Centered Leadership (1990; ISBN 0-671-79280-6). Covey discusses the difference between giving someone a map to follow or a compass. Consider the principles of EBP as territories. Maps are not territories, but only subjective attempts to describe or represent the territory. Approaches to give practitioners maps to follow will have a limited impact on our effectiveness, especially when the roads haven’t been built and detours exist. We’re very accustomed to being provided with maps to follow an educational or service delivery approach.

A compass is an instrument used to navigate or direct a course. Teaching someone to use a compass empowers them with the skills to make decisions and direct a course when the map is not helpful or obsolete. I was encouraged in my doctorate program to think about the tools and resources needed to support practitioners applying EBP principles and measuring outcomes more effectively. This is an exciting time, because tools and resources are becoming more available for EBP. All stakeholders are starting to ask questions, and the questions that are being posed are expecting better answers (evidence) from us. The most compelling questions I hear are posed by individuals with complex communication needs and/or family members. Clinical decisions have to be made today, they're made by the practitioners and clients, and cannot be deferred until a stronger research-base exists. The principles of EBP can still be applied, successful outcomes achieved, and quality of life improved for those we hold paramount. I encourage everyone to attend the Pittsburgh Employment Conference for Augmented Communicators (PEC) in Pittsburgh every other August to see the AT/AAC outcomes that are possible - up close and personal, and not just in print.

I look forward to meeting some of you as I present on performance and outcome measurement tools and other resources for EBP at conferences across the country. Please, introduce yourself as a FCTD online discussion participant. I hope many of you will consider attending the AAC Institute Symposium on Evidence-Practice Practice: Research, Practice, and Policy on June 19th in Orlando, Florida as part of the annual RESNA Conference. Find out more by visiting www.aacinstitute.org.

Best wishes for a happy and successful New Year!

Katya

This has been a great discussion, led by two committed and extremely knowledgeable experts. The discussion will remain available for review in our archives. You can access the archives in the same way you accessed the discussion. Twice a year the Family Center produces an Assistive Technology Resources CD-ROM, and both a transcript of the discussion and a summary of it will be included on our Spring 2004 disc. You can request as many free copies of this resource as you'd like.

On January 12, 2004 we'll begin our discussion of Website Accessibility, led by two of the leaders in the field - Earl Johnson, Director of Web Accessibility at Sun Microsystems, and Cyndi Rowland, Director of the WebAIM project at Utah State University. Please join us for that discussion and please pass the information along to your colleagues. Again, thanks to all.