The Family Center on Technology and Disability

Welcome to the Family Center’s first national online discussion. Each of our discussions will feature one or more experts who have agreed to commit a certain amount of time each week, to share their thoughts and experiences, and to respond to participant questions and comments. Our first discussion addresses the important and timely topic of assistive technology outcomes and is led by a leading expert on the subject, Dr. Roger O. Smith. Dr. Smith is Director of the Center for Rehabilitation Sciences and Technology at the University of Wisconsin-Milwaukee and is on the faculty of the Occupational Therapy program. Dr. Smith has been researching measurement related to disabilities for more than twenty years and has recently focused on issues surrounding the measurement of assistive technology outcomes. This discussion will be enhanced, as will all of our online conferences, by a library of online resources.

Those who have participated in this type of conferencing know that the participants themselves are an invaluable source of information. We hope that you will join the FCTD online conferences on a regular basis. Whether you’re comfortable sharing your views or merely wish to listen in, you are very welcome. Please let us know how you feel about the discussions and please feel free to suggest topics and experts for future conferences. All FCTD online discussions are free, having been made possible by the Department of Education’s Office of Special Education Program (OSEP)

How can we ensure parents have easy access to evaluation and assesments on assistive technology? Is the information written to be understandable for them.

Janet Peters, PACER Center

• Re:Parental Involvement posted by Kerry on Aug 30, 2002

  Thank you Janet, for raising an excellent point about accessibility to the evaluation info. I'd like to second that issue for teachers, also. For many teachers in the classroom, terminology and complicated data can be an obstacle to employing new technology or being able to put it to work as soon as possible for a student.

• Re:Parental Involvement posted by Roger O. Smith on Aug 30, 2002

  Janet does bring up an important question. Many assistive technology evaluations can be very technical and it is important for the technical team members to report information in ways that are understandable to each team member including the parents. This is not easy, as assessments and parents are so highly unique. But it is a challenge that must be met!

  • Re:Parental Involvement posted by Jonathan Cruce on Sep 10, 2002

    I'm coming in a little late here, but I do have a couple of cents to add to this. While it's very important for technical team members to attempt to make their explanations understandable, it's just as important for the other side (e.g. teachers and parents) to be sure they ask questions when they *don't* understand. Increased awareness of what outcomes we're trying to measure will help parents speak the same language we are speaking, but in order to increase their awareness, we have to define our terms very clearly up front, and make it known to all parties involved that we are not offended by questions. I like to start evaluations with a disclaimer that "I do tend to go off into my own world of terminology, so if I say something that you don't understand, stop me right then so we can be clear on what I'm talking about. And I'll do the same if you use abbreviations and terms that I don't understand."
    
    Still figuring out all of that Occupational Therapist shorthand...
    

    • Re:Parental Involvement posted by Discussion Board Guest on Sep 10, 2002

      Great point! I printed your language, which I thought was particularly apt, and plan to include it in the workshops I conduct (with attribution of course). Thanks.

• Re:Parental Involvement posted by Nathan Lemmon on Sep 16, 2002

  
  I don't think that the interpretation of assessments is done at a layman's level. Depending on who is explaining the results of assessments, you may find a variety of ways to say the same thing. As a parent, I'm concerned with providing the best, most up-to-date technologically advanced assistance I can. I want results I can see. Don't try to give my a "snow job" w/ technical jargon and expect to hold my attention or gain my respect. If something isn't working out that well, I'd rather hear about it in a straightforward manner.
  
  I don't believe in making parents feel better by spending money on something that doesn't work very well. Be honest about the prospect of progress and never prop up expectations beyond what is realistic.
  
  School systems shouln't feel afraid to disagree when someone proposes a magic bullet to solve problems.
  
  Assistive technololgy should be used and assessments made to measure gains in the same areas as other students. Quality of life is too subjective an area to measure and even if objective measures were devised for quality of life, I see mounds of paperwork and beuraucracy coming to support new assessments and new requirements.
  
  I would agree to support new assessments meant (in part) to continue providing funding for technologies that obviously work, but by definition, if the the positive effect is obvious, you don't need a fancy test to measure it, or fancy words to expain it to me.

  • Re:Parental Involvement posted by Jackie Hess on Sep 17, 2002

    Mr. Lemmon,
    Thank you for your thoughtful comments. This discussion, like most national online discussions, is being monitored by hundreds of individuals who read the posts (we know from Webalyzer statistics) but who don't feel comfortable leaving a message themselves. So your remarks will be read and hopefully taken to heart by many people.

  • Re:Parental Involvement posted by Joan on Sep 17, 2002

    I have been on every side of the table when it comes to evaluation reports-I have delivered them, and had results about my own child delivered to me. There are times that as a parent, I have had to say "Stop- let's go over that again" and as the evaluator, I have been stopped several times and asked to make more sense, or to be more clear. Teachers and other professionals do not try to snow job parents with jargon- but may use the language that is part of their job- and most try to be clear. But, technical terms are used, and can be hard to understand. All members of the evaluation process need to work on communication skills, both delivering and receiving the message. Yes- it is MUCH harder to be the parent- it is a position of less power and sometimes decreased input. But, I also try as a state consultant in AT to make sure parent training and input into the process is integral. I am able to bring what I learned as the parent in the process to helping making the whole process more collegial.
    I am an advocate for the use of data to prove effectiveness- results can be in the eye of the beholder- both good and bad results, so I encourage the gathering of empirical data, while telling teams to be ready to recognize and take advantage of the "unexpected gifts" which can also occur outside of the measured outcomes- and make sure that these are counted into the determination of value of use of the product or device.

  • Re:Parental Involvement posted by Roger O. Smith on Sep 21, 2002

    Maybe communicating the results of evauations and outcomes should be considered as important the results of a test. Unfortunately, assistive technology professionals spend a lot of time learning how to help identify the proper assistive technology devices and performing the best tests, but less emphasis is made on test interpretation and communicating the results.
    
    Perhaps assistive technology professional training programs should spend more time on what do we do with the information after we collect it.

Who is the most appropriate person or persons to do the evaluation/assessment? What perspectives are gathered to include in the evaluation for that student?

• Re:Evaluator posted by Kerry on Aug 30, 2002

  I think that this is a very valuable point - who should be included in the evaluation process? Obviously the technically trained personnel, but what about the parents and families and other people who are more closely involved with the person and the technology being used. How are these perpectives being incorporated, or how should they be?

• Re:Evaluator posted by Roger O. Smith on Aug 30, 2002

  Assistive technology assessments are ideally custom delivered for each individual. This means the type and background of the person(s) doing the assessment may range from a speech and language specialist, to a wheelchair technician, to a computer access specialist or combinations. Complex challenges often require more team members with more experience and specialized skills. Simple challenges may be perfomed by one therapist or a teacher with a minimum of background in assistive technology. I am a strong advocate of team-based assistive technooogy services that include a wide spectrum of experts with a breadth of professional and depth of AT background.

• Re:Evaluator posted by Roger O. Smith on Aug 30, 2002

  I missed the second question. I believe that 4 perspectives are essential. A satisfactory evaluation needs the perspective of 1) the prospective AT user, 2) experts in the potential assistive technology solutions, 3) experts in the disorder or cause of the problem/challenge, and 4) people knowledgable of the context where the AT will be applied (e.g. parents, classroom teacher.) This does not mean 4 people need to be involved as one person may be able to offer more than one perspective. For example, a knowledgable client/student may be able to provide perspectives 1,3,&4, but often this does mean the team is helpful.

Hi Roger and Kerry:

One of the largest issues I face is convincing parents that the most recent/expensive/complicated/impressive technology is not necessarily the *best* technology. This is particularly true around speech input, which people always think would be the way to go for their THIRteen year old SON whose voice is changING every five SEConds and who isn't seeing enough immediate reward from using the technology to continue practicing with it. I'm forever steering kids instead to typing tutors, Co:Writer, Inspiration, etc. and seeing positive subjective and objective results fairly quickly, but sometimes the parents still aren't convinced.

What success have people had in promoting the concept of appropriate technology?

Jane Vincent, CforAT
(slowly accepting down escalators as appropriate technology)

• Re:Appropriate Technology posted by Roger O. Smith on Sep 02, 2002

  Jane brings up a key concept. Often we become enamored by the latest and flashiest technology. High tech serves as the "silver bullet" for many, but the appropriate technology may be the tin or ceramic version. Learning new skills with older or more typical technology may be more powerful and appropriate in the long run than the latest voice or handwriting recognition software. Someday with more outcomes data we should be able to look at what works best and under what circumstances when. Until then, unfortunately, we are dependent on the experts in our midst (or finding the experts just outside our midst). One way I have found helpful in promoting appropriate technology is through use of data e.g. which system results in fastest words/minute performance for a person, or which system facilitates the most interactions or most diverse vocabulary in use. These types of data can be recorded and compared between systems being considered by a prospective user. Often data speak louder than an opinion or aesthetic preference. I believe that part of the expert evaluator role is to help identify which data to record to discover best performance.

  • Re:Appropriate Technology posted by Jane Vincent on Sep 03, 2002

    I agree, Roger, but I'm also hoping that somewhere in the outcomes data is a feel for the subjective elements--how "cool" or "normal" the student thinks the technology is. I remember one 11-year-old in particular who objectively had better cursor control with a trackball than with a joystick, but who made no bones about letting us know that he'd actually use a joystick and would never touch a trackball again. We are constantly reminded at the Center that kids inevitably put their "kid-ness" first and their disability second...

    • Re:Appropriate Technology posted by Roger O. Smith on Sep 04, 2002

      Not just kids. Personal preference is unquestionably the mosty important factor for successful AT device use by anyone. However, some potential AT users appreciate objective performance and it affects their preferences. Therefore, offering the full set of information to a potential user is important. A non-AT example would be how we buy cars. While auto color, shine, ride, and cost may be sufficeint information to decide and enjoy a car for some, others may also want to have acces to information like miles per gallon, repair history, odometer mileage, etc.

What is a the breaking edge in terms of AT for preschool and elementary students ?

• Re:toys and AT posted by Kerry on Sep 03, 2002

  Dr. Smith, I would like to add to this, what sort of outcomes criteria would indicate that AT is cutting edge or even useful with these students. What are some more specific ways to determine this?
  

  • Re:toys and AT posted by Discussion Board Guest on Sep 03, 2002

    what are some specific ways to increase parents' awareness of AT in this field and increase their involvement in continuing education?

    • Re:toys and AT posted by Roger O. Smith on Sep 04, 2002

      Great question! How do we help increase parent interest? I'll toss in my idea, but this would be a great one for other discussion board readers to respond to. I have seen parent groups work very effectively. Parents listen to other parents. Parents are excellent resources for other parents. A challenge is, what methods are best for bringing them together?

      • Re:toys and AT posted by Discussion Board Guest on Sep 06, 2002

        One way to successfully gather parents is to host an Assistive Technology Lab night (or during or after-school--whenever convenient). Parents and AT specialists can mingle, experiment, and learn about the latest and greatest in AT, as well as network together.

        • Re:toys and AT posted by Roger O. Smith on Sep 11, 2002

          Like an open house! And with the students! Plus, why not invite the general education teachers too?

  • Re:toys and AT posted by Roger O. Smith on Sep 04, 2002

    There are many types of outcomes. Let me turn the question around. What outcomes does a user or potential user of AT devices and services look for?

    • Re:toys and AT posted by Kerry on Sep 05, 2002

      As a teacher, I look for outcomes that resemble qualities of the AT; such as, is it easy to use for both me and my student (most people will not use something that is too complicated for very little return).
      And does it do what we were needing or expecting it to do. If it is a computer screen viewer that enlarges the text and graphics, does it enlarge it enough for the particular student and can I adjust it within a wide range depending on the website being viewed? So, in other words, the outcome I look for is does it work for the particular situation or person.

  • Re:toys and AT posted by Nathan Lemmon on Sep 16, 2002

    The problem being that you can't set up a control group etc. and run a little experiment with kids. If the technology is meant to increase a specific skill, you still have to measure its efficacy on a case by case basis. Example: If John's reading scores go up dramatically, can we be sure why this happened?
    
    With new technologies(and the newer the better, we tend to assume) the more difficult it is to say if the technology itself is having an impact. Add the technology, hope to see a dramatic change in skills or abilities. It would be great if every new technology provided a before I couldn't, now I can effect. Beyond that, it takes time and many trials before setting the standard for what to expect given the addition of advanced educational technologies. Money flows to the Universities in the form of research grants and everyone waits for approval, following peer review. This process takes time. Maybe we need to move to a cost/benefit analysis during the development cycle of educational technologies?

    • Re:toys and AT posted by Roger O. Smith on Sep 21, 2002

      Many researchers believe that use of control and experimental groups are the best way to understand the cause and effect of an intervention. However, many researchers also acknowledge that to really understand the effect of a device or service, we needs to examine outcomes in the natural environment and under natural circumstances. The need, then, is to better identify ways to determine outcomes on a day to day basis, without artificially injecting or removing an intervention.
      
      "before I couldn't" and "now I can" is exactly what we all hope to see! Wouldn't it be a wonderful if we could determine the outcome of an assistve technology device this way!
      
      We must be sure, though, that we understand what improved the performance. Was it the device, the service, the family, natural development of skills, other aspects of the interventions or combinations?

• Re:toys and AT posted by Roger O. Smith on Sep 04, 2002

  One of the places I have found some of the newest ideas and devices is at AT related conferences like Closing the Gap (in the midwest.) New product manufacturers often unveil their newest devices at conferences, or at a minimum they are very happy to tell you about them in those settings. Also, many conferences have a free or open day in their exhibit halls for the public.

• Re:toys and AT posted by Joan on Sep 16, 2002

  I think that it is not so much what is cutting edge as much as re-thinking how we look at technology for younger kids. In the past, I was hesistant to suggest higher end technology for little ones, for fear I was using too big a tool- however, now I am more likely to think about broader goals and look at what the more sophisticated technology migh allow the child to do. So- where I may have provided a 4 message device for a little on, I would now consider trying a dynamic display device which can grow with the child over time. BUT- and this is a big caveat- what is biggest and best is not always best. The smaller, less powerful device could be perfectly wonderful. Also, different strategies might work better in different settings. That is, there may be more than one right answer. Cutting edge is fun- but not always the one right answer.
  Joan

  • Re:toys and AT posted by Roger O. Smith on Sep 21, 2002

    Great point! The biggest tool is not always (maybe only rarely) the best tool. In the woodshop, the power table saw is a wonderful tool, but would be devastating for use with a handsize wood carving. The key in the woodshop is for the woodsmith to really understand the range of tools available, and rather intimately. Likewise, in the assistive technology shop, the ATsmiths need to fully understand the range of assistive technologies devices available and rather intimately. It is very difficult to select the best tool without knowing what relevant tools exist and how each works. Understanding outcomes is part of knowing how each works.

Dr. Smith, Have you seen a pattern of increased resistance to AT outcomes testing on the part of educators who have been pressed in recent years by a greater emphasis on performance-based student evaluations?

• Re:Increasing resistance to outcomes testing? posted by Roger O. Smith on Sep 04, 2002

  Generally, everyone like to see positive outcomes of an intervention like AT. Everyone is usually willing to spend a little bit of time to document good outcomes as long as they know what to document. The key words here I'd say are "little bit of time" and "what to document." We are still working to identfy what the most important outcomes data are and how to collcet them most efficiently so it only takes a little but of time.

  • Re:Increasing resistance to outcomes testing? posted by Jackie Hess on Sep 06, 2002

    Having spent some time in the classroom myself, I heartily agree that teachers are likelier to take the time to generate outcomes documentation if the tools are intuitive, efficient time-wise, and seen as important to a child's overall experience (education, interventions, outside therapies, etc.) What has surprised me is how often assessment efforts do not take maximum advantage of computer capabilities. When the AT tool is computer-based, we can easily capture the student's path through the material, the time taken at each step, the pattern of use (is s/he relying primarily on visual content, textual content, or graphics), number of "mis-steps," etc. Yet quite a few assessment tools don't seem to use the computer this way.

    • Re:Increasing resistance to outcomes testing? posted by Roger O. Smith on Sep 11, 2002

      I pulled out and gazed into my crystal ball. (After checking the manual to make sure I was using it correctly,) I saw that over the next few years we will increasingly see computer-based outcomes data collection. Plus, the advent of palmtop computers and wireless connections will allow us to collect outcomes in ways we can only begin to conceptualize today.
      
      Of course, collecting the data and being able to use the data are two distinct challenges. As a wise therapist once told me, put tons of data in, might get tons of data out, but so what.
      
      What data do we need and hoe do we use it?
      

We have posted an updated introduction that includes a few questions. The idea was to generate some discussion and maybe some controversy to expand our horizons. If you haven't seen the revised intro yet, we are curious what you think.

If it is not sufficiently controversial, I can recite some comments and opinions I have heard that should stimluate some bantering.

• Cost Effective as an Outcome posted by Discussion Board Guest on Sep 06, 2002

  I read the updated introduction and I have some specific questions about using cost effectiveness as an outcome with AT. Can something be considered cost effective and justify using it if it is really expensive and benefits only one person or student?

  • Re:Cost Effective as an Outcome posted by Kat on Sep 06, 2002

    As long as it helps a child or student then it should not matter how much the equpiment costs. There are so many children in our society that suffer from various disabilities. I think that if there is something that can help them communicate with their parents and friends, learn better in school, or just enhance their daily lives then that should be enough to justify the cost of the equipment. Even if it only helps one child it is a start to helping them all.

    • Re:Cost Effective as an Outcome posted by Jonathan Cruce on Sep 10, 2002

      I agree with you that cost *shouldn't* be an issue. But "shouldn't" and "isn't" are two very different words. Funding is, and will likely always be, scarce for A.T. There will always be more people whose quality of life can be enhanced by AT than there will be funding for AT. In my opinion, it's only a matter of time before outcomes measurement for AT becomes mandatory for that very reason. Rehabilitation Engineering services are not currently covered by most insurance companies. Part of the reason for their hesitation is that there is no standardized method to measure the effectiveness of the recommended equipment. The other side of that coin, however, is that the goal of many assistive technology devices is to enhance the *quality of life* of the user. It's easy to get funding for a ventilator, because the outcome is not subjective. The client is still alive; tangible, concrete outcome. Quality of life(QoL) is this big, amorphous blob that seems to slip away from us the moment we think we have it licked. I can prescribe a GameBoy(tm), and show you outcomes measures that say it is well-used, it makes the client happier, it is appropriate for the client's stage of development, and even that the "ingos" were not a factor in the increased quality of life of the client. Does that mean that the GameBoy(tm) should be funded? How can we ensure that outcomes measurements will involve the desired aspects of QoL without allowing too much 'working' of the system? (some 'working' will happen, regardless...)

      • Re:Cost Effective as an Outcome posted by Joan on Sep 11, 2002

        It seems to me to be part of the growth process in AT that outcome measures should be required. As a former AT provider, I made suggestions based on instinct- usually good suggestions, sometimes not. I would dearly love to get a "do over" on some of my early practice- I always sought conusmer input, but failed to conduct good trials, collecting data. Sometimes the seemingly "best" solution ended up being wrong- data collection over time and environments would have helped to insure appropriate choices, appropriate buy in from others. By choosing something because it seemed right, we sometimes made poor use of tax payer dollars, provided administrators with reason to distrust AT solutions, frustrated the student or family or teachers. In my current position of helping support best practice in education, I encourage and support data collection over time, in multiple environments, including full team input. To do otherwise just doesn't cut it-

      • Re:Cost Effective as an Outcome posted by Roger O. Smith on Sep 11, 2002

        I think that how we determine and state goals are critical to provide the context of the outcomes. Some funding sources support some goals more than others. Even a Gameboy could serve as the medium to achieve certain cognitive, motor or educational goals and could be fundable under some circumstances.

        • Re:Cost Effective as an Outcome posted by Discussion Board Guest on Sep 16, 2002

          How do we learn the lingo to acquire the funding for AT? In other words, what can we do as consumers to educate ourselves to "work" the system (to some extent)?

          • Re:Cost Effective as an Outcome posted by Kerry on Sep 16, 2002

            I'll be interested in seeing Dr. Smith's direct answer to this very timely question, but I think the root of learning to "work" the system is a subtext in previous postings. I think that consumers need to be involved in determining HOW the outcomes are going to be measured and WHAT we are measuring for (why is Juan using the gameboy, what cognitive gains are being sought) and then being aware of and following the data. Using data is the most powerful "tool" we have to argue for getting better AT and/or services.
            Using information sites like the FCTD webpage or NICHEY (National Information Center for Children and Youth with Disabilities) or any of our extensive partnership list can help with getting familiar with what terms mean and how main funders (like state departments of education) use them.

          • Re:Cost Effective as an Outcome posted by Roger O. Smith on Sep 21, 2002

            Funding lingo is very colloquial. Not only does it vary among vocational, medical, independent living and educational settings, even within one of these settings there are large differences. Some states have more dollars available than others. And even school district to school district the resources available to assistive technology teams and students is highly variable.
            
            Asking plentiful questions of the assistive technology team members can often help clarify what the correct lingo might be in that setting. No question can be considered a bad question when asked in a serious and concerned fashion. Most assistive technooogy teams should be able to respond in a helpful way to questions about funding.
            

In your excellent introduction, you refer to the variety of assessment types and tools being akin to looking out of different windows in a house. Which prods me to ask, what are your suggestions to those responsible for AT assessments when the experiences reported by the child's teacher differ significantly from those reported by the child's parents? So often children with disabilities display very different behaviors at home and at school.

• Re:Home versus school posted by Roger O. Smith on Sep 11, 2002

  Ah, it is so important to have parents and school team members to interact! Without this discussion plans may be set with only a partial picture.

Your description of "ingos" calls to mind the Chilean experience with computer-assisted learning. The Enlaces porject provided hundreds of computers to children in rural village schools. The results were great - significant increase in learning. However, the increases were comparable to those sustained by two of the control groups (which did not have access to computers)- the one which offered free breakfast and lunch programs and the one which provided low-tech supplies (crayons, pencils, notebooks, etc.) which had not been available previously. The Chilean administrators responsible for the program chose not to give up on the computers but to conduct additional assessments, more finely tuned, to gather more data on the specific learning gains.

• Re:Ingos posted by Roger O. Smith on Sep 11, 2002

  Interesting study! It sounds as if in this case intervention of various types was beneficial. Plus, the outcome assessments were apparently too general if more finely tuned data were needed to really see what was happening.

• Re:Ingos posted by Jackie Hess on Sep 13, 2002

  This raises a point concerning "one size fits all" evaluations. Dr. Smith, what's your experience with respect to the need for different types of evaluations for high-end (wealthier) schools and interventions versus economically at-risk schools and populations? Unfortunately in this country too there are students and schools where offering breakfast/lunch and more supplies makes (or would make) a big difference. How do we distinguish the results of AT utilizzation from non-AT support?

  • Re:Ingos posted by Discussion Board Guest on Sep 16, 2002

    To somewhat piggyback on this question: What assessments are on the market that are currently being used, and what assessments do you personally recommend? Are they the same ones or different, and if different, why do you prefer those over the ones currently used?

  • Re:Ingos posted by Kerry on Sep 16, 2002

    Which also raises the "pathos" question - that is, do we apply different standards to AT as professionals because we want to help and sometimes feel helpless. So how do feelings impact objective evaluation data? Do we sometimes feel that doing "something" or using something is better than doing nothing?

    • Re:Ingos posted by Nathan Lemmon on Sep 16, 2002

      It is very easy (and satisfying) to come across as the sage when making suggestions for the use of technology/magic. Hope springs eternal. Especially if we believe, or if we have much invested in our beliefs. I think we can and do, interpret numbers emotionally.
      
      Emotions are involved with both how we relate to the promise of technology and the hopes for individual students.
      
      What I don't think we do enough of is keep track of what our decisions were yesterday and where we were emotionally when we made them. What were the outcomes of those decisions? Do we dare take a hard look at what has happened since a decision was made? Are we ready to admit there was an emotional component to that decision?

      • Re:Ingos posted by Kerry on Sep 20, 2002

        Good question - it is hard to separate how we feel from waht is real. But, maybe emotions should be a piece of the decision making process?

  • Re:Ingos posted by Roger O. Smith on Sep 21, 2002

    The availability of assistive technology devices and services should not be dependent on the economic viability of a school district. Of course, neither should there be a difference in the availability of books music, sports or computers, but there are. This is a huge policy issue as students across the nation do not have equal access to assistive technology.
    
    Research to investigate the question you raise is very needed and likely needs to have several foci including:
    a) what is the nature of differences of AT utilization across districts of various resource levels?
    b) do the utilization differences result in outcomes differences?
    c) do personnel providing assistive technology services have skill levels and different levels of training?
    d) what are strategies for equalizing the potentials of AT across districts?

    • Re:Ingos posted by Kerry on Sep 30, 2002

      I am particularly interested in the personnel training. Even the best technology is useless unless someone knows how to use it, yet we tend to measure outcomes based exclusively on the person be served by the technology when we should also be looking at those who are using AT to serve someone else.
      I absolutely believe that utilization and training differences make a huge difference in outcomes.

      • Re:Ingos posted by Roger O. Smith on Oct 03, 2002

        Ah, outcomes of the device and the person using the device is important, but yes, a more difficult problem looms in outcomes measurement. How do we identify and measure the outcomes of a service? For one, we must know who is providing the service including the backgrounds and training of personnel. Of course, we also need to know how they work as a team, who all are team members, how well is the AT user and parents/family included, etc.
        
        Many more focused research studies need to be funded to examine the effects of varable provider skills and background and team process on the outcomes of services.

Our server was down from Friday night until this morning - sorry for the interruption. The discussion is really getting interesting so welcome back or welcome for the first time.

Kerry Venegas
Moderator

• Re:Apologies for the Server being down! posted by Jackie Hess on Sep 26, 2002

  Let me add my apologies. The host of our site is located in another state and they have had a series of router and server problems. They have been working hard to resolve the issues and it's our hope that things will stabilize at their end. Thank you for staying with us.

The thoughtful and interesting comments by Joan made me think about some of my AT-related experiences in the classroom which produced unintended results. I'm thinking specifically of accommodations that were integrated into classroom activities for the benefit of one specific child but which produced remarkable outcomes in one or more other children. In most cases the other children had some learning, behavior or neurological issue. However, for a variety of reasons the specific AT intervention was not aimed at them and we were not measuring results as we were for the initial child. I felt then as I do now that there should have been a non-anecdotal way of capturing the success of the intervention with a larger group. The issue was generally complicated by the fact that the parents of the other children were often in denial about their child's issues and were sometimes offended when it was suggested. Any thoughts?

• Re:Unintended outcomes posted by Joan on Oct 01, 2002

  I have no solutions to Jackie's situation, but a thought- we need to provide a more universal design to learning, so that children have more than one way to show they are successful. What is AT to me could merely be a tool for another student. Make that tool- and other tools available to all the students, and let them learn without labels when we have to. I feel for the teachers who are trying to get the supports for students- and knowing that a tool or strategy will help- and the documentation of the child's needs will help insure it gets provided. We have to be sympathetic to the parents, but it is tough when we are trying to do our best for the student and aren't able to do all we want to or can.

  • Re:Unintended outcomes posted by Roger O. Smith on Oct 05, 2002

    Universal design in education can be thought of as the inclusive extreme of assistive technology. When we think assistive technology, we usually think a device or system used by one person. However, if an assistive technolgy intervention becomes useful for eveyone it becomes transparent as an assistive technology system although technical is still may be one.
    
    A classroom example might be when a teacher provides electronic hand-outs to the class. Not only does this help the student who is blind and is not able to see the overheads and powerpoint slides during the presentation, but it helps the student with a learning disability who needs to review via audio, and it gives all students the opportunity to have the same information in a form that they can print, enlarge, have it spoken, or whatever.
    
    What we do not know how to do very well yet is how do you measure the outcomes of a group of individuals that may be benefitting form such an intervention. But how about this for an idea in this example:
    
    Test the success (or lack of) of the class using the usual presentation style without the UD stratgey of e-hand-outs. Provide the e-hand-outs and document when they were provided. Test the success of the class using the same testing methods after the intervention. Analysis of the data could go several ways. One could examine the success of teaching by looking at the entire class before and after. One could compare the improvements of those noted to have disabilites with those who are not know to have disabilities. Or one could examine one student as might be needed in an IEP.

I am a senior at Bowling Green State University studing Advanced Technological Education. I am currently researching training opportunities for individuals with physical limitations. My quetion is "Are you aware of opportunites made available by employers to employees with physical challenges for training?" Any information you may be able to pass along would be appreciated.

• Re:Employer training opportunities posted by Kerry on Sep 30, 2002

  This has a lot to do with ADA issues, that is, the Americans with Disabilities Act. I'm assuming that you are talking about training either for a job or on the job for a necessary skill (such as operating a new printing press).If an employee has a physical limitation which makes traditional training methods difficult, the employer is responsible for providing appropriate accomendations for the employee's training. Do a search under ADA-employers and you will find information on what accomendations employers need to provide and more that may be linked to your question.

• Re:Employer training opportunities posted by Roger O. Smith on Oct 05, 2002

  As Kerry suggests the web has excellent resources for ADA employment related information. Resources that you will encounter if you have not already are the nationally funded DBTAC's.
  These centers are charged to assist with information about the ADA, especially in emplyoment and in education.

We've talked about some funding issues, the need for outcomes measurement, and a little about what to measure, i.e. whether cost effectiveness is a fundamental issue in outcomes. But I haven't seen much about what, exactly, we *should* be measuring. My opinion:

We need to measure both the efficiency of the intervention and the effectiveness of it. Efficiency should be relatively easy to measure when the intervention is a device. Computer access can be measured in words per minute, minimum accessible icon size, time to travel from one icon to another, all relative to the same values prior to the intervention. Augmentative communication devices are rather platform-dependent, but many of the high-tech devices on the market now can log conversation activity, requiring some interpretation by a professional as far as how efficient the intervention is, but the data is there nonetheless. Effectiveness, again in the case of a device, can be measured by how often it is used and whether the device changes the individual's quality of life (QoL). QoL is a difficult thing to measure, but some of the reference material for this discussion group includes some rather heroic attempts, I think, to measure quality of life dependent on the individual's self-image.

I think that if we are going to measure outcomes, we must be certain that we include qualitative measures such as QoL as well as statistical measures like efficiency. If AT funding sources decide to purchase a device based solely on efficiency, we'll see more instances of abandonment, and less ownership in our interventions by the entire AT Team.

Discussion? Thoughts? Disagreement?

• Re:What to measure? posted by Kerry on Oct 03, 2002

  Without a doubt, we need to include QoL and balance effectiveness with efficiency. But even if we collect data, so much of the data we collect on this is persepctive dependent. Maybe the question is - WHO gets to evaluate and have the ultimate decision on the AT? Or if we take in more than one person's or one group's perspective, how do we weight it?

  • Re:What to measure? posted by Jonathan Cruce on Oct 04, 2002

    If we have a well designed outcomes measurement system, the data will be the same regardless of who takes the measurement, i.e. the measurement system will be repeatable. Efficiency data is fairly easy to make objective. As for QoL, I think those measurements should be based on the individual's self-image, and that the goal sould not be some arbitrary QoL level set by an outside group, but a positive difference between before and after measurements.
    
    That may leave the question of who gets to decide how much change is worthy of investment, but I think the answer to that question will always be the funding source. After all, they're the ones paying. However, perhaps a database of outcomes data will help provide a litmus test for the funding source, as well as evidence that this stuff really works.
    
    We will always need some funding sources that have true good will, who are willing to take a chance and make an investment regardless of the 'average' outcome. But if we can get major funders, i.e. insurance companies and government funding, to take care of 90-95214f the cases (because we'll have data to prove effectiveness), that will reduce the load on the funders that really care, and they can concentrate on funding the cases that are a bit more of a gamble.

    • Re:What to measure? posted by Roger O. Smith on Oct 05, 2002

      Jon has hit it right on the head. What he listed as efficiency examples are fairly empirical. They are easy to quantify and perspective is not highly variable. And quality of life can only be measured from the subjective perspective of the individual.
      
      And these are two VERY important outcome measurement domains.
      What is interesting is that Jon implies that these may be the most important. Interesting idea as they may be the easiest domains to measure that have the most obvious interpretations. There is a VERY powerful argument with these two domains and to focus on just these two.
      
      However, this still means we'd need to determine what comprises QoL and what specific efficiency measures are needed for any given individual, assuming that they would need to address thje specific areas of performance targeted by an individual that might vary form person to person. And who then selects what those performence thresholds should be, not only how much improvement is necessary before it becomes worthwhile?

      • Re:What to measure? posted by Jonathan Cruce on Oct 09, 2002

        There is another issue here as well. Measuring outcomes data is all good and well, but it comes, as the name implies, *after* the intervention. We can't very well go to a funding source with outcomes data before they decide to purchase equipment for the client, unless we are in the unique situation to loan equipment and/or services for extended periods while measuring the effects of the equipment/services. So this data cannot be used on a case-by-case basis, but must be averaged and analyzed. Of course, it can be used to measure the effectiveness of our intervention while a particular case is still open, but it cannot be used to convince the funder to purchase equipment from the start.
        
        What do you think? Is outcomes data more important in the case-by-case perspective, where it can be used to correct not-so-effective interventions, ex post facto? Or is it more important in the larger framework, where an organization that is measuring outcomes can learn from its 'mistakes' and keeps a longer memory of intervention effectiveness?

What to do with it? Now there's the interesting question. Besides the obvious benefits of tooting our own horns when we provide an effective intervention, how would we use outcomes measures? Of course we would look at interventions where outcomes are less than satisfactory and see if we can improve on the intervention, either for that particular client or the next similar client. We can certainly use outcomes measures when talking to third party funding sources, who always want to know that they're getting their money's worth.

Something else that I think would be good is a centralized database of outcomes information, categorized by disability type. There's always a danger with this type of compilation that people will look at the database, see that X device has been, on average, most effective for Y disability, and prescribe or purchase that device with little or no clinical data to indicate the device as an appropriate intervention in the case of that particular client. However, I would propose using the information more as a guideline to see where we can expect to go with a given intervention. If I know that, on average, clients with Y disability and X intervention have reached Z functional level using their intervention technique, I know to keep trying if my client hasn't yet reached that level. I also know not to get too discouraged if I don't reach my personal functional goal for the client if it's higher than the stats show.

Again - Questions? Discussion? Disagreement? :)

• Re:What to do with what we measure? posted by Kerry on Oct 03, 2002

  Yes, yes and yes -the info should be accessible for everyone, but where should it be stored? On Dept of Ed websites, advocacy groups, universities a combination? Or should a national database be funded and maintained specifically for this purpose?

  • Re:What to do with what we measure? posted by Jonathan Cruce on Oct 04, 2002

    I would take storage space for a database wherever I could find it. I personally think the burden of proof is on rehabilitation professionals as far as effectiveness of their interventions. Dept of Ed may be a good start, but I think this information will be useful to all ranges of the population, not just the education sector. I work with a lot of adults in vocational rehabilitation, and the outcomes measurements are just as necessary with them as with students.

• Re:What to do with what we measure? posted by Roger O. Smith on Oct 06, 2002

  Great ideas! I dream about that universal interface on the web that could be a master outcomes look-up system. Jon, you seem to have a mental model of what such a system might look like and how it might work. Maybe we have watched too much Star Trek, but the technology today does seem as if it might soon support a database where a person with a disability, a service provider, a funder, or a parent might go to the database to see what has worked (and not) for others in similar situations.
  
  Of course such a system will not be easy to create and will take substantial development attention and resources. Also, the magnitude of this development activity will likely require some political support as the database must be large in catchment area (national at least?) and have some incentives for AT users and service providers to upload data into such a database or we'd have a database with no data.

  • Re:What to do with what we measure? posted by Jonathan Cruce on Oct 09, 2002

    The database structure would be fairly easy to create, once we decided how we were going to measure our outcomes data. And once the structure was complete, it would be fairly simple to implement web-based data collection tools for use by registered persons (not that *I* could do it mind you, but I'm guessing the time for an experienced web designer to create such an interface would be minimal). As for area covered, I would like to see it as a national project eventually, but federal govt would probably want to see some examples of its use on a trial level (i.e. university project) first. I do agree that political support is desirable. The use of outcomes data is already required by CARF (Center for Accreditation of Rehabilitation Facilities, www.carf.org), though the exact vehicle for measuring outcomes is not specified. Getting a data upload attached to a requirement like that, one that is already in place and used on a widespread basis, could be one answer. If state funding agencies say, "we're not going to purchase equipment unless you can show me you're measuring these outcomes," that would be a fairly strong incentive to upload data. And I think private funding sources would soon follow.
    
    But, like CARF, we run the risk of our measurement system being seen as just one more layer of red tape that has to be broken before equipment is purchased. How can we keep that from happening? Do we *want* to keep that from happening? In my experience, it seems that people complain at first about these kinds of changes, but after a while it simply becomes the standard operating procedure. As it should be, in this case...