1st Edition

Software Test Attacks to Break Mobile and Embedded Devices

By Jon Duncan Hagar Copyright 2014
    378 Pages 80 B/W Illustrations
    by Chapman & Hall

    377 Pages
    by Chapman & Hall

    Address Errors before Users Find Them
    Using a mix-and-match approach, Software Test Attacks to Break Mobile and Embedded Devices presents an attack basis for testing mobile and embedded systems. Designed for testers working in the ever-expanding world of "smart" devices driven by software, the book focuses on attack-based testing that can be used by individuals and teams. The numerous test attacks show you when a software product does not work (i.e., has bugs) and provide you with information about the software product under test.

    The book guides you step by step starting with the basics. It explains patterns and techniques ranging from simple mind mapping to sophisticated test labs. For traditional testers moving into the mobile and embedded area, the book bridges the gap between IT and mobile/embedded system testing. It illustrates how to apply both traditional and new approaches. For those working with mobile/embedded systems without an extensive background in testing, the book brings together testing ideas, techniques, and solutions that are immediately applicable to testing smart and mobile devices.

    Setting the Mobile and Embedded Framework
    Objectives of Testing Mobile and Embedded Software Systems
    What Is Embedded Software?
    What Are "Smart" Handheld and Mobile Systems?
    Why Mobile and Embedded Attacks?
    Framework for Attacks
    Beginning Your Test Strategy
    Attacks on Mobile and Embedded Software
    If You Are New to Testing
    An Enlightened Tester Makes a Better Tester

    Developer Attacks: Taking the Code Head On
    Attack 1: Static Code Analysis
    Attack 2: Finding White-Box Data Computation Bugs
    Attack 3: White-Box Structural Logic Flow Coverage
    Test Coverage Concepts for White-Box Structural Testing
    Not e of Concern in Mobile and Embedded Environments

    Control System Attacks
    Attack 4: Finding Hardware System Unhandled Uses in Software
    Attack 5: Hardware-to-Software and Software-to-Hardware Signal Interface Bugs
    Attack 6: Long-Duration Control Attack Runs
    Attack 7: Breaking Software Logic and/or Control Laws
    Attack 8: Forcing the Unusual Bug Cases

    Hardware Software Attacks
    Attack 9: Breaking Software with Hardware and System Operations
    Attack 10: Finding Bugs in Hardware-Software Communications
    Attack 11: Breaking Software Error Recovery
    Attack 12: Interface and Integration Testing
    Attack 13: Finding Problems in Software-System Fault Tolerance

    Mobile and Embedded Software Attacks
    Attack 14: Breaking Digital Software Communications
    Attack 15: Finding Bugs in the Data
    Attack 16: Bugs in System-Software Computation
    Attack 17: Using Simulation and Stimulation to Drive Software Attacks

    Time Attacks: "It’s about Time"
    Attack 18: Bugs in Timing Interrupts and Priority Inversions
    State Modeling Example
    Attack 19: Finding Time-Related Bugs
    Attack 20: Time-Related Scenarios, Stories, and Tours
    Attack 21: Performance Testing Introduction
    Supporting Concepts
    Completing and Reporting the Performance Attack
    Wrapping Up

    Human User Interface Attacks: "The Limited (and Unlimited) User Interface"
    How to Get Started—the UI
    Attack 22: Finding Supporting (User) Documentation Problems
    Attack 23: Finding Missing or Wrong Alarms
    Attack 24: Finding Bugs in Help Files

    Smart and/or Mobile Phone Attacks
    General Notes and Attack Concepts Applicable to Most Mobile-Embedded Devices
    Attack 25: Finding Bugs in Apps
    Attack 26: Testing Mobile and Embedded Games
    Attack 27: Attacking App–Cloud Dependencies

    Mobile/Embedded Security
    The Current Situation
    Reusing Security Attacks
    Attack 28: Penetration Attack Test
    Attack 29: Information Theft—Stealing Device Data
    Attack 30: Spoofing Attacks
    Attack 31: Attacking Viruses on the Run in Factories or PLCs

    Generic Attacks
    Attack 32: Using Combinatorial Tests
    Attack 33: Attacking Functional Bugs

    Mobile and Embedded System Labs
    Introduction to Labs
    To Start
    Test Facilities
    Why Should a Tester Care?
    What Problem Does a Test Lab Solve?
    Staged Evolution of a Test Lab
    Simulation Environments
    Prototype and Early Development Labs
    Development Support Test Labs
    Integration Labs
    Pre-Product and Product Release (Full Test Lab)
    Field Labs
    Other Places Labs Can Be Realized
    Developing Labs: A Project inside of a Project
    Planning Labs
    Requirement Considerations for Labs
    Functional Elements for a Developer Support Lab
    Functional Elements for a Software Test Lab
    Test Lab Design Factors
    Lab Implementation
    Lab Certification
    Operations and Maintenance in the Lab
    Lab Lessons Learned
    Automation Concepts for Test Labs
    Tooling to Support Lab Work
    Test Data Set-Up
    Test Execution: For Developer Testing
    Test Execution: General
    Product and Security Analysis Tools
    Tools for the Lab Test Results Recording
    Performance Attack Tooling
    Basic and Generic Test Support Tools
    Automation: Test Oracles for the Lab Using Modeling Tools
    Simulation, Stimulation, and Modeling in the Lab Test Bed
    Continuous Real-Time, Closed-Loop Simulations to Support Lab Test Environments
    Keyword-Driven Test Models and Environments
    Data Collection, Analysis, and Reporting
    Post-Test Data Analysis
    Post-Test Data Reporting
    Wrap Up: N-Version Testing Problems in Labs and Modeling
    Final Thoughts: Independence, Blind Spots, and Test Lab Staffing

    Some Parting Advice
    Are We There Yet?
    Will You Get Started Today?
    Advice for the "Never Ever" Tester
    Bug Database, Taxonomies, and Learning from Your History
    Lessons Learned and Retrospectives
    Implementing Software Attack Planning
    Regression and Retest
    Where Do You Go from Here?

    Appendix A: Mobile and Embedded Error Taxonomy: A Software Error Taxonomy (for Testers)
    Appendix B: Mobile and Embedded Coding Rules
    Appendix C: Quality First: "Defending the Source Code So That Attacks Are Not So Easy"
    Appendix D: Basic Timing Concepts
    Appendix E: Detailed Mapping of Attacks
    Appendix F: UI /GUI and Game Evaluation Checklist
    Appendix G: Risk Analysis, FMEA, and Brainstorming





    Jon Hagar is the principal (CEO/CTO) and senior software test engineer at Grand Software Testing. For over 30 years he has worked on systems and software engineering, specializing in testing/verification and validation. He is the lead editor on ISO/IEC/IEEE29119 Software Test Standard, a member of the IEEE1012 V&V Plan working group, and co-chair on the OMG UML testing profile standard. Jon holds a patent on web test technologies and has published numerous articles on software reliability, testing, test tools, formal methods, and embedded systems. He has a B.S. in mathematics with a specialization in civil engineering and software from Metropolitan State College in Denver, Colorado, and an M.S. in computer science with a specialization in software engineering and testing from Colorado State University.

    "Using the framework of attacks popularized by James Whittaker’s books, Jon Duncan Hagar describes those that are relevant here and extends the approach with new attacks specifically for mobile and embedded systems. He provides detailed information and guidance on how to test more effectively and efficiently in the mobile and embedded world. … Jon shows you what to test and how to test, giving ideas that you can use to do better testing of mobile devices now and save yourself serious trouble later on.
    Jon’s extensive experience (much of it in the embedded world), his thorough research, and his deep knowledge give this book a solid foundation and provide helpful guidance and steps to take in applying testing attacks to mobile and embedded devices."
    —From the Foreword by Dorothy Graham, Software Testing Consultant

    "Every tester who wants to keep current needs to read this book, and you can read with confidence knowing you are being guided by the best in this business. … Learn by doing, with this book as your guide."
    —From the Foreword by Lisa Crispin, Agile Testing Coach and Practitioner