test 3 cs

Continuous Integration(CI)

• automatically build, scan and test the code

• for every commit to a key development branch

Continuous Delivery(CD)

automatically packaging and deploying software after CI

• enabling frequent releases to users

• Projects differ in their delivery to users:

  • installable apps

  • Internet (often cloud) deployed - such as Web apps

  • open source libraries posted to repositories (like Maven, PyPi, npm)

  But CD is always about making the delivery automatic, reliable and frequent.

Deployment

a particular system configuration for a running version of the software.

development (dev)

testing (test)

production (prod)

This pattern - called dev/test/prod - can vary by project but is a useful generalization

development (dev)

• development team needs easy access to everyone's latest changes integrated together

testing (test)

as we prepare a release, we need to freeze it for a while to run a full suite of tests

production (prod)

code and system actually in use (supporting real users)

Environments

Deployments need distinct configuration settings (e.g. hostnames, URL, passwords)

• Such settings cannot be part of the source code

  • they are often handled in the code by OS environment variables

  • So CI/CD frameworks provide an environment feature

    • secrets

secrets

passwords, tokens, etc. get special treatment so they can be:

• hidden

• changed regularly (rotated)

reliability

is one of the most difficult challenges in software development.

Useful software is:

• extensive (10-100s of packages, each 10s-1000s of source files)

• interdependent in complex ways

• frequently changing (new features, fixes, redesigns)

• a collaboration of many individuals and teams

• subject to many requirements (functional, performance, security etc.)

Failures of software have major real-world consequences.

shift left

• Find problems as early in the development cycle as possible.

• there are big design challenges that you need to consider early

Waterfall —> impractical

Upfront Design Challenges

Many software qualities that are hard (or impossible) to "bolt on" later:

• Code modularity / API design / interface evolution

• Security

• Internationalization (I8N)

• Accessibility

• Serviceability

• Thread safety / concurrency / scalability

Address non-functional requirements from the inception of your project

• Internationalization (I8N)

• supporting multiple languages and cultures

• Accessibility

• making the software usable by people with disabilities

• Serviceability

• tools to diagnose problems that occur in production

Developing Secure Software

Security requires careful design and implementation up front:

• do not take the attitude: "we will add security later"

Developing Secure Software (key)

Key practices:

• Adopt a design methodology like Threat modeling

• Ensure developer is trained on accepted principles - like OWASP for Web apps

• consider trust boundaries - the attacker can reach the mid-tier directly, so test that

• Every developer is personally responsible for product security

Threat Modeling

• Defining security requirements. 

•  Creating an application diagram. 

•  Identifying threats. 

•  Mitigating threats. 

•  Validating that threats have been mitigated. 

Individual Development Practices (and Responsibility)

The developer is primarily responsible for the reliability of her code.

• Not: "seems to work - commit it and see if it passes all the testing..."

But developers make many mistakes, so teams must also:

• review, scan and test committed code as early as possible

!!!Shift left to catch issues earlier in the development process.

Memory-safe

languages like Java and Rust prevent this:

• using invalid index on array (string)

• miscalculating the value of a pointer

• following ("dereferencing") a null pointer

Programs in system-level languages (C, assembler) can modify arbitrary memory:

• unintentionally - as a bug (more than half of all bugs in OS code)

• intentionally - such as malware that exploited Heartbleed circa 2014

Static Analysis (Scanning Software)

• static checking

• data type issues - questionable conversions, etc.

• code flow issues - "dead code" (not used), array bounds, etc.

• security issues - buffer overflow, password in source, etc

• internationalization issues - e.g. messages in english only

Weed out the many false positives (unnecessary warnings) with options and directives.

Commonly used tools (among many):

• your compiler's warnings: like -Xlint

• ruff, klockwork, findbugs, AppScan

Static checking

finding problems by examining the source code (not by running it)

Dynamic Checking - Assertions

a program statement that must always be True if the program logic is correct

• inspired by correctness proofs - but less comprehensive

• a first line of defense inside the functional code (testing assertions are different)

• Think through whether failure should stop your program or just be logged.

Dynamic checking

tools that check for problems by running the software

• Languages like C that allow memory overlays

  • buffer overrun

  • memory leak

• Memory checkers

Dynamic checking is especially useful for such languages

• but also useful for other tasks like security testing

buffer overrun

• running off the end of an array of bytes

memory leak

• not freeing memory that is no longer needed

Memory checkers

• look for memory leaks, buffer overruns, etc.

  • Valgrind is a popular tool for C/C++

Manual testing

when members of the development team act like users

• the testers follow a script to check that the software works.

  • very inefficient

  • might be omitted or done badly by the team

Automated tests

programs that check the correctness of a software product.

• sometimes called self-verifying

• check many features and conditions; produce an itemized pass/fail report

• widely applicable through the development cycle

Unit tests

e foundation for all other testing:

• makes sure the smallest pieces are correct

• they are automated

• well-suited for regression testing

• mainly applicable to functional testing

Every programming language has at least one unit testing framework.

check the smallest components or your code - methods or classes

Integration tests

check how different components work together

System tests (or end-to-end tests)

check an overall deployment of your hardware and software

developer tests

run before publishing code to coworkers

continuous integration tests

run regularly on the code base while it is under development by the team

beta test

run with a level of the software that is almost ready to release

acceptance test

run by the user organization upon receiving a new production version of the software

smoke test

a quick test often run to check proper installation of new software

Functional testing

verifies the software's features

test the actual security features of the product

• e.g. that data is actually encrypted or that only admins can perform admin actions

• e.g. that tax software correctly handles each line of each tax form

non-functional requirements

• load testing

• performance testing and benchmarking

• security testing

• internalization/localization testing

• accessibility testing

• endurance testing

load testing

• support concurrent users

• to assess performance under high traffic.

performance testing and benchmarking

good performance

security testing

• cannot be subverted by attackers

internalization/localization testing

• tests the support users in many languages

accessibility testing

• tests the support users with disabilities

endurance testing

• runs a long time without crashing

Static application security testing (SAST)

• scans for known patterns in source code like failure to check input parameters

Dynamic application security testing (DAST)

• (passive) watches app communication looking for patterns of concern.

• (active) automatically generating potentially difficult inputs

Software composition analysis (SCA)

• checks the libraries your app uses for known vulnerabilities

regression

is a bug introduced by a change which breaks the software

• adding a feature or

• fixing a bug or

• redesigning some code

Every time a programmer makes a change, there is a risk of regression.

Regression tests

are automated tests designed to be rerun whenever the code changes.

Coverage

analysis scrutinizes which lines of code are exercised when test suites are run.

• starts with reporting an overall percentage of lines covered

• but allows you to drill into different parts of your software project to see which areas are least well tested.

Coverage is the gold standard for testing completeness.

Unit Testing - Architectural Considerations

Test at all layers of your software.

• UI - will probably require a tool like Selenium

• REST APIs

• Unit test your classes/methods (sometimes called "the business logic")

• Database layer

mocking

creating a fake layer sufficient for testing higher layers

• e.g. classes that pretend to contact a database, but just return boilerplate data

• the goal is to make frequent unit testing more efficient

Test Driven Development (TDD)

"Write the test first"

When beginning work on a requirement (feature) - the developer should:

1. Write one or more tests to check that the software works according to the new requirement

2. Those tests initially fail

3. Write code until the tests succeed (and no existing tests broken)

4. Ensure that the changes are as simple as possible

kent beck made popular

Behavior-Driven Development (BDD)

What if the Business Owner could write the test?

• makes business owner think more clearly

• avoid miscommunication of the intent

• with enough rigor, the spec could form the basis of a test suite

BDD enhances TDD with a precise, executable language to specify app behavior

Tools like Cucumber were invented on this idea

pytest

• A testing framework for Python that makes it easy to write simple and scalable test cases. It supports fixtures, parameterized testing, and has a rich plugin architecture.

• more flexible and easier to use than other frameworks.