DATA ANALYTICS F200

big data

large datasets cannot be easily managed processed or analysed with traditional tools

examples of big data

• social media

• self driving cars

• IOT

• sensors

• IOE

Data

raw facts without context

Information

data organized in a meaningful way

knowledge

insights derived from info

structured data

organized data in rows and columns

unstructured data

flexible format JSON/JAVASCRIPT

semi-structured

data has no clear structure

data mining

process used by companies turn raw data into useful info

alpha numeric

combo of letters symbols or numbers

boolean

true/false

date data

date data that can be used

text

any character

currency

data in format of money

decimal format

number format shows using decimal point

interger

whole number

percentage

number format include decimal and % signs

real

any number with or without decimal places

data formats

standardized ways to structure/store data how its read and processed by software

JPEG

used for scanned photographic images not good with images with lots of solid colour vector drawings type or line art or images with web-safe colours

TIFF

designed for printing a bitmap of image format support lossless LZW comprehension makes it good archive format for photoshop documents

JSON

used for storing and exchanging data often between serve or web application/ structured data organised into key and value pairs so easily understood by people and computer

CVS

simple text format stores tabular data compatible with most data analysis tools

lacks structure and metadata inefficient for storing complex relationships

TXT

plain document contains unformatted characters

stores raw/ unstructed data

benefit / limit of JPEG

benefit is that its widely supported

limit does not support animation

benefit and limitation of PNG

benefit support transparent backgrounds

limit is that it has a very large file size

benefit and limitations of TIFF

benefit is high quality images

limitations are that it has a very large file size

what are the stages of the DLM (data lifecycle management)

• creation

• storage

• usage

• archival

• destruction

creation

data enter system in various ways

storage

data need to be stored but decisions need to be made abt where its stored

usage

employees and others access however if about humans than data protection should be followed

archival

being stored offline when data not in use

destruction

data not in use so destroyed in a way that no one can recover

data analytics pipeline

steps prepare data for analysis and use

capture

data that is captured than converted to common format before analysis

process/cleaning data

filtering grouping sorting and removing duplicates

data validation

data is as error free as possible

data analysis

mathematical/statistical method applied to help organisation find answers to questions

data use and visualisation

make trends easier to see in the form of charts and graphs

data accuracy

data is correct precise free from errors

data quality

data should be complete consistent/useful for intended purpose

data redundancy

avoid unnecessary duplicates

data reliability

dependable data come from trustworthy sources

timelessness

data should be up to date/ available when needed

data validation

checks data against rules/constraints ensure data is reasonable and acceptable

data validity

ensuring data measures what its supposed to measure valid data aligns with intended purpose of collection

data verification

checking data has been correctly entered transferred and processed wo error

advantage of onsite storage

• full control over hardware

• no dependency on internet connection

• can be customized to organizational need

disadvantage of onsite storage

• expensive

• requires ongoing maintenance

advantage of cloud storage

• scalable/flexible

• inexpensive

• accessible from anywhere with internet

• managed by provider

disadvantage of cloud provider

• need reliable internet

• buy subscription

• security/privacy concern

• less control over infrastructure

benefit of portable storage

cheap and small and easily transported

limit of portable storage

easily lost and limited storage space

advantage of hard drive

internal storage for individual computers

limitation of hard drive

difficult to upgrade and fixed location

advantage if using NAS

easy to set up small and fits into office environment

limit of nas

limited scalability

advantage of SAN

high performance and capacity

limit of san

expensive to set up and maintain

portable sotrage

taking small backup offsite

hard drive

internal storage for individual computer

file server

sharing large volumes of data between users in organisation

NAS

dedicated storage device that enables multiple users to store and share files from a central location .

SAN

sharing large amount of data in large organisation that can cope with cost

hybrid storage

public and private clouds store sensitive data using private clouds while using public cloud for less critical workloads

community storage

cloud storage shared between organizations with similar requirements

private storage

single organization more secure

public storage

makes computing resources available to users over the public internet.

what is data transformation

processing raw data to make it suitable for use e.g analysis, marketing

improves accuracy better descion making easy automation

steps in data transformation

• d collection

• d wrangling

• d integration

• d formatting and storage

data wrangling

process that transforms/prepare data important as it reduces/remove problems before data is analysed

data maintenance

running regular checks involve checking data make sure not missing/accurate and protected

descriptive data

what happened b- simple l- no conclusion

diagnostic data

why happened b- tells why trend occurred l- appears 2 factors are linked when correlation is just by chance

predictive data

predict what might happen in future b- predict future outcome/trends l- need high quality data up to date

prescriptive data

want to know best course of action b- 7 possible actions l- overreliance lead to incorrect decision

cognitive data

analyse unstructured data b- work with unstructured data l- highly complex data makes it difficult/time consuming

tabular data

look in detail at results b- detailed analysis l- trends hard to spot

graphical data

picture of results required B- make trend easier to spot L-detail of analysis not clear

textual data

send analysis to others B- create written report L- take time to create read/understand

dashboards

presentation of number of diffrent but related datasets

data tables

present data in rows/columns

digital slides

often share with a wide range of people used by digital slides

infographics

visual way of presenting info on specific topic

degaussing

remove magnetism on HDD and so remove any data on drive

drive destruction

physically destroyed

erasure/overwriting

erasure is removing data from digital media overwriting is replacing existing data with something else

shredding

destroyed by industrial shredders break drive into small pieces making data impossible to recover

ICO

ensure organization comply with data privacy/transparency rules in public interest

computer misuse act 1990

unauthorised access to computer material or with intent to commit or facilitate a crime

data protection act 1998

view data sorted about you right to make changes to your data if its inaccurate

copyright, designs and patents act 2018

prevent anyone else using invention for 20 years copyright let musicians or author decide how their work should be used

UK GDPR

govern how personal data is handled

FREEDOM OF INFORMATION ACT 2000

public entitled to request info from public authorities

PRIVACY AND ELECTRONIC COMMUNICATIONS REGULATIONS

They give people specific privacy rights in relation to electronic communications.

job roles

• AI scientist- analytical skills/problem solving

• data analyst- analytical skills

• data architect-problem solvinf/olanning and organszation

• data engineer- problem solving

• database admin- problem solving planning/organization

• ML engineer- analytical skills/problem solving

API

interface between 2 programs programmer write program that can work with and exchange data with another program

list 3 types of API

• public

• private or interna;

• partner

REST API

better performance/simple to use L- NOT AS SECURE AS SOAP

SOAP API

advanced security features L- STRICT STANDDARDS more complex and poor performance