Big Data

What is Big Data?

  • It is the process of collecting and analysing large data sets from traditional and digital sources to identify trends and patterns that can be used in decision-making

  • Most big data has been created in the last 2-3 years

  • Volume: Lower costs of data storage so data warehousing allows for greater amounts of data to be analysed

  • Velocity: Analytical software and speed of manipulation allows for rapid reporting to management

  • Variety: More unstructured and Qualitative information. E.g Use of customer feedback and social media information

Many reasons for the exponential growth of big data:

  • Retail e-commerce databases

  • Interactions with websites and mobile device apps

  • Use of logistics, transportation systems

  • Social media interaction

  • Location data (e.g. GPS-generated)

  • Internet of Things (IoT)

Uses of Big Data:

  • Analysis of Operations: Better monitoring of processes and equipment operation; speed of production, costs of manufacture, customer satisfaction: IoT Internet of things

  • Marketing Information: Loyalty cards, website use, credit cards, and customer feedback enable companies to understand their customers’ behaviour and reactions more deeply

  • Improving Decision Making: Analyse internal business information to identify successes and failures; strengths and weaknesses. Understand the reasons for success or failure

Users of Big Data:

  • Online retailers such as Amazon and Temu

  • Transport scheduling: Delivery Vehicle Operations, Aircraft Movements, Bus and Train arrivals

  • Personalised Marketing: Collect data on Sales patterns and levels of inventory held at different locations better matching capacity, stock levels, and output to demand. This results in improved availability of products

Benefits of Big Data:

  • Tracking and monitoring the performance, safety and reliability of operation equipment (e.g. data generated by sensors)

  • Generating marketing insights into the needs and wants of customers, based on the transactions, feedback, and comments (e.g. from e-commerce analytics, and social media posts). Big data is revolutionising traditional market research

  • Improved decision-making: For example analysing the real-time impact of pricing changes or other elements of the marketing mix (The use of big data to drive dynamic pricing is a great example of this)

  • More efficient management of capacity: The increasing use of big data to inform decision-making about capacity management (e.g. in transportation and logistics systems) is a great example of how big data can help a business cooperate more efficiently

Disadvantages of Big Data:

  • Privacy concerns: Collecting large amounts of personal data raises ethical issues regarding individual privacy and can lead to legal repercussions if not managed correctly. 

  • Security risks: Storing and processing large datasets increases the potential for data breaches and cyberattacks, which can damage a company's reputation and lead to financial losses. 

  • Cost of implementation: Implementing big data infrastructure requires significant investment in hardware, software, and skilled personnel. 

  • Data quality issues: Not all data collected is accurate or reliable, which can lead to misleading analysis and poor decision-making. 

  • Data integration challenges: Combining data from multiple sources can be complex and time-consuming, leading to inconsistencies and inaccurate results. 

  • Bias in algorithms: Machine learning algorithms used to analyse big data can perpetuate biases present in the data, leading to discriminatory outcomes. 

  • Information overload: Too much data can be overwhelming for decision-makers, making it difficult to identify the most relevant information. 

  • Complexity in data management: Managing large datasets requires specialised skills and expertise, which can be a challenge for many organisations. 

Ethical Issues of Big Data:

  • Privacy: The most prominent concern is the collection and storage of vast amounts of personal data without adequate user knowledge or consent, potentially leading to intrusive profiling and tracking. 

  • Algorithmic Bias: Data analysis algorithms can perpetuate existing societal biases, leading to discriminatory outcomes in areas like loan approvals or job hiring based on biased data sets. 

  • Lack of Transparency: The complex nature of big data analysis often makes it difficult for individuals to understand how their data is being used, raising concerns about accountability and control. 

  • Informed Consent: Obtaining meaningful informed consent from individuals for data collection, especially when data is aggregated from various sources, can be challenging. 

  • Data Security: The sheer volume of data handled by businesses increases the risk of data breaches and unauthorised access to sensitive personal information. 

  • Misuse of Data: Data could be used for unethical purposes like targeted manipulation, profiling, or creating unfair advantages in markets.