Data Acquisition and Management: Foundations and Course Framework

Instructor Profile and Professional Background

  • Dr. Ali El-Sharif holds several advanced degrees and titles, including:

    • Ph.D. in Information Systems, specializing in Explainable Artificial Intelligence (XAI).

    • Master of Business Administration (MBA).

    • Master of Science in Information Security.

    • Bachelor of Science in Computer Engineering.

  • Professional Roles:

    • Co-Organizer at Aggregate Intellect.

    • Advisor and Professor.

    • Author of the 1st Edition book: Building Responsible AI with Python: Learn to identify and mitigate bias with hands-on code examples (published by Packt), co-authored with Serg Masís and Nicholas Schmidt.

Course Overview and Grading Policy

  • Grading Breakdown:

    • Quizzes (Individual): 20%20\%

    • Midterm Examination (Individual): 25%25\%

    • Assignment (Group): 15%15\%

    • Project (Group): 15%15\%

    • Final Examination (Individual): 25%25\%

    • Total: 100%100\%

  • Grading Standards:

    • There will be absolutely no curves applied to grades.

    • There will be no rounding up of grades at the end of the term.

    • Assessments (quizzes, midterm, and final) are closely aligned with class discussions.

    • There is a strong and positive correlation between attendance and academic performance.

  • Learning Resources:

    • LinkedIn Learning.

    • DataCamp.

    • W3Schools.

    • Primary Textbook: Principles of Database Management: The Practical Guide to Storing, Managing and Analyzing Big and Small Data by Wilfried Lemahieu, Seppe Vanden Broucke, and Bart Baesens.

The Learning Pyramid and Pedagogical Philosophy

  • The Learning Pyramid illustrates how different activities influence memory retention and learning outcomes:

    • 10%10\% Retention (Reading): People remember 10%10\% of what they read. Outcomes include the ability to Define, List, and Describe.

    • 20%20\% Retention (Hearing): Outcomes include the ability to Explain.

    • 30%30\% Retention (Seeing): This involves viewing images, watching videos, or attending exhibits and sites.

    • 50%50\% Retention (Seeing and Hearing): This involves watching a demonstration. Outcomes include the ability to Demonstrate and Apply.

    • 70%70\% Retention (Saying and Writing): This involves participating in hands-on workshops and designing collaborative lessons. Outcomes include the ability to Practice and Analyze.

    • 90%90\% Retention (Doing): This involves simulating, modeling, experiencing a lesson, or performing a presentation ("Do the Real Thing"). Outcomes include the ability to Create and Evaluate.

Communication and Student Support

  • Preferred Communication: Email is the only preferred method; chat messages are not checked.

  • Inquiry Philosophy: Questions do not need to be specific or well-defined. Statements like "I need help with this," "I am not clear on this," or "I am not sure what to do next" are encouraged.

Group Assessments and GitHub Workflow

  • Group Formation:

    • Groups must consist of up to 44 members.

    • Students must self-enroll in their groups.

  • GitHub Requirements:

    • Groups must create two private GitHub repositories for the instructor (elsharif-UWindsor):

      1. 8730-assignment

      2. 8730-project

    • Evidence of collaboration must be visible via the repository.

  • Git Lifecycle Commands:

    • Working Directory to Staging: git add.

    • Staging to Local Repository: git commit.

    • Local Repository to Remote Repository: git push.

    • Remote Repository to Local Repository: git clone or git pull.

    • Local Repository to Staging/Working: git checkout or git revert.

  • Grading Criteria: Projects are graded on logic, process, presentation, and delivered results.

Project Proposal and Structure

  • Proposal Objectives: A group project must be approved via an ungraded proposal before submission. It must address:

    • The specific problem or insight sought through data.

    • Background and motivation.

    • Target customer and beneficiaries.

    • Objectives and specific questions to answer.

  • Target Audience: Slides should target a business audience with no code. Explain terminology for non-experts.

  • Required Documentation Structure:

    • Background/Motivation: Describes the current state and reasons for interest.

    • Problem Statement: Format: "[Business Audience] needs to [know what] in order to [make what decisions]."

    • Dataset Description and Validation: Confirming data availability and cleaning requirements.

    • Methods: Strategy to achieve results.

    • Analysis, Results, and Future Work.

  • Topic Selection Criteria:

    • Industry interest.

    • Feasibility within the time frame.

    • Meeting course requirements.

Academic Integrity and AI Policy

  • Citation Standards: All work must use APA v7 in-text citations and a reference section.

  • Data-Driven Analysis: Conclusions must be supported by the dataset or credible sources, not personal opinions.

  • Code Sharing: Code from others can be used if properly referenced.

  • Generative AI (LLMs like ChatGPT and Gemini):

    • Should be used as a tool, not a source of facts.

    • Prompts used must be cited.

    • Students remain responsible for all submitted work and must be able to answer questions about the logic/process.

  • Impact of AI on Learning:

    • Based on research by Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, \u00d6zge Kabakc\u0131, and Rei Mariman.

    • Students with GPT-4 access perform better, but they perform worse than those who never had access once the tool is removed.

    • Principle: AI should help but not replace the student.

Data Acquisition Methods

  • Methods for Acquiring Data:

    • Manually downloading data.

    • Online Data Banks.

    • Application Programming Interface (API): A set of protocols allowing two systems to communicate and share data. Interface defined as: "If you give me this input, I will give you back this output or perform this task."

    • Web Scraping: Extracting data from websites.

  • Web Scraping Tools:

    • curl Commands: Returns web content in text format. Examples:

      • curl https://www.linkedin.com (displays content).

      • curl https://www.linkedin.com > 745-test.html (saves to file).

    • Common Crawl: Found at https://commoncrawl.org/.

    • Wayback Machine: Found at https://web.archive.org/.

Data Schemas and Models

  • Relational Database Model:

    • Organizes data into tables related by Primary Keys and Foreign Keys.

    • Uses MySQL as a standard.

  • Semi-Structured Schemas (Text-Based):

    • JSON (JavaScript Object Notation): Uses key-value pairs.

      • Example: json { "Product Name": "Adjustable Race", "Product Number": "AR-5381", "Product Price": 5.00 }             

    • XML (Extensible Markup Language): Uses tags to define data structure.

      • Example: xml <Product> <ProductName>Adjustable Race</ProductName> <ProductNumber>AR-5381</ProductNumber> <ProductPrice>5.00</ProductPrice> </Product> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

  • Key Distinction: Concepts include the difference between pre-defined schemas (Relational) and flexible schemas (Semi-structured).