Comprehensive Study Notes on AI-Assisted Quantitative and Qualitative Data Analysis

Foundations of AI-Assisted Quantitative and Qualitative Data Analysis

• Program Context and Objectives:

  • The pedagogical focus centers on translating analytical questions directly into executable Python scripts, enabling researchers to process cross-sectional, time series, and panel datasets using artificial intelligence code generation while retaining methodological oversight and interpretative control.
  • Research analysis is fundamentally an experiential, hands-on discipline requiring direct experimentation with data rather than passive theoretical instruction.
  • Modern data analytics pipelines are partitioned into two core categories:
  • Quantitative Financial Time Series Analysis: Sourced from publicly available databases where data integrity, authenticity, and legal clearance are verified, allowing zero-threat computation of complex financial models.
  • Qualitative Textual Data Analysis: Involves extracting sentiment, governance tones, and linguistic indices from large unstructured text corpora (e.g., corporate filings), requiring extensive pre-processing and data pipeline structuring.

• Software Environment and Tooling Strategy:

  • Rather than relying solely on proprietary point-and-click statistical packages (such as SPSS, EViews, or Stata), modern research relies heavily on programming environments like Python and R.
  • Zero prior syntax knowledge or memorization of Python libraries is required to execute comprehensive statistical workflows; generative AI platforms (such as Claude.ai) can formulate complete, robust pipelines given precise architectural prompts.
  • Execution Platforms:
  • Command Prompt (CMD): Executed natively via Windows (Win + R, entering cmd). This local terminal method is prioritized over browser or IDE environments to minimize software overhead, eliminate configuration roadblocks, maintain data privacy, and allow offline local computation.
  • Google Colab / Jupyter Notebooks (Anaconda): Alternative environments supporting interactive, block-by-block execution and collaborative cloud-hosted computation.

Financial Time Series Analysis: End-to-End Python Workflow

• Dataset Architecture and Structure:

  • Source: Bombay Stock Exchange (BSE) of India historical equity data.
  • Observation Span: Covering the one-year duration from October 1, 2025 (10/01/2025) through September 30, 2026 (09/30/2026), comprising approximately $250$ trading day observations.
  • Dependent Variable ($Y$): BSE SENSEX index values.
  • Independent Variables ($X_1, X_2, X_3$): Closing prices of three major enterprise equities: Cipla, Mahindra & Mahindra (M&M), and Tata Consultancy Services (TCS).
  • File Formats: Sourced as a standard Excel file (BSE_sensex.xlsx) located in a local working directory (e.g., C:\Users\Ramakrishna\Desktop\ethical research\datasets).

• Prompt Specification for AI Script Generation:

  • Formulated as a concise, highly specific 100-word prompt instructing the AI model (Claude.ai) to generate a fully automated end-to-end financial time series pipeline.
  • Core Prompt Instructions:
  • State the nature of the data (financial time series) and explicitly define the exact local system path to the Excel file.
  • Design the script for direct execution inside the Windows Command Prompt.
  • Assume absolute zero prior programming knowledge on the user's part.
  • Request two separate files: an installation file/script to acquire all necessary libraries, and a core analysis execution script.
  • Mandate the calculation of full statistical phases: basic descriptive statistics, stylized facts, stationarity diagnostics, bivariate models, and multiple regression.
  • Specify variables: SENSEX as the dependent variable; Cipla, Mahindra, and TCS as independent variables.
  • Direct outputs into structured files: a multi-sheet Microsoft Excel workbook containing all tabular calculations, and high-resolution JPEG files for all graphical figures.
  • Request comprehensive, step-by-step terminal instructions.

• Required Python Libraries:

  • pandas: Ingestion, alignment, and manipulation of tabular series and time-indexed dataframes.
  • numpy: High-performance numerical computations, array transformations, and logarithmic operations.
  • scipy: Advanced scientific computing and parametric/non-parametric probability distributions.
  • statsmodels: Implementation of formal econometric models, time series tests, and ordinary least squares regressions.
  • matplotlib: Core 2D plotting engine generating publication-quality visual layouts.
  • seaborn: Statistical data visualization providing aesthetic heatmaps, distribution curves, and pair plots.
  • openpyxl: Excel spreadsheet reading and writing engine enabling dynamic multi-tab generation.
  • pillow (PIL): Image processing module for rendering and formatting high-resolution JPEG graphics.
  • arch: Specialized autoregressive conditional heteroskedasticity library used for modeling financial market volatility ($ARCH$ and $GARCH$ frameworks).

Time Series Statistical Metrics and Visualization Suite

• Computational Output Scope:

  • Execution of the generated Python script locally inside the Command Prompt runs entirely within approximately $5$ to $10$ minutes.
  • Generates an Excel workbook containing approximately $32$ distinct thematic sheets plus dynamic chart sheets, accompanied by an output folder populated with $17$ publication-ready JPEG figures.

• Statistical Phases and Tabular Sheets:

  • Data Cleaning and Return Computation: Calculation of daily log returns for all equities and indices: $\text{Return}_t = \ln\left(\frac{P_t}{P_{t-1}}\right)$.
  • Descriptive Statistics: Full distributional assessments across both raw closing price series and log return series:
  • Central Tendency: Mean, Median, Mode.
  • Dispersion: Standard Deviation, Variance, Range.
  • Higher Moments: Skewness (asymmetry) and Kurtosis (leptokurtic tail measurement).
  • Normality Tests: Jarque-Bera test statistic and Shapiro-Wilk test probability calculations.
  • Stylized Facts Analysis:
  • Volatility clustering validation.
  • Ljung-Box test for residual autocorrelation across designated lag structures.
  • Engle's Autoregressive Conditional Heteroskedasticity Lagrange Multiplier ($ARCH\text{-}LM$) test for conditional heteroskedasticity.
  • Non-Gaussianity and asymmetry evaluation.
  • Univariate Stationarity and Unit Root Tests:
  • Augmented Dickey-Fuller ($ADF$) test for assessing the presence of unit roots.
  • Kwiatkowski-Phillips-Schmidt-Shin ($KPSS$) test for confirming trend/level stationarity.
  • Phillips-Perron ($PP$) test for non-parametric handling of autocorrelation and heteroskedasticity.
  • Autocorrelation Dynamics:
  • Autocorrelation Function ($ACF$) and Partial Autocorrelation Function ($PACF$) across standard returns.
  • Autocorrelation functions evaluated over squared returns ($r_t^2$) and absolute returns ($|r_t|$) to diagnose persistent non-linear volatility dependency.
  • Volatility and Risk Modeling:
  • Generalized Autoregressive Conditional Heteroskedasticity model: $GARCH(1,1)$.
  • Dynamic variance tracking and $30\text{-day}$ rolling standard deviations.
  • Bivariate and Multivariate Dependencies:
  • Covariance matrices and Pearson correlation matrices.
  • Ordinary Least Squares ($OLS$) bivariate regressions.
  • Multiple Linear Regression modeling SENSEX against Cipla, Mahindra, and TCS simultaneously.

• Graphical Visualizations (17 Core Images):

  • Price Performance Trajectories: Comparative base-line tracking indexed to a normalized value of $100$.
  • Return Series Plots: High-frequency daily log-return fluctuation series for SENSEX, Cipla, Mahindra, and TCS.
  • Six-Panel Stylized Facts Composite Figures: Detailed individual stock diagnostics (e.g., Cipla) comprising:
  • Volatility clustering time series.
  • Empirical return distribution histogram overlaid with theoretical curves.
  • Normal Quantile-Quantile ($Q\text{-}Q$) plots.
  • Standard Return Autocorrelation Function ($ACF$).
  • Squared Return Autocorrelation Function ($ACF$).
  • Absolute Return Autocorrelation Function ($ACF$).
  • Rolling Volatility: $30\text{-day}$ moving-window volatility series tracking market regime transitions.
  • Dependency Maps: Correlation heatmaps reflecting inter-asset linear dependencies.
  • Bivariate Scatter Plots: Linear association scatter plots displaying independent assets (Cipla, Mahindra, TCS) paired against the dependent market index (SENSEX).
  • Pairwise Matrix Plots: Multi-asset cross-distribution pair plots featuring univariate density curves.

Qualitative and Panel Data Engineering: Annual Report Textual Analysis

• Corporate Corpus Overview:

  • Empirical research project investigating qualitative disclosures, governance sentiment, and corporate reporting within Indian Central Public Sector Enterprises (CPSEs)—specifically listed Navratna enterprises.
  • Dataset consists of $171$ to $172$ primary annual reports across $20$ listed Navratna firms, supplemented by secondary sets reaching $224$ to $225$ total reports spanning extensive reporting years ($2015$ through $2026$).
  • Project scope involved parsing approximately $200{,}000$ to $300{,}000$ documents over a three-month research window.

• Structural Challenges of Corporate Annual Reports:

  • Volume: Standard annual filings range from $300$ to $400$ pages each (e.g., Bharat Electronics Limited / BEL reports exceed $312$ pages).
  • Language Structure: Bilingual documentation containing English and Hindi text arranged side-by-side on individual pages.
  • File Encoding: Substantial proportions of reports are stored as scanned bitmap image files rather than native digital PDFs (e.g., Rashtriya Chemicals and Fertilizers Limited / RCFL), necessitating optical character conversion.
  • LLM Ingestion Limits: Direct upload of raw multi-hundred-page documents into Large Language Models fails due to token context window thresholds, high API costs, and context loss.

• Automated PDF Parsing and Selective Conversion Workflow:

  • Utility Employed: PDFgear—a dedicated offline conversion software.
  • Targeted Section Extraction: Rather than extracting entire documents, researchers define targeted page intervals where substantive narrative content resides:
  • Chairman's Statement / Speech.
  • Management Discussion and Analysis (MD&A).
  • Independent Auditor's Reports.
  • Corporate Social Responsibility (CSR) and Business Responsibility Reports.
  • Specific section windows (e.g., pages $21\text{ to }40$ in one report; pages $50\text{ to }75$ in another).
  • Batch Conversion Execution: Hundreds of multi-megabyte PDF files are segmented and converted into lightweight, clean plain text (.txt) files within $15\text{ to }20$ minutes.
  • Text files eliminate token bloat and optical image overhead, enabling instantaneous retrieval, indexing, and processing across Large Language Models.

• Panel Data Construction via Standardized Naming Conventions:

  • Large-scale multi-file ingestion requires strict, programmatic naming taxonomies to preserve structure without manual data entry.
  • Files are formatted systematically by entity and temporal period (e.g., BEL_2016.txt, BEL_2020.txt, BEL_2021.txt).
  • Custom Python scripts ingest the entire directory, parse company identifiers and reporting years directly from file strings, execute linguistic algorithms, and populate a structured panel dataframe.
  • Out of $172$ raw files, an econometric panel of $142$ to $143$ fully verified, balanced company-year observations was finalized for downstream modeling.

• Linguistic, Forensic, and Textual Metrics Generated:

  • Loughran-McDonald (L&M) Financial Dictionary Scoring: Specialized master financial lexicon metrics tracking specialized operational tones:
  • Net Tone ($\text{Positive} - \text{Negative} / \text{Total Words}$
  • Uncertainty Score.
  • Litigation / Litigate Exposure Score.
  • Readability and Obfuscation Metrics:
  • Gunning Fog Index: Evaluation of structural readability and syllable complexity.
  • Flesch-Kincaid Grade Level: Measure of required comprehension education levels.
  • Score Certainty Ratios.
  • Theoretical Modeling Applications: Extraction of narrative decoupling, linguistic obfuscation, and causal reasoning metrics.
  • Computational Duration: Processing and generating full L&M master scores across the extensive corpus required approximately $16\text{ to }18$ continuous hours of local Python script execution.

Methodological Rigor, Reproducibility, and Ethical AI Integration

• Mixed-Methods Research Design Integration:

  • Qualitative Extraction: Automated Python analysis extracts linguistic indices (tone, uncertainty, readability) from chairman speeches and governance sections.
  • Secondary Financial Panel Merging: Qualitative scores are merged directly with secondary firm-level accounting and market data extracted from standardized financial databases (CMIE Prowess / ProwessIQ):
  • Tobin's $Q$ (firm valuation ratio).
  • Return on Equity ($ROE$).
  • Return on Assets ($ROA$).
  • Altman $Z\text{-Score}$ (financial distress and credit risk assessment).

• Accuracy Verification and "Human-in-the-Loop" Protocols:

  • AI script outputs must be subjected to triangulation and validation before integration into formal findings.
  • Iterative validation involves executing Python routines two to three times to confirm deterministic stability.
  • Benchmarking Against Known Tools: Substantive metrics such as Flesch-Kincaid grade levels are cross-checked by pasting identical extracted text into Microsoft Word's native readability diagnostic engine. Discrepancies between the AI-generated Python script output and Microsoft Word remained tightly bounded within a $1\text{ to }2$ point margin, establishing algorithmic accuracy.

• Ethical Boundaries and Academic Transparency:

  • Acceptable and Ethical Applications:
  • Utilizing generative AI to write, debug, optimize, and modularize Python scripts.
  • Designing data conversion workflows and regex-based file extraction routines.
  • Generating preliminary interpretive outlines, thematic drafts, or visual summaries based on direct data.
  • Unacceptable and Unethical Practices:
  • Direct falsification, synthetic generation, or post-hoc manipulation of empirical data.
  • Wholly outsourcing the substantive intellectual interpretation, discussion, and manuscript authorship to AI systems without human subject-matter oversight.
  • Journal Editor Audit Readiness:
  • Rigorous journals (including Q1 publications) require full evidentiary transparency regarding AI tooling.
  • Methodological integrity demands providing a complete reproducible bundle: raw datasets, PDF source files, selective text extracts, exact Python scripts (.py), variable codebooks, extraction log files, and intermediate multi-sheet Excel workbooks.
  • Submitting complete programmatic architectures validates the work as genuine empirical research assisted by AI, disproving accusations of synthetic AI-written scholarship.

Technical Implementation: Prompt Engineering and Environment Setup

• Operational Steps for Script Execution via Command Prompt:

  • Step 1: Environment Installation: Download and install standard Python (freely accessible open-source distribution) ensuring the "Add Python to PATH" variable is toggled on.
  • Step 2: Script Preparation: Copy the AI-generated code blocks into an unformatted text editor (such as Notepad) and save the files with standard extensions:
  • install_libraries.py (or execute terminal-based pip installation commands).
  • financial_analysis.py (main analytical script).
  • Step 3: Dependency Resolution: Open the terminal window (Win + R, enter cmd), navigate or pass commands to verify and install packages:
  • pip install pandas numpy scipy statsmodels matplotlib seaborn openpyxl pillow arch
  • Step 4: Path Acquisition and Execution: Locate the directory housing the target data file and script. Right-click the Python script, select "Copy as Path", paste the full string path into the Command Prompt window, and press Enter to commence execution.

• Working with AI Service Constraints (Free-Tier Optimization):

  • Generative platforms (e.g., Claude, ChatGPT) provide robust coding support on free-tier allocations.
  • When dealing with context exhaustion or operational token time-outs ("usage limit reached; return in 2-3 hours"), work can be suspended and resumed.
  • Re-entering the session after the cooldown window and entering the simple command continue causes the model to pick up script generation precisely at the point of interruption without losing contextual continuity.

Questions & Discussion

• Use of Google Colab and Collaborative AI Platforms:

  • Participant Observation (Dr. Katherine Isaac, Symbiosis Nagpur): Analysis can be executed entirely in the cloud via Google Colab without downloading local software. Data files are uploaded into Colab's virtual drive directory, Python scripts are pasted directly into code cells, and integrated AI (e.g., Gemini) assists with inline error correction. Additionally, Claude features collaborative workspace tooling ("Co-work") allowing folder-level analysis without external upload dependencies, contingent on adequate processing speed and RAM.
  • Response: Colab, Jupyter, RStudio, and native Command Prompt are functionally valid. Local Command Prompt execution is emphasized because it runs offline, preserves data sovereignty, eliminates upload latency for massive files, and ensures researchers understand the underlying machine environment.

• Algorithmic Accuracy of AI vs. Established Software (SPSS, R, Python):

  • Question (Dr. from KL University): How accurate are analytical findings generated through AI platforms compared to traditional statistical suites like SPSS, R, or native Python, and can these AI results be trusted?
  • Response: The analytical calculations are not performed by the generative AI itself; they are executed strictly by the local Python statistical engine (statsmodels, scipy, pandas). The AI functions exclusively as a code-generation architect that writes the script. The resulting calculations represent native programmatic computations, yielding precision identical to SPSS, R, or Stata.

• Document Skimming and Page Filtering via AI vs. Specialized Software:

  • Question (Dr. J. Santoshi, Dean of Student Office, IIMC College Hyderabad): When parsing an annual report of 300 to 400 pages containing diverse corporate disclosures (Cash Flow Statements, Risk Reports, CSR Reports, Auditor Reports), can AI automatically read the index and extract only specific pages (e.g., 10 to 20 pages), or how should this be conducted?
  • Response: Relying on generative AI to ingest massive 400-page filings to locate pages introduces severe token bottlenecks and file-size upload rejections. The optimal procedural method uses PDFgear. The user references the annual report's index, enters the exact page ranges (e.g., pages 21–40) into PDFgear's batch extraction module, and converts only those target sections into .txt files. AI is subsequently deployed to analyze the targeted text.

• Handling Embedded Images and Non-Text Visuals in Filings:

  • Question: When converting mixed image/PDF documents into text files, what happens to the embedded financial charts, diagrams, and corporate images?
  • Response: The conversion routine isolates and extracts textual characters exclusively. Graphical diagrams, brand assets, and charts are omitted, producing clean, low-token text files containing only linguistic information.

• Qualitative Sentiment Analysis on Audio, Video, and Social Media (YouTube):

  • Question (Participant Registration #262396): How can the presented text-analysis pipeline be applied to qualitative finance studies examining sentiment within YouTube discussions, earnings calls, or executive interviews?
  • Response: Audio and video streams must first be transcribed into text formats using one of three conversion pathways:
  • Utilizing automated backend transcript extraction tools or web-based YouTube-to-transcript extractors.
  • Extracting the audio channel into an MP3 file and processing it through Microsoft Word's native "Detect and Transcribe" feature (as detailed by participant Rakesh), which transcribes Zoom, YouTube, or corporate earnings recordings into clean text files.
  • Processing extracted audio through automated Speech-to-Text models (e.g., OpenAI Whisper). Once text files are compiled, they enter the identical Python textual analysis pipeline.

• Empirical Research Findings and Market Signaling Mechanisms:

  • Question (Dr. Anita Yadav): What were the empirical findings of the Navratna annual report research, what theoretical frameworks were utilized, and were market impacts captured on event days or across prolonged observation windows?
  • Response: The study evaluated whether qualitative chairman disclosures in Public Sector Enterprises (PSEs) directly impacted equity valuations, drawing upon Signaling Theory, Stakeholder Theory, and Market Reaction dynamics.
  • Findings: Public sector enterprise chairmen generally operate under strict administrative oversight, bureaucratic constraints, and fixed administrative tenures, causing them to adopt conservative, risk-averse linguistic styles in annual addresses. This conservative tone fails to generate market signaling, resulting in flat or muted investor reaction and lower equity market valuations. Conversely, dynamic CPSEs with assertive leadership (e.g., Hindustan Aeronautics Limited / HAL, Mazagon Dock Shipbuilders, Garden Reach Shipbuilders) exhibit aggressive, forward-looking sentiment that corresponds with market outperformance and surging stock valuations. Impact measurements were evaluated over extended contextual event windows rather than single-day releases.

• Handling Massive Enterprise Datasets (20,000 Firms across 20 Years):

  • Question (Dr. Arunima): Can this automated extraction pipeline scale to analyze extensive institutional datasets comprising 20,000 companies over a 20-year longitudinal horizon, and must extraction be done file-by-file?
  • Response: The pipeline scales across large corpora without individual manual processing, provided standardized naming taxonomies (Company_Year.txt) are implemented across directories. The upper boundary is dictated by local hardware capability, primarily CPU multithreading and RAM capacity. Top-tier (Q1) corporate governance literature employs this batch-scripting methodology to process tens of thousands of corporate filings simultaneously.