AI-Ready Webpage Auditor
========================

This package contains Python scripts to fetch, store, analyze, and export webpage metadata and structure for SEO and content auditing.

--------------------
FILES INCLUDED
--------------------

1. fetch_store.py
   - Purpose: Fetch HTML from the given list of URLs and store raw HTML content in the `pages` table.
   - Input: List of URLs (modify inside script or read from external file).
   - Output: SQLite database file containing fetched HTML.
   - Notes: Uses settings.py for database path.

2. page_summary.py
   - Purpose: Parse stored HTML from `pages` table and extract key SEO tags (title, description, canonical, OpenGraph, etc.).
   - Output: Stores extracted values in `page_summary` table.
   - Notes: Can be expanded to check for additional tags or specific strings.

3. page_analysis_query.py
   - Purpose: Run custom SQL queries against the `page_summary` table.
   - Use Case: Filter or find pages with missing tags, short titles, or duplicate canonicals.
   - Output: Results are printed in terminal.

4. page_summary_excel.py
   - Purpose: Export `page_summary` table data to an Excel file.
   - Output: Excel file for offline review and reporting.

5. settings.py
   - Purpose: Common configuration file for all scripts.
   - Contains: `DB_PATH` for SQLite file location (change here to use MySQL or different DB).
   - Advantage: Easy switch to another database without modifying each script.

--------------------
HOW TO USE
--------------------

1. Set the database path in settings.py (`DB_PATH`).
2. Run `fetch_store.py` to collect HTML for your target URLs.
3. Run `page_summary.py` to parse and store key metadata.
4. Use `page_analysis_query.py` to run custom checks and analysis.
5. Export results with `page_summary_excel.py` if needed.

--------------------
NOTES
--------------------
- For MySQL support, replace SQLite connection code with MySQL connector code in each script and update settings.py accordingly.
- Ensure you have `requests`, `beautifulsoup4`, `lxml`, and `pandas` installed.
- Keep your database file backed up before running DELETE queries.
- Update your URL list regularly to keep audit results fresh.

--------------------
ASCII DIAGRAM (Workflow)
--------------------

       +-----------------+
       |   Web / URLs    |
       | (HTML Sources)  |
       +--------+--------+
                |
                v
       +-----------------+
       | fetch_store.py  |
       | Fetch & Store   |
       +--------+--------+
                |
                v
       +-----------------+
       | page_summary.py |
       | Extract Tags    |
       +---+--------+----+
           |        |
           v        v
+----------------+  +---------------------+
| page_analysis_ |  | page_summary_excel. |
| query.py       |  | py Export to Excel  |
+----------------+  +---------------------+

         ^  
         |  
   +------------+
   | settings.py|
   | Shared DB  |
   +------------+

--------------------
AUTHOR
--------------------
Subhendu Mohapatra
https://www.plus2net.com
