When Python Libraries Talk: Pydoxtools Writes Blog Post About Itself in Fewer Than 100 Lines of Code!

The new Tools that LLMs and AI are giving us are amazing. To demonstrate what is already possible using our new Pydoxtools library, we wrote the first blogpost about Pydoxtools completely automatically - using Pydoxtools! Here is the code how this was done.

Pydoxtools is a versatile Python library designed to streamline AI-powered document processing and information retrieval. This library is perfect for developers who want to harness the power of AI in their projects. This article showcases an efficient method to automate article writing for your project using ChatGPT and Pydoxtools in fewer than 100 lines of code. The script demonstrates the following key functionalities:

Indexing a directory containing files with PyDoxTools
Employing an agent for information retrieval within those files
Auto-generating a text based on a set objective

Most of the code below is available as a notebook or you can simply refer to our concise script, which executes these steps in less than 100 lines of code:

https://github.com/Xyntopia/pydoxtools/blob/main/examples/automatic_project_writing.py

or open this notebook in Colab: Open the notebook in Colab.

Costs and API Key

Please note that ChatGPT is a paid service, and running the script once will cost you about 2-5 cents. Pydoxtools automatically caches all calls to ChatGPT. So subsequent runs usually turn out to be a little cheaper. To use ChatGPT, you will need to generate an OpenAI API key by registering an account at https://platform.openai.com/account/api-keys. Remember to keep your API key secure and do not share it with anyone.

Pydoxtools already includes open source LLM-models which can do the same for free, locally on your computer. As of May 2023 this is being tested.

Safeguarding Your API Key in Google Colab

When working with sensitive information like API keys, it's crucial to ensure their security. In Google Colab, you can save your API key in a separate file, allowing you to share the notebook without exposing the key. To do this, follow these simple steps:

Execute the cell below to create a new file in your Colab environment. This file will store your API key, and it will be deleted automatically when the Colab runtime is terminated.

!touch /tmp/openai_api_key

Click on the following link to open the newly created file by clicking on the following link in colab: /tmp/openai_api_key

Copy and paste your API key into the file, then save it.

By following these steps, you can ensure the security of your API key while still being able to share your notebook with others. Happy coding!

Installation

Follow these simple steps to install and configure Pydoxtools for your projects:

Install the Pydoxtools library by running the following command:

%%capture
# if we want to install directly from our repository:
#!pip install -U --force-reinstall --no-deps "pydoxtools[etl,inference] @ git+https://github.com/xyntopia/pydoxtools.git"
!pip install -U pydoxtools[etl,inference]==0.6.3

After installation, restart the runtime to load the newly installed libraries into Jupyter.

Now we are loading the OPENAI_API_KEY from our file.

#load the key as an environment variable:
import os
# load the key
with open('/tmp/openai_api_key') as f:
  os.environ['OPENAI_API_KEY']=f.read()

now we can initialize pydoxtools which will automatically make use of the OPENAI_API_KEY

import logging

import dask
from chromadb.config import Settings

import pydoxtools as pdx
from pydoxtools import agent as ag
from pydoxtools.settings import settings

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)
logging.getLogger("pydoxtools.document").setLevel(logging.INFO)

configuration

Pydoxtools can be configured in various ways. But for this example we are using two settings:

Pydoxtools uses dask in the background to handle large indexes and databases. We could even index terabytes of data this way! For this example though, we are setting the dask scheduler to "synchronous" so that we can see everything thats happening locally and make it easy to debug the script.
Pydoxtools has a caching mechanism which caches calls to pydoxtoos.Document. This helps during development for much faster execution on subsequent runs (for example the vector index creation or extraction of other information from documents).

# pydoxtools.DocumentBag uses a dask scheduler for parallel computing
# in the background. For easier debugging, we set this to "synchronous"
dask.config.set(scheduler='synchronous')
# dask.config.set(scheduler='multiprocessing') # can als be used...

settings.PDX_ENABLE_DISK_CACHE = True  # turn on caching for pydoxtools

download our project

In order for our program to work, we need to provide the AI with information. In this case we are using files in a directory as a source of information! We are simply downloading the "Pydoxtools" project from github. Essentialy pydoxtools is writing about itself :-). You could also mount a google drive here or simply load a folder on your computer if you're running this notebook locally on your computer.'

!cd /content
!git clone https://github.com/Xyntopia/pydoxtools.git

Index initialization

In order for an LLM like ChatGPT to retrieve the information it needs to be saved in a "vectorformat". This way we can retrieve relevant information using nearest neighbour search. We are using ChromaDB here for this purpose, but there are many other choices available.

##### Use chromadb as a vectorstore #####
chroma_settings = Settings(
    chroma_db_impl="duckdb+parquet",
    persist_directory=str(settings.PDX_CACHE_DIR_BASE / "chromadb"),
    anonymized_telemetry=False
)

# create our source of information. It creates a list of documents
# in pydoxtools called "pydoxtools.DocumentBag" (which itself holds a list of pydoxtools.Document) and
# here we choose to use pydoxtools itself as an information source!
root_dir = "/content/pydoxtools"
ds = pdx.DocumentBag(
    source=root_dir,
    exclude=[  # ignore some files which make the indexing rather inefficient
        '.git/', '.idea/', '/node_modules', '/dist',
        '/__pycache__/', '.pytest_cache/', '.chroma', '.svg', '.lock',
        "/site/"
    ],
    forgiving_extracts=True
)

Initialize agent, give it a writing objective and compute the index

Now that we have everything setup, we can initialize our LLM Agent with the provided information. For the pydoxtools project in this example, computing the index will take about 5-10 minutes. In total there will be about 4000 text snippets in the vector index for the project after finishing the computation.. When using the pydoxtools cache, subsequent calculations will be much faster (~1 min).

final_result = []

agent = ag.LLMAgent(
    vector_store=chroma_settings,
    objective="Write a blog post, introducing a new library (which was developed by us, "
              "the company 'Xyntopia') to "
              "visitors of our corporate webpage, which might want to use the pydoxtools library but "
              "have no idea about programming. Make sure, the text is about half a page long.",
    data_source=ds
)
agent.pre_compute_index()

>
[########################################] | 100% Completed | 60.10 s

Search for relevant Information

The agent is able to store information as question/answer pairs and makes use of that information when executing tasks. In order to get our algorithm running a bit more quickly, we answer a basic question manually, to get the algorithm started more quickly... In a real app you could ask questions like this in a user-dialog.

agent.add_question(question="Can you please provide the main topic of the project or some primary "
                            "keywords related to the project, "
                            "to help with identifying the relevant files in the directory?",
                    answer="python library, AI, pipelines")

Having this information, we ask the agent to come up with a few more questions that it needs to answer before being able to write the article.

# first, gather some basic information...
questions = agent.execute_task(
  task="What additional information do you need to create a first, very short outline as a draft? " \
        "provide it as a ranked list of questions", save_task=True)

print("\n".join(questions))

>
What is the name of the new library developed by Xyntopia?
What is the purpose of the pydoxtools library?
What are some examples of how the pydoxtools library can be used in AI pipelines?
What are some benefits of using the pydoxtools library for non-programmers?
What are some potential drawbacks or limitations of the pydoxtools library?

Having created this list of questions, we can now ask the agent to research them by itself. It will automatically use the index we computed above for this task.

# we only use the first 5 provided questions to make it faster ;).
agent.research_questions(questions[:5], allowed_documents=["text/markdown"])

After retrieving this information we can tell the agent to write the text. We tell it to automatically make use of the information by setting the "context_size" parameter to a value greater than 0. This represents the pieces of stored information that it will use to fulfill the task.

txt = agent.execute_task(task="Complete the overall objective, formulate the text "
                              "based on answered questions and format it in markdown.",
                          context_size=20, max_tokens=1000, formatting="txt")
final_result.append(txt)  # add a first draft to the result

Having our first draft of the text, let's criticize it to improve the quality! Then with this critique create a new list of tasks that we can give to the agent to execute one-by-one. Gradually improving our text.

critique = agent.execute_task(task=f"Given this text:\n\n```markdown\n{txt}\n```"
                                    "\n\nlist 5 points of critique about the text",
                              context_size=0, max_tokens=1000)

tasks = agent.execute_task(
    task=f"Given this text:\n\n```markdown\n{txt}\n```\n\n"
          f"and its critique: {critique}\n\n"
          "Generate instructions that would make it better. "
          "Sort them by importance and return it as a list of tasks",
    context_size=0, max_tokens=1000)

for t in tasks:
    task = f"Given this text:\n\n" \
            f"```markdown\n{txt}\n```\n\n" \
            f"Make the text better by executing this task: '{t}' " \
            f"and integrate it into the given text, but keep the overall objective in mind."
    txt = agent.execute_task(task, context_size=10, max_tokens=1000, formatting="markdown")
    final_result.append([task, txt])

print("\n".join(str(t) for t in tasks))

{'Specify the text length': 'Write a blog post that is approximately half a page long.'}
{'Define the target audience': 'Introduce the pydoxtools library to visitors of our corporate webpage who are interested in using it but have no programming experience.'}
{'Mention the name of the library': 'Introduce our new library, pydoxtools, to readers.'}
{'Explain the benefits of using the library': 'Highlight the advantages of using pydoxtools, such as simplifying documentation generation and improving workflow efficiency.'}
{'Include a call-to-action or next steps for the reader': 'Encourage readers to try out pydoxtools by providing a link to download the library and offering support resources for beginners.'}

# for debugging, you can see all intermediate results, simply uncomment the variable to check:

#final_result  # for the evolution of the final text
#agent._debug_queue  # in order to check all requests made to llms and vectorstores etc...

Final text

And here is the generated text. Subsequent runs will usually produce differing Versions of the text, even though the temperature of ChatGPT is set to zero (which should remove randomness from output). I have not immediately found any obvious reasons why this would be the case. Pydoxtools usually creates the same input on subsequent runs, so this should rather not be the reason. Additionally, occasionally, ChatGPT creates blog posts that repeat themselves in different sections.

from IPython.display import Markdown
Markdown(txt.strip("`").replace("markdown",""))

Generated Text:

Introduction to Pydoxtools Library

Xyntopia is proud to introduce its new python library, pydoxtools, designed to provide a convenient and efficient way of handling documents. With functionalities such as pipeline management, integration with AI models, low-resource table extraction, document analysis, and question-answering, pydoxtools offers a sophisticated interface for reading and writing documents.

Where to get it

To access or download the pydoxtools library, please visit our corporate webpage at [insert webpage link here].

Some benefits of using the pydoxtools library include support for most of today's document formats, vector index creation, entity and address identification, list and keyword extraction, and data normalization, translation, and cleaning.

Users can experience a new level of convenience and efficiency in handling documents and reimagine their data extraction pipelines with pydoxtools. It is designed to work with AI models such as GPT, Alpaca, and Huggingface, making it an excellent tool for AI pipelines.

Use Cases

Examples of how the pydoxtools library can be used in AI pipelines include:

Extracting tables from PDF documents without the need for configuration and expensive layout detection algorithms
Identifying entities and addresses in documents
Creating vector indexes for document search
Extracting lists and keywords from documents
Normalizing, translating, and cleaning data

The pydoxtools library is designed to be user-friendly and accessible to users with varying levels of programming expertise. While no programming knowledge is required to use the library, users with more advanced programming skills can take advantage of its more advanced features and customize it to their specific needs.

However, it is important to note that the pydoxtools library may not be suitable for users who require highly specialized document handling functionalities or those who work with very large datasets. Additionally, while the library is designed to work with various AI models, users may still encounter compatibility issues with certain models or configurations.

Conclusion

Pydoxtools is a powerful and user-friendly Python library that makes it easy to harness the power of AI for document processing and information retrieval. Whether you are new to programming or an experienced developer, Pydoxtools can help you streamline your projects and achieve your goals. Give it a try today and experience the benefits for yourself!

Get more information under the following links: