Overview#

Description: An agent to research a given topic by doing analytical and syntopic reading, summarizing and extracting main points, connecting them together, and forming new knowledge outputs such as zettels or evergreen notes.

Lessons Learned#

  • ~~Document embeddings with transformers don’t seem to be too practical. Instead, go with a sentence embedding for the knowledgebase using note titles and search with main idea embeddings.
  • Dispatch table values need to the the function as an object, so without the parentheses. Otherwise they’ll be run when instantiated.

Project Notes#

Future Improvements#

Future improvements to the project should go here for easy reference. They should be a type of project note and link back to a Project page and any tickets that come from it.

Resources#

Project Plan#

Designs#

Task: Ingest input text Because we will be reusing the input text, store it in memory and make the embedding in pinecone(?) Task: Ingest existing knowledgebase Add all items to vector DB as embeddings for later search Strip out metadata Summarizer - summarizes text Main point/argument finder - finds the main points or arguments of the article Assertion finder - finds assertions of fact Knowledge generator - generates zettels (how? Just use main points? Expand on them?) Zettel connector - uses semantic similarity search to find related zettels and either add to them or use backlinking to connect them

Questions#

Summarizer#

The summarizer endpoint from Cohere has a limit of 100k characters. For a long text, how should we break it down? I have two medium-difficulty options I see, and 1 really easy one:

  1. EASY: Don’t summarize texts longer than 100k characters. That’s a problem for future me. One of my long RP articles with code and all (my Sentiment Analysis one) is just under 46k characters.
  2. Medium: Tokenize into sentences and do a “center crop” - assumes that least information-dense parts of an article are the ends of it, which isn’t always the case.
  3. Medium: Chunk text into sub-100k chunks, summarize them separately, then weave them back together. This can be costly for large texts and if done should warn the user.

Going to go with #1 for now, then create a Github issue for handling larger texts. Will probably warn the user and give some options?

This is due to using the Transformer architecture - it’s harder to carry long-term memory than a recurrent network (but transformers give us better outputs). What if we need context from outside documents? - Tomasso

Journal#