Polyweave
Repositories
1
 Projects Packages

@polyweave/researcher (0.1.1)

Published 2026-07-10 19:55:29 +00:00 by Dvorak

Installation

@polyweave:registry=https://hub.kl1.tenere.ai/api/packages/Polyweave/npm/
npm install @polyweave/researcher@0.1.1
"@polyweave/researcher": "0.1.1"

About this package

@polyweave/researcher

Autonomous research agent that queries arXiv and adds relevant papers to Encyclopedia when questions cannot be answered internally.

Features

  • Smart Relevance Filtering: Checks paper abstracts first, only downloads PDFs for relevant papers
  • Abstract-First Strategy: Fast rejection of irrelevant papers without downloading
  • Full-Text Verification: Downloads and checks full PDF for papers that pass abstract screening
  • Automatic Ingestion: Adds relevant papers to Encyclopedia for question re-resolution
  • Continuous Monitoring: Watches for open questions and automatically researches them
  • Metrics Tracking: Monitors papers found, evaluated, added, and rejected

Installation

npm install @polyweave/researcher

Optional: PDF Text Extraction

If you want PDF text extraction capabilities (for full-text relevance checking), install pdf-parse:

npm install pdf-parse

Without pdf-parse, the researcher will still work but will only use abstracts for relevance checking.

Quick Start

import { createResearcher } from '@polyweave/researcher';
import { createEncyclopediaMaintainer } from '@polyweave/encyclopedia';

// Create your Encyclopedia maintainer
const maintainer = createEncyclopediaMaintainer({
  stores: myStores,
  llmRunner: myLLM
});

// Create researcher
const researcher = createResearcher({
  maintainer,
  stateManager: maintainer.stateManager,
  category: 'cs.AI', // Focus on AI papers
  maxPapersPerQuestion: 5
});

// Listen for document ingestion events
researcher.subscribe((event) => {
  if (event.type === 'researcher:document_ready') {
    // Ingest into Encyclopedia
    console.log('New paper:', event.content.docId);
  }
});

// Start autonomous research
researcher.start();

How It Works

  1. Find Open Questions: Monitors Encyclopedia state for questions that are:

    • Still open (question_open)
    • Have completed internal searches (question_searches_resolved)
    • Not yet resolved (!question_resolved)

  2. Query arXiv: Searches arXiv for papers matching the question text

  3. Two-Stage Relevance Check:

    • Stage 1 (Abstract): Fast keyword/semantic matching on title + abstract
    • Stage 2 (Full Text): Downloads PDF and checks first 8KB of text

  4. Ingest Relevant Papers: Adds papers that pass both checks to Encyclopedia

  5. Trigger Re-Resolution: Marks question for re-evaluation with new documents

Configuration

const researcher = createResearcher({
  // ArXiv options
  category: 'cs.CL',           // Limit to specific category
  maxPapersPerQuestion: 10,    // Max papers to fetch per question
  sortBy: 'relevance',         // relevance | lastUpdatedDate | submittedDate
  
  // Relevance options
  minRelevanceScore: 0.4,      // Threshold for relevance (0-1)
  useLLM: true,                // Use LLM for semantic relevance
  llmRunner: async (prompt) => { /* your LLM */ },
  
  // Polling options
  pollInterval: 60000,         // Check for questions every 60s
  enableContinuous: true       // Keep running continuously
});

Events

The researcher emits events for monitoring:

researcher.subscribe((event) => {
  switch (event.type) {
    case 'researcher:started':
      console.log('Researcher started');
      break;
      
    case 'researcher:questions_found':
      console.log(`Found ${event.content.count} questions to research`);
      break;
      
    case 'researcher:papers_found':
      console.log(`Found ${event.content.count} papers`);
      break;
      
    case 'researcher:paper_rejected':
      console.log(`Rejected paper: ${event.content.reason}`);
      break;
      
    case 'researcher:document_added':
      console.log(`Added document: ${event.content.docId}`);
      break;
  }
});

Architecture

Encyclopedia State Manager
    ↓ (monitors for open questions)
Researcher
    ↓ (queries arXiv API)
arXiv Search Results
    ↓ (abstract relevance check)
Filtered Papers
    ↓ (PDF download + full-text check)
Relevant Papers
    ↓ (extract text)
Encyclopedia Ingestion
    ↓ (triggers)
Question Re-Resolution

License

MIT

API

createResearcher(options)

Parameter Type Default Description
options.encyclopedia object required Encyclopedia maintainer instance
options.llmRunner function required LLM invocation function
options.pollIntervalMs number 60000 Poll interval for open questions
options.maxResults number 10 Max arXiv results per query

Returns a researcher duplex with source and sink.

Tests

npm test

Covers researcher construction and arXiv query parsing.

Requirements

Node.js 18 or later. ESM only.

License

MIT

Dependencies

Dependencies

ID Version
@polyweave/core *
@polyweave/durability *
@polyweave/encyclopedia *
@polyweave/state *

Development Dependencies

ID Version
@rigor/core *

Keywords

research arxiv pdf encyclopedia autonomous
Details
npm
2026-07-10 19:55:29 +00:00
5
John Dvorak
SEE LICENSE IN LICENSE
latest
21 KiB
Assets (1)
researcher-0.1.1.tgz 21 KiB
Versions (1) View all
0.1.1 2026-07-10