import { v4 as uuidv4 } from 'uuid'; import { Networking } from '../../../../Network'; import { BaseTool } from './BaseTool'; import { Observation } from '../types/types'; import { ParametersType, ToolInfo } from '../types/tool_types'; const websiteInfoScraperToolParams = [ { name: 'urls', type: 'string[]', description: 'The URLs of the websites to scrape', required: true, max_inputs: 3, }, ] as const; type WebsiteInfoScraperToolParamsType = typeof websiteInfoScraperToolParams; const websiteInfoScraperToolInfo: ToolInfo = { name: 'websiteInfoScraper', description: 'Scrape detailed information from specific websites relevant to the user query. Returns the text content of the webpages for further analysis and grounding.', citationRules: ` Your task is to provide a comprehensive response to the user's prompt using the content scraped from relevant websites. Ensure you follow these guidelines for structuring your response: 1. Grounded Text Tag Structure: - Wrap all text derived from the scraped website(s) in tags. - **Do not include non-sourced information** in tags. - Use a single tag for content derived from a single website. If citing multiple websites, create new tags for each. - Ensure each tag has a citation index corresponding to the scraped URL. 2. Citation Tag Structure: - Create a tag for each distinct piece of information used from the website(s). - Each tag must reference a URL chunk using the chunk_id attribute. - For URL-based citations, leave the citation content empty, but reference the chunk_id and type as 'url'. 3. Structural Integrity Checks: - Ensure all opening and closing tags are matched properly. - Verify that all citation_index attributes in tags correspond to valid citations. - Do not over-cite—cite only the most relevant parts of the websites. Example Usage: Based on data from the World Bank, economic growth has stabilized in recent years, following a surge in investments. According to information retrieved from the International Monetary Fund, the inflation rate has been gradually decreasing since 2020. What are the long-term economic impacts of increased investments on GDP? How might inflation trends affect future monetary policy? Are there additional factors that could influence economic growth beyond investments and inflation? ***NOTE***: Ensure that the response is structured correctly and adheres to the guidelines provided. Also, if needed/possible, cite multiple websites to provide a comprehensive response. `, parameterRules: websiteInfoScraperToolParams, }; export class WebsiteInfoScraperTool extends BaseTool { private _addLinkedUrlDoc: (url: string, id: string) => void; constructor(addLinkedUrlDoc: (url: string, id: string) => void) { super(websiteInfoScraperToolInfo); this._addLinkedUrlDoc = addLinkedUrlDoc; } async execute(args: ParametersType): Promise { const urls = args.urls; // Create an array of promises, each one handling a website scrape for a URL const scrapingPromises = urls.map(async url => { try { const { website_plain_text } = await Networking.PostToServer('/scrapeWebsite', { url }); const id = uuidv4(); this._addLinkedUrlDoc(url, id); return { type: 'text', text: `\n${website_plain_text}\n`, } as Observation; } catch (error) { console.log(error); return { type: 'text', text: `An error occurred while scraping the website: ${url}`, } as Observation; } }); // Wait for all scraping promises to resolve const results = await Promise.all(scrapingPromises); return results; } }