SmartPartCollector (SPC)
- Agat Support
- Yoav Crombie
Service Purpose: A whole BL for the hiper-global project
Preparations: Prepare SmartPartEntities DB alone with our current AP DB and GW DB.
SmartPartEntities DB includes a table called MarketParts which includes the items to look for
Installation
Copy paste the desired version \\fs\Build Versions\Production\SmartPartCollector or \\fs\Build Versions\Test\SmartPartCollectortor to C:\Agat on your machine
Configuration
ConnectionStrings.config includes 3 connections strings: AccessPortalEntities, BusinessGPTGateway and something specific to this product - SmartPartEntities
Open C:\Agat\BGFileLoader\Configuration\ApplicationSettings.config and configure the following:
Key,IV- as usual
GoogleApiKey, GoogleSearchEngineId - Values in order to connect Google API for searches
RunningIntervalMinutes - How often to scan the table of hiper-global, default 1440 minutes, once a day
DoneIntervalMinutes- How often to check for done records in contnetList and then continue the flow. Default 60.
Installing the service
Open cmd as administrator, go to the service installation folder, and run the following command:
>SmartPartCollectortor.exe install
To uninstall the service run the command:
>SmartPartCollectortor.exe remove
Note: Can run in console mode (not as a Windows Service) with the command:
>SmartPartCollectortor.exe console
How it works
Flow 1 - Get Web search results -every day
Scan the MarketParts table, and for each item (CompanyPart=true), get from Google the top 10 exact results of the part number and then the top 10 results of the product description. For each link, get its content using our embedding queue
Flow 2 Get structured data - every 60 mins
For each ‘Done’ record in our ContentList (including, but not only the ones from Step 1), send the content to our GW for AI analysis which extracts info about the product such as price from the content.
Then this info will be saved to MarketParts as CompanyPart=false
Flow 3- Search routine
The Collector includes a search routine that processes each company part marked as High in the search priority field and performs a smart search in the marketplace DB.
It then saves the results in the BGT search tables.