ScrapperMin

ScrapperMin a Web Automation Language

Introduction


ScrapperMin is a lightweight minimalistic Windows and Android app that allows software developer to quickly build web automation Robot or Crawler.
Robot or Crawler can be programmed to download internet files, pages, extract information with capability to login, posting comments or articles, download and upload files automatically.


Hundred of Functions


ScrapperMin includes hundred of pre-built functions to easily scrape information from the web, posting data or files using multipart or url encoded, String operations such as Regular Expressions matching, extracting HTML, File operations such as download or upload file, HTTP operations such as changing HTTP Headers, add or modify Cookies, Open Authentication, and Logic operations such as comparisons and looping.
The built-in language and functions is sufficient for most project, developer can build or use third party Plugin too.


Multi Threading


ScrapperMin includes multi-threading capability allows developer to programmatically open many web pages at once to accumulate data faster.
The built-in thread pool is capable to have initial state such as initial cookies or can be programmed to login if there is no cookies.


C Like Language


ScrapperMin parses and interpretes a ScrapperMin C-like language from text file to process the command as function to get pages or post data to internet.
Parameters can be passed to the text file such as user name or password used for login into website before actual scraping starts


Command Line Compatible


ScrapperMin has console and IDE version (.NET). Console version is used to let third party program (non .NET or Java) to run exe (no GUI) by passing script filename and other arguments for crawling. Also can be run using Task Scheduler-like programs periodically with same arguments.
IDE version let developer build and test script before pass to production use. .NET or Java developer can easily reference .DLL or .JAR library and call using self built program.


Plugins Support


ScrapperMin supports plugin or extension built by third party developer. Third party functions .DLL files can be put in SMPlugins (Windows) folder and developer can use the function in ScrapperMin's script.
Android developer can develop plugin with dex .JAR file and placed in Plugins folder of the /sdcard/ScrapperMin/SMPlugins folder, the function is readily to be used by ScrapperMin's script.


Script Can be Compiled as APK


Android version of ScrapperMin can compile user script into APK, when run, APK will ask for parameters input and upon clicking on Run button, APK will do Web Automation process as written in the script.
Windows version support running using ScrapperMin console version.