ScrapperMin

ScrapperMin a Web Automation Language

ScrapperMin Ideas and Development

John Kenedy 2017-04-21


ScrapperMin is built with the purpose of easy changing of Web Automation process whenever target website changes in a lightweight and minimalistic piece of software.
Website in the internet often get enhanced, modified, changes of template, themes, changing of CMS, or custom built solution that is modified. A specially coded software such as written in .NET, Java, Python or other languages, can do as much as ScrapperMin when dealing with Web Automation, such as getting page, extracting information or posting information, but one drawback of coded software is it needs to be compiled using developer tools, or interpreted using developer tools which requires installation, configuration, downloading dependencies, etc. Using these tools require more knowledge about the language, the configuration and dependencies often changes too which requires time and effort to modify existing code to suit for newer version of dependencies.

A specially coded software will be able to scrape a website or automate a website, however when website changes (which often happens), the software will no longer works and software developer will need to redo the cycle of installing, configuring developer tools, upgrading dependencies, modifying code to suit with newer version of dependencies and only then change the actual code of the program in order to match with newer version of the website and compile it into a new software version. Such cycle of software development is tedious and not everyone wishes to tackle these issues especially some website changes once in a while and developer might already bought a new PC since last update of the crawler software, and then developer requires to install the developer tools, configure it before being able to write the actual program to interact with the new website.

ScrapperMin tackles these issues by centralizing all Web Automation works into a package of code with built-in function just for doing Web Automation purposes, to code the automation, developer only need a notepad or default IDE and write script which uses built-in or third party functions to do web automation. When website changes, developer only need to modify the text file which contains the script.

ScrapperMin has been developed for more than 4 years and the development still continues as more features are added. ScrapperMin has been used by many projects such as a website that automatically upload file to file hosting services (Kumpulbagi, Zippyshare, Tusfiles, etc) which the uploading process is written in text file and being interpreted on the fly. ScrapperMin also has been used by EDMW Social Match project which scrape information from Hardwarezone website to accumulate forum post and notify any user when his or her comment get quoted or the nick is mentioned.

ScrapperMin is not limited into what it can do now, as third party developer can easily develop their own plugin and ScrapperMin can easily call the functions when needed by the script. Aside of that, ScrapperMin can be integrated with a custom built software as dependency, such software can leverage the power of built-in Multi-Threading feature in order to scrape many web pages at a single time in order to automate process faster.