ScrapperMin

ScrapperMin a Web Automation Language

Important Headers when Doing Scraping

John Kenedy 2017-05-10 When doing scraping, for example getting links and try to download from links. You will encounter sometimes download is served via GET and sometimes POST depends on the website. Some websites serves download using POST because they want to make sure that you didn't accidentally download the same file twice as browser open using GET when it is relaunched (saved last browsed page), because of this POST is used with sometimes secured with extra tokens as POST parameters in order for the download to success. Some websites also has expiry for a download link,...

How to build Plugin for ScrapperMin

John Kenedy 2017-04-21 ScrapperMin can load any functions or methods you have created natively in JAVA (jar file) using EX_MethodName(param1, param2, param3) The jar file must be placed in /sdcard/ScrapperMin/SMPlugins/sample.jar The MethodName must be unique among all jar files in that directory, or the first MethodName that it encountered will be called instead, the namespace does not matter since ScrapperMin will load all classes in all namespaces however please note that the Method must be a static method that accept parameter in type String[] or String...

How to Set Parameter to run using console version (non GUI)

John Kenedy 2017-04-21 For example we have a sample script like below with filename HWZ.txt SET('PG', WC_GetPage('http://forums.hardwarezone.com.sg', '')); SET('PS', WC_GetPostStringRaw('http://forums.hardwarezone.com.sg', GET('PG'), '1', FORMAT('vb_login_username={PARAM0}&vb_login_md5password_utf={0}&vb_login_md5password={1}', SO_GetMd5Hash('{PARAM1}'),SO_GetMd5Hash('{PARAM2}')), '')); SET('PG2', WC_PostPageComplete(WC_GetLastActionUrl(), GET('PS'), 'http://forums.hardwarezone.com.sg')); IF(GET('PG2'),...

ScrapperMin Ideas and Development

John Kenedy 2017-04-21 ScrapperMin is built with the purpose of easy changing of Web Automation process whenever target website changes in a lightweight and minimalistic piece of software. Website in the internet often get enhanced, modified, changes of template, themes, changing of CMS, or custom built solution that is modified. A specially coded software such as written in .NET, Java, Python or other languages, can do as much as ScrapperMin when dealing with Web Automation, such as getting page, extracting information or posting information, but one drawback of coded...