python - Howto use scrapy to crawl a website which hides the url as href="javascript:;" in the next button -

-

i learning python , scrapy lately. googled , searched around few days, don't seem find instruction on how crawl multiple pages on website hidden urls - <a href="javascript:;". each page contains 20 listings, each time click on ">>" button, load next 20 items. can't figure out how find actual urls, below source code reference. pointers , appreciated.

visiting site web-browser , activated web-developer-tools (the following screenshots made firefox , add-on firebug) should able analyze network requests , responses. show sites pagination buttons send requests following:

enter image description here

so url seems be:

http://rent.591.com.hk/?m=home&c=search&a=rslist&type=1&shtype=list&p=2&searchtype=1

but it's not normal request. it's xmlhttprequest. indicated unter header tab. , response in json:

enter image description here

so don't need grab data complicated nested html-structures, can directly json dict.

i ended scrapy code (with room improvement):

import scrapy import json  class rentobject(scrapy.item):     address = scrapy.field()     purpose = scrapy.field()     # add more fields needed  class scrapespider(scrapy.spider):      name = "rent_hk"     allowed_domains = ['591.com.hk']     start_urls = ['http://rent.591.com.hk/?hl=en-us#list' ]      page_number = 0     page_num_max = 5 # test purposes grab 5 pages      def parse(self, response):          if 'page_number' in response.meta:             result_dict = json.loads(response.body)  # data dict             object in result_dict['items']:                 ro = rentobject()                 ro['address'] = object['address']                 ro['purpose'] = object['purpose']                 yield ro          # make request (next page) json data         self.page_number += 1          payload = {             'm': 'home',             'c': 'search',             'a': 'rslist',             'type': '1',             'p': str(self.page_number),             'searchtype': '1'         }          if self.page_number < self.page_num_max:             request = scrapy.formrequest(url='http://rent.591.com.hk/',                                          method='get',                                          formdata=payload,                                          headers={'referer': 'http://rent.591.com.hk/?hl=en-us',                                                   'x-requested-with': 'xmlhttprequest'},                                          callback=self.parse)             request.meta['page_number'] = self.page_number             yield request

the site not easy scrapy beginner - compiled detailed answer.


Comments

Post a Comment

Popular posts from this blog

Java 3D LWJGL collision -

-
i making 3d java game lwjgl library, , wondering how add collision detection, player not go through models. i using obj models. here objloader class, loads models: package renderengine; import java.io.bufferedreader; import java.io.file; import java.io.filenotfoundexception; import java.io.filereader; import java.util.arraylist; import java.util.list; import models.rawmodel; import org.lwjgl.util.vector.vector2f; import org.lwjgl.util.vector.vector3f; public class objloader { public static rawmodel loadobjmodel(string filename, loader loader){ filereader fr = null; try { fr = new filereader(new file("res/"+filename+".obj")); } catch (filenotfoundexception e) { system.err.println("couldn't load file!"); e.printstacktrace(); } bufferedreader reader = new bufferedreader(fr); string line; list<vector3f> vertices = new arraylist<vector3f...
Read more

spring - SubProtocolWebSocketHandler - No handlers -

-
i have ugly error during deploying spring app on jboss. 18:11:16,025 error [org.apache.catalina.core.containerbase.[jboss.web].[default-host].[/consumer]] (msc service thread 1-7) exception sending context initialized event listener instance of class org.springframework.web.context.contextloaderlistener: org.springframework.context.applicationcontextexception: failed start bean 'subprotocolwebsockethandler'; nested exception java.lang.illegalargumentexception: no handlers @ org.springframework.context.support.defaultlifecycleprocessor.dostart(defaultlifecycleprocessor.java:176) [spring-context-4.1.3.release.jar:4.1.3.release] @ org.springframework.context.support.defaultlifecycleprocessor.access$200(defaultlifecycleprocessor.java:51) [spring-context-4.1.3.release.jar:4.1.3.release] @ org.springframework.context.support.defaultlifecycleprocessor$lifecyclegroup.start(defaultlifecycleprocessor.java:346) [spring-context-4.1.3.release.jar:4.1.3.release] @ org.s...
Read more

methods - python can't use function in submodule -

-
my folder structure this: pythonstuff/ program.py modulea/ __init__.py foo.py bar.py here's code bar.py : def hello(): print("hello, world!") here's code program.py : #!/usr/bin/env python3 modulea import bar bar.hello() i run $ python3 program.py somehow error: file "program.py", line 3, in <module> bar.hello() attributeerror: 'module' object has no attribute 'hello' edit: __init__.py file empty. edit2: after trying realized had bar.py in root directory contained hello() method. bar.py in modulea/ directory empty. add it import os __all__ = [] module in os.listdir(os.path.dirname(__file__)): if module != '__init__.py' , module[-3:] == '.py': __all__.append(module[:-3]) the init .py files required make python treat directories containing packages; done prevent directories common name, such string, unintentionally hiding v...
Read more