Python - How To Add Response From Scrapy.request From Yield Into An Array

I am trying to collect populations of different sovereigns from wiki list of sovereigns and add them to an array on each response. In the code below allList should have a list of d

Solution 1:

It seems that this is exactly what pipelines.py is set up for in Scrapy. The issue is that the callback responses are not received in order or fast enough for them to be stored in a separate array. Once the responses are received they go through the Pipelines.py where they can be processed and stored. Below is Scrapy documentation on usage of pipeline. Very useful tool! [https://doc.scrapy.org/en/latest/topics/item-pipeline.html][1]