Half of all organisations believe web scraping technology is too complex to implement


Oxylabs’ white paper, The Growing Importance of External Data in the Retail and E-Commerce Industry, reveals that the main reason organisations have chosen not to adopt automated external data acquisition is because web scraping technology is seen as too complicated (cited by 50% of respondents).

​Furthermore, 44% of respondents in the retail and e-commerce sector listed the legal complications around web scraping as another significant factor. While attempts have been made to educate organisations about the legal nuances of web scraping, there is still a need for the message to be communicated further across the industry.

The report also went on to highlight that 41% of respondents claimed their needs were already fully covered by other means of data collection, such as third-party databases. Purchasing third-party databases may redistribute some of the weight of legal compliance on the provider instead of the receiver. 

Finally, 38% of respondents listed budget constraints and 35% listed lack of talent as reasons for not implementing web scraping. A large part of budget constraint issues can be mitigated by using out-of-the-box web scraping solutions. Managing these requires very few in-house employees but still promises effective data acquisition. 

Julius Černiauskas Chief Executive Officer at Oxylabs said: “At Oxylabs, the role of data has become even more significant in the last couple of years, with fewer businesses now relying exclusively on traditional sources to acquire this information. Therefore, as external data acquisition becomes useful, it is expected that organisations will increase their scraping budgets”.

Denas Grybauskas, the Head of Legal at Oxylabs followed up with :“However, the current web scraping landscape is burdened with a plethora of legal technicalities. A lack of industry regulation and ethics agreements makes the situation hugely complex. While many companies utilize web scraping on a daily basis, the best thing all organisations can do to avoid legal problems, other than consulting with a legal professional, is to only extract publicly available data. Trying to scrape other forms of data might seem harmless, but they are protected by a number of different laws which could lead to legal difficulties further down the line.”

Data strategies in 2022 and beyond

Over the next 12 months, 52% of e-commerce and retail organisations plan to collect more external data, with half looking into ways to collect more real-time data.

Currently, the most common reason retail and e-commerce organisations are using external data is to predict market trends (43% of businesses). The report also highlighted that 38% are benchmarking against competition and optimising customer service to help elevate the quality of service provided. Organisations view this approach as incredibly powerful, as it allows business development processes to gain a stronger foundation and build better roadmaps.

Černiauskas concluded: “It is clear to see that alternative data is playing, and will continue to play, an increasingly important role in the retail market. Whilst there are numerous incentives to acquire such data, the complex processes involved mean some companies are yet to make the most of it. With the right approaches, however, there is every possibility that businesses can take full advantage of web scraping.”

Oxylabs, in cooperation with Censuswide, conducted a survey of 252 senior data decision-makers across the UK’s thriving finance sector, including Directors of Engineering, CTOs, VPs of Engineering, Chief Data Officers, Heads of Business Intelligence and Heads of Data & Analytics, were questioned on their current approach to data management. Respondents were selected across a broad spectrum of financial service companies, including retail, commercial and private banks, investment and wealth management firms.

Add a Comment

No messages on this article yet

Editorial: +44 (0)1892 536363
Publisher: +44 (0)208 440 0372
Subscribe FREE to the weekly E-newsletter