Using NodeJS, print all HTTPS Requests a given URL performs

Closed Posted 6 years ago Paid on delivery
Closed Paid on delivery

Whenever I access a URL that is always given to me (let's call it "search URL"), it makes several subsequent HTTPS requests. Between these requests, there are 4 ones that I need (let's call them "desired URLs").

Given any "search URL" I can easily get the "desired URLs" via Chrome DevTools (Network/XHR), but I need it in nodejs.

So I need a script that outputs me ([login to view URL]) the "desired URLs" when I give it a "search URL".

I had a working script, but it broke in a recent site update (I will give you a printscreen of the new site to make sure we are on the same page).

Here is the NON-WORKING example:

//===========================================================

//===========================================================

// OLD SITE WORKING [login to view URL]

const phantom = require('phantom');

// searchURLs TO TEST THE SCRIPT (I ALWAYS HAVE THESE URLs)

var searchURL1 = "*";

var searchURL2 = "*";

var searchURL3 = "*";

var searchURL4 = "*";

// RANDOMLY PICK ONE OF THE ABOVE

var randomURLNumber= [login to view URL](([login to view URL]() * 4) + 1);

if (randomURLNumber == 1){

getMMjsonURLs(searchURL1);

} else { if (randomURLNumber == 2){

getMMjsonURLs(searchURL2);

} else {if (randomURLNumber == 3){

getMMjsonURLs(searchURL3);

} else {getMMjsonURLs(searchURL4);

};

};

};

// PRINT THE desiredURLs

async function getMMjsonURLs(url) {

const instance = await [login to view URL]();

const page = await [login to view URL]();

await [login to view URL]('onResourceRequested', function (requestData) {

//// PRINT ALL REQUESTS

//[login to view URL]("ALL_REQUESTS: " + [login to view URL]);

//// FILTER (should get 4 URLs similtar to: **)

var desiredURL = [login to view URL]('airline');

if (desiredURL != null) {

//// PRINT FILTERED REQUESTS

[login to view URL]("**desiredURL: " + [login to view URL]);

};

});

const status = await [login to view URL](url);

await [login to view URL]();

};

//===========================================================

//===========================================================

Express JS JavaScript Node.js Web Crawling Web Scraping

Project ID: #16515152

About the project

5 proposals Remote project Active 5 years ago

5 freelancers are bidding on average R$41 for this job

kkc264043kkc

I will use Phantomjs tool which will give you all the request is made. from that we can extract the desired URLs. I will do it in the node.js

R$50 BRL in 1 day
(44 Reviews)
6.0
gabrielgitonga3

This letter is to express my interest in the job posted on your website for an experienced, detailed-oriented, front-end web developer. As you'll see, I have six years of hands-on experience efficiently coding websites More

R$52 BRL in 10 days
(0 Reviews)
0.0
dejason67

Please pick me

R$30 BRL in 1 day
(0 Reviews)
0.0