Would there be any issues with this?

[Scott]

I want to fill up my tutorial site's database with tutorials (just the links to them and a title). So I was thinking about creating a PHP script which could sort of "crawl" large tutorial sites and index the URLs and titles from those sites in my own database.

Would there be any copyright issues or anything by doing this?
_________________
Tutorial Management Script - Version 1.4 Released
TutorialToday - Up and running, submit your tutorials!
Linux Tutorials - Coming Soon

[LP-SolidRaven]

Yes and no, it depends on tutorial per tutorial basis I guess.
_________________

[Scott]

How so? Would this not be like a search engine crawling pages on the internet and storing them?
_________________
Tutorial Management Script - Version 1.4 Released
TutorialToday - Up and running, submit your tutorials!
Linux Tutorials - Coming Soon

[LP-SolidRaven]

Certain sites don't allow direct linking of content or only allow certain other sites to link to them. They could ask you to remove the links. Though I think that the chances of that happening are small.
_________________

[Scott]

Now another question in regards to this topic, is it possible to find the URL a site redirects to in PHP?
_________________
Tutorial Management Script - Version 1.4 Released
TutorialToday - Up and running, submit your tutorials!
Linux Tutorials - Coming Soon

[LP-SolidRaven]

Yes, is it a http or meta redirect?
_________________

[Scott]

I believe it is using header("Location : ********"). Here is an example (although this is my site, I am looking to find it on external sites)

http://www.tutorialtoday.com/t.....it_in_PHP/

I am looking to find the URL of the tutorial it will go to, which will end up being:

http://www.tutorialtoday.com/read_tutorial/65/
_________________
Tutorial Management Script - Version 1.4 Released
TutorialToday - Up and running, submit your tutorials!
Linux Tutorials - Coming Soon

[LP-SolidRaven]

You should use normal sockets for this one. Cause then you can simply read the http headers.
_________________

[Scott]

Ok well I've been messing around with fsock and LLP people seems to block it by giving a 400 error but how do you read the headers after you have opened a site?
_________________
Tutorial Management Script - Version 1.4 Released
TutorialToday - Up and running, submit your tutorials!
Linux Tutorials - Coming Soon

[LP-SolidRaven]

Send a regular http request, like: http://pastebin.com/f685ee30b (pastebin cause of security mod of forum)
each line should end with a line break \r\n.
_________________

[krt]

Use cURL with the followlocation option on. Much simpler. cURL is also simpler when you have to deal with security, SSL, proxies and what not. This abridged list of Curl options lists the other options you may be interested in if you want to deal with what I said above or other things - POST data, headers, cookies, user agent spoofing etc.

Example:

[Scott]

Thanks for the help from both of you. I already wrote the script before I saw your post about cURL, I just used fsockopen() as SolidRaven was talking about and then used preg_match_all() to check the headers for the location.

I was thinking the same thing that they wouldn't mind, I would personally prefer if people automatically index my tutorials on their site.
_________________
Tutorial Management Script - Version 1.4 Released
TutorialToday - Up and running, submit your tutorials!
Linux Tutorials - Coming Soon

[LP-SolidRaven]

The only problem with that krt is that cURL isn't installed everywhere.
_________________

[ClickFanatic]

cURL is really useful. If it's not installed I would demand it!
_________________
Captain Jell-O Buster from the Future
[img]http://feeds.feedburner.com/sparepencil.1.gif[/img]